FSDP vs DDP: When Sharded Data Parallel Training Becomes Necessary

A DEV Community post explains when to choose sharded data parallelism (FSDP) over standard Distributed Data Parallel (DDP) for model training. With AdamW in mixed precision, DDP stops fitting on an 80 GB NVIDIA H100 at roughly 3.75 billion parameters once about 20 GB is reserved for activations. The article explains that DDP replicates the full model on every GPU and synchronizes gradients with all-reduce, while FSDP shards parameters, gradients and optimizer states, trading extra communication for lower memory use. Using about 16 bytes per parameter, the author works out per-GPU memory for 1B, 8B, 13B and 70B models across 8 GPUs. Under this arithmetic, only the 1B model fits under DDP, the 8B and 13B models fit under either FSDP strategy, and a 70B model cannot fit on an 80 GB GPU even when fully sharded. The figures are the author's own estimates and exclude activations and other overhead.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in