SShortSingh.
Back to feed

FSDP vs DDP: When Sharded Data Parallel Training Becomes Necessary

0
·5 views

A DEV Community post explains when to choose sharded data parallelism (FSDP) over standard Distributed Data Parallel (DDP) for model training. With AdamW in mixed precision, DDP stops fitting on an 80 GB NVIDIA H100 at roughly 3.75 billion parameters once about 20 GB is reserved for activations. The article explains that DDP replicates the full model on every GPU and synchronizes gradients with all-reduce, while FSDP shards parameters, gradients and optimizer states, trading extra communication for lower memory use. Using about 16 bytes per parameter, the author works out per-GPU memory for 1B, 8B, 13B and 70B models across 8 GPUs. Under this arithmetic, only the 1B model fits under DDP, the 8B and 13B models fit under either FSDP strategy, and a 70B model cannot fit on an 80 GB GPU even when fully sharded. The figures are the author's own estimates and exclude activations and other overhead.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

CVE-2026-63411: Code Injection Flaw in Jane PHP OpenAPI Generator Enables Remote Code Execution

A critical vulnerability tracked as CVE-2026-63411 (GHSA-xwcw-j52r-34x4) affects the Jane PHP code generation framework's open-api-3 component before version 7.12.0. The generator inserts parameter names and descriptions from OpenAPI specifications into PHPDoc comment blocks without sanitizing them. An attacker can embed a comment-closing sequence followed by PHP code, allowing injected code to run when the generated client class is instantiated. The flaw is rated 9.3 under CVSS v4.0 and is classified as code injection and improper escaping. Users are advised to upgrade to version 7.12.0 or later, regenerate existing SDKs, and validate OpenAPI specifications in CI/CD pipelines.

0
ProgrammingDEV Community ·

Spain's Key Tax and Visa Rules for Developers Moving There: Residency, Beckham Law, Modelo 720

A Spain-based author has published a free reference guide covering five rules that developers often misunderstand when relocating. The digital nomad visa requires income of at least 200% of the Spanish minimum wage, employment history, and limits on Spanish client work for freelancers. Tax residency applies after more than 183 days in Spain, and the first year is not split, so worldwide income for that year is declared. The Beckham law offers a flat 24% rate on employment income for six years, but applications must be filed within six months and the regime is not always cheaper. The article also explains the Modelo 720 information return for foreign assets exceeding €50,000 in any single category, with each category assessed separately.

0
ProgrammingDEV Community ·

Five research-backed patterns for orchestrating coding agent teams

An article on DEV Community outlines five patterns for running teams of AI coding agents, drawing on published research to judge when they outperform a single agent. Its first recommendation is to separate building from verification, since models tend to be poor judges of their own output and benefit from a fresh reviewer. Parallel work splits cleanly by domain, such as frontend, backend, and docs, but overlapping edits and tightly dependent steps can make multiple agents perform worse. A best-of-N approach, where several attempts run in parallel and an automated test picks the winner, is shown to raise solve rates when reliable checks exist. For tasks lacking clear tests, such as architecture or visual design, a council of several models can propose ideas, vote anonymously, and merge the strongest answers.

0
ProgrammingDEV Community ·

OpenLIT Brings OpenTelemetry Tracing to LLM Apps With Minimal Code Changes

The article explains how OpenLIT, an Apache 2.0 open-source platform, uses OpenTelemetry to record each LLM call as a trace span with its model, token count and cost. It describes SDKs for Python, TypeScript and Go, and notes that traces follow the OpenTelemetry GenAI semantic conventions and can be exported over standard OTLP. Three instrumentation approaches are presented, from a single start-up call to a CLI wrapper, depending on whether application code can be modified. The piece also flags privacy concerns, since captured prompts may contain customer data, and points to settings that disable content capture, truncate it, or redact personal information. Facts are drawn from the project's README and documentation as read in November 2026.