Medusa and Multi-Token Prediction Are Rewriting How LLMs Generate Text

Large language models traditionally generate text one token at a time, meaning producing 100 tokens effectively requires running the model 100 times sequentially. This autoregressive bottleneck means expensive hardware often spends more time moving model weights through memory than performing useful computation. Techniques like speculative decoding, Medusa-style multiple decoding heads, and multi-token prediction aim to break this constraint by predicting several future tokens simultaneously. In speculative decoding, a smaller draft model proposes multiple candidate tokens that the larger model then verifies in a single parallel pass, potentially accepting several tokens at once. These approaches are not merely speed optimizations but fundamentally alter the computational structure of text generation, making each expensive forward pass yield more output.