Speculative Decoding Explained: When It Speeds Up LLMs and When It Doesn't

Speculative decoding is a widely discussed technique for accelerating large language model (LLM) inference, where a smaller draft model generates token candidates that a larger target model then verifies in a single forward pass. Contrary to common concern, the method does not amplify hallucinations — verification is token-by-token, and any incorrect token causes the sequence to be truncated and regenerated from that point, making output mathematically equivalent to standard autoregressive generation. However, the compute cost question is more nuanced: while a full draft hit can save significant target-model compute, a full miss results in more total computation than standard generation would have required. The real-world benefit depends on the draft acceptance rate, the size ratio between draft and target models, and the draft length chosen. Speculative decoding is therefore not a guaranteed speedup but a conditional one — it pays off only when acceptance rates are consistently high enough to offset the overhead of running the draft model.