Speculative Decoding Can Speed Up LLM Inference by 20-50% Without Changing Output

Speculative decoding is a technique that accelerates large language model inference by generating multiple tokens per forward pass instead of one at a time. Standard autoregressive generation is bottlenecked by memory bandwidth, as a 70B model on an H100 GPU produces only one token every 30-50 milliseconds due to sequential dependencies. The method works in draft-verify cycles, where a lightweight draft mechanism proposes several candidate tokens and the target model verifies all of them in a single forward pass. Accepted tokens are statistically identical to those produced by standard decoding, meaning output quality is preserved. The approach is supported across popular inference frameworks including llama.cpp, vLLM, SGLang, and TensorRT-LLM, with multiple implementation variants available as of 2026.