Speculative Decoding in 2026: New Architectures and NVIDIA Guidelines Reshape LLM Inference

As of September 2026, speculative decoding (SD) has seen its most significant wave of innovation in two years, with five new research papers published on arXiv within 72 hours. The technique addresses a fundamental bottleneck in large language model inference, where weight-loading from memory — not compute — limits throughput to roughly 15–20 tokens per second on high-end GPU clusters. SD works by using a small draft model to propose candidate tokens, which a larger target model then verifies in a parallelized pass, reducing redundant memory loads without altering output distribution. Architecture evolution has progressed from EAGLE-3 through DFlash to the newer XPress framework, each iteration targeting limitations introduced by its predecessor. NVIDIA also released production co-design guidelines and a new benchmark standard, while community interest surged around running frontier-scale models on consumer Mac hardware using the technique.