Separating LLM Prefill and Decode Phases Cuts Token Latency by 66%

Large language model inference involves two distinct workloads: a compute-heavy prefill phase that processes the entire prompt, and a latency-sensitive decode phase that generates tokens one at a time. When both phases run on the same hardware, a long-context prefill can stall all active decode streams on that engine, a problem known as head-of-line blocking. Prefill/decode disaggregation addresses this by routing each phase to dedicated server pools, preventing prefills from ever queuing ahead of decodes. A Go-based simulation demonstrated that splitting these pools reduced p99 inter-token latency from 88ms to 30ms, a 66% improvement, at the cost of a modest increase in time-to-first-token. This architectural pattern is already adopted in production serving systems such as DistServe, Splitwise, and vLLM with Mooncake.