How LLM Inference Engines Determine AI Response Speed, Explained Simply

A developer building an open-source AI code review tool has published a technical explainer on how large language model inference engines work under the hood. The piece breaks down LLM text generation into two phases: prefill, where the model processes the input prompt, and decode, where it produces output one token at a time in an autoregressive loop. Central to performance is the KV cache, which stores computed vectors for each token to avoid redundant calculations, but grows large quickly and consumes scarce GPU memory. Older serving systems worsened this by pre-reserving worst-case memory for every request, leading to both internal and external memory fragmentation. The article frames these inefficiencies as the root cause of why the same underlying model can feel dramatically faster or slower depending on the serving infrastructure running it.