Why AI Responses Pause Then Stream: The Two Phases Inside Every LLM Request

Every large language model request is processed in two distinct phases that rely on different parts of a GPU. In the first phase, called prefill, the entire user prompt is processed simultaneously by GPU cores to produce the first output token and generate a memory structure called the KV cache. The longer the input prompt, the longer this initial pause lasts — a metric known as time to first token (TTFT). In the second phase, called decode, the model generates one word at a time by repeatedly reading the KV cache and model weights from VRAM, which is why responses stream word by word rather than appearing all at once. This distinction explains why a faster GPU does not always improve response speed, since the decode phase is bottlenecked by memory bandwidth, not raw compute power.