How LLM inference works and how to tune its latency

The article explains LLM inference, the process of running a trained model on a new prompt to generate a response. It splits inference into two phases: prefill, which processes the full prompt and determines time to first token, and decode, which generates the answer one token at a time and governs streaming speed. It identifies three latency measures a product team may track: time to first token, time between tokens, and total completion time. Drawing on two voice-agent builds, the author notes that most end-to-end delay came from stages outside the model call, such as turn detection and transcription, so each stage should be timed before tuning the model. The piece also argues that prompt and answer length are the first levers to adjust, ahead of changing hardware.