LLM Inference Costs Are Driven by Memory Bandwidth, Not Compute FLOPs

Most discussions around large language model infrastructure costs focus on FLOPs and GPU compute, but the real bottleneck during inference is memory bandwidth — specifically the cost of moving the key-value cache in and out of GPU memory on every decode step. Unlike the prefill phase, which is genuinely compute-intensive, the token-generation (decode) phase involves minimal arithmetic but requires reading gigabytes of cached data repeatedly, leaving accelerators waiting on memory rather than computing. This means infrastructure decisions based on FLOPs-per-dollar metrics are systematically misaligned with actual decode-dominated workloads. The problem compounds with longer context windows, as a growing KV cache increases memory traffic for every subsequent token generated in a session. Optimizing for memory bandwidth — through prefill-decode disaggregation, right-sized batching, and context management — is the more economically sound approach for production LLM serving.