How to Size the KV Cache for Local LLMs Before It Runs Your GPU Out of Memory

A common cause of local LLM load failures is not the model weights but the key-value cache, which stores per-token attention tensors and grows linearly with context length. The cache size can be estimated from the number of layers, key/value heads, head dimension, and bytes per element, with grouped-query attention sharply reducing the footprint. Worked examples for two 7B models show that a 32K context can require roughly 1.75 GiB or 4 GiB of VRAM for the cache alone at f16 precision. Quantizing the cache saves less than the common rule of thumb suggests, because block-scaling overhead in formats like q8_0 and q4_0 reduces the savings slightly. The article notes that these figures exclude weights, compute buffers, and runtime overhead, which must be added when planning memory on consumer GPUs.