Self-Hosting LLMs: Why GPU Memory Planning Goes Beyond Model Weights

Most tutorials for running large language models on a GPU focus only on model weight size, overlooking three other critical memory consumers: the KV cache, runtime activations, and memory fragmentation. The KV cache, which stores key and value tensors to avoid recomputing tokens, scales with context length and batch size, and can easily exceed weight memory under real traffic conditions. For a Llama-3-8B-class model in FP16, a single 8,192-token request consumes roughly 1GB of KV cache, meaning a batch of 16 concurrent requests at that length requires around 16GB for cache alone. Developers can reduce this pressure by limiting maximum context length and enabling KV-cache quantization, both supported by frameworks like vLLM and TGI. The article argues that accurate GPU memory budgeting requires calculating all four memory buckets before provisioning hardware, not just checking whether model weights fit.