How to Estimate GPU Memory Needs for AI Models Before You Deploy

A technical guide aimed at site reliability engineers explains how to systematically size GPU infrastructure for hosting AI models, rather than relying on guesswork. The method focuses on two main memory consumers: model weights, which are a fixed cost loaded once into VRAM, and the KV cache, which holds working memory for concurrent requests and directly determines throughput. Using Hugging Face's Qwen2.5-7B-Instruct-AWQ as a demonstration, the guide shows how quantization precision — fp16, int8, or 4-bit AWQ — dramatically changes a model's memory footprint, from roughly 15 GB down to under 6 GB for the same parameter count. The article notes that advertised weight sizes can differ from actual file sizes, and recommends verifying by summing model file sizes directly from the repository. The sizing process is described as a rule-of-thumb filter to narrow GPU family choices, with precise measurements recommended once the model is running via tools like vLLM.