How to Choose the Right Gemma 4 Quantization Without Running Out of VRAM

Running large language models locally requires careful quantization choices, as the popular advice to default to Q4 GGUF can fall short during long-context tasks. A model's memory footprint depends on parameters multiplied by bits per weight, with Q4_K_M (~7.2 GB for a 12B model) generally offering the best quality-to-size tradeoff. However, guides often overlook the KV cache, which grows linearly with context length and can more than double a model's total memory use at long sequences. Two practical levers — limiting context size to actual needs and quantizing the KV cache — can recover significant memory without the permanent quality loss that comes from dropping to Q3. Experts recommend exhausting these options before downgrading quantization, as falling to Q3 to gain headroom is a common but costly mistake.