Quantization Can Cut LLM Size by 75% With Minimal Quality Loss

Quantization is a technique that reduces the number of bits used to store neural network parameters, shrinking large language models to as little as one-quarter of their original size. This allows models that would otherwise require data-center-grade hardware to run on consumer GPUs or even laptops by cutting VRAM and RAM requirements significantly. However, reducing numerical precision introduces errors, so modern methods like GPTQ and AWQ use calibration datasets and activation-aware strategies to protect the most critical weights. GPTQ works layer-by-layer using second-order mathematical information to minimize rounding error, while AWQ identifies and preserves high-impact weights based on how activations flow through the model. Common file formats like GGUF, used by tools such as Ollama and LM Studio, support multiple quantization levels, letting users balance model size against output quality for their specific hardware.