GGUF, GPTQ, AWQ: A Practical Guide to LLM Quantization Formats

Running large language models locally has become more accessible thanks to three quantization formats — GGUF, GPTQ, and AWQ — each suited to different hardware setups. GGUF, developed by the llama.cpp team, allows models to split workloads between GPU and system RAM, making it ideal for consumer laptops and Apple Silicon devices even with limited VRAM. GPTQ targets dedicated GPU environments, using a calibration dataset to compress model weights to 4-bit integers for fast inference, but requires all data to fit entirely within VRAM. AWQ, the newest of the three, improves on GPTQ's accuracy by identifying and preserving roughly 1% of critical model weights during quantization, making it well-suited for enterprise tasks like complex reasoning and code generation. Choosing the right format depends on available hardware — GGUF for flexibility, GPTQ for GPU-bound production APIs, and AWQ where accuracy is the top priority.