How to Run GGUF Models Locally Using Ollama, llama.cpp, and vLLM

GGUF is a self-contained file format for quantised language models, originally developed by llama.cpp, that bundles weights, tokenizer, and metadata into a single file. Quantisation compresses model weights from 16-bit floats to as low as 4-bit integers, allowing a 9B-parameter model to shrink from roughly 18 GB to around 5.5 GB and run on consumer GPUs or even CPUs. Tools like Ollama, llama.cpp, LM Studio, and Jan all support the format natively; Ollama can pull GGUF files directly from Hugging Face since 2024, while llama.cpp's llama-server offers finer control over GPU layer offloading. The Q4_K_M quantisation level is widely recommended as the best balance between quality and file size for 7–14B models, while Q8_0 suits users with ample VRAM who want near-lossless output. Choosing the right quantisation depends on available VRAM, with options ranging from Q8_0 for high-end setups down to Q2_K for very constrained hardware.