Developer Benchmarks Local LLMs on Consumer GPUs, Picks Gemma 4 26B as Top Model

A developer has documented running large language models locally on two consumer-grade machines — a primary desktop with an AMD RX 6900XT and a secondary box with an NVIDIA GTX 1060 — using Ollama as the model runner. Several models were benchmarked for both throughput and output quality, with smaller models like Llama 3.2 generating tokens significantly faster but producing shallower, less accurate responses. Gemma 4 26B, a mixture-of-experts model quantized to Q4_K_M, scored 50/50 on a five-task quality suite and 99/100 on a harder follow-up, while sustaining around 17–18 tokens per second. To fit the model within the GPU's 16 GB VRAM, the developer enabled flash attention and an 8-bit KV cache to reduce memory overhead. The series will continue with topics including context-length trade-offs, prefill versus generation performance, and strategies for avoiding cold-start reload delays.