Why Memory Bandwidth, Not Model Size, Determines LLM Speed on Local Hardware

A software developer running a 27-billion-parameter quantized AI model on an Apple Mac Mini M4 found it generated only 5.6 tokens per second despite fitting entirely within the device's 24GB unified memory. The core issue was memory bandwidth: at roughly 120GB/s, the M4's theoretical ceiling for that model size is about 8 tokens/second, making the measured result 70% of the maximum possible. This ratio indicates a bandwidth-bound bottleneck, meaning reducing model size through lower quantization would yield less than 9% speed improvement while noticeably degrading output quality. The developer proposes a practical rule — if measured throughput divided by the bandwidth ceiling is 0.5 or higher, size-reduction strategies should be abandoned in favour of hardware upgrades or workload reassignment. The key takeaway is that confirming a model 'fits' in memory answers only the capacity question, leaving throughput — governed by a separate hardware resource — entirely unresolved.