A Simple Formula to Calculate Your CPU's Local AI Inference Speed Limit

Running large language models locally on a CPU is often slow, and the key bottleneck during text generation is memory bandwidth, not processing power. For every token generated, the CPU must read all active model weights from RAM into cache, meaning decode speed is capped by how fast data crosses the memory bus. A straightforward ceiling can be calculated by dividing a machine's memory bandwidth — derivable from its DDR standard — by the model's file size in gigabytes. For example, a 70B parameter model at Q4_K_M quantization cannot exceed roughly two tokens per second on a dual-channel DDR5 desktop, regardless of CPU core count. Apple Silicon chips perform notably better at local inference due to their significantly higher unified memory bandwidth, not superior CPU architecture.