Why Moving Model Weights, Not Math, Drives AI Token Energy Cost on Devices

On-device AI inference consumes most of its energy moving model weights through memory, not performing arithmetic, making energy cost roughly proportional to bytes read per token. Research grounded in Mark Horowitz's 2014 ISSCC findings shows that a DRAM memory access costs thousands of times more energy than a basic arithmetic operation, a gap that remains relevant today. For a 3-billion-parameter model quantized to 4-bit precision, generating a single token is estimated to consume between 0.028 and 0.278 joules in memory-system energy alone, meaning a 300-token response could draw 8 to 83 joules. This structure reveals why quantization is so effective on battery-powered devices: halving the bytes per weight approximately halves energy per token, far outweighing any arithmetic savings. Factors such as on-chip caches, KV cache growth at long contexts, and display or radio power draw add nuance but do not change the core linear relationship between model size, token count, and energy use.