Why LLMs Are ALUs, Not CPUs: The Hidden Cost of Agent Architecture

A software developer argues that large language models are fundamentally misunderstood as general-purpose processors, when they are better compared to arithmetic-logic units that transform input in a single pass without retaining state. Every tool call in an LLM agent forces data through the model's context window unnecessarily, incurring costs in tokens, latency, and reliability even when no real reasoning is needed. The author contends that treating a stateless function like a state machine is the root architectural flaw in most agent frameworks, not a problem solvable by larger models or longer context windows. Because LLMs have no internal program counter or persistent memory between calls, compounding tool use degrades reliability and makes failures harder to diagnose. The piece concludes that waiting for more powerful models is a category error — a faster ALU is still an ALU, and the missing components must be built around it.