Why AI Infrastructure's Real Bottleneck Is Memory Bandwidth, Not Raw Compute

The true constraint in AI infrastructure is not processing power but the physical layer — chips, power, cooling, and data centers — which analysts argue represents the more investable side of the AI boom. Running a large language model involves two distinct tasks: prefill, which processes input tokens in parallel and suits GPUs well, and decode, which generates output tokens sequentially and is bottlenecked by memory bandwidth. During decode, the chip must read an entire model's weights from memory to produce each single token, making memory speed the limiting factor rather than raw computation. Power availability has emerged as an equally critical constraint, since grid capacity cannot be rapidly expanded regardless of capital, making performance-per-watt a core business metric for data center operators. Groq is cited as a notable example of a company that identified and targeted this memory-bandwidth bottleneck rather than competing on conventional compute metrics.