Splitting LLM prefill and decode adds a KV cache transfer cost that can erode gains

Separating prefill and decode onto different GPU pools removes compute stalls, but the design also has to move the KV cache between them. For a 32-layer model with 8 KV heads, each token adds about 64 KB of cache, so an 8,000-token prompt produces roughly 512 MB that must cross the network before decoding begins. On a 10 Gb/s link this transfer takes about 410 ms, longer than the 320 ms prefill it was meant to offload, while 25 Gb/s roughly halves the benefit and 100 Gb/s clearly wins. The author argues that cache size and interconnect bandwidth should be calculated before choosing disaggregation, and that grouped-query attention and KV cache compression can substantially reduce the transfer cost.