DeepSeek's MLA Cuts KV Cache Memory by 93% Over Standard Transformer Attention

DeepSeek's Multi-Head Latent Attention (MLA), used in DeepSeek-V2 and V3, dramatically reduces the memory footprint of the key-value cache during transformer inference. Instead of storing full key and value tensors for all attention heads, MLA compresses the hidden state into a low-dimensional latent vector of just 512 scalars per token. This brings the KV cache cost down to roughly 135 KB per token in FP16, compared to 3.84 MB for a standard multi-head attention model like DeepSeek's 67B baseline — a reduction of over 93%. The improvement directly addresses the memory-bandwidth bottleneck that limits token generation speed during the autoregressive decode phase. By contrast, competing approaches like Multi-Query Attention achieve similar compression but at a measurable cost to model quality, making MLA a more efficient trade-off.