QuantMLA's INT4/INT2 dual-path cache lifts long-context serving throughput over fivefold
Serving transformers with very long token windows often runs into memory bottlenecks that hinder real-time deployment. The QuantMLA approach uses a dual-path INT4/INT2 key-value cache, which reportedly yields more than five times the whole-job output throughput of a BF16 baseline under long-sequence pressure tests. Conventional BF16 caches grow linearly with sequence length, causing memory preemptions and recomputation that limit practical window sizes. At a 128k-token window, QuantMLA compresses the combined content and RoPE cache by 3.59 times and reportedly avoids preemption entirely. The authors stress that the gain comes from avoiding capacity-driven recomputation rather than from a faster attention kernel, and they caution that positional signals may degrade on tasks requiring fine-grained geometry.
