PIVOT technique cuts long-context AI inference latency without retraining models

Researchers have developed PIVOT, a training-free method that addresses a core inefficiency in Dynamic Sparse Attention (DSA) used by large language models during long-context inference. Although DSA reduces attention computation, its indexer still performs a full token scan per query, keeping overall complexity at O(L²) and dominating latency at sequence lengths like 100K tokens. PIVOT exploits the observation that adjacent queries share roughly 90% of their top-k token selections, allowing a single proxy scan per group of queries instead of one scan per query. This reduces indexer cost from O(L²) to O(L²/g), delivering a reported 4× indexer speedup and 1.6× end-to-end latency reduction on models including DeepSeek-V3.2 and GLM-5.1. The method offers two modes — PIVOT-Reuse for maximum speed and PIVOT-Refine for accuracy closer to a dense indexer — and can be plugged into existing DSA models at inference time.