Kimi K3's Linear Attention Explained: Why Architecture Beats Parameter Count

Kimi K3, Moonshot AI's 2.8-trillion-parameter model, uses a custom mechanism called KDA (Kimi Delta Attention), a variant of linear attention designed to handle context windows of up to one million tokens. Standard transformer attention requires an N×N matrix across all token pairs, making very long sequences computationally infeasible. Linear attention sidesteps this by maintaining a running state updated token by token, keeping the cost roughly constant regardless of sequence length. KDA builds on linear attention by adding a delta term intended to recover expressiveness that plain linear attention loses over long sequences. A short Python example using NumPy illustrates the core difference between standard and linear attention, showing that both produce identically shaped outputs while the linear version never constructs the full matrix.