Mixture-of-Experts Routing Extended to Attention Layer, Halving Query Head Compute

A new paper titled 'Grouped Query Experts' (arXiv 2506.20945), published in June 2026, applies mixture-of-experts routing to the attention layer of large language models — a part of the architecture this technique had not previously reached. Instead of activating all query heads for every token, a small router selects only the relevant heads per token, while the shared key-value memory remains fully active. The approach matches the output quality of standard all-active attention while activating roughly half the query heads, effectively cutting that portion of compute in half. Since attention costs scale rapidly with longer inputs, efficiency gains there could translate to cheaper training, faster inference, and capable models on less powerful hardware. The authors caution that results were demonstrated at relatively small scale, and it remains to be seen whether the gains hold as models are trained on more data at frontier sizes.