Study Finds Mixing Attention Types Matters More Than Layer Order in Transformers

Researchers at VIDRAFT AI Research used a Latin square design to test whether the arrangement of different attention mechanisms across Transformer layers affects model performance. By structurally preventing any single mechanism from clustering in one area, they isolated placement from mechanism choice as variables. Ablation experiments on a 700.9M-parameter proxy model showed that shuffling a balanced mix of mechanisms had no measurable effect, but replacing all mechanisms with a single type raised cross-entropy loss by 1.68%. Removing state-space model Mamba-2 from the stack caused a 2.14% performance drop, while removing attention-family variants had negligible impact, suggesting cross-family diversity is the key driver. Both penalties grew larger when scaled to 1.514B parameters, lending additional credibility to the findings.