FlashMorph automates hybrid attention design using just 20M tokens and 2 GPU-hours

Researchers have introduced FlashMorph, a method that automates the conversion of standard attention models into hybrid architectures by framing layer selection as a global subset-optimization problem. Unlike previous approaches that relied on costly grid searches or hand-crafted layer patterns, FlashMorph requires only 20 million tokens and approximately 2.1 GPU-hours to determine the optimal mix of full and linear attention layers. The system freezes pretrained weights and jointly learns binary gates on synthetic retrieval data, followed by standard distillation to finalize the architecture. In evaluations, FlashMorph matched dense baseline performance on short and medium context retrieval benchmarks with negligible accuracy loss. However, the study has not yet tested generative tasks, and the authors note that synthetic training data may not fully reflect real-world distribution shifts.