PyTorch 2.13 FlexAttention hits 7.83x speedup on M1 Max for sparse attention

PyTorch 2.13, released on July 8, 2026, introduced Metal/MPS kernel support for FlexAttention on Apple Silicon, enabling custom attention rules to run as compiled GPU kernels on devices like the M1 Max. A benchmark comparing FlexAttention against standard Scaled Dot-Product Attention (SDPA) found that at 32,768 tokens with a 256-token local window, FlexAttention completed in 75.27 ms versus 589.05 ms for SDPA, a 7.83x speedup. However, for standard causal attention, SDPA was approximately 19 times faster, showing that FlexAttention's advantage is limited to long, highly sparse attention patterns. FlexAttention works by expressing an attention rule as a Python function, which PyTorch's compiler converts into a specialized fused kernel that can skip unnecessary token comparisons. The API and kernel options are still marked unstable in PyTorch 2.13, and the test was a kernel benchmark using random inputs rather than a real pretrained model.