How FlashAttention Speeds Up Transformers by Reducing GPU Memory Traffic

Attention in Transformers scales quadratically with sequence length, and on modern GPUs the real bottleneck is often memory bandwidth rather than raw compute. Standard implementations write large intermediate attention matrices to slow high-bandwidth memory, forcing repeated costly reads and writes. FlashAttention restructures the computation through tiling, processing small blocks that fit in fast on-chip SRAM, and uses online softmax to keep the exact result without materializing the full matrix. The approach avoids storing that matrix during the forward pass, reducing memory use for long contexts. The article attributes its large speedups to this IO-aware reorganization rather than any approximation.