The 2017 Transformer Paper Explained: Why Attention Replaced Recurrence

The 2017 paper 'Attention Is All You Need' proposed replacing recurrent neural networks with a purely attention-based architecture for sequence tasks like translation and summarisation. Its core argument was about parallelism: recurrent models must process sequences one step at a time, making training time scale with sequence length regardless of available hardware, while self-attention computes all positions simultaneously. The architecture consists of encoder and decoder stacks of six identical layers each, combining scaled dot-product attention, multi-head attention, feed-forward networks, residual connections, and positional encodings. A key section compares layer types on complexity, parallelisability, and maximum path length between positions, showing that self-attention achieves constant path length versus the linear path length of recurrence, making long-range dependencies easier to learn. This single property — full parallelism during training — underpinned the subsequent decade of scaling larger models on larger datasets.