How the 2017 Transformer Paper Actually Argued for Speed, Not Superiority

The landmark paper 'Attention Is All You Need' was published on arXiv on 12 June 2017 and presented at NIPS that December, primarily as a machine translation paper. Its central argument was not that transformers produce better outputs, but that replacing recurrence with self-attention makes training significantly faster and more parallelisable on modern hardware. Prior state-of-the-art models relied on recurrent networks, which required sequential processing steps that could not be parallelised, making them inefficient on GPU-scale hardware. Self-attention relates any two positions in a constant number of operations, though at the cost of quadratic complexity in sequence length — a trade-off that was negligible with short sentences in 2017 but became a major engineering challenge as context lengths grew into the hundreds of thousands in the 2020s. Most of what made transformers foundational to modern large language models was contributed by other researchers in the eighteen months following the paper's release.