Summary
'Attention Is All You Need' (Vaswani et al., 2017) proposed an architecture built entirely on attention — no recurrence, no convolutions. It was designed for translation. It became the substrate of every frontier language model, every foundation model, and most of modern AI.
Objectives
- 01State what the transformer removed and what it kept.
- 02Explain why removing recurrence mattered.
- 03Trace the seven-year path from translation to ChatGPT.
The Lesson
The move
Attention lets every position in a sequence directly attend to every other position, in parallel. Recurrence forced sequential computation and long-range dependency loss. Removing it made training massively parallelizable and long-range coherence tractable.
The consequence
GPT (2018), BERT (2018), GPT-2 (2019), GPT-3 (2020), ChatGPT (2022). Every one is a transformer. The 2017 paper is the closest thing modern AI has to a founding document.
Key Ideas
- Attention replaces recurrence, enabling parallel training at scale.
- The transformer is the substrate of the foundation-model era.
References
- Vaswani et al., 'Attention Is All You Need,' NeurIPS (2017)