Reading: Vaswani et al., Attention Is All You Need

The founding paper of the era.

Reading

Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin, 'Attention Is All You Need,' NeurIPS (2017).

The founding paper of the modern era. Fifteen pages that define the architecture behind every frontier model. Read Section 3 (Model Architecture) with a diagram open. Read Section 4 for why attention won.

Reading

Primary and secondary sources for this lesson.

Your record

Enrol as a Scholar to keep a record of the lessons you complete and the credentials they lead to.

Enrol in the Academy

← Layer normalization and residual connections Pretraining →