The landmark 2017 research paper by Vaswani et al. that introduced the Transformer architecture based entirely on self-attention mechanisms, replacing recurrence and convolutions.
Published at NeurIPS 2017 by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin, 'Attention Is All You Need' is the foundational paper of modern AI. Reading the original 15-page paper after implementing attention in Phase 5 reveals how simple and clean the core equations (`Attention(Q,K,V) = softmax(QK^T / sqrt(d_k))V`), multi-head projections, and complexity tradeoffs truly are.
Course status is tracked independently from roadmap topic checkboxes so you can mark external courses without altering your topic mastery percentage.
Phase 5 Foundational Research Paper (Modules 5.1–5.3)
100% free open-access PDF and HTML paper on arXiv (arXiv:1706.03762).
Status: Verified (2026-11-09)
Review these foundations before starting so you get the most out of this material:
In Phase 5: Transformers From Scratch
Read Sections 1–4 of the paper during Modules 5.1–5.3 after you have coded scaled dot-product attention so every equation feels familiar.
Why attention replaced recurrence, Queries/Keys/Values, scaled dot-product attention, softmax normalization, and causal lower-triangular masking.
Multi-head attention, positional representations (sinusoidal, learned, RoPE), FFN blocks (GELU, SwiGLU), Pre-LN LayerNorm & RMSNorm, residual connections, weight tying, and initialization.
Comparing Encoder-only (BERT), Decoder-only (GPT/LLaMA), and Encoder-Decoder (T5) architectures, attention masking patterns, and block stacking.
Line-by-line PyTorch implementation and mathematical walkthrough of the original 'Attention Is All You Need' Transformer paper.
A code-first introduction to neural networks covering automatic differentiation, backpropagation, language models, and GPT implementations.
Graduate-level Stanford course walking through every aspect of building language models from scratch: Transformer architecture, tokenization, pretraining data pipelines, GPU systems, parallelism, and scaling laws.