Line-by-line PyTorch implementation and mathematical walkthrough of the original 'Attention Is All You Need' Transformer paper.
Written by Sasha Rush and collaborators at Harvard NLP (updated to modern PyTorch), The Annotated Transformer presents the seminal 2017 Vaswani et al. paper as an executable literacy document. Every mathematical equation for scaled dot-product attention, multi-head projection, positional encoding, residual layer normalization, and label smoothing is immediately followed by clean PyTorch code.
Course status is tracked independently from roadmap topic checkboxes so you can mark external courses without altering your topic mastery percentage.
Phase 5 Line-by-Line Code & Paper Companion (Modules 5.1–5.3)
100% free open-source technical article and executable Jupyter notebook.
Status: Verified (2026-11-09)
Review these foundations before starting so you get the most out of this material:
In Phase 5: Transformers From Scratch
Read alongside Modules 5.1–5.3 and the 'Attention Is All You Need' paper to see how every equation maps line-by-line into PyTorch tensor operations.
Why attention replaced recurrence, Queries/Keys/Values, scaled dot-product attention, softmax normalization, and causal lower-triangular masking.
Multi-head attention, positional representations (sinusoidal, learned, RoPE), FFN blocks (GELU, SwiGLU), Pre-LN LayerNorm & RMSNorm, residual connections, weight tying, and initialization.
Comparing Encoder-only (BERT), Decoder-only (GPT/LLaMA), and Encoder-Decoder (T5) architectures, attention masking patterns, and block stacking.
The landmark 2017 research paper by Vaswani et al. that introduced the Transformer architecture based entirely on self-attention mechanisms, replacing recurrence and convolutions.
A code-first introduction to neural networks covering automatic differentiation, backpropagation, language models, and GPT implementations.
The simplest, fastest, and most readable open-source repository for training and fine-tuning medium-sized GPT language models (~300 lines of model code and ~300 lines of training loop).