Graduate-level Stanford course walking through every aspect of building language models from scratch: Transformer architecture, tokenization, pretraining data pipelines, GPU systems, parallelism, and scaling laws.
Stanford CS336 (Spring 2025) is the definitive university engineering curriculum for building Large Language Models from first principles without relying on opaque high-level wrappers. Across its lectures and hands-on programming assignments, students implement a BPE tokenizer, a modern Transformer (RMSNorm, RoPE, SwiGLU), FlashAttention/Triton kernels, distributed multi-GPU training (DDP/FSDP/tensor parallelism), compute-optimal scaling laws, pretraining data curation/deduplication, and RLHF/DPO alignment.
Course status is tracked independently from roadmap topic checkboxes so you can mark external courses without altering your topic mastery percentage.
Primary Systems & Pretraining Spine across Phase 5 (Architecture), Phase 6 (Pretraining), and Phase 8 (Advanced LLM Systems)
Course syllabus, detailed lecture slides/notes, reading lists, and comprehensive open-source assignment repositories on GitHub are publicly accessible.
Status: Verified (2026-11-09)
Review these foundations before starting so you get the most out of this material:
In Phase 5: Transformers From Scratch
Study the Architecture lectures and Assignment 1 (implementing Byte-Pair Encoding, RMSNorm, RoPE, SwiGLU, Multi-Head Causal Self-Attention, and the full Transformer block from scratch).
In Phase 6: Train Your Own LLM
Start Phase 6 here! Focus on the lectures and assignments covering Pretraining Data Pipelines (filtering, MinHash deduplication, quality classifiers), Mixed-Precision Training Loops, and Chinchilla Scaling Laws.
In Phase 8: Advanced LLM Systems & Research
Co-recommended starting point for Phase 8. Study the Systems & Parallelism lectures and Assignment 2 (GPU memory hierarchy, roofline arithmetic intensity, Triton kernels, FlashAttention-2, and distributed multi-GPU training).
Why attention replaced recurrence, Queries/Keys/Values, scaled dot-product attention, softmax normalization, and causal lower-triangular masking.
Multi-head attention, positional representations (sinusoidal, learned, RoPE), FFN blocks (GELU, SwiGLU), Pre-LN LayerNorm & RMSNorm, residual connections, weight tying, and initialization.
Comparing Encoder-only (BERT), Decoder-only (GPT/LLaMA), and Encoder-Decoder (T5) architectures, attention masking patterns, and block stacking.
Hands-on PyTorch implementation of a complete GPT-style language model from scratch—from embeddings and attention blocks to loss calculation, training loop, and generation.
Corpus curation, licensing, cleaning, deduplication, PII filtering, tokenization, sequence packing, and memory-mapped binary streaming.
Random weight initialization, token-budget batch sizing, gradient accumulation, AdamW decoupled weight decay, warmup-cosine schedules, gradient clipping, and fault-tolerant checkpointing.
Progressive pretraining experiments across Model A (100K–1M params), Model B (5M–20M params), and Model C (30M–100M params), tailored to your available hardware.
Training vs. validation loss, perplexity, baselines, memorization vs. generalization, data contamination, hallucinations, robustness, speed/memory profiling, and ablations.
GPU hardware architecture, CUDA execution hierarchy, HBM vs. SRAM memory bandwidth, kernel fusion, FP16/BF16, activation checkpointing, FlashAttention, Triton, and PyTorch Profiler.
Scaling training across multiple GPUs and nodes: DDP, FSDP (ZeRO-1/2/3), Tensor Parallelism, Pipeline Parallelism, collective communication primitives, and fault tolerance.
Mixture of Experts (MoE), Grouped-Query Attention (GQA), Multi-Query Attention (MQA), long-context modeling, Multimodal Vision/Audio Transformers, Diffusion fundamentals, and Scaling Laws.
A code-first introduction to neural networks covering automatic differentiation, backpropagation, language models, and GPT implementations.
The simplest, fastest, and most readable open-source repository for training and fine-tuning medium-sized GPT language models (~300 lines of model code and ~300 lines of training loop).
Community-driven advanced research and engineering lecture series covering GPU hardware architecture, CUDA kernels, Triton, FlashAttention, quantization, and PyTorch compiler internals.
Official hands-on tutorials for OpenAI's Triton language: vector addition, fused softmax, tiled matrix multiplication, LayerNorm, and FlashAttention kernels in Python.