The simplest, fastest, and most readable open-source repository for training and fine-tuning medium-sized GPT language models (~300 lines of model code and ~300 lines of training loop).
Andrej Karpathy's `nanoGPT` strips away framework bloat and implements the complete GPT-2 architecture in a single readable `model.py` file (~300 lines) and a production-capable distributed mixed-precision training loop in `train.py` (~300 lines). It serves as both a Phase 5 architectural blueprint and a Phase 6 pretraining reference implementation.
Course status is tracked independently from roadmap topic checkboxes so you can mark external courses without altering your topic mastery percentage.
Reference Codebase for Phase 5 (`model.py`) and Phase 6 (`train.py`)
100% free MIT-licensed open-source GitHub repository.
Status: Verified (2026-11-09)
Review these foundations before starting so you get the most out of this material:
In Phase 5: Transformers From Scratch
Study `model.py` line by line while building Milestone #10 (Mini GPT Language Model) in Modules 5.2–5.5.
In Phase 6: Train Your Own LLM
Study `train.py` and `data/openwebtext/prepare.py` during Modules 6.1–6.3 to see how memory-mapped `.bin` token shards, gradient accumulation, BF16 autocast, cosine learning rate warmup, and DDP work in practice.
Multi-head attention, positional representations (sinusoidal, learned, RoPE), FFN blocks (GELU, SwiGLU), Pre-LN LayerNorm & RMSNorm, residual connections, weight tying, and initialization.
Comparing Encoder-only (BERT), Decoder-only (GPT/LLaMA), and Encoder-Decoder (T5) architectures, attention masking patterns, and block stacking.
Hands-on PyTorch implementation of a complete GPT-style language model from scratch—from embeddings and attention blocks to loss calculation, training loop, and generation.
Decoding strategies (greedy, temperature, top-k, top-p nucleus sampling), context window limits, stopping criteria, KV caching, and repetition mitigation.
Random weight initialization, token-budget batch sizing, gradient accumulation, AdamW decoupled weight decay, warmup-cosine schedules, gradient clipping, and fault-tolerant checkpointing.
Progressive pretraining experiments across Model A (100K–1M params), Model B (5M–20M params), and Model C (30M–100M params), tailored to your available hardware.
Serializing model weights, tokenizer artifacts, configuration JSONs, model cards, and standalone CLI inference scripts for reproducible distribution.
A code-first introduction to neural networks covering automatic differentiation, backpropagation, language models, and GPT implementations.
Standalone 2-hour hands-on video lecture building and training a generative pretrained Transformer (GPT) from an empty Python script.
Graduate-level Stanford course walking through every aspect of building language models from scratch: Transformer architecture, tokenization, pretraining data pipelines, GPU systems, parallelism, and scaling laws.