A code-first introduction to neural networks covering automatic differentiation, backpropagation, language models, and GPT implementations.
Andrej Karpathy's flagship code-first lecture series is the backbone of Phases 3, 4, and 5 of this curriculum. Starting from an empty Python file, you build `micrograd` (a scalar reverse-mode autograd engine), progress through `makemore` (character-level language models using bigrams, MLPs, BatchNorm, diagnostic histograms, backprop ninja, and WaveNet), build a GPT tokenizer from scratch (`minbpe`), and culminate in implementing and training a GPT-2 decoder-only Transformer in PyTorch.
Course status is tracked independently from roadmap topic checkboxes so you can mark external courses without altering your topic mastery percentage.
Core Spine across Phase 3 (Autograd/DL), Phase 4 (Makemore/Tokenizers), and Phase 5 (GPT From Scratch)
All video lectures, GitHub repositories (micrograd, makemore, minbpe, nanoGPT), and Jupyter notebooks are 100% free and open-source.
Status: Verified (2026-11-09)
Review these foundations before starting so you get the most out of this material:
In Phase 3: Deep Learning & Neural Networks
Start Phase 3 here! Watch Lecture 1 ('The spelled-out intro to neural networks and backpropagation: building micrograd') and Lectures 4–5 ('Becoming a Backprop Ninja' & activation/gradient diagnostics).
In Phase 4: NLP & Language Modeling
In Phase 4, focus on the Makemore lectures (Bigrams, MLP character language model, and 'Let's build the GPT Tokenizer') to implement BPE tokenization and autoregressive character/subword modeling from scratch.
In Phase 5: Transformers From Scratch
Start Phase 5 with the final two lectures of Zero to Hero: 'Let's build GPT: from scratch, in code, spelled out' and 'Let's reproduce GPT-2 (124M)' to implement every block of your Mini GPT.
Neurons, feedforward layers, modern activations (ReLU, GELU, SiLU), forward/backward passes, initialization, normalization, and residual streams.
Building a dynamic reverse-mode autodiff engine (micrograd-style) with computational graphs, topological sorting, and gradient accumulation.
PyTorch tensors, GPU acceleration, autograd mechanics, nn.Module architecture, DataLoaders, checkpointing, and mixed precision.
Bag of Words, TF-IDF weighting, N-gram statistical language models with smoothing, and dense static embeddings (Word2Vec, GloVe).
Character, word, and subword tokenization; Byte Pair Encoding (BPE), WordPiece, Unigram, vocabulary sizing, and special control tokens.
Autoregressive next-token prediction, conditional probability factorization, cross-entropy training, perplexity evaluation, and sampling.
Why attention replaced recurrence, Queries/Keys/Values, scaled dot-product attention, softmax normalization, and causal lower-triangular masking.
Multi-head attention, positional representations (sinusoidal, learned, RoPE), FFN blocks (GELU, SwiGLU), Pre-LN LayerNorm & RMSNorm, residual connections, weight tying, and initialization.
Comparing Encoder-only (BERT), Decoder-only (GPT/LLaMA), and Encoder-Decoder (T5) architectures, attention masking patterns, and block stacking.
Hands-on PyTorch implementation of a complete GPT-style language model from scratch—from embeddings and attention blocks to loss calculation, training loop, and generation.
Decoding strategies (greedy, temperature, top-k, top-p nucleus sampling), context window limits, stopping criteria, KV caching, and repetition mitigation.
Standalone 2-hour hands-on video lecture building and training a generative pretrained Transformer (GPT) from an empty Python script.
The simplest, fastest, and most readable open-source repository for training and fine-tuning medium-sized GPT language models (~300 lines of model code and ~300 lines of training loop).
Official 'Learn the Basics' tutorial series covering PyTorch Tensors, Datasets/DataLoaders, `nn.Module`, `torch.autograd`, optimization loops, and model checkpointing.
Graduate-level Stanford course walking through every aspect of building language models from scratch: Transformer architecture, tokenization, pretraining data pipelines, GPU systems, parallelism, and scaling laws.