Standalone 2-hour hands-on video lecture building and training a generative pretrained Transformer (GPT) from an empty Python script.
In this famous standalone coding lecture, Andrej Karpathy starts with a blank file, loads the TinyShakespeare dataset, constructs a baseline bigram language model, writes the training and evaluation loop, and incrementally builds up self-attention, multi-head causal attention, feed-forward blocks, residual connections, LayerNorm, and autoregressive token generation.
Course status is tracked independently from roadmap topic checkboxes so you can mark external courses without altering your topic mastery percentage.
Phase 5–6 Bridge Video Lecture (Building & Training GPT End-to-End)
100% free full-length video lecture on YouTube with open-source Colab notebook and GitHub code (`ng-video-lecture`).
Status: Verified (2026-11-09)
Review these foundations before starting so you get the most out of this material:
In Phase 6: Train Your Own LLM
Follow along and code every line before scaling up your Phase 6 pretraining pipeline (`MyLLM v1`).
Hands-on PyTorch implementation of a complete GPT-style language model from scratch—from embeddings and attention blocks to loss calculation, training loop, and generation.
Decoding strategies (greedy, temperature, top-k, top-p nucleus sampling), context window limits, stopping criteria, KV caching, and repetition mitigation.
Random weight initialization, token-budget batch sizing, gradient accumulation, AdamW decoupled weight decay, warmup-cosine schedules, gradient clipping, and fault-tolerant checkpointing.
Progressive pretraining experiments across Model A (100K–1M params), Model B (5M–20M params), and Model C (30M–100M params), tailored to your available hardware.
Serializing model weights, tokenizer artifacts, configuration JSONs, model cards, and standalone CLI inference scripts for reproducible distribution.
A code-first introduction to neural networks covering automatic differentiation, backpropagation, language models, and GPT implementations.
The simplest, fastest, and most readable open-source repository for training and fine-tuning medium-sized GPT language models (~300 lines of model code and ~300 lines of training loop).
Graduate-level Stanford course walking through every aspect of building language models from scratch: Transformer architecture, tokenization, pretraining data pipelines, GPU systems, parallelism, and scaling laws.