Skip to main content
Curated Open-Access DirectoryPhase 0 – Phase 844 Canonical Resources · 51 Phase Placements

Learn From the Best. Completely Free.

Explore carefully curated courses, tutorials, lectures, documentation, and research resources from leading universities, educators, and open-source communities — organized around your AI and LLM learning journey.

44

Canonical Resources

9 Phases

51 Phase Recommendations

24

Universities & Labs

10 Primary

“Start Here” Picks

How to Use This Library: You do not need to complete every resource listed here. Each phase highlights a primary Start Here resource alongside optional lectures, technical documentation, and reference codebases so you can tailor your study depth.

My Learning Resources

Personal Resource Collection

Local-First Sync
Independent Resource Tracking44 Not Started

Bookmark courses and track your resource status (Not Started, In Progress, or Completed) separately from your roadmap topic checklist.

Bookmarks & course statuses persist locally in your browser.
Filter by Roadmap Phase
Quick Filters:
Showing 44 of 44 resources
PHASE 0 · P04 Weeks in RoadmapBeginner5 curated resources

Phase 0 — Programming Foundations

Develop the Python, algorithmic, and numerical computing fluency required to implement machine learning algorithms from first principles.

View Phase 0 Modules & Checklists

Primary Starting Resource for Phase 0

Begin with this foundational pick before exploring optional references
Resource 0.1Start Here · RecommendedFull Course
Harvard University
BeginnerFree Learning Material

CS50's Introduction to Programming with Python

A comprehensive introduction to Python programming, covering functions, variables, control flow, exceptions, libraries, unit testing, file I/O, and object-oriented programming.

Phase 0 Study Focus: Start here for Phase 0. Complete Weeks 0–8 and solve the programming problem sets to build rock-solid Python syntax, debugging, pytest unit testing, and object-oriented design fluency.
PythonCS50PUnit TestingOOPFile I/O
Your Status:

Additional Learning, Practice & Reference Materials (4)

Select selectively based on the specific modules you are studying
Resource 0.2Additional LearningFull Course
University of Michigan / Dr. Charles Severance
BeginnerFree Learning Material

Python for Everybody

Beginner-friendly Python instruction covering programming fundamentals, data structures, files, web data, and databases.

Phase 0 Study Focus: Use Lessons 1–11 if you want extra practice with Python strings, dictionaries, tuples, and file parsing at a gentler pace.
PythonData StructuresDictionariesBeginner Python
Your Status:
Resource 0.3Practice & ReferenceDocumentation / Tutorials
NumPy
BeginnerFree Learning Material

NumPy Learning Resources

Learn numerical arrays, broadcasting, vectorized computation, matrix operations, and scientific Python fundamentals.

Phase 0 Study Focus: Essential companion for Module 0.3. Work through 'NumPy: the absolute basics for beginners' and the broadcasting guides.
NumPyVectorsMatricesBroadcastingVectorization
Your Status:
Resource 0.4Practice & ReferenceDocumentation / Tutorials
Pandas
BeginnerFree Learning Material

Pandas Getting Started Tutorials

Learn to load, inspect, filter, clean, and manipulate structured datasets.

Phase 0 Study Focus: Study alongside Module 0.4 to practice loading, inspecting, filtering, and cleaning tabular CSV datasets.
PandasDataFramesData CleaningCSVTabular Data
Your Status:
Resource 0.5Practice & ReferenceDocumentation / Tutorials
Matplotlib
BeginnerFree Learning Material

Matplotlib Tutorials

Learn scientific plotting, mathematical visualization, and data visualization in Python.

Phase 0 Study Focus: Use during Module 0.4 to plot mathematical functions, numerical derivatives, and dataset distributions.
MatplotlibPlottingVisualizationLoss Curves
Your Status:
PHASE 1 · P18 Weeks in RoadmapBeginner → Intermediate7 curated resources

Phase 1 — Mathematics for AI

Master the linear algebra, multivariable calculus, probability theory, and numerical optimization that govern how neural networks represent and learn.

View Phase 1 Modules & Checklists

Primary Starting Resources for Phase 1

Begin with these foundational picks before exploring optional references
Resource 1.1Start Here · RecommendedVideo Series
3Blue1Brown
BeginnerFree Learning Material

Essence of Linear Algebra

Visual explanations of vectors, transformations, matrix multiplication, eigenvectors, and the intuition behind linear algebra.

Phase 1 Study Focus: Start Phase 1 here before diving into proof-heavy problem sets. Watch all chapters alongside Module 1.1 to build geometric intuition for matrix multiplication and eigen-decompositions.
Linear AlgebraVectorsMatricesEigenvectorsGeometric Intuition
Your Status:
Resource 1.2Start Here · RecommendedFull Course
MIT OpenCourseWare
IntermediateFree Learning Material

Linear Algebra (18.06SC)

A university-level linear algebra course covering vector spaces, matrices, linear systems, eigenvalues, and related topics.

Phase 1 Study Focus: Pair directly with 3Blue1Brown during Module 1.1. Focus on Matrix Multiplication, Orthogonality/Projections, Eigenvalues, and Singular Value Decomposition (SVD).
MIT OCWGilbert StrangLinear AlgebraSVDEigenvalues
Your Status:

Additional Learning, Practice & Reference Materials (5)

Select selectively based on the specific modules you are studying
Resource 1.3Additional LearningVideo Series
3Blue1Brown
BeginnerFree Learning Material

Essence of Calculus

Intuitive visual lessons explaining derivatives, integrals, and fundamental calculus concepts.

Phase 1 Study Focus: Watch at the start of Module 1.2, paying special attention to the episodes on the Chain Rule, Exponential derivatives, and Taylor series.
CalculusDerivativesChain Rule3Blue1Brown
Your Status:
Resource 1.4Additional LearningFull Course
MIT OpenCourseWare
BeginnerFree Learning Material

Single Variable Calculus

Comprehensive MIT OpenCourseWare Scholar course covering differentiation, the chain rule, optimization of single-variable functions, and integration.

Phase 1 Study Focus: Focus on Part I (Differentiation) and Part II (Applications of Differentiation & Optimization) if you need rigorous practice with single-variable derivatives before multivariable gradients.
MIT OCWCalculusDifferentiationOptimization
Your Status:
Resource 1.5Additional LearningFull Course
MIT OpenCourseWare
IntermediateFree Learning Material

Multivariable Calculus

University course on vectors, matrices, partial derivatives, gradient vectors, directional derivatives, and multivariable optimization.

Phase 1 Study Focus: Focus heavily on Part 2 (Partial Derivatives, Gradient Vectors, Chain Rule, and Max-Min Optimization Problems) alongside Modules 1.2 and 1.4.
MIT OCWMultivariable CalculusPartial DerivativesGradientsJacobians
Your Status:
Resource 1.6Additional LearningFull Course
MIT OpenCourseWare
IntermediateFree Learning Material

Introduction to Probability and Statistics

Rigorous introduction to discrete and continuous probability distributions, conditional probability, Bayes' theorem, expectation, variance, and maximum likelihood estimation.

Phase 1 Study Focus: Study the class reading notes on Random Variables, Expectation/Variance, Conditional Probability, and Maximum Likelihood Estimation during Module 1.3.
MIT OCWProbabilityStatisticsBayes TheoremMaximum Likelihood
Your Status:
Resource 1.7Practice & ReferenceFree Textbook
Mathematics for Machine Learning textbook authors
IntermediateFree Learning Material

Mathematics for Machine Learning

Mathematical foundations of machine learning, including linear algebra, vector calculus, probability, and optimization.

Phase 1 Study Focus: Use Chapters 2–7 as your primary written textbook reference across all four modules of Phase 1 (Linear Algebra, Vector Calculus, Probability, and Optimization).
TextbookLinear AlgebraVector CalculusOptimizationProbability
Your Status:
PHASE 2 · P28 Weeks in RoadmapIntermediate6 curated resources

Phase 2 — Machine Learning From Scratch

Understand how algorithms learn patterns from data, optimize loss functions, avoid overfitting and data leakage, and generalize to unseen examples.

View Phase 2 Modules & Checklists

Primary Starting Resource for Phase 2

Begin with this foundational pick before exploring optional references
Resource 2.1Start Here · RecommendedInteractive Course
Google
BeginnerFree Learning Material

Machine Learning Crash Course

Google's practical, interactive introduction to machine learning fundamentals, linear and logistic regression, gradient descent, generalization, regularization, and neural networks.

Phase 2 Study Focus: Start Phase 2 here. Complete the modules on Linear Regression, Logistic Regression, Classification Metrics, Overfitting, and Regularization alongside Modules 2.1, 2.2, and 2.4.
GoogleMachine LearningLinear RegressionLogistic RegressionGradient Descent
Your Status:

Additional Learning, Practice & Reference Materials (5)

Select selectively based on the specific modules you are studying
Resource 2.2Additional LearningFull Course
Harvard University
IntermediateFree Learning Material

CS50's Introduction to Artificial Intelligence with Python

University course exploring the foundational concepts and algorithms of modern artificial intelligence, including search, optimization, machine learning, neural networks, and natural language processing.

Phase 2 Study Focus: Focus on the lectures and projects covering Optimization, Learning (Supervised ML & Regularization), and Neural Networks.
HarvardCS50 AIMachine LearningOptimizationPython
Your Status:
Resource 2.3Practice & ReferenceInteractive Course
Kaggle
BeginnerFree Learning Material

Intro to Machine Learning

Hands-on interactive micro-course covering decision trees, model validation, underfitting vs. overfitting, and random forests.

Phase 2 Study Focus: Use during Modules 2.1 and 2.3 for rapid hands-on practice with decision trees, random forests, and validation splits.
KaggleDecision TreesRandom ForestsModel ValidationOverfitting
Your Status:
Resource 2.4Practice & ReferenceInteractive Course
Kaggle
IntermediateFree Learning Material

Intermediate Machine Learning

Practical interactive course covering missing values, categorical variables, pipelines, cross-validation, gradient boosting (XGBoost), and data leakage.

Phase 2 Study Focus: Complete alongside Modules 2.3 and 2.4 to master cross-validation, gradient boosting, and preventing data leakage.
KaggleCross-ValidationData LeakageGradient BoostingPipelines
Your Status:
Resource 2.5Practice & ReferenceDocumentation / Tutorials
scikit-learn
IntermediateFree Learning Material

scikit-learn Tutorials

Official scikit-learn getting started tutorials and user guide for classical supervised/unsupervised learning, preprocessing, cross-validation, and evaluation metrics.

Phase 2 Study Focus: Use as your reference benchmark when verifying your from-scratch NumPy implementations of Linear/Logistic Regression, Decision Trees, PCA, and k-Means.
scikit-learnClassical MLClassificationRegressionClustering
Your Status:
Resource 2.6Advanced ReadingInteractive Textbook
D2L
IntermediateFree Learning Material

Dive into Deep Learning

Free interactive textbook combining mathematical explanations, code, and practical deep-learning concepts.

Phase 2 Study Focus: Read Chapters 3 and 4 (Linear Neural Networks for Regression and Classification) to see how classical linear and softmax regression transition into deep learning.
D2LInteractive TextbookDeep LearningPyTorchMLP
Your Status:
PHASE 3 · P38 Weeks in RoadmapIntermediate → Advanced5 curated resources

Phase 3 — Deep Learning & Neural Networks

Implement multi-layer neural networks, reverse-mode automatic differentiation, and PyTorch training pipelines from the ground up.

View Phase 3 Modules & Checklists

Primary Starting Resource for Phase 3

Begin with this foundational pick before exploring optional references
Resource 3.1Start Here · RecommendedVideo Series
Andrej Karpathy
IntermediateFree Learning Material

Neural Networks: Zero to Hero

A code-first introduction to neural networks covering automatic differentiation, backpropagation, language models, and GPT implementations.

Phase 3 Study Focus: Start Phase 3 here! Watch Lecture 1 ('The spelled-out intro to neural networks and backpropagation: building micrograd') and Lectures 4–5 ('Becoming a Backprop Ninja' & activation/gradient diagnostics).
Andrej KarpathyMicrogradMakemoreBackpropagationAutograd
Your Status:

Additional Learning, Practice & Reference Materials (4)

Select selectively based on the specific modules you are studying
Resource 3.2Additional LearningFull Course
fast.ai
IntermediateFree Learning Material

Practical Deep Learning for Coders

Top-down, code-first course teaching practical deep learning, neural network training dynamics, transfer learning, and PyTorch foundations.

Phase 3 Study Focus: Use Part 1 Lessons 3–5 for practical intuition on SGD, learning rates, matrix multiplication, and building neural nets from the foundations.
fast.aiJeremy HowardDeep LearningPyTorchSGD
Your Status:
Resource 3.3Practice & ReferenceDocumentation / Tutorials
PyTorch
IntermediateFree Learning Material

PyTorch Beginner Tutorials

Official 'Learn the Basics' tutorial series covering PyTorch Tensors, Datasets/DataLoaders, `nn.Module`, `torch.autograd`, optimization loops, and model checkpointing.

Phase 3 Study Focus: Required hands-on tutorial for Module 3.4 (PyTorch Fundamentals). Complete all 7 sections of 'Learn the Basics'.
PyTorchTensorsAutogradnn.ModuleDataLoader
Your Status:
Resource 3.4Additional LearningInteractive Textbook
D2L
IntermediateFree Learning Material

Dive into Deep Learning

Free interactive textbook combining mathematical explanations, code, and practical deep-learning concepts.

Phase 3 Study Focus: Study Chapters 5, 6, and 12 (Multilayer Perceptrons, Forward/Backward Propagation, Numerical Stability/Initialization, and Optimization Algorithms) alongside Phase 3.
D2LInteractive TextbookDeep LearningPyTorchMLP
Your Status:
Resource 3.5Advanced ReadingUniversity Course Materials
Stanford University
IntermediateFree Learning Material

CS231n — Deep Learning for Computer Vision

Stanford's foundational deep learning course materials covering linear classifiers, optimization, backpropagation computational graphs, neural network training dynamics, and representation learning.

Phase 3 Study Focus: Read the publicly accessible CS231n written notes on Optimization, Backpropagation Intuition, and Neural Networks Part 1–3 (Activations, Initialization, Babysitting the Learning Process).
StanfordCS231nBackpropagationComputational GraphsWeight Initialization
Your Status:
PHASE 4 · P46 Weeks in RoadmapIntermediate → Advanced5 curated resources

Phase 4 — NLP & Language Modeling

Understand how raw human text is normalized, tokenized into subword integer IDs, embedded into dense vectors, and modeled autoregressively.

View Phase 4 Modules & Checklists

Primary Starting Resource for Phase 4

Begin with this foundational pick before exploring optional references
Resource 4.1Start Here · RecommendedInteractive Course
Hugging Face
IntermediateFree Learning Material

Hugging Face LLM Course

Comprehensive open interactive course covering Transformer models, tokenization pipelines, datasets, fine-tuning, and the open-source LLM ecosystem.

Phase 4 Study Focus: Start Phase 4 here (Chapters 1 & 2) to understand how raw text flows through tokenizers, embedding layers, and language modeling heads.
Hugging FaceNLPLanguage ModelingTokenizationTransformers
Your Status:

Additional Learning, Practice & Reference Materials (4)

Select selectively based on the specific modules you are studying
Resource 4.2Additional LearningUniversity Course Materials
Stanford University
AdvancedFree Learning Material

CS224N — Natural Language Processing with Deep Learning

Stanford's flagship graduate NLP curriculum covering word vectors (Word2Vec), neural language models, recurrent networks (RNN/LSTM), attention mechanisms, and pretrained Transformers.

Phase 4 Study Focus: Read the lecture notes and slides for Lectures 1–7 (Word Vectors, Neural Classifiers, RNNs, Language Models, Perplexity, and Vanishing Gradients) alongside Modules 4.3–4.5.
StanfordCS224NWord EmbeddingsWord2VecLanguage Models
Your Status:
Resource 4.3Additional LearningVideo Series
Andrej Karpathy
IntermediateFree Learning Material

Makemore — Neural Networks: Zero to Hero

Canonical Series: Neural Networks: Zero to Hero

A code-first introduction to neural networks covering automatic differentiation, backpropagation, language models, and GPT implementations.

Phase 4 Study Focus: In Phase 4, focus on the Makemore lectures (Bigrams, MLP character language model, and 'Let's build the GPT Tokenizer') to implement BPE tokenization and autoregressive character/subword modeling from scratch.
Andrej KarpathyMicrogradMakemoreBackpropagationAutograd
Your Status:
Resource 4.4Practice & ReferenceTutorial
PyTorch
IntermediateFree Learning Material

Character-Level RNN Classification Tutorial

Step-by-step PyTorch tutorial building a character-level Recurrent Neural Network from scratch without high-level RNN wrappers.

Phase 4 Study Focus: Complete this coding tutorial during Modules 4.4 and 4.5 to experience building explicit recurrent hidden-state loops in PyTorch.
PyTorchRNNCharacter-LevelSequence ModelingUnicode
Your Status:
Resource 4.5Practice & ReferenceCourse Module
Hugging Face
IntermediateFree Learning Material

Tokenizers — Hugging Face LLM Course

Dedicated technical module diving deep into subword tokenization algorithms: Byte-Pair Encoding (BPE), WordPiece, Unigram, normalization, pre-tokenization, and building tokenizers from scratch.

Phase 4 Study Focus: Study directly alongside Module 4.2 when implementing your Custom BPE Tokenizer (Milestone #7).
TokenizersBPEByte-Pair EncodingWordPieceUnigram
Your Status:
PHASE 5 · P510 Weeks in RoadmapAdvanced5 curated resources

Phase 5 — Transformers From Scratch

Understand and implement every component of modern GPT-style decoder-only Transformer architectures from first principles in PyTorch.

View Phase 5 Modules & Checklists

Primary Starting Resource for Phase 5

Begin with this foundational pick before exploring optional references
Resource 5.1Start Here · RecommendedVideo Series
Andrej Karpathy
IntermediateFree Learning Material

Neural Networks: Zero to Hero — GPT

Canonical Series: Neural Networks: Zero to Hero

A code-first introduction to neural networks covering automatic differentiation, backpropagation, language models, and GPT implementations.

Phase 5 Study Focus: Start Phase 5 with the final two lectures of Zero to Hero: 'Let's build GPT: from scratch, in code, spelled out' and 'Let's reproduce GPT-2 (124M)' to implement every block of your Mini GPT.
Andrej KarpathyMicrogradMakemoreBackpropagationAutograd
Your Status:

Additional Learning, Practice & Reference Materials (4)

Select selectively based on the specific modules you are studying
Resource 5.2Additional LearningUniversity Course Materials
Stanford University
AdvancedFree Learning Material

CS336 — Language Modeling from Scratch

Graduate-level Stanford course walking through every aspect of building language models from scratch: Transformer architecture, tokenization, pretraining data pipelines, GPU systems, parallelism, and scaling laws.

Phase 5 Study Focus: Study the Architecture lectures and Assignment 1 (implementing Byte-Pair Encoding, RMSNorm, RoPE, SwiGLU, Multi-Head Causal Self-Attention, and the full Transformer block from scratch).
StanfordCS336Language ModelingPretrainingScaling Laws
Your Status:
Resource 5.3Practice & ReferenceTechnical Tutorial
Harvard NLP
AdvancedFree Learning Material

The Annotated Transformer

Line-by-line PyTorch implementation and mathematical walkthrough of the original 'Attention Is All You Need' Transformer paper.

Phase 5 Study Focus: Read alongside Modules 5.1–5.3 and the 'Attention Is All You Need' paper to see how every equation maps line-by-line into PyTorch tensor operations.
Harvard NLPAnnotated TransformerSelf-AttentionMulti-Head AttentionPyTorch
Your Status:
Resource 5.4Practice & ReferenceGitHub Repository
Andrej Karpathy
AdvancedFree Learning Material

nanoGPT

The simplest, fastest, and most readable open-source repository for training and fine-tuning medium-sized GPT language models (~300 lines of model code and ~300 lines of training loop).

Phase 5 Study Focus: Study `model.py` line by line while building Milestone #10 (Mini GPT Language Model) in Modules 5.2–5.5.
nanoGPTAndrej KarpathyGPT-2PyTorchPretraining
Your Status:
Resource 5.5Advanced ReadingResearch Paper
Original Research Authors
AdvancedFree Learning Material

Attention Is All You Need

The landmark 2017 research paper by Vaswani et al. that introduced the Transformer architecture based entirely on self-attention mechanisms, replacing recurrence and convolutions.

Phase 5 Study Focus: Read Sections 1–4 of the paper during Modules 5.1–5.3 after you have coded scaled dot-product attention so every equation feels familiar.
Research PaperarXivAttention Is All You NeedTransformersSelf-Attention
Your Status:
PHASE 6 · P610 Weeks in RoadmapAdvanced5 curated resources

Phase 6 — Train Your Own LLM

Engineer end-to-end dataset pipelines, pretraining loops, evaluation suites, and reproducible packaging to train your own GPT-style language model from random weights.

View Phase 6 Modules & Checklists

Primary Starting Resource for Phase 6

Begin with this foundational pick before exploring optional references
Resource 6.1Start Here · RecommendedUniversity Course Materials
Stanford University
AdvancedFree Learning Material

CS336 — Language Modeling from Scratch

Graduate-level Stanford course walking through every aspect of building language models from scratch: Transformer architecture, tokenization, pretraining data pipelines, GPU systems, parallelism, and scaling laws.

Phase 6 Study Focus: Start Phase 6 here! Focus on the lectures and assignments covering Pretraining Data Pipelines (filtering, MinHash deduplication, quality classifiers), Mixed-Precision Training Loops, and Chinchilla Scaling Laws.
StanfordCS336Language ModelingPretrainingScaling Laws
Your Status:

Additional Learning, Practice & Reference Materials (4)

Select selectively based on the specific modules you are studying
Resource 6.2Additional LearningVideo Lecture
Andrej Karpathy
AdvancedFree Learning Material

Let's Build GPT: From Scratch

Standalone 2-hour hands-on video lecture building and training a generative pretrained Transformer (GPT) from an empty Python script.

Phase 6 Study Focus: Follow along and code every line before scaling up your Phase 6 pretraining pipeline (`MyLLM v1`).
Andrej KarpathyGPTVideo LecturePretrainingPyTorch
Your Status:
Resource 6.3Practice & ReferenceGitHub Repository
Andrej Karpathy
AdvancedFree Learning Material

nanoGPT Training Implementation

Canonical Series: nanoGPT

The simplest, fastest, and most readable open-source repository for training and fine-tuning medium-sized GPT language models (~300 lines of model code and ~300 lines of training loop).

Phase 6 Study Focus: Study `train.py` and `data/openwebtext/prepare.py` during Modules 6.1–6.3 to see how memory-mapped `.bin` token shards, gradient accumulation, BF16 autocast, cosine learning rate warmup, and DDP work in practice.
nanoGPTAndrej KarpathyGPT-2PyTorchPretraining
Your Status:
Resource 6.4Practice & ReferenceCourse Module
Hugging Face
IntermediateFree Learning Material

Hugging Face Datasets Course Material

Dedicated course chapter on loading, slicing, filtering, deduplicating, streaming, and memory-mapping massive text datasets using Apache Arrow and Hugging Face Datasets.

Phase 6 Study Focus: Study directly alongside Module 6.1 (Dataset Engineering) to build memory-mapped and streaming data pipelines for `MyLLM v1`.
DatasetsData EngineeringMemory MappingStreamingPretraining Data
Your Status:
Resource 6.5Additional LearningFull Course
Full Stack Deep Learning
AdvancedFree Learning Material

Full Stack Deep Learning

Comprehensive engineering course covering the end-to-end lifecycle of production deep learning and LLM systems: data management, experiment tracking, troubleshooting, testing, deployment, and continual learning.

Phase 6 Study Focus: Use the lectures on Data Management, Troubleshooting & Testing, and Experiment Management during Modules 6.3–6.5 to structure your `MyLLM v1` training runs cleanly.
Full Stack Deep LearningMLOpsExperiment TrackingDeploymentTesting
Your Status:
PHASE 7 · P78 Weeks in RoadmapAdvanced5 curated resources

Phase 7 — Post-training & Alignment

Transform pretrained base language models into helpful, safe, instruction-following assistants using SFT, LoRA/QLoRA, DPO, RLHF, and reasoning RL.

View Phase 7 Modules & Checklists

Primary Starting Resource for Phase 7

Begin with this foundational pick before exploring optional references
Resource 7.1Start Here · RecommendedInteractive Course
Hugging Face
AdvancedFree Learning Material

smol-course — Fine-Tuning Language Models

Practical hands-on course focused on post-training small and medium open-weight language models: chat templates, Supervised Fine-Tuning (SFT), LoRA/QLoRA parameter-efficient adaptation, preference alignment (DPO/ORPO), and evaluation.

Phase 7 Study Focus: Start Phase 7 here! Work through the units on Chat Templates & SFT, Parameter-Efficient Fine-Tuning (LoRA/QLoRA), and Preference Alignment (DPO) as you build Milestone #12 (Instruction-Tuned Assistant Model).
smol-courseSFTSupervised Fine-TuningLoRAQLoRA
Your Status:

Additional Learning, Practice & Reference Materials (4)

Select selectively based on the specific modules you are studying
Resource 7.2Additional LearningCourse Module
Hugging Face
IntermediateFree Learning Material

Fine-Tuning LLMs — Hugging Face Course

Foundational course module covering dataset preprocessing, dynamic padding, the Trainer API, custom PyTorch fine-tuning loops, and learning rate schedules for adapting pretrained Transformers.

Phase 7 Study Focus: Review during Modules 7.1 and 7.2 for clear coverage of dynamic batch collation, padding masks, and native PyTorch fine-tuning loops.
Fine-TuningPyTorchHugging FaceDynamic PaddingSFT
Your Status:
Resource 7.3Practice & ReferenceDocumentation / Tutorials
Hugging Face
AdvancedFree Learning Material

TRL Documentation and Tutorials

Official documentation and technical guides for TRL (Transformer Reinforcement Learning), covering `SFTTrainer`, `RewardTrainer`, `PPOTrainer`, `DPOTrainer`, and `GRPOTrainer`.

Phase 7 Study Focus: Use as your primary API and mathematical reference during Modules 7.1, 7.3, and 7.4 when running SFT, DPO, PPO, and GRPO experiments.
TRLSFTDPORLHFPPO
Your Status:
Resource 7.4Practice & ReferenceGitHub Repository / Handbook
Hugging Face
AdvancedFree Learning Material

Alignment Handbook

Production recipes, hyperparameter configurations, and open-source training scripts for aligning language models (such as Zephyr and SmolLM-Instruct) using SFT, LoRA/QLoRA, and DPO.

Phase 7 Study Focus: Inspect the `recipes/` directory when choosing learning rates, LoRA ranks, and DPO `beta` hyperparameters for your Phase 7 assistant model.
Alignment HandbookZephyrSFTDPOQLoRA
Your Status:
Resource 7.5Advanced ReadingInteractive Course
Hugging Face
AdvancedFree Learning Material

Deep Reinforcement Learning Course

Free interactive course teaching foundational and advanced Reinforcement Learning: Markov Decision Processes, Policy Gradients, Actor-Critic methods, and Proximal Policy Optimization (PPO).

Phase 7 Study Focus: Focus on Unit 1 (Introduction to RL), Unit 4 (Policy Gradient), Unit 6 (Actor-Critic), and Unit 8 (Proximal Policy Optimization — PPO) alongside Modules 7.3 and 7.4.
Reinforcement LearningPPOPolicy GradientsActor-CriticRLHF
Your Status:
PHASE 8 · P816 Weeks in RoadmapAdvanced → Research8 curated resources

Phase 8 — Advanced LLM Systems & Research

Master GPU kernel architecture, distributed multi-GPU training, Mixture-of-Experts and GQA architectures, high-throughput inference serving, and rigorous AI research methodology.

View Phase 8 Modules & Checklists

Primary Starting Resources for Phase 8

Begin with these foundational picks before exploring optional references
Resource 8.1Start Here · RecommendedUniversity Course Materials
Stanford University
AdvancedFree Learning Material

CS336 — Advanced Language Modeling Systems

Canonical Series: CS336 — Language Modeling from Scratch

Graduate-level Stanford course walking through every aspect of building language models from scratch: Transformer architecture, tokenization, pretraining data pipelines, GPU systems, parallelism, and scaling laws.

Phase 8 Study Focus: Co-recommended starting point for Phase 8. Study the Systems & Parallelism lectures and Assignment 2 (GPU memory hierarchy, roofline arithmetic intensity, Triton kernels, FlashAttention-2, and distributed multi-GPU training).
StanfordCS336Language ModelingPretrainingScaling Laws
Your Status:
Resource 8.6Start Here · RecommendedVideo Lectures
GPU MODE
ResearchFree Learning Material

GPU Programming Lectures

Community-driven advanced research and engineering lecture series covering GPU hardware architecture, CUDA kernels, Triton, FlashAttention, quantization, and PyTorch compiler internals.

Phase 8 Study Focus: Co-recommended starting point for Phase 8 (alongside Stanford CS336). Watch the foundational lectures on GPU Architecture, Profiling, Triton Kernels, FlashAttention, and Quantization during Modules 8.1 and 8.4.
GPU MODECUDATritonFlashAttentionGPU Architecture
Your Status:

Additional Learning, Practice & Reference Materials (6)

Select selectively based on the specific modules you are studying
Resource 8.2Additional LearningTraining Resources
NVIDIA
AdvancedFree Learning Material

CUDA Programming Resources

Official NVIDIA Developer & Educator teaching kits, university course materials, and technical resources on parallel computing and CUDA GPU programming.

Phase 8 Study Focus: Use during Module 8.1 (GPU & Training Systems) to understand thread blocks, warps, shared memory SRAM, and memory coalescing.
NVIDIACUDAParallel ComputingGPU MemoryKernels
Your Status:
Resource 8.3Additional LearningMixed Free/Paid Training Catalog
NVIDIA
AdvancedMixed Free / Paid Catalog

NVIDIA Online Training Catalog

NVIDIA Deep Learning Institute (DLI) self-paced training catalog featuring both free introductory courses and paid hands-on GPU cluster labs on CUDA, accelerated computing, and LLM deployment.

Phase 8 Study Focus: Optional supplementary catalog for Phase 8. Note: This is a mixed free/paid catalog—filter for free self-paced courses if you want zero-cost modules on CUDA and accelerated AI.
NVIDIADLITraining CatalogCUDAAccelerated Computing
Your Status:
Resource 8.4Practice & ReferenceDocumentation / Tutorials
PyTorch
AdvancedFree Learning Material

Distributed Training Tutorials

Official PyTorch documentation and step-by-step tutorials for DistributedDataParallel (DDP), Fully Sharded Data Parallel (FSDP2), Tensor Parallelism, and multi-GPU collective communication.

Phase 8 Study Focus: Core technical reference for Module 8.2 (Distributed Training). Work through the DDP and FSDP tutorials.
PyTorchDistributed TrainingDDPFSDPZeRO
Your Status:
Resource 8.5Additional LearningFull Course
Full Stack Deep Learning
AdvancedFree Learning Material

Full Stack Deep Learning

Comprehensive engineering course covering the end-to-end lifecycle of production deep learning and LLM systems: data management, experiment tracking, troubleshooting, testing, deployment, and continual learning.

Phase 8 Study Focus: In Phase 8, study the lectures on Infrastructure & Tooling, Deployment, Continual Learning, and Foundation Model Systems alongside Modules 8.4 and 8.5.
Full Stack Deep LearningMLOpsExperiment TrackingDeploymentTesting
Your Status:
Resource 8.7Practice & ReferenceTechnical Tutorials
Triton
ResearchFree Learning Material

Triton Programming Tutorials

Official hands-on tutorials for OpenAI's Triton language: vector addition, fused softmax, tiled matrix multiplication, LayerNorm, and FlashAttention kernels in Python.

Phase 8 Study Focus: Work through Tutorials 01 (Vector Add), 02 (Fused Softmax), 03 (Matrix Multiplication), and 06 (Fused Attention) during Modules 8.1 and 8.3.
TritonGPU KernelsFused SoftmaxFlashAttentionSRAM Tiling
Your Status:
Resource 8.8Practice & ReferenceDocumentation
vLLM
AdvancedFree Learning Material

vLLM Documentation

Official documentation for vLLM, the high-throughput open-source LLM inference and serving engine powered by PagedAttention, continuous batching, speculative decoding, and quantized kernels.

Phase 8 Study Focus: Study alongside Module 8.4 (Inference & Optimization) and Milestone #13 (Quantized & Optimized Inference Benchmark).
vLLMPagedAttentionKV CacheContinuous BatchingInference
Your Status: