Official PyTorch documentation and step-by-step tutorials for DistributedDataParallel (DDP), Fully Sharded Data Parallel (FSDP2), Tensor Parallelism, and multi-GPU collective communication.
When a language model or its optimizer states exceed the VRAM of a single GPU, training must be parallelized across devices. The official PyTorch Distributed Overview and tutorial suite teach collective primitives (`AllReduce`, `AllGather`, `ReduceScatter`), `DistributedDataParallel` (DDP), `FullyShardedDataParallel` (FSDP / ZeRO-3 sharding), and Pipeline/Tensor Parallelism.
Course status is tracked independently from roadmap topic checkboxes so you can mark external courses without altering your topic mastery percentage.
Phase 8 Module 8.2 Core Reference (Distributed Training)
100% free official PyTorch distributed training tutorials.
Status: Verified (2026-11-09)
Review these foundations before starting so you get the most out of this material:
In Phase 8: Advanced LLM Systems & Research
Core technical reference for Module 8.2 (Distributed Training). Work through the DDP and FSDP tutorials.
Graduate-level Stanford course walking through every aspect of building language models from scratch: Transformer architecture, tokenization, pretraining data pipelines, GPU systems, parallelism, and scaling laws.
Community-driven advanced research and engineering lecture series covering GPU hardware architecture, CUDA kernels, Triton, FlashAttention, quantization, and PyTorch compiler internals.
Official hands-on tutorials for OpenAI's Triton language: vector addition, fused softmax, tiled matrix multiplication, LayerNorm, and FlashAttention kernels in Python.