Community-driven advanced research and engineering lecture series covering GPU hardware architecture, CUDA kernels, Triton, FlashAttention, quantization, and PyTorch compiler internals.
GPU MODE is an open research and systems engineering community featuring deep technical lectures by leading kernel engineers, PyTorch core developers, and LLM systems researchers. Topics span SRAM vs. HBM memory hierarchies, roofline profiling, custom CUDA and Triton kernels, FlashAttention algorithms, FP8/INT4 quantization, and distributed ring/tensor parallelism.
Course status is tracked independently from roadmap topic checkboxes so you can mark external courses without altering your topic mastery percentage.
Phase 8 Co-Primary Starting Resource (Modules 8.1, 8.2 & 8.4)
100% free full-length technical lectures and open-source companion repositories.
Status: Verified (2026-11-09)
Review these foundations before starting so you get the most out of this material:
In Phase 8: Advanced LLM Systems & Research
Co-recommended starting point for Phase 8 (alongside Stanford CS336). Watch the foundational lectures on GPU Architecture, Profiling, Triton Kernels, FlashAttention, and Quantization during Modules 8.1 and 8.4.
GPU hardware architecture, CUDA execution hierarchy, HBM vs. SRAM memory bandwidth, kernel fusion, FP16/BF16, activation checkpointing, FlashAttention, Triton, and PyTorch Profiler.
Scaling training across multiple GPUs and nodes: DDP, FSDP (ZeRO-1/2/3), Tensor Parallelism, Pipeline Parallelism, collective communication primitives, and fault tolerance.
Post-training quantization (INT8, INT4, AWQ, GGUF), KV cache management, continuous batching, PagedAttention, speculative decoding, serving APIs, observability, and cost optimization.
Graduate-level Stanford course walking through every aspect of building language models from scratch: Transformer architecture, tokenization, pretraining data pipelines, GPU systems, parallelism, and scaling laws.
Official hands-on tutorials for OpenAI's Triton language: vector addition, fused softmax, tiled matrix multiplication, LayerNorm, and FlashAttention kernels in Python.
Official NVIDIA Developer & Educator teaching kits, university course materials, and technical resources on parallel computing and CUDA GPU programming.