Official hands-on tutorials for OpenAI's Triton language: vector addition, fused softmax, tiled matrix multiplication, LayerNorm, and FlashAttention kernels in Python.
Triton allows AI engineers to write high-efficiency block-tiled GPU kernels directly in Python without managing low-level CUDA warp synchronization manually. The official Triton tutorial sequence walks from a simple vector addition kernel to a fused row-wise softmax (demonstrating massive memory-bandwidth speedups over naive PyTorch), tiled matrix multiplication in SRAM, fused LayerNorm, and FlashAttention.
Course status is tracked independently from roadmap topic checkboxes so you can mark external courses without altering your topic mastery percentage.
Phase 8 Hands-On Kernel Engineering (Modules 8.1 & 8.3)
100% free open-source documentation and executable Python kernel scripts.
Status: Verified (2026-11-09)
Review these foundations before starting so you get the most out of this material:
In Phase 8: Advanced LLM Systems & Research
Work through Tutorials 01 (Vector Add), 02 (Fused Softmax), 03 (Matrix Multiplication), and 06 (Fused Attention) during Modules 8.1 and 8.3.
GPU hardware architecture, CUDA execution hierarchy, HBM vs. SRAM memory bandwidth, kernel fusion, FP16/BF16, activation checkpointing, FlashAttention, Triton, and PyTorch Profiler.
Mixture of Experts (MoE), Grouped-Query Attention (GQA), Multi-Query Attention (MQA), long-context modeling, Multimodal Vision/Audio Transformers, Diffusion fundamentals, and Scaling Laws.
Community-driven advanced research and engineering lecture series covering GPU hardware architecture, CUDA kernels, Triton, FlashAttention, quantization, and PyTorch compiler internals.
Graduate-level Stanford course walking through every aspect of building language models from scratch: Transformer architecture, tokenization, pretraining data pipelines, GPU systems, parallelism, and scaling laws.
Official documentation for vLLM, the high-throughput open-source LLM inference and serving engine powered by PagedAttention, continuous batching, speculative decoding, and quantized kernels.