Practical hands-on course focused on post-training small and medium open-weight language models: chat templates, Supervised Fine-Tuning (SFT), LoRA/QLoRA parameter-efficient adaptation, preference alignment (DPO/ORPO), and evaluation.
The Hugging Face `smol-course` is purpose-built for Phase 7 of this roadmap. Designed so every exercise runs on consumer or free cloud GPUs, it guides you step by step through formatting multi-turn instruction datasets with chat templates, running Supervised Fine-Tuning (SFT), applying Low-Rank Adaptation (LoRA and 4-bit QLoRA), aligning models with Direct Preference Optimization (DPO), and evaluating instruction-following quality.
Course status is tracked independently from roadmap topic checkboxes so you can mark external courses without altering your topic mastery percentage.
Phase 7 Primary Starting Course (Modules 7.1–7.5)
100% free open-source interactive course and GitHub notebooks on Hugging Face.
Status: Verified (2026-11-09)
Review these foundations before starting so you get the most out of this material:
In Phase 7: Post-training & Alignment
Start Phase 7 here! Work through the units on Chat Templates & SFT, Parameter-Efficient Fine-Tuning (LoRA/QLoRA), and Preference Alignment (DPO) as you build Milestone #12 (Instruction-Tuned Assistant Model).
Base vs. instruction-tuned models, chat templates, prompt-response loss masking, Supervised Fine-Tuning (SFT), and LIMA-style dataset quality.
Full fine-tuning memory costs, Low-Rank Adaptation (LoRA), Quantized LoRA (QLoRA), adapters, and memory-performance tradeoffs.
Human preference datasets, Bradley-Terry reward modeling, RLHF with PPO, Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO).
Safe behavior, harmful output evaluation, calibrated refusals, bias mitigation, instruction hierarchy, red teaming, reward hacking, catastrophic forgetting, and calibration.
Verifiable rewards (RLVR), outcome vs. process supervision, chain-of-thought reasoning traces, reinforcement learning for reasoning, and evaluation pitfalls.
Official documentation and technical guides for TRL (Transformer Reinforcement Learning), covering `SFTTrainer`, `RewardTrainer`, `PPOTrainer`, `DPOTrainer`, and `GRPOTrainer`.
Production recipes, hyperparameter configurations, and open-source training scripts for aligning language models (such as Zephyr and SmolLM-Instruct) using SFT, LoRA/QLoRA, and DPO.
Foundational course module covering dataset preprocessing, dynamic padding, the Trainer API, custom PyTorch fine-tuning loops, and learning rate schedules for adapting pretrained Transformers.