Production recipes, hyperparameter configurations, and open-source training scripts for aligning language models (such as Zephyr and SmolLM-Instruct) using SFT, LoRA/QLoRA, and DPO.
Maintained by the Hugging Face H4 team, the Alignment Handbook contains the exact reproducible recipes, YAML hyperparameter configs, and multi-GPU/QLoRA scripts used to train top-performing open instruction models. It demonstrates why learning rates must drop an order of magnitude from SFT to DPO, how chat templates are validated, and how to prevent catastrophic forgetting during post-training.
Course status is tracked independently from roadmap topic checkboxes so you can mark external courses without altering your topic mastery percentage.
Phase 7 Production Recipe Codebase (Modules 7.1–7.5)
100% free Apache-2.0 licensed open-source GitHub repository and technical handbook.
Status: Verified (2026-11-09)
Review these foundations before starting so you get the most out of this material:
In Phase 7: Post-training & Alignment
Inspect the `recipes/` directory when choosing learning rates, LoRA ranks, and DPO `beta` hyperparameters for your Phase 7 assistant model.
Base vs. instruction-tuned models, chat templates, prompt-response loss masking, Supervised Fine-Tuning (SFT), and LIMA-style dataset quality.
Full fine-tuning memory costs, Low-Rank Adaptation (LoRA), Quantized LoRA (QLoRA), adapters, and memory-performance tradeoffs.
Human preference datasets, Bradley-Terry reward modeling, RLHF with PPO, Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO).
Verifiable rewards (RLVR), outcome vs. process supervision, chain-of-thought reasoning traces, reinforcement learning for reasoning, and evaluation pitfalls.
Practical hands-on course focused on post-training small and medium open-weight language models: chat templates, Supervised Fine-Tuning (SFT), LoRA/QLoRA parameter-efficient adaptation, preference alignment (DPO/ORPO), and evaluation.
Official documentation and technical guides for TRL (Transformer Reinforcement Learning), covering `SFTTrainer`, `RewardTrainer`, `PPOTrainer`, `DPOTrainer`, and `GRPOTrainer`.