Dedicated technical module diving deep into subword tokenization algorithms: Byte-Pair Encoding (BPE), WordPiece, Unigram, normalization, pre-tokenization, and building tokenizers from scratch.
Chapter 6 of the Hugging Face LLM Course focuses exclusively on the internals of modern tokenizers. It walks step by step through the tokenization pipeline (normalization, pre-tokenization, model, post-processing) and compares BPE, WordPiece, and Unigram algorithms with code implementations—making it the ideal companion to Milestone #7 (Custom BPE Tokenizer).
Course status is tracked independently from roadmap topic checkboxes so you can mark external courses without altering your topic mastery percentage.
Phase 4 Module 4.2 Dedicated Tokenization Reference
100% free interactive module with runnable code notebooks on Hugging Face.
Status: Verified (2026-11-09)
Review these foundations before starting so you get the most out of this material:
In Phase 4: NLP & Language Modeling
Study directly alongside Module 4.2 when implementing your Custom BPE Tokenizer (Milestone #7).
Unicode code points, UTF-8 byte streams, normalization forms, sentence segmentation, corpus cleaning, and vocabulary construction.
Bag of Words, TF-IDF weighting, N-gram statistical language models with smoothing, and dense static embeddings (Word2Vec, GloVe).
Comprehensive open interactive course covering Transformer models, tokenization pipelines, datasets, fine-tuning, and the open-source LLM ecosystem.
A code-first introduction to neural networks covering automatic differentiation, backpropagation, language models, and GPT implementations.
Dedicated course chapter on loading, slicing, filtering, deduplicating, streaming, and memory-mapping massive text datasets using Apache Arrow and Hugging Face Datasets.