Dedicated course chapter on loading, slicing, filtering, deduplicating, streaming, and memory-mapping massive text datasets using Apache Arrow and Hugging Face Datasets.
Pretraining a language model requires processing gigabytes or terabytes of text without running out of system RAM. Chapter 5 of the Hugging Face LLM Course teaches you how Apache Arrow memory-mapping works under the hood, how to stream multi-hundred-gigabyte corpora without downloading them to disk, and how to apply parallelized `.map()` and `.filter()` transformations for dataset curation.
Course status is tracked independently from roadmap topic checkboxes so you can mark external courses without altering your topic mastery percentage.
Phase 6 Module 6.1 (Dataset Engineering & Sharding)
100% free interactive chapters and code notebooks on Hugging Face.
Status: Verified (2026-11-09)
Review these foundations before starting so you get the most out of this material:
In Phase 6: Train Your Own LLM
Study directly alongside Module 6.1 (Dataset Engineering) to build memory-mapped and streaming data pipelines for `MyLLM v1`.
Comprehensive open interactive course covering Transformer models, tokenization pipelines, datasets, fine-tuning, and the open-source LLM ecosystem.
Dedicated technical module diving deep into subword tokenization algorithms: Byte-Pair Encoding (BPE), WordPiece, Unigram, normalization, pre-tokenization, and building tokenizers from scratch.
Graduate-level Stanford course walking through every aspect of building language models from scratch: Transformer architecture, tokenization, pretraining data pipelines, GPU systems, parallelism, and scaling laws.