Character Language Model
Train an autoregressive character-level language model and generate text
What You Will Build
Autoregressive language modeling—predicting the probability distribution of the next token given the preceding context `P(x_t | x_{<t})`—is the foundational objective behind GPT-4, Claude, and Llama. Before scaling to subword BPE vocabularies and multi-layer Transformers, you can experience the entire lifecycle of generative language modeling at the character level (`vocab_size ≈ 65`). Inspired by Andrej Karpathy's `makemore`, you will prepare a text corpus, encode characters to integer IDs, slice `(X, Y)` context windows, build a count-based bigram baseline, train a neural character language model (embedding + MLP or RNN/GRU) using Cross-Entropy loss, evaluate validation perplexity `exp(loss)`, and generate text using multinomial temperature and Top-k sampling.
Why This Project Matters in the AI/LLM Stack
Every data formatting, loss computation, and sampling concept in this project—input/target shifting by 1 position, `nn.Embedding`, cross-entropy over vocabulary logits, perplexity, and temperature-scaled `torch.multinomial`—carries over unchanged to Project 10 (Mini GPT) and Project 11 (Pretrained Mini LLM).
Learning Objectives
- Build a bidirectional character vocabulary (`stoi` and `itos`) and construct shifted `(context, target)` training tensors.
- Implement a count-based or lookup bigram baseline to establish a concrete validation loss benchmark.
- Build and train a neural character language model with `nn.Embedding` that beats the bigram baseline on held-out validation loss.
- Implement autoregressive text generation comparing greedy decoding, temperature scaling (`τ = 0.5, 0.8, 1.2`), and Top-k sampling.
Prerequisites
- Project 6 (PyTorch training loops, `nn.Module`, `CrossEntropyLoss`)
- Phase 4 Module 4.1 & 4.3 (Tokenization, Embeddings, and Autoregressive Language Modeling)
Required Mathematical Concepts
- Chain rule of probability for sequences: P(x₁,…,x_T) = ∏_{t=1}^T P(x_t | x₁,…,x_{t-1})
- Negative Log-Likelihood (Cross-Entropy) loss: L = -(1/T) ∑_t log P_θ(x_t | x_{<t})
- Perplexity: PPL = exp(L) (effective branching factor per character)
- Temperature-scaled Softmax: P_i = exp(z_i / τ) / ∑_j exp(z_j / τ)
Step-by-Step Implementation Tasks (12 Steps)
Check off each engineering task as your code passes its acceptance criteria. Progress is saved automatically in your browser.
Corpus Preparation, Vocabulary & Sequence Slicing
Load a text dataset, build `stoi`/`itos` mappings, and construct shifted input-target sequences.
A cohesive public-domain text file (such as TinyShakespeare ~1MB or a list of 32,000 names) has clear orthographic and structural patterns that a small character model can learn quickly.
Every unique character in the corpus gets a deterministic integer index `0 .. vocab_size - 1`.
Neural networks ingest integer token ID tensors (`torch.long`), not Python strings.
In next-token prediction with context length `block_size`, a single chunk of `block_size + 1` tokens provides `block_size` supervised training examples simultaneously.
Bigram Baseline & Neural Language Model Architecture
Implement a simple character-level baseline, then build a neural language model with learned embeddings.
A bigram model predicts the next character using only the single current character `P(x_t | x_{t-1})`. Measuring its validation loss gives you a concrete floor that your neural model must beat.
Conditioning on `block_size > 1` previous characters via learned dense embeddings (`nn.Embedding(vocab_size, d_emb)`) and hidden non-linear layers allows the model to learn syllable structure, words, and punctuation.
Minimizing Cross-Entropy over the vocabulary logits maximizes the log-likelihood assigned to the true next character.
Training, Validation Perplexity & Autoregressive Generation
Train the model, track validation loss/perplexity, and implement autoregressive sampling strategies.
Training with `AdamW` over several thousand mini-batches progressively lowers training and validation cross-entropy.
Perplexity `PPL = exp(cross_entropy_loss)` measures how many characters the model is effectively choosing among at each step.
Autoregressive generation feeds a context sequence into the model, takes the logits for the final step, converts them to probabilities via Softmax, samples 1 character ID, appends it to the sequence, and repeats.
Temperature `τ → 0` (greedy) often gets stuck in repetitive loops, `τ = 0.8` with `top_k = 10` balances coherence and diversity, and `τ = 1.5` produces chaotic spelling.
Character-level models must spend sequence length and model capacity spelling individual words letter-by-letter, resulting in short effective semantic context.
Starter & Verification Code Examples
Autoregressive Sampling Loop with Temperature & Top-k Filtering
Shows how to sample tokens one step at a time with temperature scaling and optional Top-k logit masking.
import torch
import torch.nn.functional as F
@torch.no_grad()
def generate_characters(
model: torch.nn.Module,
idx: torch.Tensor,
max_new_tokens: int,
block_size: int,
temperature: float = 1.0,
top_k: int | None = None,
) -> torch.Tensor:
model.eval()
for _ in range(max_new_tokens):
idx_cond = idx[:, -block_size:]
logits = model(idx_cond) # Shape: (B, T, V) or (B, V)
if logits.ndim == 3:
logits = logits[:, -1, :] # Take final time-step: (B, V)
logits = logits / max(temperature, 1e-5)
if top_k is not None:
v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
logits[logits < v[:, [-1]]] = -float("Inf")
probs = F.softmax(logits, dim=-1)
idx_next = torch.multinomial(probs, num_samples=1)
idx = torch.cat((idx, idx_next), dim=1)
return idxHow to Test Your Implementation
Verify lossless encode/decode round-trips, target sequence alignment, initial uniform loss check, and generation output lengths.
Shifted Context-Target Alignment Test
Ensure `y[b, t]` is always the exact next token `x[b, t + 1]` in the raw stream.
Steps: Construct a batch `(x, y)` from `[10, 20, 30, 40, 50]` with `block_size=4`.
Pass Criteria: `x` is `[10, 20, 30, 40]` and `y` is `[20, 30, 40, 50]`.
Autoregressive Generation Length & Temperature Test
Confirm `generate()` appends exactly `max_new_tokens` valid token IDs without index errors.
Steps: Pass a 3-character prompt with `max_new_tokens=25`, `temperature=0.8`, `top_k=5`.
Pass Criteria: Output tensor has length `28` and decodes to a valid 28-character string.
Overall Project Acceptance Criteria
- Bidirectional character vocabulary and shifted `(X, Y)` sequence slicing implemented and tested.
- Bigram baseline validation loss and perplexity recorded.
- Neural character language model trained to achieve lower validation perplexity than the bigram baseline.
- Autoregressive text generation implemented with greedy, temperature, and Top-k sampling comparisons.
Expandable Hints & Debugging Tips
Try implementing each stage on your own first. Expand a hint below if you want to verify tensor shapes, mathematical identities, or common edge cases.
Optional Stretch Challenges
2D Character Embedding Visualization (t-SNE / PCA)
Project the learned `vocab_size × d_emb` character embedding matrix down to 2D and plot where vowels, consonants, digits, and punctuation cluster.
What You Learn: How distributional similarity emerges automatically in learned embedding tables.
WaveNet-Style Hierarchical Dilated Convolutions
Fuse pairs of adjacent characters hierarchically across layers (following Karpathy's `makemore` Part 3/5) to expand context length to `16` or `32` characters.
What You Learn: How hierarchical sequence composition bridges simple MLPs and full self-attention.
What Your Completed Repository Should Contain
A functioning character-level language model.
Deliverable Artifacts
- Character vocabulary encoder/decoder and shifted batch loader
- Bigram baseline and `CharNeuralLM` PyTorch implementation
- Validation perplexity comparison and multi-temperature generation showcase
After Completing This Project, You Should Be Able To...
- Formulate text generation as autoregressive next-token probability estimation trained with Cross-Entropy loss.
- Construct shifted `(context, target)` sequences and evaluate language models using validation Perplexity.
- Build and train an embedding-based neural language model that outperforms a count-based bigram baseline.
- Implement autoregressive decoding with greedy search, temperature scaling, and Top-k sampling.
GitHub Repository Submission
Link your public or private GitHub repository URL to keep a structured portfolio of your from-scratch AI builds. Stored locally in your browser (self-reported; no GitHub tokens or remote code execution).
Suggested Repository: character-language-model-pytorch
Recommended Directory Layout:
char_lm/dataset.py char_lm/model.py char_lm/generate.py tests/test_char_lm.py README.md
Recommended README Documentation Checklist
- Loss & Perplexity comparison table (Uniform vs Bigram vs Neural LM)
- Generated text samples across temperatures τ = 0.5, 0.8, 1.2
- Analysis of character-level context limitations