Your Complete Journey Into AI & LLM Development

Variables bind names to objects in memory. Core scalar types include integers (`int`), IEEE-754 floating-point numbers (`float`), booleans (`bool`), and Unicode strings (`str`).

Conditional branches (`if`, `elif`, `else`) and loops (`for`, `while`, `break`, `continue`) direct execution order—such as iterating over training epochs and stopping early when validation loss plateaus.

Functions encapsulate reusable logic with explicit inputs, default parameters, keyword arguments, and return values, keeping data preprocessing and mathematical formulas modular.

Lists store ordered mutable sequences; tuples represent fixed immutable records (like tensor shapes `(batch, seq_len, d_model)`); sets track unique tokens; dictionaries map keys to values in O(1) average time (like token-to-ID vocabularies).

Comprehensions provide a concise, readable syntax for transforming and filtering sequences or inverting token-to-ID lookup tables in a single expression.

Classes bundle state (attributes like layer weights and biases) with behavior (methods like `forward()` and `parameters()`). Every PyTorch `nn.Module` you write later is a Python class.

Generators use `yield` to produce items lazily one batch at a time rather than loading terabytes of training text into RAM all at once.

Decorators wrap functions to add cross-cutting behavior—such as timing execution, caching results, or disabling gradient tracking with `@torch.no_grad()`.

Structured `try/except/finally` blocks and custom `ValueError` assertions catch mismatched matrix dimensions or corrupted dataset rows immediately with informative diagnostics.

Organizing code across separate `.py` modules and packages (`__init__.py`) separates model definitions, data pipelines, training loops, and evaluation suites.

Context managers (`with open(...)`) and `pathlib.Path` safely read and write UTF-8 text corpora, JSON configurations, and binary model checkpoints.

Static type annotations (`def matmul(a: list[list[float]], b: list[list[float]]) -> list[list[float]]:`) document contracts and catch bugs before a long training run starts.

Unit tests (`pytest`), assertions, and interactive debuggers (`pdb` / IDE breakpoints) verify that custom mathematical implementations match analytical reference values.

A scalar is a single number (0D); a vector is an ordered 1D array of numbers; a matrix is a 2D grid of rows and columns; a tensor generalizes arrays to 3 or more axes (such as `[batch, seq_len, d_model]`).

A vector space $\mathbb{R}^n$ is a collection of $n$-dimensional vectors closed under vector addition and scalar multiplication, serving as the coordinate system where feature representations and embeddings live.

Adding two vectors of the same dimension sums their corresponding components ($u + v = [u_1+v_1, \dots, u_n+v_n]^T$), geometrically translating one vector along the other—used directly in residual connections (`x + sublayer(x)`).

Multiplying a vector by a scalar $c$ stretches or shrinks its magnitude by $|c|$ (and reverses its direction if $c < 0$) without changing the line it lies on.

A linear combination $c_1 v_1 + c_2 v_2 + \dots + c_k v_k$ blends multiple vectors using scalar weights. In self-attention, every output token is a weighted linear combination of Value vectors.

The dot product $u \cdot v = \sum_{i=1}^n u_i v_i = \|u\| \|v\| \cos(\theta)$ combines algebraic multiplication with geometric angle: it is positive when vectors point in similar directions, zero when orthogonal, and negative when opposed.

For $A \in \mathbb{R}^{m \times k}$ and $B \in \mathbb{R}^{k \times n}$, entry $(i, j)$ of $C = AB$ is the dot product of row $i$ of $A$ with column $j$ of $B$, transforming a batch of $m$ input vectors of dimension $k$ into dimension $n$.

The transpose $A^T$ swaps rows and columns ($(A^T)_{ij} = A_{ji}$) and satisfies $(AB)^T = B^T A^T$—used constantly when computing attention scores $QK^T$ and weight gradients during backpropagation.

A square matrix $A$ is invertible if there exists $A^{-1}$ such that $AA^{-1} = I$. In practice, explicit inversion is numerically unstable and avoided in favor of solving linear systems $Ax = b$ or gradient optimization.

The rank of a matrix is the number of linearly independent rows or columns—the true dimensionality of the subspace onto which the matrix maps inputs. Low-Rank Adaptation (LoRA) exploits low-rank weight updates.

Every matrix represents a linear transformation that maps vectors from one space to another while preserving grid lines parallel and evenly spaced (rotations, reflections, shears, projections).

Two vectors are orthogonal if $u \cdot v = 0$. Orthogonal projection finds the closest point in a subspace to a target vector, forming the geometric foundation of least-squares linear regression.

Norms measure vector magnitude: the $L_1$ norm ($\sum |x_i|$) promotes sparsity, the $L_2$ Euclidean norm ($\sqrt{\sum x_i^2}$) measures straight-line length and powers weight decay and gradient clipping, and $L_\infty$ tracks the maximum absolute entry.

The Euclidean distance $\|u - v\|_2 = \sqrt{\sum (u_i - v_i)^2}$ measures straight-line separation between two points in feature space, used in k-nearest neighbors and k-means clustering.

Cosine similarity $\frac{u \cdot v}{\|u\|_2 \|v\|_2} \in [-1, 1]$ normalizes the dot product by vector lengths, measuring pure directional alignment between semantic embeddings regardless of magnitude.

An eigenvector $v$ of square matrix $A$ only gets scaled (not rotated) when multiplied by $A$: $Av = \lambda v$. Eigenvalues $\lambda$ reveal variance directions in PCA and curvature in Hessian matrices.

Any $m \times n$ matrix factors as $A = U \Sigma V^T$ into orthogonal rotation $V^T$, diagonal scaling $\Sigma$ (singular values), and orthogonal rotation $U$—enabling dimensionality reduction and low-rank matrix approximation.

Rather than hand-coding explicit decision rules, machine learning optimizes parameterized mathematical functions so they extract statistical regularities directly from data.

Learning a mapping $f_\theta: X \to Y$ from labeled input-output pairs $(x_i, y_i)$—predicting continuous targets (regression) or discrete categories (classification).

Discovering latent structure, clusters, or low-dimensional manifolds in unlabeled data $X$ without human annotations (clustering, dimensionality reduction, density estimation).

Generating supervisory labels automatically from the structure of raw data itself—such as predicting the next token from preceding tokens in raw text corpora, enabling LLM pretraining on trillions of words without human labeling.

Features ($X \in \mathbb{R}^{N \times d}$) are the measurable input attributes of each example; labels ($y$) are the target outcomes the model is trained to predict.

The training set fits parameters $\theta$; the validation set tunes hyperparameters and selects architectures; the held-out test set is evaluated once at the end to measure unbiased generalization.

Parameters (weights $W$ and biases $b$) are internal variables updated automatically by optimization algorithms during training.

Hyperparameters (learning rate $\eta$, regularization strength $\lambda$, polynomial degree, number of layers) are configuration choices set prior to training and tuned on the validation split.

Overfitting occurs when a model memorizes noise and idiosyncrasies in the training set, achieving low training loss but high validation/test loss.

Underfitting occurs when a model is too simple or insufficiently trained to capture the underlying signal, resulting in high error on both training and validation data.

Expected generalization error decomposes into Bias² (systematic error from overly restrictive assumptions), Variance (sensitivity to fluctuations in the training sample), and irreducible noise.

Generalization is a model's ability to perform accurately on new, unseen examples drawn from the same underlying distribution.

Data leakage happens when information from the validation or test set inadvertently enters training—such as computing normalization mean/std across the whole dataset before splitting, or target leakage in features.

Standardizing features to zero mean and unit variance ($z = (x - \mu_{\text{train}})/\sigma_{\text{train}}$) or min-max scaling transforms elongated loss contours into circular ones so gradient descent converges rapidly.

Adding a complexity penalty to the loss—$L_2$ Ridge ($\lambda \|w\|_2^2$, equivalent to weight decay) or $L_1$ Lasso ($\lambda \|w\|_1$, inducing sparse zero weights)—constrains parameter magnitude to reduce variance.

$K$-fold cross-validation partitions training data into $K$ folds, training $K$ times while rotating the validation fold to obtain a low-variance estimate of generalization performance.

A computational unit that computes a weighted sum of inputs plus a bias ($z = w^T x + b$) and applies a non-linear activation function $a = \phi(z)$.

Rosenblatt's classical binary threshold classifier; its inability to solve non-linearly separable tasks like XOR proved the necessity of multi-layer networks with smooth differentiable activations.

For a layer mapping $d_{\text{in}} \to d_{\text{out}}$, the weight matrix $W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$ rotates/scales inputs and the bias vector $b \in \mathbb{R}^{d_{\text{out}}}$ shifts activation thresholds.

Multi-Layer Perceptrons (MLPs) compose sequential layers $h^{(l)} = \phi(h^{(l-1)} W^{(l)} + b^{(l)})$ without feedback loops, forming both standalone classifiers and the FFN sublayer inside every Transformer block.

Dense linear projections (`nn.Linear`) where every output unit connects to every input unit, executing batched matrix multiplication $Y = XW + b$ accompanied by backward gradients $dX = dY \cdot W^T$ and $dW = X^T \cdot dY$.

Element-wise non-linear functions that break linearity between matrix multiplications so stacked layers learn hierarchical features.

$\sigma(x) = \frac{1}{1 + e^{-x}} \in (0, 1)$ has derivative $\sigma(x)(1 - \sigma(x)) \le 0.25$, which saturates near 0 and 1 and causes vanishing gradients in deep hidden layers, though it remains ideal for binary gates.

$\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} \in (-1, 1)$ is zero-centered with derivative $1 - \tanh^2(x) \le 1$, improving over sigmoid for hidden states in RNNs/LSTMs.

Rectified Linear Unit $\text{ReLU}(x) = \max(0, x)$ has derivative $1$ for $x > 0$ and $0$ for $x < 0$, eliminating positive-side saturation at the risk of 'dead ReLUs' when units stay negative.

Gaussian Error Linear Unit $\text{GELU}(x) = x \Phi(x)$ smoothly weights inputs by their Gaussian cumulative probability, serving as the standard activation in GPT-2, BERT, and Vision Transformers.

Sigmoid Linear Unit (Swish) $\text{SiLU}(x) = x \cdot \sigma(x)$ pairs with gating in SwiGLU feed-forward networks used across modern LLaMA, Mistral, and DeepSeek LLMs.

Converts a vector of arbitrary real logits $z \in \mathbb{R}^V$ into a normalized probability distribution $p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$ summing to 1.

Combining Softmax and Negative Log-Likelihood into a single fused operation yields the clean upstream logit gradient $\frac{\partial L}{\partial z_i} = p_i - y_i$.

Sequentially passing input tensors through each layer while caching intermediate activations needed later during the backward pass.

Propagating upstream loss sensitivities $\frac{\partial L}{\partial a^{(l)}}$ backward from the output layer to the first layer using cached forward activations and local Jacobians.

Initializing weights all to zero causes symmetry (every neuron learns the same thing); too large causes exploding activations; Xavier ($1/\sqrt{d_{\text{in}}}$) and He ($\sqrt{2/d_{\text{in}}}$) preserve activation and gradient variance across depth.

Randomly zeroes a fraction $p$ of activations during training and scales remaining activations by $\frac{1}{1-p}$ (inverted dropout) to prevent co-adaptation, while acting as an identity pass during inference.

BatchNorm normalizes across the batch dimension for CNNs; LayerNorm and RMSNorm normalize across the feature dimension independently for each token, making them standard in Transformers.

Skip connections $y = x + F(x)$ have gradient $\frac{\partial y}{\partial x} = I + \frac{\partial F}{\partial x}$, ensuring gradients flow unimpeded through the identity term $I$ even in 100-layer networks.

The Unicode standard assigns a unique integer code point (`U+0000` to `U+10FFFF`) to over 150,000 characters across human scripts, mathematical symbols, and emojis.

A variable-width encoding that represents every Unicode code point as 1 to 4 bytes (`0`–`255`), preserving 1-byte ASCII compatibility while enabling byte-level tokenization with a base vocabulary of just 256 byte values.

Visually identical characters (like `é` as a single code point vs. `e` + combining acute accent) have different byte sequences unless canonicalized via Unicode normalization (`unicodedata.normalize('NFC', text)`).

Splitting documents into sentences or semantic paragraphs while handling abbreviations (`Dr.`, `e.g.`, `3.14`) and code blocks accurately.

Removing broken HTML markup, control characters, and boilerplate navigation text while carefully preserving indentation and punctuation needed for code and prose.

Collecting unique tokens from a training corpus, sorting or ranking them by frequency, and assigning contiguous integer IDs `0 .. V-1` via bidirectional `stoi` and `itos` maps.

Word-level vocabularies map out-of-vocabulary (OOV) words to a shared `<unk>` token, destroying information—a flaw solved by character-level or byte-level subword tokenization.

Originally introduced to let seq2seq translation models look back at all source tokens without compressing a whole sentence into a single vector, before Vaswani et al. (2017) showed that self-attention alone ('Attention Is All You Need') eliminates recurrence entirely.

Projected from token representations via $Q = X W_Q$, a Query vector $q_t \in \mathbb{R}^{d_k}$ represents the question or lookup criteria issued by token position $t$.

Projected via $K = X W_K$, a Key vector $k_j \in \mathbb{R}^{d_k}$ represents the label or address advertised by token position $j$ to match against incoming Queries.

Projected via $V = X W_V$, a Value vector $v_j \in \mathbb{R}^{d_v}$ contains the actual semantic content contributed by token $j$ once its Key matches a Query.

The raw matrix product $S = Q K^T$ of shape `(B, H, T, T)` where entry $S_{i,j} = q_i \cdot k_j$ measures raw affinity between query token $i$ and key token $j$.

When $q$ and $k$ have zero mean and unit variance, their dot product has variance $d_k$. Dividing by $\sqrt{d_k}$ restores unit variance so softmax inputs do not grow large and push gradients to zero.

Applied row-wise along the last dimension (`dim=-1`) of the scaled score matrix so that for each query token $i$, the attention weights over all attended keys $j$ are non-negative and sum to $1.0$.

In autoregressive language modeling, token $i$ must never peek at future tokens $j > i$. Using `torch.tril` and `masked_fill(mask == 0, float('-inf'))` before softmax forces future weights to $\exp(-\infty) = 0$.

Visualizing the $T \times T$ attention matrix reveals interpretable routing patterns—such as induction heads, previous-token heads, and delimiter-attending heads.

Sourcing domain-appropriate text corpora—such as TinyStories, FineWeb-Edu subsets, open-access textbooks, or permissive code repositories—matched to your target model size and compute budget.

Inspecting dataset licenses (CC-BY, CC0, MIT, Apache-2.0 vs. non-commercial or restrictive terms) so you know what rights apply to models trained on the corpus.

Recording the exact origin URL, snapshot date, hash checksum, and preprocessing script version for every shard entering your training set.

Extracting clean prose and code from raw HTML, Markdown, or PDF sources while stripping navigation menus, ads, and broken markup.

Applying heuristic quality filters (symbol-to-word ratios, stop-word presence, line-length distributions) to discard gibberish, machine-generated spam, and corrupted encodings.

Removing exact duplicate documents/paragraphs (via SHA-256 hashes) and near-duplicates (via MinHash LSH) prevents the model from memorizing repeated boilerplate and reduces train-test leakage.

Filtering documents by target language probability using fast classifiers so a small model does not waste capacity across dozens of unintended scripts.

Redacting email addresses, phone numbers, IP addresses, and secret API keys using regex patterns before tokenization.

Splitting at the document level (never slicing inside a single document!) after deduplication so validation perplexity measures true generalization to unseen documents.

Encoding millions of documents in parallel across CPU worker processes into compact `uint16` (for $V < 65536$) or `uint32` integer arrays.

Concatenating tokenized documents separated by `<|eos|>` into a single continuous stream and slicing fixed-length `block_size` windows eliminates wasted `<|pad|>` compute during pretraining.

Using `np.memmap('train.bin', dtype=np.uint16, mode='r')` maps binary token files directly from disk into virtual address space, letting you train on datasets larger than system RAM.

Recording the tokenizer hash, shard checksums, total token counts, and filtering statistics in a `dataset_card.json` alongside the binary shards.

Pretraining optimizes next-token prediction over billions/trillions of raw web tokens using massive compute to acquire world knowledge; fine-tuning adapts those weights on thousands of curated examples at a smaller learning rate.

Collections of `(system, user_instruction, assistant_response)` records covering question answering, coding, summarization, and multi-turn dialogue.

Deterministic serialization formats (such as ChatML `<|im_start|>role\ncontent<|im_end|>`) that convert structured message arrays into a single token stream with unambiguous turn boundaries.

Constructing input `input_ids` and target `labels` where user prompt tokens are masked with `-100` (`ignore_index`) so cross-entropy loss is computed strictly on the assistant's response tokens.

Standard autoregressive cross-entropy training over formatted instruction-response trajectories for 1–3 epochs with a small learning rate (e.g., `2e-5`).

During SFT, the model is conditioned on the ground-truth prefix at every token position of the assistant's target answer, enabling parallel loss calculation in a single forward pass.

A few thousand carefully written, accurate, and well-formatted demonstrations consistently outperform 100,000 noisy or contradictory examples during SFT.

GPUs consist of dozens of Streaming Multiprocessors (SMs) containing specialized Tensor Cores for matrix multiply-accumulate ($D = A B + C$) alongside high-bandwidth off-chip HBM memory.

The parallel programming model organizing threads into warps, thread blocks (scheduled onto a single SM), and grids, indexing tensors via `blockIdx.x * blockDim.x + threadIdx.x`.

Off-chip HBM is large (16–80+ GB) but relatively slow (~1–3 TB/s), whereas on-chip SRAM / Shared Memory is tiny (~20–50 MB across SMs) but ~10× faster (~19+ TB/s).

Every un-fused PyTorch operation (`x * 0.5`, `torch.tanh`, `+`) launches a separate GPU kernel that reads inputs from HBM, computes in registers, and writes outputs back to HBM.

Combining sequences of element-wise and reduction operations (like `RMSNorm` or `GeLU`) into a single GPU kernel so intermediate values stay in ultra-fast registers/SRAM (`torch.compile`).

Performing matrix multiplications in 16-bit precision on Tensor Cores while accumulating products and maintaining master weights in FP32.

IEEE 16-bit float (1 sign, 5 exponent, 10 mantissa bits): high precision but narrow dynamic range (max value `65,504`), requiring dynamic loss scaling (`GradScaler`) to prevent small gradients from underflowing to zero.

Brain Float 16 (1 sign, 8 exponent, 7 mantissa bits): matches FP32's exact 8-bit exponent dynamic range, eliminating `GradScaler` and overflow headaches during LLM pretraining.

Activation checkpointing discards intermediate sublayer activations during the forward pass and recomputes them on the fly during the backward pass—trading ~25–33% extra compute to reduce activation VRAM by 4×–8×.

Tiles $Q, K, V$ into SRAM blocks and uses the online softmax rescaling trick to compute exact attention without ever materializing the $O(T^2)$ attention matrix in HBM (`F.scaled_dot_product_attention`).

A Python-based GPU kernel language that operates on tile blocks (`tl.load`, `tl.store`, `tl.dot`) rather than individual scalar threads, making custom fused kernels (fused softmax, RMSNorm, cross-entropy) accessible without raw C++ CUDA.

Using `torch.profiler` and Chrome trace timelines (`trace.json`) to inspect GPU kernel occupancy, CPU launch gaps, and memory allocation lifecycles.