Neural Network From Scratch
Implement and train a vectorized multilayer neural network in NumPy
What You Will Build
Scalar autodiff (Project 4) teaches how computational graphs work on individual numbers, but real deep learning operates on batched tensors using vectorized matrix calculus. In this project, you will implement a modular, multi-layer neural network in pure NumPy. You will design `Linear` layers (`Z = XW + b`), nonlinear activations (`ReLU`, `Sigmoid`, `Softmax`), numerically stable Cross-Entropy loss, He/Xavier weight initialization, explicit backward pass Jacobians (`dX = dZ @ W.T`, `dW = X.T @ dZ`, `db = sum(dZ, axis=0)`), and mini-batch training—verifying your analytical tensor gradients against both numerical finite differences and PyTorch.
Why This Project Matters in the AI/LLM Stack
Deriving and coding the matrix backward pass (`dW = Xᵀ dZ`, `dX = dZ Wᵀ`) makes tensor shape duality second nature: you will immediately see why weight gradients match the shape of `W` and input gradients match the shape of `X`.
Learning Objectives
- Implement modular Dense (`Linear`) and activation layers with forward activation caching and explicit `.backward(d_out)` methods.
- Derive and code numerically stable Softmax Cross-Entropy loss and its combined logit gradient `(probs - one_hot) / B`.
- Validate every layer's backward pass using central finite differences and compare learned trajectories against an identical PyTorch model.
- Train a multi-layer neural network with mini-batch gradient descent and weight initialization that prevents vanishing/exploding activations.
Prerequisites
- Project 1 (Matrix Operations) and Project 4 (Autodiff Engine)
- Phase 3 Module 3.1 & 3.2 (Forward pass, matrix Jacobians, He/Xavier initialization)
Required Mathematical Concepts
- Batched affine transformation: Z = XW + b with X ∈ ℝ^{B×D_in}, W ∈ ℝ^{D_in×D_out}, b ∈ ℝ^{1×D_out}
- Linear backward pass: dW = Xᵀ dZ, db = ∑_{i=1}^B dZ_i, dX = dZ Wᵀ
- Elementwise activation backward pass: dZ = dA ⊙ f'(Z)
- Log-sum-exp stabilized Softmax and Cross-Entropy gradient: dZ_logits = (P - Y_one_hot) / B
- Variance-preserving initialization: He normal N(0, √(2/D_in)) for ReLU
Step-by-Step Implementation Tasks (14 Steps)
Check off each engineering task as your code passes its acceptance criteria. Progress is saved automatically in your browser.
Architecture Design, Weight Initialization & Forward Pass
Implement `Linear` dense layers with He/Xavier initialization, `ReLU`/`Sigmoid` activations, and forward caching.
A clean layer interface (`forward(x)` and `backward(dout)`) lets you stack arbitrary sequences of dense and activation layers inside a `Sequential` container.
Initializing weights to all zeros causes all hidden neurons to compute identical gradients (symmetry breaking failure); overly large weights saturate activations.
A dense (`Linear`) layer computes `Z = X @ W + b` and caches `X` during the forward pass so `dW = X.T @ dZ` can be computed during the backward pass.
Elementwise nonlinearities allow multi-layer networks to approximate non-linear decision boundaries.
Chaining layers transforms raw input features `(B, D_in)` into class logits `(B, C)`.
Loss Functions & Matrix Backpropagation
Implement stabilized Cross-Entropy / MSE losses, derive layer Jacobians, and code backward propagation.
Fusing Softmax and Cross-Entropy avoids computing `exp(z)` without max-subtraction and yields the remarkably clean gradient `(probs - one_hot) / B`.
Dimensional analysis confirms the matrix chain rule: for `Z = XW + b`, since `W` is `(D_in, D_out)`, `dW` must be `X.T @ dZ` (`(D_in, B) @ (B, D_out)`).
Propagating `d_logits` backward through `reversed(layers)` populates `layer.dW` and `layer.db` on every `Linear` layer.
Mini-Batch Optimization & Training Loop
Implement parameter updates, mini-batch shuffling, and loss/accuracy tracking on a classification dataset.
An optimizer iterates through all trainable layers and updates `W -= lr * dW` and `b -= lr * db`.
Non-linearly separable datasets (like concentric circles, spirals, two-moons, or multi-class digits) prove that your hidden ReLU layers learn non-linear representations.
Recording epoch-by-epoch training loss, validation loss, and classification accuracy verifies healthy learning dynamics.
Numerical Gradient Checking, PyTorch Comparison & Docs
Verify gradients numerically, compare step-for-step against PyTorch, and document results.
Perturbing each entry `W[i, j]` by `±1e-5` and comparing `(L_plus - L_minus) / (2e-5)` against `layer.dW[i, j]` catches any transpose or scaling bug.
Initializing a `torch.nn.Sequential` model with the exact same initial weight matrices `W` and biases `b` allows verifying that forward logits, loss, and backward `.grad` tensors match PyTorch to floating-point precision.
Clear documentation of the backward pass equations, gradient checks, and training curves completes the project.
Starter & Verification Code Examples
Dense Layer Forward & Backward Pass with Stabilized Softmax Cross-Entropy
Shows the exact NumPy matrix operations for a Linear layer and fused Softmax Cross-Entropy loss.
import numpy as np
class Linear:
def __init__(self, in_features: int, out_features: int, rng: np.random.Generator) -> None:
# He normal initialization
self.W = rng.normal(0.0, np.sqrt(2.0 / in_features), size=(in_features, out_features))
self.b = np.zeros((1, out_features), dtype=np.float64)
self.dW = np.zeros_like(self.W)
self.db = np.zeros_like(self.b)
self._cache_x: np.ndarray | None = None
def forward(self, x: np.ndarray) -> np.ndarray:
self._cache_x = x
return x @ self.W + self.b
def backward(self, dout: np.ndarray) -> np.ndarray:
assert self._cache_x is not None
self.dW = self._cache_x.T @ dout
self.db = np.sum(dout, axis=0, keepdims=True)
return dout @ self.W.T
def softmax_cross_entropy(logits: np.ndarray, y_indices: np.ndarray) -> tuple[float, np.ndarray]:
batch_size = logits.shape[0]
shifted = logits - np.max(logits, axis=1, keepdims=True)
exp_scores = np.exp(shifted)
probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)
loss = float(-np.mean(np.log(probs[np.arange(batch_size), y_indices] + 1e-12)))
dlogits = probs.copy()
dlogits[np.arange(batch_size), y_indices] -= 1.0
dlogits /= batch_size
return loss, dlogitsHow to Test Your Implementation
Run finite-difference gradient checks on every layer and verify exact forward/backward numerical parity against PyTorch.
Matrix Finite-Difference Gradient Check
Verify that `Linear.backward` and `SoftmaxCrossEntropyLoss.backward` compute exact analytical gradients.
Steps: For a random batch X (5, 4) and labels y (5,) through Linear(4, 6) -> ReLU -> Linear(6, 3), compare analytical dW1, db1, dW2, db2 with central finite differences (h=1e-5).
Pass Criteria: Relative error < 1e-6 across all weight and bias tensors.
Single-Step PyTorch Parity Test
Prove your NumPy forward logits, Cross-Entropy loss, and backward gradients match `torch.nn`.
Steps: Copy W and b into `torch.nn.Linear` (`linear_torch.weight.data = torch.from_numpy(W.T)`), run `F.cross_entropy` + `.backward()`, and compare gradients.
Pass Criteria: `np.allclose(dW_numpy, linear_torch.weight.grad.T, atol=1e-6)` is True.
Overall Project Acceptance Criteria
- Dense layers, activations, and Softmax Cross-Entropy loss implemented in pure NumPy.
- Analytical matrix gradients `dW`, `db`, and `dX` verified against numerical finite differences (`< 1e-6` relative error).
- Forward and backward passes verified for exact parity against PyTorch given identical initial weights.
- Network trained with mini-batch gradient descent to `> 95%` accuracy on a non-linear classification task.
Expandable Hints & Debugging Tips
Try implementing each stage on your own first. Expand a hint below if you want to verify tensor shapes, mathematical identities, or common edge cases.
Optional Stretch Challenges
Implement Adam & Momentum Optimizers in NumPy
Implement SGD with Momentum and the Adam optimizer (tracking first moment `m`, second moment `v`, and bias corrections `1 - β₁ᵗ`, `1 - β₂ᵗ`) and compare convergence speed against vanilla SGD.
What You Learn: How adaptive per-parameter learning rates accelerate training across ill-conditioned loss surfaces.
Inverted Dropout & Layer Normalization From Scratch
Implement `Dropout(p=0.2)` (scaling active neurons by `1/(1-p)` during training and identity during evaluation) and `LayerNorm(d_model)` with its backward pass.
What You Learn: How regularization and normalization layers behave differently in `train` vs `eval` modes.
What Your Completed Repository Should Contain
A fully working NumPy neural network implementation.
Deliverable Artifacts
- Modular NumPy neural network library (`Linear`, `ReLU`, `Sigmoid`, `SoftmaxCrossEntropyLoss`, `Sequential`, `SGD`)
- Numerical gradient checker and PyTorch parity test suite
- Training script and loss/decision-boundary visualizations
After Completing This Project, You Should Be Able To...
- Derive and implement vectorized forward and backward passes for dense layers, activations, and Softmax Cross-Entropy loss in NumPy.
- Initialize neural network weights using variance-preserving He and Xavier schemes.
- Verify analytical tensor gradients using finite differences and step-by-step PyTorch parity tests.
- Train a multi-layer neural network using mini-batch gradient descent to solve non-linear classification tasks.
GitHub Repository Submission
Link your public or private GitHub repository URL to keep a structured portfolio of your from-scratch AI builds. Stored locally in your browser (self-reported; no GitHub tokens or remote code execution).
Suggested Repository: neural-network-from-scratch-numpy
Recommended Directory Layout:
nn_numpy/layers.py nn_numpy/losses.py nn_numpy/optim.py tests/test_nn_numpy.py README.md
Recommended README Documentation Checklist
- Forward and backward matrix equations with tensor shapes
- Finite-difference and PyTorch parity verification output
- Training loss curve and accuracy results