Linear Regression From Scratch
Derive analytical gradients and train a regression model via gradient descent
What You Will Build
Linear regression is the foundational prototype of supervised machine learning: you define a parameterized hypothesis `ŷ = Xw + b`, measure prediction error with a differentiable loss function (Mean Squared Error), derive the partial derivatives `∂L/∂w` and `∂L/∂b` using multivariable calculus, and iteratively step weights opposite the gradient until the loss converges. In this project, you will implement the entire training, validation, and diagnostic visualization pipeline from scratch using Python, NumPy, and Matplotlib—without calling `sklearn.linear_model` during training.
Why This Project Matters in the AI/LLM Stack
The training loop you build here—forward prediction, scalar loss computation, analytical gradient computation, parameter update `θ ← θ - η∇L`, and held-out validation tracking—is identical in structure to the training loop of a 70-billion-parameter language model.
Learning Objectives
- Derive and implement the exact vectorized gradients of Mean Squared Error with respect to weights and bias.
- Build an iterative gradient descent optimizer and diagnose convergence, underfitting, and divergence across learning rates.
- Plot training and validation loss curves and regression fit lines using Matplotlib.
- Verify your trained parameters and held-out MSE/R² metrics against a reference library implementation.
Prerequisites
- Project 1 (Matrix Operations Engine) or fluency with NumPy array shapes and dot products
- Partial derivatives and the chain rule from Phase 1 Calculus
Required Mathematical Concepts
- Linear hypothesis function: ŷ = Xw + b
- Mean Squared Error (MSE): L(w, b) = (1/N) ∑ᵢ (ŷᵢ - yᵢ)²
- Analytical weight gradient: ∂L/∂w = (2/N) Xᵀ(Xw + b - y)
- Analytical bias gradient: ∂L/∂b = (2/N) ∑ᵢ (ŷᵢ - yᵢ)
- Gradient descent update rule and learning rate η sensitivity
- Train/validation split and feature standardization
Step-by-Step Implementation Tasks (12 Steps)
Check off each engineering task as your code passes its acceptance criteria. Progress is saved automatically in your browser.
Data Generation, Exploration & Visualization
Create or load a regression dataset, split into train/validation sets, and visualize feature-target relationships.
Starting with a synthetic dataset with known ground-truth weights `w_true` and Gaussian noise `ε` lets you verify whether your optimizer recovers the exact underlying parameters.
Inspecting raw scatter plots reveals scale differences, outliers, and linear trends before training begins.
Forward Model, MSE Loss & Manual Gradient Derivation
Implement vectorized linear predictions, MSE loss, and exact analytical gradients.
The linear hypothesis computes `ŷ = X @ w + b` across a batch of `N` examples and `D` features simultaneously.
Mean Squared Error `L = (1/N) ∑ (ŷ_i - y_i)²` penalizes large residuals quadratically and has a unique global minimum for linear models.
Applying the chain rule to `L(w, b) = (1/N) ||Xw + b - y||₂²` yields the exact gradient vector `dw` and scalar `db`.
Gradient Descent Training & Loss Curve Diagnostics
Implement the gradient descent loop, train the model, and plot training/validation loss convergence.
Gradient descent iteratively updates parameters opposite the gradient: `w ← w - lr * dw`, `b ← b - lr * db`.
Running the training loop to convergence on standardized features recovers the true linear relationship.
Loss curves immediately reveal whether optimization converged smoothly, plateaued early, or diverged.
Held-Out Evaluation, Library Comparison & LR Experiments
Evaluate on unseen test data, compare against closed-form/scikit-learn baselines, test learning rate regimes, and document derivations.
Reporting MSE, Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and R² coefficient of determination on unseen data measures true generalization.
Comparing your gradient-descent solution with the closed-form Normal Equation `w = (XᵀX)⁻¹Xᵀy` or `np.linalg.lstsq` / `scikit-learn` confirms your optimizer reached the global optimum.
If `η` is too small, training crawls; if `η > 2 / λ_max(XᵀX/N)`, gradient descent oscillates and diverges to `NaN`/`Inf`.
Writing out the matrix calculus derivation cements your understanding before moving to multilayer backpropagation.
Starter & Verification Code Examples
Vectorized MSE Loss & Analytical Gradients in NumPy
Shows the exact vectorized forward pass, MSE computation, and analytical gradient equations.
import numpy as np
def forward_and_gradients(
X: np.ndarray, y: np.ndarray, w: np.ndarray, b: float
) -> tuple[float, np.ndarray, float]:
"""
X: (N, D) feature matrix
y: (N,) target vector
w: (D,) weight vector
b: scalar bias
"""
n_samples = X.shape[0]
y_pred = X @ w + b # Shape: (N,)
residuals = y_pred - y # Shape: (N,)
loss = float(np.mean(residuals ** 2)) # Scalar MSE
dw = (2.0 / n_samples) * (X.T @ residuals) # Shape: (D,)
db = (2.0 / n_samples) * float(np.sum(residuals))
return loss, dw, dbHow to Test Your Implementation
Verify analytical gradient accuracy via finite differences and assert that gradient descent recovers exact synthetic parameters.
Finite-Difference Gradient Verification
Prove that analytical dw and db match numerical central differences.
Steps: Perturb each weight w[j] by ±eps (eps=1e-6), compute (L_plus - L_minus)/(2*eps), and compare with dw[j].
Pass Criteria: Relative error between analytical and numerical gradients < 1e-7.
Noiseless Parameter Recovery Test
Confirm optimizer converges to exact ground-truth weights when noise is zero.
Steps: Generate y = X @ [3.5, -1.2] + 0.75 with zero noise. Train for 800 epochs at lr=0.1.
Pass Criteria: Recovered w matches [3.5, -1.2] and b matches 0.75 within 1e-4; MSE < 1e-8.
Overall Project Acceptance Criteria
- Linear prediction, MSE loss, and gradient descent implemented from scratch without predefined ML estimators.
- Analytical gradients verified against numerical finite differences within 1e-7.
- Loss curve and regression fit visualizations generated and saved.
- Trained weights and test MSE verified against closed-form lstsq or scikit-learn baseline.
Expandable Hints & Debugging Tips
Try implementing each stage on your own first. Expand a hint below if you want to verify tensor shapes, mathematical identities, or common edge cases.
Optional Stretch Challenges
Add L2 (Ridge) and L1 (Lasso) Regularization
Extend your loss and gradient functions to support L2 penalty `λ||w||₂²` (`dw += 2λw`) and L1 subgradient penalty `λ||w||₁` (`dw += λ * np.sign(w)`).
What You Learn: How weight decay prevents overfitting on collinear or high-dimensional features.
Mini-Batch Stochastic Gradient Descent
Implement shuffled mini-batch iteration (`batch_size=32`) and compare its noisy loss trajectory against full-batch gradient descent.
What You Learn: How stochastic mini-batching works in deep neural network training loops.
What Your Completed Repository Should Contain
A complete linear regression training pipeline.
Deliverable Artifacts
- From-scratch NumPy linear regression trainer with gradient checking
- Diagnostic loss curve and learning-rate comparison plots
- Evaluation report comparing custom gradient descent against closed-form/library baseline
After Completing This Project, You Should Be Able To...
- Derive and vectorize analytical gradients of Mean Squared Error loss with respect to weights and biases.
- Implement gradient descent from scratch and verify gradients using central finite differences.
- Diagnose learning-rate instability, feature scaling effects, and train vs. validation generalization.
- Compare iterative gradient-based optimization against closed-form linear algebra solutions.
GitHub Repository Submission
Link your public or private GitHub repository URL to keep a structured portfolio of your from-scratch AI builds. Stored locally in your browser (self-reported; no GitHub tokens or remote code execution).
Suggested Repository: linear-regression-from-scratch
Recommended Directory Layout:
regression/model.py regression/metrics.py plots/loss_curve.png tests/test_linear_regression.py README.md
Recommended README Documentation Checklist
- Mathematical derivation of ∂L/∂w and ∂L/∂b
- Loss convergence plot and learning-rate sensitivity chart
- Held-out MSE/R² comparison table against closed-form / scikit-learn