Matrix Operations Engine
Build a pure-Python linear algebra library verified against NumPy
What You Will Build
Every forward and backward pass in modern deep learning and Large Language Models is built on vector dot products, matrix transpositions, and matrix multiplications. Before relying on C++/CUDA-accelerated libraries like NumPy or PyTorch, you will implement a clean, reusable linear algebra library in pure Python from first principles. You will design explicit Vector and Matrix data structures, enforce strict shape invariants, implement O(n³) matrix multiplication with nested loops, and write an automated test suite comparing your outputs against NumPy within floating-point tolerances.
Why This Project Matters in the AI/LLM Stack
When debugging tensor shape mismatches in multi-head self-attention or backpropagation, engineers who have written matrix multiplication and transposition from scratch can immediately trace which inner dimensions must align and why incompatible shapes fail.
Learning Objectives
- Design clean, immutable or well-validated Vector and Matrix classes in pure Python.
- Enforce strict dimensionality checks and raise descriptive ValueError exceptions on shape mismatches.
- Implement vector addition, scalar multiplication, dot products, matrix addition, transpose, and matrix multiplication without external libraries.
- Verify numerical correctness across edge cases (rectangular, square, identity, and zero matrices) against NumPy.
Prerequisites
- Core Python syntax: lists, nested loops, functions, classes, and custom exceptions
- Basic understanding of 1D arrays (vectors) and 2D grids (matrices)
Required Mathematical Concepts
- Vectors and vector dimensions (ℝⁿ)
- Matrices and matrix dimensions (ℝᵐˣⁿ)
- Elementwise addition, subtraction, and scalar multiplication
- Inner (dot) product of two equal-length vectors
- Matrix multiplication rule (Cᵢⱼ = ∑ₖ Aᵢₖ Bₖⱼ) and inner dimension compatibility
- Matrix transposition ((Aᵀ)ᵢⱼ = Aⱼᵢ)
- Floating-point precision and relative/absolute numerical tolerance
Step-by-Step Implementation Tasks (12 Steps)
Check off each engineering task as your code passes its acceptance criteria. Progress is saved automatically in your browser.
Core Data Structures & Shape Validation
Define Vector and Matrix representations and validate row/column regularity.
A reliable linear algebra library starts with clear representations for 1D vectors (`list[float]`) and 2D matrices (`list[list[float]]`) that expose `.shape` properties.
Jagged nested lists like `[[1, 2], [3]]` are not valid matrices and will cause index errors during multiplication.
Vector Operations & Inner Products
Implement elementwise vector arithmetic, scalar scaling, and dot products.
Vector addition combines two vectors of equal dimension component-by-component: `(u + v)_i = u_i + v_i`. It is used in bias addition and residual connections.
Multiplying a vector or matrix by a scalar `alpha` scales every entry by `alpha` without changing its dimensions.
The dot product `u · v = ∑ u_i * v_i` measures alignment between two vectors and forms the core computation of every artificial neuron and matrix multiply.
Matrix Arithmetic, Transpose & Multiplication
Build 2D matrix addition, transposition, and pure-Python matrix multiplication with dimension guards.
Two matrices `A` and `B` can be added or subtracted if and only if they have identical `(rows, cols)` dimensions.
Transposing an `(m, n)` matrix flips its rows and columns into an `(n, m)` matrix where `(Aᵀ)_{j,i} = A_{i,j}`.
For `A` of shape `(m, k)` and `B` of shape `(k, n)`, entry `(i, j)` of `C = A @ B` is the dot product of row `i` of `A` with column `j` of `B`, producing an `(m, n)` matrix.
Clear dimension error messages save hours of debugging when chaining linear transformations.
NumPy Verification, Unit Testing & Documentation
Verify outputs against NumPy, build a comprehensive pytest suite, and document your API.
Floating-point arithmetic can differ by ~1e-15 due to operation ordering. Comparing against `numpy` with `np.allclose` validates numerical accuracy.
A structured test suite ensures edge cases remain unbroken as you extend your library.
Clear docstrings and complexity notes make your library reusable in later phases.
Starter & Verification Code Examples
Shape Validation & Pure-Python Matrix Multiplication Skeleton
Demonstrates how to validate 2D matrix regularity and structure the inner dimension check for matrix multiplication.
class Matrix:
def __init__(self, rows: list[list[float]]) -> None:
if not rows or not rows[0]:
raise ValueError("Matrix must contain at least one non-empty row.")
num_cols = len(rows[0])
cleaned: list[list[float]] = []
for r_idx, row in enumerate(rows):
if len(row) != num_cols:
raise ValueError(
f"Row {r_idx} has length {len(row)}, expected {num_cols}."
)
cleaned.append([float(val) for val in row])
self._data = cleaned
self.shape = (len(cleaned), num_cols)
def matmul(self, other: "Matrix") -> "Matrix":
m, k1 = self.shape
k2, n = other.shape
if k1 != k2:
raise ValueError(
f"Incompatible shapes for matmul: {self.shape} and {other.shape} (inner dims {k1} != {k2})"
)
# Compute C[i][j] = sum_p A[i][p] * B[p][j]
result = [
[
sum(self._data[i][p] * other._data[p][j] for p in range(k1))
for j in range(n)
]
for i in range(m)
]
return Matrix(result)How to Test Your Implementation
Run your pytest suite to verify algebraic identities, shape error handling, and floating-point equivalence against NumPy.
Identity Matrix Multiplication Invariant
Verify that multiplying an (m, n) matrix A by I_n on the right and I_m on the left returns A.
Steps: Construct a (3, 5) matrix A and 5x5 identity matrix I_5. Compute matmul(A, I_5) and assert equality with A.
Pass Criteria: Output shape is (3, 5) and all elements exactly equal A.
Incompatible Inner Dimensions Guard
Confirm that matmul((2, 3), (2, 3)) fails cleanly before partial computation.
Steps: Call matmul(Matrix([[1, 2, 3], [4, 5, 6]]), Matrix([[1, 2, 3], [4, 5, 6]])) inside pytest.raises(ValueError).
Pass Criteria: Raises ValueError mentioning inner dimensions 3 and 2.
Rectangular Matrix Multiplication vs NumPy
Validate triple-loop indexing on non-square dimensions (4, 7) @ (7, 2).
Steps: Generate random floats for A (4, 7) and B (7, 2), compute matmul(A, B), and compare with np.matmul(A_np, B_np).
Pass Criteria: Maximum absolute difference < 1e-12.
Overall Project Acceptance Criteria
- Correct results for all valid vector and matrix input shapes.
- Clear, descriptive ValueError exceptions raised for any incompatible shapes.
- Automated unit tests covering rectangular, square, identity, and zero matrices.
- Numerical verification against NumPy passing with `np.testing.assert_allclose`.
Expandable Hints & Debugging Tips
Try implementing each stage on your own first. Expand a hint below if you want to verify tensor shapes, mathematical identities, or common edge cases.
Optional Stretch Challenges
Support Batched 3D Matrix Multiplication
Extend your engine to multiply batched 3D tensors of shape (B, M, K) and (B, K, N) producing (B, M, N), mirroring how Transformer attention processes batches of sequences.
What You Learn: How batch dimensions wrap standard 2D linear algebra operations in deep learning.
Vector Norms & Cosine Similarity
Implement L1 norm, L2 Euclidean norm, and cosine similarity `cos(θ) = (u · v) / (||u||_2 ||v||_2)` with zero-norm guards.
What You Learn: How embedding similarity search and vector databases compare semantic representations.
What Your Completed Repository Should Contain
A reusable Python matrix operations library.
Deliverable Artifacts
- Pure-Python `linalg` module with `Vector` and `Matrix` classes
- Automated `pytest` verification suite comparing outputs with NumPy
- README documentation with API examples and complexity analysis
After Completing This Project, You Should Be Able To...
- Implement vector addition, scalar multiplication, dot products, transposes, and matrix multiplication from scratch in pure Python.
- Predict output dimensions and diagnose shape mismatches across chained linear transformations.
- Write automated numerical unit tests using floating-point tolerances against a reference library.
- Explain why contiguous C memory buffers in NumPy outperform pure-Python nested loops.
GitHub Repository Submission
Link your public or private GitHub repository URL to keep a structured portfolio of your from-scratch AI builds. Stored locally in your browser (self-reported; no GitHub tokens or remote code execution).
Suggested Repository: matrix-operations-engine
Recommended Directory Layout:
linalg/__init__.py linalg/core.py tests/test_linalg.py README.md
Recommended README Documentation Checklist
- Overview of implemented vector and matrix operations
- How to run the pytest verification suite against NumPy
- Time complexity table and pure-Python vs NumPy timing comparison