Spam Classifier
Build an end-to-end text classification pipeline with Precision, Recall, and F1 analysis
What You Will Build
Before deep neural networks, statistical text classification powered spam filters, document routing, and sentiment analysis—and its core concepts (tokenization, vocabulary indexing, log-probabilities, binary cross-entropy, and Precision/Recall tradeoffs) remain essential in modern LLM pipelines. In this project, you will build a complete spam classification system from raw text messages to calibrated probability predictions. You will preprocess text, extract Bag-of-Words or TF-IDF features, implement Multinomial Naive Bayes (with Laplace smoothing in log-space) or binary Logistic Regression from scratch in NumPy, evaluate Precision, Recall, and F1 on a stratified test split, inspect false positives and false negatives, and benchmark against `scikit-learn`.
Why This Project Matters in the AI/LLM Stack
Accuracy alone is misleading on imbalanced datasets (if 87% of messages are legitimate ham, predicting 'ham' every time gets 87% accuracy with 0% spam recall). Building a spam filter teaches you how to build leak-free text vocabularies, prevent underflow with log-probabilities, and tune decision thresholds using Precision and Recall.
Learning Objectives
- Inspect class imbalance and construct a stratified train/validation/test split without vocabulary leakage.
- Convert variable-length text messages into numerical Bag-of-Words count vectors or TF-IDF matrices.
- Implement Multinomial Naive Bayes or binary Logistic Regression from scratch in NumPy.
- Compute Precision, Recall, F1, and confusion matrices, analyze misclassified messages, and compare with a scikit-learn baseline.
Prerequisites
- Phase 1 Probability & Statistics (conditional probability, Bayes' theorem, log-likelihood)
- Phase 2 Supervised Learning fundamentals (train/test splits, overfitting, classification metrics)
Required Mathematical Concepts
- Bayes' Theorem and conditional independence: P(y | x₁,…,x_T) ∝ P(y) ∏_t P(x_t | y)
- Log-space computation to prevent floating-point underflow: log P(y) + ∑_t log P(x_t | y)
- Laplace (add-α) smoothing for unseen vocabulary words
- Logistic sigmoid σ(z) = 1 / (1 + e⁻ᶻ) and Binary Cross-Entropy loss
- Confusion Matrix (TP, FP, TN, FN), Precision, Recall, and F1 harmonic mean
Step-by-Step Implementation Tasks (11 Steps)
Check off each engineering task as your code passes its acceptance criteria. Progress is saved automatically in your browser.
Dataset Curation, Class Distribution & Text Preprocessing
Load a labeled SMS/email dataset, inspect class ratios, and build a clean text tokenizer.
Public benchmark datasets like the UCI SMS Spam Collection (CC BY 4.0 / public domain research corpus) provide thousands of real labeled `ham` and `spam` messages.
Real-world spam datasets are typically skewed (e.g., ~86.6% ham vs. ~13.4% spam), requiring stratified splitting and non-accuracy metrics.
Consistent normalization prevents `'FREE!'`, `'free'`, and `'Free'` from being treated as unrelated features while preserving spam-indicative tokens like currency symbols or URLs.
Feature Extraction & From-Scratch Classifier Implementation
Build leak-free Bag-of-Words/TF-IDF features and implement your classifier from scratch.
Classifiers operate on numerical vectors `x ∈ ℝ^V`. Building the vocabulary strictly on `X_train` prevents test-set data leakage.
You may implement either Multinomial Naive Bayes (computing log class priors and Laplace-smoothed log word likelihoods) or binary Logistic Regression (`ŷ = σ(Xw + b)` with BCE gradient descent) from scratch in NumPy.
Fitting the model on the vectorized training corpus estimates the spam vs. ham token distributions or feature weights.
Converting raw scores into calibrated probabilities allows adjusting the classification threshold (`0.5` vs `0.8`) to prioritize high precision.
Evaluation Metrics, Error Analysis & Baseline Comparison
Compute Precision/Recall/F1, inspect false positives/negatives, compare against scikit-learn, and document limitations.
In spam filtering, a False Positive (legitimate email sent to spam) is often more costly than a False Negative.
Reading the actual messages your model got wrong reveals dataset quirks, sarcasm, short messages, or missing features.
Benchmarking against `sklearn.naive_bayes.MultinomialNB` or `sklearn.linear_model.LogisticRegression` verifies your implementation's statistical performance.
Bag-of-Words models ignore word order (`'not free'` vs `'free'`), context, and adversarial character substitutions (`'fr33'`).
Starter & Verification Code Examples
Log-Space Multinomial Naive Bayes with Laplace Smoothing
Shows how to estimate class priors and Laplace-smoothed token log-likelihoods in vectorized NumPy without floating-point underflow.
import numpy as np
class MultinomialNBScratch:
def __init__(self, alpha: float = 1.0) -> None:
self.alpha = alpha
self.log_priors: np.ndarray | None = None # Shape: (2,)
self.log_likelihoods: np.ndarray | None = None # Shape: (2, V)
def fit(self, X_counts: np.ndarray, y: np.ndarray) -> "MultinomialNBScratch":
n_samples, vocab_size = X_counts.shape
self.log_priors = np.zeros(2, dtype=np.float64)
self.log_likelihoods = np.zeros((2, vocab_size), dtype=np.float64)
for c in (0, 1):
X_c = X_counts[y == c]
self.log_priors[c] = np.log(X_c.shape[0] / n_samples)
word_counts = np.sum(X_c, axis=0) + self.alpha
total_words = np.sum(word_counts)
self.log_likelihoods[c] = np.log(word_counts / total_words)
return self
def predict_log_scores(self, X_counts: np.ndarray) -> np.ndarray:
assert self.log_priors is not None and self.log_likelihoods is not None
# (N, V) @ (V, 2) + (2,) -> (N, 2)
return X_counts @ self.log_likelihoods.T + self.log_priorsHow to Test Your Implementation
Test tokenizer normalization, Laplace smoothing on unseen words, and Precision/Recall/F1 formulas on known synthetic confusion matrices.
Unseen Vocabulary Token Stability (Laplace Smoothing)
Ensure messages containing words never seen in training do not produce -inf or NaN probabilities.
Steps: Fit classifier on a small toy corpus, then call predict_proba on a message containing 100% novel words.
Pass Criteria: Returns finite probabilities in (0, 1) without ZeroDivisionError or NaN.
Precision, Recall, and F1 Exact Calculation
Verify metric formulas on a known ground-truth vector.
Steps: Pass y_true = [1, 1, 1, 0, 0] and y_pred = [1, 1, 0, 1, 0] (TP=2, FN=1, FP=1, TN=1).
Pass Criteria: Precision = 2/3 (0.6667), Recall = 2/3 (0.6667), F1 = 2/3 (0.6667).
Overall Project Acceptance Criteria
- Leak-free vocabulary and feature extraction fit strictly on the training split.
- Classifier implemented from scratch in NumPy without calling scikit-learn estimators.
- Precision, Recall, F1, and confusion matrix calculated and verified.
- Misclassification analysis and scikit-learn baseline comparison documented.
Expandable Hints & Debugging Tips
Try implementing each stage on your own first. Expand a hint below if you want to verify tensor shapes, mathematical identities, or common edge cases.
Optional Stretch Challenges
Bigram Features (Character or Word N-Grams)
Extend your vectorizer to extract word bigrams (`n=2`, e.g., `'not free'`, `'claim now'`) and measure the resulting F1 improvement.
What You Learn: How local n-gram context captures negation and multi-word phrases.
Precision-Recall Curve & Threshold Sweeping
Sweep decision thresholds from `0.05` to `0.95` and plot the Precision-Recall curve to find the threshold that achieves `>= 99%` precision.
What You Learn: How production classifiers tune operating thresholds for asymmetric business costs.
What Your Completed Repository Should Contain
A working text classification system with evaluation results.
Deliverable Artifacts
- Text tokenizer and leak-free Bag-of-Words / TF-IDF vectorizer
- From-scratch Naive Bayes or Logistic Regression classifier in NumPy
- Evaluation report with Confusion Matrix, Precision, Recall, F1, and misclassification analysis
After Completing This Project, You Should Be Able To...
- Preprocess raw text and construct leak-free numerical feature matrices using Bag-of-Words and TF-IDF.
- Implement probabilistic classification in log-space with Laplace smoothing or Binary Cross-Entropy optimization.
- Evaluate classifiers on imbalanced data using Confusion Matrices, Precision, Recall, and F1.
- Perform qualitative error analysis on false positives and false negatives to identify model limitations.
GitHub Repository Submission
Link your public or private GitHub repository URL to keep a structured portfolio of your from-scratch AI builds. Stored locally in your browser (self-reported; no GitHub tokens or remote code execution).
Suggested Repository: spam-classifier-from-scratch
Recommended Directory Layout:
spam_filter/vectorizer.py spam_filter/classifier.py spam_filter/metrics.py tests/test_spam_classifier.py README.md
Recommended README Documentation Checklist
- Dataset attribution and class balance summary
- Precision, Recall, and F1 comparison table (Scratch vs scikit-learn)
- Error analysis of false positives and false negatives