Traditional ML Classification Metrics: Confusion Matrix, Accuracy, Precision, Recall & F1-Score

Traditional ML Evaluation Metrics — Foundations Guide

This document covers the metrics used to evaluate classical machine learning models, organized as classification, regression, and computer vision metrics.

Traditional ML
├── Classification  → Accuracy, Precision, Recall, F1, ROC-AUC
├── Regression       → MAE, MSE, RMSE, R²
└── Computer Vision  → IoU, Dice, mAP

#1. Classification Metrics

Classification metrics almost all derive from the confusion matrix, so start there.

1.1 The Confusion Matrix

For a binary classifier predicting "positive" vs "negative":

Example dataset: 100 emails, 20 are spam (positive class). A spam filter predicts 25 as spam; 18 of those are truly spam.

1.2 Accuracy

Definition: Fraction of total predictions that were correct.

Example: (18 + 73) / 100 = 0.91 (91%)

When to use: Balanced classes only. When to avoid: Imbalanced datasets — e.g., a fraud detector with 1% fraud rate gets 99% accuracy by predicting "not fraud" every time, which is useless.

from sklearn.metrics import accuracy_score
accuracy_score(y_true, y_pred)

1.3 Precision

Definition: Of everything predicted positive, how much was actually positive. Answers: "When the model says yes, how often is it right?"

Example: 18 / (18 + 7) = 0.72 (72%)

When to prioritize: False positives are costly. E.g., spam filter (don't want real emails marked spam), or a medical test that triggers invasive follow-up procedures on a false alarm.

from sklearn.metrics import precision_score
precision_score(y_true, y_pred)

1.4 Recall (Sensitivity / True Positive Rate)

Definition: Of everything that was actually positive, how much did the model catch. Answers: "Of all real positives, how many did we find?"

Example: 18 / (18 + 2) = 0.90 (90%)

When to prioritize: False negatives are costly. E.g., cancer screening, fraud detection, security threat detection — missing a real case is worse than a false alarm.

from sklearn.metrics import recall_score
recall_score(y_true, y_pred)

1.5 F1 Score

Definition: Harmonic mean of precision and recall — a single number that balances both. Harmonic mean (not arithmetic) punishes large imbalances between the two.

Example: 2 × (0.72 × 0.90) / (0.72 + 0.90) = 0.80

When to use: You need one summary metric and both false positives and false negatives matter, especially on imbalanced data. There is also a general F-beta score where β weights recall β times as important as precision:

from sklearn.metrics import f1_score
f1_score(y_true, y_pred)