Four numbers that accuracy hides
A confusion matrix is a small table of counts: for every row in your test set, it records what the true label was and what the model predicted. For a two-class problem that gives four numbers. True negatives and true positives are the agreements. False positives are false alarms, where the model shouted and nothing happened. False negatives are misses, where something happened and the model stayed quiet.
Accuracy collapses those four numbers into one, and in doing so throws away the only thing you usually care about: which kind of mistake the model prefers to make. A courier with 4,000 delivery attempts logged in a parcel-locker trial found this out the hard way. Their model scored 79.5% accuracy on held-out data. Always predicting “the delivery will succeed”, with no model at all, scores 78.7% on the same rows. The matrix shows why.
Building the example
import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, accuracy_score
rng = np.random.default_rng(75)
n = 4000
slot_hour = rng.integers(8, 20, n)
prior_failures = rng.poisson(0.6, n)
has_concierge = rng.binomial(1, 0.28, n)
block_flats = rng.binomial(1, 0.55, n)
logit = (-1.9
+ 0.11 * (slot_hour - 13)
+ 0.62 * prior_failures
- 1.35 * has_concierge
+ 0.48 * block_flats)
attempt_failed = rng.binomial(1, 1 / (1 + np.exp(-logit)))
deliveries = pd.DataFrame({
"slot_hour": slot_hour,
"prior_failures": prior_failures,
"has_concierge": has_concierge,
"block_flats": block_flats,
"attempt_failed": attempt_failed,
})
print("rows:", len(deliveries))
print("failure rate:", round(deliveries["attempt_failed"].mean(), 4))
# rows: 4000
# failure rate: 0.213
X = deliveries.drop(columns="attempt_failed")
y = deliveries["attempt_failed"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=0, stratify=y)
model = LogisticRegression(max_iter=1000, random_state=0).fit(X_train, y_train)
y_pred = model.predict(X_test)
cm = confusion_matrix(y_test, y_pred)
print(cm)
# [[929 15]
# [231 25]]
tn, fp, fn, tp = cm.ravel()
print("tn, fp, fn, tp =", tn, fp, fn, tp)
# tn, fp, fn, tp = 929 15 231 25
print("accuracy:", round(accuracy_score(y_test, y_pred), 4))
# accuracy: 0.795
A full description of attempt_failed and why a binary target beats a three-level one lives in defining the target variable. Here the target is given.
Reading a confusion matrix row by row
Rows are truth, columns are prediction. confusion_matrix sorts the labels numerically, so for a 0/1 target the top-left cell is true negatives and the bottom-right is true positives. That ordering trips people up constantly, because most textbook diagrams put positives first. If you remember one thing about the scikit-learn implementation, remember that cm.ravel() on a 2×2 result hands back tn, fp, fn, tp in that order, and that the official confusion_matrix reference says so explicitly.
Normalising by row turns counts into rates per true class, which is where the courier’s problem becomes obvious.
print(confusion_matrix(y_test, y_pred, normalize="true").round(3))
# [[0.984 0.016]
# [0.902 0.098]]
print("miss rate:", round(fn / (fn + tp), 3))
# miss rate: 0.902
print("false alarm rate:", round(fp / (fp + tn), 3))
# false alarm rate: 0.016
print("baseline accuracy, predict 'succeeds' always:",
round(1 - y_test.mean(), 4))
# baseline accuracy, predict 'succeeds' always: 0.7867
Of the 256 attempts that genuinely failed, the model flagged 25. It misses nine out of ten of the events it was built to predict, and the accuracy figure never mentions it. This is the same failure mode described in handling imbalanced classes, arriving here through the matrix rather than through the class ratio.
Two things worth noticing. The false alarm rate is tiny, 1.6%, because the model almost never predicts failure at all. And the model is not broken: its probability estimates may well be good. What is broken is the decision rule bolted onto them, which labels a row positive only when the predicted probability clears 0.5.
The matrix moves when you move the threshold
The confusion matrix is not a property of the model. It is a property of the model plus a cut-off. Change the cut-off and all four cells change, which means you can trade misses for false alarms directly. For the courier the exchange rate is known: diverting a parcel to a locker costs about £1.20 in handling, and a failed attempt costs £4.80 in redelivery.
proba = model.predict_proba(X_test)[:, 1]
rows = []
for t in [0.50, 0.40, 0.30, 0.25, 0.20, 0.15, 0.10]:
tn_t, fp_t, fn_t, tp_t = confusion_matrix(
y_test, (proba >= t).astype(int), labels=[0, 1]).ravel()
rows.append((t, tn_t, fp_t, fn_t, tp_t,
round((tn_t + tp_t) / len(y_test), 3),
round(1.20 * fp_t + 4.80 * fn_t, 2)))
print(pd.DataFrame(rows, columns=["thr", "TN", "FP", "FN", "TP",
"acc", "cost_gbp"]).to_string(index=False))
# thr TN FP FN TP acc cost_gbp
# 0.50 929 15 231 25 0.795 1126.8
# 0.40 897 47 208 48 0.788 1054.8
# 0.30 778 166 146 110 0.740 900.0
# 0.25 685 259 111 145 0.692 843.6
# 0.20 548 396 83 173 0.601 873.6
# 0.15 389 555 46 210 0.499 886.8
# 0.10 232 712 15 241 0.394 926.4
The cheapest rule on these 1,200 rows sits at a threshold of 0.25, where accuracy has fallen to 69.2%. Accuracy drops ten points and the bill drops by £283. If you tuned this model by accuracy you would ship the most expensive version of it.
| Threshold | What the matrix looks like | When it is the right call |
|---|---|---|
| 0.50 | 15 false alarms, 231 misses | Only if a false alarm is far dearer than a miss |
| 0.25 | 259 false alarms, 111 misses | Lowest cost here, and what I would deploy |
| 0.10 | 712 false alarms, 15 misses | A near-free intervention, or a legal duty to catch everything |
Where the matrix stops being enough
Three limits are worth stating plainly.
The matrix is computed on one sample, so each cell carries sampling noise. With 25 true positives in the default-threshold matrix, a handful of rows moving either way shifts the miss rate by whole percentage points. Averaging matrices across folds, as described in cross-validation strategies, is more honest than one matrix from one split.
Four raw counts do not compare well across datasets with different base rates. That is what the ratios in the next chapter are for.
And for more than two classes the table grows to k by k, where the interesting structure is which classes get mistaken for which. That is covered in multi-class classification, and the reading rule stays the same: rows are truth, columns are prediction.
Frequently Asked Questions
Why is my confusion matrix in a different order than the one in my textbook?
Scikit-learn orders labels by sorted value, so with a 0/1 target true negatives land top-left and true positives bottom-right. Many textbooks and most medical literature put the positive class first, which flips the whole table. Pass labels=[1, 0] if you need their layout.
Can I get a confusion matrix from predicted probabilities?
Not directly. You have to pick a threshold first, because the matrix counts hard decisions. Call predict_proba, compare against your chosen cut-off, then pass the resulting 0/1 array in. The default predict method is just this with the cut-off fixed at 0.5.
Should the matrix come from the validation set or the test set?
Use validation data while you are still choosing a threshold, since choosing one on test data leaks the test set into your decisions. Report the final matrix on test data once, at the end. The split discipline is set out in testing and validation.
Key Takeaways
- Read the confusion matrix before any single-number metric, because it is the only view that shows you which of the two mistakes your model is making.
- Remember that
cm.ravel()returnstn, fp, fn, tpfor a 0/1 target, since the scikit-learn ordering differs from most textbook diagrams. - Treat the matrix as a property of the model plus a threshold, not of the model alone, and recompute it whenever you move the cut-off.
- Attach different costs to false positives and false negatives before you tune anything, or you will optimise accuracy and ship the expensive model by mistake.
- Normalise by row when classes are unbalanced, because raw counts in a rare-event problem make a useless classifier look respectable.