Two questions, asked of the same column
Precision and recall are both ratios built from the positive column and the positive row of a confusion matrix. Precision asks: of everything the model flagged, what fraction was real? Recall asks: of everything that was real, what fraction did the model flag? They answer to different people. Precision belongs to whoever has to act on the flags, because false alarms waste their time. Recall belongs to whoever bears the cost of the events you missed.
A water utility gave me a good case. They shortlist 6,000 household meters a quarter on a crude night-flow alarm, send inspectors to some of them, and about 23% of the shortlist turns out to have a real leak. Each inspection visit costs £38. A leak left running for a quarter costs roughly £520 in lost water and eventual repair. Those two numbers are what decide which of precision and recall matters more, and no metric computed from labels alone can tell you.
The two rates, by hand and by scikit-learn
import numpy as np, pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import (confusion_matrix, classification_report,
precision_score, recall_score, f1_score, fbeta_score)
rng = np.random.default_rng(76)
n = 6000
night_flow = np.round(rng.gamma(2.0, 9.0, n), 1)
pressure_var = np.round(rng.normal(0.8, 0.3, n).clip(0.05), 3)
pipe_age_years = rng.integers(2, 71, n)
meter_reads_gap = rng.poisson(1.4, n)
logit = (-6.8
+ 0.090 * night_flow
+ 2.40 * pressure_var
+ 0.040 * pipe_age_years
+ 0.10 * meter_reads_gap)
leak_confirmed = rng.binomial(1, 1 / (1 + np.exp(-logit)))
meters = pd.DataFrame({
"night_flow_litres": night_flow, "pressure_var": pressure_var,
"pipe_age_years": pipe_age_years, "meter_reads_gap": meter_reads_gap,
"leak_confirmed": leak_confirmed,
})
print("rows:", len(meters), "leak rate:", round(meters["leak_confirmed"].mean(), 4))
# rows: 6000 leak rate: 0.23
X = meters.drop(columns="leak_confirmed")
y = meters["leak_confirmed"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3,
random_state=0, stratify=y)
clf = RandomForestClassifier(n_estimators=300, min_samples_leaf=5,
random_state=0).fit(X_tr, y_tr)
pred = clf.predict(X_te)
tn, fp, fn, tp = confusion_matrix(y_te, pred).ravel()
print("tn, fp, fn, tp =", tn, fp, fn, tp)
# tn, fp, fn, tp = 1297 89 252 162
print("precision by hand:", round(tp / (tp + fp), 4))
# precision by hand: 0.6454
print("recall by hand: ", round(tp / (tp + fn), 4))
# recall by hand: 0.3913
print("f1_score:", round(f1_score(y_te, pred), 4))
# f1_score: 0.4872
print(classification_report(y_te, pred, digits=3))
# precision recall f1-score support
# 0 0.837 0.936 0.884 1386
# 1 0.645 0.391 0.487 414
# accuracy 0.811 1800
# macro avg 0.741 0.664 0.686 1800
# weighted avg 0.793 0.811 0.793 1800
Two of every three visits find a leak, and six of every ten leaks go unfound. Read classification_report carefully: it prints both classes, and the row for class 0 looks far healthier simply because negatives are the majority. The macro avg row averages the two classes with equal weight, the weighted avg row weights by support. For a rare-positive problem the macro row is the one that moves when your model gets better at the thing you care about.
Why F1 is a harmonic mean and not an average
F1 is the harmonic mean of precision and recall: 2PR / (P + R). The harmonic mean is dragged down hard by whichever of the two is smaller, which is the whole point. An arithmetic average lets a model buy a respectable score with one extreme.
for p_, r_ in [(0.90, 0.10), (0.50, 0.50), (0.65, 0.39), (0.30, 0.95)]:
print(f"P={p_:.2f} R={r_:.2f} mean={(p_ + r_) / 2:.3f}"
f" F1={2 * p_ * r_ / (p_ + r_):.3f}")
# P=0.90 R=0.10 mean=0.500 F1=0.180
# P=0.50 R=0.50 mean=0.500 F1=0.500
# P=0.65 R=0.39 mean=0.520 F1=0.487
# P=0.30 R=0.95 mean=0.625 F1=0.456
print("F0.5:", round(fbeta_score(y_te, pred, beta=0.5), 4))
# F0.5: 0.5712
print("F2: ", round(fbeta_score(y_te, pred, beta=2.0), 4))
# F2: 0.4248
The first and second rows score identically on an arithmetic average and are not remotely comparable as classifiers. F1 separates them by 32 points. When you genuinely want to lean one way, fbeta_score takes a beta: values above 1 weight recall, values below 1 weight precision. Beta is the ratio of how much you value recall over precision, so beta=2 says a miss hurts twice as much as a false alarm. Pick it from your costs, not from a blog post.
| Metric | Formula | Use it when |
|---|---|---|
| Precision | TP / (TP + FP) |
Acting on a flag is expensive or intrusive |
| Recall | TP / (TP + FN) |
Missing a positive is the expensive outcome |
| F1 | 2PR / (P + R) |
You need one number and the two costs are roughly equal |
| F-beta | weighted harmonic mean | You know the cost ratio and can set beta |
Where precision and recall beat F1 on a real bill
This is the part most tutorials skip. F1 treats a false positive and a false negative as equally bad. The utility’s own numbers say a miss is nearly fourteen times worse than a wasted visit. So the threshold that maximises F1 and the threshold that minimises cost are different thresholds, and following F1 is simply the wrong call.
proba = clf.predict_proba(X_te)[:, 1]
VISIT, MISSED_LEAK = 38, 520
rows = []
for t in [0.02, 0.05, 0.10, 0.15, 0.20, 0.25, 0.35, 0.50, 0.65]:
tn_t, fp_t, fn_t, tp_t = confusion_matrix(
y_te, (proba >= t).astype(int), labels=[0, 1]).ravel()
prec, rec = tp_t / (tp_t + fp_t), tp_t / (tp_t + fn_t)
rows.append((t, tp_t, fp_t, fn_t, round(prec, 3), round(rec, 3),
round(2 * prec * rec / (prec + rec), 3),
VISIT * (tp_t + fp_t) + MISSED_LEAK * fn_t))
sweep = pd.DataFrame(rows, columns=["thr", "TP", "FP", "FN",
"precision", "recall", "F1", "cost_gbp"])
print(sweep.to_string(index=False))
# thr TP FP FN precision recall F1 cost_gbp
# 0.02 410 1176 4 0.259 0.990 0.410 62348
# 0.05 387 910 27 0.298 0.935 0.452 63326
# 0.10 363 674 51 0.350 0.877 0.500 65926
# 0.15 345 509 69 0.404 0.833 0.544 68332
# 0.20 314 393 100 0.444 0.758 0.560 78866
# 0.25 290 300 124 0.492 0.700 0.578 86900
# 0.35 231 196 183 0.541 0.558 0.549 111386
# 0.50 162 89 252 0.645 0.391 0.487 140578
# 0.65 114 28 300 0.803 0.275 0.410 161396
print("best F1 at threshold ", sweep.loc[sweep["F1"].idxmax(), "thr"])
# best F1 at threshold 0.25
print("best cost at threshold", sweep.loc[sweep["cost_gbp"].idxmin(), "thr"])
# best cost at threshold 0.02
print("extra cost of using the F1 threshold: GBP",
int(sweep.loc[sweep["F1"].idxmax(), "cost_gbp"] - sweep["cost_gbp"].min()))
# extra cost of using the F1 threshold: GBP 24552
Maximising F1 picks 0.25 and costs £24,552 more on one quarter of meters than the cheapest rule in the sweep. The cost column is still falling at 0.02, which is the honest reading: with a £38 visit against a £520 leak, this utility should inspect nearly the whole shortlist and the model’s job is only to order the queue. That is an unglamorous conclusion and it is the right one. F1 is a reasonable default when you have no cost numbers at all. The moment you have them, use them.
Report these metrics from averaged folds rather than a single split, following cross-validation strategies, because precision on a rare class is noisy. The scikit-learn classification metrics guide documents the zero-division behaviour you will eventually hit when a threshold flags nothing at all.
Frequently Asked Questions
Is a high F1 score good?
Only relative to the base rate and to what the alternatives score. In the sweep above, a threshold of 0.02 flags almost every meter and still scores 0.410 without doing anything clever, which makes the default threshold’s 0.487 a modest result. Compute the trivial baseline first.
Can precision and recall both go up at once?
Yes, but not by moving the threshold. Threshold changes trade one against the other along a fixed curve. Lifting both at the same time needs a better model: stronger features, more data, or a different algorithm. If someone reports both improving after a threshold tweak, something is wrong.
Which average should I use for multiple classes?
Macro when every class matters equally, weighted when you want the result dominated by the common classes, micro when you want global counts pooled. For imbalanced problems macro is the honest default, and handling imbalanced classes covers the resampling choices that change it.
Key Takeaways
- Quote precision and recall together, never one alone, because either can be pushed near 1.0 by a threshold that destroys the other.
- Use the harmonic mean rather than an arithmetic average when you fold the two into one number, since it refuses to reward a lopsided classifier.
- Set
betainfbeta_scorefrom your actual cost ratio, and treat plain F1 as the special case where a miss and a false alarm cost the same. - Compare any score against the flag-everything baseline for your base rate before you call it good.
- Choose your operating threshold on expected cost when you have the numbers, because the F1-maximising threshold can be thousands of pounds worse than the cost-minimising one.