One curve, every threshold at once
An ROC curve and AUC answer a question that a single confusion matrix cannot: how good is the model before you commit to a cut-off? The curve is built by sweeping the threshold from high to low and plotting two rates at each step. On the vertical axis, the true positive rate, which is recall. On the horizontal axis, the false positive rate, the share of genuine negatives that got flagged anyway. Every point on the curve is one confusion matrix. The curve is the whole family of them.
AUC is the area under that curve, and it has an exact interpretation worth memorising: it is the probability that a randomly chosen positive case gets a higher score than a randomly chosen negative one. A model at 0.83 puts the positive first in 83% of such pairs. That is a statement about ordering, and it says nothing at all about whether your probabilities are calibrated or your labels correct.
Sweeping the threshold
A bike-share operator logged 3,500 station-hours and wanted to know, an hour ahead, which docks would run out of bikes so a van could rebalance them.
import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_curve, roc_auc_score, confusion_matrix, accuracy_score
rng = np.random.default_rng(77)
n = 3500
bikes_at_start = rng.integers(0, 25, n)
is_weekday = rng.binomial(1, 0.71, n)
temp_c = np.round(rng.normal(13.5, 6.0, n), 1)
near_stations = rng.integers(0, 6, n)
hour = rng.integers(6, 22, n)
logit = (0.9
- 0.21 * bikes_at_start
+ 0.75 * is_weekday
+ 0.055 * temp_c
- 0.28 * near_stations
- 0.09 * np.abs(hour - 9))
ran_empty = rng.binomial(1, 1 / (1 + np.exp(-logit)))
stations = pd.DataFrame({
"bikes_at_start": bikes_at_start, "is_weekday": is_weekday,
"temp_c": temp_c, "near_stations": near_stations, "hour": hour,
"ran_empty": ran_empty,
})
print("station-hours:", len(stations),
"empty rate:", round(stations["ran_empty"].mean(), 4))
# station-hours: 3500 empty rate: 0.2769
X = stations.drop(columns="ran_empty")
y = stations["ran_empty"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3,
random_state=0, stratify=y)
model = LogisticRegression(max_iter=1000, random_state=0).fit(X_tr, y_tr)
scores = model.predict_proba(X_te)[:, 1]
fpr, tpr, thresholds = roc_curve(y_te, scores)
print("points on the curve:", len(fpr))
# points on the curve: 298
print("thresholds[0]:", thresholds[0])
# thresholds[0]: inf
print(pd.DataFrame({"threshold": thresholds, "fpr": fpr, "tpr": tpr})
.head(6).round(4).to_string(index=False))
# threshold fpr tpr
# inf 0.0000 0.0000
# 0.9062 0.0000 0.0034
# 0.7858 0.0000 0.0825
# 0.7839 0.0013 0.0825
# 0.7723 0.0013 0.0962
# 0.7697 0.0040 0.0962
print("auc:", round(roc_auc_score(y_te, scores), 4))
# auc: 0.8333
roc_curve returns 298 points for 1,050 test rows, one per distinct score plus a starting point at infinity where nothing is flagged and both rates are zero. Notice how the rates move in alternating steps: dropping the threshold past a positive case lifts tpr and leaves fpr unchanged, and passing a negative case does the reverse. A perfect ranker would climb straight up the left edge before turning right.
Each point is a confusion matrix you already know how to read
Nothing new is being measured. Pick three thresholds off the curve and the matrices behind them are the same tables from the last two chapters.
for t in [0.70, 0.50, 0.30]:
tn, fp, fn, tp = confusion_matrix(
y_te, (scores >= t).astype(int), labels=[0, 1]).ravel()
print(f"threshold {t:.2f} TN={tn} FP={fp} FN={fn} TP={tp}"
f" fpr={fp / (fp + tn):.4f} tpr={tp / (tp + fn):.4f}")
# threshold 0.70 TN=747 FP=12 FN=233 TP=58 fpr=0.0158 tpr=0.1993
# threshold 0.50 TN=682 FP=77 FN=138 TP=153 fpr=0.1014 tpr=0.5258
# threshold 0.30 TN=582 FP=177 FN=80 TP=211 fpr=0.2332 tpr=0.7251
Those three (fpr, tpr) pairs are three of the 298 points. The curve exists so you can choose among them with the trade-off visible, instead of accepting whatever 0.5 happens to give. The operator’s van can service about 20% of stations per hour, which fixes the false positive budget and picks the threshold off the horizontal axis rather than by taste. Precision and recall at a chosen point are then read as in precision, recall and F1 score.
What the ROC curve and AUC actually measure
The ranking interpretation is not an analogy. Count every positive-negative pair in the test set and check which one scored higher, and you recover AUC exactly.
pos = scores[y_te.to_numpy() == 1]
neg = scores[y_te.to_numpy() == 0]
wins = (pos[:, None] > neg[None, :]).sum()
ties = (pos[:, None] == neg[None, :]).sum()
pairs = pos.size * neg.size
print("positive/negative pairs:", pairs)
# positive/negative pairs: 220869
print("positive ranked higher in:", int(wins), "pairs")
# positive ranked higher in: 184061 pairs
print("pairwise estimate:", round((wins + 0.5 * ties) / pairs, 6))
# pairwise estimate: 0.833349
print("roc_auc_score: ", round(roc_auc_score(y_te, scores), 6))
# roc_auc_score: 0.833349
Agreement to six decimal places, because they are the same quantity computed two ways. Ties count half, which is why a model that gives every row an identical score scores 0.5 rather than 0. This equivalence to the Mann-Whitney U statistic was set out by Hanley and McNeil in their 1982 Radiology paper.
Rank-only has consequences
Because AUC sees only the ordering, any strictly increasing transform of your scores leaves it untouched.
squared, shifted = scores ** 2, scores * 0.1 + 0.4
print("auc raw, squared, shifted:",
round(roc_auc_score(y_te, scores), 6),
round(roc_auc_score(y_te, squared), 6),
round(roc_auc_score(y_te, shifted), 6))
# auc raw, squared, shifted: 0.833349 0.833349 0.833349
print("accuracy at 0.5:",
round(accuracy_score(y_te, scores >= 0.5), 4),
round(accuracy_score(y_te, squared >= 0.5), 4),
round(accuracy_score(y_te, shifted >= 0.5), 4))
# accuracy at 0.5: 0.7952 0.7619 0.7229
print("auc of reversed scores:", round(roc_auc_score(y_te, -scores), 6))
# auc of reversed scores: 0.166651
print("auc of random scores: ",
round(roc_auc_score(y_te, rng.random(len(y_te))), 6))
# auc of random scores: 0.492391
Three score sets with identical AUC and accuracies spanning seven points. The shifted version never exceeds 0.49, so at a 0.5 cut-off it predicts “not empty” for every single row, and AUC still reports 0.833. Read that as the warning it is: a strong AUC tells you the model orders cases well and tells you nothing about whether the deployed decision rule works. Calibration and threshold choice are separate jobs.
Two more readings. Reversing the scores gives 0.1667, exactly one minus the original, so an AUC well below 0.5 means your signal is real and your sign is wrong. And random scores land at 0.4924 rather than 0.5000, which is the sampling noise you should expect on 1,050 rows before you get excited about a model at 0.52.
| AUC | What it means | Caveat |
|---|---|---|
| 0.5 | No ranking ability | Expect roughly plus or minus 0.02 of noise on a sample this size |
| 0.7 to 0.85 | Useful ordering, typical of a decent tabular model | Says nothing about calibration |
| above 0.95 | Suspiciously good | Check for data leakage first |
The case against ROC on heavily skewed data, and why a precision-recall curve is the honest choice when positives are under a few per cent, is made with worked numbers in handling imbalanced classes. At 27.7% positives here, ROC is the right tool. The scikit-learn roc_auc_score reference covers the multi-class average and multi_class options when you need them.
Frequently Asked Questions
Why does roc_curve return a threshold of inf?
So the curve starts at the origin. At an infinite threshold nothing is flagged, so both rates are zero and the curve has a defined left-hand end. Older scikit-learn versions used max(score) + 1 instead. It is an artefact of plotting, not a usable cut-off.
Is AUC better than accuracy?
For comparing models it usually is, because it does not depend on an arbitrary threshold and is unaffected by the class balance of your test set. For deciding whether to ship, it is worse, because you deploy a threshold rather than a ranking and AUC cannot see your costs.
Can two models with the same AUC behave differently?
Yes, and often. Equal areas can come from very different curve shapes: one model strong at low false positive rates, another strong only once you flag half the negatives. If you operate in a narrow region, compare the curves there or use partial AUC rather than the whole area.
Key Takeaways
- Treat the ROC curve as the set of every confusion matrix the model can produce, so you choose a threshold with the trade-off in front of you rather than inheriting 0.5.
- Read AUC as the probability that a random positive outranks a random negative, which you can verify by counting pairs directly.
- Remember that AUC is invariant to any monotone transform of the scores, so a model can score 0.83 while its default decision rule predicts one class for every row.
- Treat an AUC below 0.5 as an inverted signal rather than a useless model, since reversing the scores gives exactly one minus the area.
- Benchmark against random scores on your own sample size before calling a small AUC gain real, because a few hundred rows of noise moves the figure by a point or two.