A model that is 98.9% accurate and completely useless
Imbalanced classes are what you have when the thing you want to predict almost never happens: chargebacks, equipment failures, rare diagnoses, churn in a sticky subscription business. The standard advice is to reweight or resample until the classes look even. That advice is half right, and the half that is wrong costs people a lot of wasted effort, so this chapter measures it rather than repeating it.
Short version: rebalancing does not make your model better at separating the classes. It moves where the model’s decision threshold effectively sits. You can usually get the same result by moving the threshold yourself, and keep a model whose probabilities still mean something.
Start with a card issuer’s authorisation file.
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(67)
n = 40000
amount_z = rng.normal(0, 1, n)
hours_since_last = rng.gamma(2.0, 6.0, n)
new_device = (rng.random(n) < 0.14).astype(int)
mismatch_country = (rng.random(n) < 0.07).astype(int)
logit = (-4.9 + 0.85 * amount_z - 0.040 * hours_since_last
+ 1.25 * new_device + 1.60 * mismatch_country)
charged_back = rng.binomial(1, 1 / (1 + np.exp(-logit)))
auths = pd.DataFrame({"amount_z": amount_z.round(3),
"hours_since_last": hours_since_last.round(2),
"new_device": new_device,
"mismatch_country": mismatch_country,
"charged_back": charged_back})
rate = auths["charged_back"].mean()
print("rows:", len(auths), "| positives:", int(auths["charged_back"].sum()),
"| positive rate:", f"{rate:.4f}")
print("always-predict-legitimate accuracy:", round(1 - rate, 4))
# rows: 40000 | positives: 447 | positive rate: 0.0112
# always-predict-legitimate accuracy: 0.9888
features = ["amount_z", "hours_since_last", "new_device", "mismatch_country"]
X_tr, X_te, y_tr, y_te = train_test_split(auths[features], auths["charged_back"],
test_size=0.3, random_state=0,
stratify=auths["charged_back"])
447 chargebacks in 40,000 authorisations. A model that predicts “legitimate” for every row scores 98.88%, which is the number you must beat before accuracy means anything at all. Note the stratify argument: without it a random split can hand you a test set with a materially different positive rate, and on 1% labels that happens often enough to matter.
What the plain model does
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (precision_score, recall_score, roc_auc_score,
average_precision_score)
def report(name, y_true, y_pred, scores):
print(f"{name:22s} acc {(y_true == y_pred).mean():.4f} "
f"precision {precision_score(y_true, y_pred, zero_division=0):.3f} "
f"recall {recall_score(y_true, y_pred):.3f} "
f"ROC AUC {roc_auc_score(y_true, scores):.3f} "
f"PR AUC {average_precision_score(y_true, scores):.3f}")
plain = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000, random_state=0))
plain.fit(X_tr, y_tr)
s_plain = plain.predict_proba(X_te)[:, 1]
report("plain", y_te, plain.predict(X_te), s_plain)
print("positives predicted:", int(plain.predict(X_te).sum()), "of", int(y_te.sum()), "actual")
# plain acc 0.9888 precision 0.000 recall 0.000 ROC AUC 0.783 PR AUC 0.079
# positives predicted: 0 of 134 actual
Zero. The fitted model never once crosses 0.5, so predict returns the majority class for all 12,000 test rows and recall is exactly nil. The model is not broken: its ROC AUC of 0.783 says the scores rank chargebacks above legitimate authorisations far better than chance. The ranking is fine and the default cut-off is wrong. Those are separate problems and the fix is different for each.
Two scores are worth tracking on imbalanced classes. ROC AUC asks how well the model ranks a random positive above a random negative. Average precision, the area under the precision-recall curve, is harsher, because its baseline is the positive rate itself. Here that baseline is 0.0112 and the model reaches 0.079, so roughly seven times better than guessing at the top of the ranking. Both are covered properly later in the course; treat them here as the two numbers that tell you whether a change helped the model or only moved the threshold.
Reweighting and resampling, measured
Three standard remedies, applied to the same data and scored on the same untouched test set.
weighted = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=1000, class_weight="balanced",
random_state=0)).fit(X_tr, y_tr)
report("class_weight balanced", y_te, weighted.predict(X_te),
weighted.predict_proba(X_te)[:, 1])
# class_weight balanced acc 0.7244 precision 0.027 recall 0.664 ROC AUC 0.785 PR AUC 0.082
from sklearn.utils import resample
tr = X_tr.copy(); tr["charged_back"] = y_tr.to_numpy()
pos = tr[tr["charged_back"] == 1]; neg = tr[tr["charged_back"] == 0]
up = pd.concat([neg, resample(pos, replace=True, n_samples=len(neg), random_state=0)])
down = pd.concat([pos, resample(neg, replace=False, n_samples=len(pos), random_state=0)])
for name, frame in [("random oversample", up), ("random undersample", down)]:
m = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=1000, random_state=0))
m.fit(frame[features], frame["charged_back"])
report(name, y_te, m.predict(X_te), m.predict_proba(X_te)[:, 1])
# random oversample acc 0.7238 precision 0.027 recall 0.672 ROC AUC 0.784 PR AUC 0.082
# random undersample acc 0.7298 precision 0.027 recall 0.664 ROC AUC 0.785 PR AUC 0.082
Recall goes from 0.000 to about 0.67. That looks like a triumph until you read the rest of the row. Precision is 0.027, meaning 37 false alarms for every genuine chargeback caught. Accuracy fell from 0.9888 to 0.724. And the two numbers that measure the model’s actual discriminative power barely budged: ROC AUC moved from 0.783 to 0.785, average precision from 0.079 to 0.082. Differences that small on 134 test positives are not results.
All three methods landed in the same place, within a thousandth of each other. That is the finding. Reweighting the loss, duplicating minority rows and discarding majority rows are three routes to the same effect, which is to shift the point where the model says yes.
Moving the threshold instead
If the ranking is unchanged, the threshold is the only real lever, so pull it directly.
rows = []
for t in [0.5, 0.2, 0.1, 0.06, 0.04, 0.02]:
pred = (s_plain >= t).astype(int)
rows.append({"threshold": t, "flagged": int(pred.sum()),
"precision": round(precision_score(y_te, pred, zero_division=0), 3),
"recall": round(recall_score(y_te, pred), 3)})
print(pd.DataFrame(rows).to_string(index=False))
# threshold flagged precision recall
# 0.50 0 0.000 0.000
# 0.20 20 0.200 0.030
# 0.10 77 0.156 0.090
# 0.06 230 0.104 0.179
# 0.04 517 0.074 0.284
# 0.02 1633 0.043 0.522
target = recall_score(y_te, weighted.predict(X_te))
grid = np.unique(np.round(s_plain, 5))
best = max(t for t in grid
if recall_score(y_te, (s_plain >= t).astype(int)) >= target)
matched = (s_plain >= best).astype(int)
print(f"weighted model recall {target:.3f} "
f"precision {precision_score(y_te, weighted.predict(X_te), zero_division=0):.4f}")
print(f"plain + threshold {best:.4f} recall {recall_score(y_te, matched):.3f} "
f"precision {precision_score(y_te, matched, zero_division=0):.4f}")
# weighted model recall 0.664 precision 0.0266
# plain + threshold 0.0110 recall 0.664 precision 0.0264
The untouched model, cut at 0.011 instead of 0.5, reaches exactly the same recall as the reweighted one at essentially the same precision: 0.0264 against 0.0266. The reweighting bought nothing that a threshold could not. What it cost is calibration. The plain model’s score of 0.011 is a real estimate of chargeback probability for that authorisation, usable for expected-loss arithmetic. The reweighted model’s 0.5 is not a probability of anything, because the training distribution it was fitted on no longer matches the world.
The threshold table is also the useful artefact for a conversation with the fraud team. At 0.06 you review 230 authorisations to catch 24 chargebacks. At 0.02 you review 1,633 to catch 70. Nobody in that room wants to discuss class weights; they want to know how many reviews per catch, and that choice belongs to them rather than to you.
A working order for imbalanced classes
What I actually do, in this order.
| Step | Action | Why |
|---|---|---|
| 1 | Record the majority-class accuracy | 0.9888 here, the floor any model must beat |
| 2 | Fit the plain model, score with ROC AUC and average precision | Measures ranking without threshold effects |
| 3 | Improve the ranking first | Better features and a stronger learner, not reweighting |
| 4 | Tune the threshold against the real cost ratio | Converts ranking into a decision |
| 5 | Reach for resampling only if steps 3 and 4 stall | It changes the operating point, not the model’s power |
Step 3 is where the gains live. A tree-based model that can represent interactions between new_device and mismatch_country will lift ROC AUC in a way no amount of reweighting will, which is one reason how decision trees split the data is worth revisiting here, and why the binary probabilities from how logistic regression produces probabilities are the thing you want to preserve. On an uneven label with more than two classes, read the per-class table as described in multi-class classification before averaging anything. For the business framing, cost matrices and the SMOTE family, the imbalanced datasets chapter in the Predictive Modelling course goes further than this one does.
SMOTE itself needs pip install imbalanced-learn, which this sandbox did not have, so the examples above use plain random resampling from sklearn.utils.resample rather than printing output from code that was never run. Scikit-learn’s own guide on tuning the decision threshold documents TunedThresholdClassifierCV, which wraps step 4 into a cross-validated estimator and is the cleanest way to do this in a pipeline.
Frequently Asked Questions
Does class_weight=”balanced” fix imbalanced data?
It changes the operating point, not the model’s ability to separate the classes. On the chargeback data it lifted recall from 0.000 to 0.664 while ROC AUC moved only from 0.783 to 0.785, and a threshold of 0.011 on the plain model matched it exactly. Use it as a convenience, not as a cure.
Should I oversample before or after the train-test split?
After, and only on the training portion. Oversampling before the split copies minority rows into both sides, so the same authorisation appears in training and test, and your scores come out optimistically wrong. The same applies to cross-validation folds, which is what makes a pipeline the safe way to do it.
What is a good threshold for a rare positive class?
Whatever the cost ratio implies. If reviewing an authorisation costs five pounds and a missed chargeback costs two hundred, you can justify about forty reviews per catch, which on this model lands just above a threshold of 0.011, where recall is 0.664 at 38 reviews per catch. Derive it from costs, not from a default.
Key Takeaways
- Write down the majority-class accuracy before you model anything, because 98.88% here was the score to beat rather than a good result.
- Judge any change by ROC AUC or average precision, since those measure ranking while accuracy, precision and recall all move when the threshold moves.
- Expect class weights, oversampling and undersampling to land in the same place, as all three did to within 0.003 average precision on this data.
- Tune the threshold on an untouched model when you want a specific recall, and you keep probabilities that still support expected-cost arithmetic.
- Spend your effort on better features and a stronger learner, because improving the ranking is the only change that moves the ceiling.