A model that is 98.9% accurate and completely useless

Imbalanced classes are what you have when the thing you want to predict almost never happens: chargebacks, equipment failures, rare diagnoses, churn in a sticky subscription business. The standard advice is to reweight or resample until the classes look even. That advice is half right, and the half that is wrong costs people a lot of wasted effort, so this chapter measures it rather than repeating it.

Short version: rebalancing does not make your model better at separating the classes. It moves where the model’s decision threshold effectively sits. You can usually get the same result by moving the threshold yourself, and keep a model whose probabilities still mean something.

Start with a card issuer’s authorisation file.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(67)
n = 40000

amount_z = rng.normal(0, 1, n)
hours_since_last = rng.gamma(2.0, 6.0, n)
new_device = (rng.random(n) < 0.14).astype(int)
mismatch_country = (rng.random(n) < 0.07).astype(int)

logit = (-4.9 + 0.85 * amount_z - 0.040 * hours_since_last
         + 1.25 * new_device + 1.60 * mismatch_country)
charged_back = rng.binomial(1, 1 / (1 + np.exp(-logit)))

auths = pd.DataFrame({"amount_z": amount_z.round(3),
                      "hours_since_last": hours_since_last.round(2),
                      "new_device": new_device,
                      "mismatch_country": mismatch_country,
                      "charged_back": charged_back})
rate = auths["charged_back"].mean()
print("rows:", len(auths), "| positives:", int(auths["charged_back"].sum()),
      "| positive rate:", f"{rate:.4f}")
print("always-predict-legitimate accuracy:", round(1 - rate, 4))
# rows: 40000 | positives: 447 | positive rate: 0.0112
# always-predict-legitimate accuracy: 0.9888

features = ["amount_z", "hours_since_last", "new_device", "mismatch_country"]
X_tr, X_te, y_tr, y_te = train_test_split(auths[features], auths["charged_back"],
                                          test_size=0.3, random_state=0,
                                          stratify=auths["charged_back"])

447 chargebacks in 40,000 authorisations. A model that predicts “legitimate” for every row scores 98.88%, which is the number you must beat before accuracy means anything at all. Note the stratify argument: without it a random split can hand you a test set with a materially different positive rate, and on 1% labels that happens often enough to matter.

What the plain model does

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (precision_score, recall_score, roc_auc_score,
                             average_precision_score)

def report(name, y_true, y_pred, scores):
    print(f"{name:22s} acc {(y_true == y_pred).mean():.4f}  "
          f"precision {precision_score(y_true, y_pred, zero_division=0):.3f}  "
          f"recall {recall_score(y_true, y_pred):.3f}  "
          f"ROC AUC {roc_auc_score(y_true, scores):.3f}  "
          f"PR AUC {average_precision_score(y_true, scores):.3f}")

plain = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000, random_state=0))
plain.fit(X_tr, y_tr)
s_plain = plain.predict_proba(X_te)[:, 1]
report("plain", y_te, plain.predict(X_te), s_plain)
print("positives predicted:", int(plain.predict(X_te).sum()), "of", int(y_te.sum()), "actual")
# plain                  acc 0.9888  precision 0.000  recall 0.000  ROC AUC 0.783  PR AUC 0.079
# positives predicted: 0 of 134 actual

Zero. The fitted model never once crosses 0.5, so predict returns the majority class for all 12,000 test rows and recall is exactly nil. The model is not broken: its ROC AUC of 0.783 says the scores rank chargebacks above legitimate authorisations far better than chance. The ranking is fine and the default cut-off is wrong. Those are separate problems and the fix is different for each.

Two scores are worth tracking on imbalanced classes. ROC AUC asks how well the model ranks a random positive above a random negative. Average precision, the area under the precision-recall curve, is harsher, because its baseline is the positive rate itself. Here that baseline is 0.0112 and the model reaches 0.079, so roughly seven times better than guessing at the top of the ranking. Both are covered properly later in the course; treat them here as the two numbers that tell you whether a change helped the model or only moved the threshold.

Reweighting and resampling, measured

Three standard remedies, applied to the same data and scored on the same untouched test set.

weighted = make_pipeline(StandardScaler(),
                         LogisticRegression(max_iter=1000, class_weight="balanced",
                                            random_state=0)).fit(X_tr, y_tr)
report("class_weight balanced", y_te, weighted.predict(X_te),
       weighted.predict_proba(X_te)[:, 1])
# class_weight balanced  acc 0.7244  precision 0.027  recall 0.664  ROC AUC 0.785  PR AUC 0.082

from sklearn.utils import resample
tr = X_tr.copy(); tr["charged_back"] = y_tr.to_numpy()
pos = tr[tr["charged_back"] == 1]; neg = tr[tr["charged_back"] == 0]
up = pd.concat([neg, resample(pos, replace=True, n_samples=len(neg), random_state=0)])
down = pd.concat([pos, resample(neg, replace=False, n_samples=len(pos), random_state=0)])

for name, frame in [("random oversample", up), ("random undersample", down)]:
    m = make_pipeline(StandardScaler(),
                      LogisticRegression(max_iter=1000, random_state=0))
    m.fit(frame[features], frame["charged_back"])
    report(name, y_te, m.predict(X_te), m.predict_proba(X_te)[:, 1])
# random oversample      acc 0.7238  precision 0.027  recall 0.672  ROC AUC 0.784  PR AUC 0.082
# random undersample     acc 0.7298  precision 0.027  recall 0.664  ROC AUC 0.785  PR AUC 0.082

Recall goes from 0.000 to about 0.67. That looks like a triumph until you read the rest of the row. Precision is 0.027, meaning 37 false alarms for every genuine chargeback caught. Accuracy fell from 0.9888 to 0.724. And the two numbers that measure the model’s actual discriminative power barely budged: ROC AUC moved from 0.783 to 0.785, average precision from 0.079 to 0.082. Differences that small on 134 test positives are not results.

All three methods landed in the same place, within a thousandth of each other. That is the finding. Reweighting the loss, duplicating minority rows and discarding majority rows are three routes to the same effect, which is to shift the point where the model says yes.

Moving the threshold instead

If the ranking is unchanged, the threshold is the only real lever, so pull it directly.

rows = []
for t in [0.5, 0.2, 0.1, 0.06, 0.04, 0.02]:
    pred = (s_plain >= t).astype(int)
    rows.append({"threshold": t, "flagged": int(pred.sum()),
                 "precision": round(precision_score(y_te, pred, zero_division=0), 3),
                 "recall": round(recall_score(y_te, pred), 3)})
print(pd.DataFrame(rows).to_string(index=False))
#  threshold  flagged  precision  recall
#       0.50        0      0.000   0.000
#       0.20       20      0.200   0.030
#       0.10       77      0.156   0.090
#       0.06      230      0.104   0.179
#       0.04      517      0.074   0.284
#       0.02     1633      0.043   0.522

target = recall_score(y_te, weighted.predict(X_te))
grid = np.unique(np.round(s_plain, 5))
best = max(t for t in grid
           if recall_score(y_te, (s_plain >= t).astype(int)) >= target)
matched = (s_plain >= best).astype(int)
print(f"weighted model      recall {target:.3f}  "
      f"precision {precision_score(y_te, weighted.predict(X_te), zero_division=0):.4f}")
print(f"plain + threshold {best:.4f}  recall {recall_score(y_te, matched):.3f}  "
      f"precision {precision_score(y_te, matched, zero_division=0):.4f}")
# weighted model      recall 0.664  precision 0.0266
# plain + threshold 0.0110  recall 0.664  precision 0.0264

The untouched model, cut at 0.011 instead of 0.5, reaches exactly the same recall as the reweighted one at essentially the same precision: 0.0264 against 0.0266. The reweighting bought nothing that a threshold could not. What it cost is calibration. The plain model’s score of 0.011 is a real estimate of chargeback probability for that authorisation, usable for expected-loss arithmetic. The reweighted model’s 0.5 is not a probability of anything, because the training distribution it was fitted on no longer matches the world.

The threshold table is also the useful artefact for a conversation with the fraud team. At 0.06 you review 230 authorisations to catch 24 chargebacks. At 0.02 you review 1,633 to catch 70. Nobody in that room wants to discuss class weights; they want to know how many reviews per catch, and that choice belongs to them rather than to you.

A working order for imbalanced classes

What I actually do, in this order.

Step Action Why
1 Record the majority-class accuracy 0.9888 here, the floor any model must beat
2 Fit the plain model, score with ROC AUC and average precision Measures ranking without threshold effects
3 Improve the ranking first Better features and a stronger learner, not reweighting
4 Tune the threshold against the real cost ratio Converts ranking into a decision
5 Reach for resampling only if steps 3 and 4 stall It changes the operating point, not the model’s power

Step 3 is where the gains live. A tree-based model that can represent interactions between new_device and mismatch_country will lift ROC AUC in a way no amount of reweighting will, which is one reason how decision trees split the data is worth revisiting here, and why the binary probabilities from how logistic regression produces probabilities are the thing you want to preserve. On an uneven label with more than two classes, read the per-class table as described in multi-class classification before averaging anything. For the business framing, cost matrices and the SMOTE family, the imbalanced datasets chapter in the Predictive Modelling course goes further than this one does.

SMOTE itself needs pip install imbalanced-learn, which this sandbox did not have, so the examples above use plain random resampling from sklearn.utils.resample rather than printing output from code that was never run. Scikit-learn’s own guide on tuning the decision threshold documents TunedThresholdClassifierCV, which wraps step 4 into a cross-validated estimator and is the cleanest way to do this in a pipeline.

Frequently Asked Questions

Does class_weight=”balanced” fix imbalanced data?

It changes the operating point, not the model’s ability to separate the classes. On the chargeback data it lifted recall from 0.000 to 0.664 while ROC AUC moved only from 0.783 to 0.785, and a threshold of 0.011 on the plain model matched it exactly. Use it as a convenience, not as a cure.

Should I oversample before or after the train-test split?

After, and only on the training portion. Oversampling before the split copies minority rows into both sides, so the same authorisation appears in training and test, and your scores come out optimistically wrong. The same applies to cross-validation folds, which is what makes a pipeline the safe way to do it.

What is a good threshold for a rare positive class?

Whatever the cost ratio implies. If reviewing an authorisation costs five pounds and a missed chargeback costs two hundred, you can justify about forty reviews per catch, which on this model lands just above a threshold of 0.011, where recall is 0.664 at 38 reviews per catch. Derive it from costs, not from a default.

Key Takeaways

  • Write down the majority-class accuracy before you model anything, because 98.88% here was the score to beat rather than a good result.
  • Judge any change by ROC AUC or average precision, since those measure ranking while accuracy, precision and recall all move when the threshold moves.
  • Expect class weights, oversampling and undersampling to land in the same place, as all three did to within 0.003 average precision on this data.
  • Tune the threshold on an untouched model when you want a specific recall, and you keep probabilities that still support expected-cost arithmetic.
  • Spend your effort on better features and a stronger learner, because improving the ranking is the only change that moves the ceiling.