The symptom is a pair of numbers, not one
Overfitting and underfitting are not two points on a dial you nudge by feel. They are two distinct diagnoses, and you separate them with two numbers: the training score and the held-out score. A wide gap between them, with training near perfect, is overfitting. Both scores stuck at a low level, close together, is underfitting. The fixes point in opposite directions, so guessing wrong costs you a week.
The number nobody computes but everybody needs is the third one: the best score achievable on your data at all. Without it you cannot tell a model that is underfitting from a model that has already extracted everything available. The statistical framing of this, including the business consequences, is covered in overfitting, underfitting and the bias-variance tradeoff in predictive modelling. This chapter owns the diagnostic procedure: learning curves, the gap test, and what to do next.
Two earlier chapters showed the overfitting mechanism on a single knob. The depth sweep in the decision trees chapter drove training accuracy to 1.000 while test accuracy sagged to 0.650, and the polynomial degree sweep in main challenges of machine learning did the same for regression. Neither told you how to decide which problem a real project has. That is the gap here.
A water utility and 1,500 buried pipe sections
A water company has condition records for 1,500 sections of buried main and wants to flag which will burst in the next twelve months so it can schedule replacement. Five columns: age, operating pressure, a soil corrosivity index, diameter, and the count of past bursts. The simulated truth includes an interaction, since old pipe in aggressive made ground fails far faster than age or soil alone would suggest.
import numpy as np
import pandas as pd
rng = np.random.default_rng(73)
n = 1500 # buried pipe sections on a water network
age = rng.uniform(5, 85, n)
pressure = rng.normal(4.2, 0.9, n).clip(1.5, 7.5)
soil = rng.uniform(0, 1, n) # 0 benign clay, 1 aggressive made ground
diameter = rng.choice([80, 100, 150, 225, 300], n)
past_bursts = rng.poisson(0.6 + 0.02 * age, n)
risk = (-9.6 + 0.055 * age + 2.4 * soil
+ 0.085 * age * soil # the interaction a straight line cannot see
+ 0.42 * pressure - 0.004 * diameter + 0.55 * past_bursts)
p_fail = 1 / (1 + np.exp(-risk))
fails = (rng.uniform(0, 1, n) < p_fail).astype(int)
pipes = pd.DataFrame({"age_years": age.round(1), "pressure_bar": pressure.round(2),
"soil_corrosivity": soil.round(3), "diameter_mm": diameter,
"past_bursts": past_bursts, "failed_12m": fails})
X = pipes.drop(columns="failed_12m")
y = pipes["failed_12m"]
print("rows", len(pipes), "| failure rate", round(y.mean(), 3),
"| best possible accuracy", round(float(((p_fail > 0.5) == y).mean()), 3))
# rows 1500 | failure rate 0.288 | best possible accuracy 0.885
That last figure is the luxury a simulation gives you. Because the generating probabilities are known, the best accuracy any model could reach is 0.885. Guessing the majority class scores 0.712. So the usable range is 17 points wide, and a validation accuracy of 0.88 is not “good”; it is finished.
A diagnosis workflow for overfitting and underfitting
A learning curve refits the model at increasing training-set sizes and reports training and validation scores at each. One call does the resampling for you. Three configurations here: a deliberately starved model using only the age column, the full five-column logistic regression, and a one-nearest-neighbour classifier with no capacity control at all.
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import learning_curve, StratifiedKFold
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
sizes = [0.2, 0.4, 0.6, 0.8, 1.0]
runs = {
"too simple": (make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)),
X[["age_years"]]),
"about right": (make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)), X),
"too flexible": (make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=1)), X),
}
for label, (model, features) in runs.items():
rows, train_s, val_s = learning_curve(model, features, y, train_sizes=sizes,
cv=cv, scoring="accuracy")
print(label)
for i, r in enumerate(rows):
print(f" {r:4d} rows train {train_s[i].mean():.3f} val {val_s[i].mean():.3f}"
f" gap {train_s[i].mean() - val_s[i].mean():+.3f}")
# too simple
# 240 rows train 0.822 val 0.771 gap +0.050
# 480 rows train 0.773 val 0.775 gap -0.002
# 720 rows train 0.779 val 0.772 gap +0.007
# 960 rows train 0.775 val 0.775 gap -0.000
# 1200 rows train 0.773 val 0.770 gap +0.003
# about right
# 240 rows train 0.910 val 0.869 gap +0.041
# 480 rows train 0.887 val 0.873 gap +0.014
# 720 rows train 0.880 val 0.873 gap +0.007
# 960 rows train 0.885 val 0.883 gap +0.002
# 1200 rows train 0.881 val 0.883 gap -0.001
# too flexible
# 240 rows train 1.000 val 0.826 gap +0.174
# 480 rows train 1.000 val 0.811 gap +0.189
# 720 rows train 1.000 val 0.807 gap +0.193
# 960 rows train 1.000 val 0.807 gap +0.193
# 1200 rows train 1.000 val 0.818 gap +0.182
Three curves, three verdicts, and all the information is in the shape.
The starved model converges to 0.773 training and 0.770 validation, and stays there from 480 rows onwards. Flat, together, and 11 points below the 0.885 ceiling. That is underfitting, and the flatness is the important part: more rows will not move it. The fix is more signal, not more data.
The full logistic regression converges to 0.881 and 0.883, a gap of essentially zero, within 0.002 of the achievable ceiling. Nothing left on the table.
The one-nearest-neighbour model memorises every training row, so its training accuracy is exactly 1.000 at every size, and validation sits around 0.81. The gap barely narrows from 240 to 1,200 rows, moving from 0.174 to 0.182. Doubling the dataset again would not rescue it; the capacity has to come down.
| What you see | Diagnosis | What helps | What does not |
|---|---|---|---|
| Both scores low, gap near zero, curve flat | Underfitting | Better features, more flexible model, weaker regularisation | More rows |
| Training near perfect, wide gap | Overfitting | Stronger regularisation, fewer features, simpler model, more rows | More features |
| Scores converging, validation still rising | Not enough data yet | More rows | Changing the model |
| Both scores high and close | Nothing to fix | Shipping it | Further tuning |
The third row is the one people miss. If the validation curve is still climbing at your largest training size, the honest answer is that you cannot yet tell whether the model is right, and collecting data beats tuning. The underlying arithmetic for all four cases is set out in the bias variance decomposition.
When a structural error costs almost nothing
The generating rule contained an age_years * soil_corrosivity interaction, and a plain logistic regression has no way to represent it. By the usual story that model is structurally wrong and should underfit. Add the missing term and see what it buys.
from sklearn.model_selection import cross_val_score
X_int = X.copy()
X_int["age_x_soil"] = X["age_years"] * X["soil_corrosivity"]
plain = cross_val_score(make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)),
X, y, cv=cv, scoring="accuracy")
with_int = cross_val_score(make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)),
X_int, y, cv=cv, scoring="accuracy")
print(f"five raw columns {plain.mean():.3f}")
print(f"plus the interaction {with_int.mean():.3f}")
print(f"gain from fixing the structural error {with_int.mean() - plain.mean():+.3f}")
# five raw columns 0.883
# plus the interaction 0.886
# gain from fixing the structural error +0.003
Three thousandths of a point. The model was missing a term that genuinely exists in the data-generating process, and repairing it changed nothing a utility would notice. Age and soil both carry strong main effects, so the linear surface already lands on the right side of the threshold for almost every section.
The lesson is uncomfortable and worth sitting with: being structurally wrong is not the same as underfitting. Underfitting is a measured shortfall against an achievable score. A misspecified model that still reaches the ceiling is not underfitting anything, and chasing its misspecification is wasted effort. Judge by the held-out number from a sealed test slice, not by whether the functional form looks right. The resampling behaviour of these curves is documented under learning_curve in scikit-learn.
Frequently Asked Questions
How big a train-validation gap counts as overfitting?
There is no universal threshold, which is why the gap alone is not enough. Judge it against the validation score: a gap of 0.18 with validation at 0.81, as the one-nearest-neighbour model shows, is serious. A gap of 0.02 with validation at 0.88 and no headroom left is noise.
Can a model overfit and underfit at the same time?
Yes, in different regions of the input space. A tree can memorise a dense region while leaving a sparse one crudely approximated. Aggregate scores hide this, so when a model behaves oddly in production, score it separately on slices of the data rather than only overall.
Will more training data fix overfitting?
Sometimes, and the learning curve tells you whether. If the validation score is still rising at your largest training size, extra rows will help. If the curve has gone flat with the gap intact, as above, adding rows is wasted spend and you need to reduce capacity instead.
Key Takeaways
- Read the training score, the held-out score and their gap together; any one of the three on its own is uninterpretable.
- Underfitting looks like two low scores that have converged and stopped moving, and more rows will not shift it.
- Overfitting looks like a near-perfect training score beside a much lower held-out score, and the learning curve tells you whether more data can close it.
- Estimate the best achievable score before deciding a model underfits, or you will tune against a ceiling you have already reached.
- A structurally misspecified model is not automatically underfitting; here the missing interaction was worth only 0.003 in accuracy.