Measuring the bias variance tradeoff instead of describing it
Expected squared error on new data splits into exactly three parts: bias squared, variance, and noise you can never remove. Bias is how far the average model is from the truth. Variance is how much any single model wobbles when you refit it on a different sample. Noise is the scatter in the target that no feature explains. The bias variance tradeoff is the observation that most ways of reducing one of the first two increase the other.
Almost every treatment of this stops at the diagram. You can actually compute the three numbers, as long as you can draw repeated training samples from the same process, and the arithmetic then does something useful: it tells you which of the two problems you have, and therefore whether your next hour is better spent on a simpler model or on more data.
Why the decomposition needs repeated samples
With one training set you see one model, and one model cannot tell you how much your procedure varies. So you simulate. Draw 300 independent training sets from the same generator, fit the same estimator on each, and look at the spread of predictions at each test point. The spread is variance. The distance from the average prediction to the truth is bias. This is the same expectation arithmetic as in the statistics you actually need, applied to predictions rather than to sample means.
A district heating network and 300 parallel universes
A district heating scheme in a northern town bills on metered heat and wants to forecast daily gas burn from the outside temperature. The real relationship is a smooth S-curve: burn saturates around 100 MWh in hard frost, falls away through the middle of the range, and flattens near 7 MWh in summer when only hot water remains. Each year it sees 60 usable days of readings.
import numpy as np
from sklearn.neighbors import KNeighborsRegressor
rng = np.random.default_rng(72)
def true_burn(temp_c):
"""Daily gas burn in MWh for a district heating scheme, as a function of temperature."""
return 95 / (1 + np.exp(0.55 * (temp_c - 11.0))) + 6.0
SIGMA = 3.5 # metering and behaviour noise, MWh
grid = np.linspace(-6, 24, 200).reshape(-1, 1)
truth = true_burn(grid.ravel())
def draw_winter(m=60):
"""One year's worth of readings, 60 days sampled from the same process."""
t = rng.uniform(-6, 24, m).reshape(-1, 1)
return t, true_burn(t.ravel()) + rng.normal(0, SIGMA, m)
print([round(float(v), 2) for v in true_burn(np.array([-5.0, 11.0, 20.0]))])
# [100.99, 53.5, 6.67]
Knowing true_burn is the trick that makes this measurable. On real data you never have it, which is why the decomposition is a teaching instrument rather than a diagnostic you run at work. The lesson it teaches transfers; the numbers do not.
The decomposition in numbers
The knob is n_neighbors. Small k means a flexible model that chases individual readings. Large k means a stiff model that averages across wide temperature bands. If you want the algorithm itself rather than its error behaviour, see how k-nearest neighbours makes a prediction.
REPEATS = 300
table = []
for k in (1, 3, 5, 10, 20, 40):
preds = np.empty((REPEATS, len(grid)))
for r in range(REPEATS):
t, y = draw_winter()
preds[r] = KNeighborsRegressor(n_neighbors=k).fit(t, y).predict(grid)
bias2 = np.mean((preds.mean(axis=0) - truth) ** 2)
variance = np.mean(preds.var(axis=0))
table.append((k, bias2, variance, bias2 + variance + SIGMA ** 2))
print(f"{'k':>3} {'bias^2':>8} {'variance':>9} {'noise':>7} {'total':>8}")
for k, b, v, tot in table:
print(f"{k:>3} {b:8.2f} {v:9.2f} {SIGMA**2:7.2f} {tot:8.2f}")
# k bias^2 variance noise total
# 1 0.05 15.74 12.25 28.04
# 3 0.08 7.63 12.25 19.96
# 5 0.26 7.12 12.25 19.63
# 10 2.13 8.23 12.25 22.61
# 20 24.70 15.46 12.25 52.41
# 40 480.46 47.40 12.25 540.11
Read down the columns. Bias squared starts at essentially zero and stays tiny until k reaches 10, then explodes: 2.13, then 24.70, then 480.46. Averaging 40 of 60 points smears the S-curve into something close to a flat line, and no amount of extra data fixes that. Total error is lowest at k equal to 5, at 19.63, with k equal to 3 a close second at 19.96. The floor is 12.25, the noise term, so even the best setting carries about 7.4 of avoidable error.
One result here contradicts the standard picture. Variance is supposed to fall monotonically as k rises. It does not: it bottoms out at 7.12 for k equal to 5 and climbs back to 15.46 and then 47.40. The reason is that the temperature readings themselves land in random places each year. At large k the prediction is close to a global average whose value depends heavily on where those 60 days happened to fall on a steep curve. The textbook monotone picture assumes a fixed design; with random sampling, a very stiff model can be both biased and unstable.
Checking the sum against measured error
A decomposition you cannot verify is just a story. Measure the real thing and compare.
def measured_mse(k, repeats=300):
errs = []
for r in range(repeats):
t, y = draw_winter()
model = KNeighborsRegressor(n_neighbors=k).fit(t, y)
fresh = truth + rng.normal(0, SIGMA, len(grid))
errs.append(np.mean((model.predict(grid) - fresh) ** 2))
return float(np.mean(errs))
predicted = {k: tot for k, _, _, tot in table}
for k in (1, 5, 40):
print(f"k={k:<3} decomposition {predicted[k]:7.2f} measured {measured_mse(k):7.2f}")
# k=1 decomposition 28.04 measured 27.54
# k=5 decomposition 19.63 measured 19.53
# k=40 decomposition 540.11 measured 534.31
Within Monte Carlo noise, the three parts add up to the error you actually suffer. That is the claim worth carrying: error has a budget, and every modelling choice moves money between two of the three lines.
What the tradeoff changes in practice
Diagnosis comes first, and the signature of each problem is different. High variance shows as a large gap between training and held-out performance, and it responds to more rows, fewer features, stronger regularisation or a simpler model. High bias shows as poor performance on both, and more rows do nothing for it; you need a richer model or better features. The full diagnostic routine, including learning curves, sits in the overfitting chapter that follows.
The reason regularisation is so widely used is that it buys a controlled trade: add a penalty, accept a little bias, cut variance by more than you paid. That is the whole argument behind ridge and lasso regularisation. Ensembles attack the same term from the other side by averaging many high-variance models.
Where I would push back on the usual advice: the tradeoff is not a dial you set once. It depends on sample size. At 60 rows per year, k equal to 5 wins; at 6,000 rows, k equal to 1 becomes competitive because variance shrinks with the density of the data while bias does not. Any statement of the form “k equal to 5 is the right amount of flexibility” is incomplete without the row count beside it. Keep the held-out slice from a proper train, validation and test split as your referee, and check the KNeighborsRegressor documentation for what the other parameters do.
Frequently Asked Questions
Can I calculate bias and variance on my own dataset?
Not directly, because you would need the true function and the ability to resample from the generating process. What you can do is approximate the variance term with repeated resampling: fit the same model on many bootstrap samples and look at the spread of predictions for a fixed row. Bias stays unmeasurable without the truth.
Does more data fix high bias?
No. Bias is the error of the average model, and averaging more data does not make a straight line curve. More rows cut variance only. If both training and validation scores are poor and close together, spend your effort on features or a more flexible model.
Is low bias always better than low variance?
Neither is a goal on its own, since only their sum plus noise reaches your users. A model with bias squared of 0.05 and variance of 15.74 is worse overall than one at 0.26 and 7.12, which is exactly what k equal to 1 and k equal to 5 show above.
Key Takeaways
- Expected squared error equals bias squared plus variance plus irreducible noise, and the three terms can be computed and verified in simulation.
- A flat gap between poor training and poor validation scores points to bias; a wide gap between good training and poor validation points to variance.
- More data reduces variance and leaves bias untouched, so diagnose before you go hunting for rows.
- The best complexity setting is a function of sample size, so quote a row count whenever you quote a tuned hyperparameter.
- Regularisation and ensembling are both deliberate trades: accept a little bias to buy a larger reduction in variance.