The instability a single tree leaves behind

Fit a tree on one bootstrap sample of your data and the root splits on distance_km < 14.11. Resample and refit, and the root moves to bins_lifted < 359.5. How decision trees pick their splits ended on exactly that observation and left it hanging. Ensemble learning is the answer: fit many unstable models on perturbed copies of the same training set, then average what they say.

Averaging does not improve any individual tree. It improves the thing you ship. On the route data below, the average test MSE of a single tree is 117.07 minutes squared. Average twenty-five of them and that falls to 59.59, a reduction of 49 per cent, with no new features, no tuning and no extra data. The reason is specific and measurable: almost all of a deep tree’s error is variance, the part that changes when the training sample changes, and averaging twenty-five independent-ish estimates of the same quantity shrinks variance without touching bias.

A waste-collection contractor keeps 3,000 route records and wants to predict how many minutes a round takes.

import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(14)
n = 3000
bins_lifted   = rng.integers(180, 520, n)
distance_km   = rng.uniform(4, 26, n)
terrace_share = rng.beta(2, 3, n)
crew_size     = rng.integers(2, 4, n)
minutes = (22 + 0.11*bins_lifted + 1.9*distance_km + 38*terrace_share
           - 6*crew_size + rng.normal(0, 7, n))

X = np.column_stack([bins_lifted, distance_km, terrace_share, crew_size])
X_train, X_test, y_train, y_test = train_test_split(
    X, minutes, test_size=0.3, random_state=0)

names = ["bins_lifted", "distance_km", "terrace_share", "crew_size"]
boot = np.random.default_rng(7)
preds, roots, single_mse = [], [], []
for _ in range(25):
    idx = boot.integers(0, len(X_train), len(X_train))
    t = DecisionTreeRegressor(random_state=0).fit(X_train[idx], y_train[idx])
    preds.append(t.predict(X_test))
    single_mse.append(mean_squared_error(y_test, preds[-1]))
    roots.append((names[t.tree_.feature[0]], round(float(t.tree_.threshold[0]), 2)))

preds = np.array(preds)
print("distinct root thresholds in 25 trees:", len({r[1] for r in roots}))
print("mean MSE of one tree      : %.2f" % np.mean(single_mse))
print("MSE of the 25-tree average: %.2f" % mean_squared_error(y_test, preds.mean(axis=0)))
print("mean per-point variance   : %.2f" % preds.var(axis=0).mean())
# distinct root thresholds in 25 trees: 20
# mean MSE of one tree      : 117.07
# MSE of the 25-tree average: 59.59
# mean per-point variance   : 57.48

Twenty of the twenty-five trees chose a different root threshold, and two different root features appeared. That is not a bug in the algorithm. It is what a greedy splitter does when two features carry similar information and the sample shifts slightly.

What ensemble learning actually removes

The numbers above already tell the story if you line them up. Mean single-tree MSE is 117.07. Mean per-point variance across the twenty-five trees is 57.48. The ensemble’s MSE is 59.59, and 117.07 minus 57.48 is 59.59 to the penny. The variance term did not shrink. It vanished.

Because this data was generated from a formula, you can go further and measure bias and variance separately against the true function rather than against noisy observations.

f_true = (22 + 0.11*X_test[:, 0] + 1.9*X_test[:, 1]
          + 38*X_test[:, 2] - 6*X_test[:, 3])
dev = preds - f_true

print("irreducible noise in y        : %.2f" % np.mean((y_test - f_true)**2))
print("one tree,  error against f    : %.2f" % np.mean(dev**2))
print("   of which variance          : %.2f" % dev.var(axis=0).mean())
print("   of which squared bias      : %.2f" % (dev.mean(axis=0)**2).mean())
print("25 trees,  error against f    : %.2f" % np.mean((preds.mean(axis=0) - f_true)**2))

second = []
boot3 = np.random.default_rng(31)
for _ in range(25):
    idx = boot3.integers(0, len(X_train), len(X_train))
    second.append(DecisionTreeRegressor(random_state=0)
                  .fit(X_train[idx], y_train[idx]).predict(X_test))
second = np.array(second)
print("two single trees disagree by    : %.2f minutes" % np.abs(preds[0] - second[0]).mean())
print("two 25-tree averages disagree by: %.2f minutes"
      % np.abs(preds.mean(axis=0) - second.mean(axis=0)).mean())
for k in (1, 3, 5, 10, 25):
    print("%2d trees -> MSE %.2f"
          % (k, mean_squared_error(y_test, preds[:k].mean(axis=0))))
# irreducible noise in y        : 47.95
# one tree,  error against f    : 72.13
#    of which variance          : 57.48
#    of which squared bias      : 14.64
# 25 trees,  error against f    : 14.64
# two single trees disagree by    : 8.03 minutes
# two 25-tree averages disagree by: 1.68 minutes
#  1 trees -> MSE 118.36
#  3 trees -> MSE 79.72
#  5 trees -> MSE 70.27
# 10 trees -> MSE 64.42
# 25 trees -> MSE 59.59

One tree misses the true route time by 72.13 on a squared scale, split 57.48 variance and 14.64 squared bias. The twenty-five-tree average misses by 14.64. Every unit of variance is gone and every unit of bias remains, which is the whole mechanism in one line. The disagreement figures say the same thing in units a depot manager would use: two single trees quote arrival times 8.03 minutes apart on average, two ensembles 1.68 minutes apart.

Returns arrive fast and then flatten:

Trees averaged Test MSE
1 118.36
3 79.72
5 70.27
10 64.42
25 59.59

Going from one tree to five buys 48 points. Going from ten to twenty-five buys under 5. This is why the variance half of the bias-variance tradeoff matters more than tree count once you are past a few dozen trees.

Where averaging buys you nothing

Averaging only removes variance, so a model with no variance to remove gains nothing. The same twenty-five bootstrap samples, fitted with ordinary least squares instead of trees, produce a mean single-model MSE of 48.15 and an averaged MSE of 48.03. Per-point variance across the twenty-five linear fits is 0.122, against 57.48 for the trees. There is simply nothing there to average away.

from sklearn.linear_model import LinearRegression

lin, lin_mse = [], []
boot2 = np.random.default_rng(7)
for _ in range(25):
    idx = boot2.integers(0, len(X_train), len(X_train))
    p = LinearRegression().fit(X_train[idx], y_train[idx]).predict(X_test)
    lin.append(p)
    lin_mse.append(mean_squared_error(y_test, p))
lin = np.array(lin)
print("linear: one model %.2f | averaged %.2f | variance %.3f"
      % (np.mean(lin_mse), mean_squared_error(y_test, lin.mean(axis=0)),
         lin.var(axis=0).mean()))
# linear: one model 48.15 | averaged 48.03 | variance 0.122

Look harder at those numbers and something awkward falls out. The single linear model scores 48.15 and the twenty-five-tree ensemble scores 59.59. The ensemble loses, badly, to the simplest model in the course. That is not a flaw in the method. The route times were generated as a linear function plus noise, so the trees carry 14.64 of structural bias that no amount of averaging touches, while linear regression carries almost none. Ensembling a high-bias learner on a problem that suits a low-bias learner wastes compute and still comes second.

So the rule is narrower than the enthusiasm around ensembles suggests. Ensemble learning pays when your base learner is unstable and roughly unbiased: deep trees, nearest-neighbour models with tiny k, anything whose predictions swing when you shuffle the data. It pays very little on linear or heavily regularised models, and it cannot rescue a base learner that is systematically wrong. Check the variance before you reach for it. If two models fitted on resamples of your data agree to within a rounding error, averaging them is a no-op with twenty-five times the training cost.

A second condition hides in the arithmetic. Averaging works because the trees’ errors partly cancel, which requires them to be different. Twenty-five identical trees average to one tree. The bootstrap gives you some of that difference for free, and the next chapter adds a second source of it deliberately by starving each tree of features. The tree-based models chapter in the Predictive Modelling course covers how to read these models in a business report; here the concern is the mechanics. The scikit-learn ensemble user guide lists which estimators implement the averaging and which implement the sequential alternative.

Frequently Asked Questions

How many models do I need in an ensemble?

More is never worse for accuracy, only slower. On the route data, five trees captured 82 per cent of the total gain and twenty-five captured all of it. Start at 100 for trees, plot your score against tree count once, and stop where the curve flattens. Past that point you are paying for prediction latency.

Does an ensemble reduce overfitting?

It reduces variance, which is part of overfitting but not all of it. An averaged ensemble of deep trees still memorises the training set almost perfectly. What changes is that the memorisation differs between members and cancels, so held-out error drops while training error stays near zero.

Can I ensemble completely different model types together?

Yes, and mixing families often decorrelates errors more than resampling does. Averaging the predictions of a tree, a linear model and a nearest-neighbour model is the crude version. Weighting them by a second model is stacking, which has its own chapter in this section.

Key Takeaways

  • Check your base learner’s variance before ensembling, because averaging removes only the variance component and leaves bias untouched: the route trees went from 57.48 variance to effectively none while 14.64 of squared bias survived.
  • Expect most of the gain in the first handful of models; one to five trees cut MSE by 48 points, ten to twenty-five cut it by under 5.
  • Do not ensemble a stable learner. Twenty-five averaged linear models scored 48.03 against 48.15 for one, a gain that does not pay for the training time.
  • Remember that an ensemble of a biased learner stays biased, which is why a single linear regression beat the twenty-five-tree average on this linear problem.
  • Make the members disagree if you want the averaging to work, since identical models average to one model.