Fitting the mistakes instead of the target

Harrowgate Scaffolding hires out tower kits across three counties, and every contract carries a booked return date that the yard plans the next delivery around. They want to know, at the moment a kit goes out, how many hours late it will come back. Gradient boosting is what I would reach for first on a table like this: six columns, a couple of sharp thresholds, one interaction, 6,000 completed hires.

The mechanism is almost disappointingly plain. Start with a single number, the average overrun. Measure how wrong that number is on every row. Fit a small tree to those errors rather than to the target. Add a fraction of that tree’s output to your running prediction, recompute the errors, repeat. The finished model is a sum of hundreds of small corrections. The word “gradient” is there because, for squared error, the residual you are fitting is exactly the negative gradient of the loss, so each tree is a step downhill in function space rather than in parameter space. If you have read how gradient descent works, this is the same idea with trees standing in for the step direction.

The sequential idea itself belongs to boosting fundamentals, which covers why fitting models one after another beats fitting them side by side. This chapter is about the gradient variant specifically: the arithmetic, the four hyperparameters that matter, and where it breaks.

Watching the residuals shrink

Five depth-one trees, each fitted to what the previous four got wrong. No library magic, just a loop.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import root_mean_squared_error

rng = np.random.default_rng(84)
n = 6000
hire_days = rng.integers(3, 61, n)
kit_tonnes = rng.gamma(3.0, 1.4, n).round(2)
distance_km = rng.uniform(2, 180, n).round(1)
crew_size = rng.integers(2, 9, n)
prior_overruns = rng.poisson(0.7, n)

overrun_hours = (
    0.22 * hire_days
    + 1.8 * kit_tonnes
    + 0.04 * distance_km
    - 1.1 * crew_size
    + 3.4 * prior_overruns
    + 0.09 * hire_days * prior_overruns   # interaction a straight line cannot see
    + rng.normal(0, 4, n)                 # irreducible noise, sd 4 hours
).round(2)

hires = pd.DataFrame({
    "hire_days": hire_days, "kit_tonnes": kit_tonnes,
    "distance_km": distance_km, "crew_size": crew_size,
    "prior_overruns": prior_overruns, "overrun_hours": overrun_hours,
})
X = hires.drop(columns="overrun_hours")
y = hires["overrun_hours"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)

pred_tr = np.full(len(y_tr), y_tr.mean())
pred_te = np.full(len(y_te), y_tr.mean())
print("round 0 (mean only)  test RMSE", round(root_mean_squared_error(y_te, pred_te), 3))

for r in range(1, 6):
    residual = y_tr - pred_tr
    stump = DecisionTreeRegressor(max_depth=1, random_state=0).fit(X_tr, residual)
    pred_tr = pred_tr + 0.5 * stump.predict(X_tr)
    pred_te = pred_te + 0.5 * stump.predict(X_te)
    split_on = X_tr.columns[stump.tree_.feature[0]]
    print(f"round {r} split on {split_on:14s} test RMSE",
          round(root_mean_squared_error(y_te, pred_te), 3))

# round 0 (mean only)  test RMSE 9.684
# round 1 split on prior_overruns test RMSE 8.987
# round 2 split on hire_days      test RMSE 8.284
# round 3 split on kit_tonnes     test RMSE 7.707
# round 4 split on prior_overruns test RMSE 7.212
# round 5 split on hire_days      test RMSE 6.883

Two things in that output are worth more than the falling error. The first is that round 4 splits on prior_overruns again, a column round 1 already used. Nothing stops it: once the easy signal in that column is absorbed, what is left of it is still the largest remaining slice of residual. The second is how slow the progress is. Five stumps took the error from 9.68 to 6.88 hours, and the noise floor built into the data is 4.0. Hundreds of rounds are normal, which is the whole design: each tree is deliberately too weak to do damage.

What the gradient boosting hyperparameters actually do

Learning rate and tree count are a single dial

That 0.5 multiplier is the learning rate, often written as shrinkage. Halve it and you need roughly twice as many trees for the same fit. Readers often tune the two separately and then wonder why nothing behaves monotonically.

from sklearn.ensemble import GradientBoostingRegressor

for lr in (0.5, 0.1, 0.05):
    for ne in (50, 200, 800):
        m = GradientBoostingRegressor(learning_rate=lr, n_estimators=ne,
                                      max_depth=3, random_state=0).fit(X_tr, y_tr)
        print(f"lr={lr:<5} n={ne:<4}",
              "train", round(root_mean_squared_error(y_tr, m.predict(X_tr)), 3),
              "test", round(root_mean_squared_error(y_te, m.predict(X_te)), 3))

# lr=0.5   n=50   train 3.758   test 4.417
# lr=0.5   n=200  train 3.019   test 4.624
# lr=0.5   n=800  train 1.715   test 5.021
# lr=0.1   n=50   train 4.056   test 4.437
# lr=0.1   n=200  train 3.646   test 4.252
# lr=0.1   n=800  train 3.086   test 4.385
# lr=0.05  n=50   train 4.957   test 5.153
# lr=0.05  n=200  train 3.762   test 4.21
# lr=0.05  n=800  train 3.431   test 4.282
learning_rate best n_estimators tried test RMSE train RMSE at n=800
0.5 50 4.417 1.715
0.1 200 4.252 3.086
0.05 200 4.210 3.431

Read the top row again. At learning_rate=0.5 the training error keeps falling all the way to 1.715 while the test error climbs to 5.021, which is worse than the 50-tree model it started from. More boosting rounds make a gradient boosted model worse, reliably, past a point. That is the opposite of a random forest, where extra trees plateau harmlessly, and it is the single most important practical difference between bagging and random forests and this family.

The best score here is 4.210 against a noise floor of 4.0, so there is very little left on the table. Spending a week tuning further would buy nothing real.

Depth, subsampling and early stopping

max_depth controls how many features a single tree can combine. Depth 1 can never represent the hire_days by prior_overruns interaction; depth 3 can. Three to six is the usual range, and going deeper usually calls for a lower learning rate to compensate.

Early stopping removes the need to guess n_estimators at all. Hold back a slice of the training data, watch the score on it, and stop when it stalls.

from sklearn.ensemble import HistGradientBoostingRegressor

h = HistGradientBoostingRegressor(learning_rate=0.05, max_iter=2000,
                                  early_stopping=True, n_iter_no_change=25,
                                  validation_fraction=0.15,
                                  random_state=0).fit(X_tr, y_tr)
print("trees actually built:", h.n_iter_)
print("test RMSE:", round(root_mean_squared_error(y_te, h.predict(X_te)), 3))
# trees actually built: 163
# test RMSE: 4.31

for ss in (1.0, 0.6):
    m = GradientBoostingRegressor(learning_rate=0.05, n_estimators=800, max_depth=3,
                                  subsample=ss, random_state=0).fit(X_tr, y_tr)
    print(f"subsample={ss} test RMSE",
          round(root_mean_squared_error(y_te, m.predict(X_te)), 3))
# subsample=1.0 test RMSE 4.282
# subsample=0.6 test RMSE 4.277

It asked for a budget of 2,000 and stopped at 163. Use HistGradientBoostingRegressor rather than the original GradientBoostingRegressor for anything above a few thousand rows: it bins each feature into at most 255 buckets before splitting, which makes it an order of magnitude faster at the same accuracy.

The subsampling result is the honest negative of this chapter. Stochastic gradient boosting, where each tree sees only 60 per cent of the rows, moved test RMSE from 4.282 to 4.277. That is nothing. Subsampling earns its keep on noisier, wider, more correlated data than this, and on a clean six-column table you should not expect it to rescue anything.

Where it is the wrong choice

Three cases, stated plainly. Very wide and sparse data, such as bag-of-words with 50,000 columns, where a linear model fits in seconds and boosting grinds. Data with strong extrapolation requirements, because a sum of trees is piecewise constant and will happily predict the same overrun for a 90 day hire as for a 60 day one it has never seen beyond. And any setting where you must explain a single prediction to a regulator by hand, since 163 trees of depth 3 is not a story anyone can tell without extra tooling.

It also punishes carelessness about leakage harder than most algorithms, because it is good enough to find a leaking column and lean on it entirely. The scikit-learn ensemble user guide documents the loss functions available, including loss="absolute_error" and quantile losses that are worth knowing when overruns have a long right tail.

Frequently Asked Questions

Is gradient boosting better than random forest?

On tabular data with a tuned learning rate, usually yes, by a small margin. A random forest is better when you have no time to tune, because its defaults are nearly unimprovable, and when you need the training to be trivially parallel. Boosting is sequential by construction, so a single model cannot be spread across machines the same way.

How many trees should I use?

Do not pick a number. Set early_stopping=True with a patience of 20 to 50 rounds and a generous max_iter, then read off what it built. Guessing n_estimators by hand is how people end up on the wrong side of the overfitting curve shown above.

Why does my training error keep falling while test error rises?

That is boosting working as designed and being allowed to run too long. Each round fits the remaining residual, and after the signal is used up the only residual left is noise. Lower the learning rate, cap the depth, or stop earlier.

Key Takeaways

  • Gradient boosting fits each new tree to the errors of the trees before it, so the model is a sum of small corrections rather than an average of independent guesses.
  • Treat learning_rate and n_estimators as one dial: lowering the rate means raising the count, and tuning them separately produces results that look random.
  • Set early_stopping=True with a patience and a generous iteration budget instead of guessing how many trees you need; on the scaffolding data that turned 2,000 into 163.
  • Extra trees can make a boosted model actively worse, which is the clearest practical difference between boosting and bagging.
  • Reach for HistGradientBoostingRegressor above a few thousand rows, and do not expect subsample to do much on clean, narrow tables.