Fixing your own mistakes, one stump at a time

Bagging fits its members independently and averages them, so every tree is ignorant of what the others got wrong. Boosting in machine learning does the opposite. Models are fitted one after another, and before each new model is trained the rows that the current committee misclassifies have their weights raised, so the next model is pushed to attend to exactly those rows. The output is a weighted vote, not an average, with better models getting louder votes.

The effect on a deliberately feeble base learner is large. A single decision stump, one split and two leaves, classifies punnets at 0.819 test accuracy below. Two hundred stumps combined by AdaBoost reach 0.939. No stump in that sequence is individually any good. The sequence is good because each one is fitted on a problem the previous ones have already deformed in its favour. Where bagging works by cancelling variance, boosting works by chipping away at bias, which is why its base learners are deliberately shallow rather than fully grown.

A fruit packhouse grades 3,600 punnets. A punnet is rejected if the berries are undersized, or too bruised, or soft and several days old, with 4 per cent of the labels wrong because graders disagree.

import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

rng = np.random.default_rng(83)
n = 3600
mean_berry_g    = rng.normal(12.5, 1.9, n)
bruise_count    = rng.poisson(1.8, n)
firmness        = rng.normal(58, 9, n)
sugar_brix      = rng.normal(9.2, 1.1, n)
days_since_pick = rng.integers(0, 6, n)

reject = ((mean_berry_g < 11.0) | (bruise_count > 3) |
          ((firmness < 50) & (days_since_pick >= 3))).astype(int)
reject = np.where(rng.random(n) < 0.04, 1 - reject, reject)

X = np.column_stack([mean_berry_g, bruise_count, firmness,
                     sugar_brix, days_since_pick])
X_tr, X_te, y_tr, y_te = train_test_split(X, reject, test_size=0.3,
                                          random_state=0, stratify=reject)
stump = DecisionTreeClassifier(max_depth=1, random_state=0).fit(X_tr, y_tr)
print("one stump, test accuracy %.3f" % accuracy_score(y_te, stump.predict(X_te)))
# one stump, test accuracy 0.819

How boosting in machine learning reweights the rows

AdaBoost is worth writing out by hand once, because the whole algorithm is four lines and the usual explanations make it sound mysterious. Every row starts with equal weight. Fit a stump on the weighted data, measure its weighted error, convert that error into a vote strength alpha, then multiply the weight of every row by exp(-alpha) if the stump got it right and exp(+alpha) if it got it wrong. Renormalise and repeat.

w = np.ones(len(X_tr)) / len(X_tr)
signed = np.where(y_tr == 1, 1, -1)
models, alphas = [], []

for round_no in range(1, 201):
    s = DecisionTreeClassifier(max_depth=1, random_state=0).fit(X_tr, y_tr, sample_weight=w)
    pred = np.where(s.predict(X_tr) == 1, 1, -1)
    err = max(w[pred != signed].sum(), 1e-10)
    alpha = 0.5 * np.log((1 - err) / err)
    w = w * np.exp(-alpha * signed * pred)
    w /= w.sum()
    models.append(s)
    alphas.append(alpha)
    if round_no in (1, 5, 25, 100, 200):
        def vote(Xs):
            agg = sum(a * np.where(m.predict(Xs) == 1, 1, -1)
                      for m, a in zip(models, alphas))
            return np.where(agg > 0, 1, 0)
        print("%4d stumps  train %.3f  test %.3f  weighted error %.3f"
              % (round_no, accuracy_score(y_tr, vote(X_tr)),
                 accuracy_score(y_te, vote(X_te)), err))

hardest = int(0.05 * len(w))
print("weight on the hardest %d rows: %.3f"
      % (hardest, np.sort(w)[::-1][:hardest].sum()))
#    1 stumps  train 0.822  test 0.819  weighted error 0.178
#    5 stumps  train 0.835  test 0.834  weighted error 0.402
#   25 stumps  train 0.925  test 0.931  weighted error 0.470
#  100 stumps  train 0.935  test 0.941  weighted error 0.468
#  200 stumps  train 0.934  test 0.939  weighted error 0.497
# weight on the hardest 126 rows: 0.364

The alpha formula is less arbitrary than it looks. A stump with weighted error 0.178 gets alpha of 0.765, a loud vote. A stump with error 0.497 gets 0.006, effectively silenced. A stump worse than chance would get a negative alpha and have its predictions inverted, which is correct behaviour.

Watch the weighted error column. It starts at 0.178 and climbs to 0.497, almost exactly chance. That is the algorithm working, not failing. Each round reweights the data until the previous solution looks no better than guessing on the new weighting, so the next stump is forced somewhere new. By round 200, 36.4 per cent of all sample weight sits on the hardest 5 per cent of rows: the mislabelled punnets and the genuinely ambiguous ones near the size threshold.

from sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier

stump_maker = DecisionTreeClassifier(max_depth=1, random_state=0)
ada = AdaBoostClassifier(stump_maker, n_estimators=200, random_state=0).fit(X_tr, y_tr)
forest = RandomForestClassifier(n_estimators=200, random_state=0, n_jobs=-1).fit(X_tr, y_tr)
print("scikit-learn AdaBoost %.3f" % accuracy_score(y_te, ada.predict(X_te)))
print("random forest         %.3f" % accuracy_score(y_te, forest.predict(X_te)))
for lr in (0.1, 0.5, 1.0):
    row = [accuracy_score(y_te, AdaBoostClassifier(stump_maker, n_estimators=ne,
           learning_rate=lr, random_state=0).fit(X_tr, y_tr).predict(X_te))
           for ne in (50, 200)]
    print("learning_rate=%.1f  50 rounds %.3f  200 rounds %.3f" % (lr, row[0], row[1]))
# scikit-learn AdaBoost 0.939
# random forest         0.968
# learning_rate=0.1  50 rounds 0.819  200 rounds 0.898
# learning_rate=0.5  50 rounds 0.898  200 rounds 0.937
# learning_rate=1.0  50 rounds 0.941  200 rounds 0.939

The hand-written loop and AdaBoostClassifier agree to three decimals at 0.939, which is the point of writing it out. The learning_rate table is less flattering. Shrinkage trades rounds for caution, and at 0.1 fifty rounds gets you precisely one stump’s worth of accuracy, 0.819. Full-strength boosting at 50 rounds scored 0.941, the best number in the whole chapter. Shrinkage did not help here, and that is the usual result for AdaBoost with stumps on clean tabular data; it earns its keep with the gradient-based variants covered next, where thousands of rounds are normal.

The uncomfortable comparison is the random forest at 0.968 against boosted stumps at 0.939. The generating rule contains firmness < 50 AND days_since_pick >= 3, a two-way interaction. A stump cannot represent a conjunction at all, so the boosted model has to approximate it with a sum of many single splits, while an unpruned tree captures it in two levels. Boosting is not uniformly better than bagging. On problems whose structure needs depth, the shallow base learner is the binding constraint, and the fix is a deeper base learner rather than more rounds.

Where boosting is the wrong choice

The standard warning is that boosting chases mislabelled rows, since the exponential weight update grows without bound on a row that can never be classified correctly. Testing that produced a result I did not expect.

for frac in (0.15, 0.30):
    noisy = y_tr.copy()
    flipped = np.random.default_rng(5).choice(len(noisy), int(frac*len(noisy)), replace=False)
    noisy[flipped] = 1 - noisy[flipped]
    a = AdaBoostClassifier(stump_maker, n_estimators=200, random_state=0).fit(X_tr, noisy)
    f = RandomForestClassifier(n_estimators=200, random_state=0, n_jobs=-1).fit(X_tr, noisy)
    print("%.0f%% flipped: test acc  adaboost %.3f  forest %.3f"
          % (frac*100, accuracy_score(y_te, a.predict(X_te)),
             accuracy_score(y_te, f.predict(X_te))))
    print("            agrees with flipped labels  adaboost %.3f  forest %.3f"
          % (accuracy_score(noisy[flipped], a.predict(X_tr[flipped])),
             accuracy_score(noisy[flipped], f.predict(X_tr[flipped]))))
# 15% flipped: test acc  adaboost 0.933  forest 0.956
#             agrees with flipped labels  adaboost 0.079  forest 1.000
# 30% flipped: test acc  adaboost 0.894  forest 0.852
#             agrees with flipped labels  adaboost 0.112  forest 1.000

At 15 per cent corruption the forest still wins, 0.956 against 0.933. At 30 per cent the ranking flips and AdaBoost wins by four points, 0.894 against 0.852. The last column explains why. The forest reproduces every single flipped label it was given, because fully grown trees have the capacity to isolate each corrupted row in its own leaf. AdaBoost with stumps reproduces only 7.9 per cent and 11.2 per cent of them, because a one-split model cannot carve out individual rows no matter how much weight they carry.

So the warning is about the combination, not about boosting alone. Exponential loss plus a base learner with capacity to fit noise is what collapses. Exponential loss plus a stump mostly just wastes rounds. If your labels are dirty, the real lever is base-learner depth, and the corresponding risk on the bagging side is unlimited tree growth, which is also what overfitting looks like in a forest.

Use boosting when your base learner is high-bias and low-variance, which means shallow trees rather than deep ones. Avoid it when you cannot afford sequential training, since rounds cannot be parallelised the way bagging can, and when label quality is unknown and you have no held-out set to catch the degradation. Freund and Schapire’s original AdaBoost paper and the AdaBoostClassifier reference are the sources to read next.

Frequently Asked Questions

Is boosting always better than bagging?

No. On the packhouse data a random forest scored 0.968 against 0.939 for 200 boosted stumps, because the true rule contains a two-way interaction that a one-split model cannot express. Boosting wins when you can afford a slightly deeper base learner and enough rounds; bagging wins when the structure needs depth immediately.

Why must the base learner be weak?

Because boosting reduces bias by accumulation. A strong base learner fits the training data in round one, leaving nothing for later rounds to correct, and the reweighting then amplifies noise instead of signal. Stumps and depth-2 or depth-3 trees are the usual choices.

What does learning_rate actually do?

It scales each model’s vote, so more rounds are needed to reach the same total fit. It is insurance against overshooting on noisy data. On this clean dataset it cost accuracy: 0.1 with 50 rounds gave 0.819 while 1.0 with 50 rounds gave 0.941.

Key Takeaways

  • Reach for boosting when the base learner is high-bias, since 200 stumps lifted accuracy from 0.819 to 0.939 on a problem where a single stump was nearly useless.
  • Expect the weighted error of each new model to drift towards 0.5, as it did from 0.178 to 0.497 here, because that is the sign the reweighting is forcing genuinely new models.
  • Do not assume shrinkage helps; learning_rate=1.0 with 50 rounds beat every slower setting tested, and low rates only pay off over thousands of rounds.
  • Check whether your problem needs depth before blaming the number of rounds, because an unpruned random forest beat boosted stumps 0.968 to 0.939 on a rule containing a two-way interaction.
  • Control base-learner capacity rather than the loss when labels are dirty, since the forest reproduced 100 per cent of flipped labels at 30 per cent corruption while stumps reproduced 11.2 per cent.