From one tree to a bag of them

Bagging is the mechanical version of the variance argument from the previous chapter: draw bootstrap samples, fit one unpruned tree per sample, average or vote. A random forest keeps that and adds one line of extra randomness. At every split, each tree is allowed to consider only a random subset of the features rather than all of them, which forces the trees to disagree more than resampling alone achieves.

That single change is worth having. On the plant-hire data below, a lone tree reaches 0.582 AUC, bagging 300 of them reaches 0.671, and a random forest with the same 300 trees reaches 0.682. Measured pairwise correlation between member trees drops from 0.160 under bagging to 0.136 under the forest, and that drop is where the extra AUC comes from. Nothing else differs between the two models.

A plant hire firm has 4,400 excavator returns and wants to flag which machines will come back needing unscheduled repair. The asset_tag column is deliberately useless, a five-digit identifier with no predictive content, because it will matter later.

import numpy as np, pandas as pd
from sklearn.ensemble import RandomForestClassifier, BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, roc_auc_score

rng = np.random.default_rng(82)
n = 4400
hours_used     = rng.gamma(4, 320, n).round()
machine_age    = rng.uniform(0.5, 14, n).round(1)
operator_grade = rng.integers(1, 5, n)
wet_days       = rng.poisson(6, n)
site_type      = rng.choice([0, 1, 2], n, p=[0.45, 0.35, 0.20])
asset_tag      = rng.integers(10000, 99999, n)

logit = (-3.4 + 0.00052*hours_used + 0.16*machine_age - 0.30*operator_grade
         + 0.055*wet_days + 0.55*site_type)
needs_repair = (rng.random(n) < 1/(1 + np.exp(-logit))).astype(int)

df = pd.DataFrame({"hours_used": hours_used, "machine_age": machine_age,
                   "operator_grade": operator_grade, "wet_days": wet_days,
                   "site_type": site_type, "asset_tag": asset_tag})
X_tr, X_te, y_tr, y_te = train_test_split(df, needs_repair, test_size=0.3,
                                          random_state=0, stratify=needs_repair)

tree = DecisionTreeClassifier(random_state=0).fit(X_tr, y_tr)
bag = BaggingClassifier(DecisionTreeClassifier(random_state=0),
                        n_estimators=300, random_state=0, n_jobs=-1).fit(X_tr, y_tr)
forest = RandomForestClassifier(n_estimators=300, random_state=0,
                                n_jobs=-1, oob_score=True).fit(X_tr, y_tr)
for label, m in [("single tree", tree), ("bagging", bag), ("random forest", forest)]:
    print("%-14s accuracy %.3f  auc %.3f"
          % (label, accuracy_score(y_te, m.predict(X_te)),
             roc_auc_score(y_te, m.predict_proba(X_te)[:, 1])))
print("forest out-of-bag score %.3f" % forest.oob_score_)
# single tree    accuracy 0.708  auc 0.582
# bagging        accuracy 0.784  auc 0.671
# random forest  accuracy 0.799  auc 0.682
# forest out-of-bag score 0.786

oob_score=True is the part people skip. Each bootstrap sample leaves roughly a third of the rows untouched, so every row has a subset of trees that never saw it, and scoring those gives you a held-out estimate for free. Here out-of-bag accuracy came in at 0.786 against a test accuracy of 0.799, close enough to trust on a dataset too small to split three ways. Read the AUC and threshold material if the 0.68 figure needs interpreting; on this data the signal is genuinely weak and no ensemble fixes that.

What a random forest adds to bagging

The lever is max_features, the number of candidate features drawn at each split. Lower means more decorrelation and weaker individual trees. Higher means stronger trees that all look alike. With six features the choice spans the whole range from one to six, so you can watch both effects at once.

def tree_correlation(model):
    P = np.array([t.predict_proba(X_te.values)[:, 1] for t in model.estimators_[:60]])
    C = np.corrcoef(P)
    return (C.sum() - len(C)) / (len(C) * (len(C) - 1))

print("mean pairwise tree correlation: bagging %.3f | forest %.3f"
      % (tree_correlation(bag), tree_correlation(forest)))
for mf in (1, 2, 3, 4, 6):
    m = RandomForestClassifier(n_estimators=300, max_features=mf, random_state=0,
                               n_jobs=-1, oob_score=True).fit(X_tr, y_tr)
    print("max_features=%d  oob %.3f  test auc %.3f  tree corr %.3f"
          % (mf, m.oob_score_, roc_auc_score(y_te, m.predict_proba(X_te)[:, 1]),
             tree_correlation(m)))
# mean pairwise tree correlation: bagging 0.160 | forest 0.136
# max_features=1  oob 0.795  test auc 0.690  tree corr 0.134
# max_features=2  oob 0.786  test auc 0.682  tree corr 0.136
# max_features=3  oob 0.789  test auc 0.678  tree corr 0.145
# max_features=4  oob 0.790  test auc 0.675  tree corr 0.156
# max_features=6  oob 0.790  test auc 0.667  tree corr 0.160

Correlation climbs monotonically with max_features and test AUC falls monotonically with it. max_features=6 is bagging by another name, and it is the worst setting of the five. One feature per split, the most aggressive decorrelation available, is the best.

Two honest observations about that table. First, the whole range spans 0.023 AUC, so max_features is a real but modest lever, not a rescue. Second, and more awkwardly, out-of-bag accuracy barely moved across the sweep: 0.786 to 0.795, with the ranking almost reversed against test AUC. Out-of-bag scoring is excellent as a sanity check on whether the model learned anything, and it was a poor guide to choosing max_features here, because accuracy on a 20 per cent positive rate is dominated by the majority class. If you tune on out-of-bag, tune on an out-of-bag probability score, not accuracy.

Reading the importances without being fooled

feature_importances_ on a fitted forest reports mean impurity decrease, and it has a known bias towards features with many distinct values. The useless asset_tag column is there to expose it.

from sklearn.inspection import permutation_importance
impurity = pd.Series(forest.feature_importances_, index=df.columns)
perm = permutation_importance(forest, X_te, y_te, n_repeats=10,
                              random_state=0, scoring="roc_auc")
print(pd.DataFrame({"impurity": impurity.round(3),
                    "permutation": pd.Series(perm.importances_mean,
                                             index=df.columns).round(4)})
      .sort_values("impurity", ascending=False).to_string())
Feature Impurity Permutation
hours_used 0.274 0.0179
asset_tag 0.244 0.0047
machine_age 0.243 0.0739
wet_days 0.123 0.0059
operator_grade 0.059 0.0178
site_type 0.057 0.0212

Impurity importance ranks a random five-digit identifier second out of six, above machine_age, which is the strongest genuine predictor in the generating formula. It does that because a high-cardinality column offers thousands of candidate thresholds, so it wins splits by luck deep in the tree where the sample is small. Permutation importance, which shuffles one column on held-out data and measures the AUC it costs, puts asset_tag last at 0.0047 and machine_age first at 0.0739. Use permutation importance, or drop identifier columns before fitting, or both. This also explains why feature selection by model importance goes wrong on raw forests.

Where a random forest is the wrong choice: anything needing extrapolation, because the prediction is an average of training leaves and cannot go above the highest or below the lowest value it saw. Smooth monotone relationships also cost a forest more trees than a linear model needs parameters. The tree-based models chapter in the Predictive Modelling course covers the reporting side. For the algorithm itself, Breiman’s 1996 paper Bagging Predictors is the origin, and the scikit-learn guide to randomised tree ensembles plus the RandomForestClassifier parameter reference are the references worth keeping open.

Frequently Asked Questions

Is a random forest just bagging with trees?

Nearly, but the extra feature subsampling at each split is what separates them. Set max_features to the full feature count and a forest becomes bagging exactly. On the excavator data that change cost 0.023 AUC and raised tree correlation from 0.134 to 0.160, so the distinction is worth keeping.

What should I set max_features to?

Start at the scikit-learn default, the square root of the feature count for classification, then sweep it. On wide data with many weakly informative columns, lower values usually win. Here the lowest possible value, one feature per split, was best, which is common when features are few and correlated.

Do I still need cross-validation if I have the out-of-bag score?

For a go or no-go read on whether the model works, the out-of-bag score is enough and free. For model selection, prefer proper validation with a probability-based metric, since out-of-bag accuracy was almost uninformative across the max_features sweep above.

Key Takeaways

  • Set oob_score=True on every forest you fit, because the out-of-bag estimate costs nothing and caught a 0.786 against 0.799 agreement with the test set here.
  • Treat max_features as the one hyperparameter that matters most, since it controls member correlation directly: 0.134 at one feature per split against 0.160 using all six, worth 0.023 AUC.
  • Never read feature_importances_ on data containing identifiers or other high-cardinality columns, which ranked a meaningless asset_tag second out of six features.
  • Use permutation_importance on held-out data instead, which correctly demoted asset_tag to last and promoted machine_age to first.
  • Reach for a linear model instead when you need to predict beyond the range of the training target, because forests average training leaves and cannot extrapolate.