Letting a model decide which models to trust
A soft-voting ensemble averages its members’ probabilities with equal weight. Model stacking replaces that average with a second model, trained on the first models’ predictions, which learns the weights from data instead of assuming them. The predictions it trains on must come from cross-validation rather than from the same rows the base models were fitted on, and that one detail is the whole technique.
So the honest framing before any code: stacking is a small, reliable gain bought with a large increase in moving parts. On the gym renewal data below it added 0.0008 to 0.0027 AUC over the best single model across five splits. Positive every time, and never enough to matter unless you are competing on a leaderboard or the metric converts directly into money.
The dataset is 9,000 memberships from a four-site gym chain, predicting whether a member fails to renew at twelve months. It is built so that different model families have genuinely different strengths: visits per week and tenure act linearly, while distance over seven kilometres and the combination of under-30 with zero personal-training sessions act as sharp thresholds.
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split, cross_val_predict, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import (RandomForestClassifier, HistGradientBoostingClassifier,
StackingClassifier, VotingClassifier)
from sklearn.metrics import roc_auc_score
rng = np.random.default_rng(86)
n = 9000
visits_per_week = rng.gamma(2.2, 1.1, n).round(2)
months_held = rng.integers(12, 61, n)
pt_sessions = rng.poisson(1.3, n)
distance_km = rng.exponential(4.0, n).round(2)
age = rng.integers(18, 71, n)
price_rise_pct = rng.choice([0, 3, 6, 11], n, p=[.4, .3, .2, .1])
logit = (0.1
- 0.95 * visits_per_week - 0.012 * months_held + 0.07 * price_rise_pct
+ 1.3 * (distance_km > 7)
+ 0.9 * ((age < 30) & (pt_sessions == 0))
+ rng.normal(0, 0.6, n))
y = (rng.uniform(size=n) < 1 / (1 + np.exp(-logit))).astype(int)
X = pd.DataFrame({"visits_per_week": visits_per_week, "months_held": months_held,
"pt_sessions": pt_sessions, "distance_km": distance_km,
"age": age, "price_rise_pct": price_rise_pct})
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
random_state=0, stratify=y)
def base_models():
return [
("logit", make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))),
("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=45))),
("forest", RandomForestClassifier(n_estimators=400, min_samples_leaf=5,
random_state=0)),
("hgb", HistGradientBoostingClassifier(max_iter=300, learning_rate=0.06,
random_state=0)),
]
for name, m in base_models():
m.fit(X_tr, y_tr)
print(f"{name:8s} test AUC {roc_auc_score(y_te, m.predict_proba(X_te)[:, 1]):.4f}")
# logit test AUC 0.7780
# knn test AUC 0.7477
# forest test AUC 0.7718
# hgb test AUC 0.7564
Logistic regression leads. That is worth sitting with for a moment: a linear model beating both tree ensembles on data that contains two deliberate threshold effects. The linear part of the signal is simply stronger than the threshold part, and how gradient boosting machines fit each tree does not help when most of the structure is a straight line.
What model stacking actually trains
The meta-learner needs one column per base model, holding that model’s prediction for rows it did not see during fitting. cross_val_predict produces exactly that, and StackingClassifier does it internally.
cv = StratifiedKFold(5, shuffle=True, random_state=0)
oof = pd.DataFrame({name: cross_val_predict(m, X_tr, y_tr, cv=cv,
method="predict_proba")[:, 1]
for name, m in base_models()})
print(oof.corr().round(3).to_string())
# logit knn forest hgb
# logit 1.000 0.864 0.909 0.819
# knn 0.864 1.000 0.875 0.780
# forest 0.909 0.875 1.000 0.918
# hgb 0.819 0.780 0.918 1.000
vote = VotingClassifier(base_models(), voting="soft").fit(X_tr, y_tr)
print("soft vote ", round(roc_auc_score(y_te, vote.predict_proba(X_te)[:, 1]), 4))
stack = StackingClassifier(base_models(),
final_estimator=LogisticRegression(max_iter=1000),
cv=5, n_jobs=-1).fit(X_tr, y_tr)
print("stack (logit) ", round(roc_auc_score(y_te, stack.predict_proba(X_te)[:, 1]), 4))
print("meta weights ", dict(zip([n for n, _ in base_models()],
stack.final_estimator_.coef_[0].round(3).tolist())))
stack2 = StackingClassifier(base_models(),
final_estimator=LogisticRegression(max_iter=1000),
cv=5, passthrough=True, n_jobs=-1).fit(X_tr, y_tr)
print("stack passthrough", round(roc_auc_score(y_te, stack2.predict_proba(X_te)[:, 1]), 4))
stack3 = StackingClassifier(base_models(), cv=5, n_jobs=-1,
final_estimator=RandomForestClassifier(
n_estimators=300, min_samples_leaf=20,
random_state=0)).fit(X_tr, y_tr)
print("stack forest meta", round(roc_auc_score(y_te, stack3.predict_proba(X_te)[:, 1]), 4))
# soft vote 0.774
# stack (logit) 0.7765
# meta weights {'logit': 3.329, 'knn': 2.401, 'forest': 1.787, 'hgb': -0.413}
# stack passthrough 0.7789
# stack forest meta 0.7591
Read the table of correlations first. Every pair sits between 0.78 and 0.92, which is the problem. Stacking pays for disagreement, and these four models mostly agree. The 0.780 pair, k-nearest neighbours against histogram boosting, is the most useful thing in the matrix because it is the only one even close to independent.
Now the results. Plain soft voting scored 0.7740, which is worse than logistic regression alone at 0.7780. The stack with a logistic meta-learner reached 0.7765, still short of the single model. Only passthrough=True, which hands the meta-learner the original six features alongside the four predictions, got ahead at 0.7789. A random forest meta-learner was the worst option at 0.7591, which is the usual outcome: the meta-learner should be low variance, and a flexible model fitted to four highly correlated columns of probabilities overfits them.
The negative weight on hgb is informative rather than a bug. The meta-learner has decided that where boosting disagrees with the other three, the other three are right, so it subtracts.
The leakage that destroys it
Skip the cross-validation step, train the meta-learner on predictions the base models made for their own training rows, and this happens.
fitted = [(nm, m.fit(X_tr, y_tr)) for nm, m in base_models()]
insample = np.column_stack([m.predict_proba(X_tr)[:, 1] for _, m in fitted])
leaky = LogisticRegression(max_iter=1000).fit(insample, y_tr)
test_feats = np.column_stack([m.predict_proba(X_te)[:, 1] for _, m in fitted])
print("leaky meta ", round(roc_auc_score(y_te, leaky.predict_proba(test_feats)[:, 1]), 4))
print("leaky weights", dict(zip([n for n, _ in base_models()],
leaky.coef_[0].round(3).tolist())))
# leaky meta 0.7182
# leaky weights {'logit': -7.696, 'knn': -5.656, 'forest': 12.489, 'hgb': 10.595}
0.7182 against 0.7789, and look at those weights. The random forest almost memorises its training rows, so in-sample it looks nearly perfect and the meta-learner assigns it a weight of 12.5 while actively subtracting the two models that were honest. At prediction time the forest is merely good, and the whole arrangement collapses. This is data leakage in its purest form, and it is why the out-of-fold step is not optional. Use StackingClassifier and you cannot make this mistake; hand-roll it and you can.
Blending, and whether the gain survives
Blending is the cheaper variant: split the training data once, fit base models on one part, fit the meta-learner on their predictions for the other. One fit per base model instead of k + 1.
Xa, Xb, ya, yb = train_test_split(X_tr, y_tr, test_size=0.3,
random_state=0, stratify=y_tr)
fb = [(nm, m.fit(Xa, ya)) for nm, m in base_models()]
meta = LogisticRegression(max_iter=1000).fit(
np.column_stack([m.predict_proba(Xb)[:, 1] for _, m in fb]), yb)
blend_feats = np.column_stack([m.predict_proba(X_te)[:, 1] for _, m in fb])
print("blend holdout", round(roc_auc_score(y_te, meta.predict_proba(blend_feats)[:, 1]), 4))
# blend holdout 0.7804
for split in range(3):
Xa2, Xb2, ya2, yb2 = train_test_split(X, y, test_size=0.25,
random_state=split, stratify=y)
singles = {nm: roc_auc_score(yb2, m.fit(Xa2, ya2).predict_proba(Xb2)[:, 1])
for nm, m in base_models()}
st = StackingClassifier(base_models(), cv=5, passthrough=True, n_jobs=-1,
final_estimator=LogisticRegression(max_iter=1000)).fit(Xa2, ya2)
s = roc_auc_score(yb2, st.predict_proba(Xb2)[:, 1])
best = max(singles, key=singles.get)
print(f"split {split}: best single {best} {singles[best]:.4f} "
f"| stack {s:.4f} | gain {s - singles[best]:+.4f}")
# split 0: best single logit 0.7780 | stack 0.7789 | gain +0.0008
# split 1: best single logit 0.7810 | stack 0.7835 | gain +0.0024
# split 2: best single logit 0.7920 | stack 0.7941 | gain +0.0021
Blending scored 0.7804, slightly ahead of the five-fold stack. Do not read anything into that ordering; with 2,250 test rows the standard error on AUC here is larger than the difference. What the three-split sweep does establish is that the gain is consistently positive and consistently tiny. Four models, five folds each, roughly twenty-five times the training cost of logistic regression, for about two thousandths of AUC.
When I would and would not bother
Stack when the base models are genuinely different in kind and their out-of-fold correlation is below roughly 0.9, when a small metric gain has real value, and when you have the serving budget to run every base model in production. Do not stack to rescue a weak feature set. A new feature that captures something the data does not yet contain beats any ensemble of models over the same columns, every time. Picking a better single library, covered in the XGBoost, LightGBM and CatBoost comparison, is also usually cheaper than stacking three mediocre ones.
Two practical notes. Set cv with a StratifiedKFold object rather than an integer when classes are skewed, and keep the folds consistent with whatever you use elsewhere so the comparison is fair; cross-validation strategies covers the choices. And prefer passthrough=True by default, since it was the only configuration above that beat the best single model. The scikit-learn stacked generalization guide documents how multi-layer stacks are composed if you want to go further, though I would not.
Frequently Asked Questions
How much does stacking usually improve a model?
In the low single digits of a percentage point on a well-tuned baseline, and sometimes nothing. The 0.0008 to 0.0027 AUC seen here is typical for tabular data with four correlated base models. Kaggle-winning stacks get more because they combine dozens of deliberately diverse models and the margin between top entries is minuscule.
What should the meta-learner be?
Logistic regression for classification, ridge for regression. Both are low variance, and the meta-learner is fitted to few, highly correlated columns. A random forest meta-learner cost 0.0198 AUC in the run above. Keep the second layer boring.
What is the difference between stacking and blending?
Stacking builds the meta-features with k-fold cross-validation, so every training row contributes. Blending uses one holdout split instead, which is k times cheaper but wastes part of the data and gives a noisier meta-learner. Blending is a reasonable first attempt; stacking is what you ship.
Key Takeaways
- Model stacking trains a second-level model on out-of-fold predictions from the base models, which is what lets it learn weights instead of assuming equal ones.
- Train the meta-learner on in-sample predictions and it will overweight whichever model memorises hardest; that mistake cost 0.0607 AUC here and produced weights of plus 12.5 and minus 7.7.
- Check the out-of-fold correlation matrix before building anything: four models correlated 0.78 to 0.92 left almost nothing for a stack to exploit.
- Use
passthrough=Trueand a linear meta-learner, since that was the only setup that beat the single best base model, and a random forest second layer was the worst. - Expect a gain in the third decimal place for twenty-five times the training cost, and spend the effort on features first.