The splitter is a modelling decision

Cross validation rotates the held-out slice: split the rows into k folds, train on k minus one of them, score on the one left out, repeat until every row has been scored exactly once, then average. You get an estimate built from all your data instead of one arbitrary slice, plus a spread that tells you how much to trust it.

The part that gets skipped is that the splitter matters as much as k. Plain KFold assumes rows are independent and exchangeable. When they are not, which is most business data, it reports a score your model will never achieve in production. Choosing the right splitter is not housekeeping; on the dataset below it is worth 0.150 of accuracy, which is larger than any model change in this chapter.

The five splitters worth knowing

Splitter Use when What it guarantees
KFold Rows are independent, target is balanced Every row scored once
StratifiedKFold Classification, especially rare classes Each fold has the same class mix
GroupKFold Repeated records per customer, shop or device No group spans train and test
TimeSeriesSplit Anything ordered in time Training always precedes scoring
RepeatedStratifiedKFold Small data, close comparisons A lower-variance mean score

If this is new ground, the single held-out slice it replaces is described in testing and validation, and the statistical treatment of fold design sits in cross-validation strategies in predictive modelling.

A tutoring company with 1,800 sessions and 120 tutors

An online tutoring firm wants to predict whether a session will be rated five stars, so it can intervene before a pupil churns. It has 1,800 sessions across 120 tutors, 15 each. The dominant driver of a rating is tutor quality, which nobody has ever measured. What the table does hold is two tutor-level columns, years of experience and hourly rate, which together nearly identify each tutor.

import numpy as np
import pandas as pd

rng = np.random.default_rng(74)
n_tutors, per_tutor = 120, 15
n = n_tutors * per_tutor                   # 1,800 tutoring sessions

skill = rng.normal(0, 1.5, n_tutors)       # never recorded anywhere
tutor_years = rng.uniform(0.5, 18, n_tutors).round(2)
tutor_rate = (24 + 1.1 * tutor_years + rng.normal(0, 4, n_tutors)).round(2)

tutor_id = np.repeat(np.arange(n_tutors), per_tutor)
session_mins = rng.choice([30, 45, 60, 90], n)
booked_ahead = rng.integers(0, 21, n)
repeat_pupil = rng.binomial(1, 0.45, n)

logit = (-0.9 + 1.35 * skill[tutor_id] + 0.011 * session_mins
         + 0.040 * booked_ahead + 0.55 * repeat_pupil)
five_star = (rng.uniform(0, 1, n) < 1 / (1 + np.exp(-logit))).astype(int)

sessions = pd.DataFrame({"tutor_id": tutor_id,
                         "tutor_years": tutor_years[tutor_id],
                         "tutor_rate": tutor_rate[tutor_id],
                         "session_mins": session_mins,
                         "booked_ahead": booked_ahead,
                         "repeat_pupil": repeat_pupil,
                         "five_star": five_star})

X = sessions.drop(columns=["tutor_id", "five_star"])
y = sessions["five_star"]
groups = sessions["tutor_id"]
print(X.shape, "| five-star rate", round(y.mean(), 3), "| tutors", groups.nunique())
# (1800, 5) | five-star rate 0.596 | tutors 120

tutor_id is dropped from the features, as any careful analyst would do. The leak survives anyway.

Choosing a cross validation splitter that matches the data

Score a random forest and a logistic regression two ways: with folds that mix sessions freely, and with folds that hold out whole tutors.

from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import (cross_val_score, KFold, StratifiedKFold,
                                     GroupKFold, RepeatedStratifiedKFold)

forest = RandomForestClassifier(n_estimators=300, min_samples_leaf=2, random_state=0)
logreg = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))

for name, model in [("forest", forest), ("logreg", logreg)]:
    naive = cross_val_score(model, X, y,
                            cv=StratifiedKFold(5, shuffle=True, random_state=0))
    honest = cross_val_score(model, X, y, groups=groups,
                             cv=GroupKFold(5, shuffle=True, random_state=0))
    print(f"{name}: rows mixed {naive.mean():.3f}  tutors held out {honest.mean():.3f}"
          f"  optimism {naive.mean() - honest.mean():+.3f}")
# forest: rows mixed 0.716  tutors held out 0.566  optimism +0.150
# logreg: rows mixed 0.598  tutors held out 0.575  optimism +0.023

Two things in those four numbers deserve a slow read.

The forest looks excellent under mixed folds at 0.716 and collapses to 0.566 when tutors are held out, a 15-point drop. It found that the pair tutor_years and tutor_rate picks out individual tutors, memorised each one’s typical rating from their other 12 sessions sitting in the training folds, and recited it. For a brand new tutor, which is the only case the business cares about, it knows nothing.

The second thing is worse. Under mixed folds the forest beats logistic regression by 12 points. Under grouped folds the forest at 0.566 is behind logistic regression at 0.575. The wrong splitter did not just inflate a score; it selected the wrong model. This is a close cousin of the problem in data leakage and how to avoid it, arriving through fold design rather than through a feature.

The rule I apply: if any entity contributes more than one row, pass groups and use GroupKFold. Pupils, patients, shops, vehicles, sensors, documents by author. The GroupKFold reference page lists the related splitters, including GroupKFold in the scikit-learn API.

Stratification, and what it actually buys you

Thin the positive class to 11 per cent and compare KFold with StratifiedKFold.

pos = np.flatnonzero(y.to_numpy() == 1)
neg = np.flatnonzero(y.to_numpy() == 0)
keep = np.sort(np.concatenate([rng.choice(pos, 90, replace=False), neg]))
X_rare, y_rare = X.iloc[keep], y.iloc[keep]
print("rare-event subset:", len(X_rare), "rows, positive rate", round(y_rare.mean(), 3))

for name, splitter in [("KFold", KFold(5, shuffle=True, random_state=0)),
                       ("StratifiedKFold", StratifiedKFold(5, shuffle=True, random_state=0))]:
    rates = [round(float(y_rare.iloc[te].mean()), 3) for _, te in splitter.split(X_rare, y_rare)]
    s = cross_val_score(forest, X_rare, y_rare, cv=splitter, scoring="roc_auc")
    print(f"{name:16s} {rates}  auc {s.mean():.3f}  sd {s.std():.3f}")
# rare-event subset: 817 rows, positive rate 0.11
# KFold            [0.14, 0.104, 0.074, 0.123, 0.11]  auc 0.681  sd 0.050
# StratifiedKFold  [0.11, 0.11, 0.11, 0.11, 0.11]  auc 0.631  sd 0.035

Unstratified folds carry between 7.4 and 14.0 per cent positives, nearly a two-fold range. Stratified folds carry exactly 11.0 per cent each.

The honest result is that stratification made the headline number worse, 0.631 against 0.681. Stratification does not buy you a better model or a better score. It buys a comparable one: the standard deviation falls from 0.050 to 0.035, and every fold now measures the same quantity. The 0.681 is the flattering accident you get when a couple of easy folds happen to be positive-rich. For classification I use StratifiedKFold by default and would not bother justifying it, and for severe imbalance the techniques in handling imbalanced classes come on top of it, not instead.

How many folds, and how many repeats

for name, splitter in [
        ("5-fold", StratifiedKFold(5, shuffle=True, random_state=0)),
        ("10-fold", StratifiedKFold(10, shuffle=True, random_state=0)),
        ("5-fold, 10 repeats", RepeatedStratifiedKFold(n_splits=5, n_repeats=10,
                                                       random_state=0))]:
    s = cross_val_score(forest, X_rare, y_rare, cv=splitter, scoring="roc_auc")
    print(f"{name:19s} fits {len(s):3d}  mean auc {s.mean():.3f}  sd {s.std():.3f}")
# 5-fold              fits   5  mean auc 0.631  sd 0.035
# 10-fold             fits  10  mean auc 0.656  sd 0.096
# 5-fold, 10 repeats  fits  50  mean auc 0.664  sd 0.065

Ten folds gave a higher spread than five, 0.096 against 0.035, which surprises people who expect more folds to mean more precision. The arithmetic is simple once stated: ten folds means each test fold holds only 82 rows and about 9 positives, so each individual fold score is wildly noisy. The reported standard deviation is the scatter of fold scores, not the standard error of their mean.

More folds give each model more training data, which raises the mean, and noisier individual estimates. Repeats are the honest way to tighten the mean: 50 fits across 10 different fold assignments, and the spread drops to 0.065 while averaging over the luck of any one partition. The cost is ten times the compute, which on 817 rows is seconds and on 8 million rows is your afternoon.

My defaults: five stratified folds while iterating, repeated five-fold when deciding between two close candidates, and grouped folds whenever an entity repeats. Nested validation for hyperparameter search belongs with the bias variance arithmetic and the tuning chapters later in the course.

Frequently Asked Questions

How many folds should I use?

Five is the right default. Ten gives each model slightly more training data and a modestly higher mean score, at twice the compute and noisier individual folds. Leave-one-out validation is rarely worth it: n fits for an estimate whose fold scores are either 0 or 1.

Does cross validation replace the test set?

No. It replaces the single validation slice you choose models with. Keep a test set sealed, run folds inside the rest, then score the chosen model on the test set once. Reporting a cross-validated score as your final number is optimistic, because you selected on it.

What if I have both groups and class imbalance?

Use StratifiedGroupKFold, which honours both constraints at once. It cannot always satisfy them exactly, since whole groups move together, so check the realised class balance per fold rather than assuming it worked.

Key Takeaways

  • Pick the splitter from the structure of your data, not by habit; here the wrong choice inflated accuracy by 0.150 and selected the wrong model.
  • Pass groups and use GroupKFold whenever a customer, tutor or device contributes more than one row, even after dropping the identifier column.
  • Stratification stabilises the estimate rather than improving it, and here it lowered the reported AUC while cutting the spread from 0.050 to 0.035.
  • The standard deviation across folds is the scatter of fold scores, not the standard error of the mean, so more folds can mean a larger spread.
  • Use repeated folds when you need to separate two close candidates, and keep a sealed test set regardless.