What a train test validation split is actually for

A train test validation split cuts your rows into three pieces because you have three different questions and only one supply of data. The training rows answer “what are the coefficients”. The validation rows answer “which of these six candidates should I ship”. The test rows answer “how well will the thing I shipped actually do”, and they can only answer it once.

That last point is the whole reason the third piece exists. The moment you use a set of rows to choose something, that set stops being an unbiased estimate of future performance. Choose a model on your test set and the number it reports is no longer a forecast; it is the best of several draws from a noisy process, which is systematically flattering. Keeping a third slice sealed is the cheapest insurance in applied machine learning.

Three roles, three slices

Slice Typical share Used for Touched how often
Train 50 to 70 per cent Fitting parameters Every experiment
Validation 15 to 25 per cent Comparing models, tuning Dozens of times
Test 15 to 25 per cent Final honest estimate Once, at the end

Scaling, imputation and encoding belong inside the training slice too. Fit a StandardScaler on all the rows before splitting and you have quietly leaked test information into training, which is covered properly in data leakage and how to avoid it. The habit worth building is to wrap every preprocessing step in a Pipeline so that fit never sees a row it should not.

Building the split on eight years of bike-hire data

A council-run bike-hire scheme has 2,920 daily records: temperature, rainfall, wind speed, whether it was a weekend, and the number of hires. The scheme wants a model to forecast tomorrow’s demand so it can plan redistribution vans.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(71)
n = 2920                                   # eight years of daily records

day = np.arange(n)
season = 10 * np.sin(2 * np.pi * day / 365.25 - 1.3)
temp_c = 10.5 + season + rng.normal(0, 2.4, n)
rain_mm = rng.gamma(0.7, 3.1, n).round(1)
wind_kph = rng.normal(17, 5.5, n).clip(0).round(1)
is_weekend = ((day % 7) >= 5).astype(int)

hires = (420 + 31 * temp_c - 24 * rain_mm - 3.1 * wind_kph
         + 150 * is_weekend + rng.normal(0, 95, n))

bikes = pd.DataFrame({"temp_c": temp_c.round(1), "rain_mm": rain_mm,
                      "wind_kph": wind_kph, "is_weekend": is_weekend,
                      "hires": hires.round().astype(int)})

X = bikes.drop(columns="hires")
y = bikes["hires"]

X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
X_train, X_val, y_train, y_val = train_test_split(X_rest, y_rest, test_size=0.25,
                                                  random_state=0)
print(len(X_train), len(X_val), len(X_test))
# 1752 584 584

Two calls to train_test_split, not one. The first carves off the test set; the second splits what is left. Note that test_size=0.25 on the second call gives 20 per cent of the original, because it is a quarter of the 80 per cent that remains. Getting that arithmetic wrong is a common way to end up with a 15 per cent validation set when you meant 20.

Now four candidates, scored on all three slices so you can see what each slice tells you. In practice you would never look at the test column at this stage; it is printed here to make the point.

from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import Ridge
from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_absolute_error

candidates = {
    "ridge":  make_pipeline(StandardScaler(), Ridge(alpha=1.0)),
    "knn_5":  make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=5)),
    "knn_25": make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=25)),
    "forest": RandomForestRegressor(n_estimators=300, random_state=0),
}

scores = {}
for name, model in candidates.items():
    model.fit(X_train, y_train)
    scores[name] = (mean_absolute_error(y_train, model.predict(X_train)),
                    mean_absolute_error(y_val, model.predict(X_val)),
                    mean_absolute_error(y_test, model.predict(X_test)))

for name, (tr, va, te) in scores.items():
    print(f"{name:7s} train {tr:6.1f}  val {va:6.1f}  test {te:6.1f}")

best = min(scores, key=lambda k: scores[k][1])
print("chosen on validation:", best, "| its honest test MAE:", round(scores[best][2], 1))
# ridge   train   78.0  val   78.0  test   77.9
# knn_5   train   71.8  val   92.1  test   86.6
# knn_25  train   82.4  val   85.1  test   84.3
# forest  train   31.2  val   87.4  test   85.0
# chosen on validation: ridge | its honest test MAE: 77.9

The forest scores 31.2 on its training rows and 87.4 on validation. Nobody should ship that. Ridge scores 78.0 on both, and 77.9 on the sealed test set, which is as close to boring as evaluation gets and exactly what you want. The reason is that the data really was generated by a linear rule, so the extra flexibility buys nothing. Choosing on the validation column picks ridge, and the test set then agrees.

The validation set is a noisy ruler

Textbooks warn about the winner’s curse: pick the best of many candidates on validation and the winner’s validation score is optimistic. Here is that experiment with 40 nearest-neighbour models of random size.

pool = []
for i in range(40):
    k = int(rng.integers(2, 60))
    pool.append((f"knn_{k}", make_pipeline(StandardScaler(),
                                           KNeighborsRegressor(n_neighbors=k))))

val_scores, test_scores = [], []
for name, model in pool:
    model.fit(X_train, y_train)
    val_scores.append(mean_absolute_error(y_val, model.predict(X_val)))
    test_scores.append(mean_absolute_error(y_test, model.predict(X_test)))

val_scores, test_scores = np.array(val_scores), np.array(test_scores)
w = val_scores.argmin()
print("winner by validation :", pool[w][0])
print("its validation MAE   :", round(val_scores[w], 1))
print("its test MAE         :", round(test_scores[w], 1))
print("best test MAE in pool:", round(test_scores.min(), 1))
# winner by validation : knn_18
# its validation MAE   : 84.7
# its test MAE         : 82.9
# best test MAE in pool: 82.3

The honest result here goes against the lecture. The winner’s test error, 82.9, is slightly better than its validation error of 84.7, not worse. With 584 validation rows and only 40 fairly similar candidates, the curse is swamped by ordinary sampling noise. It bites when the validation set is small and the candidate pool is large, which is precisely the situation a long hyperparameter search creates.

So how small is too small? Refit one model on different random splits and watch the spread of the score.

for v_size in (60, 200, 584):
    diffs = []
    for seed in range(20):
        Xa, Xb, ya, yb = train_test_split(X_rest, y_rest, test_size=v_size,
                                          random_state=seed)
        m = make_pipeline(StandardScaler(), Ridge(alpha=1.0)).fit(Xa, ya)
        diffs.append(mean_absolute_error(yb, m.predict(Xb)))
    print(f"validation rows {v_size:4d}  mean MAE {np.mean(diffs):6.1f}"
          f"  sd across splits {np.std(diffs):5.2f}")
# validation rows   60  mean MAE   76.9  sd across splits  7.94
# validation rows  200  mean MAE   78.8  sd across splits  4.88
# validation rows  584  mean MAE   78.6  sd across splits  2.01

Sixty validation rows give a standard deviation of 7.94 on a score of about 77. A one-point improvement in that regime means nothing at all. At 584 rows the spread drops to 2.01. If your validation set is in the low hundreds, do not trust small differences between models, and read what the regression metrics actually measure before deciding a gap matters.

How big to make each slice

With a few thousand rows, 60/20/20 is a sensible default and what I use unless something forces a change. Below roughly 2,000 rows the validation slice gets too noisy to choose with, and you should fold training and validation together and use repeated resampling instead, which is what the statistical case for cross-validation sets out. Above a few hundred thousand rows you can shrink the held-out slices to a few per cent, because 10,000 rows measure a score precisely enough for any decision you are likely to make.

Two situations break the random split outright. Time-ordered data must be split by date, or the model trains on next month and predicts last month. Repeated measurements on the same person, shop or machine must be split by that entity. Both have their own splitters, covered in the cross-validation chapter.

Scaling always goes inside the pipeline, as in feature scaling and normalisation, and the train_test_split parameters are documented in the scikit-learn model selection guide. If you are still getting comfortable with fit and predict, the mechanics are in your first model in scikit-learn.

Frequently Asked Questions

Do I need a validation set if I use cross-validation?

Cross-validation replaces the single validation slice, not the test slice. You still hold back a test set, run cross-validation inside the remaining rows to choose a model, then score the chosen model once on the test set. Skipping that final step is the most common evaluation mistake in portfolio projects.

What ratio should I use for train, validation and test?

Start at 60/20/20 for a few thousand rows. Shift towards 80/10/10 as the dataset grows past a hundred thousand rows, since a smaller percentage still gives plenty of held-out rows. Below about 2,000 rows, drop the separate validation slice and use repeated cross-validation instead.

Can I look at the test set more than once?

Once per project, ideally. Each extra look leaks a little information through the decisions you make afterwards. If you must re-test after a change, say so when you report the number and treat it as optimistic rather than pretending otherwise.

Key Takeaways

  • Split into three slices because fitting, choosing and estimating are three different jobs, and the slice you choose with cannot also be the slice you report.
  • Call train_test_split twice and check the arithmetic; test_size=0.25 on the second call yields 20 per cent of the original data, not 25.
  • A validation set in the low hundreds carries a standard deviation of several points, so ignore small differences between candidate models.
  • Wrap scaling and imputation in a Pipeline so that no preprocessing step ever sees a held-out row.
  • Report the test score once, and say plainly if you have looked at it more than once.