What a train test validation split is actually for
A train test validation split cuts your rows into three pieces because you have three different questions and only one supply of data. The training rows answer “what are the coefficients”. The validation rows answer “which of these six candidates should I ship”. The test rows answer “how well will the thing I shipped actually do”, and they can only answer it once.
That last point is the whole reason the third piece exists. The moment you use a set of rows to choose something, that set stops being an unbiased estimate of future performance. Choose a model on your test set and the number it reports is no longer a forecast; it is the best of several draws from a noisy process, which is systematically flattering. Keeping a third slice sealed is the cheapest insurance in applied machine learning.
Three roles, three slices
| Slice | Typical share | Used for | Touched how often |
|---|---|---|---|
| Train | 50 to 70 per cent | Fitting parameters | Every experiment |
| Validation | 15 to 25 per cent | Comparing models, tuning | Dozens of times |
| Test | 15 to 25 per cent | Final honest estimate | Once, at the end |
Scaling, imputation and encoding belong inside the training slice too. Fit a StandardScaler on all the rows before splitting and you have quietly leaked test information into training, which is covered properly in data leakage and how to avoid it. The habit worth building is to wrap every preprocessing step in a Pipeline so that fit never sees a row it should not.
Building the split on eight years of bike-hire data
A council-run bike-hire scheme has 2,920 daily records: temperature, rainfall, wind speed, whether it was a weekend, and the number of hires. The scheme wants a model to forecast tomorrow’s demand so it can plan redistribution vans.
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(71)
n = 2920 # eight years of daily records
day = np.arange(n)
season = 10 * np.sin(2 * np.pi * day / 365.25 - 1.3)
temp_c = 10.5 + season + rng.normal(0, 2.4, n)
rain_mm = rng.gamma(0.7, 3.1, n).round(1)
wind_kph = rng.normal(17, 5.5, n).clip(0).round(1)
is_weekend = ((day % 7) >= 5).astype(int)
hires = (420 + 31 * temp_c - 24 * rain_mm - 3.1 * wind_kph
+ 150 * is_weekend + rng.normal(0, 95, n))
bikes = pd.DataFrame({"temp_c": temp_c.round(1), "rain_mm": rain_mm,
"wind_kph": wind_kph, "is_weekend": is_weekend,
"hires": hires.round().astype(int)})
X = bikes.drop(columns="hires")
y = bikes["hires"]
X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
X_train, X_val, y_train, y_val = train_test_split(X_rest, y_rest, test_size=0.25,
random_state=0)
print(len(X_train), len(X_val), len(X_test))
# 1752 584 584
Two calls to train_test_split, not one. The first carves off the test set; the second splits what is left. Note that test_size=0.25 on the second call gives 20 per cent of the original, because it is a quarter of the 80 per cent that remains. Getting that arithmetic wrong is a common way to end up with a 15 per cent validation set when you meant 20.
Now four candidates, scored on all three slices so you can see what each slice tells you. In practice you would never look at the test column at this stage; it is printed here to make the point.
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import Ridge
from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_absolute_error
candidates = {
"ridge": make_pipeline(StandardScaler(), Ridge(alpha=1.0)),
"knn_5": make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=5)),
"knn_25": make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=25)),
"forest": RandomForestRegressor(n_estimators=300, random_state=0),
}
scores = {}
for name, model in candidates.items():
model.fit(X_train, y_train)
scores[name] = (mean_absolute_error(y_train, model.predict(X_train)),
mean_absolute_error(y_val, model.predict(X_val)),
mean_absolute_error(y_test, model.predict(X_test)))
for name, (tr, va, te) in scores.items():
print(f"{name:7s} train {tr:6.1f} val {va:6.1f} test {te:6.1f}")
best = min(scores, key=lambda k: scores[k][1])
print("chosen on validation:", best, "| its honest test MAE:", round(scores[best][2], 1))
# ridge train 78.0 val 78.0 test 77.9
# knn_5 train 71.8 val 92.1 test 86.6
# knn_25 train 82.4 val 85.1 test 84.3
# forest train 31.2 val 87.4 test 85.0
# chosen on validation: ridge | its honest test MAE: 77.9
The forest scores 31.2 on its training rows and 87.4 on validation. Nobody should ship that. Ridge scores 78.0 on both, and 77.9 on the sealed test set, which is as close to boring as evaluation gets and exactly what you want. The reason is that the data really was generated by a linear rule, so the extra flexibility buys nothing. Choosing on the validation column picks ridge, and the test set then agrees.
The validation set is a noisy ruler
Textbooks warn about the winner’s curse: pick the best of many candidates on validation and the winner’s validation score is optimistic. Here is that experiment with 40 nearest-neighbour models of random size.
pool = []
for i in range(40):
k = int(rng.integers(2, 60))
pool.append((f"knn_{k}", make_pipeline(StandardScaler(),
KNeighborsRegressor(n_neighbors=k))))
val_scores, test_scores = [], []
for name, model in pool:
model.fit(X_train, y_train)
val_scores.append(mean_absolute_error(y_val, model.predict(X_val)))
test_scores.append(mean_absolute_error(y_test, model.predict(X_test)))
val_scores, test_scores = np.array(val_scores), np.array(test_scores)
w = val_scores.argmin()
print("winner by validation :", pool[w][0])
print("its validation MAE :", round(val_scores[w], 1))
print("its test MAE :", round(test_scores[w], 1))
print("best test MAE in pool:", round(test_scores.min(), 1))
# winner by validation : knn_18
# its validation MAE : 84.7
# its test MAE : 82.9
# best test MAE in pool: 82.3
The honest result here goes against the lecture. The winner’s test error, 82.9, is slightly better than its validation error of 84.7, not worse. With 584 validation rows and only 40 fairly similar candidates, the curse is swamped by ordinary sampling noise. It bites when the validation set is small and the candidate pool is large, which is precisely the situation a long hyperparameter search creates.
So how small is too small? Refit one model on different random splits and watch the spread of the score.
for v_size in (60, 200, 584):
diffs = []
for seed in range(20):
Xa, Xb, ya, yb = train_test_split(X_rest, y_rest, test_size=v_size,
random_state=seed)
m = make_pipeline(StandardScaler(), Ridge(alpha=1.0)).fit(Xa, ya)
diffs.append(mean_absolute_error(yb, m.predict(Xb)))
print(f"validation rows {v_size:4d} mean MAE {np.mean(diffs):6.1f}"
f" sd across splits {np.std(diffs):5.2f}")
# validation rows 60 mean MAE 76.9 sd across splits 7.94
# validation rows 200 mean MAE 78.8 sd across splits 4.88
# validation rows 584 mean MAE 78.6 sd across splits 2.01
Sixty validation rows give a standard deviation of 7.94 on a score of about 77. A one-point improvement in that regime means nothing at all. At 584 rows the spread drops to 2.01. If your validation set is in the low hundreds, do not trust small differences between models, and read what the regression metrics actually measure before deciding a gap matters.
How big to make each slice
With a few thousand rows, 60/20/20 is a sensible default and what I use unless something forces a change. Below roughly 2,000 rows the validation slice gets too noisy to choose with, and you should fold training and validation together and use repeated resampling instead, which is what the statistical case for cross-validation sets out. Above a few hundred thousand rows you can shrink the held-out slices to a few per cent, because 10,000 rows measure a score precisely enough for any decision you are likely to make.
Two situations break the random split outright. Time-ordered data must be split by date, or the model trains on next month and predicts last month. Repeated measurements on the same person, shop or machine must be split by that entity. Both have their own splitters, covered in the cross-validation chapter.
Scaling always goes inside the pipeline, as in feature scaling and normalisation, and the train_test_split parameters are documented in the scikit-learn model selection guide. If you are still getting comfortable with fit and predict, the mechanics are in your first model in scikit-learn.
Frequently Asked Questions
Do I need a validation set if I use cross-validation?
Cross-validation replaces the single validation slice, not the test slice. You still hold back a test set, run cross-validation inside the remaining rows to choose a model, then score the chosen model once on the test set. Skipping that final step is the most common evaluation mistake in portfolio projects.
What ratio should I use for train, validation and test?
Start at 60/20/20 for a few thousand rows. Shift towards 80/10/10 as the dataset grows past a hundred thousand rows, since a smaller percentage still gives plenty of held-out rows. Below about 2,000 rows, drop the separate validation slice and use repeated cross-validation instead.
Can I look at the test set more than once?
Once per project, ideally. Each extra look leaks a little information through the decisions you make afterwards. If you must re-test after a change, say so when you report the number and treat it as optimistic rather than pretending otherwise.
Key Takeaways
- Split into three slices because fitting, choosing and estimating are three different jobs, and the slice you choose with cannot also be the slice you report.
- Call
train_test_splittwice and check the arithmetic;test_size=0.25on the second call yields 20 per cent of the original data, not 25. - A validation set in the low hundreds carries a standard deviation of several points, so ignore small differences between candidate models.
- Wrap scaling and imputation in a
Pipelineso that no preprocessing step ever sees a held-out row. - Report the test score once, and say plainly if you have looked at it more than once.