The Statistics You Actually Need

The statistics for machine learning that earns its place

It is not the syllabus you would sit an exam on. It is a small set of tools for one recurring question: my model scored 0.847 on a test set — how much of that is real?

You almost never have a population. You have a sample, and every number computed from it is an estimate that would come out differently with different rows. Four tools tell you how much differently: the standard error, the confidence interval, the bootstrap, and a distribution comparison. Hypothesis tests matter less than your degree suggested, and I will say where they still apply.

The probability groundwork runs from assumed mechanism to expected data. Statistics runs backwards, from observed data to a claim about the mechanism, and that direction is always uncertain.

Sampling error is not a mistake

Take repeated samples from the same population and each gives a different mean. The spread of those means is the sampling distribution, and its standard deviation is the standard error.

import numpy as np

rng = np.random.default_rng(11)
population = rng.gamma(shape=2.0, scale=120.0, size=200_000)  # skewed spend data
print(round(population.mean(), 2))            # 239.77

means = np.array([rng.choice(population, 200, replace=False).mean()
                  for _ in range(2000)])
print(round(means.mean(), 2), round(means.std(ddof=1), 2))    # 239.45 12.48

# the formula: population sd divided by the square root of sample size
print(round(population.std(ddof=1) / np.sqrt(200), 2))        # 11.96

Two things to take from those numbers. The simulated spread of 12.48 matches the formula’s 11.96, so the standard error is predictable rather than mysterious. And it shrinks with the square root of sample size — quartering your error takes sixteen times the data, which is why the third significant figure of a metric computed on two hundred rows is decoration.

Note that the population here is heavily skewed, yet the distribution of its sample means is near-symmetric. That is the central limit theorem, and it is why normal-based intervals work on averages taken from decidedly non-normal data.

Confidence intervals and the bootstrap

A confidence interval puts a range around an estimate. The standard reading: if you repeated the whole procedure many times, 95% of the intervals produced would contain the true value. It does not say there is a 95% chance the true value sits inside this particular interval, though almost everyone treats it that way in conversation.

The formula version needs assumptions about the underlying distribution. The bootstrap needs almost none — resample your own data with replacement, many times, and look at the spread of the results.

sample = rng.choice(population, 200, replace=False)
boot = np.array([rng.choice(sample, 200, replace=True).mean()
                 for _ in range(5000)])

print(round(sample.mean(), 2))                        # 249.9
print(np.percentile(boot, [2.5, 97.5]).round(2))      # [227.2  275.28]

The point estimate is 249.9 and the interval runs from 227 to 275 — about £48 wide on a sample of two hundred. The true population mean, 239.77, sits inside it.

The bootstrap is the technique I reach for most, and it is underused. It works for medians, ratios, model scores, differences between models, and anything else where no textbook formula exists. Resample, recompute, take percentiles. Twenty lines and no distributional assumptions.

Apply it to your model metrics. Reporting “accuracy 0.847, bootstrap interval 0.81 to 0.88” is honest in a way that “accuracy 0.847” is not, and it stops arguments about whether a 0.003 improvement means anything.

Correlation, and its limits

Correlation measures how strongly two variables move together, on a scale from -1 to 1. Pearson’s version captures straight-line association; Spearman’s works on ranks and handles curved-but-monotonic relationships.

from scipy import stats

n = 300
tenure = rng.uniform(1, 60, n)                             # months as a customer
spend  = 40 + 1.8 * tenure + rng.normal(0, 25, n)

r = float(np.corrcoef(tenure, spend)[0, 1])
print(round(r, 4), round(r ** 2, 4))                       # 0.7783 0.6057
print(round(float(stats.spearmanr(tenure, spend).statistic), 4))  # 0.7932

An r of 0.78 sounds strong. Square it and you get 0.61, meaning tenure accounts for about 61% of the variation in spend — a more sober framing, and the one worth quoting.

Two failures to keep in mind. Correlation near zero does not mean unrelated; it means no linear relationship, and a clean U-shape can score almost exactly zero. And correlation says nothing about causation, which matters practically rather than philosophically: a feature correlated with your target because it is a consequence of the target will produce a superb test score and a useless model, which is the leakage trap described in the main challenges of machine learning.

Where hypothesis tests still belong

Most of applied machine learning does not need p-values. You are not testing whether an effect exists; you are measuring how well a model trained on labelled examples predicts, and held-out performance answers that better than any test.

Two places they remain genuinely useful.

Comparing two models properly. Run cross-validation, collect the per-fold scores for each model, and test whether the difference could plausibly be noise. Otherwise you ship the model that won by luck on one split.

Detecting distribution shift. Comparing this month’s input data against the training data is precisely a two-sample test, and the Kolmogorov-Smirnov test is the standard tool:

last_month = rng.normal(64, 9, 1500)
this_month = rng.normal(67, 9, 1500)

result = stats.ks_2samp(last_month, this_month)
print(round(float(result.statistic), 4), f"{result.pvalue:.2e}")
# 0.1427 1.00e-13

A shift of three units in the mean, invisible in a summary table, is flagged decisively. Automate this per feature and you have a drift monitor.

One caution on p-values: with a few hundred thousand rows, almost any difference reaches statistical significance while remaining far too small to act on. Significance is not importance. The scipy.stats.ks_2samp reference covers the test’s options and assumptions.

Frequently Asked Questions

How much statistics do you need for machine learning?

Sampling variability, standard error, confidence intervals, correlation and the bootstrap cover most practical needs. You should also know why statistical significance is not practical importance. Formal hypothesis testing matters far less than in traditional analysis, because held-out performance answers most questions directly.

What is the bootstrap and why is it useful?

Resample your own data with replacement many times, recompute the statistic each time, and read the spread. It gives you a confidence interval for medians, ratios, model scores and differences between models, with no distributional assumptions and about twenty lines of code.

Does correlation imply causation in machine learning?

No, and the practical danger is specific. A feature correlated with your target because it is caused by the target will score brilliantly in testing and fail in production, because its value is not available at prediction time. Always ask when a feature becomes known.

Key Takeaways

• Report a bootstrap interval alongside every model metric, because a bare accuracy figure hides how much of it is sampling noise.
• Stop reading past the second significant figure on any score computed from a few hundred rows, since standard error only shrinks with the square root of sample size.
• Square your correlation coefficient before describing a relationship as strong, as r = 0.78 explains only about 61% of the variation.
• Run a Kolmogorov-Smirnov test per feature against your training data on a schedule, which turns drift from an invisible failure into an alert.
• Treat statistical significance and practical importance as separate questions, because large datasets make trivial differences significant.