What a model score actually claims

A classifier tells you a transaction has a 0.83 chance of being fraud. Probability for machine learning is the vocabulary that makes that number mean something rather than being a score between zero and one that happens to sound confident.

Three pieces cover most of what you need: distributions describe the shape of a random quantity, expectation and variance summarise that shape in two numbers, and Bayes’ theorem tells you how to revise a belief when evidence arrives. Each one shows up directly in models you will build.

Statistics reasons from a sample back to the world. Probability runs the other direction: assume a mechanism, then work out what data it would produce. Models are built on the second and evaluated with the first, and the next chapter takes the other half.

Distributions are shapes with parameters

A distribution is a rule assigning probability across possible outcomes. Four cover most of what you meet in practice.

Bernoulli — one trial, two outcomes, one parameter p. A single click, a single default. This is exactly what a binary classifier from supervised learning models.

Binomial — the number of successes in n independent Bernoulli trials. Out of ten emails sent, how many get opened?

Poisson — counts of events in a fixed window, with one parameter for the average rate. Support tickets per hour, defects per batch.

Normal — the familiar symmetric bell, parameterised by mean and standard deviation. Measurement error, aggregate quantities, and the sampling distributions that statistical tests rely on.

from scipy import stats

# 3 opens out of 10 emails, when the true open rate is 20%
print(round(stats.binom.pmf(3, n=10, p=0.2), 4))    # 0.2013

# exactly 2 tickets in an hour, when the average is 1.4
print(round(stats.poisson.pmf(2, mu=1.4), 4))       # 0.2417

# central 95% of a normal with mean 64, sd 9
print(stats.norm(loc=64, scale=9).ppf([0.025, 0.975]).round(2))
# [46.36 81.64]

pmf gives the probability of an exact count for discrete distributions. For continuous ones you use pdf for density and cdf for the probability of falling below a value, and ppf inverts the cdf to answer “which value sits at this percentile”.

The one to be wary of is the normal. It is the default assumption in a great deal of software, and a large share of real data is not remotely normal — income, session length, insurance claims and basket value are all right-skewed with long tails. Plot a histogram before assuming symmetry. When the tail matters, a log-normal or gamma is usually the better description.

Expectation and variance

Expectation is the long-run average outcome: every value weighted by its probability, summed. Variance is the average squared distance from that expectation, and its square root, the standard deviation, is in the same units as the data.

import numpy as np

values = np.array([0, 1, 2, 3])                # no-shows per morning clinic
probs  = np.array([0.55, 0.30, 0.12, 0.03])

ev  = float((values * probs).sum())
var = float((probs * (values - ev) ** 2).sum())
print(round(ev, 4), round(var, 4), round(var ** 0.5, 4))
# 0.63 0.6531 0.8081

Expect 0.63 no-shows per clinic, with a standard deviation of 0.81. The expectation is not a possible outcome — you never observe 0.63 patients missing an appointment — and that is normal. It is a summary of many mornings, which is also the correct way to read a probability from a classifier.

Bayes’ theorem and the base rate

Bayes’ theorem revises a belief in light of evidence. Its practical importance is that it explains why accurate tests produce mostly false alarms when the thing being detected is rare.

A screening test catches 98% of cases and wrongly flags 5% of healthy people. The condition affects 0.4% of the population. A patient tests positive. What is the chance they have it?

prevalence = 0.004      # prior probability
sensitivity = 0.98      # P(positive | has condition)
false_positive = 0.05   # P(positive | healthy)

p_positive = sensitivity * prevalence + false_positive * (1 - prevalence)
posterior = sensitivity * prevalence / p_positive

print(round(p_positive, 5), round(posterior, 5))
# 0.05372 0.07297

Roughly 7%. The test is good and the answer is still overwhelmingly “probably not”, because healthy people vastly outnumber sick ones and 5% of a huge group beats 98% of a tiny one. Most people guess above 90% when asked this, which is why it is worth working through the arithmetic once.

Reading the formula in words: the updated belief equals how likely the evidence was under your hypothesis, times how likely the hypothesis was beforehand, divided by how likely the evidence was overall.

This is not an abstract puzzle. It is the reason a fraud model with excellent recall still drowns an operations team in false positives, and the reason you should never quote a classifier’s accuracy without stating the base rate, a point made in the main challenges of machine learning. Naive Bayes classifiers apply this theorem directly, treating features as independent.

Likelihood: where probability for machine learning meets model fitting

Turn the question around. Instead of asking how probable this data is given known parameters, ask which parameter value makes the observed data most probable. That quantity is the likelihood, and maximising it is how a large share of models are fitted.

data = np.array([61.2, 66.8, 59.4, 71.0, 64.3])

for mu in [60, 64, 68]:
    ll = stats.norm(loc=mu, scale=5).logpdf(data).sum()
    print(mu, round(float(ll), 4))
# 60 -16.3925
# 64 -14.3605
# 68 -15.5285

Of the three candidates, 64 makes this sample most plausible. Search over all values of mu rather than three and you have maximum likelihood estimation. Logistic regression is fitted this way, and minimising cross-entropy loss is exactly maximising likelihood with a sign flipped — the same loss-driven search that trains every model, with probability supplying the loss.

Log probabilities are used rather than raw ones for a practical reason: multiplying a thousand numbers below one underflows to zero in floating point, while adding their logarithms does not. When you see log_loss or logpdf in a library, this is why. The scipy.stats reference for the normal distribution lists every method these objects expose.

Frequently Asked Questions

Why is Bayes theorem important in machine learning?

It formalises how evidence should update a belief, and it explains base rate effects — why a test with 98% sensitivity still yields mostly false positives for a rare condition. Naive Bayes classifiers apply it directly, and it underpins how you interpret any classifier’s output on imbalanced data.

What probability distributions should I know for machine learning?

Bernoulli and binomial for binary outcomes and counts of successes, Poisson for event counts in a window, and normal for measurement error and sampling distributions. Knowing log-normal and gamma helps, because a lot of real business data is right-skewed rather than symmetric.

What is the difference between probability and likelihood?

Probability fixes the parameters and asks how likely some data is. Likelihood fixes the observed data and asks which parameter values make it most plausible. Same formula, opposite thing held constant. Maximum likelihood estimation searches for the parameters that maximise the second.

Key Takeaways

  • Plot a histogram before assuming a normal distribution, because business quantities such as revenue and session length are usually right-skewed with long tails.
  • Quote the base rate alongside any classifier result, since Bayes’ theorem shows that rare positives produce mostly false alarms even with an accurate model.
  • Read an expected value as a long-run average rather than a possible outcome, and apply the same reading to predicted probabilities.
  • Recognise that maximising likelihood and minimising cross-entropy loss are the same procedure, which connects probability directly to model fitting.
  • Work in log probabilities whenever many terms are multiplied, or floating-point underflow will silently turn your result into zero.