Guess, measure, adjust

An electricity network operator needs to know how much power the city will draw tomorrow. Demand tracks temperature — hot days mean air conditioning — but the exact relationship shifts with the city, the season and the building stock.

Suppose you start with a deliberately terrible guess: demand does not depend on temperature at all, and is always zero. You can measure exactly how wrong that guess is against yesterday’s records. Then you can nudge the guess in whichever direction makes it less wrong, measure again, and repeat.

That loop — guess, measure the error, adjust — is what “training a model” means. Nearly every algorithm you will meet is a variation on it. In what machine learning actually is you saw .fit() produce a set of coefficients. This chapter opens that method up.

Parameters are the knobs

Before an algorithm can adjust anything, someone has to decide what is adjustable. Choosing LinearRegression says: the rule has the form

prediction = w × temperature + b

w and b are the parameters. w is how many megawatts each extra degree adds. b is the baseline draw at average temperature. Training means finding the pair of numbers that fits the records best. Nothing else about the shape of the rule is up for negotiation — a straight line is a straight line.

This is the division of labour that runs through all of machine learning. You choose the family of rules. The algorithm chooses the member of that family. A decision tree has a different set of knobs — which column to split on, at what threshold — but the principle is identical.

The loss function turns “wrong” into a number

You cannot adjust towards “better” until better is measurable. A loss function takes your predictions and the true answers and returns a single number, where lower is better.

The standard choice for a numeric target is mean squared error:

MSE = average of (prediction − actual)²

Read the terms in words: for every record, subtract the true value from what the model said, square that difference so overshooting and undershooting both count as error, and take the average across all records.

Squaring matters more than it looks. It means one badly missed day hurts more than several slightly missed days, so training will sacrifice small accuracy everywhere to avoid a large miss anywhere. That is usually what you want for electricity demand, where a big shortfall is a blackout. It is a poor choice when your data contains genuine outliers you would rather the model ignored, because the model will contort itself to accommodate them.

The loss function is a design decision, not a technical detail. It defines what the model is being told to care about.

Following the slope downhill

Given a loss, training becomes a search: find the parameters with the lowest loss. The workhorse method is gradient descent.

Picture the loss as a landscape, with w running east–west and b running north–south, and the height at each point being how wrong that combination is. You are standing somewhere on that surface in fog. You can feel which way the ground slopes. So you step downhill, then feel again, then step again.

The gradient is the mathematical version of feeling the slope: for each parameter, it tells you how much the loss would change if you nudged that parameter up. Move the opposite way and the loss goes down.

Here is the whole thing written out, with no library doing the work:

import numpy as np

temp   = np.array([18., 21., 24., 27., 30., 33., 36.])
demand = np.array([412., 430., 468., 521., 590., 664., 742.])

# Scale the input so both parameters move at a comparable rate
x = (temp - temp.mean()) / temp.std()

w, b = 0.0, 0.0     # the deliberately terrible starting guess
lr   = 0.1          # how big a step to take

for step in range(300):
    pred  = w * x + b
    error = pred - demand
    loss  = (error ** 2).mean()

    grad_w = 2 * (error * x).mean()   # slope of loss with respect to w
    grad_b = 2 * error.mean()         # slope of loss with respect to b

    w -= lr * grad_w                  # step downhill
    b -= lr * grad_b

    if step % 100 == 0:
        print(step, round(loss, 2), round(w, 3), round(b, 3))

# 0   312095.57  22.571   109.343
# 100 462.33     112.857  546.714
# 200 462.33     112.857  546.714

The loss falls from 312,095 to 462 and then stops moving, because the algorithm has reached the bottom of the valley and every direction is uphill.

Now confirm that scikit-learn arrives at the same place:

from sklearn.linear_model import LinearRegression

model = LinearRegression().fit(x.reshape(-1, 1), demand)
print(round(model.coef_[0], 2), round(model.intercept_, 2))
# 112.86 546.71

Identical to two decimal places. When you call .fit(), this is the kind of search that runs. The library is faster and handles awkward cases, but it is not doing anything conceptually beyond the loop above.

What the learning rate controls

lr is the step size, and it is the first hyperparameter most people meet. Set it too small and training crawls — you will run out of iterations halfway down the slope. Set it too large and each step overshoots the valley floor, bouncing to the far wall and back, with the loss oscillating or exploding to nan. There is no universally correct value; it depends on the scale of your data, which is exactly why the snippet standardises temp first.

If you take one diagnostic habit from this chapter, take this one: print the loss during training. A loss that is falling means learning. A loss that is flat from step one usually means the learning rate is far too small or the features carry no signal. A loss that becomes nan means it is far too large.

Why the starting guess usually does not matter

The loop above started at w = 0, b = 0, which is an arbitrary choice. For linear regression it makes no difference where you start, because the loss surface is a bowl: there is exactly one lowest point and every downhill path leads to it. Start anywhere, take enough steps, arrive at the same answer. That is why the manual loop and scikit-learn agreed to two decimal places.

This pleasant property is called convexity, and it belongs to a small family of models — linear regression, logistic regression, linear support vector machines. For neural networks the surface is not a bowl but a range of hills and valleys with many separate low points. Where you start decides which valley you fall into, and two training runs from different random starting values will produce genuinely different models.

The practical consequence shows up as reproducibility. When you can retrain a linear model and get identical coefficients, that is convexity, not luck. When a neural network gives you slightly different results each run, that is the surface, and the fix is to set a random seed rather than to hunt for a bug.

Where this recipe does not apply

Gradient descent is the dominant story, not the only one, and assuming it is universal will confuse you later.

Algorithm family How it finds its parameters
Linear and logistic regression Gradient descent, or a closed-form equation solved directly
Neural networks Gradient descent, with gradients computed by backpropagation
Decision trees and forests Greedy search: try candidate splits, keep the one that reduces impurity most
k-nearest neighbours No parameters are fitted at all; the training data is simply stored
Naive Bayes Counts and frequencies computed in a single pass

Two consequences worth holding on to. First, small linear problems do not need iteration at all — there is an exact formula, and scikit-learn uses it. Gradient descent wins when the dataset is too large for that formula or the model is too complex for one to exist. Second, tree-based models never compute a gradient, so concepts like learning rate and feature scaling do not transfer to them.

The underlying frame still holds everywhere: define what a good rule looks like, then search for the best one you can find. Only the search changes.

Key Takeaways

  • Name the parameters of your model before training, because training is nothing more than a search for good values of those specific numbers.
  • Choose the loss function deliberately — it is the definition of “wrong” that the model will optimise against, and squared error will chase outliers hard.
  • Print the loss on each pass while training, since a falling loss confirms learning and a flat or nan loss diagnoses your learning rate immediately.
  • Scale your inputs before running any gradient-based algorithm, or parameters on different scales will force you into a learning rate that suits neither.
  • Remember that trees, k-nearest neighbours and Naive Bayes fit without gradients, so do not carry learning rates and scaling assumptions across to them.