Gradient Descent Explained

Three algorithms wearing one name

You have already met the basic loop in how machines learn from data: compute the gradient, step against it, repeat. This chapter is about the engineering decisions that loop hides. Gradient descent is not one algorithm but a family, and the differences between members determine whether your model trains in thirty seconds or not at all.

Three choices matter. How much data you use per step. How large a step you take. Whether you carry momentum from previous steps. Get the second one wrong and nothing else helps, which is why it deserves most of your attention.

How much data per step

Each variant computes the same gradient formula — the one derived in calculus for optimisation, expressed with the matrix and transpose operations from linear algebra — on a different amount of data.

Full batch uses every row for every step. The gradient is exact and each step is expensive.

Stochastic uses one row at a time. Steps are cheap and noisy, and it makes as many updates per pass through the data as you have rows.

Mini-batch uses a chunk, typically 32 to 256 rows. This is what everyone actually uses.

import numpy as np

rng = np.random.default_rng(5)
n = 400
X = rng.normal(0, 1, (n, 3))
y = X @ np.array([2.0, -1.5, 0.5]) + 4.0 + rng.normal(0, 0.5, n)
X = np.hstack([np.ones((n, 1)), X])          # intercept column

def mse(X, y, w):
    err = X @ w - y
    return float((err ** 2).mean())

def descend(X, y, lr=0.05, epochs=30, batch=None, seed=0):
    r = np.random.default_rng(seed)
    w = np.zeros(X.shape[1])
    m = len(y)
    history = []
    for epoch in range(epochs):
        if batch is None:                     # full batch: one step per epoch
            w -= lr * (2 / m * X.T @ (X @ w - y))
        else:
            for start in range(0, m, batch):  # many steps per epoch
                idx = r.permutation(m)[start:start + batch]
                w -= lr * (2 / len(idx) * X[idx].T @ (X[idx] @ w - y[idx]))
        history.append(mse(X, y, w))
    return w, history

for name, size in [("full batch", None), ("mini-batch 32", 32), ("stochastic", 1)]:
    w, hist = descend(X, y, batch=size)
    print(name, "| after 1 epoch", round(hist[0], 4),
          "| after 30", round(hist[-1], 4))

# full batch    | after 1 epoch 17.9686 | after 30 0.2957
# mini-batch 32 | after 1 epoch 1.8375  | after 30 0.2431
# stochastic    | after 1 epoch 0.2959  | after 30 0.3111

Read the first-epoch column. Full batch has taken one step and is still at 17.97. Stochastic has taken four hundred and is already at 0.296 — it reached in one pass what full batch needed thirty passes to achieve.

Then read the last column. Stochastic gets slightly worse by epoch 30, because single-row gradients are noisy and it keeps jittering around the minimum rather than settling. Mini-batch ends best: enough rows per gradient to be stable, enough steps per epoch to move fast.

The learning rate is the parameter that decides everything

Step size controls whether you converge, crawl, or explode. The behaviour is not gradual — there is a cliff.

for lr in [0.001, 0.05, 0.6, 1.2]:
    w, hist = descend(X, y, lr=lr, epochs=30)
    print(lr, round(hist[-1], 4))

# 0.001    19.5873          <- far too small, barely moved
# 0.05      0.2957
# 0.6       0.2430          <- faster than 0.05
# 1.2       1064267587956.9 <- diverged

Worth sitting with. At 0.001 the model has barely left its starting point after thirty epochs. At 0.6 it converges faster than at 0.05, so larger is genuinely better right up until 1.2, where each step overshoots the valley and the loss grows without bound. There is no gentle warning between 0.6 and 1.2 — the transition from best-in-class to numerically dead spans one factor of two.

Practical approach: start at 0.01, watch the loss curve, and multiply or divide by three. A loss that falls smoothly and flattens means you are close. A loss falling in a straight line at epoch thirty means increase it. A loss that oscillates or reaches nan means decrease it, immediately.

Learning rate schedules formalise this. Start large to cover ground, shrink over time to settle precisely. That combination beats any fixed value, and it is why SGDRegressor exposes learning_rate="invscaling" rather than only a constant.

Momentum, and the thing it papers over

Momentum accumulates a running average of past gradients and steps along that instead of the raw gradient. In a long narrow valley, where plain descent zig-zags across the walls, the side-to-side components cancel and the along-the-valley component builds up.

Here is an ill-conditioned problem — one feature scaled forty times larger than the others, which is exactly what unscaled real data looks like:

rng = np.random.default_rng(5)
A = rng.normal(0, 1, (400, 2))
A[:, 1] *= 40                                   # wildly different scales
y2 = A @ np.array([3.0, 0.05]) + 4.0 + rng.normal(0, 0.5, 400)
X2 = np.hstack([np.ones((400, 1)), A])

def run(lr, epochs, beta=None):
    w = np.zeros(3); v = np.zeros(3); m = len(y2)
    for _ in range(epochs):
        g = 2 / m * X2.T @ (X2 @ w - y2)
        if beta is None:
            w -= lr * g
        else:
            v = beta * v + g                    # accumulate direction
            w -= lr * v
    err = X2 @ w - y2
    return float((err ** 2).mean())

print(round(run(3e-4, 200), 4))          # 19.7882  plain
print(round(run(3e-4, 200, 0.9), 4))     # 2.5536   with momentum

Momentum is roughly eight times better after the same 200 epochs. Now the part that rarely gets said: scale the two features to comparable ranges first and plain gradient descent reaches 0.2312 in thirty epochs, beating momentum’s 2.55 after two hundred.

So momentum genuinely helps, and on this problem it is compensating for something you should have fixed in preprocessing. Scale your features first. Then add momentum for the ill-conditioning that remains, which in deep networks is unavoidable — that is where Adam and RMSProp, both momentum variants with per-parameter step sizes, become the sensible default.

Where gradient descent stops working

Gradient descent has failure modes worth recognising by their signature.

Plateaus. Regions where the gradient is nearly zero but you are nowhere near a minimum. The loss flattens for many epochs, then resumes falling. Patience or momentum gets you across.

Saddle points. Uphill in one direction, downhill in another. In high dimensions these are far more common than local minima, and momentum escapes most of them.

Local minima. A real problem for neural networks, which is why two runs from different random seeds give different models — a reproducibility issue that belongs on the list of main challenges in machine learning. Not a problem for linear or logistic regression, whose bowl-shaped surfaces have a single bottom.

Vanishing and exploding gradients. Deep networks multiply many chain-rule factors; the product tends to zero or infinity. Gradient clipping and careful initialisation are the usual answers.

In scikit-learn, gradient descent is what you get from SGDRegressor and SGDClassifier. Everything above translates to their parameters: eta0 is the initial learning rate, learning_rate selects the schedule, and tol sets the stopping threshold.

from sklearn.linear_model import SGDRegressor
from sklearn.preprocessing import StandardScaler

Xs = StandardScaler().fit_transform(X[:, 1:])   # scale first, always
sgd = SGDRegressor(max_iter=2000, tol=1e-4, eta0=0.01, random_state=0)
sgd.fit(Xs, y)
print(sgd.n_iter_, sgd.coef_.round(3), round(float(sgd.intercept_[0]), 3))
# 15 [ 1.97  -1.52   0.473] 4.006

Fifteen passes recovered the true coefficients of 2.0, -1.5 and 0.5. Full options are in the SGDRegressor documentation.

Frequently Asked Questions

What is the difference between gradient descent and stochastic gradient descent?

Plain gradient descent computes the gradient over the entire dataset before each step, giving one exact update per pass. Stochastic gradient descent updates after every single row — far more steps per pass, each based on a noisy estimate. In practice mini-batch sits between them and is what production code uses.

How do you choose a learning rate?

Start around 0.01 and watch the loss curve. Still falling in a straight line at the last epoch means increase it. Oscillating or producing nan means decrease it. Adjust by factors of three rather than small increments, and prefer a decaying schedule over any fixed value.

Why is my loss becoming nan during training?

Almost always a learning rate large enough that each step overshoots and amplifies the error, which compounds to infinity within a few iterations. Unscaled features make this far likelier. Reduce the learning rate by a factor of ten and standardise your inputs before retrying.

Key Takeaways

• Default to mini-batches of 32 to 256 rows, because they combine stable gradients with many updates per pass and beat both extremes in practice.
• Diagnose the learning rate from the shape of the loss curve rather than guessing, since the gap between the fastest working value and divergence can be a single factor of two.
• Standardise your features before adding momentum or switching optimiser, as poor scaling is usually the real cause of slow convergence.
• Reach for Adam or RMSProp on deep networks where ill-conditioning cannot be preprocessed away, and stick with simpler methods on convex problems.
• Read a nan loss as a learning rate that is too large and treat it as an immediate signal to reduce it tenfold rather than to change models.