Calculus for Optimisation | Machine Learning | Codeayan Courses

Calculus for optimisation asks one question

Calculus for optimisation answers a single question: if I nudge this number up a little, does my error get better or worse, and by how much? That is the derivative. Everything else in this chapter is machinery for answering that question when there are a thousand numbers instead of one.

You will not integrate anything. You will not compute limits. Applied machine learning uses roughly four pieces of calculus — the derivative, partial derivatives, the gradient, and the chain rule — and the libraries compute all of them for you. Knowing what they mean is what lets you diagnose a model that refuses to train.

The dot products and matrix shapes from linear algebra tell you what a model computes. Calculus tells you how to improve it.

The derivative is a slope

Suppose your model has exactly one parameter w, and your error as a function of that parameter is (w - 3)² + 2. The derivative at any point tells you the slope of that curve there: positive means uphill to the right, negative means downhill to the right, zero means flat.

For this function the derivative is 2(w - 3). You can check that numerically by measuring the actual rise over run across a tiny interval:

def loss(w):
    return (w - 3.0) ** 2 + 2.0

def dloss(w):
    return 2 * (w - 3.0)          # the analytical derivative

h = 1e-6
for w in [0.0, 3.0, 5.0]:
    numerical = (loss(w + h) - loss(w - h)) / (2 * h)
    print(w, round(numerical, 6), dloss(w))

# 0.0  -6.0  -6.0
# 3.0   0.0   0.0
# 5.0   4.0   4.0

At w = 0 the slope is -6: the error falls steeply as w increases, so move right. At w = 5 the slope is +4, so move left. At w = 3 the slope is zero and you are at the bottom.

That trick in the middle of the snippet — measuring a derivative by nudging the input and dividing — is called a finite difference, and it is worth knowing as a debugging tool. When you have hand-derived a gradient and the model will not train, comparing your formula against a finite difference finds the algebra error in about a minute.

Partial derivatives and the gradient

Real models have many parameters. A partial derivative asks the same slope question about one parameter while holding all the others fixed. Stack all the partial derivatives into a vector and you have the gradient — a single object pointing in the direction of steepest increase.

For mean squared error the gradient has a compact matrix form. If X is your feature matrix, w your weights and y the answers, the gradient is (2/n) · Xᵀ(Xw - y). In words: compute the prediction error for every row, multiply it back through the features, and average.

import numpy as np

X = np.array([[1.0, 2.0], [2.0, 1.0], [3.0, 4.0], [4.0, 3.0]])
y = np.array([8.0, 7.0, 18.0, 17.0])
w = np.array([1.0, 1.0])              # a poor starting guess

error = X @ w - y
grad  = 2 / len(y) * (X.T @ error)

print(error.round(3))   # [ -5.  -4. -11. -10.]
print(grad.round(4))    # [-43. -44.]

Both gradient components are strongly negative, which says both weights are too small. Notice that the transpose appears for a shape reason: X is (4, 2) and error is (4,), so X.T @ error gives (2,) — one number per parameter, which is exactly what a gradient must be. If your gradient does not have the same shape as your parameter vector, you have made an error.

The chain rule composes the layers

Models are functions of functions. Logistic regression computes a dot product, then squashes it through a sigmoid. A neural network stacks that pattern many times over. The chain rule says the derivative of a composition is the product of the derivatives of its parts: to find how the final output responds to an early parameter, multiply the sensitivities along the path.

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

x, w = 2.0, 0.5
z = w * x                    # inner function
p = sigmoid(z)               # outer function

analytic = p * (1 - p) * x   # dp/dw = dp/dz × dz/dw
numeric  = (sigmoid((w + h) * x) - sigmoid((w - h) * x)) / (2 * h)

print(round(p, 6), round(analytic, 6), round(numeric, 6))
# 0.731059 0.393224 0.393224

p * (1 - p) is the sigmoid’s own derivative and x is the derivative of w * x with respect to w. Multiply them and you have the answer, confirmed by finite difference to six decimal places.

This is backpropagation. Every deep learning framework is an efficient, automatic application of the chain rule across a graph of operations, and the notorious vanishing gradient problem is what happens when you multiply many factors smaller than one: p * (1 - p) peaks at 0.25, so ten stacked sigmoids can shrink a gradient by a factor of a million.

Curvature tells you what kind of surface you are on

The second derivative measures how the slope itself is changing. Positive curvature everywhere means the surface is a bowl with exactly one bottom.

def f_convex(w): return w ** 2
def f_wiggly(w): return w ** 4 - 3 * w ** 2 + w

def second(f, w, h=1e-4):
    return (f(w + h) - 2 * f(w) + f(w - h)) / h ** 2

for w in [-1.5, 0.0, 1.5]:
    print(w, round(second(f_convex, w), 3), round(second(f_wiggly, w), 3))
# -1.5   2.0   21.0
#  0.0   2.0   -6.0
#  1.5   2.0   21.0

The bowl has curvature 2 everywhere. The second function curves upward at the edges and downward at zero, which means zero is a local maximum sitting between two separate valleys — start on the wrong side and you converge to the wrong one. Linear and logistic regression give you the first kind of surface. Neural networks give you the second, which is why their results vary between runs.

Where calculus stops helping: any model that is not differentiable. A decision tree splits on thresholds, so its loss changes in steps rather than smoothly, and there is no gradient to compute. Trees sit in a different corner of the machine learning system taxonomy and use greedy search over candidate splits instead — the same reason concepts like learning rate do not apply to them, as noted in how machines learn from data. Finite differences on real data can be checked with numpy.gradient.

Frequently Asked Questions

How much calculus do you need for machine learning?

Derivatives, partial derivatives, the gradient and the chain rule. You need to read those notations and know what they mean, not compute them by hand — libraries handle the computation. Integration, limits and formal proofs play almost no part in applied work.

What is the difference between a derivative and a gradient?

A derivative is the slope of a function with one input. A gradient is the vector of partial derivatives for a function with many inputs, one entry per parameter, pointing in the direction of steepest increase. Models have thousands of parameters, so gradients are what you actually work with.

Why does the chain rule matter in machine learning?

Models compose operations — a dot product feeding a sigmoid, layer after layer. The chain rule lets you find how an early parameter affects the final loss by multiplying sensitivities along the path. Backpropagation is exactly this, applied automatically across a computation graph.

Key Takeaways

Read a gradient as one slope per parameter, and check that its shape matches your parameter vector before debugging anything else.
Verify any hand-derived derivative against a finite difference, because the comparison locates algebra errors in seconds and costs three lines of code.
Expect the chain rule to multiply small factors when layers stack, which is the mechanism behind vanishing gradients in deep networks.
Check whether your loss surface is convex, since a bowl gives reproducible results from any starting point and a wiggly surface does not.
Skip gradient reasoning entirely for tree-based models, which split on thresholds and have no gradient to follow.