Calculus: Which way does the error move?

Newton and Leibniz invented calculus independently in the second half of the seventeenth century, to describe the motion of the planets and the shape of curves. Three centuries later, their central tool, the derivative, is what allows a neural network to learn. The idea is simple: if you know how the error changes when you nudge each parameter a little, you know how to move them so as to be less wrong.

The derivative: instantaneous sensitivity

The derivative of $f$ at a point $x$ measures how much $f (x)$ changes when $x$ moves by a very small amount:

f^{'} (x) = lim_{h \to 0} \frac{f (x + h) - f (x)}{h} .

Geometrically, it is the slope of the tangent line. The quotient with a finite $h$ is the slope of a secant line, which crosses the curve at two points. As $h$ shrinks, the secant turns until it coincides with the tangent.

The secant (dashed) passes through

x

and

x + h

. As

h

shrinks, its slope approaches the derivative, the slope of the tangent. Look at the point where the tangent is horizontal (

f^{'} (x) = 0

): there the function has a local minimum or maximum.

The gradient: the derivative with many variables

A loss function does not depend on one number but on millions: $ℒ (θ_{1}, \dots, θ_{n})$ . Each partial derivative $\partial ℒ / \partial θ_{i}$ measures the sensitivity to one parameter with the others held fixed, and the vector collecting them all is the gradient:

\nabla ℒ (θ) = (\frac{\partial ℒ}{\partial θ_{1}}, \dots, \frac{\partial ℒ}{\partial θ_{n}}) .

Theorem (direction of steepest ascent)

If $ℒ$ is differentiable at $θ$ and $\nabla ℒ (θ) \neq 𝟎$ , then among all unit directions $𝐮$ the directional derivative $D_{𝐮} ℒ = \nabla ℒ \cdot 𝐮$ is largest when $𝐮$ points along $\nabla ℒ$ and smallest when it points the opposite way.

Proof

By Cauchy–Schwarz, $- ‖ \nabla ℒ ‖ \leq \nabla ℒ \cdot 𝐮 \leq ‖ \nabla ℒ ‖$ for $‖ 𝐮 ‖ = 1$ , and the extremes are reached at $𝐮 = \pm \nabla ℒ / ‖ \nabla ℒ ‖$ .

Here is the recipe for learning: move in the direction of $- \nabla ℒ$ , the direction in which the error falls fastest. What we need is an efficient way to compute the gradient of a function with billions of variables.

The chain rule

A neural network is a composition of many simple functions, like an assembly line. The chain rule says how sensitivity propagates along that line. Leibniz used it in notes from 1676, and his notation makes it look like simply cancelling fractions:

Theorem (chain rule)

If $y = g (x)$ and $z = f (y)$ are differentiable, then

\frac{d z}{d x} = \frac{d z}{d y} \cdot \frac{d y}{d x}, that is, (f \circ g)^{'} (x) = f^{'} (g (x)) g^{'} (x) .

With several variables the derivatives become Jacobian matrices and the product becomes a matrix product: $J_{f \circ g} = J_{f} J_{g}$ .

For a long chain, $ℒ = f_{L} (f_{L - 1} (\dots f_{1} (x)))$ , the derivative is a product of many factors. The order in which they are multiplied does not change the result, but it changes the cost of the computation enormously.

Computational graphs and backpropagation

Any computation can be drawn as a directed acyclic graph: the nodes are elementary operations and the edges carry the intermediate results. This is where graph theory enters AI. For a neuron that predicts $\hat{y} = σ (w x + b)$ and makes an error $ℒ = (\hat{y} - y)^{2}$ , the graph is the one in the figure.

Backpropagation step by step. The forward pass computes the values (below each node). The backward pass starts from

\partial ℒ / \partial ℒ = 1

and multiplies by each node's local derivative in reverse topological order, leaving the gradient above each one. “Learn” takes one gradient-descent step on

w

and

b

: repeat it and watch the loss go down.

This algorithm, reverse-mode automatic differentiation, was described by Seppo Linnainmaa in his 1970 master's thesis. Paul Werbos proposed applying it to neural networks in his 1974 thesis. The paper by David Rumelhart, Geoffrey Hinton and Ronald Williams in Nature (1986) showed that networks trained this way learn useful internal representations, and it popularized the method under the name backpropagation.

Theorem (cheap gradient principle; Baur–Strassen, 1983)

If a function $f : ℝ^{n} \to ℝ$ is computed with $T$ elementary operations, its full gradient, all $n$ partial derivatives, can be computed in reverse mode with $c \cdot T$ operations, where $c$ is a small constant (around 3 to 5) that does not depend on $n$ .

This is the result that makes deep learning possible. Computing each derivative separately, by moving one parameter and measuring the change, would cost $n$ evaluations of the network: with $10^{11}$ parameters, impossible. Backpropagation delivers every derivative for roughly the price of two or three evaluations. Every current AI library (PyTorch, JAX, TensorFlow) is, at heart, an automatic differentiation engine.

When the gradient vanishes

The chain rule has a dark side. The sigmoid $σ (z) = 1 / (1 + e^{- z})$ , the classic activation function, has derivative $σ^{'} (z) = σ (z) (1 - σ (z)) \leq \frac{1}{4}$ . In a network with $L$ sigmoid layers, the gradient reaching the first layers contains a product of $L$ such factors:

| \frac{\partial ℒ}{\partial h_{1}} | ≲ \prod_{ℓ = 1}^{L} | σ^{'} (z_{ℓ}) | ‖ W_{ℓ} ‖ \leq {(\frac{1}{4} max_{ℓ} ‖ W_{ℓ} ‖)}^{L} .

If the weights are moderate, the gradient shrinks exponentially with depth and the first layers do not learn. Sepp Hochreiter analysed this vanishing gradient problem in his 1991 thesis. The solutions came in several waves: LSTMs (1997), the ReLU activation, $max (0, z)$ , whose derivative is 1 on the positive side, and residual connections (2015), $h_{ℓ + 1} = h_{ℓ} + F (h_{ℓ})$ , which open a highway along which the gradient travels without fading. Transformers use the last two.

We now know how to compute which way the error falls. What remains is to decide how far to go in that direction, and what guarantees there are of getting anywhere. That is optimization.

References

S. Linnainmaa (1970). The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's thesis, University of Helsinki.
P. J. Werbos (1974). Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thesis, Harvard.
W. Baur and V. Strassen (1983). “The complexity of partial derivatives”. Theoretical Computer Science, 22(3).
D. E. Rumelhart, G. E. Hinton and R. J. Williams (1986). “Learning representations by back-propagating errors”. Nature, 323.
S. Hochreiter (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TU Munich.
A. Griewank and A. Walther (2008). Evaluating Derivatives, 2nd ed. SIAM.
K. He, X. Zhang, S. Ren and J. Sun (2016). “Deep Residual Learning for Image Recognition”. CVPR.

§The derivative: instantaneous sensitivity

§The gradient: the derivative with many variables

§The chain rule

§Computational graphs and backpropagation

§When the gradient vanishes