Chain rule

The derivative of a composition is the product of the derivatives: $(g\circ f)'(x) = g'(f(x))\,f'(x)$. Rates of change multiply along a chain — and backpropagation is this rule applied, very efficiently, to a neural network.

Almost every function we meet is built by composing simpler ones. Without the chain rule we would have to go back to limits for each new formula; with it, knowing the derivatives of a few primitives is enough to differentiate everything built from them — including a program.

Gears. If $u$ turns 3 times as fast as $x$, and $y$ turns 2 times as fast as $u$, then $y$ turns $2 \times 3 = 6$ times as fast as $x$. In Leibniz notation the rule looks like cancelling fractions: $\frac{\dd y}{\dd x} = \frac{\dd y}{\dd u}\cdot\frac{\dd u}{\dd x}$. Along a chain of $n$ functions you multiply $n$ local rates — and if they are all smaller than 1, the product vanishes (that is the vanishing-gradient problem).

If $f$ is differentiable at $x$ and $g$ is differentiable at $f(x)$, then $g \circ f$ is differentiable at $x$ and

Write $g(f(x+h)) - g(f(x)) = \big(g'(f(x)) + \eta(k)\big)\,k$ with $k = f(x+h) - f(x)$ and $\eta(k) \to 0$; divide by $h$ and let $h \to 0$. (Dividing and multiplying by $k$ directly fails when $k = 0$, which is why the auxiliary function $\eta$ is needed.)

Name the intermediate values: $u_1 = f_1(x)$, $u_2 = f_2(u_1)$, … Compute the local derivatives $f_k'(u_{k-1})$ and multiply. The order in which you multiply does not change the result, but it changes the cost when the functions have many inputs and outputs: left-to-right is forward mode, right-to-left is reverse mode (backpropagation).

$y = \sigma(wx + b)$ with $\sigma$ the sigmoid. Set $z = wx + b$. Then $\frac{\partial y}{\partial w} = \sigma'(z)\cdot x = \sigma(z)(1 - \sigma(z))\,x$. This is literally the gradient a neural network computes for one weight of one neuron.

Backpropagation (Rumelhart, Hinton and Williams, 1986; Linnainmaa's reverse-mode AD, 1970) is the chain rule organised so that the gradient with respect to all weights costs about as much as one evaluation of the network. Without that trick, training modern models would be impossible.