Multivariable chain rule

When a variable influences the output through several paths, add the contributions of every path, each the product of the local derivatives along it: $\frac{\partial z}{\partial x} = \sum_i \frac{\partial z}{\partial u_i}\frac{\partial u_i}{\partial x}$. In matrix form, Jacobians multiply. This is exactly what backpropagation computes on a network's graph.

Real computations branch and merge: a weight affects many neurons, which all affect the loss. The one-variable chain rule handles a single path; the multivariable version handles any directed acyclic graph of operations — any program without loops, or any unrolled one.

Draw the computation as a graph: inputs at the bottom, the output at the top, each edge labelled with a local partial derivative. $\partial(\text{output})/\partial(\text{input})$ is the sum, over all paths, of the products of labels along the path. Backpropagation computes all these sums at once by sweeping the graph from the top, accumulating at each node "how much the output cares about me" (the adjoint).

If $g : \R^n \to \R^m$ is differentiable at $x$ and $f : \R^m \to \R^p$ at $g(x)$, then

For scalar $z = f(u_1, \dots, u_m)$ with $u_i = u_i(x)$: $\frac{\partial z}{\partial x_j} = \sum_{i=1}^{m}\frac{\partial z}{\partial u_i}\frac{\partial u_i}{\partial x_j}$.

The product $J_L\cdots J_1$ can be evaluated right-to-left (forward mode: cost ∝ number of inputs) or left-to-right (reverse mode: cost ∝ number of outputs). A loss has one output and millions of inputs, so reverse mode wins by a factor of millions — that choice is backpropagation.

$z = xy + \sin x$ with $x = t^2$, $y = e^t$. Two paths from $t$ to $z$ (through $x$ and through $y$): $\frac{\dd z}{\dd t} = (y + \cos x)\cdot 2t + x\cdot e^t = (e^t + \cos t^2)\,2t + t^2e^t$.

It is the mathematical content of backpropagation and of every autodiff library (PyTorch, JAX, TensorFlow). It also gives robot velocities through the kinematic chain and the sensitivities used in engineering design (adjoint methods in CFD and weather forecasting are the same reverse-mode idea).