Backpropagation

The algorithm that computes $\partial L/\partial W$ and $\partial L/\partial b$ for every layer of a network: one forward pass storing intermediate values, then one backward pass applying the chain rule from the loss down to the inputs. Cost: about twice the forward pass, whatever the number of parameters.

Gradient descent needs the gradient with respect to every weight. Computing each partial derivative separately (or by finite differences) costs one full network evaluation per weight: impossible with $10^9$ weights. Backprop shares the work: the derivative of the loss with respect to a layer's output is computed once and reused for all the weights feeding into it.

Blame assignment. The loss says "the output was too high by this much". Each layer passes the blame back to its inputs in proportion to how much they influenced it (the local derivatives), and each weight receives (blame arriving at its neuron) × (the input it multiplied). That product is $\partial L/\partial w$.

Where does it come from? Follow the prerequisites: backpropagation ← chain rule ← derivatives; ← partial derivatives ← gradient ← functions of several variables; ← gradient descent ← optimization; ← neural networks.

For $\hat y = f(z)$, $z = Wx + b$ and loss $L(\hat y, y)$, with $\delta = \frac{\partial L}{\partial z} = \frac{\partial L}{\partial\hat y}\odot f'(z)$:

For a deep network the last formula hands $\delta$ to the previous layer: $\delta^{(\ell)} = \big(W^{(\ell+1)\mathsf T}\delta^{(\ell+1)}\big)\odot\sigma'(z^{(\ell)})$.

forward:  a0 = x;  for ℓ = 1..L:  zℓ = Wℓ aℓ₋₁ + bℓ;  aℓ = σ(zℓ)     # keep every zℓ, aℓ
loss:     L = ℓ(aL, y);  δ = ∂ℓ/∂aL ⊙ σ'(zL)
backward: for ℓ = L..1:
              ∂L/∂Wℓ = δ aℓ₋₁ᵀ ;  ∂L/∂bℓ = δ
              δ = (Wℓᵀ δ) ⊙ σ'(zℓ₋₁)
update:   Wℓ ← Wℓ − η ∂L/∂Wℓ ;  bℓ ← bℓ − η ∂L/∂bℓ

One neuron, $\hat y = \sigma(wx + b)$, squared loss $L = \frac12(\hat y - y)^2$, with $x = 1.5$, $y = 1$, $w = 0.8$, $b = -0.2$. Forward: $z = 1.0$, $\hat y = \sigma(1) \approx 0.731$, $L \approx 0.036$. Backward: $\partial L/\partial\hat y = -0.269$, $\sigma'(z) = 0.197$, $\delta \approx -0.053$, so $\partial L/\partial w = \delta x \approx -0.080$, $\partial L/\partial b \approx -0.053$. Both negative: increase $w$ and $b$. Reproduce it step by step in the demo.

Popularized by Rumelhart, Hinton and Williams (1986), with roots in Linnainmaa's reverse-mode AD (1970) and control theory. It made multilayer networks trainable and, scaled up on GPUs, made deep learning possible. Every loss.backward() call runs it.