Automatic differentiation

What is it?

Formulas

$$(a + b\varepsilon)(c + d\varepsilon) = ac + (ad + bc)\varepsilon, \quad \varepsilon^2 = 0$$

forward mode with dual numbers

$$\bar v_i = \sum_{j:\ i \to j} \bar v_j\,\frac{\partial v_j}{\partial v_i}$$

reverse mode: adjoints propagate backwards

Why does it matter?

The mathematics behind it

• Derivative★★★★★fundamental

  AD computes exact derivatives of programs by propagating them through each operation.

• Differentiation rules★★★★★fundamental

  AD applies exactly these rules, one elementary operation at a time, to numbers instead of formulas.

• Chain rule★★★★★fundamental

  Forward and reverse mode are two orders of multiplying the same chain of local derivatives.

• Multivariable chain rule★★★★★fundamental

  Forward and reverse mode are the two natural orders of multiplying the Jacobian chain.

• Composition★★★★★fundamental

  AD sees a program as a composition of primitive operations and differentiates each one.

• Derivatives of elementary functions★★★★★fundamental

  Every AD system ships a table of derivatives of its primitive operations.

• Directional derivative★★★★★frequent

  Forward-mode AD with dual numbers computes $D_u f$ (a JVP) in one pass.

• Jacobian matrix★★★★★fundamental

  AD computes JVPs (forward mode) and VJPs (reverse mode) without materializing $J$.

• Numerical differentiation★★★★★frequent

  "Gradient checking" compares autodiff gradients against finite differences to catch bugs.

Where is it used?

Computing topics reachable from here, through the chain of ideas that leads to them:

What depends on it

• Backpropagation