Second-order (Hessian-based) optimization

Level SpecializationDifficulty ★★★★★Application⌖ Open in the map

What is it?

Use curvature to choose the step: $\theta \leftarrow \theta - H^{-1}\nabla L$ . Quadratic convergence near a minimum and immune to ill-conditioning, but the Hessian of a large model cannot even be stored — so practice uses approximations (L-BFGS, Gauss–Newton, K-FAC, Shampoo).

Formulas

\theta_{k+1} = \theta_k - \big(\nabla^2 L(\theta_k)\big)^{-1}\nabla L(\theta_k)

The mathematics behind it

Newton's method★★★★★fundamental
Newton's method on $\nabla f = 0$ uses the Hessian: $x \leftarrow x - H^{-1}\nabla f$ .
Taylor polynomial★★★★★fundamental
Newton and trust-region methods minimize the quadratic Taylor model $f + g^{\mathsf T}h + \frac12 h^{\mathsf T}Hh$ .
Hessian matrix★★★★★fundamental
Newton, Gauss–Newton, natural gradient and K-FAC use the Hessian or an approximation of it.
Higher-order derivatives★★★★★fundamental
Second derivatives measure curvature; Newton-type methods use them to choose the step.

This page has the essentials. A fuller treatment (intuition, formal definition, worked example) is on the way.