Hessian matrix

The matrix of second partial derivatives $\partial^2 f/\partial x_i\partial x_j$: the curvature of $f$ in every direction. Its eigenvalues classify critical points (minimum, maximum, saddle) and control how fast optimizers can go.

The gradient says which way is downhill but not how the slope changes — whether the valley is a gentle bowl, a narrow ravine or a saddle. That second-order information decides whether a critical point is a minimum and how big a step is safe.

Near a critical point, $f(x + h) \approx f(x) + \tfrac12 h^{\mathsf T}Hh$: a quadratic bowl whose axes are the eigenvectors of $H$ and whose steepness along each axis is the eigenvalue. All positive: a bowl (minimum). All negative: a dome. Mixed signs: a saddle. Very different eigenvalues: a narrow ravine where gradient descent bounces from wall to wall.

For $f \in C^2$, $H_f(x) = \nabla^2 f(x) = \big(\partial_i\partial_j f(x)\big)_{ij}$, symmetric by Schwarz. At a critical point $a$: $H \succ 0$ (positive definite) ⇒ strict local minimum; $H \prec 0$ ⇒ strict local maximum; $H$ indefinite ⇒ saddle point.

$f(x, y) = x^2 - y^2$: $\nabla f = 0$ at the origin, $H = \operatorname{diag}(2, -2)$, indefinite: a saddle (a Pringles chip). Gradient descent started exactly on the $x$-axis converges to it; any tiny perturbation in $y$ escapes. That is why saddle points slow training down but rarely trap it.

Newton's method uses $H^{-1}\nabla f$; quasi-Newton methods (BFGS, L-BFGS) approximate it; Adam and other adaptive optimizers can be read as cheap diagonal approximations of curvature. In deep learning, the "sharpness" (largest Hessian eigenvalue) is linked to generalization and to the edge-of-stability phenomenon. The full Hessian of a billion-parameter model is never formed, but Hessian–vector products are cheap with autodiff.