Lagrange multipliers

At a constrained optimum of $f$ subject to $g = 0$, the gradients are parallel: $\nabla f = \lambda\nabla g$. The multiplier $\lambda$ measures how much the optimum would improve if the constraint were relaxed.

Substituting the constraint to eliminate a variable works only in easy cases. Lagrange's method keeps all the variables and adds one unknown per constraint, turning a constrained problem into an unconstrained system of equations — and the new unknowns turn out to carry economic and physical meaning.

Walk along the curve $g = 0$ looking at the level sets of $f$. While the curve crosses level sets, you can still go lower. At the optimum the curve is tangent to a level set of $f$, so their normals — $\nabla f$ and $\nabla g$ — point along the same line.

If $x^\ast$ is a local extremum of $f$ on $\{g = 0\}$, with $f, g$ continuously differentiable and $\nabla g(x^\ast) \ne 0$, then there is $\lambda$ with $\nabla f(x^\ast) = \lambda\,\nabla g(x^\ast)$. Equivalently, $(x^\ast, \lambda)$ is a critical point of the Lagrangian

Maximize entropy $H(p) = -\sum_i p_i\ln p_i$ subject to $\sum_i p_i = 1$. $\partial_{p_i}\big(H - \lambda(\sum p - 1)\big) = -\ln p_i - 1 - \lambda = 0$, so all $p_i$ are equal: the uniform distribution. Add a constraint on the expected energy and the same computation produces the softmax / Boltzmann distribution $p_i \propto e^{-\beta E_i}$.

Support vector machines are derived through their Lagrangian dual (where the kernel trick appears). Constrained training, fairness constraints, the maximum-entropy justification of softmax, and duality in operations research all use the same idea. In physics, Lagrange's other great idea — the Lagrangian of mechanics — runs robot dynamics.