Critical points and derivative tests

Where $f'(x) = 0$ (or does not exist). Fermat: an interior extremum of a differentiable function is a critical point. The sign of $f''$ then tells minimum ($f'' > 0$), maximum ($f'' < 0$) or "look closer" ($f'' = 0$).

Checking every point to find the lowest is impossible on a continuum. Fermat's observation reduces the search to solving an equation, $f'(x) = 0$, usually with finitely many solutions. Optimization algorithms are, at heart, methods to solve that equation when it cannot be solved by hand.

At the bottom of a valley or the top of a hill the tangent is horizontal. But a horizontal tangent can also be an inflection point ($x^3$ at 0) — and in many dimensions, a saddle: a minimum in one direction and a maximum in another. In high-dimensional loss landscapes most critical points are saddles.

Fermat. If $f$ has a local extremum at an interior point $c$ and $f'(c)$ exists, then $f'(c) = 0$.

Second derivative test. If $f'(c) = 0$ and $f''(c) > 0$, $c$ is a strict local minimum; if $f''(c) < 0$, a strict local maximum.

1. Compute $f'$ and solve $f'(x) = 0$; add points where $f'$ does not exist.
2. Classify each with the sign of $f''$ (or the sign change of $f'$).
3. For global extrema on $[a,b]$, also compare the endpoint values. When step 1 has no closed-form solution, iterate: gradient descent or Newton's method on $f'$.

Fit $y \approx w x$ to the points $(1, 2), (2, 3), (3, 7)$. Loss $L(w) = (2 - w)^2 + (3 - 2w)^2 + (7 - 3w)^2$. $L'(w) = -2(2 - w) - 4(3 - 2w) - 6(7 - 3w) = 28w - 58 = 0 \Rightarrow w = 58/28 \approx 2.07$, and $L'' = 28 > 0$: a minimum. This is linear regression in one line.

"Set the derivative to zero" is the most reused recipe of applied mathematics: least squares, maximum likelihood, optimal control and economics all start there. Gradient-based training is that recipe executed numerically when the equation is too big to solve.