Derivative

$f'(a)$ is the instantaneous rate of change of $f$ at $a$: the slope of the tangent line to the graph, defined as the limit of slopes of secant lines.

Average rates are easy: distance over time. But "how fast is the car going now?" asks for a rate over an interval of length zero — $0/0$. Newton needed it for mechanics, Leibniz for tangents; the derivative is their common answer, made rigorous by the limit. It solves the problem of quantifying local change.

Geometrically: zoom in on a smooth curve and it looks like a straight line; the derivative is that line's slope. Physically: if $s(t)$ is position, $s'(t)$ is velocity and $s''(t)$ acceleration. As sensitivity: $f(a + h) \approx f(a) + f'(a)\,h$ — the output moves $f'(a)$ times as much as the input, for small nudges. That last reading is the one machine learning uses: the derivative of the loss with respect to a weight says which way to move the weight.

when the limit exists (then $f$ is differentiable at $a$). The function $f' : x \mapsto f'(x)$ is the derivative of $f$; Leibniz writes it $\frac{\dd f}{\dd x}$.

Three ways, and all three are used by software:

1. Symbolically, applying rules (power, product, quotient, chain) to a formula — what a CAS does.
2. Numerically, with a finite difference $\frac{f(a+h) - f(a - h)}{2h}$ — easy but inexact (truncation error for large $h$, rounding error for tiny $h$).
3. Automatically: propagate exact derivatives through each elementary operation of a program — automatic differentiation, the engine of PyTorch and JAX.

$f(x) = x^2$ at $a = 3$: $\frac{(3 + h)^2 - 9}{h} = \frac{6h + h^2}{h} = 6 + h \to 6$. So $f'(3) = 6$ and the tangent is $y = 9 + 6(x - 3)$. In code, the central difference with $h = 10^{-5}$ gives $6.000000000039…$, while $h = 10^{-15}$ gives garbage — the subtraction cancels all the digits.

Optimization is following derivatives downhill; simulation is integrating them forward in time; rendering needs them to orient surfaces; control uses them to anticipate. Modern deep learning is, at its computational core, an extremely efficient machine for computing derivatives of one number (the loss) with respect to billions of inputs (the weights).