Lab

$f'(a)$ is the instantaneous rate of change of $f$ at $a$: the slope of the tangent line to the graph, defined as the limit of slopes of secant lines.

Approximate the area under a curve by $n$ thin rectangles, $\sum f(x_i^\ast)\,\Delta x$. As $n \to \infty$ the sum converges to the integral — slowly for the left/right rule ($O(1/n)$), faster for the midpoint ($O(1/n^2)$).

To solve $f(x) = 0$, replace $f$ by its tangent line at the current guess and jump to where the tangent hits zero: $x_{k+1} = x_k - f(x_k)/f'(x_k)$. Near a simple root the number of correct digits doubles every step.

The polynomial of degree $n$ that matches $f$ and its first $n$ derivatives at a point $a$. Degree 1 is the tangent line, degree 2 adds curvature; the higher the degree, the wider the region where it is a good approximation. At $a = 0$ it is called a Maclaurin polynomial.

Repeat $\theta \leftarrow \theta - \eta\,\nabla L(\theta)$: take a small step against the gradient. Cauchy proposed it in 1847; today it (and its stochastic, adaptive variants) trains essentially every neural network.

The algorithm that computes $\partial L/\partial W$ and $\partial L/\partial b$ for every layer of a network: one forward pass storing intermediate values, then one backward pass applying the chain rule from the loss down to the inputs. Cost: about twice the forward pass, whatever the number of parameters.