Directional derivative

Level UniversityDifficulty ★★★★★Concept⌖ Open in the map

What is it?

The rate of change of $f$ moving in a unit direction $u$ : $D_u f = \nabla f\cdot u$ . It proves that $-\nabla f$ is the direction of steepest descent, and it is what forward-mode AD computes (a Jacobian–vector product).

Formulas

D_u f(a) = \lim_{t\to0}\frac{f(a + t u) - f(a)}{t} = \nabla f(a)\cdot u

Where it shows up in AI

Gradient descent★★★★★fundamentalAI and machine learning
Among unit steps, $u = -\nabla f/\norm{\nabla f}$ minimizes $D_u f$ : the justification of the method.
Automatic differentiation★★★★★frequentAI and machine learning
Forward-mode AD with dual numbers computes $D_u f$ (a JVP) in one pass.

Where is it used?

Computing topics reachable from here, through the chain of ideas that leads to them:

ℒ AI and machine learning

Gradient descent★★★★★
Automatic differentiation★★★★★
Gradient descent→Learning rate★★★★★
Automatic differentiation→Backpropagation★★★★★
Gradient descent→Stochastic gradient descent (SGD)★★★★★
Gradient descent→Loss landscape★★★★★
+7

⇄ Optimization and systems

Gradient descent→Recommender systems★★★★★

This page has the essentials. A fuller treatment (intuition, formal definition, worked example) is on the way.