Calculus×Computing
ES

Gradient

Level UniversityDifficulty ★★★★★Concept⌖ Open in the map

What is it?

∇f=(∂f∂x1,…,∂f∂xn)\nabla f = \left(\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n}\right): the vector of all partial derivatives. It points in the direction of steepest ascent, its length is that steepest slope, and it is perpendicular to the level sets. Walk against it and you go downhill fastest.

Why does it exist?

We want one object that answers "which way is up, and how steep?" for a function of many variables. The partial derivatives are nn separate numbers; assembled into a vector they gain a geometric meaning that does not depend on the coordinate system — and that turns optimization into "follow an arrow".

Intuition

Stand on a hillside in fog. The gradient is the arrow on the ground pointing straight uphill; contour lines run perpendicular to it. Its length is the slope in that direction. Gradient descent is: look at the arrow, take a step the other way, repeat.

               ∇f
               ↑
               │
       ╭───────●───────╮   ← level curve f = c
     ╭─┴───────────────┴─╮

Formal definition

For f:ℝn→ℝf : \R^n \to \R differentiable at aa, ∇f(a)\nabla f(a) is the unique vector with

f(a+h)=f(a)+∇f(a)⋅h+o(∥h∥).f(a + h) = f(a) + \nabla f(a)\cdot h + o(\norm h).

The directional derivative in a unit direction uu is Duf(a)=∇f(a)⋅uD_u f(a) = \nabla f(a)\cdot u, maximized by u=∇f/∥∇f∥u = \nabla f/\norm{\nabla f} (Cauchy–Schwarz), with maximum value ∥∇f(a)∥\norm{\nabla f(a)}. If ∇f(a)≠0\nabla f(a) \ne 0, it is orthogonal to the level set {f=f(a)}\{f = f(a)\}.

Formulas

∇f=(∂f∂x1,…,∂f∂xn)\nabla f = \left(\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n}\right)
Duf=∇f⋅u≤∥∇f∥D_u f = \nabla f\cdot u \le \norm{\nabla f}
steepest ascent along ∇f\nabla f
θk+1=θk−η ∇L(θk)\theta_{k+1} = \theta_k - \eta\,\nabla L(\theta_k)
gradient descent
n=∇F∥∇F∥n = \frac{\nabla F}{\norm{\nabla F}}
unit normal of an implicit surface F(x,y,z)=0F(x,y,z) = 0

How is it computed?

By hand: all partial derivatives. In software, for a scalar loss of nn parameters, reverse-mode automatic differentiation computes the whole gradient for a small constant times the cost of evaluating ff (Baur–Strassen), while finite differences would need n+1n + 1 evaluations.

Example

f(x,y)=x2+4y2f(x, y) = x^2 + 4y^2 (an elongated bowl). ∇f=(2x,8y)\nabla f = (2x, 8y). At (2,1)(2, 1): ∇f=(4,8)\nabla f = (4, 8), which does not point at the minimum (0,0)(0, 0) — it points across the narrow valley. That is why plain gradient descent zigzags on badly conditioned problems (try it in the Gradient descent demo).

Why does it matter?

The gradient is the bridge from calculus to modern AI: every model trained by gradient descent — from linear regression to large language models — follows −∇L-\nabla L. It is also how renderers get surface normals from implicit shapes, how edge detectors find boundaries, and how physics derives forces from potentials (F=−∇UF = -\nabla U).

Where it shows up in computing

  • Surface normals★★★★★fundamentalComputer graphics

    The normal of an implicit surface F=0F = 0 is ∇F/∥∇F∥\nabla F/\norm{\nabla F}.

  • Signed distance fields and ray marching★★★★★fundamentalComputer graphics

    For a signed distance dd, ∥∇d∥=1\norm{\nabla d} = 1 and ∇d\nabla d is the surface normal; ray marchers estimate it by finite differences.

  • Image processing and computer vision★★★★★frequentSignals, media and vision

    Edges are large ∥∇I∥\norm{\nabla I}; HOG features are histograms of gradient directions.

  • Classical mechanics★★★★★frequentPhysics and simulation

    Conservative forces are minus the gradient of a potential: F=−∇UF = -\nabla U.

Where it shows up in AI

  • Gradient descent★★★★★fundamentalAI and machine learning

    The update θ←θ−η∇L(θ)\theta \leftarrow \theta - \eta\nabla L(\theta) is the gradient, used as a direction.

Where is it used?

Computing topics reachable from here, through the chain of ideas that leads to them:

ℒ AI and machine learning

3D Computer graphics

⚙ Robotics and control

⚛ Physics and simulation

⇄ Optimization and systems

ψ Quantum computing and physics

∿ Signals, media and vision

What depends on it

Exercises

1Computation

Find ∇f\nabla f for f(x,y,z)=x2y+yz3f(x,y,z) = x^2y + yz^3 at (1,2,−1)(1, 2, -1) and the rate of increase in the direction (1,1,1)/3(1,1,1)/\sqrt3.

Solution

∇f=(2xy,x2+z3,3yz2)=(4,0,6)\nabla f = (2xy, x^2 + z^3, 3yz^2) = (4, 0, 6). Duf=(4+0+6)/3=10/3≈5.77D_u f = (4 + 0 + 6)/\sqrt3 = 10/\sqrt3 \approx 5.77; the maximum possible is ∥∇f∥=52≈7.21\norm{\nabla f} = \sqrt{52} \approx 7.21.

2Graphical

Draw the level curves of f(x,y)=x2+4y2f(x,y) = x^2 + 4y^2 and the gradient at (2,1)(2, 1). Why does the arrow not point to the origin?

Solution

Level curves are ellipses, wider in xx. The gradient (4,8)(4, 8) is perpendicular to the ellipse through (2,1)(2,1), which is not the radial direction (2,1)(2, 1) because the ellipse is not a circle.

3AI

Why does reverse-mode AD compute ∇L∈ℝ109\nabla L \in \R^{10^9} in roughly the time of 3–5 evaluations of LL, while finite differences would need 10910^9?

Solution

Finite differences perturb one parameter at a time. Reverse mode runs the program once forward, then once backward propagating ∂L/∂(intermediate)\partial L/\partial(\text{intermediate}); each intermediate is visited once, so the cost is a constant multiple of the forward pass, independent of the number of parameters (one scalar output).

↑ ↓ to navigate · ↵ · Esc