Learning rate

Level UniversityDifficulty ★★★★★Application⌖ Open in the map

What is it?

The step size $\eta$ of gradient descent — the most important hyperparameter. Stability requires $\eta < 2/\lambda_{\max}$ (the largest curvature); schedules (warm-up, cosine decay) start careful, go fast, and end small to settle into a minimum.

Formulas

$\eta_t = \eta_{\min} + \tfrac12(\eta_{\max} - \eta_{\min})\big(1 + \cos(\pi t/T)\big)$
cosine schedule

Where is it used?

Computing topics reachable from here, through the chain of ideas that leads to them:

ℒ AI and machine learning

Stochastic gradient descent (SGD)★★★★★
Stochastic gradient descent (SGD)→Momentum and Adam★★★★★
Stochastic gradient descent (SGD)→Momentum and Adam→Deep learning★★★★★
Stochastic gradient descent (SGD)→Momentum and Adam→Deep learning→Convolutional networks (CNNs)★★★★★
Stochastic gradient descent (SGD)→Momentum and Adam→Deep learning→Generative models★★★★★
Stochastic gradient descent (SGD)→Momentum and Adam→Deep learning→Neural ODEs★★★★★

What depends on it

Stochastic gradient descent (SGD)

This page has the essentials. A fuller treatment (intuition, formal definition, worked example) is on the way.