Regularization

Level AdvancedDifficulty ★★★★★Application⌖ Open in the map

What is it?

Add a penalty to the loss, $L + \lambda\norm\theta^2$ (weight decay) or $\lambda\norm\theta_1$ (sparsity), to prefer simpler models and reduce overfitting. Its gradient $2\lambda\theta$ shrinks every weight a little each step.

Formulas

$\nabla\big(L + \lambda\norm\theta^2\big) = \nabla L + 2\lambda\theta$; $w^\ast_{\text{ridge}} = (X^{\mathsf T}X + \lambda I)^{-1}X^{\mathsf T}y$
ridge regression also cures ill-conditioning

The mathematics behind it

Lagrange multipliers★★★★★advanced
Penalized training $L + \lambda\norm w^2$ is the Lagrangian of training under a norm constraint $\norm w \le r$ .

Where is it used?

Computing topics reachable from here, through the chain of ideas that leads to them:

ℒ AI and machine learning

Deep learning★★★★★
Deep learning→Convolutional networks (CNNs)★★★★★
Deep learning→Generative models★★★★★
Deep learning→Neural ODEs★★★★★

What depends on it

Deep learning

This page has the essentials. A fuller treatment (intuition, formal definition, worked example) is on the way.