Variance

Level UniversityDifficulty ★★★★★Concept⌖ Open in the map

What is it?

$\Var(X) = \E[(X - \E X)^2]$ : the average squared distance from the mean. The variance of an average of $N$ independent samples is $\sigma^2/N$ — the $1/\sqrt N$ law that governs Monte Carlo error, batch sizes and A/B tests.

Formulas

\Var(X) = \E[X^2] - (\E X)^2, \qquad \Var\Big(\frac1N\sum_{i=1}^{N}X_i\Big) = \frac{\sigma^2}{N}

Where it shows up in computing

Monte Carlo methods★★★★★fundamentalScientific computing and algorithms
MC error is $\sigma/\sqrt N$ ; variance reduction (importance sampling, control variates) is the main lever.
Kalman filter★★★★★fundamentalRobotics and control
The Kalman gain weighs prediction and measurement by their variances.

Where it shows up in AI

Stochastic gradient descent (SGD)★★★★★frequentAI and machine learning
Gradient noise variance falls as $1/|B|$ ; it sets the useful learning rate and batch size.
Momentum and Adam★★★★★frequentAI and machine learning
Adam divides by a running estimate of the gradient's second moment.
Neural networks★★★★★frequentAI and machine learning
Xavier/He initialization chooses weight variances so activations keep a stable variance through layers.

Where is it used?

Computing topics reachable from here, through the chain of ideas that leads to them:

λ Scientific computing and algorithms

Monte Carlo methods★★★★★

3D Computer graphics

Monte Carlo methods→The rendering equation★★★★★

⚙ Robotics and control

Kalman filter★★★★★

ℒ AI and machine learning

Stochastic gradient descent (SGD)★★★★★
Momentum and Adam★★★★★
Momentum and Adam→Deep learning★★★★★
Momentum and Adam→Deep learning→Convolutional networks (CNNs)★★★★★
Momentum and Adam→Deep learning→Generative models★★★★★
Momentum and Adam→Deep learning→Neural ODEs★★★★★
+4

This page has the essentials. A fuller treatment (intuition, formal definition, worked example) is on the way.