Calculus×Computing
ES

Numerical stability

Level UniversityDifficulty ★★★★★Concept⌖ Open in the map

What is it?

Whether an algorithm keeps rounding errors under control. Two mathematically equal formulas can behave very differently: subtracting nearly equal numbers (catastrophic cancellation) or iterating an unstable recurrence destroys accuracy.

Formulas

x1,2=−b∓b2−4ac2a ⟶ x1=−b−sign⁡(b)b2−4ac2a,  x2=ca x1x_{1,2} = \frac{-b \mp \sqrt{b^2 - 4ac}}{2a} \ \longrightarrow\ x_1 = \frac{-b - \operatorname{sign}(b)\sqrt{b^2 - 4ac}}{2a},\ \ x_2 = \frac{c}{a\,x_1}
the stable way to solve a quadratic

Where it shows up in computing

  • Floating point (IEEE 754)★★★★★fundamentalScientific computing and algorithms

    Catastrophic cancellation is the classic floating-point pitfall; stable reformulations avoid it.

  • Physics engines★★★★★fundamentalPhysics and simulation

    Explicit integrators with too large a time step blow up; engines pick stable schemes and sub-step.

Where it shows up in AI

  • Loss function★★★★★frequentAI and machine learning

    Libraries fuse softmax and cross-entropy (log-sum-exp) because the naive composition overflows.

Where is it used?

Computing topics reachable from here, through the chain of ideas that leads to them:

λ Scientific computing and algorithms

⚛ Physics and simulation

ℒ AI and machine learning

∿ Signals, media and vision

⇄ Optimization and systems

What depends on it

This page has the essentials. A fuller treatment (intuition, formal definition, worked example) is on the way.

↑ ↓ to navigate · ↵ · Esc