Lipschitz continuity

What is it?

Formulas

$$|f(x) - f(y)| \le L\,|x - y|$$

$$\norm{\nabla f(x) - \nabla f(y)} \le L\norm{x - y} \implies f(y) \le f(x) + \nabla f(x)\cdot(y - x) + \tfrac L2\norm{y - x}^2$$

descent lemma

Where it shows up in AI

• Gradient descent★★★★★fundamentalAI and machine learning

  The safe learning rate is set by the Lipschitz constant of the gradient: $\eta < 2/L$.

• Generative models★★★★★advancedAI and machine learning

  Wasserstein GANs require a 1-Lipschitz critic, enforced by weight clipping, gradient penalties or spectral normalization.

Where is it used?

Computing topics reachable from here, through the chain of ideas that leads to them:

What depends on it

• Initial value problems: existence and uniqueness