What is it?
A function is differentiable at when both one-sided derivatives exist and agree. Differentiable implies continuous, but not conversely: and are continuous with a kink at 0.
Formulas
- subdifferential: the set of slopes at a kink
Where it shows up in AI
ReLU has ; frameworks simply pick a value (usually 0) at the kink.
L1 and hinge losses are non-differentiable at a point and are optimized with subgradients.
Where is it used?
Computing topics reachable from here, through the chain of ideas that leads to them:
λ Scientific computing and algorithms
- Rolle's theorem→Mean value theorem→Taylor's theorem and the remainder→Scientific computing★★★★★
- Rolle's theorem→Mean value theorem→Fundamental theorem of calculus→Cumulative distribution function→Monte Carlo methods★★★★★
- Rolle's theorem→Mean value theorem→Fundamental theorem of calculus→Symbolic computation (CAS)★★★★★
- Rolle's theorem→Mean value theorem→Taylor's theorem and the remainder→Floating point (IEEE 754)★★★★★
- Rolle's theorem→Mean value theorem→Cauchy's mean value theorem→L'Hôpital's rule→Algorithm analysis and complexity★★★★★
ℒ AI and machine learning
- Rolle's theorem→Mean value theorem→Lipschitz continuity→Gradient descent★★★★★
- Rolle's theorem→Mean value theorem→Lipschitz continuity→Gradient descent→Learning rate★★★★★
- Rolle's theorem→Mean value theorem→Lipschitz continuity→Gradient descent→Backpropagation★★★★★
- Rolle's theorem→Mean value theorem→Lipschitz continuity→Gradient descent→Stochastic gradient descent (SGD)★★★★★
- Rolle's theorem→Mean value theorem→Lipschitz continuity→Gradient descent→Loss landscape★★★★★
- Rolle's theorem→Mean value theorem→Lipschitz continuity→Gradient descent→Reinforcement learning★★★★★
- +12
⚛ Physics and simulation
- Rolle's theorem→Mean value theorem→Fundamental theorem of calculus→Physics engines★★★★★
- Rolle's theorem→Mean value theorem→Fundamental theorem of calculus→Physics engines→N-body gravitational simulation★★★★★
- Rolle's theorem→Mean value theorem→Fundamental theorem of calculus→Physics engines→Fluid dynamics and CFD★★★★★
What depends on it
This page has the essentials. A fuller treatment (intuition, formal definition, worked example) is on the way.