Skip to content

Derivatives of elementary functions

Level FundamentalDifficulty ★★★★★Concept⌖ Open in the map

What is it?

The table every differentiation engine starts from: powers, exponentials, logarithms, trigonometric and hyperbolic functions — and the activation functions built from them.

Formulas

$(x^r)' = r x^{r-1}, \quad (a^x)' = a^x \ln a, \quad (\log_a x)' = \frac{1}{x \ln a}$; $(\tan x)' = 1 + \tan^2 x, \quad (\arctan x)' = \frac{1}{1 + x^2}$; $\sigma(x) = \frac{1}{1 + e^{-x}} \implies \sigma'(x) = \sigma(x)\,\big(1 - \sigma(x)\big)$
the sigmoid: its derivative comes for free from its value

Where it shows up in AI

Activation functions★★★★★fundamentalAI and machine learning
$\sigma' = \sigma(1 - \sigma)$ and $\tanh' = 1 - \tanh^2$ : backprop reuses the forward value.
Automatic differentiation★★★★★fundamentalAI and machine learning
Every AD system ships a table of derivatives of its primitive operations.

Where is it used?

Computing topics reachable from here, through the chain of ideas that leads to them:

λ Scientific computing and algorithms

Antiderivatives and indefinite integrals→Symbolic computation (CAS)★★★★★
Antiderivatives and indefinite integrals→Fundamental theorem of calculus→Cumulative distribution function→Monte Carlo methods★★★★★
Antiderivatives and indefinite integrals→Ordinary differential equations→Euler's method→Runge–Kutta methods→Scientific computing★★★★★
Antiderivatives and indefinite integrals→Fundamental theorem of calculus→Improper integrals→Integral test→Algorithm analysis and complexity★★★★★

ℒ AI and machine learning

Activation functions★★★★★
Activation functions→Neural networks★★★★★
Activation functions→Neural networks→Backpropagation★★★★★
Activation functions→Neural networks→Loss landscape★★★★★
Antiderivatives and indefinite integrals→Ordinary differential equations→Neural ODEs★★★★★
Activation functions→Neural networks→Backpropagation→Deep learning★★★★★
+11

3D Computer graphics

Antiderivatives and indefinite integrals→Fundamental theorem of calculus→Cumulative distribution function→Monte Carlo methods→The rendering equation★★★★★

∿ Signals, media and vision

Antiderivatives and indefinite integrals→Integration by substitution→Trigonometric integrals→Fourier series→Signal processing★★★★★
Antiderivatives and indefinite integrals→Fundamental theorem of calculus→Improper integrals→Fourier transform→Sampling theorem (Nyquist–Shannon)★★★★★
Antiderivatives and indefinite integrals→Fundamental theorem of calculus→Improper integrals→Fourier transform→Fast Fourier transform (FFT)★★★★★
Antiderivatives and indefinite integrals→Ordinary differential equations→Laplace transform→Z-transform→Digital filters★★★★★
Antiderivatives and indefinite integrals→Fundamental theorem of calculus→Improper integrals→Fourier transform→Telecommunications (modulation, OFDM)★★★★★
Antiderivatives and indefinite integrals→Fundamental theorem of calculus→Improper integrals→Fourier transform→Image processing and computer vision★★★★★
+1

ψ Quantum computing and physics

Antiderivatives and indefinite integrals→Fundamental theorem of calculus→Improper integrals→Fourier transform→Uncertainty principle★★★★★

⇄ Optimization and systems

Antiderivatives and indefinite integrals→Ordinary differential equations→Euler's method→Gradient descent→Recommender systems★★★★★
Antiderivatives and indefinite integrals→Fundamental theorem of calculus→Cumulative distribution function→Queueing theory and performance★★★★★

What depends on it

Antiderivatives and indefinite integrals

Exercises

1Proof

Prove that $\sigma'(x) = \sigma(x)(1 - \sigma(x))$ for $\sigma(x) = 1/(1 + e^{-x})$ . What is the maximum of $\sigma'$ ?

Solution

$\sigma'(x) = \frac{e^{-x}}{(1 + e^{-x})^2} = \frac{1}{1 + e^{-x}}\cdot\frac{e^{-x}}{1 + e^{-x}} = \sigma(1 - \sigma)$ . Since $s(1 - s) \le \frac14$ , the maximum is $\frac14$ at $x = 0$ — so each sigmoid layer shrinks gradients by at least a factor 4, one cause of vanishing gradients.

This page has the essentials. A fuller treatment (intuition, formal definition, worked example) is on the way.