Activation functions

What is it?

Formulas

$$\sigma(z) = \frac{1}{1 + e^{-z}}, \quad \tanh z, \quad \operatorname{ReLU}(z) = \max(0, z), \quad \operatorname{GELU}(z) = z\,\Phi(z)$$

$$\sigma' = \sigma(1-\sigma) \le \tfrac14, \qquad \operatorname{ReLU}'(z) = \mathbf 1[z > 0]$$

Why does it matter?

The mathematics behind it

• Derivatives of elementary functions★★★★★fundamental

  $\sigma' = \sigma(1 - \sigma)$ and $\tanh' = 1 - \tanh^2$: backprop reuses the forward value.

• Exponential functions★★★★★frequent

  Sigmoid, softmax and tanh are all built from $e^x$.

• Continuity★★★★★fundamental

  Activations must be continuous (and almost everywhere differentiable) for gradient training to work.

• Discontinuities★★★★★historical

  The perceptron's step activation is discontinuous; replacing it by the sigmoid made backpropagation possible.

• Differentiability and one-sided derivatives★★★★★fundamental

  ReLU has $f'_-(0) = 0 \ne 1 = f'_+(0)$; frameworks simply pick a value (usually 0) at the kink.

• Hyperbolic functions★★★★★frequent

  $\tanh$ squashes to $(-1,1)$ and its derivative $1 - \tanh^2$ is cheap to compute from the output.

• Domain and range★★★★★frequent

  The range of the sigmoid, $(0,1)$, is why its output can be read as a probability.

• Limits at infinity★★★★★frequent

  The sigmoid saturates: as $|x| \to \infty$ its slope tends to 0, the root of vanishing gradients.

Where is it used?

Computing topics reachable from here, through the chain of ideas that leads to them:

What depends on it

• Neural networks