Neural ODEs

Level SpecializationDifficulty ★★★★★Application⌖ Open in the map

What is it?

A residual network $x_{\ell+1} = x_\ell + hF(x_\ell)$ is Euler's method; let $h \to 0$ and the network becomes an ODE $\dot x = F_\theta(x, t)$ , evaluated by an ODE solver and trained by the adjoint method (reverse-mode AD in continuous time).

Formulas

\frac{\dd h}{\dd t} = f_\theta(h, t), \qquad \frac{\dd a}{\dd t} = -a^{\mathsf T}\frac{\partial f_\theta}{\partial h}, \quad a = \frac{\partial L}{\partial h}

The mathematics behind it

Ordinary differential equations★★★★★fundamental
A neural ODE defines the hidden state by $h'(t) = f_\theta(h, t)$ and calls an ODE solver as a layer.
Runge–Kutta methods★★★★★frequent
Neural ODE libraries (torchdiffeq, Diffrax) default to adaptive Runge–Kutta solvers.

This page has the essentials. A fuller treatment (intuition, formal definition, worked example) is on the way.