Probability density function

A function $p(x) \ge 0$ with $\int p = 1$ whose integral over a set is the probability of that set. A density is not a probability: it can exceed 1; it is probability per unit length.

With infinitely many possible values, each individual value must have probability zero, yet some regions are more likely than others. The density describes that relative likelihood, and integration recovers actual probabilities.

Think of a histogram of many samples with bins getting thinner and thinner, normalized so the total area is 1. In the limit the bars become a curve: the density. Probability is area under it; $p(x)\,\dd x$ is the probability of landing in a tiny interval near $x$.

$X$ has density $p$ if $P(X \in A) = \int_A p(x)\,\dd x$ for every (measurable) set $A$, with $p \ge 0$ and $\int_{-\infty}^{\infty}p = 1$. Then $p = F'$ where $F$ is the cumulative distribution function.

Uniform on $[0, 0.5]$: $p(x) = 2$ there. The density is 2 (> 1) but every probability is fine: $P(X \le 0.1) = \int_0^{0.1}2\,\dd x = 0.2$.

Generative AI is density modelling: VAEs, normalizing flows and diffusion models all learn a density $p_\theta(x)$ of images or sounds (explicitly or through its score $\nabla_x\log p$). Bayesian inference updates densities with data, and the normalizing integral in the denominator is usually the hard part.