Calculus×Computing
ES

Image processing and computer vision

Level UniversityDifficulty ★★★★★Application⌖ Open in the map

What is it?

An image is a function I(x,y)I(x, y). Blurring is convolution, edges are large gradients ∥∇I∥\norm{\nabla I}, corners come from second derivatives, motion between frames from the optical-flow equation. Classical vision is applied calculus; modern vision learns the filters (CNNs).

Formulas

Gx=(−101−202−101)∗I,∥∇I∥≈Gx2+Gy2G_x = \begin{pmatrix}-1 & 0 & 1\\ -2 & 0 & 2\\ -1 & 0 & 1\end{pmatrix} * I, \qquad \norm{\nabla I} \approx \sqrt{G_x^2 + G_y^2}
Sobel edge detector
Ix u+Iy v+It=0I_x\,u + I_y\,v + I_t = 0
optical flow constraint (brightness constancy + chain rule)

The mathematics behind it

  • Interpolation★★★★★fundamental

    Resizing and rotating images samples between pixels with bilinear or bicubic interpolation.

  • Partial derivatives★★★★★fundamental

    Edge detection estimates ∂I/∂x\partial I/\partial x and ∂I/∂y\partial I/\partial y of the intensity.

  • Convolution★★★★★fundamental

    Blur, sharpen and edge filters are convolutions with small kernels.

  • Discontinuities★★★★★frequent

    Edges in an image are (approximate) jump discontinuities of intensity; edge detectors look for them.

  • Derivative★★★★★frequent

    Edges are where intensity changes fast: detectors (Sobel, Canny) estimate derivatives of the image.

  • Numerical differentiation★★★★★frequent

    Sobel and Prewitt filters are smoothed finite differences of pixel intensities.

  • Functions of several variables★★★★★frequent

    A grayscale image is a function I(x,y)I(x, y) sampled on a grid; a colour image is ℝ2→ℝ3\R^2 \to \R^3.

  • Gradient★★★★★frequent

    Edges are large ∥∇I∥\norm{\nabla I}; HOG features are histograms of gradient directions.

  • Laplacian★★★★★frequent

    Laplacian-of-Gaussian edge detection, sharpening and Poisson image blending.

  • Fourier transform★★★★★frequent

    Frequency-domain filtering, deblurring and image registration (phase correlation).

  • Transformations of functions★★★★★frequent

    Brightness and contrast adjustments are a I(x,y)+ba\,I(x,y) + b.

  • Hessian matrix★★★★★advanced

    Hessian-based detectors (determinant of Hessian in SURF, Frangi vesselness) find blobs and ridges.

  • Partial differential equations★★★★★advanced

    Anisotropic (Perona–Malik) diffusion denoises images while preserving edges by solving a nonlinear heat equation.

  • Wavelets★★★★★frequent

    Wavelet shrinkage is a classic denoising method.

Where is it used?

Computing topics reachable from here, through the chain of ideas that leads to them:

∿ Signals, media and vision

What depends on it

Exercises

1Computing

Why do edge detectors blur the image (e.g. with a Gaussian) before differentiating?

Solution

Differentiation amplifies high-frequency noise (in Fourier terms it multiplies by 2πiξ2\pi i\xi). Smoothing first suppresses those frequencies; since derivative and convolution commute, ∇(G∗I)=(∇G)∗I\nabla(G * I) = (\nabla G) * I, so one convolution with the derivative of a Gaussian does both.

This page has the essentials. A fuller treatment (intuition, formal definition, worked example) is on the way.

↑ ↓ to navigate · ↵ · Esc