Skip to content

Matrices and linear maps

Level UniversityDifficulty ★★★★★Concept⌖ Open in the map

What is it?

A matrix $A \in \R^{m \times n}$ is a linear map $x \mapsto Ax$ from $\R^n$ to $\R^m$ . The derivative of a function of several variables is a matrix (the Jacobian), and a layer of a neural network is $Wx + b$ .

Formulas

$(Ax)_i = \sum_j a_{ij} x_j, \qquad (AB)x = A(Bx)$; $\det\begin{pmatrix} a & b \\ c & d \end{pmatrix} = ad - bc$
area scaling factor of the map

Where it shows up in computing

Kalman filter★★★★★frequentRobotics and control
State transition, observation and covariance updates are all matrix products.
Control theory★★★★★frequentRobotics and control
State-space models $\dot x = Ax + Bu$ are linear maps.

Where it shows up in AI

Neural networks★★★★★fundamentalAI and machine learning
A dense layer is $\sigma(Wx + b)$ ; GPUs exist to multiply these matrices fast.

Where is it used?

Computing topics reachable from here, through the chain of ideas that leads to them:

λ Scientific computing and algorithms

Jacobian matrix→Multiple integrals and change of variables→Monte Carlo methods★★★★★
Jacobian matrix→Equilibria and stability→Stiffness and implicit methods→Scientific computing★★★★★
Jacobian matrix→Equilibria and stability→Attractors→Chaos and sensitivity to initial conditions→Floating point (IEEE 754)★★★★★

ℒ AI and machine learning

Neural networks★★★★★
Neural networks→Backpropagation★★★★★
Hessian matrix→Loss landscape★★★★★
Hessian matrix→Second-order (Hessian-based) optimization★★★★★
Jacobian matrix→Generative models★★★★★
Hessian matrix→Extrema in several variables→Linear regression★★★★★
+10

3D Computer graphics

Jacobian matrix→Multiple integrals and change of variables→Monte Carlo methods→The rendering equation★★★★★
Jacobian matrix→Multiple integrals and change of variables→Green's theorem→Mesh processing (discrete differential geometry)★★★★★

⚙ Robotics and control

Jacobian matrix→Robot Jacobian (velocity kinematics)★★★★★
Jacobian matrix→Inverse kinematics★★★★★
Systems of ODEs→Dynamical systems→Control theory★★★★★
Systems of ODEs→Dynamical systems→Control theory→Trajectory optimization and MPC★★★★★
Kalman filter★★★★★
Systems of ODEs→Robot dynamics★★★★★

⚛ Physics and simulation

Systems of ODEs→Population and epidemic models★★★★★
Systems of ODEs→N-body gravitational simulation★★★★★
Systems of ODEs→Dynamical systems→Weather and climate modelling★★★★★
Jacobian matrix→Multiple integrals and change of variables→Surface integrals and flux→Divergence theorem (Gauss)→Electromagnetism (Maxwell's equations)★★★★★
Jacobian matrix→Multiple integrals and change of variables→Surface integrals and flux→Divergence theorem (Gauss)→Fluid dynamics and CFD★★★★★
Jacobian matrix→Equilibria and stability→Stiffness and implicit methods→Physics engines★★★★★

⇄ Optimization and systems

Jacobian matrix→Equilibria and stability→Gradient descent→Recommender systems★★★★★

∿ Signals, media and vision

Hessian matrix→Image processing and computer vision★★★★★
Hessian matrix→Image processing and computer vision→Media compression (JPEG, MP3, video)★★★★★

What depends on it

This page has the essentials. A fuller treatment (intuition, formal definition, worked example) is on the way.