Upstream
Where does backpropagation come from?
- Functions
- Derivative
- Chain rule
- Functions of several variables
- Partial derivatives
- Gradient
- Multivariable chain rule
- Loss function
- Gradient descent
- Neural networks
- Backpropagation
Overview
Calculus × Computing
All of university calculus — from limits to vector calculus, differential equations and Fourier — and where every idea shows up in AI, graphics, simulation, robotics and scientific computing.
- topics
- 208
- connections
- 640
- areas
- 26
- interactive figures
- 6
From mathematics to computing
The thesis of this portal
Downstream
Where is the derivative used?
- λ Scientific computing and algorithmsSymbolic computation (CAS) · Scientific computing · Floating point (IEEE 754)
- ℒ AI and machine learningGradient descent · Automatic differentiation · Learning rate
- 3D Computer graphicsSurface normals · Signed distance fields and ray marching · Mesh processing (discrete differential geometry)
- ⚙ Robotics and controlKinematics: position, velocity, acceleration · Robot Jacobian (velocity kinematics) · Robot dynamics
- ⚛ Physics and simulationPhysics engines · Classical mechanics · Population and epidemic models
- ∿ Signals, media and visionImage processing and computer vision · Media compression (JPEG, MP3, video) · Digital filters
- ⇄ Optimization and systemsRecommender systems · Operations research and logistics · Queueing theory and performance
Pick a route
Learning routes →∫
Recover university calculus
The classic first-year sequence, from functions to several variables, for anyone who wants the foundations back.
∇Mathematics for AI
The shortest honest path from derivatives to deep learning: gradients, optimization, likelihood and backpropagation.
3DComputer graphics
From derivatives to rendering: vectors, normals, surfaces, lighting and the integral that gives each pixel its colour.
ẏSimulation
How a computer predicts motion: differential equations, numerical integrators, stability and chaos.
⚛Computational physics
Fields and waves: vector calculus, Fourier analysis and partial differential equations, up to fluids.
Try it
Lab →Derivative
is the instantaneous rate of change of at : the slope of the tangent line to the graph, defined as the limit of slopes of secant lines.
Riemann sums
Approximate the area under a curve by thin rectangles, . As the sum converges to the integral — slowly for the left/right rule (), faster for the midpoint ().
Newton's method
To solve , replace by its tangent line at the current guess and jump to where the tangent hits zero: . Near a simple root the number of correct digits doubles every step.
Taylor polynomial
The polynomial of degree that matches and its first derivatives at a point . Degree 1 is the tangent line, degree 2 adds curvature; the higher the degree, the wider the region where it is a good approximation. At it is called a Maclaurin polynomial.
Gradient descent
Repeat : take a small step against the gradient. Cauchy proposed it in 1847; today it (and its stochastic, adaptive variants) trains essentially every neural network.
Backpropagation
The algorithm that computes and for every layer of a network: one forward pass storing intermediate values, then one backward pass applying the chain rule from the loss down to the inputs. Cost: about twice the forward pass, whatever the number of parameters.
All areas
Calculus and its neighbours
ƒFoundations16v⃗Linear algebra (bridge)3limSequences and limits11εContinuity7f′Derivatives8∎Fundamental theorems5minExtrema and optimization6≈Numerical methods10TₙTaylor series5∫Integrals8ΣSeries8∇Multivariable calculus13∮Vector calculus10ẏDifferential equations10φDynamical systems and chaos7ℱTransforms6ℙProbability and calculus7