Timeline
Four centuries to ChatGPT
Modern AI is the confluence of ideas born centuries apart and for very different reasons: splitting wagers, computing orbits, sending messages over the telephone. Here they are in order, two winters included. Filter by chapter to follow the thread of each theory.
1654 – 1927
The foundations
Probability, calculus and linear algebra: tools created for gambling, physics and astronomy, which three centuries later would serve to learn from data.
-
1654
Pascal and Fermat invent probability
Blaise Pascal, Pierre de Fermat
A correspondence about how to split the stakes of an interrupted game founds the calculus of probability.
Chapter 03 · Probability → -
1676
The chain rule
Gottfried W. Leibniz
Leibniz differentiates composite functions in his notes. Three centuries later, backpropagation will be this rule applied systematically.
Chapter 06 · Calculus → -
1713
Law of large numbers
Jakob Bernoulli
Ars Conjectandi is published posthumously: the observed frequency converges to the true probability.
Chapter 03 · Probability → -
1763
Bayes' theorem
Thomas Bayes, Richard Price
Price publishes Bayes's posthumous essay on reasoning from effects back to causes.
Chapter 03 · Probability → -
1805
Least squares
Adrien-Marie Legendre, Carl F. Gauss
Legendre publishes the method for fitting the orbits of comets. Gauss had used it to recover the dwarf planet Ceres in 1801. It is the first “loss function”.
Chapter 07 · Optimization → -
1844
Spaces of any dimension
Hermann Grassmann
Die lineale Ausdehnungslehre introduces vector spaces with any number of dimensions.
Chapter 05 · Linear algebra → -
1847
Gradient descent
Augustin-Louis Cauchy
To solve astronomical equations, Cauchy proposes walking down step by step in the direction of steepest slope.
Chapter 07 · Optimization → -
1854
The laws of thought
George Boole
Boole writes logic as algebra: variables that are worth 0 or 1, and operations that obey laws like those of numbers.
Chapter 00 · Circuits and automata → -
1858
The theory of matrices
Arthur Cayley
Cayley defines the matrix product and treats matrices as algebraic objects in their own right.
Chapter 05 · Linear algebra → -
1879
The Begriffsschrift
Gottlob Frege
The first complete formal language for mathematics, with variables, quantifiers and rules of inference.
Chapter 02 · Logic → -
1922
Maximum likelihood
Ronald A. Fisher
Fisher formalizes the principle of choosing the parameters that make the data most probable: the objective every LLM is trained with.
Chapter 03 · Probability →
1928 – 1955
Computation and information are born
Turing defines what computing is, Shannon what information is, and McCulloch and Pitts imagine neurons that compute.
-
1928
The decision problem
David Hilbert, Wilhelm Ackermann
Is there a mechanical procedure that decides whether any logical statement can be proved?
Chapter 01 · Computation → -
1929
The completeness theorem
Kurt Gödel
In first-order logic, everything that is true in every model can be proved: truth and proof coincide.
Chapter 02 · Logic → -
1931
The incompleteness theorems
Kurt Gödel
Every sufficiently powerful consistent formal system contains truths it cannot prove.
Chapter 01 · Computation → -
1936
The Turing machine
Alan Turing, Alonzo Church
Turing defines computation with a tape and a table of rules, describes the universal machine and proves that the halting problem is undecidable.
Chapter 01 · Computation → -
1937
Circuits obey Boole's algebra
Claude Shannon
In his master's thesis Shannon shows that relay circuits compute Boolean functions, and that designing them is manipulating formulas.
Chapter 00 · Circuits and automata → -
1943
The logical neuron
Warren McCulloch, Walter Pitts
The first mathematical model of a network of neurons: thresholds that compute logical functions.
Chapter 08 · Neural networks → -
1945
The stored-program computer
John von Neumann
The EDVAC report describes the architecture in which program and data share memory.
Chapter 01 · Computation → -
1948
A mathematical theory of communication
Claude Shannon
Shannon defines entropy, proves the limits of compression and generates text with -gram models.
Chapter 04 · Information → -
1948
The Manchester Baby
Frederic Williams, Tom Kilburn, Geoff Tootill
On 21 June, the first computer to run a program stored in its own memory.
Chapter 00 · Circuits and automata → -
1949
Hebb's rule
Donald Hebb
“Neurons that fire together wire together”: the first hypothesis about how a network learns.
Chapter 08 · Neural networks → -
1950
The imitation game
Alan Turing
“Computing Machinery and Intelligence” proposes the Turing test and suggests building machines that learn like a child.
Chapter 01 · Computation → -
1951
Stochastic approximation
Herbert Robbins, Sutton Monro
You can optimize with noisy estimates of the gradient if the learning rate decreases appropriately.
Chapter 07 · Optimization → -
1951
The Kullback–Leibler divergence
Solomon Kullback, Richard Leibler
A measure of distance between distributions that reappears in training and in RLHF.
Chapter 04 · Information → -
1951
Regular events
Stephen Kleene
Analysing McCulloch and Pitts' nets, Kleene characterises what finite automata recognise: the languages of regular expressions.
Chapter 00 · Circuits and automata → -
1952
Huffman codes
David Huffman
A simple algorithm builds the optimal prefix-free code for a given distribution.
Chapter 04 · Information →
1956 – 1973
The first spring
AI gets its name. Boundless optimism: perceptrons, programs that play checkers and chatbots that imitate a psychotherapist.
-
1956
“Artificial intelligence” is born
John McCarthy, Marvin Minsky, Nathaniel Rochester, Claude Shannon
The Dartmouth summer workshop gives the field its name. The proposal (1955) estimated that a select group could make “a significant advance” in two months.
-
1956
The Logic Theorist
Allen Newell, Herbert Simon, Cliff Shaw
The first program that proves theorems: 38 of the first 52 in chapter 2 of the Principia Mathematica.
Chapter 02 · Logic → -
1956
Three models for language
Noam Chomsky
Chomsky compares finite automata and grammars as models of language; by 1959 they form his four-level hierarchy.
Chapter 00 · Circuits and automata → -
1957
The perceptron
Frank Rosenblatt
An artificial neuron with a learning rule. The following year the Mark I Perceptron is unveiled.
Chapter 08 · Neural networks → -
1957
Dynamic programming
Richard Bellman
The Bellman equation founds sequential decision-making, the basis of reinforcement learning.
Chapter 11 · Language models → -
1957
The maximum entropy principle
Edwin T. Jaynes
The least committed distribution compatible with what we know is exponential: the softmax with temperature.
Chapter 11 · Language models → -
1959
“Machine learning”
Arthur Samuel
A checkers program that improves by playing against itself popularizes the term machine learning.
Chapter 09 · Learning theory → -
1959
Nondeterministic automata
Michael Rabin, Dana Scott
Automata that guess, and the subset construction that shows they recognise nothing new. Turing Award in 1976.
Chapter 00 · Circuits and automata → -
1962
Perceptron convergence
Albert Novikoff
If the data are separable with margin , the perceptron makes at most mistakes.
Chapter 08 · Neural networks → -
1964
The heavy ball
Boris Polyak
The momentum method speeds up gradient descent in elongated valleys.
Chapter 07 · Optimization → -
1965
Resolution and unification
John Alan Robinson
A single inference rule, made for machines, is enough to prove any consequence in first-order logic.
Chapter 02 · Logic → -
1966
ELIZA
Joseph Weizenbaum
A program of simple rules that imitates a psychotherapist. Many users credit it with real understanding, to its author's alarm.
-
1969
Perceptrons
Marvin Minsky, Seymour Papert
A rigorous analysis of the limitations of the single-layer perceptron, with XOR as the famous example.
Chapter 08 · Neural networks → -
1970
Reverse-mode automatic differentiation
Seppo Linnainmaa
A Finnish master's thesis describes the algorithm we now call backpropagation.
Chapter 06 · Calculus → -
1971
The VC dimension
Vladimir Vapnik, Alexey Chervonenkis
A combinatorial measure of the capacity of a class of models that determines when generalization is possible.
Chapter 09 · Learning theory → -
1972
Prolog
Alain Colmerauer, Philippe Roussel, Robert Kowalski
Programming in logic: a program is a set of Horn clauses, and running it is searching for a proof.
Chapter 02 · Logic → -
1973
The Lighthill report
James Lighthill
A highly critical report for the British government triggers cuts in AI research funding.
1974 – 1979
The first winter
The promises are not kept, funding is cut and neural networks fall out of favour after Minsky and Papert's book. Quietly, backpropagation is invented.
-
1974
Backpropagation for neural networks
Paul Werbos
Werbos proposes in his thesis training multi-layer networks with reverse-mode differentiation. It goes almost unnoticed.
Chapter 06 · Calculus → -
1976
MYCIN
Edward Shortliffe
An expert system with some 600 rules recommends antibiotics for blood infections as well as specialists do.
Chapter 02 · Logic →
1980 – 1986
Expert systems and connectionism
Companies invest in systems built on hand-written rules. In parallel, neural networks are reborn with backpropagation.
-
1980
The Neocognitron
Kunihiko Fukushima
A hierarchical network inspired by the visual cortex: a forerunner of convolutional networks.
Chapter 08 · Neural networks → -
1982
Hopfield networks
John Hopfield
A recurrent network that works as an associative memory, analysed with tools from statistical physics.
Chapter 08 · Neural networks → -
1982
The Fifth Generation project
Japan's MITI
Ten years of public money to build parallel machines for logic programming. It falls short of its goals.
Chapter 02 · Logic → -
1983
The accelerated method
Yurii Nesterov
A first-order method with error , the best possible for smooth convex functions.
Chapter 07 · Optimization → -
1984
PAC learning
Leslie Valiant
“A Theory of the Learnable” gives a formal definition of learning: probably approximately correct.
Chapter 09 · Learning theory → -
1984
The Johnson–Lindenstrauss lemma
William B. Johnson, Joram Lindenstrauss
Many points fit in few dimensions without distorting their distances: the geometry that embeddings exploit.
Chapter 05 · Linear algebra → -
1986
Backpropagation goes mainstream
David Rumelhart, Geoffrey Hinton, Ronald Williams
A paper in Nature shows that multi-layer networks learn useful internal representations.
Chapter 06 · Calculus →
1987 – 1992
The second winter
The Lisp machine market collapses and expert systems prove expensive and brittle. Theory, however, moves forward: universal approximation, vanishing gradients, support vectors.
-
1987
The Lisp machine market collapses
General-purpose computers catch up with machines specialized for AI and drag the expert-systems industry down with them.
-
1989
Universal approximation
George Cybenko; Kurt Hornik, Maxwell Stinchcombe, Halbert White
A single hidden layer is enough to approximate any continuous function.
Chapter 08 · Neural networks → -
1989
Convolutional networks read ZIP codes
Yann LeCun and colleagues
A network trained with backpropagation reads handwritten digits for the US Postal Service.
Chapter 08 · Neural networks → -
1990
Simple recurrent networks
Jeffrey Elman
A network with memory of its own state discovers structure in sequences of words.
Chapter 10 · Transformers → -
1991
The vanishing gradient
Sepp Hochreiter
A diploma thesis explains why deep and recurrent networks fail to learn long-range dependencies.
Chapter 06 · Calculus → -
1992
Support vector machines
Bernhard Boser, Isabelle Guyon, Vladimir Vapnik
Maximum-margin classifiers with theoretical guarantees: the star method of the following decade.
Chapter 09 · Learning theory → -
1992
TD-Gammon
Gerald Tesauro
A neural network trained by reinforcement, playing against itself, reaches the level of the best backgammon players.
Chapter 11 · Language models →
1993 – 2011
Statistical learning
AI becomes probabilistic and data-driven. Methods with theoretical guarantees dominate, while neural networks wait for their moment: data and compute.
-
1994
Byte Pair Encoding
Philip Gage
A compression algorithm that, twenty years later, will be used to cut text into tokens.
Chapter 11 · Language models → -
1996
No free lunch
David Wolpert
No learning algorithm beats another when averaged over all problems: an inductive bias is needed.
Chapter 09 · Learning theory → -
1997
LSTM
Sepp Hochreiter, Jürgen Schmidhuber
The gates of long short-term memory let recurrent networks remember for hundreds of steps.
Chapter 10 · Transformers → -
1997
Deep Blue beats Kasparov
IBM
Massive search and hand-designed evaluation, with no deep learning: the other tradition of AI.
-
2003
A neural language model
Yoshua Bengio and colleagues
Word embeddings learned together with a network that predicts the next word.
Chapter 05 · Linear algebra → -
2006
Deep belief networks
Geoffrey Hinton, Simon Osindero, Yee-Whye Teh
Layer-by-layer pretraining revives interest in deep networks.
Chapter 08 · Neural networks → -
2009
ImageNet
Fei-Fei Li and colleagues
A database of millions of labelled images, and a yearly competition that will measure progress in vision.
Chapter 08 · Neural networks →
2012 – 2016
Deep learning
GPUs, large datasets and a few technical tricks: deep networks sweep vision, speech and games.
-
2012
AlexNet
Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton
A convolutional network trained on GPUs wins ImageNet with a top-5 error of 15.3%, ten points below the runner-up.
Chapter 08 · Neural networks → -
2013
word2vec
Tomas Mikolov and colleagues
Embeddings trained at scale in which semantic relations become directions: king − man + woman ≈ queen.
Chapter 05 · Linear algebra → -
2014
Attention for translation
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio
The decoder learns to focus on the relevant words of the input sentence.
Chapter 10 · Transformers → -
2014
Adam
Diederik Kingma, Jimmy Ba
An optimizer with momentum and an adaptive learning rate for each parameter.
Chapter 07 · Optimization → -
2015
Residual networks
Kaiming He and colleagues
Residual connections make it possible to train networks of over a hundred layers without the gradient vanishing.
Chapter 06 · Calculus → -
2016
AlphaGo beats Lee Sedol
DeepMind
Deep networks, search and reinforcement learning defeat the champion at the game of Go, a decade earlier than expected.
Chapter 11 · Language models →
2017 – today
Transformers and language models
Attention replaces recurrence, scaling laws guide investment, and next-word prediction becomes general-purpose assistants.
-
2017
“Attention Is All You Need”
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, Illia Polosukhin
The Transformer drops recurrence: only attention, in parallel.
Chapter 10 · Transformers → -
2017
Reinforcement from human preferences
Paul Christiano and colleagues
A reward model trained on human comparisons guides reinforcement learning.
Chapter 11 · Language models → -
2017
Rethinking generalization
Chiyuan Zhang and colleagues
Networks memorize random labels perfectly: classical bounds do not explain their success.
Chapter 09 · Learning theory → -
2018
GPT and BERT
OpenAI; Google
Pretraining a Transformer on unlabelled text and then fine-tuning it breaks records on almost every language task.
Chapter 10 · Transformers → -
2019
GPT-2 and double descent
OpenAI; Mikhail Belkin and colleagues
A 1.5-billion-parameter model writes surprisingly coherent text; in parallel, overparameterized models are shown to improve again.
Chapter 09 · Learning theory → -
2020
Scaling laws and GPT-3
Jared Kaplan and colleagues; Tom Brown and colleagues
Loss falls as a power law. GPT-3, with 175 billion parameters, learns new tasks from a few examples in the prompt itself.
Chapter 11 · Language models → -
2021
LoRA
Edward Hu and colleagues
Adapting a huge model by training only low-rank corrections.
Chapter 05 · Linear algebra → -
2022
Chinchilla
Jordan Hoffmann and colleagues
With fixed compute, parameters and data should grow together: about 20 tokens per parameter.
Chapter 11 · Language models → -
2022
ChatGPT
OpenAI
On 30 November a model fine-tuned with RLHF opens to the public. A million users in five days.
Chapter 11 · Language models → -
2023
Direct preference optimization
Rafael Rafailov and colleagues
The exact solution of the KL-regularized objective makes it possible to align models without explicit reinforcement learning.
Chapter 11 · Language models → -
2024
Nobel prizes for neural networks
John Hopfield, Geoffrey Hinton; Demis Hassabis, John Jumper, David Baker
The Nobel Prize in Physics rewards the foundations of artificial neural networks, and the Chemistry prize protein structure prediction with AlphaFold (together with protein design).
Chapter 08 · Neural networks → -
2024
Calibrated models hallucinate
Adam Tauman Kalai, Santosh Vempala
A statistical lower bound on the hallucination rate of calibrated language models.
Chapter 11 · Language models → -
2024
The 5-state busy beaver is settled
The bbchallenge collaboration
A formally verified proof establishes that the 5-state machine that takes longest to halt runs for 47,176,870 steps.
Chapter 01 · Computation → -
2024
Reasoning models
OpenAI, DeepSeek and others
Models trained with reinforcement on verifiable problems that “think” before answering, trading compute at answer time for accuracy.
Chapter 11 · Language models →