Temporal-difference learning

Sutton showed that a prediction can be corrected against the next prediction rather than the final outcome, and proved it works.

Why it matters

An agent stopped waiting for the end of an episode: it learns from every step instead of from the finish.

The paper formalises the TD(lambda) family, where lambda governs how far back a correction propagates. Sutton proves convergence for the linear case and shows TD beats Monte Carlo on problems with Markov structure. This is the core TD-Gammon was built on four years later, and deep reinforcement learning thirty years later.

Event record

Event date

August 1988

Timeline date

Event date

Verification

Sources gathered automatically · September 17, 2026

Lines

Learning theory

ID

evt-0157

Machine Learning volume 3, August 1988.

Richard Sutton

Sources

primary Learning to Predict by the Methods of Temporal Differences
Machine Learning · Published August 1988
archive Learning to Predict by the Methods of Temporal Differences (full text)
Richard Sutton · Published August 1988

Related events

Earlier

Extends Actor and critic 1983
It formalises the critic described in the actor-critic scheme.
- Learning to Predict by the Methods of Temporal Differences · Machine Learning
Builds on Some Studies in Machine Learning Using the Game of Checkers 1959
It traces its rule directly to what Samuel's checkers program did.
- Learning to Predict by the Methods of Temporal Differences · Machine Learning
Builds on Dynamic Programming 1957
The method rests on Bellman's decomposition for sequential decisions.
- Learning to Predict by the Methods of Temporal Differences · Machine Learning

Later

Builds on Q-learning 1989links to this record
The method develops temporal-difference learning to the case of choosing actions.
- Learning from Delayed Rewards · University of Cambridge
Builds on TD-Gammon 1992links to this record
The program learns by temporal-difference methods precisely.
- Practical Issues in Temporal Difference Learning · Machine Learning
Extends Reinforcement Learning: An Introduction 1998links to this record
The book systematises the methods the temporal-difference paper began.
- Reinforcement Learning: An Introduction · MIT Press