Research · February 2003

A neural probabilistic language model

Bengio and co-authors replaced n-gram tables with a network that learns a vector for each word and predicts the next word from those vectors.

Why it matters

Words became points in a space where nearness means similar meaning, the basis of all later language modelling.

The problem with n-grams is sparsity: most word combinations never occur in a corpus at all. A distributed representation solves it by generalisation, so a model that has seen a cat running gives a sensible probability to a dog running. Computationally the model was too expensive for its time. Word2vec in 2013 and ELMo are direct descendants of the idea.

Event record

Event date

February 2003

Timeline date

Event date

Verification

Sources gathered automatically · September 17, 2026

Lines

Learning theory

ID

evt-0205

The February 2003 issue of the Journal of Machine Learning Research; an earlier version was presented at NIPS 2000.

Yoshua Bengio

Sources

primary A Neural Probabilistic Language Model
Journal of Machine Learning Research · Published February 2003

Related events

Builds on Back-propagation
The model is trained by back-propagation.
A Neural Probabilistic Language Model
Related Statistical speech recognition
Instead of counting n-grams, a network with distributed representations estimates word probability.

Records that link to this one

Related word2vec
The word vector idea is reduced to a model trainable on billions of words.
Enables Latent semantic analysis
Bengio and colleagues name Deerwester et al.'s latent semantic indexing as where feature vectors for words were already learned from co-occurrence in documents, and tried principal components of co-occurrence as fixed word features.
A Neural Probabilistic Language Model
Enables Word classes from mutual information
Bengio and colleagues start from the word clustering of Brown et al., replace a word's discrete class with a continuous vector, and compare their model with class-based n-grams.
A Neural Probabilistic Language Model
Enables Chen and Goodman's comparison of smoothing
Bengio's neural language model is compared with n-grams using modified Kneser-Ney smoothing, citing Kneser and Ney 1995 and the 1999 journal version of Chen and Goodman's work.
A Neural Probabilistic Language Model