Atlas record: https://www.aitimeline.dev/en/events/evt-0205/
Research · February 2003
A neural probabilistic language model
Bengio and co-authors replaced n-gram tables with a network that learns a vector for each word and predicts the next word from those vectors.
Why it matters
Words became points in a space where nearness means similar meaning, the basis of all later language modelling.
The problem with n-grams is sparsity: most word combinations never occur in a corpus at all. A distributed representation solves it by generalisation, so a model that has seen a cat running gives a sensible probability to a dog running. Computationally the model was too expensive for its time. Word2vec in 2013 and ELMo are direct descendants of the idea.
Event record
- Event date
- February 2003
- Timeline date
- Event date
- Verification
- Sources gathered automatically · September 17, 2026
- Lines
- ID
- evt-0205
The February 2003 issue of the Journal of Machine Learning Research; an earlier version was presented at NIPS 2000.
Related events
Earlier
- Enables Chen and Goodman's comparison of smoothing 1998links to this record
Bengio's neural language model is compared with n-grams using modified Kneser-Ney smoothing, citing Kneser and Ney 1995 and the 1999 journal version of Chen and Goodman's work.
- Enables Word classes from mutual information 1992links to this record
Bengio and colleagues start from the word clustering of Brown et al., replace a word's discrete class with a continuous vector, and compare their model with class-based n-grams.
- Enables Latent semantic analysis 1990links to this record
Bengio and colleagues name Deerwester et al.'s latent semantic indexing as where feature vectors for words were already learned from co-occurrence in documents, and tried principal components of co-occurrence as fixed word features.
- Builds on Back-propagation 1986
The model is trained by back-propagation.
- Related Statistical speech recognition 1976
Instead of counting n-grams, a network with distributed representations estimates word probability.
Later