Finding Structure in Time
Elman gave a network a context layer holding its own previous state, and showed that it then learns the structure of a sequence.
Why it matters
Time stopped being an extra input and became internal state, the form recurrent networks still work in.
The network predicted the next word in a sentence. Analysis of the hidden layer showed it had grouped words into nouns, verbs, and animate against inanimate by itself, though nobody had supplied categories or labels. The scheme with recurrent links was first described by Jordan in 1986, feeding back the network's output; Elman feeds back the hidden layer. On limits Elman is cautious: memory is bounded because repeated application of the activation function gives exponential decay, and he calls memory capacity an open question; Hochreiter explained it formally the next year.