Atlas record: https://www.aitimeline.dev/en/events/evt-0009/
Research · September 1, 2014
An attention mechanism weighted parts of an input sequence.
It exposed the fixed-vector bottleneck and added soft alignment over source words.
It is an encoder-decoder translation result, not the later Transformer architecture.
arXiv · Published September 1, 2014
International Conference on Learning Representations 2015 (conference archive, iclr.cc) · Published May 2015
The attention paper for translation (v1, September 2014) uses as the activation of its recurrent network the gated hidden unit recently proposed by Cho et al. (2014a), with reset and update gates.
Two papers from the same month on neural sequence translation.
End-to-end sequence learning is taken from machine translation.
The GNMT decoder reaches the encoder through attention, citing Bahdanau, Cho and Bengio.
Tacotron's backbone is a seq2seq model with attention after Bahdanau et al.: text characters are translated into spectrogram frames.
Transformer develops attention for selecting relevant sequence information while dispensing with recurrence. The paper cites Bahdanau and colleagues.