Research · September 1, 2014
An attention mechanism weighted parts of an input sequence.
It exposed the fixed-vector bottleneck and added soft alignment over source words.
It is an encoder-decoder translation result, not the later Transformer architecture.
arXiv · Published September 1, 2014
Two papers from the same month on neural sequence translation.
Transformer develops attention for selecting relevant sequence information while dispensing with recurrence. The paper cites Bahdanau and colleagues.
Attention Is All You NeedEnd-to-end sequence learning is taken from machine translation.
The attention paper for translation (v1, September 2014) uses as the activation of its recurrent network the gated hidden unit recently proposed by Cho et al. (2014a), with reset and update gates.
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine TranslationNeural Machine TranslationThe GNMT decoder reaches the encoder through attention, citing Bahdanau, Cho and Bengio.
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (arXiv:1609.08144v1)Tacotron's backbone is a seq2seq model with attention after Bahdanau et al.: text characters are translated into spectrogram frames.
Tacotron: A Fully End-to-End Text-to-Speech Synthesis Model (arXiv:1703.10135v1)