Atlas record: https://www.aitimeline.dev/en/events/evt-0011/
Research · June 12, 2017
The Transformer used attention without recurrent layers.
It replaced recurrence with attention and made sequence computation more parallelizable.
The paper reports translation and parsing experiments, not a universal capability claim.
arXiv · Published June 12, 2017
Advances in Neural Information Processing Systems 30 (NIPS 2017), NeurIPS proceedings · Published 2017
Transformer develops attention for selecting relevant sequence information while dispensing with recurrence. The paper cites Bahdanau and colleagues.
The model is a transformer decoder.
BERT uses a bidirectional Transformer encoder, explicitly described in the paper.
The library gathered under one interface exactly the architectures that grew out of that paper.
The GPT-2 technical description identifies Transformer-based language modeling.
NeRF encodes coordinates with sinusoids and notes that a similar mapping in the Transformer is called positional encoding, though there it serves another purpose.
The transformer is applied to images cut into patches.
The attention mechanism is applied to pairs of amino acid residues.
CLIP uses a Transformer text encoder, documented in the technical paper included alongside its announcement.
RoPE changes how the Transformer encodes position: instead of a vector added to the input, position enters self-attention itself through a rotation matrix.
The hardware unit is designed for the attention mechanism.
FlashAttention computes the same exact Transformer attention as the standard implementation, changing only the order of reads and writes to GPU memory.
It manages the key-value cache of exactly the attention that paper described.
The first serious alternative to attention without quadratic cost in length.
The quadratic cost of attention was the main obstacle to long context.
The theoretical outstanding paper measures the power of this architecture by how succinctly it encodes concepts against recurrent networks.
One of the three best papers characterises the expressivity of local attention in this architecture.