Research · June 12, 2017
The Transformer used attention without recurrent layers.
It replaced recurrence with attention and made sequence computation more parallelizable.
The paper reports translation and parsing experiments, not a universal capability claim.
arXiv · Published June 12, 2017
Transformer develops attention for selecting relevant sequence information while dispensing with recurrence. The paper cites Bahdanau and colleagues.
Attention Is All You NeedBERT uses a bidirectional Transformer encoder, explicitly described in the paper.
BERTThe GPT-2 technical description identifies Transformer-based language modeling.
GPT-2CLIP uses a Transformer text encoder, documented in the technical paper included alongside its announcement.
CLIPThe model is a transformer decoder.
Improving Language Understanding by Generative Pre-TrainingThe attention mechanism is applied to pairs of amino acid residues.
AlphaFold: a solution to a 50-year-old grand challenge in biologyThe transformer is applied to images cut into patches.
An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleThe hardware unit is designed for the attention mechanism.
The first serious alternative to attention without quadratic cost in length.
Mamba: Linear-Time Sequence Modeling with Selective State SpacesThe quadratic cost of attention was the main obstacle to long context.
The theoretical outstanding paper measures the power of this architecture by how succinctly it encodes concepts against recurrent networks.
Announcing the ICLR 2026 Outstanding PapersOne of the three best papers characterises the expressivity of local attention in this architecture.
Best Paper Awards - ACL 2026The library gathered under one interface exactly the architectures that grew out of that paper.
HuggingFace's Transformers: State-of-the-art Natural Language ProcessingIt manages the key-value cache of exactly the attention that paper described.
Efficient Memory Management for Large Language Model Serving with PagedAttentionRoPE changes how the Transformer encodes position: instead of a vector added to the input, position enters self-attention itself through a rotation matrix.
RoFormer: Enhanced Transformer with Rotary Position EmbeddingFlashAttention computes the same exact Transformer attention as the standard implementation, changing only the order of reads and writes to GPU memory.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessNeRF encodes coordinates with sinusoids and notes that a similar mapping in the Transformer is called positional encoding, though there it serves another purpose.
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (arXiv:2003.08934v1)