BERT
BERT introduced bidirectional pre-training.
Why it matters
It established a reusable bidirectional pre-training recipe for language tasks.
Its reported benchmarks depend on the model and evaluation tasks described in the paper.
Atlas record: https://www.aitimeline.dev/en/events/evt-0012/
Research · October 11, 2018
Event record
- Event date
- October 11, 2018
- Timeline date
- Event date
- Verification
- Sources gathered automatically · September 11, 2026
- Lines
- ID
- evt-0012
Sources
- primary BERT
arXiv · Published October 11, 2018
- primary BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
ACL Anthology (NAACL-HLT 2019) · Published 2019
Related events
Earlier
- Contrasts GPT-1 2018links to this record
A different bet on the same mechanism: generation instead of bidirectional encoding.
- Related GLUE: nine tasks, one score, closed tests 2018links to this record
The aggregate score over nine tasks was the number pretrained models were measured by half a year later. No source in this record asserts causation.
- Builds on Transformer 2017
BERT uses a bidirectional Transformer encoder, explicitly described in the paper.
- BERT · arXiv
Later
- Builds on ERNIE 2.0: pre-training that adds tasks 2019links to this record
The model takes the same transformer settings as BERT and is measured against it task by task.
- Related wav2vec 2.0 2020links to this record
Masked self-supervision is taken from the approach shown on text.