BERT
BERT introduced bidirectional pre-training.
Why it matters
It established a reusable bidirectional pre-training recipe for language tasks.
Its reported benchmarks depend on the model and evaluation tasks described in the paper.
Research · October 11, 2018
Event record
- Event date
- October 11, 2018
- Timeline date
- Event date
- Verification
- Sources gathered automatically · September 11, 2026
- Lines
- ID
- evt-0012
Sources
- primary BERT
arXiv · Published October 11, 2018
Related events
- Builds on Transformer
BERT uses a bidirectional Transformer encoder, explicitly described in the paper.
BERT
Records that link to this one
- Contrasts GPT-1
A different bet on the same mechanism: generation instead of bidirectional encoding.
Improving Language Understanding by Generative Pre-Training - Related wav2vec 2.0
Masked self-supervision is taken from the approach shown on text.
- Related GLUE: nine tasks, one score, closed tests
The aggregate score over nine tasks was the number pretrained models were measured by half a year later. No source in this record asserts causation.
- Builds on ERNIE 2.0: pre-training that adds tasks
The model takes the same transformer settings as BERT and is measured against it task by task.
ERNIE 2.0: A Continual Pre-training Framework for Language Understanding (arXiv:1907.12412v1)