GPT-1
OpenAI showed that a transformer pre-trained to predict the next word on unannotated text beats specialised models after fine-tuning.
Why it matters
The scheme of training on all text then fine-tuning on a task worked for language for the first time, and became the only scheme for the next decade.
The model is a 12-layer decoder-only transformer (768-dimensional states, 12 attention heads) trained on the BooksCorpus (over 7,000 unpublished books). The paper gives no parameter count: the figure of 117 million is from Table 2 of the GPT-2 paper (2019), which lists a 12-layer, 768-wide model of 117M and calls its smallest model equivalent to the original GPT. What matters is not the architecture but the order of operations: next-word prediction on an enormous unannotated corpus first, then brief fine-tuning for a task. Four months later BERT showed the same for bidirectional encoding, and the argument over which scheme was better ran until GPT-3.