GPT-3

A model with 175 billion parameters performed tasks from a few examples in the prompt, with no weight updates at all.

Why it matters

A capability nobody programmed appeared: the model learns from the statement of the task rather than from a gradient step.

A hundredfold jump over GPT-2 produced not merely better quality but qualitatively different behaviour. In-context learning meant a new task needed neither data nor retraining, only a description in the prompt. That changed how a model is used: the prompt replaced fine-tuning. Together with the scaling laws of the same year, GPT-3 fixed the direction toward size.

Event record

Event date

May 2020

Timeline date

Event date

Verification

Sources gathered automatically · September 17, 2026

Lines

Learning theory

ID

evt-0238

Preprint of 28 May 2020; presented at NeurIPS 2020.

Sources

primary Language Models are Few-Shot Learners
arXiv · Published May 28, 2020

Related events

Earlier

Builds on Scaling laws 2020
The model size was chosen from the scaling laws published the same year.
- Language Models are Few-Shot Learners · arXiv
Extends GPT-2 2019
The same architecture, scaled up a hundredfold.
- Language Models are Few-Shot Learners · arXiv
Enables A crawl of the web becomes available to anyone 2012links to this record
In Table 2.2 of the GPT-3 paper, filtered Common Crawl supplies 410 billion of the 499 billion tokens in the assembled dataset.
- Language Models are Few-Shot Learners · arXiv

Same period

Related NVIDIA A100 2020links to this record
The hardware the GPT-3 generation of models was trained on.

Later

Related MMLU: 57 subjects the model did not study 2020links to this record
The largest model of the day scored 43.9 per cent on this test against 25 per cent for random guessing, and that gap was the reason for publishing the set.
- Measuring Massive Multitask Language Understanding · arXiv
Contrasts Stochastic Parrots: can a language model be too big 2021links to this record
The paper names that model among those whose scaling it questions, and asks whether enough thought has gone into the risks of building them.
- On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? · ACM
Related GPT-3 in megawatt-hours 2021links to this record
The model announced a year earlier is named here for the first time in megawatt-hours and tonnes.
- Carbon Emissions and Large Neural Network Training (arXiv:2104.10350v1) · arXiv
Related Low-rank adaptation (LoRA) 2021links to this record
The paper's main case is GPT-3 175B: deploying many independently fine-tuned instances of it, each with 175B parameters, is what the authors call extremely expensive.
- LoRA: Low-Rank Adaptation of Large Language Models · arXiv
Extends Codex and Copilot 2021links to this record
The same model, fine-tuned on public code.
- OpenAI Codex · OpenAI
Related Chain-of-thought prompting 2022links to this record
The first version also tests the method on GPT-3 davinci, whose results it calls comparable with those of the 137-billion-parameter model.
- Chain of Thought Prompting Elicits Reasoning in Large Language Models · arXiv
Extends InstructGPT and learning from human ratings 2022links to this record
The same model, trained to carry out a request rather than continue text.
- Training language models to follow instructions with human feedback · arXiv
Enables ChatGPT 2022
The model ChatGPT was built on two years later.
- Language Models are Few-Shot Learners · arXiv
Related LLaMA 2023links to this record
A 13-billion-parameter model beats the 175-billion GPT-3 on most measures.
Related Segment Anything 2023links to this record
The same move as in language: one promptable model instead of a model per task.