Research · May 2020

GPT-3

A model with 175 billion parameters performed tasks from a few examples in the prompt, with no weight updates at all.

Why it matters

A capability nobody programmed appeared: the model learns from the statement of the task rather than from a gradient step.

A hundredfold jump over GPT-2 produced not merely better quality but qualitatively different behaviour. In-context learning meant a new task needed neither data nor retraining, only a description in the prompt. That changed how a model is used: the prompt replaced fine-tuning. Together with the scaling laws of the same year, GPT-3 fixed the direction toward size.

Event record

Event date

May 2020

Timeline date

Event date

Verification

Sources gathered automatically · September 17, 2026

Lines

Learning theory

ID

evt-0238

Preprint of 28 May 2020; presented at NeurIPS 2020.

Sources

primary Language Models are Few-Shot Learners
arXiv · Published May 2020

Related events

Extends GPT-2
The same architecture, scaled up a hundredfold.
Language Models are Few-Shot Learners
Builds on Scaling laws
The model size was chosen from the scaling laws published the same year.
Language Models are Few-Shot Learners
Enables ChatGPT
The model ChatGPT was built on two years later.
Language Models are Few-Shot Learners

Records that link to this one

Extends Codex and Copilot
The same model, fine-tuned on public code.
OpenAI Codex
Extends InstructGPT and learning from human ratings
The same model, trained to carry out a request rather than continue text.
Training language models to follow instructions with human feedback
Related NVIDIA A100
The hardware the GPT-3 generation of models was trained on.
Related LLaMA
A 13-billion-parameter model beats the 175-billion GPT-3 on most measures.
Related Segment Anything
The same move as in language: one promptable model instead of a model per task.
Related MMLU: 57 subjects the model did not study
The largest model of the day scored 43.9 per cent on this test against 25 per cent for random guessing, and that gap was the reason for publishing the set.
Measuring Massive Multitask Language Understanding
Enables A crawl of the web becomes available to anyone
In Table 2.2 of the GPT-3 paper, filtered Common Crawl supplies 410 billion of the 499 billion tokens in the assembled dataset.
Language Models are Few-Shot Learners
Contrasts Stochastic Parrots: can a language model be too big
The paper names that model among those whose scaling it questions, and asks whether enough thought has gone into the risks of building them.
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
Related GPT-3 in megawatt-hours
The model announced a year earlier is named here for the first time in megawatt-hours and tonnes.
Carbon Emissions and Large Neural Network Training (arXiv:2104.10350v1)
Related Low-rank adaptation (LoRA)
The paper's main case is GPT-3 175B: deploying many independently fine-tuned instances of it, each with 175B parameters, is what the authors call extremely expensive.
LoRA: Low-Rank Adaptation of Large Language Models
Related Chain-of-thought prompting
The first version also tests the method on GPT-3 davinci, whose results it calls comparable with those of the 137-billion-parameter model.
Chain of Thought Prompting Elicits Reasoning in Large Language Models