Scaling laws

The study measured loss relative to scale.

Why it matters

It made scale a measurable design variable for language-model training.

The relationships reported are empirical fits over the studied training range.

Event date

January 23, 2020

Timeline date

Event date

Verification

Sources gathered automatically · September 11, 2026

Lines

ID

evt-0014

Related Microsoft's billion for OpenAI 2019links to this record
Computation on that scale made the scaling experiments possible.
Related GPT-2 2019
The study measured models of the same architectural family.

Builds on GPT-3 2020links to this record
The model size was chosen from the scaling laws published the same year.
- Language Models are Few-Shot Learners · arXiv
Related Three eras of compute: 21, 6 and 10 months 2022links to this record
The scaling law says how much compute is worth spending; this series says how much was actually spent and how fast that grew.
- Compute Trends Across Three Eras of Machine Learning (arXiv:2202.05924v1) · arXiv
Contrasts Chinchilla: compute-optimal training 2022links to this record
Kaplan and co-authors advised, for a tenfold budget, a 5.5-fold larger model on only 1.8 times more tokens; this paper finds that model size and training tokens should be scaled in equal proportions.
- Training Compute-Optimal Large Language Models · arXiv
Extends BIG-bench: 204 tasks assembled by 442 authors 2022links to this record
The scaling laws described quality rising smoothly with compute; this set looked for tasks where it does not, and named brittle metrics as one cause of the jumps.
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models · arXiv
Related o1 and compute at answering time 2024links to this record
Quality grows with compute spent at answering time, not only at training time.
Related Jalapeño chip 2026links to this record
Both records concern computing resources. Training scaling laws do not establish the lineage of this inference chip.