Scaling laws
The study measured loss relative to scale.
Why it matters
It made scale a measurable design variable for language-model training.
The relationships reported are empirical fits over the studied training range.
Research · January 23, 2020
Event record
- Event date
- January 23, 2020
- Timeline date
- Event date
- Verification
- Sources gathered automatically · September 11, 2026
- Lines
- ID
- evt-0014
Sources
- primary Scaling Laws
arXiv · Published January 23, 2020
Related events
- Related GPT-2
The study measured models of the same architectural family.
Records that link to this one
- Related Jalapeño chip
Both records concern computing resources. Training scaling laws do not establish the lineage of this inference chip.
- Related Microsoft's billion for OpenAI
Computation on that scale made the scaling experiments possible.
- Builds on GPT-3
The model size was chosen from the scaling laws published the same year.
Language Models are Few-Shot Learners - Related o1 and compute at answering time
Quality grows with compute spent at answering time, not only at training time.
- Extends BIG-bench: 204 tasks assembled by 442 authors
The scaling laws described quality rising smoothly with compute; this set looked for tasks where it does not, and named brittle metrics as one cause of the jumps.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models - Related Three eras of compute: 21, 6 and 10 months
The scaling law says how much compute is worth spending; this series says how much was actually spent and how fast that grew.
Compute Trends Across Three Eras of Machine Learning (arXiv:2202.05924v1) - Contrasts Chinchilla: compute-optimal training
Kaplan and co-authors advised, for a tenfold budget, a 5.5-fold larger model on only 1.8 times more tokens; this paper finds that model size and training tokens should be scaled in equal proportions.
Training Compute-Optimal Large Language Models