Research · March 2022

InstructGPT and learning from human ratings

OpenAI fine-tuned a model to carry out requests by training a separate reward model on human comparisons of answers.

Why it matters

A 1.3-billion-parameter model became more useful than a 175-billion one: alignment mattered more than size.

The scheme has three steps: fine-tuning on examples of desired behaviour, training a reward model on human rankings of answers, then optimising the policy against that reward. The result was not about the quality of the language but about whether the model does what it is asked. Eight months later the same scheme turned GPT-3.5 into ChatGPT, and it is this, not size, that made the model broadly usable.

Event record

Event date

March 2022

Timeline date

Event date

Verification

Sources gathered automatically · September 17, 2026

Lines

ID

evt-0243

Preprint of 4 March 2022; presented at NeurIPS 2022.

Sources

primary Training language models to follow instructions with human feedback
arXiv · Published March 2022

Related events

Extends GPT-3
The same model, trained to carry out a request rather than continue text.
Training language models to follow instructions with human feedback
Enables ChatGPT
The method ChatGPT was built on eight months later.
Training language models to follow instructions with human feedback
Related Reinforcement Learning: An Introduction
The fine-tuning is reinforcement learning against a reward derived from human ratings.

Records that link to this one

Related Alpaca
The same alignment result, obtained without human raters.
Related Claude
The alignment is built on the same line of learning from human preferences.
Contrasts Direct preference optimisation
The same alignment result without a separate reward model and without reinforcement learning.
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Related o1 and compute at answering time
Reinforcement learning applied not to rater preference but to the correctness of a solution.
Contrasts A model that writes no text: TypeSafe releases Jev
TypeSafe's founder helped create the method that record describes; the company defines Jev's training method straight against it, calibrated decisions in place of human preference.
A new kind of AI model from a ChatGPT inventor is thrilling developers Introducing System One Models & Jev
Enables Learning from human preferences
InstructGPT fine-tunes GPT-3 with reinforcement learning from human feedback, citing 'RLHF; Christiano et al., 2017'; Christiano is among the InstructGPT authors.
Training language models to follow instructions with human feedback
Enables Proximal policy optimisation (PPO)
InstructGPT v1 optimises its policy against the reward model 'using the PPO algorithm (Schulman et al., 2017)', and the models themselves are called PPO and PPO-ptx.
Training language models to follow instructions with human feedback
Builds on Constitutional AI: harmlessness from AI feedback
InstructGPT is named among the works in which learning from human feedback typically uses at least tens of thousands of human labels; this work replaces the harmlessness labels with the model's own judgements.
Constitutional AI: Harmlessness from AI Feedback (arXiv:2212.08073v1)
Contrasts OpenAI assembles a superalignment team
The post names reinforcement learning from human feedback as a current alignment technique that will not scale to systems smarter than humans, because it relies on human supervision.
Introducing Superalignment (archived copy)
Related The superalignment leads leave OpenAI
Among his team's achievements Leike names InstructGPT first, the first language model trained with human feedback.
Jan Leike on X: last day as head of alignment (thread)