Atlas record: https://www.aitimeline.dev/en/events/evt-0243/
Research · March 2022
InstructGPT and learning from human ratings
OpenAI fine-tuned a model to carry out requests by training a separate reward model on human comparisons of answers.
Why it matters
A 1.3-billion-parameter model became more useful than a 175-billion one: alignment mattered more than size.
The scheme has three steps: fine-tuning on examples of desired behaviour, training a reward model on human rankings of answers, then optimising the policy against that reward. The result was not about the quality of the language but about whether the model does what it is asked. Eight months later the same scheme turned GPT-3.5 into ChatGPT, and it is this, not size, that made the model broadly usable.
Event record
- Event date
- March 2022
- Timeline date
- Event date
- Verification
- Sources gathered automatically · September 17, 2026
- Lines
- ID
- evt-0243
Preprint of 4 March 2022; presented at NeurIPS 2022. OpenAI announced the InstructGPT models on its blog on 27 January 2022.
Related events
Earlier
- Extends GPT-3 2020
The same model, trained to carry out a request rather than continue text.
- Enables Proximal policy optimisation (PPO) 2017links to this record
InstructGPT v1 optimises its policy against the reward model 'using the PPO algorithm (Schulman et al., 2017)', and the models themselves are called PPO and PPO-ptx.
- Enables Learning from human preferences 2017links to this record
InstructGPT fine-tunes GPT-3 with reinforcement learning from human feedback, citing 'RLHF; Christiano et al., 2017'; Christiano is among the InstructGPT authors.
- Related Reinforcement Learning: An Introduction 1998
The fine-tuning is reinforcement learning against a reward derived from human ratings.
Later
- Enables ChatGPT 2022
The method ChatGPT was built on eight months later.
- Builds on Constitutional AI: harmlessness from AI feedback 2022links to this record
InstructGPT is named among the works in which learning from human feedback typically uses at least tens of thousands of human labels; this work replaces the harmlessness labels with the model's own judgements.
- Related Alpaca 2023links to this record
The same alignment result, obtained without human raters.
- Related Claude 2023links to this record
The alignment is built on the same line of learning from human preferences.
- Contrasts Direct preference optimisation 2023links to this record
The same alignment result without a separate reward model and without reinforcement learning.
- Contrasts OpenAI assembles a superalignment team 2023links to this record
The post names reinforcement learning from human feedback as a current alignment technique that will not scale to systems smarter than humans, because it relies on human supervision.
- Related The superalignment leads leave OpenAI 2024links to this record
Among his team's achievements Leike names InstructGPT first, the first language model trained with human feedback.
- Related o1 and compute at answering time 2024links to this record
Reinforcement learning applied not to rater preference but to the correctness of a solution.
- Contrasts A model that writes no text: TypeSafe releases Jev 2026links to this record
TypeSafe's founder helped create the method that record describes; the company defines Jev's training method straight against it, calibrated decisions in place of human preference.