GPQA: questions you cannot google
On 20 November 2023 David Rein and colleagues at New York University released GPQA, 448 multiple-choice questions in biology, physics and chemistry written by specialists. Experts with a PhD in the field answered 65% correctly, skilled non-experts with over 30 minutes of web access 34%, and the best GPT-4 baseline 39%.
Why it matters
The set was designed for research on scalable oversight: how people can check the answers of systems that know more than they do. The questions resist web search, so they test knowledge and reasoning rather than finding the answer.
Experts reach 74% once the mistakes they themselves later admitted are set aside. Besides the main set the authors released an extended one (546 questions) and the strictest, the diamond set (198), on which reasoning models later reported. The questions were written and checked by 61 contractors hired through Upwork; the authors estimate an average pay of about $95 an hour and a maximum of $150, most of it bonuses for quality.