This research paper describes the creation and analysis of GPQA, a new set of multiple-choice questions designed to be very hard to answer, even with the help of Google. The questions cover advanced topics in biology, physics, and chemistry, and were written and checked for accuracy by experts with PhDs in those fields. The researchers made sure the questions were extra tough by having other experts, called non-experts, try to answer them using the internet. These non-experts also had PhDs, but in different subjects. The goal was to create questions that would be challenging even for very smart people who don't have specific knowledge in the subject. The researchers also tested the questions on advanced AI systems, like GPT-4, to see how well they could answer them. They found that even with access to the interne ...