GPQA
Abstract. GPQA, short for Graduate-Level Google-Proof Question Answering, is a compact but unusually deep benchmark family for testing whether AI systems can answer expert-written graduate-level science questions in biology, chemistry, and physics. Its central methodological claim is not that the questions are literally impossible to search, but that skilled non-experts with unrestricted web access remain near chance while domain experts do substantially better. This article covers the broader GPQA family — Extended, Main, Diamond, and the non-canonical “Reduced” label sometimes seen downstream — rather than duplicating a focused gpqa-diamond.md scorecard.
Coverage note: verified through May 11, 2026.
What GPQA is
GPQA is a multiple-choice question-answering benchmark introduced by Rein et al. in the paper “GPQA: A Graduate-Level Google-Proof Q&A Benchmark.” The dataset contains graduate-level questions written by domain experts in biology, physics, and chemistry, with the explicit goal of creating questions that are answerable by experts but resistant to solution by non-experts using ordinary web search. The original paper reports 448 questions in the recommended Main set, 546 questions in the Extended set, and 198 questions in the Diamond subset. arXiv
The benchmark is best understood as a Scalable Oversight probe: can models answer questions that non-expert human evaluators cannot reliably judge or solve, even with search? This makes GPQA adjacent to evaluation ideas such as Sandwiching Evaluations, Expert Validation, and AI Capability Benchmarks, rather than a general-purpose “science trivia” test. Rein et al. explicitly situate GPQA in the problem of evaluating systems whose outputs may exceed the competence of ordinary annotators. arXiv
The phrase Google-proof is operational, not metaphysical. GPQA questions are not guaranteed to be impossible to solve with arbitrary search, textbooks, expert consultation, symbolic tools, or enough time. Instead, the original validation procedure showed that non-expert PhD-level validators using unrestricted internet access averaged about 34% accuracy on the Extended set, only modestly above the 25% random baseline for four-choice questions, while expert validators performed much better. arXiv
Why GPQA exists
Many language-model benchmarks are vulnerable to three related failures. First, they may test skills below the frontier, causing fast saturation. Second, they may use questions whose answers can be recovered by shallow search or memorization. Third, they may depend on labels produced by annotators who are not competent to judge expert-level outputs.
GPQA was designed as a countermeasure to those failures. Its questions are narrow, technical, and often require graduate-level conceptual integration. The original study paid PhD-level contributors to write and validate questions, asked non-expert PhDs to attempt them with internet access, and filtered the released sets according to expert correctness and non-expert failure. ar5iv
That design makes GPQA useful for a particular kind of claim: whether a system can solve compact, expert-authored scientific reasoning questions that are hard for non-experts to validate by web search. It is much less useful for claims about broad scientific research ability, experimental design, theorem proving, literature synthesis, or production engineering.
Relationship to gpqa-diamond.md
This article covers GPQA as a benchmark family: its construction methodology, split semantics, validation logic, strengths, weaknesses, and how Main and Diamond should be used by practitioners. A separate gpqa-diamond.md entry should focus on GPQA Diamond specifically: its current leaderboard role, saturation behavior, residual hard items, per-item error structure, and how frontier labs report Diamond scores.
The distinction matters because GPQA Diamond is not the whole benchmark. Diamond is a highly filtered 198-question subset selected for expert agreement and non-expert failure. The broader family includes the larger Main and Extended sets, whose different selection criteria make them better suited to some comparisons and worse suited to others. arXiv
Construction methodology
GPQA’s methodology is the main reason the benchmark became important. The dataset was not produced by scraping exams or prompting crowdworkers for science questions. It was built through a multi-stage process involving expert authorship, expert validation, revision, and non-expert validation with web access.
Expert authorship
Rein et al. recruited 61 contractors through Upwork, each of whom either had or was pursuing a PhD and had English proficiency. Contributors were paid at an average rate of roughly $95 per hour. The authors asked question writers to produce difficult graduate-level questions in their area of expertise, with the important constraint that questions should be answerable by experts but difficult for non-experts even with search. ar5iv
Question writers were instructed to make questions answerable from the question alone, not from exploiting the answer choices. They also wrote answer explanations and labeled questions by domain and subdomain. This matters because multiple-choice benchmarks can otherwise become tests of distractor artifacts: a model may infer the answer from the style, granularity, or implausibility of the options rather than from domain reasoning. ar5iv
Expert validation and revision
The expert validation loop had three core stages. A first expert attempted the question and gave feedback. The original writer then revised the item. A second expert attempted the revised question. Rein et al. used this expert-agreement process to estimate whether questions were objective and correctly keyed. ar5iv
This is a stronger methodology than ordinary crowd-labeling, but it does not remove ambiguity. The paper reports that expert validators were not always correct, that some disagreements were due to mistakes by validators rather than faulty questions, and that the authors performed post-hoc analysis to distinguish demonstrable validator errors from item problems. Their conservative estimate of question objectivity was about 74%, with higher estimated quality in more selective subsets. arXiv
Non-expert validation with web access
GPQA’s most distinctive step is non-expert validation. The authors recruited PhD-level validators outside the relevant subdomain and gave them unrestricted internet access, except that they were not allowed to use language-model assistants. Validators were required to spend at least 15 minutes per question; the reported average time was about 37 minutes, with a median of about 30 minutes. ar5iv
The benchmark’s “Google-proof” claim depends on this step. Non-experts were not merely glancing at questions without help. They were allowed to search the web, read papers, and spend substantial time. Despite that, their average accuracy remained low: about 34.1% on Extended, 30.4% on Main, and 22.1% on Diamond, compared with a 25% random baseline. arXiv
Gated release and anti-contamination norms
The GPQA dataset is gated on Hugging Face, and the dataset card asks users not to reveal examples online. The official release lists gpqa_diamond.csv, gpqa_extended.csv, gpqa_main.csv, and gpqa_experts.csv. This is a soft but meaningful anti-contamination measure: the benchmark’s value depends on item secrecy, especially because the dataset is small. Hugging Face
For that reason, this article does not reproduce sample questions. Publishing examples would directly weaken the benchmark.
The GPQA subsets
The original GPQA family has three canonical public question subsets: Extended, Main, and Diamond. A fourth label, Reduced, sometimes appears in downstream contexts, but it is not a canonical Rein et al. release subset.
| Subset | Size | Canonical status | Selection logic | Intended use |
|---|---|---|---|---|
| GPQA Extended | 546 | Canonical | Broadest released set after removing held-out questions from the collected pool | Maximum coverage; useful for broader analysis, but includes more variable item quality |
| GPQA Main | 448 | Canonical; recommended primary set in the paper | At least one expert agrees with the answer and no more than two of three non-experts answer correctly, with post-hoc handling of demonstrable expert mistakes | General GPQA reporting; better statistical power than Diamond |
| GPQA Diamond | 198 | Canonical; highest-quality subset | Both experts answer correctly and a majority of non-experts answer incorrectly, again with handling for demonstrable expert mistakes | High-confidence, hard subset; de facto leaderboard split |
| GPQA Reduced | Not stable | Non-canonical | No Rein et al. public split by this name; downstream harnesses may use local subsamples or cleaned reductions | Only comparable when the harness defines exactly what was reduced and how |
The official paper defines the Extended, Main, and Diamond sets, and the canonical Hugging Face release lists files for those three question subsets. It does not list a canonical gpqa_reduced.csv. Therefore, “GPQA Reduced” should be treated as an evaluation-local label, not as a stable member of the original benchmark family. arXiv
Extended
GPQA Extended contains 546 questions. It is the broadest released set and therefore gives the most coverage across domains and subdomains. But because it is less selectively filtered, it also contains more items where expert agreement, objectivity, or non-expert difficulty is less strong. arXiv
Extended is useful when a researcher wants more examples for analysis or when the cost of evaluating additional items is low. It is less attractive as a headline metric because its item quality is more heterogeneous.
Main
GPQA Main, often referred to simply as GPQA in the original paper, contains 448 questions. Rein et al. recommend this as the primary set. Its inclusion rule is less strict than Diamond’s but stricter than Extended’s: it keeps questions with at least one expert agreeing with the answer and filters out questions that too many non-experts could solve. arXiv
Main is often the better choice for statistical comparisons because it has more than twice as many questions as Diamond. However, that larger size comes from including items with less stringent expert-agreement and difficulty filters.
Diamond
GPQA Diamond contains 198 questions and is the highest-confidence subset. The paper defines it as questions where both expert validators answered correctly and where a majority of non-expert validators answered incorrectly, with post-hoc handling for cases where an expert validator made a demonstrable mistake. arXiv
Diamond became the most visible leaderboard split because it is compact, high-quality, and harder for non-experts. But its small size makes it statistically fragile for comparing current frontier systems whose scores differ by only a few percentage points.
Reduced
There is no canonical Rein et al. GPQA Reduced split in the paper’s subset definitions or in the official Hugging Face file list. When a dashboard, paper, or evaluation harness reports “Reduced,” it should be read as a local evaluation artifact unless the source defines the exact sampling or cleaning rule. arXiv
This is not a minor bookkeeping issue. A reduced split may be easier, harder, decontaminated, cheaper to run, or simply a convenience sample. Without a definition, it is not comparable to Main or Diamond.
Domain composition
GPQA is deep but narrow. It covers biology, physics, and chemistry, with heavy representation from some subfields and very little representation from others.
| Domain | Extended-set count | Expert accuracy | Non-expert accuracy | Expert–non-expert gap |
|---|---|---|---|---|
| Biology | 105 | 66.7% | 43.2% | 23.5 points |
| Physics | 227 | 57.3% | 32.5% | 24.8 points |
| Chemistry | 214 | 72.0% | 31.4% | 40.6 points |
The paper’s subdomain distribution is uneven. Chemistry is heavily weighted toward organic chemistry; physics includes quantum mechanics, high-energy physics, astrophysics, and smaller numbers from other subfields; biology includes molecular biology and genetics. arXiv
This unevenness is not a flaw by itself, but it limits what GPQA can support. A high GPQA score does not imply broad scientific competence across medicine, engineering, mathematics, materials science, ecology, statistics, or experimental research. It primarily supports claims about a particular kind of compact, graduate-level, multiple-choice scientific reasoning.
Original baseline results
The original paper’s baselines are historically important because they show where GPQA sat relative to GPT-4-era systems. Rein et al. report the following results:
| Evaluated system or group | Extended | Main | Diamond |
|---|---|---|---|
| Llama-2-70B-chat, few-shot chain-of-thought | 30.4% | 29.1% | 28.1% |
| GPT-3.5-turbo-16k, few-shot chain-of-thought | 28.2% | 28.0% | 29.6% |
| GPT-4, few-shot chain-of-thought | 38.7% | 39.7% | 38.8% |
| GPT-4 with search and fallback | 39.4% | 41.0% | 38.8% |
| Expert human validators | 65.4% | 72.5% | 81.2% |
| Non-expert human validators | 33.9% | 30.5% | 21.9% |
The striking result is that GPT-4 with search barely improved over GPT-4 without search, and both remained well below expert validators. In the original paper’s regime, GPQA was not saturated. It separated GPT-4 from expert humans and showed that search augmentation alone did not close the gap. arXiv
This is also where a common misconception enters. GPQA was not originally a benchmark where models beat human experts. That became true only later for some reasoning models and some evaluation protocols.
Empirical trajectory: from GPT-4 to frontier reasoning models
GPQA’s history is a compressed example of benchmark lifecycle dynamics. It began as a hard, expert-level evaluation in the GPT-4 era. By 2025–2026, GPQA Diamond had moved toward saturation for top frontier reasoning models.
| Period | Representative result | Interpretation |
|---|---|---|
| 2023 paper baseline | GPT-4 few-shot CoT: about 39.7% Main, 38.8% Diamond | GPT-4 remained far below expert validators |
| 2024 reasoning-model transition | OpenAI reported o1-preview 73.3% pass@1 and o1 77.3% pass@1 on GPQA Diamond | Test-time reasoning sharply changed the benchmark’s difficulty profile |
| 2025 frontier clustering | Epoch AI reported state-of-the-art models clustered around 83% on GPQA Diamond in May 2025 | Diamond still had signal, but top models were compressing the range |
| Late 2025 frontier reports | OpenAI reported GPT-5.1 Thinking 88.1%, GPT-5.2 Thinking 92.4%, and GPT-5.2 Pro 93.2% on GPQA Diamond, no tools | Diamond entered a high-score regime for top reasoning systems |
| 2026 independent leaderboard regime | Artificial Analysis reported Gemini 3.1 Pro Preview 94.1%, GPT-5.5 xhigh 93.5%, and GPT-5.5 high 93.2% on GPQA Diamond | Diamond is now a coarse discriminator among the strongest models, not a clean separator |
OpenAI’s o1 results were the first widely visible sign that GPQA Diamond could be pushed well above the GPT-4 baseline by models optimized for deliberate reasoning. The o1 system card reports GPT-4o at 50.6% pass@1, o1-preview at 73.3% pass@1, and o1 at 77.3% pass@1 on GPQA Diamond. OpenAI
By late 2025, OpenAI reported 92.4% for GPT-5.2 Thinking and 93.2% for GPT-5.2 Pro on GPQA Diamond, with no tools and maximum reasoning effort. The same table reports 88.1% for GPT-5.1 Thinking, making the prompt’s “~88–90%+ in 2026” framing conservative. OpenAI
Google’s Gemini 3 announcement reported 93.8% on GPQA Diamond for Gemini 3 Pro in the context of its Deep Think discussion. Artificial Analysis, using an independent five-repeat no-tools methodology, reported a current top score of 94.1% for Gemini 3.1 Pro Preview, followed by 93.5% for GPT-5.5 xhigh and 93.2% for GPT-5.5 high. blog.google
The important conclusion is not “GPQA is useless.” The conclusion is narrower: GPQA Diamond is no longer a high-resolution discriminator among the very top 2026 frontier models unless evaluated with careful paired methods, repeated runs, and transparent protocols.
Why reasoning models changed the benchmark
GPQA rewards several capabilities that improved sharply in reasoning-oriented systems:
-
Longer deliberation. Many GPQA questions require multi-step scientific reasoning rather than short factual recall.
-
Error checking. Frontier reasoning models can catch some of their own false starts.
-
Latent tool-like computation. Even without external tools, models can simulate calculations, compare mechanisms, and reason over constraints.
-
Domain-language familiarity. Pretraining and post-training have improved models’ fluency in specialized scientific text.
-
Test-time compute scaling. Some reported results use higher reasoning effort, consensus, or repeated sampling, which can materially change scores.
This is why GPQA results must be read with the evaluation protocol attached. A pass@1 no-tools result, a consensus@64 result, a tool-assisted result, and a best-of-N result are different measurements. Reporting them as if they were interchangeable destroys the benchmark’s meaning.
Artificial Analysis, for example, reports GPQA Diamond with 198 questions, five repeats, four-option multiple choice, no tools, regex extraction, and pass@1 scoring. That is a specific protocol, not a universal definition of “the GPQA score.” artificialanalysis.ai
Methodological strengths
Expert validation
GPQA’s central strength is that experts are involved at the point where expertise matters: writing and validating the questions. This makes it a better fit for evaluating scientific reasoning than benchmarks generated from general web text, textbook snippets, or crowdworker-authored questions. ar5iv
The expert process also creates a useful asymmetry. If experts can answer a question and non-experts with web access cannot, then the item is more relevant to oversight problems where ordinary evaluators cannot directly verify an AI system’s output.
Web-search resistance
The non-expert validation step is unusually strong. Validators had advanced degrees, substantial time, and web access. The fact that they still performed poorly supports the benchmark’s claim to test more than shallow search. ar5iv
This does not mean GPQA is immune to retrieval-augmented systems. It means that ordinary search by non-experts did not solve the benchmark during validation.
Narrow but deep coverage
GPQA’s narrowness is a strength for certain research questions. A benchmark that tries to cover all science may end up shallow everywhere. GPQA instead asks whether a system can handle compact, technical questions in a few expert domains.
This makes GPQA a useful companion to broader evaluations such as MMLU, MMMU, Humanity’s Last Exam, and domain-specific expert benchmarks. It should not replace them.
Contamination resistance
GPQA’s gated release, canary-style norms, and request not to publish examples help reduce contamination risk. This is especially important because small multiple-choice benchmarks can be badly compromised if examples appear in training data, eval preparation corpora, or public prompt libraries. Hugging Face
The resistance is partial. Gating and norms are not cryptographic protections. But they are stronger than the fully public, widely mirrored benchmark pattern that caused many older evaluations to degrade.
Reproducibility
Because GPQA is multiple-choice, it is easier to score reproducibly than open-ended scientific reasoning tasks. A model either selects the correct option or it does not. This makes GPQA cheap to run and easy to compare inside a single harness.
That same feature is also a weakness. Multiple-choice format introduces a 25% random baseline, allows partial exploitation of answer-choice artifacts, and compresses the space of possible reasoning failures.
Weaknesses and validity threats
Small N
The largest canonical public question set has 546 questions, Main has 448, and Diamond has 198. This is small for distinguishing frontier models separated by a few percentage points. arXiv
A simple binomial approximation illustrates the issue. At 90% accuracy, a 198-question Diamond run has a rough 95% interval of about ±4 percentage points. A 448-question Main run has a rough interval of about ±3 percentage points. Repeated runs can reduce some sampling variance, but repeated prompts over the same questions are correlated; they do not create fully independent new benchmark items.
Narrow domain coverage
GPQA is not a general scientific intelligence benchmark. It covers biology, chemistry, and physics, with uneven subdomain representation. Organic chemistry is particularly prominent within chemistry. ar5iv
This limits external validity. A model can do well on GPQA while still being unreliable at medical diagnosis, engineering design, lab protocol planning, theorem proving, causal inference, or scientific literature review.
Item-quality variance
GPQA’s validation process is strong, but the paper itself reports nontrivial expert disagreement and estimates objectivity conservatively. Diamond reduces this problem but does not eliminate it. arXiv
Epoch AI’s 2025 analysis of remaining GPQA Diamond errors found that models were clustered on a minority of hard questions and investigated whether the remaining items were valid. The article’s conclusion was that Diamond still had some signal, but it also illustrates the need for per-item analysis once top models are above 80%. Epoch AI
Selection effects
Diamond is not just “a smaller random sample of Main.” It is selected for expert agreement and non-expert failure. This makes it higher quality and harder for humans, but it also changes the distribution.
Selection effects complicate interpretation. A model can perform similarly on Main and Diamond for reasons that are not obvious: the harder subset may also contain clearer expert-agreed questions, while the larger set may contain more ambiguous or lower-quality items.
Saturation
By 2026, GPQA Diamond is close to saturated for the strongest reported frontier systems. Vals.ai’s 2026 GPQA Diamond page explicitly frames the benchmark as largely saturated, and Artificial Analysis’s leaderboard shows top models in the low-to-mid 90% range. vals.ai
Saturation does not mean every model gets 95%. It means the benchmark no longer provides fine-grained separation among the best systems. It remains useful for distinguishing weaker systems, tracking open-weight progress, and checking whether a model has serious gaps in compact scientific reasoning.
Protocol variance
Modern GPQA results differ along many dimensions:
| Protocol dimension | Why it matters |
|---|---|
| Subset | Extended, Main, Diamond, and local “Reduced” variants are not interchangeable |
| Prompting | Zero-shot, few-shot, chain-of-thought, hidden reasoning, and instruction templates can change results |
| Test-time compute | Higher reasoning effort, longer thinking, or more samples can improve accuracy |
| Tools | Search, calculators, code, retrieval, and scientific databases change what is being measured |
| Scoring | Pass@1, consensus@N, best-of-N, and fallback protocols are different metrics |
| Answer extraction | Regex extraction, forced-choice decoding, and manual parsing can differ on edge cases |
| Model version | Frontier APIs change; model names alone may not be enough |
| Date | Current-production results may differ from research-system or preview results |
For serious comparison, “Model X scored Y on GPQA” is under-specified. A useful report says: which subset, which model version, which date, which prompting, which tools, which repeats, which scoring rule, and which harness.
Main vs Diamond as discriminators
The practitioner question is not “Which subset is better?” It is “Which subset discriminates the models I care about under the constraints I have?”
When Main is better
Use GPQA Main when you want more statistical power, broader coverage, or continuity with the original paper’s recommended primary set. Main’s 448 questions give it a lower sampling error than Diamond and make it more useful for ablations, mid-tier model comparisons, and internal regression testing. arXiv
Main is also preferable when the question is: “How does this model perform on the benchmark family as originally recommended?” The original paper identifies Main as the primary GPQA set, not Diamond. arXiv
The cost is that Main includes more item-quality variance. It is less strictly filtered for both expert agreement and non-expert failure. For top frontier models, this can make Main noisy in a different way: errors may reflect ambiguous items or lower-quality questions rather than genuine capability gaps.
When Diamond is better
Use GPQA Diamond when you want the highest-confidence, hardest, most leaderboard-standard subset. Diamond is selected for both expert correctness and non-expert failure, so it better captures the original “expert-solvable, non-expert-hard” design goal. arXiv
Diamond is also the split most commonly used in current frontier-model reporting. OpenAI, Google, Artificial Analysis, Epoch AI, and Vals.ai all discuss or report GPQA Diamond results. vals.ai+4OpenAI+4OpenAI+4
The cost is small N and saturation. With only 198 questions, Diamond is a weak discriminator when two frontier models differ by only one or two points. A 93.5% vs 94.1% comparison is not, by itself, strong evidence that one model is scientifically more capable than the other.
Practical rule
For 2026 frontier models, Diamond is best treated as a coarse gate:
| Diamond score band | Practical interpretation |
|---|---|
| Below ~50% | Below early reasoning-model frontier; likely weak at compact expert science questions |
| ~50–75% | Stronger than GPT-4-era baselines, but below top reasoning systems |
| ~75–88% | Competitive reasoning-model range, still meaningfully below the saturated frontier |
| ~88–94% | Frontier or near-frontier; differences require careful paired analysis |
| Above ~94% | Possible saturation regime; investigate contamination, protocol, item overlap, and confidence intervals |
For model selection, GPQA should be one component in a larger evaluation suite. A production team choosing a model for scientific or technical workflows should combine GPQA with domain-specific tasks, tool-use evaluations, calibration tests, hallucination audits, long-context retrieval tests, and human expert review.
What GPQA does and does not measure
It measures
GPQA is evidence for:
-
compact scientific reasoning under multiple-choice constraints;
-
fluency with graduate-level concepts in parts of biology, chemistry, and physics;
-
ability to avoid some shallow search and memorization shortcuts;
-
performance on questions that non-expert PhDs with web access struggled to solve;
-
progress in test-time reasoning for expert-level science problems.
It does not measure
GPQA is not strong evidence for:
-
broad scientific discovery ability;
-
correctness in open-ended research settings;
-
lab competence or experimental design;
-
literature review quality;
-
mathematical theorem proving;
-
medical reliability;
-
engineering reliability;
-
calibrated uncertainty;
-
safe deployment in scientific workflows.
The distinction is important because GPQA’s success made it rhetorically tempting. A model that scores above expert validators on a small multiple-choice benchmark has not thereby become a better scientist than a scientist. It has shown strong performance on a particular benchmark distribution.
Reporting checklist
A rigorous GPQA result should include the following:
| Field | Example |
|---|---|
| Subset | Main, Extended, Diamond, or explicitly defined Reduced |
| Dataset source | Official GPQA release, Hugging Face, OpenAI simple-evals, local copy |
| Model | Exact model name and version |
| Date | API or checkpoint date |
| Prompt | Zero-shot, few-shot, CoT, hidden reasoning, system instructions |
| Tools | None, search, code, calculator, retrieval, or other |
| Test-time compute | Reasoning effort, sample count, consensus rule |
| Metric | Pass@1, consensus@N, best-of-N, accuracy with fallback |
| Repeats | Number of runs and aggregation rule |
| Answer extraction | Forced-choice, regex, logprob, manual, or parser |
| Confidence interval | Prefer paired bootstrap or per-item comparison |
| Item disclosure policy | Do not publish question text |
This checklist is especially important for frontier comparisons because current top systems are clustered tightly. The Stanford AI Index 2026 notes that benchmark saturation and reporting discrepancies have made model comparison harder across several leading evaluations, and that cost, latency, reliability, and domain-specific performance increasingly matter when benchmark scores compress. hai.stanford.edu
Interpretation for AI engineering
For AI engineers, GPQA is useful less as a standalone scoreboard and more as a case study in benchmark design.
It shows the value of expert-authored, expert-validated, contamination-aware evaluation. It also shows the limits of any small static benchmark once frontier labs optimize toward the distribution. GPQA moved from “GPT-4 is far below expert humans” to “frontier reasoning models are above 90% on Diamond” in roughly three years.
That trajectory is instructive for Evaluation Design. Benchmarks for frontier systems need either hidden item pools, continuously refreshed tasks, open-ended expert grading, adversarial item generation, or task distributions tied to real workflows. Static public multiple-choice datasets can be valuable, but their highest-value period may be short.
GPQA also illustrates a deeper AI Safety Evaluation problem: when models surpass non-expert validators, evaluation must either involve experts, use tasks with objectively checkable answers, or develop scalable oversight mechanisms. GPQA chose the objectively checkable route: expert-written multiple-choice questions. That made it practical, but also limited its scope.
Selected references
[Rein et al., GPQA paper] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman, “GPQA: A Graduate-Level Google-Proof Q&A Benchmark.” arXiv
[GPQA official dataset release] Hugging Face dataset card listing gpqa_diamond.csv, gpqa_extended.csv, gpqa_main.csv, and gpqa_experts.csv, with gating and anti-disclosure notice. Hugging Face
[OpenAI o1 system card] OpenAI’s reported GPQA Diamond results for GPT-4o, o1-preview, and o1. OpenAI
[OpenAI GPT-5.2 release] OpenAI’s reported GPQA Diamond results for GPT-5.1 Thinking, GPT-5.2 Thinking, and GPT-5.2 Pro. OpenAI
[Google Gemini 3 announcement] Google’s reported GPQA Diamond result in the Gemini 3 / Deep Think context. blog.google
[Artificial Analysis GPQA Diamond leaderboard and methodology] Independent 2026 leaderboard and no-tools, five-repeat pass@1 methodology. artificialanalysis.ai
[Epoch AI GPQA Diamond analysis] Analysis of what remains difficult in GPQA Diamond after frontier progress. Epoch AI
[Vals.ai GPQA Diamond benchmark page] 2026 benchmark page describing GPQA Diamond’s saturation regime and evaluation methodology. vals.ai
Companion entries
Core theory: Scalable Oversight, Sandwiching Evaluations, Expert Validation, Benchmark Saturation, Evaluation Contamination, AI Capability Benchmarks
GPQA family: GPQA Diamond, GPQA Main, GPQA Extended, Google-Proof Evaluation, Graduate-Level QA Benchmarks
Related benchmarks: MMLU, MMMU, Humanity’s Last Exam, AIME, FrontierMath, SWE-bench
Practice: Evaluation Harnesses, Pass@1, Consensus@N, Test-Time Compute, Reasoning Effort, Model Leaderboards
Counterarguments and limitations: Small-N Benchmarks, Multiple-Choice Artifacts, Benchmark Overfitting, Leaderboard Noise, Scientific Reasoning Evaluation