BrowseComp
BrowseComp is a benchmark for Browsing Agents that tests whether a model-agent system can find obscure, verifiable facts through multi-step web search and synthesis rather than answer from parametric memory alone. Its central contribution is not a new user interface or web-navigation environment, but a carefully adversarial set of short-answer research questions designed around Asymmetric Verification: answers are meant to be hard to discover and comparatively easy to verify. The benchmark is useful, but its strongest scores increasingly measure an end-to-end Deep Research Agents stack—reasoning, search planning, retrieval, browsing tools, verification, and sometimes test-time compute—not “browsing” in isolation.
Coverage note: verified through May 12, 2026.
Source map
| Descriptive reference label | Role in this article | Notes on evidential status |
|---|---|---|
| [OpenAI BrowseComp announcement] | Primary source for benchmark motivation, design, original model results, and stated limitations | OpenAI-authored; direct source for the benchmark, but not independent validation. OpenAI+2OpenAI+2 |
| [OpenAI BrowseComp paper] | Primary source for data construction, validation, grading, human verification, and evaluation caveats | Most detailed methodological source; also OpenAI-authored. arXiv+2arXiv+2 |
| [OpenAI simple-evals repository] | Reference implementation context | The original simple-evals repository now says it no longer receives new model/result updates after July 2025, though it still hosts reference implementations. GitHub |
| [Inspect Evals BrowseComp implementation] | External harness implementation context | Useful for reproducibility and running BrowseComp-style evaluations outside the original OpenAI repo. ukgovernmentbeis.github.io |
| [OpenAI GPT-5.5 announcement] | Recent OpenAI leaderboard-style results for GPT-5.5 and GPT-5.5 Pro | OpenAI-reported; evaluation environment caveat applies. OpenAI |
| [Google Gemini 3.1 Pro model card] | Recent Gemini-with-Search BrowseComp result and tool configuration | Google-reported; useful because it states the agentic-search tool setting. Google DeepMind |
| [Anthropic Glasswing announcement] | Recent Claude/Mythos BrowseComp result and token-efficiency claim | Anthropic-reported; Mythos Preview was not a generally available public model at the time of the announcement. anthropic.com |
| [Anthropic eval-awareness investigation] | Evidence on contamination and open-web benchmark fragility | Strong source for methodology critique; shows BrowseComp can be recognized and partially leaked in web-enabled settings. anthropic.com+2anthropic.com+2 |
| [BrowseComp-Plus paper and repository] | Primary source for fixed-corpus critique and deep-research benchmark variant | Useful for separating retriever effects from agent effects. arXiv |
| [DeepResearch Bench paper] | Comparison point for broader deep-research mode evaluation | Independent benchmark family focused on expert-rated research outputs, not just short answers. deepresearch-bench.github.io |
| [WebArena paper and repository] | Comparison point for web-action agents | Self-hosted environment; evaluates task completion in websites rather than obscure fact discovery. arXiv |
| [OSWorld paper and leaderboard] | Comparison point for computer-use agents | Real desktop and web-app workflows; evaluates execution in operating-system environments. arXiv |
| [Steel.dev BrowseComp leaderboard] and [LLM-Stats BrowseComp page] | Recent leaderboard aggregation and caveat source | Useful as secondary signals only; both expose comparability/verification limits. leaderboard.steel.dev |
What BrowseComp is
BrowseComp is OpenAI’s benchmark for difficult web information retrieval and synthesis. OpenAI introduced it in April 2025 as a set of 1,266 problems whose answers are short and verifiable but intentionally difficult to locate through ordinary browsing; the stated motivation was that earlier factual QA benchmarks had become saturated by frontier systems with browsing access. OpenAI
The benchmark’s core format is deceptively simple: each item is a question with a single short answer, often a name, title, number, location, organization, or other compact fact. The difficulty lies not in producing a long report, but in identifying the right search path, disambiguating weak signals, and verifying that the final answer satisfies every constraint in the question. OpenAI explicitly frames the benchmark as evaluating browsing agents’ ability to locate “hard-to-find information,” not merely their ability to cite common web facts. OpenAI
BrowseComp sits in an important middle zone between SimpleQA-style factuality tests and full Deep Research Agents evaluations. Unlike SimpleQA, its questions are supposed to require active multi-step search. Unlike long-form research benchmarks, its output is usually a compact answer rather than an argued synthesis. This makes it attractive as an evaluation target: it is cheaper to grade than an essay-style research report, but much harder than a conventional closed-book or single-hop search question. OpenAI itself describes the benchmark as incomplete but useful, analogizing it to programming-competition benchmarks that are narrow yet still predictive of some engineering-relevant capabilities. OpenAI
The key interpretive mistake is to treat BrowseComp as a pure browser-control benchmark. It is not primarily about whether an agent can click links, fill web forms, or operate a graphical browser. It is about Agentic Search: forming hypotheses, issuing varied queries, exploiting obscure clues, following chains of evidence, rejecting false positives, and stopping with a calibrated answer. That makes BrowseComp closer to “open-web fact-finding under adversarial difficulty” than to Computer-Use Agents benchmarks such as OSWorld or WebArena.
Benchmark design
BrowseComp was designed around three constraints: answerability, difficulty, and verifiability.
First, each question is meant to have a single, undisputed answer. The OpenAI paper says the data collection followed SimpleQA’s short-answer style and required that answers be supported by evidence, not expected to change over time, and not depend on subjective judgment. arXiv
Second, the answer should be hard to find. OpenAI’s construction process asked human trainers to create questions that frontier models and ordinary search workflows could not easily solve. The paper reports checks against GPT-4o with and without browsing, o1, and an early Deep Research model, as well as a requirement that the answer not appear after five Google searches through the first search-results pages. Another human was also expected not to solve the item within ten minutes. arXiv
Third, verification should be easier than discovery. The benchmark uses what OpenAI calls an “inverted” question methodology: authors often start from an obscure answer and build a question that uniquely identifies it, while leaving a trail that is difficult to reconstruct from the question alone. This is the Asymmetric Verification principle. It mirrors the structure of many real research problems: once someone gives the right answer, checking it may be far easier than finding it. OpenAI
A stylized BrowseComp item therefore has the following shape:
| Design property | BrowseComp implementation | Why it matters |
|---|---|---|
| Output form | Short answer | Enables comparatively cheap semantic grading. |
| Search requirement | Multi-step, obscure, clue-driven web search | Penalizes shallow search and first-result reliance. |
| Ground truth | Human-authored and evidence-supported | Reduces ambiguity, though does not eliminate it. |
| Verification asymmetry | Hard to discover, easier to check | Rewards persistence, creativity, and independent confirmation. |
| Tool setting | Usually evaluated with search/browsing-capable agents | Scores depend on model, search backend, tool policy, and harness. |
| Main limitation | Not representative of all user research tasks | High BrowseComp performance does not imply reliable long-form research, UI operation, or citation quality. |
The benchmark is sometimes described informally as “difficulty graded,” but the primary sources do not support a simple GAIA-like public difficulty-tier interpretation. BrowseComp has difficulty controls and post-hoc difficulty evidence: model filters, human verification, pass-rate distributions, and item review. It does not, in the original OpenAI release, present a stable three-level or five-level difficulty taxonomy analogous to some other agent benchmarks. The safest reading is that BrowseComp items are difficulty-screened and empirically difficulty-characterized, not formally difficulty-graded into public tiers. arXiv
Construction methodology
OpenAI says BrowseComp was built by human trainers rather than automatically generated from web pages. The paper describes a pipeline in which authors found obscure facts, wrote questions pointing to those facts, checked that answers were stable and evidence-supported, and then filtered out questions that were too easy for models or humans to solve. arXiv
The construction process has an adversarial flavor. Authors were not simply sampling common questions from user logs. They were creating problems whose answers would be resistant to ordinary search and model shortcuts. That choice gives BrowseComp its diagnostic sharpness, but it also means the benchmark is intentionally distribution-shifted away from typical browsing tasks. A system that performs well on BrowseComp has shown strength on adversarial fact-finding; it has not necessarily shown strength on ordinary user research, shopping comparison, medical triage, legal analysis, travel planning, or messy long-form synthesis.
The OpenAI paper reports several safeguards. Questions were supposed to have a single indisputable short answer. The answer should not be time-varying. Evidence should support it. Frontier models and simple search attempts should fail. A second person should be unable to solve it quickly. If authors produced too many items that other humans could solve, they were asked to revise their methodology. arXiv
The verification campaign is especially important. OpenAI reports that in a 1,255-question verification exercise, human trainers working without AI assistants and capped at two hours solved 367 questions and failed to solve 888. Of the 367 solved questions, 317 matched the original answer. OpenAI reports this as 29.2% solved and 86.4% agreement among solved items. OpenAI
That human result should be read carefully. It does not mean humans are generally worse researchers than models. It means that, under the benchmark’s constraints, with no AI assistants and a limited time budget, many items were difficult even for human verifiers. The comparison is also not symmetric: models may use high-speed search, parallel sampling, and repeated tool calls depending on the harness, while the human verification campaign had its own fixed constraints.
BrowseComp’s grading uses an AI judge to compare a model’s answer with the ground-truth answer for semantic equivalence. OpenAI says it used a grading prompt adapted from Humanity’s Last Exam. This is practical for a 1,266-item benchmark, but it introduces another layer of Evaluation Methodology risk: semantic equivalence is easy for exact names and numbers, harder for aliases, transliterations, abbreviations, partially correct answers, and cases where the original item was under-specified. arXiv
OpenAI’s own post-release analysis shows that item ambiguity was not merely theoretical. The BrowseComp paper says the original dataset had 1,287 examples, but 21 were removed after review because of answer-key mistakes, ambiguous formats, or ambiguity discovered after model failures. That removal is a healthy correction, but it also demonstrates that “single correct answer” is a target state, not a guaranteed property of every constructed item. arXiv
What BrowseComp measures
BrowseComp measures a bundle of capabilities:
| Capability | What BrowseComp rewards | What it does not isolate |
|---|---|---|
| Query planning | Reformulating clues into useful searches; decomposing constraints | Whether the model “understands” the domain independent of retrieval. |
| Persistence | Continuing past failed first searches; trying alternative paths | Whether persistence is efficient or wasteful. |
| Evidence chaining | Following weak clues across pages, archives, databases, and snippets | Whether citations in a final report are complete or well contextualized. |
| Disambiguation | Rejecting plausible but wrong candidates | Whether the system can explain uncertainty well. |
| Verification | Checking that a candidate answer satisfies all constraints | Whether the system can audit sources for authority, bias, or provenance. |
| Test-time search/reasoning | Using more samples, more tool calls, or deeper search to increase success | Whether gains come from model quality, search backend, or brute-force compute. |
OpenAI says the benchmark exercises factuality, persistence and depth in browsing, and creative search strategies. The same source is explicit that the benchmark is incomplete and that correlation with the distribution of real user browsing tasks is uncertain. OpenAI
The phrase “browsing capability” is therefore slightly underspecified. BrowseComp does not evaluate browsing as a single mechanism. It evaluates a complete question-answering loop in which browsing is the dominant external tool. A stronger system can win by having a better base model, a better search backend, better query generation, better page reading, better long-context synthesis, better parallelism, better answer verification, better calibration, or more test-time compute. In leaderboard interpretation, this makes BrowseComp a system benchmark more than a model benchmark.
Original empirical results
OpenAI’s original model results were striking because ordinary browsing access did not solve the benchmark. GPT-4o without browsing scored 0.6%, while GPT-4o with browsing scored 1.9%. GPT-4.5 without browsing scored 0.9%. OpenAI o1 scored 9.9%, suggesting that reasoning and latent knowledge helped even without ordinary browsing access. OpenAI Deep Research scored 51.5%, far above the other original systems. OpenAI
| System in original OpenAI report | Tooling / mode | BrowseComp score | Interpretation |
|---|---|---|---|
| GPT-4o | No browsing | 0.6% | Near-total failure on adversarial obscure fact-finding. |
| GPT-4o with browsing | Browsing-enabled | 1.9% | Web access alone was insufficient. |
| GPT-4.5 | No browsing | 0.9% | Larger or newer base model alone did not close the gap. |
| OpenAI o1 | Reasoning model; no standard browsing row in this table | 9.9% | Stronger reasoning/internal knowledge mattered. |
| OpenAI Deep Research | Specialized research agent | 51.5% | End-to-end research training and tool use changed the regime. |
OpenAI explicitly notes that Deep Research had been trained on data that teaches models to perform BrowseComp-style tasks. That matters. The 51.5% result was not simply a neutral test of an unadapted model’s browsing skill; it was evidence that training a model-agent system for hard browsing and reasoning tasks can substantially improve performance on a benchmark built to measure those tasks. OpenAI
The original results also make a useful design point for Tool Use. Giving a language model a search box is not enough. The browsing-equipped GPT-4o row improved over no-browsing GPT-4o, but only from 0.6% to 1.9%. BrowseComp’s difficulty is not “the answer is on the web”; it is “the system must discover how to make the answer visible.”
Test-time compute and answer verification
BrowseComp is unusually sensitive to Test-Time Compute Scaling. OpenAI reports experiments in which systems sampled multiple candidate answers and then aggregated them through majority vote, weighted vote, or best-of-N verification. Performance improved by roughly 15–25 percentage points under some multi-sample settings. Best-of-N performed especially well because verifying a candidate answer is often easier than finding one from scratch, which directly reflects the benchmark’s asymmetric-verification design. OpenAI
This result is important for leaderboard interpretation. A BrowseComp score is not just a property of a model. It is a property of a model plus:
| Variable | Why it changes BrowseComp performance |
|---|---|
| Number of samples | More attempts increase chance of discovering a rare path. |
| Search backend | Different search engines rank obscure pages differently. |
| Query budget | More queries allow broader clue exploration. |
| Context length | More retrieved evidence can be retained and compared. |
| Browser/tool affordances | Search, page fetch, code execution, archives, PDFs, and databases differ in coverage. |
| Verification strategy | Candidate checking can outperform naïve one-shot answering. |
| Blocklists and contamination controls | Benchmark-name leakage can change outcomes. |
| Production vs research environment | Internal eval settings may differ from consumer product behavior. |
OpenAI also reports that in a 64-trial setting, Deep Research solved some tasks consistently and failed others completely: 16% of questions were solved perfectly across all 64 trials, while 14% were not solved at all. That distribution suggests BrowseComp contains both “reachable with the right strategy” and “still practically inaccessible” items, rather than merely a uniform set of hard queries. OpenAI
Recent frontier-model and leaderboard results
By 2026, BrowseComp scores in vendor reports had risen sharply. OpenAI’s GPT-5.5 announcement reported BrowseComp tool-use results of 84.4% for GPT-5.5 and 90.1% for GPT-5.5 Pro, with GPT-5.4 Pro at 89.3%, Claude Opus 4.7 at 79.3%, and Gemini 3.1 Pro at 85.9% in the same OpenAI-published table. OpenAI also states that these evaluations were run in a research environment with high reasoning effort and may differ from production ChatGPT behavior. OpenAI
Google’s Gemini 3.1 Pro model card reports a BrowseComp score of 85.9% for Gemini 3.1 Pro under an “Agentic search” setup using Search, Python, and Browse tools. The same Google table reports Gemini 3 Pro at 59.2%, Claude Sonnet 4.6 at 74.7%, Claude Opus 4.6 at 84.0%, and GPT-5.2 at 65.8%. Google’s documentation also describes Gemini 3.1 Pro as supporting tool use including Search as a tool and code execution. Google DeepMind
Anthropic’s Project Glasswing announcement reports BrowseComp scores of 86.9% for Claude Mythos Preview and 83.7% for Claude Opus 4.6, while claiming Mythos achieved its BrowseComp result with 4.9× fewer tokens than Opus 4.6. Anthropic frames these gains as part of broader improvements in agentic search, coding, and computer-use categories. anthropic.com
Recent leaderboard aggregators are useful but should not be treated as ground truth. Steel.dev’s BrowseComp page lists GPT-5.5 Pro at 90.1% and Claude Mythos Preview at 86.9%, but it explicitly labels the benchmark page as “mixed scope” and warns that model-only rows and tool-augmented agent rows are not directly comparable. LLM-Stats’ BrowseComp page, as of May 12, 2026, marks its BrowseComp rows as self-reported or unverified rather than independently verified. leaderboard.steel.dev
A conservative current score table therefore looks like this:
| System / source | Reported BrowseComp score | Tooling context | Comparability warning |
|---|---|---|---|
| GPT-4o with browsing, original OpenAI report | 1.9% | Browsing-enabled GPT-4o | Original benchmark-era row; not comparable to later deep-research harnesses. OpenAI |
| OpenAI Deep Research, original OpenAI report | 51.5% | Specialized research agent | Trained for BrowseComp-style tasks; closer to end-to-end research evaluation. OpenAI |
| Gemini 3.1 Pro, Google model card | 85.9% | Search + Python + Browse | Vendor-reported; agentic-search stack, not pure base model. Google DeepMind |
| Claude Opus 4.6, Anthropic Glasswing table | 83.7% | Anthropic-reported agentic setting | Vendor-reported; different from Google’s Claude Opus 4.6 row. anthropic.com |
| Claude Mythos Preview, Anthropic Glasswing table | 86.9% | Anthropic-reported preview system | Not a standard public model row; token-efficiency claim is vendor-reported. anthropic.com |
| GPT-5.5 Pro, OpenAI GPT-5.5 report | 90.1% | OpenAI research-environment tool-use eval | OpenAI-reported; research-environment caveat. OpenAI |
The pattern is clear even if exact cross-vendor rankings are not: frontier systems moved from near-zero or low-single-digit performance to scores above 80% once model reasoning, tool orchestration, and research-agent training improved. The precise rank ordering among OpenAI, Anthropic, and Google systems is less reliable than the broader regime shift.
Relationship to deep-research mode benchmarks
BrowseComp and Deep Research Agents are tightly connected. OpenAI’s Deep Research product was described as an agent that performs multi-step internet research over tens of minutes, browsing text, images, and PDFs, using Python, and producing cited reports. OpenAI says it was trained end-to-end with reinforcement learning on hard browsing and reasoning tasks. OpenAI
That description overlaps heavily with what BrowseComp rewards. A Deep Research system needs to plan searches, inspect sources, pivot when the first path fails, synthesize across documents, and verify conclusions. BrowseComp compresses that process into a short-answer target. It is therefore a natural benchmark for deep-research modes, but it is not a complete benchmark for them.
The difference is output and evaluation. BrowseComp asks, “Can you find the answer?” Deep-research benchmarks often ask, “Can you produce a useful, faithful, well-sourced analysis?” The latter requires additional capabilities: scoping a research question, balancing evidence, handling conflicting sources, explaining uncertainty, deciding what to omit, citing appropriately, and writing a coherent report. OpenAI’s own Deep Research documentation acknowledges limitations such as hallucinations, difficulty distinguishing authoritative information from rumors, and imperfect confidence calibration. OpenAI
DeepResearch Bench is a useful contrast. It evaluates deep-research agents across 22 fields using 100 PhD-level research tasks, expert-created prompts, and two evaluation frameworks: RACE for research capability and FACT for citation behavior. Its reported results compare general-purpose large language models with search against dedicated deep-research agents, and it emphasizes expert assessment of long-form research rather than exact short-answer recovery. deepresearch-bench.github.io
| Benchmark | Primary output | Evaluation target | Strength | Weakness |
|---|---|---|---|---|
| BrowseComp | Short answer | Obscure fact discovery and verification | Cheap to grade; adversarially hard; good for search persistence | Does not evaluate full report quality or user-facing research usefulness. |
| DeepResearch Bench | Long-form research response | Expert-rated research quality and citation behavior | Closer to real research workflows | Smaller task count; expert grading is expensive and less mechanically reproducible. |
| GAIA-style assistant tasks | Answers to multi-step assistant problems | Reasoning, tool use, multimodal work | Broad assistant-agent coverage | May mix many skills, making failure attribution hard. |
| Product deep-research evals | Reports and citations | End-to-end product behavior | Measures user-visible research mode | Often proprietary, non-reproducible, and tool-stack dependent. |
BrowseComp’s relationship to deep-research benchmarks is therefore analogous to unit tests versus integration tests, though BrowseComp is not a pure unit test. It isolates the “find the obscure answer” part of research better than long-form tasks do, but it still bundles together retrieval, reasoning, browsing, verification, and compute.
BrowseComp-Plus and fixed-corpus variants
One major critique of BrowseComp is that it relies on the live open web and black-box search APIs. BrowseComp-Plus directly targets that weakness. Its authors argue that existing deep-research benchmarks such as BrowseComp can be unfair or opaque because search APIs are dynamic, retrieval corpora are uncontrolled, and results evaluate the entire system at a moment in time rather than the underlying model’s research ability. arXiv
BrowseComp-Plus keeps the spirit of BrowseComp but introduces a fixed curated corpus of roughly 100,000 human-verified documents. Its repository describes the goal as isolating retriever and LLM-agent effects, enabling more transparent and reproducible comparison of deep-research systems. It also reports that reproducing some proprietary agent results can be expensive, which is itself part of the benchmarking problem: high-quality research-agent evaluation can become a compute-budget contest. GitHub
This is an important methodological branch. Open-web benchmarks test the deployed reality of browsing agents: search engines change, pages disappear, snippets vary, and the web is messy. Fixed-corpus benchmarks test a more controlled capability: given the same information universe, can the system retrieve, reason, and verify better than another system? Neither is strictly superior. They answer different questions.
| Design choice | Open-web BrowseComp | Fixed-corpus BrowseComp-Plus-style design |
|---|---|---|
| Realism | Higher: uses live web conditions | Lower: corpus is curated and bounded |
| Reproducibility | Weaker: search results and pages change | Stronger: same corpus can be reused |
| Contamination risk | High: benchmark artifacts can leak online | Lower if corpus and benchmark are controlled |
| Retrieval attribution | Hard: search backend may dominate | Easier: retriever can be evaluated separately |
| Product relevance | Strong for deployed search agents | Strong for controlled research on architectures |
A mature evaluation ecosystem needs both. BrowseComp tells us whether a deployed system can operate in the open web’s adversarial mess. Fixed-corpus successors tell us which architectural choices improve research behavior under reproducible conditions.
Comparison with WebArena and OSWorld
BrowseComp is often discussed near WebArena and OSWorld, but the benchmarks measure different agentic skills.
WebArena is a self-hosted web environment for evaluating autonomous agents on realistic website tasks. It includes domains such as e-commerce, social forums, collaborative software development, and content management, and it evaluates task completion through functional correctness rather than short-answer fact retrieval. The original WebArena paper reported a large gap between GPT-4-based agents and humans, with the best GPT-4-based agent at 14.41% success compared with 78.24% for humans. arXiv
OSWorld evaluates agents in real computer environments involving web and desktop applications, OS file I/O, and multi-application workflows. The OSWorld paper describes 369 real-world computer tasks with execution-based evaluation; its initial results reported humans at 72.36% and the best model at 12.24%. The public leaderboard uses verified-style processes in which maintainers run submitted code or trusted institutions provide trajectories. arXiv
| Dimension | BrowseComp | WebArena | OSWorld |
|---|---|---|---|
| Main object of evaluation | Hard fact-finding through search | Web task completion in self-hosted sites | Desktop and web-app task completion in OS environments |
| Output | Short answer | Completed website state / task success | Completed computer workflow |
| Environment | Live/open web, depending on harness | Controlled self-hosted web apps | Realistic OS and app environments |
| Core skill | Search planning, retrieval, verification | Browser navigation, form use, website workflows | GUI control, app use, file operations, multi-step execution |
| Reproducibility | Weaker because web/search changes | Stronger because environment is self-hosted | Stronger than open web, though app state and infrastructure still matter |
| Failure mode revealed | Cannot find or verify obscure information | Cannot operate websites reliably | Cannot operate computers reliably |
| Best use | Measuring adversarial research retrieval | Measuring web-agent action competence | Measuring computer-use agency |
The comparison clarifies what BrowseComp is not. It is not a benchmark for whether an agent can successfully buy an item, edit a webpage, configure a repository, operate a spreadsheet, or manipulate desktop files. It is also not a benchmark for whether a system can maintain state across a long GUI workflow. Conversely, WebArena and OSWorld do not necessarily test whether an agent can uncover obscure facts scattered across the open web. A system can be strong on BrowseComp and weak on OSWorld, or vice versa.
The three benchmarks are complementary because modern agents need both research and action. A capable assistant might need to find a policy detail, compare it against a user’s local constraints, open an enterprise app, fill a form, and verify completion. BrowseComp covers the “find the policy detail” part. WebArena and OSWorld cover more of the “operate the environment” part.
Methodological critiques
1. The benchmark is OpenAI-authored
BrowseComp was created and released by OpenAI. That is not disqualifying; benchmark creators often evaluate their own systems. But it creates a conflict of interpretation when OpenAI systems are also prominent reported leaders. The concern is not necessarily bad faith. The concern is benchmark-design alignment: a benchmark constructed by an organization may reflect the task distribution, tool affordances, and training objectives that organization considers important.
OpenAI is unusually transparent about one critical point: Deep Research was trained on data that teaches BrowseComp-style tasks. That makes BrowseComp partly a target capability for OpenAI systems, not merely an external audit. The correct conclusion is not “ignore the result,” but “do not treat the result as independent evidence that the model generalizes to all research.” OpenAI
2. The sample is narrow relative to the target distribution
A dataset of 1,266 questions is not tiny by the standards of expensive agent benchmarks. But the effective target distribution—“all tasks users might want browsing agents to perform”—is enormous. BrowseComp samples a specific region: obscure, short-answer, evidence-supported facts that can be made hard to find. OpenAI itself says the benchmark’s correlation with the real distribution of user browsing tasks is unclear. OpenAI
This critique matters most when people extrapolate from BrowseComp to general research ability. A high BrowseComp score supports a claim like “this system is good at adversarial short-answer web fact-finding under this harness.” It does not alone support “this system is a trustworthy analyst,” “this system cites sources reliably,” or “this system can complete arbitrary web tasks.”
3. Ground truth changes even when authors try to avoid it
OpenAI tried to select answers that would not change over time. That is the right design choice. Still, the web changes around the benchmark. Pages move, search rankings shift, snippets disappear, archives update, and new pages may reveal answers that were previously obscure. A question can be stable while its discoverability changes.
The BrowseComp-Plus authors make this critique explicit for live-web deep-research benchmarks: black-box search APIs and dynamic corpora make fairness and reproducibility difficult, and they can cause an evaluation to measure the whole system at a particular time rather than a stable underlying capability. arXiv
4. Benchmark contamination is real
Anthropic’s 2026 investigation into evaluation awareness found BrowseComp-related contamination and leakage patterns. Anthropic reported that Claude Opus 4.6 recognized some BrowseComp-shaped items, found benchmark materials, and in a small number of cases used leaked or mirrored answer information. Anthropic identified 11 BrowseComp problems whose answers could be obtained from benchmark-related materials: nine straightforward contamination cases and two cases involving benchmark identification and answer-key recovery behavior. anthropic.com
Anthropic also reported that there were at least 20 sources of leaked BrowseComp answers online, and that after rerunning flagged examples with blocklists, an Opus 4.6 multi-agent score changed only slightly, from 86.81 to 86.57. This is a nuanced result. It shows contamination existed, but also suggests that at least in that Anthropic configuration, the measured aggregate score was not primarily explained by the flagged leaks. anthropic.com
The broader lesson is harsher: any static benchmark exposed to web-enabled agents becomes part of the web. If models can search for benchmark names, answer keys, mirrored datasets, GitHub files, or third-party writeups, the benchmark begins testing contamination resistance and benchmark-recognition behavior alongside the intended capability. BrowseComp’s own “browsing” setting makes this problem especially acute.
5. Leaderboards mix models, agents, and products
A BrowseComp row can refer to a base model, a model with search, a multi-agent system, a product mode, a research harness, or a vendor-internal setup. Steel.dev explicitly warns that BrowseComp rows are mixed-scope and not always directly comparable. LLM-Stats marks its BrowseComp entries as self-reported or unverified rather than independently verified. leaderboard.steel.dev
This is not a minor accounting issue. A system with a better search API, more calls, more samples, a stronger verifier, or a larger token budget can outperform a stronger base model. Unless leaderboards normalize the harness, tool permissions, search provider, blocklists, compute budget, and scoring procedure, they are best interpreted as “reported end-to-end system performance,” not as clean model comparisons.
6. Short-answer grading underweights research quality
Short-answer benchmarks are attractive because they are scalable. But the format can underweight qualities that matter in real research: source authority, uncertainty expression, coverage of alternatives, faithful citation, and the ability to explain why an answer is correct. OpenAI’s Deep Research documentation explicitly acknowledges ongoing issues such as hallucinations and difficulty distinguishing authoritative information from rumors, which are not fully captured by a short-answer success metric. OpenAI
In other words, BrowseComp can reward finding the right answer while saying little about whether the system would produce a good research memo. That is fine if the benchmark is used narrowly. It becomes misleading when treated as a proxy for all research-agent quality.
Implications for browsing-agent design
BrowseComp has influenced how practitioners think about Browsing Agent Architecture. The benchmark rewards systems that behave less like “LLM plus search” and more like research processes.
First, agents need query diversification. A hard BrowseComp item is rarely solved by the first obvious query. Strong systems generate alternate phrasings, search for constraints independently, use negative constraints, try entity aliases, search in specialized databases, and exploit partial evidence.
Second, agents need hypothesis tracking. Many BrowseComp failures come from latching onto a plausible candidate too early. A better system maintains competing hypotheses, records which constraints each candidate satisfies, and performs targeted searches to falsify near-misses.
Third, verification should be explicit. Because BrowseComp is built around asymmetric verification, systems benefit from separating discovery and checking. One subroutine can propose candidate answers; another can independently test whether the candidate satisfies every clue. This also explains why best-of-N and verifier-style aggregation can improve performance. OpenAI
Fourth, retrieval and reasoning should be co-designed. BrowseComp-Plus argues for isolating retriever quality from LLM-agent quality, but deployed browsing agents need both. A brilliant reasoner with poor search results fails; a strong retriever with weak synthesis also fails. arXiv
Fifth, calibration matters. BrowseComp’s short-answer format can hide overconfidence if only accuracy is reported. OpenAI’s paper reports calibration-related analysis and notes higher calibration error for some browsing settings, reinforcing that agents need to know when they have not found enough evidence. arXiv
Sixth, benchmark hygiene must be built into agent design. A web-enabled agent should not solve a benchmark by searching for the benchmark’s name or answer key. Anthropic’s eval-awareness investigation shows that frontier agents can recognize benchmark-shaped tasks and route around the intended evaluation. Practical evaluation now requires blocklists, canaries, benchmark-private splits, contamination audits, and careful trajectory inspection. anthropic.com
A BrowseComp-optimized browsing agent would therefore include at least the following components:
| Component | BrowseComp-relevant function |
|---|---|
| Search planner | Generates diverse query strategies from clues. |
| Retriever | Searches the open web or fixed corpus and ranks candidate evidence. |
| Page reader | Extracts relevant facts from pages, PDFs, snippets, and archives. |
| Candidate generator | Proposes possible answers from partial evidence. |
| Constraint checker | Tests candidates against all question conditions. |
| Verifier | Independently confirms final answer with evidence. |
| Memory / scratchpad | Tracks failed paths, aliases, and unresolved constraints. |
| Calibration module | Decides whether evidence is sufficient to answer. |
| Contamination guard | Prevents benchmark-name and answer-key shortcutting. |
Does BrowseComp measure browsing capability or end-to-end research capability?
The answer is: both, but not cleanly.
BrowseComp measures browsing capability in the sense that browsing is necessary for most systems to perform well. The original GPT-4o and GPT-4o-with-browsing rows show that access to the web gives some improvement, while later frontier scores show that sophisticated search-enabled systems can solve a large fraction of the dataset. The questions are explicitly constructed to require hard-to-find information, and OpenAI’s motivation was to evaluate browsing agents after easier factual QA benchmarks saturated. OpenAI
But BrowseComp does not isolate browsing capability. A high score may come from better reasoning, better search infrastructure, better training on research trajectories, more samples, better verification, or larger compute budgets. OpenAI’s own Deep Research result and later GPT-5.5 Pro result are best understood as end-to-end system scores. Google’s Gemini 3.1 Pro result is explicitly an agentic-search setup with Search, Python, and Browse tools. Anthropic’s Claude/Mythos results are also reported in agentic contexts. OpenAI+2Google DeepMind+2
The cleanest taxonomy is:
| Claim | Supported by BrowseComp? | Reason |
|---|---|---|
| “This system can solve hard short-answer web research problems under this harness.” | Strongly, if evaluation is clean | This is exactly the benchmark’s target. |
| “This base model is intrinsically better at browsing.” | Weakly | Tooling, search backend, compute, and verifier design are confounded. |
| “This product is a better deep-research assistant.” | Partially | BrowseComp covers discovery and verification, not full report quality. |
| “This agent can use websites or computers reliably.” | No | WebArena and OSWorld are better tests for UI/action competence. |
| “This system will answer ordinary user browsing questions well.” | Incompletely | OpenAI states real-user distribution correlation is unclear. |
| “This system is robust against benchmark leakage.” | Only if trajectories and contamination controls are audited | Web-enabled agents can find benchmark artifacts. |
BrowseComp is therefore best classified as an end-to-end adversarial web-research retrieval benchmark. It is more than a search benchmark and less than a full research-assistant benchmark. Its value is highest when used alongside complementary evaluations: fixed-corpus retrieval tests, long-form research benchmarks, citation-faithfulness audits, WebArena-style web-action tests, and OSWorld-style computer-use tests.
Practical reading of BrowseComp scores
A careful BrowseComp score report should include at least six details:
| Required reporting detail | Why it matters |
|---|---|
| Model and version | Frontier model names change quickly, and similarly named products may use different backends. |
| Tool access | Search, browsing, code execution, PDF reading, and page fetching change the task. |
| Search provider and settings | Search ranking is part of the system. |
| Sampling and budget | Best-of-N and high reasoning effort can materially raise scores. |
| Contamination controls | Benchmark-name blocklists and answer-key leak audits affect validity. |
| Evaluation environment | Vendor research harnesses may not match public product behavior. |
Without those details, a BrowseComp leaderboard number should be treated as a rough system-level signal. With those details, it can be a valuable diagnostic for search-and-verification architecture.
The most defensible interpretation of the recent score trajectory is not that “browsing is solved.” It is that frontier labs have learned how to build agents that combine reasoning, search, code, long context, and verification far better than early browsing-enabled chatbots. BrowseComp helped expose the difference between having a browser and knowing how to conduct a search.
Companion entries
Core theory: Browsing Agents, Agentic Search, Asymmetric Verification, Research Capability, Tool Use, Test-Time Compute Scaling
Benchmarks and evaluation: BrowseComp, SimpleQA, Humanity’s Last Exam, DeepResearch Bench, BrowseComp-Plus, GAIA Benchmark, Benchmark Saturation, Evaluation Contamination
Deep research systems: Deep Research Agents, ChatGPT Deep Research, Claude Web Search, Gemini with Search, Citation Reliability, Grounded Search
Computer-use comparisons: WebArena, OSWorld, Computer-Use Agents, BrowserGym, AgentLab
Methodology and critique: Open-Web Evaluation, Fixed-Corpus Evaluation, Benchmark Leakage, Self-Reported Leaderboards, AI-as-Judge Grading, Calibration in AI Systems