Swe-Bench-Verified
Abstract. SWE-bench Verified is a 500-instance, human-validated subset of SWE-bench designed to remove problems whose issue statements, hidden tests, or execution environments make them poor measurements of software-engineering ability. Its methodological importance is not that it is “smaller SWE-bench,” but that it tries to separate model competence from benchmark noise: ambiguous tasks, invalid test oracles, and infeasible instances. As of 2026, Verified remains a useful historical and diagnostic benchmark, but not an uncontested frontier benchmark because it is saturated enough, public enough, and still imperfect enough that the full distribution and newer live/held-out variants matter.
Coverage note: verified through May 12, 2026.
SWE-bench Verified: The Curated Subset
1. What SWE-bench Verified is
SWE-bench Verified is the human-filtered 500-task subset of the original SWE-bench benchmark. The official SWE-bench site describes it as “a human-validated subset of 500 SWE-bench instances” created with OpenAI, where annotators checked that problem descriptions were clear, test patches were correct, and tasks were solvable. SWE-bench OpenAI’s release similarly framed it as a subset of SWE-bench samples “verified to be non-problematic,” after a human review pass over randomly sampled instances from the original benchmark. OpenAI
The parent benchmark, SWE-bench, evaluates whether an AI system can modify a real GitHub repository to resolve an issue, using hidden unit tests derived from the associated pull request as the grading oracle. The original test split contains 2,294 issue–pull-request instances across 12 Python repositories; models receive the issue text and repository state, produce a patch, and are scored by whether the patch makes previously failing tests pass while preserving previously passing tests. SWE-bench+1
Verified therefore changes the denominator, not the scoring rule. It keeps the same broad setup—issue text, codebase, hidden tests, Dockerized execution, and percentage resolved—but excludes instances that human reviewers judged to be too underspecified, too test-fragile, or otherwise unsuitable as evaluation items. OpenAI The core claim is methodological: a smaller but cleaner sample can be a better measurement instrument when the full distribution contains Oracle Problem noise.
2. Background: why SWE-bench needed a curated subset
SWE-bench became influential because it moved coding evaluation away from isolated programming puzzles and toward realistic repository repair. Each task is drawn from a real issue and merged pull request; the model must inspect and edit a codebase rather than simply emit a function body. SWE-bench+1 In benchmark terms, this makes SWE-bench an Execution-Based Evaluation: the answer is not judged by textual similarity to a reference patch, but by whether the candidate patch passes tests in the repository environment. SWE-bench
That realism also creates a measurement problem. Real GitHub issues are not always self-contained specifications, and real pull-request tests are not always implementation-agnostic. OpenAI’s Verified writeup identified three recurring failure modes in the original SWE-bench: tests could be overly specific to the original contributor’s implementation, issue descriptions could omit information necessary to infer the intended fix, and some environments could fail for reasons unrelated to the candidate patch. OpenAI
The most important failure mode is not merely “hardness.” A hard instance is still valuable if the issue is clear and the tests correctly distinguish valid from invalid fixes. A bad benchmark instance is one where a competent engineer or model can produce a reasonable fix and still fail because the hidden oracle encodes unstated requirements, requires an arbitrary implementation detail, or tests behavior outside the issue statement. OpenAI’s example from scikit-learn showed a task whose tests required a specific DeprecationWarning message not stated in the issue, making the task “almost impossible” to infer from the provided problem statement. OpenAI
This matters because benchmark scores are used as proxies for model and agent capability. If the denominator includes many unsolvable or oracle-corrupted tasks, the score no longer estimates “ability to solve software-engineering issues under the benchmark interface”; it estimates a mixture of coding skill, inference under missing information, test-oracle luck, environment robustness, and sometimes benchmark-specific memorization. Verified was an attempt to reduce that confounding.
3. Design goal: remove pathological instances, not redefine the task
Verified’s design goal was to preserve the SWE-bench task format while filtering out cases where the grading setup was not a fair test of issue resolution. The official SWE-bench page says annotators ensured that problem descriptions were clear, test patches were correct, and tasks were solvable. SWE-bench OpenAI described the resulting set as 500 samples “verified to be non-problematic,” plus “easy” and “hard” subsets defined by estimated human completion time. OpenAI
That distinction is important. Verified is not a new benchmark family like SWE-bench Live, nor a language transfer benchmark like SWE-bench-Java. It is a curated slice through the original Python SWE-bench distribution. It still inherits many properties of the parent benchmark: Python projects, real issue/PR provenance, hidden tests, Docker execution, and scoring by resolved-rate. SWE-bench+1
The intended improvement is instance validity. In measurement terms, Verified tries to make “unresolved” mean “the system failed to solve a well-posed repository repair task,” rather than “the system encountered an ambiguous specification, invalid oracle, or broken execution setup.” That does not make Verified perfect. It makes the benchmark’s target estimand narrower and cleaner.
4. Construction methodology
OpenAI’s construction process began with a human-verification pass over 1,699 randomly sampled original SWE-bench instances. The review involved 93 software developers who annotated whether each task was underspecified, whether the FAIL_TO_PASS tests filtered out otherwise valid solutions, whether there were other major issues, and how difficult the task would be for an experienced human engineer. Each sample was labeled three times by separate annotators, and OpenAI used the highest-severity label across annotators as its conservative ensemble judgment. OpenAI
The filtering rule excluded instances with ensemble severity at or above 2 for underspecified problem statement, invalid or overly restrictive FAIL_TO_PASS tests, or other major problems. OpenAI reported that 38.3% of reviewed samples were underspecified, 61.1% had tests that unfairly filtered out valid solutions, and 68.3% were filtered out by the final criteria. OpenAI
After filtering for validity, OpenAI sampled the final 500 tasks. The procedure intentionally retained as many harder tasks as possible—especially tasks estimated to require one to four hours or more than four hours—before filling the remainder with randomly sampled eligible tasks. OpenAI This matters because Verified was not simply “the easiest 500 tasks.” Its construction attempted to remove bad items while preserving enough difficulty to remain useful.
OpenAI also released annotations, the rubric, Docker execution assets, and the dataset artifacts. The Verified release included 500 samples, an easy subset of 196 tasks estimated under 15 minutes for a human developer, a hard subset of 45 tasks estimated above one hour, and a new Docker-based harness. OpenAI
Construction summary
| Dimension | SWE-bench Verified design choice | Why it matters |
|---|---|---|
| Source pool | Randomly sampled instances from original SWE-bench | Keeps the benchmark connected to the original distribution rather than hand-authoring a separate task set. OpenAI |
| Reviewers | 93 software developers | Uses human software-engineering judgment rather than automated heuristics alone. OpenAI |
| Redundant labeling | Three annotators per sample | Reduces dependence on any single reviewer’s interpretation. OpenAI |
| Ensemble rule | Highest-severity label across annotators | Conservative: one serious objection can remove an instance. OpenAI |
| Exclusion targets | Underspecified issue statements, unfair tests, other major issues | Directly addresses oracle noise and infeasible tasks. OpenAI |
| Final size | 500 test instances | Smaller than full SWE-bench, but large enough for common leaderboard reporting. SWE-bench |
| Difficulty treatment | Retained harder valid tasks where possible | Avoids equating “verified” with “easy.” OpenAI |
5. What “verified” does and does not mean
“Verified” means “human-reviewed for benchmark validity under the release rubric.” It does not mean every instance is unambiguously specified in an absolute sense, every hidden test is implementation-agnostic, or every passing patch is semantically correct outside the test suite.
This distinction became more important after OpenAI’s 2026 reassessment. In February 2026, OpenAI argued that SWE-bench Verified no longer measured frontier coding capability well enough, citing both residual test-quality problems and contamination risks from public benchmark artifacts. OpenAI In that reassessment, OpenAI audited 138 Verified problems that OpenAI o3 did not consistently solve across repeated runs; each was reviewed by at least six experienced engineers, and 59.4% of that targeted failure set had material issues. OpenAI
That audit should not be read as “59.4% of all Verified tasks are bad.” The sample was not a random sample of Verified; it was selected from tasks a frontier model did not consistently solve. But it is strong evidence that human curation reduced noise rather than eliminated it. It also reinforces the general lesson: Human Verification improves benchmark quality, but it does not abolish the Test Oracle problem.
6. Scoring: the resolved-rate metric
SWE-bench variants generally report the percentage of instances resolved. A task is resolved when the system’s patch passes the relevant tests in the Dockerized evaluation environment: previously failing tests associated with the issue should pass, and previously passing tests should remain passing. SWE-bench The official leaderboard reports % Resolved over different denominators: 2,294 for Full, 500 for Verified, 300 for Lite and Multilingual, and 517 for Multimodal. SWE-bench
For Verified, the denominator is cleaner than the full set because instances with serious annotation problems were removed. This is why Verified scores are typically higher than full SWE-bench scores for comparable systems. A higher Verified score does not automatically mean the system improved; it can mean the benchmark variant has fewer impossible or unfair cases.
The best way to read a Verified score is: “Within this curated 500-instance subset, under this agent scaffold, model, tool budget, and evaluation harness, what fraction of human-screened repository repair tasks were resolved?” That is a useful measurement. It is not the same as “probability that this model can resolve an arbitrary GitHub issue.”
7. Empirical trajectory: why Verified scores rose above Full
The original SWE-bench was extremely difficult for early language-model agents. The official page reports that an early retrieval-augmented Claude 2 baseline solved 1.96% of the test set, while SWE-agent later reached 12.47% on the same benchmark family. SWE-bench OpenAI’s 2024 Verified writeup noted that, as of August 5, 2024, the top full SWE-bench submission was below 20%, while the top SWE-bench Lite submission was below 43%. OpenAI
When OpenAI compared GPT-4o under its best scaffold across benchmark variants, performance rose substantially on Verified. The writeup reports GPT-4o at 33.2% on SWE-bench Verified, compared with 16% on full SWE-bench. OpenAI OpenAI also analyzed performance by difficulty category and argued that the Verified gain was not merely due to selecting easier tasks: performance improved within difficulty categories as well, consistent with the hypothesis that curation removed infeasible or unfair instances. OpenAI
The empirical pattern is therefore not surprising. If a benchmark contains a mixture of fair tasks and oracle-corrupted tasks, then removing the corrupted tasks raises measured performance even when the underlying model is unchanged. Verified’s higher scores are partly a sign of easier measurement conditions and partly a sign of better construct validity.
By 2026, the trajectory had changed again. OpenAI wrote that SWE-bench Verified had become a standard metric, but that frontier progress on the benchmark had slowed from 74.9% to 80.9% over the prior six months. OpenAI Third-party comparative leaderboards on May 12, 2026 reported even higher headline numbers—for example, LLM-Stats listed a top SWE-bench Verified score of 93.9—but the same page marked all 89 listed results as self-reported and zero as independently verified, so those numbers should be treated as a dynamic leaderboard snapshot rather than a primary scientific result. LLM Stats
8. Methodological contribution: reducing oracle noise
The central contribution of Verified is methodological, not just statistical. It made benchmark curation itself part of the SWE-bench conversation. Instead of treating the original scraped distribution as ground truth, OpenAI asked whether each instance was actually a fair evaluation item. OpenAI
This was important because SWE-bench-style benchmarks depend on a proxy oracle. The “correctness” of a patch is not judged by maintainers, user outcomes, or broad semantic equivalence; it is judged by a hidden test suite. Hidden tests are better than string matching, but they are still incomplete. If tests encode unstated implementation details, then the benchmark rewards recovering the original pull request rather than solving the described problem. That failure mode is especially serious once models can inspect large contexts and infer benchmark artifacts.
Verified’s curation pass improved the signal in three ways.
First, it reduced underspecification. If an issue statement does not contain enough information to infer the intended behavior, then failure is not necessarily evidence of weak coding ability. OpenAI’s annotation rubric explicitly targeted such cases. OpenAI
Second, it reduced overly narrow test oracles. If FAIL_TO_PASS tests reject valid alternative implementations, the benchmark becomes a test of reproducing the original patch. Verified filtered for cases where tests were judged to be fairer. OpenAI
Third, it forced clearer benchmark reporting. Once Full, Lite, Verified, Multimodal, Multilingual, Live, and Java variants exist, a single “SWE-bench score” is underspecified. The variant, scaffold, model, budget, date, and evaluation harness become part of the result.
9. Verified versus Full SWE-bench
The full SWE-bench distribution is larger and closer to the original scraped population of real Python GitHub issue fixes. It includes 2,294 test instances across 12 repositories, and it preserves more of the messiness of real issue/PR data. SWE-bench Verified is smaller, at 500 instances, but intentionally cleaner. SWE-bench
The case for Full is representativeness. If the goal is to estimate performance on the broad population of GitHub-derived issue repairs that SWE-bench originally sampled, Full matters. Removing problematic tasks may also remove some authentic forms of software work: ambiguous tickets, partial specifications, noisy tests, and environment issues are all real features of production engineering.
The case for Verified is measurement validity. If the goal is to compare coding agents on well-posed tasks with less oracle noise, Full is too contaminated by invalid instances. OpenAI’s human review found high rates of underspecification and unfair tests in the sampled original pool, which supports using Verified for cleaner model comparison. OpenAI
The practical conclusion is that neither variant is “the real benchmark” in all contexts. Full better represents the original scraped distribution; Verified better represents the subset of tasks where the benchmark oracle is plausibly fair. The difference is an estimand choice.
Full versus Verified
| Question | Full SWE-bench | SWE-bench Verified |
|---|---|---|
| What is the denominator? | 2,294 test instances from the original Python SWE-bench test split. SWE-bench+1 | 500 human-validated instances. SWE-bench+1 |
| What does it measure best? | Performance over the original scraped issue/PR distribution. | Performance on a curated set of clearer, more solvable issue-resolution tasks. |
| Main strength | Breadth and continuity with the original benchmark. | Reduced noise from underspecified tasks and unfair tests. |
| Main weakness | Confounds model weakness with bad benchmark instances. | Smaller and less representative of the full scraped distribution. |
| Best use | Broad historical comparison and distributional stress testing. | Cleaner model/scaffold comparison and benchmark-methodology work. |
| Frontier status in 2026 | Still useful, but old and public. | Historically important, but OpenAI says it no longer suffices for frontier claims. OpenAI |
10. Verified versus SWE-bench Lite
SWE-bench Lite is often confused with Verified because both are smaller than Full and both are frequently used for faster evaluation. But their selection criteria differ.
The official Lite page describes Lite as a 300-task subset created to reduce evaluation cost, support faster iteration, and make SWE-bench more accessible, while preserving a distribution and difficulty spectrum. SWE-bench Lite filters for tasks that are more self-contained and functional-bug-fix-like: it removes examples with images, external links, commit SHAs, references to other pull requests/issues, very short problem statements, multi-file edits, large patches, file creation/removal, and tests with error-message checks. SWE-bench
Verified’s criterion is different: it is a human-validated quality filter. It removes tasks judged underspecified, unfairly tested, or otherwise problematic. OpenAI Lite is primarily a lower-cost and more tractable subset; Verified is primarily a benchmark-validity subset.
OpenAI’s writeup explicitly distinguished the two. It noted that both Lite and Verified skew easier than the original benchmark, leaving fewer than 10% of tasks above one hour of estimated human time, but that Lite was subsampled to make the benchmark easier whereas Verified attempted to remove infeasible samples. OpenAI
Lite versus Verified
| Dimension | SWE-bench Lite | SWE-bench Verified |
|---|---|---|
| Size | 300 test tasks. SWE-bench+1 | 500 test tasks. SWE-bench+1 |
| Primary purpose | Lower-cost, faster, more accessible evaluation. SWE-bench | Human-validated reduction of problematic instances. SWE-bench+1 |
| Filter type | Heuristic task-scope filters. | Human software-engineer review. |
| Typical effect | Easier and cheaper to run. | Cleaner solvability/test-quality floor. |
| Best use | Rapid iteration, budget-constrained experiments, early agent debugging. | More defensible headline comparison than Full or Lite when reporting a legacy SWE-bench-family score. |
| Main risk | May overrepresent simple, self-contained bug fixes. | May still contain residual oracle issues and is now public enough to face contamination risk. OpenAI |
11. Which variant should practitioners report?
A practitioner should not report “SWE-bench” without specifying the variant. The official leaderboard now separates tabs for Verified, Multilingual, Lite, Full, and Multimodal, and reports different denominators for each. SWE-bench A meaningful result should at minimum include the benchmark variant, model, agent scaffold, number of attempts or rollouts, tool budget, evaluation harness, date, and whether the result is self-reported or independently reproduced.
For historical comparison on Python issue repair, Verified is usually the most interpretable single number because it reduces known instance-quality noise. For fast iteration, Lite is more practical. For broad original-distribution comparison, Full still matters. For claims about modern frontier coding ability, Verified should not be the only reported benchmark: OpenAI explicitly stopped reporting SWE-bench Verified for frontier launches and recommended newer, more contamination-resistant evaluations such as SWE-bench Pro until better public uncontaminated evaluations are available. OpenAI
A robust report should therefore include multiple views:
| Reporting goal | Recommended benchmark choice | Rationale |
|---|---|---|
| Legacy comparability | Verified, plus Full if budget allows | Verified is widely used and less noisy; Full preserves continuity with the original distribution. |
| Low-cost agent iteration | Lite | Smaller, faster, and explicitly designed for more accessible evaluation. SWE-bench |
| Claims about frontier coding agents | Verified only as a historical diagnostic; prefer Live, Pro, private held-out, or freshly curated tasks | Verified is public, increasingly saturated, and subject to contamination concerns. OpenAI |
| Language breadth | Multilingual or language-specific variants such as Java | Original SWE-bench and Verified are Python-centered. SWE-bench+1 |
| Contamination resistance | SWE-bench Live or held-out/private variants | Live is designed around monthly updates and more recent issues. SWE-bench Live+1 |
| Scaffold comparison | Same benchmark, same scaffold, same budget | SWE-bench results are strongly scaffold-dependent, not just model-dependent. OpenAI emphasized the importance of understanding scaffolding differences when interpreting results. OpenAI |
The strongest practitioner convention is to report a panel rather than a single number: Verified for comparability, Lite for cheap iteration, Full for original-distribution stress, and a newer contamination-resistant benchmark for frontier claims. A single Verified score is still useful, but it is no longer sufficient evidence of general software-engineering competence.
12. Relationship to SWE-bench Live
SWE-bench Live is best understood as a response to the static-benchmark problem. Its official page describes it as a live benchmark with monthly updates via automated curation, intended to provide up-to-date tasks and more contamination-resistant evaluation. SWE-bench Live The SWE-bench Live paper argues that standard SWE-bench variants suffer from being static, narrowly distributed, and manually curated, which increases risks of overfitting and contamination. Its initial release included 1,319 tasks from GitHub issues since 2024, spanning 93 repositories, with dedicated Docker environments and automated curation. arXiv
Live and Verified address different weaknesses. Verified addresses instance validity: “Are these tasks fair and solvable under the benchmark interface?” Live addresses staleness and contamination: “Are these tasks recent enough and refreshed often enough that models are less likely to have seen them?” A high-quality future benchmark ideally needs both properties.
The official Live page also shows the benchmark expanding across operating systems and languages, with updates adding newly verified high-quality issues monthly and supporting categories such as C/C++, C#, TypeScript/JavaScript, Go, Rust, Java, and Windows tasks. SWE-bench Live This places Verified in historical context: Verified was the key curated subset of the original Python SWE-bench era, while Live pushes toward continuously refreshed software-engineering evaluation.
13. Relationship to SWE-bench-Java
SWE-bench-Java addresses a different limitation: language scope. The original SWE-bench family was centered on Python repositories. The SWE-bench-Java paper describes the Java variant as a dataset, Docker-based evaluation framework, and leaderboard for real-world Java GitHub issues, presented as a step toward more multilingual software-engineering evaluation. arXiv
Its construction pipeline collects candidate Java repositories and pull requests, links pull requests to issues, selects cases with test changes, and builds an evaluation setup where resolution is measured by passing tests. arXiv Like Verified, SWE-bench-Java emphasizes manual verification and benchmark quality, but it is not simply a Java translation of Verified. It is a language-specific benchmark built from Java projects, with its own dataset construction constraints and ecosystem assumptions.
The broader methodological point is that Verified solved only one axis of the benchmark-design problem. It improved curation for a static Python subset. It did not solve multilingual coverage, contamination, long-horizon task realism, or the mismatch between unit-test pass rate and production-quality software changes.
14. Comparative map of SWE-bench-family variants
| Variant | Main purpose | Size / scope | Primary caveat |
|---|---|---|---|
| SWE-bench Full | Original real-world Python issue-resolution benchmark | 2,294 test instances across 12 Python repositories. SWE-bench+1 | Larger but noisier; includes ambiguous or unfairly tested instances. |
| SWE-bench Lite | Lower-cost and more accessible subset | 300 test tasks, selected with scope and tractability filters. SWE-bench+1 | Easier and narrower; not mainly a human-validity filter. |
| SWE-bench Verified | Human-validated subset | 500 human-reviewed tasks. SWE-bench | Cleaner but still public, static, and imperfect. |
| SWE-bench Multilingual | Language breadth beyond Python | 300 curated tasks from 42 repositories and 9 languages. SWE-bench+1 | Smaller per-language coverage; cross-language infrastructure complexity. |
| SWE-bench Live | Freshness and contamination resistance | Monthly-updated live benchmark; initial release reported 1,319 tasks from issues since 2024 across 93 repositories. SWE-bench Live+1 | Automated curation and freshness do not automatically guarantee perfect oracle quality. |
| SWE-bench-Java | Java-specific repository repair | Java benchmark with dataset, Docker evaluation, and leaderboard. arXiv | Ecosystem-specific; not a direct replacement for Python Verified. |
15. The open question: is Verified the “real” benchmark?
There are two defensible answers, depending on the target.
If the target is fair measurement of code-agent capability on well-posed tasks, Verified is closer to the “real” benchmark than Full. The original benchmark includes many items where a system can fail because the issue is underspecified or the tests encode unstated requirements. OpenAI’s human review found those problems at high rates in the sampled original pool, and Verified was constructed specifically to reduce them. OpenAI
If the target is performance on the messy distribution of real issue reports, Full still matters. Real software engineering includes ambiguous tickets, failing environments, incomplete repro steps, and brittle tests. A system that only performs well on carefully filtered tasks may not be robust to the full socio-technical mess of real maintenance. The problem is that Full does not cleanly separate “model cannot handle ambiguity” from “benchmark instance is not a fair task.”
A better framing is that Verified and Full answer different questions:
| Benchmark | Approximate estimand |
|---|---|
| Full | “How often does the system resolve instances from the original SWE-bench scraped distribution?” |
| Verified | “How often does the system resolve human-screened SWE-bench instances that are less likely to be underspecified or unfairly tested?” |
| Lite | “How often does the system resolve cheaper, more self-contained SWE-bench-like tasks?” |
| Live | “How often does the system resolve newer, less-contaminated SWE-bench-like tasks?” |
| Java / Multilingual | “How well does the system transfer repository repair to non-Python ecosystems?” |
Under this framing, there is no single real benchmark. There are multiple benchmark instruments, each with a different failure surface.
16. Current status and interpretation in 2026
By 2026, SWE-bench Verified had become both influential and compromised as a frontier metric. It remains a clean historical reference point because many systems report it and because the construction methodology is documented. But OpenAI’s February 2026 reassessment argued that Verified had two serious problems for frontier evaluation: some tests still reject correct solutions, and public benchmark artifacts create contamination risk. OpenAI
OpenAI’s contamination analysis included examples where models could reproduce exact gold-patch-like details from short hints or benchmark-specific prompts, and the company concluded that publicly available benchmark material carries contamination risks for frontier model evaluation. OpenAI+2OpenAI+2 The same writeup recommended stronger evaluation practices: privately authored tasks, robust implementation-agnostic tests, and benchmarks less exposed to public training data. OpenAI
That does not make Verified obsolete. It makes it a legacy benchmark with a clear use profile. Verified is still valuable for comparing open scaffolds, debugging agent designs, and studying the effects of curation on benchmark validity. It is weaker as a claim about the current frontier, especially when results are self-reported, highly scaffolded, or obtained after many rollouts.
17. Practical reporting checklist
A serious SWE-bench Verified report should state the following:
| Field | Why it matters |
|---|---|
| Benchmark variant and split | “SWE-bench” alone is ambiguous across Full, Lite, Verified, Multimodal, Multilingual, Live, and other variants. SWE-bench |
| Dataset date or version | Static and live variants differ; leaderboard snapshots change. |
| Agent scaffold | OpenAI emphasized that scaffold differences materially affect SWE-bench performance. OpenAI |
| Model checkpoint | Scores are model-version-specific. |
| Number of rollouts / attempts | Multi-rollout systems can improve pass rates in ways that are not comparable to single-attempt systems. |
| Tool access and context budget | Repository repair depends heavily on search, editing, execution, and context-management tools. |
| Evaluation harness | Docker image and test execution details can affect reproducibility. |
| Verification status | Independent reproduction is stronger than self-reported leaderboard entries; some comparative leaderboards explicitly mark results as self-reported. LLM Stats |
| Per-repository and per-difficulty breakdowns | Aggregate resolved-rate can hide scaffold or repository-specific weaknesses. |
| Contamination controls | Public static benchmarks are vulnerable to training-data and prompt-leakage effects. OpenAI |
The simplest honest headline is not “Model X gets Y on SWE-bench.” It is: “Model X with scaffold S resolves Y% on SWE-bench Verified, evaluated on date D with harness H and attempt budget B.” For frontier claims, add a fresher benchmark such as Live or a private held-out set.
18. Bottom line
SWE-bench Verified is best understood as the curation turn in SWE-bench evaluation. It showed that benchmark quality is not just a matter of task realism or dataset size; it also depends on whether each instance is a fair, solvable, implementation-agnostic test of the stated problem. OpenAI’s human-verification pass made the benchmark more interpretable by removing many underspecified and unfairly tested tasks, which explains why Verified scores are typically higher than Full scores under comparable systems. OpenAI
The unresolved issue is whether a curated subset should become the canonical benchmark. Verified is cleaner than Full, but Full better preserves the original scraped distribution. Verified is more interpretable than Lite, but Lite is cheaper. Verified is historically important, but Live, Java, Multilingual, Pro-style held-out evaluations, and private benchmarks address weaknesses that Verified does not. In 2026, the intellectually honest position is that Verified is not “the real SWE-bench” or “obsolete”; it is a useful, human-curated measurement instrument whose score must be reported with variant, scaffold, contamination, and frontier-status caveats.
Companion entries
Core theory:
-
Execution-Based Evaluation
-
Test Oracle
-
Oracle Problem
-
Benchmark Validity
-
Human Verification
SWE-bench variants:
-
SWE-bench Lite
-
SWE-bench Full
-
SWE-bench-Java
-
SWE-bench Multilingual
-
SWE-bench Multimodal
Practice:
-
Benchmark Reporting for Coding Agents
-
Agent Scaffold
-
Dockerized Evaluation Harness
-
FAIL_TO_PASS and PASS_TO_PASS
-
Repository-Level Code Editing
-
Multi-Rollout Evaluation
Counterarguments and limitations:
-
The Full Distribution Problem
-
Human Verification Does Not Eliminate Oracle Risk
-
Static Benchmark Saturation
-
Leaderboard Overfitting
-
Unit Tests as Incomplete Specifications