SWE-bench Pro: Larger Repositories, Harder Issues
SWE-bench Pro is a benchmark for Coding Agents that asks whether high scores on earlier repository-repair benchmarks generalize to larger, less familiar, more industrial software systems. Its central claim is not merely that “harder issues lower scores,” but that benchmark construction must test long-horizon repository navigation, multi-file edits, contamination resistance, and robust test verification together.
Coverage note: verified through May 19, 2026. Current leaderboard values should be treated as provisional where noted, because the SWE-bench Pro repository posted a May 18 notice that it had “identified some issues with the leaderboard” and was working on them. GitHub
Position in the SWE-bench family
SWE-bench established repository-level issue resolution as a standard coding-agent task: an agent receives a codebase and issue description, edits the repository, and is judged by whether hidden tests pass. The original benchmark used 2,294 real GitHub issues from 12 Python repositories, making it substantially more realistic than function-level coding tests such as HumanEval, but still concentrated in a small number of popular open-source Python projects. arXiv
SWE-bench Verified was introduced later as a human-filtered subset of 500 SWE-bench instances intended to remove underspecified or unfair tasks. OpenAI and the SWE-bench authors built it by reviewing 1,699 SWE-bench samples, filtering out instances with severe problem-statement or test-validity issues, and retaining a 500-task subset with professional-developer annotation. OpenAI
SWE-bench Pro extends the issue-resolution paradigm in the opposite direction: instead of mainly cleaning the original task pool, it changes the task distribution. The benchmark contains 1,865 problems from 41 actively maintained repositories, divided into a 731-instance public set, a 276-instance commercial/private set, and an 858-instance held-out set. Its public and held-out sets use copyleft-licensed open-source repositories, while the commercial set uses proprietary startup repositories. arXiv
The practical question is whether SWE-bench Pro should replace SWE-bench Verified as the main score reported for frontier coding agents. The best answer is conditional: Pro is the stronger flagship benchmark for long-horizon autonomous patching, but it should remain part of a benchmark portfolio because it still evaluates a narrow slice of software engineering: single-turn issue resolution by test-passing patches.
Design thesis: generalization from small repositories to large ones
SWE-bench Pro is built around a methodological pressure test: do agent systems that perform well on relatively compact, public, Python-heavy issue-resolution tasks still perform well when the repository is larger, the edit is longer, the language distribution is broader, and contamination is harder?
The Scale paper states this explicitly. SWE-bench Pro is “designed to capture realistic, complex, enterprise-level problems beyond the scope of SWE-bench,” with tasks that may take professional engineers “hours to days,” often requiring multi-file and substantial code modifications. The paper reports that models exceeding 70% on SWE-bench Verified scored below 25% on the initial SWE-bench Pro evaluation under a unified SWE-Agent scaffold. arXiv
The benchmark’s design has four linked components:
| Design dimension | SWE-bench Verified tendency | SWE-bench Pro design choice | Why it matters |
|---|---|---|---|
| Repository distribution | 12 open-source Python repositories | 41 professional repositories across public, held-out, and private subsets | Tests repository generalization rather than familiarity with a small repo family |
| Task size | Many small patches; Pro paper notes 161/500 Verified tasks require one- or two-line modifications | Every Pro task requires at least 10 lines changed; average gold patch is 107.4 LOC across 4.1 files | Forces multi-file localization, planning, and integration |
| Contamination surface | Public GitHub repos that may be in pretraining corpora | GPL/copyleft public and held-out repos plus proprietary startup repos | Reduces, but does not eliminate, training-data leakage risk |
| Specification style | Ambiguous tasks filtered out | Ambiguous tasks clarified through human-written requirements and optional interface information | Preserves harder tasks while making them resolvable |
The important methodological move is that SWE-bench Pro does not merely “make SWE-bench bigger.” It changes the failure regime. A small-repository patch often tests whether an agent can locate a bug and write a local fix. A large-repository Pro task more often tests whether an agent can build an internal model of unfamiliar architecture, select the relevant files without drowning in context, implement coordinated changes, and avoid regressions.
Dataset construction
Repository selection
SWE-bench Pro selects repositories for professional realism, popularity, project scale, domain diversity, and the ability to extract end-to-end problems from commit history. The public repositories are intended to represent a spread of programming languages, project scales, and application domains; private repositories are sourced from Scale’s internal assets, affiliated startups, acquisitions, and external data partnerships. arXiv
The benchmark’s public and held-out open-source repositories are selected from strong copyleft licenses such as GPL. The stated rationale is contamination resistance: permissively licensed public code is more likely to appear in commercial pretraining corpora, while copyleft licensing creates legal friction for inclusion. This is not a mathematical guarantee of non-contamination, but it is a concrete design improvement over benchmark data scraped from widely used permissive repositories. arXiv
The commercial/private subset is even more important conceptually. Because those repositories are not publicly accessible, they function as a stronger test of generalization to unseen codebases. The private leaderboard describes this subset as 276 instances from 18 proprietary startup codebases acquired through partnerships and not publicly accessible. Scale Labs
Issue harvesting
SWE-bench Pro extracts problems from commit history. The construction process identifies pairs of consecutive commits, treating the older commit as the base and the newer commit as the resolved instance. Test-file diffs become the test patch; non-test diffs become the gold patch. A valid problem must correspond to a bug fix or feature addition and must exhibit a fail-to-pass transition: applying the test patch to the base commit should fail, while applying both the test patch and gold patch should pass. arXiv
This preserves the basic SWE-bench logic: a model sees the repository state before the fix and a natural-language task description, then must generate a patch. The evaluation system, not the model, sees the tests.
Difficulty calibration
SWE-bench Pro calibrates difficulty by filtering for nontrivial reference solutions. The paper says it excludes trivial 1–10 line edits and keeps problems requiring substantial multi-file modifications; the average reference solution spans 107.4 lines across 4.1 files, every task involves at least 10 changed lines, and more than 100 tasks require over 100 lines of modification. arXiv
This is a blunt but useful proxy. Lines changed and files touched are not equivalent to conceptual difficulty: a ten-line type-system fix can be harder than a hundred-line mechanical migration. But for agent evaluation, larger patches are more likely to expose the failure modes that matter in practice: context loss, incomplete call-site updates, incorrect assumptions about architecture, fragile search behavior, and regressions outside the local bug.
The benchmark also caps repository contribution to reduce overfitting. The paper says each repository contributes roughly 50–100 instances, with a strict cap of 100, so a system cannot dominate the leaderboard merely by specializing in one repository. arXiv
Human augmentation
A central tension in repository-level benchmarks is ambiguity. Real issues are often under-specified; benchmark tasks must be solvable without hidden human conversation. SWE-bench Verified mostly addressed this by filtering out ambiguous or unfair tasks. SWE-bench Pro instead uses human augmentation to retain harder tasks while clarifying their requirements.
Each SWE-bench Pro task includes a human-augmented problem statement, requirements, and optional interface specification. The problem statement is derived from commits, pull requests, and issues, then rewritten into issue style with missing information added when necessary. Requirements clarify expected behavior but are not supposed to prescribe the implementation. Interface specifications name expected classes or functions when misnaming would otherwise create false negatives. arXiv
This design choice is double-edged. It makes the benchmark more reliable as an implementation test: agents are asked to implement a specified repair rather than infer hidden maintainer intent. But it also removes part of real software engineering: discovering what should be built through exploration, product judgment, and follow-up questions. The paper itself acknowledges that reducing ambiguity may make tasks too prescriptive relative to real-world work. arXiv
Environment and test verification
SWE-bench Pro uses Docker-based, language-specific environments with dependency resolution. Python tasks use isolated virtual environments, JavaScript and TypeScript tasks use Node.js with npm or yarn, and Go tasks use module-aware environments. arXiv
The evaluation relies on two test classes inherited from SWE-bench: fail-to-pass tests and pass-to-pass tests. Fail-to-pass tests verify that the issue has been fixed; pass-to-pass tests verify that unrelated existing functionality still works. SWE-bench Pro runs tests multiple times and filters out flaky tests, then sends fail-to-pass tests through human verification to remove tests that are irrelevant or too broad. If all relevant tests for a problem are too broad or irrelevant, the problem is dropped. arXiv
This is one of the benchmark’s most valuable engineering contributions. In code-agent evaluation, a poor test suite can make the score meaningless in both directions: correct patches may fail overspecified tests, and bad patches may pass undercovered tests. SWE-bench Pro does not solve the verifier problem, but it raises the standard for test construction by combining repeated execution with human review.
Methodology contribution: from “can patch” to “can generalize”
SWE-bench Pro’s main scientific contribution is not a new metric. The primary metric remains resolve rate: the percentage of tasks for which the submitted patch fixes the issue and preserves existing functionality. The Scale leaderboard defines a resolved task as one where the patch passes fail-to-pass tests and pass-to-pass tests inside the evaluation environment. scale.com
The contribution is distributional. SWE-bench Pro asks whether success on earlier issue-resolution benchmarks transfers to larger, more complex, less contaminated, more industrial repositories. That matters because earlier SWE-bench-family scores were increasingly hard to interpret at the frontier.
OpenAI’s February 2026 audit argued that SWE-bench Verified no longer measures frontier coding capability well. It reported two problems: flawed tests that reject functionally correct submissions, and evidence that frontier models had seen at least some benchmark problems or solutions during training. OpenAI said it stopped reporting SWE-bench Verified scores and recommended reporting SWE-bench Pro until better uncontaminated evaluations are available. OpenAI
This recommendation should not be read as “SWE-bench Pro is perfect.” OpenAI’s own audit says Pro appears to suffer less from contamination, but also says it is not perfect. Its contamination pipeline found fewer and less severe contamination cases on Pro than on Verified, and no tested model reproduced a complete verbatim gold patch. OpenAI
The deeper lesson is that benchmark validity decays. A static public benchmark can be a good research instrument when released, then become a contaminated optimization target as models, scaffolds, and training corpora adapt. SWE-bench Pro is a response to that decay: harder tasks, stronger contamination controls, private splits, and held-out repositories.
Empirical scores
Paper baseline: September 2025 unified SWE-Agent scaffold
The SWE-bench Pro paper evaluated models using SWE-Agent, with a maximum of 200 turns, the same prompt for all models, and augmented task descriptions that included problem statements, requirements, and interface specifications. This setting aimed to reduce ambiguity and compare models under a common scaffold. arXiv
On the 731-task public set, the initial reported scores were:
| Model | SWE-bench Pro public resolve rate |
|---|---|
| OpenAI GPT-5 | 23.3% |
| Claude Opus 4.1 | 22.7% |
| Claude Sonnet 4 | 17.6% |
| Gemini 2.5 Pro Preview | 13.5% |
| SWE-Smith-32B | 6.8% |
| OpenAI GPT-4o | 4.9% |
| Qwen-3 32B | 3.4% |
On the 276-task commercial/private set, scores were lower:
| Model | SWE-bench Pro commercial/private resolve rate |
|---|---|
| Claude Opus 4.1 | 17.8% |
| OpenAI GPT-5 | 14.9% |
| Gemini 2.5 Pro Preview | 10.1% |
| Claude Sonnet 4 | 9.1% |
| OpenAI GPT-4o | 3.6% |
The performance drop from public to commercial/private tasks is important. It suggests that private, enterprise-like repositories are not just less contaminated; they may be structurally harder for agents because of build complexity, domain-specific abstractions, dependency layering, and less familiar code patterns. arXiv
Current Scale public leaderboard: May 2026 snapshot
By May 2026, current leaderboard scores were much higher than the paper baseline. On the Scale SWE-bench Pro public leaderboard, the top visible entries were:
| Rank group | Model / system label | Public-set score |
|---|---|---|
| 1 | gpt-5.4 (xHigh) | 59.10 ± 3.56 |
| 1 | Muse Spark | 55.00 ± 3.60 |
| 2 | claude-opus-4-6 (thinking) | 51.90 ± 3.61 |
| 3 | gemini-3.1-pro (thinking) | 46.10 ± 3.60 |
| 3 | claude-opus-4-5-20251101 | 45.89 ± 3.60 |
| 4 | claude-4-5-Sonnet | 43.60 ± 3.60 |
| 4 | gemini-3-pro-preview | 43.30 ± 3.60 |
| 4 | claude-4-Sonnet | 42.70 ± 3.59 |
| 4 | gpt-5-2025-08-07 (High) | 41.78 ± 3.49 |
| 4 | gpt-5.2-codex | 41.04 ± 3.57 |
Scale’s public leaderboard notes that grayed-out results were run with capped cost and 50-turn limits, while other visible results used uncapped cost and a 250-turn limit; starred entries were run with the mini-SWE-agent harness. scale.com
This comparison highlights a subtle but crucial point for Agent Scaffolding: the benchmark did not become easy simply because the top number rose from ~23% to ~59%. The 2025 paper baseline and 2026 leaderboard are not identical experimental conditions. Newer models, higher turn limits, mini-SWE-agent runs, and scaffold changes all contribute. A leaderboard is an ecosystem measure; a controlled benchmark run is a model-comparison measure.
Current Scale private leaderboard: May 2026 snapshot
The private/commercial leaderboard remained lower than the public leaderboard. The top visible private-set entries were:
| Rank group | Model / system label | Private-set score |
|---|---|---|
| 1 | claude-opus-4-6 (thinking) | 47.10 ± 6.07 |
| 1 | Muse Spark | 44.70 ± 6.05 |
| 1 | gpt-5.4 (xHigh) | 43.40 ± 6.03 |
| 3 | gemini-3.1-pro (thinking) | 32.20 ± 5.69 |
| 4 | GPT 5.2 Codex | 27.74 ± 5.09 |
| 4 | GPT 5.2 | 23.81 ± 5.09 |
| 4 | Opus 4.5 | 23.44 ± 5.07 |
| 5 | Gemini 3 Pro | 17.95 ± 4.78 |
| 6 | Claude Opus 4.1 | 17.75 ± 4.51 |
| 6 | OpenAI GPT-5 | 14.86 ± 4.20 |
The private leaderboard describes the private dataset as commercial-grade repositories and repeats the benchmark’s four challenges: contamination, limited task diversity, oversimplified problems, and unreliable or irreproducible testing. Scale Labs
The private-set gap is one of the benchmark’s strongest signals. Public Pro can be studied, optimized against, and eventually overfit. Private Pro is closer to the deployment scenario: unfamiliar code, unfamiliar conventions, fewer public traces, and less chance that the model has memorized the fix.
Comparison with SWE-bench Verified
SWE-bench Verified remains widely recognized and useful historically, but its frontier-model role is now contested. The official SWE-bench page describes Verified as a 500-instance human-filtered subset and reports entries by percentage resolved; it also distinguishes the full leaderboard from mini-SWE-agent comparisons that aim to evaluate language models under a simpler common bash-like setup. SWE-bench+1
Recent official SWE-bench leaderboard snippets show top mini-SWE-agent Verified scores in the mid-70s, including Claude 4.5 Opus at 76.80% and Gemini 3 Flash at 75.80%. SWE-bench Some vendor or custom-scaffold reports claim higher Verified numbers; for example, MiniMax reported 80.2% on SWE-bench Verified for M2.5. MiniMax
Those scores should not be compared naively to SWE-bench Pro scores. Verified is smaller, Python-only, public, older, and more contaminated. Pro is larger, more multilingual within its four-language range, less contaminated by design, and more demanding in patch size. OpenAI’s 2026 analysis explicitly concluded that Verified improvements increasingly reflect training exposure and flawed tests rather than real frontier coding progress, and recommended Pro instead. OpenAI
What the score gap means
The empirical pattern is clear: SWE-bench Pro lowers success rates relative to SWE-bench Verified. The interpretation is more nuanced.
First, the gap shows that agents do not simply “know how to fix GitHub issues” in a repository-independent way. Repository size, architecture, language ecosystem, and task length matter. The Pro paper reports that Go and Python tasks generally showed higher resolve rates, while JavaScript and TypeScript performance varied more widely. It also reports strong repository-specific variation: some repositories stayed below 10% across models, while others allowed some models to exceed 50%. arXiv
Second, the gap separates model capability from scaffold capability only imperfectly. The same underlying model can score differently depending on search tools, context management, turn limits, retry policies, and validation loops. OpenAI’s original Verified writeup warned that scaffold differences can substantially change scores for the same model, and its preparedness framing calls for evaluations to account for external system enhancements. OpenAI
Third, the gap may partly reflect benchmark design choices rather than raw “real-world difficulty.” SWE-bench Pro clarifies requirements and interfaces, which can make some tasks less ambiguous than real work. But it also relies on tests extracted from real changes, which may still under-specify acceptable alternative solutions. The paper lists test-suite dependence, reduced ambiguity, and limited language coverage as limitations. arXiv
The safest reading is this: SWE-bench Pro is a stronger test of long-horizon repository repair than SWE-bench Verified, but its score is still a partial proxy. A 50% Pro score does not mean an agent can autonomously do half of a professional engineer’s work. It means that, under the benchmark harness, it can solve about half of a particular distribution of issue-resolution tasks with test-verifiable patches.
Failure modes
SWE-bench Pro’s failure analysis is useful because it moves beyond a single pass rate. The paper uses an LLM-as-judge analysis over failed trajectories and reports different failure profiles by model.
For frontier models, failures are often semantic. Claude Opus 4.1’s unresolved submitted patches were heavily categorized as wrong solution or syntax error; GPT-5 showed fewer wrong-solution failures among submitted outputs but a high not-submitted fraction. Claude Sonnet 4 showed context-overflow and endless-file-reading behaviors, suggesting that repository navigation and context management are first-class bottlenecks. Qwen3 32B showed high tool-error rates, emphasizing that tool integration is not a solved problem for open-weight agents. arXiv
These failure modes align with what practitioners observe in real coding-agent use:
| Failure mode | What it looks like in SWE-bench Pro | Why it matters in practice |
|---|---|---|
| Wrong semantic model | Patch compiles but implements the wrong behavior | The agent found code, but misunderstood the system |
| Context thrashing | Repeated file reading, failure to converge, context overflow | Larger repos punish weak search and summarization |
| Tool-use failure | Bad commands, malformed edits, inability to run tests | Agent capability depends on execution discipline |
| Incomplete multi-file edit | One call site fixed, others missed | Larger patches require change propagation |
| Test overfitting | Patch passes known shape but is brittle | Hidden tests are necessary but insufficient |
The most interesting implication is that future gains may not come only from larger models. They may come from better context selection, repository maps, persistent scratchpads, multi-pass validation, dynamic tracing, and agents that can form and revise hypotheses about the codebase. That is why Evaluation Harnesses and Agent Scaffolding are inseparable from model evaluation in this domain.
Should SWE-bench Pro become the flagship benchmark?
The case for flagship status
SWE-bench Pro is the best candidate flagship for long-horizon autonomous patching because it fixes several weaknesses that now limit SWE-bench Verified.
It is harder by construction. Its tasks require larger patches, more files, and more repository-level reasoning. It is more contamination-resistant by construction, using copyleft public repositories and private commercial codebases. It includes public, private, and held-out splits, enabling both open research and future overfitting checks. It also improves task resolvability through human augmentation and test verification. arXiv
It is also already being treated as the preferred frontier benchmark by at least one major model developer. OpenAI’s 2026 SWE-bench Verified audit says OpenAI stopped reporting Verified scores and recommends reporting results for SWE-bench Pro. OpenAI
For frontier launches, this matters. A benchmark that still meaningfully separates top systems is more useful than one whose remaining unsolved tasks are increasingly explained by flawed tests, public contamination, or underspecification. Pro currently gives a wider and more realistic performance distribution.
The case for supplement status
SWE-bench Pro should not become the only flagship for software-engineering agents, because issue resolution is not the whole job.
The benchmark still asks for a patch to a specified issue. It does not directly evaluate system design, prioritization, product tradeoffs, code review, incident response, documentation, team coordination, architectural refactoring, or long-running maintenance. The Pro paper explicitly lists issue scope as a limitation, noting that real software engineering includes broader activities such as system design, code review, documentation, and architectural decisions. arXiv
It also remains test-based. Test suites are concrete and scalable, but they are imperfect verifiers of software quality. A patch can pass tests while being unmaintainable, insecure, overfit, or architecturally wrong. Conversely, a valid alternative solution can fail if the tests encode implementation-specific assumptions. The Pro paper acknowledges this verifier limitation directly. arXiv
Finally, public Pro will itself become an optimization target. The benchmark’s current contamination defenses are stronger than Verified’s, but public tasks can still be studied, scaffolded against, and indirectly learned. The held-out and private splits exist precisely because public leaderboards decay.
Practical recommendation
For practitioners evaluating coding agents, SWE-bench Pro should be treated as the flagship benchmark for long-horizon repository patching, not for “software engineering” as a whole.
A reasonable evaluation stack is:
| Evaluation purpose | Best current role |
|---|---|
| Historical comparison with older coding-agent literature | SWE-bench Verified, with contamination caveats |
| Frontier long-horizon patching | SWE-bench Pro public plus private/commercial where available |
| Internal deployment readiness | Organization-specific private repositories and tasks |
| Codebase understanding, test writing, refactoring | SWE Atlas-style workflow benchmarks |
| Robustness to benchmark decay | Live or continuously refreshed benchmarks such as SWE-rebench |
| Goal-directed engineering behavior | Competitive or objective-driven benchmarks such as CodeClash |
This stack treats Pro as necessary but not sufficient. A system that cannot do well on SWE-bench Pro probably lacks robust autonomous patching ability. A system that does well on SWE-bench Pro may still be poor at code review, designing new systems, writing high-quality tests, or maintaining long-lived codebases.
What comes after SWE-bench Pro saturates?
SWE-bench Pro saturation is not hypothetical. The paper baseline was below 25%; current public leaderboard top scores are near 60%. If the field keeps improving at the same pace, public Pro will eventually lose differentiating power for frontier systems. scale.com
The next benchmark generation should move along at least five axes.
1. Private and rotating evaluation sets
The held-out and commercial/private splits are the immediate answer. A static public set is valuable for research reproducibility, but it cannot remain uncontaminated indefinitely. A benchmark that wants to measure frontier systems must maintain hidden or rotating tasks, ideally with release schedules that allow retrospective analysis without compromising current evaluations.
SWE-rebench represents one version of this idea: a continuously updated, decontaminated benchmark with standardized scaffolding, frequent dataset updates from live repositories, and contamination tracking tied to model release dates. Nebius
2. Broader task types beyond patching
The next generation cannot be only “fix this issue.” SWE Atlas is an early example of the broader direction: Codebase Q&A, Test Writing, and Refactoring. Its paper argues that focusing only on functional resolution risks producing agents that are good patchers but poor engineers, and it introduces 284 expert-authored tasks across those three workflows with category-specific rubrics and programmatic checks. arXiv
Scale’s SWE Atlas writeup frames it as a complement to SWE-bench Pro: it draws from similar production repositories and environments but expands evaluation to investigative and maintenance workflows. scale.com
3. Better verifiers
Unit tests are useful because they are executable, scalable, and hard to argue with when well written. But professional engineering quality often cannot be reduced to hidden tests. Future benchmarks need hybrid verification: deterministic tests, mutation testing, static analysis, security checks, performance checks, expert rubrics, and maybe adversarial review agents.
SWE Atlas explicitly combines programmatic checks with rubric-based evaluation to capture maintainability, code placement, anti-patterns, and refactor quality. arXiv
4. Runtime investigation and evidence
A stronger coding agent should not merely read files and guess. It should run the system, trace execution, inspect logs, form hypotheses, test them, and cite runtime evidence. SWE Atlas Codebase Q&A tasks point in this direction: agents must answer questions by exploring code, setting up and running applications, and tracing behavior through live executions. arXiv
This matters because many real engineering tasks start before the patch: understanding why a system behaves the way it does.
5. Goal-directed and competitive software development
SWE-bench-style tasks are instruction-following tasks. Real software engineering is often goal-driven: reduce latency, improve user retention, prevent fraud, lower cost, improve test coverage, increase reliability. CodeClash pushes this direction by evaluating agents in goal-oriented multi-round software-engineering competitions, where agents iteratively improve a codebase to win in simulated arenas. GitHub
Goal-directed benchmarks are messier than patch benchmarks, but they may be closer to the economic question: can an agent improve a system under incomplete instructions and feedback loops?
Interpretive cautions
SWE-bench Pro is a better benchmark, not an oracle.
Do not compare scores across harnesses as if they were pure model scores. A score may reflect the model, scaffold, context policy, tool design, retry budget, validation strategy, and cost cap. The Scale public leaderboard itself distinguishes capped and uncapped runs and marks mini-SWE-agent harness results. scale.com
Do not equate public-set performance with private deployment performance. The private/commercial set is lower-scoring and more deployment-like. Public Pro is useful for reproducibility; private Pro is more useful for generalization.
Do not treat test-passing as full correctness. Tests are necessary but incomplete. SWE-bench Pro’s test verification makes the metric more credible, but a hidden test suite can still miss maintainability, security, performance, and architectural quality.
Do not overstate contamination resistance. Copyleft licensing and private repos reduce contamination risk; they do not prove absence of leakage. OpenAI’s audit found Pro contamination cases to be rarer and less severe than Verified, not nonexistent. OpenAI
Do not mistake benchmark saturation for solved engineering. If Pro reaches 80–90%, that will mean agents are strong on this distribution of patch tasks. It will not mean they can own production systems, negotiate requirements, design architecture, or take responsibility for long-lived code health.
Bottom line
SWE-bench Pro is the clearest successor to SWE-bench Verified for evaluating frontier coding agents on autonomous repository repair. It tests a sharper question: whether agents can generalize from familiar, smaller, public issue-resolution tasks to larger, more complex, less contaminated, multi-file professional codebases.
Its empirical contribution is the performance cliff: high Verified scores did not initially transfer to Pro, and private-set results remain harder than public-set results even as public leaderboard scores rise. Its methodological contribution is the construction pattern: contamination-aware repository selection, difficulty filtering by substantial edits, human-augmented task specification, reproducible environments, repeated test execution, and human test verification.
The benchmark should be the flagship for long-horizon patching, but not the flagship for all software engineering. The next frontier is a portfolio: Pro-style private repository repair, live decontaminated issue streams, codebase investigation, test writing, refactoring, goal-directed development, and stronger verifiers for engineering quality.
Companion entries
Core theory: SWE-bench, SWE-bench Verified, Benchmark Saturation, Benchmark Contamination, Long-Horizon Agency, Software Engineering Evals
Benchmark design: Test-Based Verification, Fail-to-Pass Tests, Pass-to-Pass Tests, Held-Out Evaluation Sets, Private Benchmarking, Evaluation Harnesses
Agent systems: Coding Agents, Agent Scaffolding, Repository-Level Code Understanding, Tool Use in LLM Agents, Context Management for Coding Agents
Practice: How to Evaluate Coding Agents, Internal Software Engineering Benchmarks, AI Code Review, Regression Testing for Agentic Code, Deployment Readiness for Coding Agents
Adjacent benchmarks: SWE Atlas, SWE-rebench, CodeClash, Multi-SWE-bench, SWE-bench Multimodal
Counterarguments: Goodharting Benchmarks, Limits of Unit-Test Evaluation, Why Test-Passing Is Not Software Engineering, Private Leaderboard Transparency, Benchmark Leakage