SWE-bench Live: Continuously Refreshed Software-Engineering Eval
SWE-bench Live is best understood as a shift from one-off software-engineering benchmarks to benchmark operations: a pipeline that continually mines recent GitHub issue/PR pairs, builds runnable environments, and verifies test behavior so coding agents can be measured on fresher repository-level tasks. Its significance is not only that it is harder than older SWE-bench variants, but that it makes contamination resistance, rolling releases, and infrastructure maintenance part of the benchmark design itself.
Coverage note: verified through May 18, 2026.
Why SWE-bench Live exists
SWE-bench made repository-level software engineering a standard model-evaluation target: given a real GitHub issue and the pre-fix repository state, an agent must edit the codebase so previously failing tests pass without breaking previously passing tests. The original benchmark contained 2,294 real issue-resolution tasks from 12 Python repositories, and early systems solved only a small fraction of them; the original paper reported Claude 2 at 1.96% resolved, while later scaffolds such as SWE-agent raised the bar on the fixed benchmark. arXiv
The problem is that fixed public benchmarks age. Once benchmark instances, gold patches, issue texts, and evaluation harnesses are public, frontier models and agents can be exposed to them directly or indirectly through training data, benchmark-driven development, blog posts, trajectories, bug reports, and leaderboard submissions. This is not a SWE-bench-specific issue: LiveCodeBench, LatestEval, LiveBench, and other contamination-resistant-evaluation projects all start from the same observation that static public test sets become less reliable as evidence about model generalization over time. arXiv+2arXiv+2
SWE-bench Live’s answer is temporal freshness. Instead of treating the benchmark as a fixed corpus, it treats benchmark construction as a continuous pipeline: discover newly resolved GitHub issues, construct executable environments, verify fail-to-pass and pass-to-pass tests, and publish rolling releases. The initial paper describes an automatic curation pipeline and an initial release of 1,319 tasks from issues created after January 2024 across 93 repositories, with monthly updates as the intended maintenance mode. arXiv
That changes the evaluation question. A static SWE-bench score asks, “How well does this agent perform on a known public corpus?” A SWE-bench Live score asks, “How well does this agent perform on recent, real, executable repository issues under a particular benchmark snapshot?” The latter is closer to measuring current generalization, but it is also less stable: scores must be interpreted relative to a release date, split definition, execution environment, and validation denominator.
Core design: live issue mining plus executable verification
SWE-bench Live keeps the basic SWE-bench task format: an agent receives a natural-language issue description and a codebase snapshot before the human fix, then must produce a patch that passes relevant tests. The design change is upstream and operational. Rather than manually constructing a fixed benchmark once, SWE-bench Live automates most of the collection, environment-building, and validation loop. arXiv
At a high level, the pipeline has four stages:
| Stage | What it does | Why it matters |
|---|---|---|
| Repository selection | Starts from popular GitHub repositories and filters for sufficient activity, language composition, licensing, and test-bearing pull requests | Ensures there are enough real issue/PR pairs and runnable tests, but introduces selection bias toward large, active, open-source projects |
| Issue/PR extraction | Finds issues resolved by pull requests and links natural-language problem statements to human patches | Preserves the SWE-bench premise: real user-facing or maintainer-facing software changes rather than synthetic edits |
| Environment construction | Uses RepoLaunch to build a Dockerized environment for each repository snapshot | Converts GitHub history into executable evaluation infrastructure |
| Test verification | Runs tests before and after the human patch, extracts fail-to-pass and pass-to-pass tests, and repeats validation to filter flaky or invalid instances | Makes the benchmark automatically scorable rather than merely descriptive |
The repository filter is important. The initial SWE-bench Live Python pipeline began with GitHub repositories above 1,000 stars, then filtered for issue/PR activity, forks, Python composition, open-source license, and pull requests that modify tests; it included issues created after January 2024. arXiv This is a reasonable engineering choice for executable benchmark construction, but it means SWE-bench Live is not a uniform sample of “software engineering.” It is a sample of recent, test-bearing, open-source GitHub maintenance in repositories that are sufficiently active and popular to support automation.
The test-verification step is the benchmark’s oracle. SWE-bench Live validates that at least one test fails before the human patch and passes after it, and that previously passing tests remain stable; it also reruns tests to reduce flakiness. arXiv This preserves the central SWE-bench scoring idea: a candidate patch is not judged by textual similarity to the human patch, but by whether it satisfies executable tests.
The architecture is therefore closer to Benchmark CI than to a conventional dataset release. The benchmark has crawlers, filters, Docker builders, dependency controls, test parsers, rerun policies, and release procedures. SWE-bench Live’s contribution is not just a new set of tasks; it is a model for keeping a software-engineering benchmark alive.
RepoLaunch as the architectural contribution
The key enabling system is RepoLaunch, the environment-construction pipeline described in the SWE-bench Live work. Prior SWE-bench-style benchmarks depended heavily on human effort to make repository snapshots executable. SWE-bench Live argues that this manual burden is one reason static benchmarks remain static: every new repository or snapshot can require dependency debugging, test-command discovery, build-tool configuration, and framework-specific log parsing. arXiv
RepoLaunch attacks this by automating the setup loop. It identifies relevant project files such as CI configuration and README files, chooses a base image, uses an interactive setup agent to install dependencies and build the repository, generates candidate test commands, verifies that the source can be installed and tests can run, and commits a Docker image for later evaluation. arXiv
One subtle design feature is the “time-machine” dependency mechanism: the setup restricts package installation to versions no later than the base commit timestamp, reducing dependency drift from future package releases. arXiv This matters because live benchmarks are especially exposed to time-dependent breakage. A task created from a 2024 repository snapshot can fail in 2026 because a dependency was yanked, a package changed behavior, a system image updated, or an external service disappeared. Dockerization helps, but it does not eliminate temporal dependency risk.
This is the sense in which SWE-bench Live is CI-style. It does not merely host benchmark files; it continuously performs the work that software projects themselves perform in CI: build the code, install dependencies, run tests, detect failures, and record reproducible artifacts. The benchmark’s quality depends on whether that operational loop keeps functioning.
Rolling release model and contamination resistance
SWE-bench Live’s contamination resistance is temporal, not magical. It reduces contamination by adding newly created and newly resolved issues after earlier model-training cutoffs and after prior benchmark releases. This is the same broad logic used by LiveCodeBench, which continuously collects recent contest problems, and LatestEval, which uses recent text sources to build less-contaminated evaluation items. arXiv+2livecodebench.github.io+2
The public SWE-bench Live repository describes the benchmark as a monthly-updated benchmark and records later expansions beyond the initial Python-only release. As of the project’s 2026 updates, the repository notes a Multi-Language release covering C/C++, C#, Java, TypeScript/JavaScript, Go, and Rust; a Windows release; and a policy of adding 50 newly verified Python issues per month to the test split while keeping some splits frozen. GitHub
That split policy is important. A live benchmark has two competing goals:
| Goal | Static benchmark behavior | Live benchmark behavior |
|---|---|---|
| Comparability | High: everyone can run the same items indefinitely | Lower unless the release snapshot is specified |
| Contamination resistance | Degrades over time | Renewed by adding recent items |
| Engineering cost | Paid mostly once | Paid continuously |
| Leaderboard stability | Stable but increasingly gameable | More robust to memorization but harder to compare across dates |
| Reproducibility | Easier if environments are preserved | Depends on Docker images, dependency freezing, and release metadata |
SWE-bench Live’s rolling release model therefore trades one methodological problem for another. Static SWE-bench has rising contamination risk; SWE-bench Live has moving-target risk. The correct unit of reporting is not “SWE-bench Live” in the abstract, but “SWE-bench Live, split X, release window Y, evaluated under harness revision Z.”
Empirical results: lower scores than static SWE-bench Verified
The initial SWE-bench Live paper evaluated several agent scaffolds and frontier models, including OpenHands, SWE-agent, and Agentless with GPT-4o, GPT-4.1, Claude 3.7 Sonnet, and DeepSeek V3. It reported three main metrics: resolved rate, patch-apply rate, and localization-success rate. arXiv
On the 300-instance Lite subset, the reported resolved rates were:
| Agent scaffold | Model | Resolved | Patch applied | Localization success |
|---|---|---|---|---|
| OpenHands | GPT-4o | 7.00% | 72.00% | 30.33% |
| OpenHands | GPT-4.1 | 11.33% | 59.33% | 28.67% |
| OpenHands | Claude 3.7 Sonnet | 17.67% | 84.00% | 48.00% |
| OpenHands | DeepSeek V3 | 13.00% | 81.00% | 38.33% |
| SWE-agent | GPT-4o | 10.00% | 93.33% | 40.33% |
| SWE-agent | GPT-4.1 | 16.33% | 95.00% | 47.33% |
| SWE-agent | Claude 3.7 Sonnet | 17.67% | 84.67% | 46.33% |
| SWE-agent | DeepSeek V3 | 15.33% | 92.00% | 44.00% |
| Agentless | GPT-4o | 11.67% | 91.67% | 37.67% |
| Agentless | GPT-4.1 | 12.00% | 84.33% | 39.00% |
| Agentless | Claude 3.7 Sonnet | 11.33% | 68.00% | 30.00% |
| Agentless | DeepSeek V3 | 13.33% | 83.33% | 40.67% |
The full-set results reported in the same work were also modest by static-leaderboard standards: OpenHands with Claude 3.7 Sonnet reached 19.25% resolved, SWE-agent with GPT-4.1 reached 18.57%, and SWE-agent with Claude 3.7 Sonnet reached 17.13%. The same OpenHands plus Claude 3.7 Sonnet setup scored 43.20% on SWE-bench Verified under the paper’s controlled comparison, while the authors noted that static SWE-bench Verified leaderboard results had exceeded 60%. arXiv
The empirical takeaway is not simply “SWE-bench Live is harder.” Several effects are entangled:
-
Freshness effect: recent tasks are less likely to have been seen in model training or benchmark-specific agent tuning.
-
Repository-diversity effect: SWE-bench Live uses more repositories than the original SWE-bench family, reducing repository-specific overfitting.
-
Infrastructure effect: automatically built environments may include different failure modes from manually curated static tasks.
-
Selection effect: requiring test-modifying pull requests may select for certain types of bugs and exclude others.
-
Agent-scaffold effect: different scaffolds fail differently; patch application, localization, and test-passing are separable capabilities.
The paper’s analysis also reports that performance tends to fall as repository size and patch complexity rise, especially for multi-file or cross-file fixes. arXiv That matters for Software Engineering Agents because many leaderboard discussions collapse performance into a single resolved-rate number. SWE-bench Live’s more useful diagnostic lesson is that current agents still struggle with localization, repository navigation, and multi-step codebase reasoning, not merely with syntax or patch generation.
Relationship to SWE-bench variants
SWE-bench Live is not a replacement for every SWE-bench variant. It answers a different evaluation question. The table below summarizes the relationship.
| Benchmark | Main purpose | Corpus style | Strength | Main weakness |
|---|---|---|---|---|
| SWE-bench Full | Broad fixed benchmark of real Python GitHub issue-resolution tasks | 2,294 tasks from 12 Python repositories | Historical comparability; canonical task format | Public and static; contamination risk grows |
| SWE-bench Lite | Cheaper, faster subset for iteration | 300 selected tasks | Lower evaluation cost; useful for development loops | Smaller and deliberately filtered; not enough for strong final claims |
| SWE-bench Verified | Human-validated subset of SWE-bench | 500 tasks reviewed by developers | Higher task-quality confidence than raw Full/Lite | Still public and static; increasingly contaminated |
| SWE-bench-Java | Java adaptation of SWE-bench-style issue resolution | Java repositories and Dockerized Java evaluation | Tests whether agents transfer beyond Python | Smaller and language-specific; not directly comparable to Python SWE-bench |
| SWE-bench Multilingual | Multi-language SWE-bench-style benchmark | Curated tasks across several languages | Broader language coverage | Static benchmark dynamics still apply |
| SWE-bench Live | Continuously refreshed, contamination-resistant issue-resolution eval | Rolling GitHub issue/PR corpus with automated environments | Temporal freshness; broader repository coverage; operational benchmark maintenance | Moving target; infrastructure-dependent; harder to reproduce across time |
The original SWE-bench Full benchmark established the task. SWE-bench Lite reduced evaluation cost by selecting 300 tasks that were more self-contained and easier to iterate on; its selection procedure filtered out tasks with images, long or underspecified issue descriptions, large multi-file edits, excessive hunks, file creation/removal, and other features that would complicate cheap evaluation. SWE-bench
SWE-bench Verified addressed a different problem: task quality. OpenAI’s Verified subset used human annotation to identify 500 samples that developers judged less problematic. The annotation campaign found that many candidate tasks had issues such as underspecified problem statements or tests that unfairly rejected valid solutions, and the resulting subset was intended to be more reliable than raw SWE-bench or Lite. OpenAI
By 2026, however, OpenAI argued that SWE-bench Verified itself had become increasingly contaminated and stopped reporting it for some of its own model evaluations, recommending newer alternatives such as SWE-bench Pro and private/post-cutoff evaluations. OpenAI That critique reinforces SWE-bench Live’s motivation: human validation improves task quality, but it does not solve the problem of a public benchmark aging into the training-and-tuning ecosystem.
SWE-bench-Java is best understood as a language-transfer variant. It ports the SWE-bench-style task to Java repositories, with Dockerized evaluation and Java-specific build/test concerns. The public SWE-bench-Java work describes a verified Java issue-resolution set, F2P/P2P-style validation, and initial agent results that were low in absolute terms, underscoring that Python SWE-bench success does not automatically transfer to other build systems and ecosystems. arXiv
SWE-bench Live now overlaps with that space through its multi-language expansion, but the comparison is not one-to-one. SWE-bench-Java asks, “How does a SWE-bench-style agent handle Java?” SWE-bench Live asks, “How does an agent handle fresh repository-level issues under a maintained rolling benchmark?” The Java slice of SWE-bench Live may become more important for contamination-resistant Java evaluation, while SWE-bench-Java remains useful as a stable language-specific reference point.
Methodological critiques
1. Continuous infrastructure becomes part of the benchmark
SWE-bench Live’s greatest strength is also its greatest operational liability. A static benchmark can be archived, mirrored, and rerun. A live benchmark must keep crawling repositories, building containers, validating tests, updating splits, hosting images, handling invalid tasks, and documenting changes.
The project documentation itself notes practical evaluation hazards: resource needs can be substantial, some instances require large memory or disk footprints, tests may become invalid over time, Docker does not provide perfect isolation, and researchers may need to rerun gold patches and report the denominator that actually passed on their own machine at evaluation time. GitHub These are not footnotes. They are central to interpreting scores.
The benchmark therefore inherits Operational Debt. If maintainers stop updating it, the contamination-resistance advantage decays. If they continue updating it without strict versioning, comparability suffers. Sustainable live benchmarks need release manifests, immutable Docker digests, archived dependency artifacts, gold-patch validation logs, and clear policies for task retirement.
2. Reproducibility is weaker across time
SWE-bench Live makes “the benchmark” time-indexed. A score from April 2025, August 2025, and May 2026 may refer to different task sets, different Docker images, different repositories, and different validation outcomes. This is not a flaw in the concept; it is the unavoidable cost of freshness.
The reporting burden is therefore higher. A credible SWE-bench Live result should specify:
| Field | Why it matters |
|---|---|
| Release date or commit hash | Identifies the task set |
| Split name | Distinguishes frozen Lite/Verified-style splits from rolling Full/test splits |
| Date range of issues | Clarifies temporal holdout strength |
| Agent scaffold and version | Separates model capability from tool-use loop |
| Model version and sampling policy | Prevents hidden differences in inference behavior |
| Docker image digests | Supports environment reproducibility |
| Gold-patch pass denominator | Handles tasks that fail validation on a particular machine/time |
| Number of rollouts and selection policy | Prevents cherry-picking across attempts |
| Compute budget and time limit | Normalizes agent search effort |
Without those fields, “Model X scored Y on SWE-bench Live” is under-specified.
3. Item selection bias is real
SWE-bench Live’s issue selection is not neutral. It prefers repositories that are popular, active, open-source, sufficiently Python-heavy in the initial release, and rich in issue-linked pull requests with test modifications. arXiv These filters improve benchmark executability, but they exclude large regions of real software engineering:
| Excluded or underrepresented work | Why it may be missing |
|---|---|
| Private enterprise repositories | Not visible on public GitHub |
| Small or inactive projects | Fail activity thresholds |
| Maintenance without tests | Cannot produce reliable F2P/P2P evaluation |
| UI, product, and design bugs | Often hard to capture with unit tests |
| Security fixes without public issue detail | May be embargoed or underspecified |
| Build-system and deployment work | May require external services or heavyweight infrastructure |
| Changes not linked cleanly to issues | Hard to mine automatically |
The result is a benchmark of executable open-source issue resolution, not of all software engineering. That is still valuable, but it should be named precisely.
4. Test oracles remain imperfect
SWE-bench-style evaluation depends on tests as the oracle. That is stronger than comparing generated patches to gold patches, but it is not equivalent to human code review. A patch can pass tests while being brittle, overfit, stylistically unacceptable, insecure, or semantically incomplete. Conversely, a valid patch can fail because tests are too narrow, too broad, flaky, or tied to the human solution.
This issue is visible in the history of SWE-bench itself. OpenAI’s SWE-bench Verified work found many original candidate tasks problematic enough to filter, including underspecified statements and tests that could reject valid solutions. OpenAI Later OpenAI analysis argued that Verified had both test-quality problems and contamination problems, despite its human curation. OpenAI SWE-bench Live reduces memorization risk through freshness, but it does not eliminate the deeper problem of Test Oracle Fragility.
5. Freshness is not secrecy
A live public benchmark is less likely to be in a model’s pretraining set at the moment it is released, but after release it becomes public data. Future models can train on released tasks, agent traces, solution discussions, and leaderboard artifacts. The live strategy works only if the benchmark continues to add new tasks faster than contamination accumulates.
This matches the broader contamination-resistant-evaluation literature. LiveCodeBench uses contest release dates to evaluate post-cutoff performance; LatestEval uses recent sources to reduce memorization; LiveBench emphasizes frequently updated, objectively scored questions. livecodebench.github.io+2arXiv+2 Other work on benchmark contamination mitigation finds that many static mitigation strategies either fail to remove contamination or sacrifice benchmark fidelity, which makes continuous refresh attractive but not free. OpenReview
Which SWE-bench variant should practitioners report?
The right answer depends on the claim.
| Claim being made | Recommended benchmark report | Rationale |
|---|---|---|
| “Our agent improves on historical SWE-bench systems” | SWE-bench Verified, possibly Full and Lite | Maximizes comparability with prior work |
| “Our agent generalizes to fresh real-world issues” | SWE-bench Live, with exact release metadata | Better contamination resistance |
| “Our iteration loop is cheaper/faster” | SWE-bench Lite and/or SWE-bench Live Lite | Lower cost, but weaker final evidence |
| “Our model handles Java” | SWE-bench-Java or a Java slice of SWE-bench Live Multi-Language | Language-specific evidence |
| “Our agent handles many programming ecosystems” | SWE-bench Multilingual plus SWE-bench Live Multi-Language | Avoids Python-only overclaiming |
| “Our frontier model is state of the art” | At least one static comparable benchmark plus one fresh/live or private benchmark | Static scores alone are no longer persuasive |
A strong modern report should include both a static and a live result. Static SWE-bench Verified remains useful because it is widely understood and historically comparable, but it should be labeled as contaminated or contamination-prone when used for frontier claims. SWE-bench Live is better evidence for current generalization, but it must be reported with the release snapshot and infrastructure details. OpenAI
A minimal credible reporting block for SWE-bench Live should look like:
| Reporting field | Example format |
|---|---|
| Benchmark | SWE-bench Live |
| Split | Full / Lite / monthly test split |
| Release | Commit hash or dated release |
| Issue window | e.g., January 2024–April 2025 |
| Language scope | Python-only / Multi-Language / Windows |
| Agent | OpenHands, SWE-agent, Agentless, or custom scaffold |
| Model | Exact model identifier and date |
| Budget | Iterations, tool calls, wall-clock limit, token budget |
| Rollouts | Number of attempts per task and selection rule |
| Denominator | Number of tasks whose gold patch passed in the local run |
| Metrics | Resolved rate, patch apply rate, localization success, cost per resolved task |
| Artifacts | Predictions, logs, trajectories, environment revisions |
The denominator field is especially important for live evaluation. The SWE-bench Live repository explicitly warns that some tests may become invalid over time and suggests rerunning gold patches and reporting the number of instances that passed under the experiment’s machine and time. GitHub That turns benchmark reporting into something closer to scientific instrumentation: the apparatus matters.
What SWE-bench Live teaches about agent capability
The headline score gap between SWE-bench Verified and SWE-bench Live suggests that part of the apparent progress on static SWE-bench may come from benchmark familiarity, task selection, repository familiarity, scaffold tuning, or contamination. But the more interesting lesson is diagnostic.
SWE-bench Live separates several abilities that are often conflated:
| Capability | Observable proxy |
|---|---|
| Repository comprehension | Can the agent find the relevant modules, conventions, and abstractions? |
| Fault localization | Does it edit the right files and functions? |
| Patch construction | Does it produce syntactically valid and semantically plausible changes? |
| Test use | Can it run, interpret, and respond to tests effectively? |
| Regression avoidance | Does it preserve pass-to-pass behavior? |
| Long-horizon control | Can it recover from failed attempts and partial fixes? |
| Environment robustness | Can it work inside real dependency and build constraints? |
The reported patch-apply rates are often much higher than resolved rates, which means many agents can emit syntactically applicable patches but still fail the behavioral task. arXiv This is a useful corrective to patch-generation demos. The hard part is not merely producing a diff; it is producing the right diff in a real codebase under executable constraints.
The localization-success metric is also revealing. Agents that fail to localize correctly often cannot recover through generation quality alone. This supports a broader view of AI Coding Agents: model intelligence, retrieval, repository indexing, test execution, planning, and tool feedback are coupled. SWE-bench Live is an evaluation of the whole agent system, not just the base model.
Relationship to contamination-resistant evaluation literature
SWE-bench Live belongs to a growing family of dynamic or freshness-oriented evaluations.
LiveCodeBench continuously collects new coding problems from contest platforms such as LeetCode, AtCoder, and Codeforces, using release dates to evaluate whether models solve post-cutoff tasks rather than memorized benchmark items. arXiv Its domain is algorithmic coding and code-generation capabilities, whereas SWE-bench Live targets repository-level issue resolution.
LatestEval creates reading-comprehension evaluations from recent sources, including arXiv, BBC, and GitHub, with the explicit goal of reducing contamination by using new data. arXiv Its task format is not software engineering, but the contamination model is similar: use temporal windows and automatic pipelines to keep evaluation ahead of training-data exposure.
LiveBench similarly frames itself around frequently updated questions with objective scoring, spanning categories such as math, coding, reasoning, language, instruction following, and data analysis. OpenReview It generalizes the live-benchmark idea across domains, while SWE-bench Live specializes it for executable software maintenance.
The broader lesson from this literature is that contamination resistance is an operational property, not a one-time dataset property. A benchmark is contamination-resistant only relative to model training windows, public exposure, update cadence, and the availability of solution artifacts. SWE-bench Live’s strongest contribution is making that operational property explicit in the software-engineering domain.
Open question: can continuously refreshed SWE benchmarks be sustained?
The unresolved question is whether continuously refreshed software-engineering benchmarks are operationally sustainable.
There are reasons for optimism. GitHub issue/PR history is abundant; many projects have CI metadata; Dockerization can preserve environments; agentic setup systems can reduce human labor; and monthly updates may be enough to keep the evaluation ahead of contamination for frontier-model reporting. SWE-bench Live’s initial scale—1,319 tasks across 93 repositories—and later expansions toward multi-language and Windows tasks show that the approach can move beyond a toy proof of concept. arXiv
There are also hard constraints. Live SWE benchmarks require continuing compute, storage, dependency management, Docker hosting, test-rerun infrastructure, flaky-test triage, repository-policy monitoring, benchmark governance, and documentation. They may also need maintainer opt-out policies, license handling, and protections against benchmark-specific gaming. A live benchmark that becomes central to model releases will attract optimization pressure; once agents are tuned to its repository-selection and task-validation patterns, the benchmark may need to diversify again.
The sustainable version of SWE-bench Live is therefore not merely “SWE-bench, but updated.” It is a benchmark service with versioned releases, frozen comparison tracks, rolling fresh tracks, public artifacts, retirement rules, and enough funding or community support to keep the CI loop healthy. The benchmark’s future value depends less on any single score and more on whether the community adopts reporting norms that preserve both comparability and freshness.
Bottom line
SWE-bench Live reframes software-engineering evaluation as a maintained system rather than a static dataset. Its rolling GitHub issue pipeline, RepoLaunch environment automation, and repeated test verification make it one of the clearest examples of Continuously Refreshed Benchmarks applied to real repository-level coding agents.
Its advantage is contamination resistance through freshness. Its cost is methodological complexity: less stable comparability, higher infrastructure dependence, more reporting burden, and persistent test-oracle limitations. For practitioners, the best current norm is dual reporting: use static SWE-bench variants for historical comparability, and use SWE-bench Live or another post-cutoff/live benchmark for claims about present-day generalization.
References
[1] [SWE-bench Live paper: “SWE-bench Goes Live!”] — primary writeup for the benchmark design, RepoLaunch pipeline, initial dataset statistics, and reported model results. arXiv+3arXiv+3arXiv+3
[2] [SWE-bench Live GitHub repository] — release notes, submission format, rolling-update policy, multi-language and Windows updates, and evaluation caveats. GitHub
[3] [SWE-bench original paper and benchmark site] — canonical definition of the repository-level issue-resolution task and original 2,294-task Python benchmark. arXiv
[4] [SWE-bench Lite] — subset-construction rationale for lower-cost SWE-bench evaluation. SWE-bench
[5] [OpenAI SWE-bench Verified report] — human-validation methodology and quality issues found in raw SWE-bench tasks. OpenAI
[6] [OpenAI SWE-bench Verified contamination critique] — later argument that SWE-bench Verified had become increasingly contaminated and less suitable for frontier claims. OpenAI
[7] [SWE-bench-Java] — Java-specific SWE-bench-style benchmark construction and initial agent results. arXiv
[8] [LiveCodeBench] — continuously updated coding benchmark using recent contest problems for contamination-resistant code evaluation. arXiv
[9] [LatestEval] — dynamic recent-data evaluation pipeline for reducing benchmark contamination. arXiv
[10] [LiveBench] — frequently updated objective benchmark across multiple model-capability domains. OpenReview
[11] [Benchmark contamination mitigation study] — evidence that many static contamination-mitigation strategies fail to balance fidelity and contamination resistance. OpenReview
Companion entries
Core theory: Benchmark Contamination, Temporal Holdout Sets, Dynamic Evaluation, Continuously Refreshed Benchmarks, Evaluation Fidelity versus Contamination Resistance
Software-engineering evals: SWE-bench, SWE-bench Verified, SWE-bench Lite, SWE-bench-Java, SWE-bench Multilingual, LiveCodeBench
Agent systems: AI Coding Agents, Repository-Level Code Understanding, Agent Scaffolds, Fault Localization, Test-Time Tool Use
Evaluation practice: SWE-bench Reporting Checklist, Dockerized Evaluation Harnesses, Gold Patch Validation, Benchmark CI, Patch Apply Rate
Counterarguments and failure modes: Operational Debt, Test Oracle Fragility, Evaluation Selection Bias, Leaderboard Overfitting, Private Evals versus Public Benchmarks