LiveCodeBench: Contamination-Resistant Code Benchmarking
1. Why LiveCodeBench mattered
By 2024, code-model evaluation had a familiar failure mode. Static benchmarks such as HumanEval, MBPP, and APPS were convenient, reproducible, and widely reported, but their public test items had become part of the ambient internet corpus that modern pretraining pipelines scrape. A model could score well because it generalized, because it had learned recurring problem patterns, or because some benchmark content or near-duplicates appeared in training. The measurement problem was that benchmark users often could not distinguish those cases.
LiveCodeBench was introduced by Jain et al. in 2024 to make that distinction more observable. The paper’s core design is a “live” benchmark of competitive-programming problems collected from LeetCode, AtCoder, and Codeforces, each tagged with a contest or release date. This enables time-gated evaluation: when a model has a known training cutoff or release date, evaluators can compare performance on problems published before and after that date. The original paper describes the benchmark as collecting new problems over time from those platforms and broadening evaluation beyond simple natural-language-to-code generation into self-repair, code execution, and test-output prediction. arXiv
The important shift is methodological. LiveCodeBench does not claim that public benchmarks can remain clean forever. Instead, it treats contamination as an empirical variable. If a model performs substantially better on problems released before its cutoff than after it, that temporal asymmetry is evidence worth investigating. If the asymmetry disappears under rephrasing, perturbation, or different construction methods, that too is a measurement result rather than a mere nuisance.
2. Benchmark design: live, dated, executable
LiveCodeBench’s benchmark design combines four ideas that are separately common but rarely packaged together: dated item provenance, executable correctness checks, competitive-programming difficulty, and multiple code-reasoning task formats.
The paper’s data pipeline collects contest problems and metadata from LeetCode, AtCoder, and Codeforces. It filters out problems that depend on images or ambiguous statements, records contest or release dates, and stores problem statements, public examples, hidden or generated tests, and ground-truth solutions where available. The original paper describes platform-specific collection from LeetCode weekly and biweekly contests, AtCoder Beginner Contests, and selected Codeforces divisions after an April–May 2023 starting period. arXiv
| Design axis | LiveCodeBench choice | Why it matters |
|---|---|---|
| Item source | Recent LeetCode, AtCoder, and Codeforces contest problems | Uses problems with public release dates and executable judges rather than hand-written static prompts |
| Temporal metadata | Each problem is associated with a contest or publication date | Enables Time-Gated Evaluation against model cutoffs or release dates |
| Main metric | Execution-based pass@1: generated program must pass all tests | Measures functional correctness, not surface similarity |
| Task formats | Code generation, self-repair, code execution, and test-output prediction | Tests more than “write a function from a docstring” |
| Difficulty structure | Competitive-programming difficulty bins and platform metadata | Reduces the risk that a benchmark is solved by shallow pattern matching alone |
| Updating pattern | New contest problems can be added over time | Makes benchmark maintenance part of the evaluation design rather than an afterthought |
LiveCodeBench evaluates code generation in the familiar way: given a natural-language problem statement, a model must produce a program that passes tests. The benchmark then extends the same problem pool into related tasks. In self-repair, the model receives buggy code and must fix it. In code execution, the model predicts the result of running a program. In test-output prediction, the model reasons about outputs for given inputs. These variants are useful because contamination can affect them differently: memorizing a solution may help code generation, while robust execution reasoning may require a different internal capability. arXiv
The benchmark is still contest-programming-heavy. That is a strength for clean measurement because problems have judges, dates, and well-defined outputs. It is also a limitation. Contest problems emphasize algorithms, edge cases, and compact solutions; they do not fully represent production software engineering, large-repository maintenance, API integration, or long-horizon debugging. The LiveCodeBench paper explicitly frames competition problems as a starting point and notes that domain-specific evaluations remain necessary. arXiv
3. The contamination problem LiveCodeBench targets
Benchmark Contamination is not one problem. It is a family of failure modes:
| Contamination mode | Example in code benchmarks | Why ordinary reporting misses it |
|---|---|---|
| Exact item memorization | The benchmark prompt or solution appears in pretraining data | A high pass rate looks identical to generalization unless provenance is checked |
| Near-duplicate exposure | A contest solution, editorial, or translated statement appears online | String matching may fail even when the semantic solution is seen |
| Test-suite overfitting | Fine-tuning on benchmark-style prompts or public tests | The model learns benchmark quirks rather than general programming |
| Leaderboard leakage | Public benchmark items become training data for later models | Old benchmarks become less informative precisely because they are popular |
| Distributional shortcut | Model sees many similar tasks before cutoff | Not “leakage” in a narrow sense, but still changes what a benchmark measures |
The literature around contamination-resistant evaluation had already shown why simple decontamination is fragile. Work on benchmark contamination argues that exact or fuzzy n-gram matching can miss paraphrased, translated, or transformed benchmark examples, and that models can overfit public test sets in ways that produce inflated benchmark scores. Other work introduced dynamic or temporally separated benchmarks such as WIKIMIA and LiveBench, where examples are chosen relative to model training dates or updated from recent sources. arXiv+2arXiv+2
LiveCodeBench’s contribution is to bring that dynamic-benchmark logic to code evaluation. It does not merely say “we decontaminated the benchmark.” It says: here are the problem release dates, here are the model dates, and here is how performance changes when evaluation windows move. That makes contamination analysis inspectable.
4. Time-gated evaluation: contamination as a testable variable
The central LiveCodeBench analysis compares model performance across problem publication windows. Suppose a model’s training cutoff is September 2023. A benchmark can report results on problems released before September 2023, after September 2023, and across the whole set. If the model is much better before the cutoff than after it, then at least three explanations become live: training-data exposure, distributional familiarity with pre-cutoff problems, or temporal shifts in problem difficulty and style.
The original paper’s headline empirical result is that some models show suspicious temporal drops. Jain et al. report that DeepSeek-Instruct models perform worse on LeetCode problems released after an August/September cutoff, and that GPT-4o shows a drop after its November cutoff, while several other models do not show comparably drastic variation. The authors interpret these drops as evidence of potential contamination or exposure effects, while also recognizing that temporal comparisons are not proof by themselves. arXiv
This is a more disciplined claim than “benchmark X is clean.” LiveCodeBench makes a weaker but more useful statement: contamination status is partly observable because each item has provenance. The benchmark can be sliced by time, and the slice can be aligned with model release dates or claimed training cutoffs. In practice, exact cutoffs are often unavailable for frontier models, so release date is sometimes used as an imperfect proxy. That proxy is not ideal, but it is still more informative than evaluating a 2025 model on a fully static 2021 benchmark and pretending that item exposure is unknowable.
A subtle point: a post-cutoff problem can still leak if it appears in post-training, synthetic data, tool-use traces, reinforcement-learning environments, or benchmark-specific fine-tuning after pretraining. Conversely, a pre-cutoff problem is not necessarily memorized. Time gating is not a binary cleanliness certificate. It is a way to make the causal structure less opaque.
5. Original empirical findings
The original LiveCodeBench paper evaluated many base and instruction-tuned models across code-generation and code-reasoning tasks. Its findings can be grouped into three categories: temporal contamination signals, task-portfolio effects, and static-benchmark overfitting.
| Finding | Evidence pattern | Interpretation |
|---|---|---|
| Some models show temporal drops | DeepSeek and GPT-4o variants performed worse after relevant cutoff windows | Potential contamination or exposure effects; not conclusive without additional probes |
| Closed/API models were generally stronger | Closed frontier systems outperformed open models on several tasks, especially reasoning-like code tasks | Frontier training and post-training scale remained important in 2024 |
| Task variants reveal different strengths | Code generation, repair, execution, and test-output prediction did not collapse into one identical ranking | “Coding ability” is not a single scalar |
| HumanEval-style scores can mislead | Some models with high HumanEval+ scores clustered with much lower LiveCodeBench scores | Static small benchmarks can reward overfitting or narrow competence |
The HumanEval comparison is especially important. The paper reports two clusters: models that perform well on both HumanEval+ and LiveCodeBench, and a red cluster of models with high HumanEval+ but low LiveCodeBench. The authors interpret the latter as potential overfitting to HumanEval-style evaluation, especially among fine-tuned open models. They also note that HumanEval is easier, smaller, and more isolated than LiveCodeBench’s contest problems. arXiv
LiveCodeBench’s broader task portfolio also surfaced differences among frontier models. The paper reports that Claude 3 Opus and Mistral-Large were strong on chain-of-thought-style tasks, and that Claude 3 Opus surpassed GPT-4 Turbo on test-output prediction in the reported setting. This matters because test-output prediction is closer to execution reasoning than to solution memorization. arXiv
6. Relationship to HumanEval
HumanEval was introduced with OpenAI Codex as a functional-correctness benchmark: 164 hand-written programming problems with unit tests, designed to evaluate whether a model can synthesize programs from docstrings. It was a major improvement over surface-form metrics because it judged generated code by execution rather than token overlap. arXiv
But HumanEval’s strengths became weaknesses once it became ubiquitous. It is small. It is public. It has been used heavily in model development. Its tests are limited compared with adversarial or industrial-scale test suites. EvalPlus later showed that augmenting HumanEval with much larger test suites can reduce apparent pass rates and change rankings, indicating that some benchmark success was due to insufficient tests rather than robust correctness. OpenReview
LiveCodeBench does not reject HumanEval’s premise. It inherits the idea that code should be judged by executable correctness. What it rejects is the assumption that a static, public, compact benchmark can indefinitely serve as a clean proxy for coding ability. In that sense, LiveCodeBench treats HumanEval-style contamination as a measurement problem:
| HumanEval issue | LiveCodeBench response |
|---|---|
| Small public item set | Larger and continually extensible pool of contest problems |
| Unknown item exposure | Explicit contest/release dates |
| Narrow function-synthesis format | Multiple task variants using the same problem pool |
| Limited tests | Platform tests and generated hidden tests where necessary |
| Saturation and overfitting | Time-windowed analysis and comparison against newer problems |
The right relationship is complementary rather than adversarial. HumanEval remains useful for fast regression tests and historical comparison. LiveCodeBench is more useful when the question is whether current models can solve recent, dated, executable programming problems under contamination-aware conditions.
7. Frontier-model trajectory: GPT-4 to DeepSeek-V3, o-series, and Claude reasoning
LiveCodeBench arrived just as code evaluation shifted from “large instruction model writes code” to “reasoning model spends more inference compute solving hard tasks.” That makes its leaderboard history unusually informative. It tracks not only code pretraining, but also Inference-Time Scaling, reinforcement-learning post-training, and long-chain reasoning.
7.1 GPT-4 and early closed-model dominance
In the original LiveCodeBench era, GPT-4-family models and Claude 3 Opus were among the strongest systems. The official LiveCodeBench site summarized early findings with GPT-4 Turbo and Claude 3 Opus near the top and also highlighted temporal contamination analysis for DeepSeek and GPT-family models. LiveCodeBench
This was the last phase in which many public code leaderboards could be read primarily as “which general instruction model writes better code?” Soon after, o-series-style reasoning models and DeepSeek-R1-style reinforcement-learning models changed the scoring regime. Longer inference traces, explicit reasoning budgets, and coding-oriented RL began to matter as much as pretraining scale.
7.2 DeepSeek-V3 and the pre-reasoning frontier
DeepSeek-V3 was released as a large mixture-of-experts model with 671B total parameters, 37B active parameters, and pretraining on 14.8T tokens. DeepSeek presented it as competitive with leading closed models while using efficient architecture choices such as multi-head latent attention and DeepSeekMoE. arXiv
On LiveCodeBench-style evaluations, however, DeepSeek-V3 is best understood as the bridge between strong coding LLMs and reasoning-specialized systems. In the DeepSeek-R1 report’s LiveCodeBench window from August 2024 to January 2025, DeepSeek-V3 scored 36.2 pass@1 with chain-of-thought prompting, compared with 32.9 for GPT-4o-0513, 38.9 for Claude 3.5 Sonnet-1022, 53.8 for o1-mini, 63.4 for o1-1217, and 65.9 for DeepSeek-R1. arXiv
| Model/report setting | LiveCodeBench result reported | What it suggests |
|---|---|---|
| GPT-4o-0513, DeepSeek-R1 report window | 32.9 pass@1-COT | Strong general model, but below reasoning-specialized systems in this setting |
| Claude 3.5 Sonnet-1022, same window | 38.9 pass@1-COT | Stronger than GPT-4o and DeepSeek-V3 in that table, still below o-series/R1 |
| DeepSeek-V3, same window | 36.2 pass@1-COT | Competitive non-reasoning frontier model, not yet the main jump |
| o1-mini, same window | 53.8 pass@1-COT | Inference-time reasoning changes the regime |
| o1-1217, same window | 63.4 pass@1-COT | Strong frontier reasoning baseline |
| DeepSeek-R1, same window | 65.9 pass@1-COT | Open reasoning model reaches or exceeds o1-level coding performance in reported setup |
The table should not be read as a universal ranking. It is a specific evaluation window, prompt style, and metric. Its importance is directional: by early 2025, LiveCodeBench rewarded models that could deliberate, search internally, and recover from reasoning errors, not merely models with broad code pretraining.
DeepSeek later reported a March 2025 upgrade from DeepSeek-V3 to DeepSeek-V3-0324, with LiveCodeBench improving from 39.2 to 49.2 in its official changelog. That reinforces the same point: code-benchmark performance was becoming a post-training and reasoning-behavior target, not just a pretraining artifact. api-docs.deepseek.com
7.3 OpenAI o1, o3, and o4-mini
OpenAI’s o-series made inference-time reasoning central. OpenAI describes o1 models as trained with large-scale reinforcement learning to reason using chain-of-thought before answering, with o1-mini positioned as faster and especially effective for coding. OpenAI’s o3 and o4-mini system card similarly describes those models as reasoning systems trained with large-scale reinforcement learning and as strong in complex math, coding, and science tasks. OpenAI
LiveCodeBench leaderboards reflect this shift. Official leaderboard snippets for the 2025-era windows show o-series models near the top: one official v5 snapshot lists o4-mini high at 76.5, o1-2024-12-17 high at 73.1, and o3-mini-2025-01-31 high at 71.6; another current official snippet lists o3 high at 75.8 and o4-mini medium at 74.2. These values are leaderboard-window-specific, not directly comparable to older LiveCodeBench releases. LiveCodeBench
The trajectory is clear even if exact cross-window comparisons are not. GPT-4-era systems made LiveCodeBench difficult but tractable. o-series systems made it a reasoning benchmark. A score on LiveCodeBench increasingly reflected a model’s ability to allocate inference compute, maintain long problem-solving traces, and handle adversarial edge cases.
7.4 DeepSeek-R1 and R1-0528
DeepSeek-R1 made the reasoning-model shift explicit in open-weight form. The R1 paper describes reinforcement learning applied to DeepSeek-V3-Base, with R1 and R1-Zero released alongside distilled variants; it reports that R1 achieved performance comparable to OpenAI o1-1217 on reasoning benchmarks. arXiv
For LiveCodeBench, the headline was that R1 outperformed DeepSeek-V3 by a wide margin in the reported August 2024–January 2025 window. The paper also reports distilled models with strong LiveCodeBench results, including a DeepSeek-R1-Distill-Qwen-32B score of 57.2, comparable to o1-mini in that table. arXiv
DeepSeek’s later R1-0528 update continued that pattern. DeepSeek’s official changelog reports LiveCodeBench v6 improving from 63.5 to 73.3, and Reuters reported that the updated R1 model ranked just behind OpenAI’s o4-mini and o3 on LiveCodeBench code generation at the time of release. api-docs.deepseek.com+1
The contamination-resistant angle matters here. A public benchmark that simply listed “R1 got 73.3” would invite the same ambiguity as older leaderboards. LiveCodeBench’s value is that the score can be interpreted relative to specific release windows and problem dates. It is still not immune to benchmark optimization, but it gives evaluators more structure for asking whether gains are broad or temporally local.
7.5 Claude reasoning and the contest-vs-real-world split
Anthropic’s Claude line complicates the story because Anthropic has emphasized real-world coding and tool use as much as contest-style programming. Claude 3.7 Sonnet was presented as both an ordinary language model and a reasoning model, with an extended-thinking mode that lets users control a thinking-token budget. Anthropic also stated that it was less focused on math and computer-science competition problems and more focused on real-world tasks, while introducing Claude Code for repository-level coding workflows. Anthropic
That positioning helps explain why Claude reasoning models can be highly valuable for engineering while not always topping LiveCodeBench. Official LiveCodeBench snippets for the current leaderboard list Claude Opus 4 Thinking around 56.6, Claude Sonnet 4 Thinking around 55.9, and Claude Sonnet 4 around 47.1 in the retrieved current window, below top o-series and DeepSeek-R1-0528 entries in the same snippet context. LiveCodeBench
This is not a contradiction. LiveCodeBench measures competitive-programming problem solving under executable tests. Claude Code and similar agentic coding products target repository navigation, editing, testing, and iterative development. The overlap is real but partial. A model can be excellent at production coding assistance and still be less optimized for Codeforces-style algorithmic puzzles than a model trained or post-trained for contest reasoning.
8. What LiveCodeBench contributes methodologically
The strongest version of LiveCodeBench’s contribution is not “we built a better coding leaderboard.” It is this:
A benchmark item should carry provenance metadata rich enough that contamination hypotheses become empirically testable.
That principle generalizes beyond coding. It suggests that serious benchmarks should expose, at minimum, item creation date, publication date, source domain, transformation history, and whether ground truth is public or private. In code evaluation, it also suggests recording test-suite origin, judge format, known editorials, and whether canonical solutions are public.
| Methodological contribution | Why it matters |
|---|---|
| Explicit release dates | Enables pre/post cutoff analysis instead of untestable cleanliness claims |
| Time-windowed reporting | Prevents a single scalar score from hiding temporal asymmetries |
| Continual item addition | Reduces immediate saturation and forces models to face newer problems |
| Executable correctness | Avoids evaluator-model subjectivity for many tasks |
| Multi-task reuse of the same problem pool | Separates solution generation from code reasoning and execution understanding |
| Public contamination analysis | Makes benchmark governance visible to users |
LiveCodeBench therefore belongs in the same family as Dynamic Benchmarks such as LiveBench and WIKIMIA-style temporal datasets. LiveBench uses recent and frequently updated questions with objective ground-truth answers to reduce contamination risk across broader capability categories, while WIKIMIA uses pre- and post-training temporal splits to support membership-inference-style contamination detection. arXiv+2LiveBench+2
The distinction is that LiveCodeBench uses an existing ecosystem of competitive-programming platforms where problem dates and executable judges are already part of the culture. That makes it unusually practical. The benchmark did not need to invent all items from scratch; it needed to curate, tag, test, and update them in a way aligned with model-release timelines.
9. The limits of time-gated public benchmarks
Time-gated evaluation is an advance, but it is not a permanent solution. It has at least five failure modes.
First, public “fresh” items become old. A problem released after a model’s pretraining cutoff may be clean for that model, but once the benchmark is public, later models can train on the problem statements, solutions, editorials, discussions, and leaderboard traces. This means LiveCodeBench items decay as measurement instruments.
Second, model cutoffs are often incomplete or strategically vague. A provider may disclose a pretraining cutoff but not post-training data, RL environments, synthetic data generation dates, tool-use traces, or benchmark-filtering procedures. Time-gated analysis is strongest when model data provenance is also transparent.
Third, recent-only windows shrink the sample. The LiveCodeBench paper itself notes that evaluating newer models only on post-cutoff problems can reduce the available evaluation set, making small score differences less reliable. arXiv
Fourth, temporal performance differences can be confounded by construction artifacts. A 2026 paper, A Test of Time: Rethinking the Temporal Signal of LLM Benchmarks, argues that post-cutoff performance decay is often interpreted as contamination evidence, but the signal is sensitive to benchmark construction. The authors report that transforming LiveCodeBench problems with an LLM while preserving solutions can remove the clear post-cutoff decay pattern, implying that temporal decay is a fragile signal rather than a definitive contamination test. arXiv
Fifth, contest-programming recency is not the same as software-engineering capability. A model can improve on LiveCodeBench by learning algorithmic templates, search strategies, and edge-case reasoning, while still struggling with multi-file refactors, ambiguous requirements, dependency conflicts, or long-running project work. That is not a defect in LiveCodeBench; it is a scope boundary.
The right conclusion is neither “time-gated benchmarks solve contamination” nor “time-gated benchmarks are useless.” The useful pattern is layered evaluation:
| Layer | Role |
|---|---|
| Public rolling benchmark | Supports reproducible comparison and community debugging |
| Private heldout set | Protects against direct leaderboard overfitting |
| Dated item metadata | Enables temporal contamination analysis |
| Perturbation audits | Tests whether scores survive rephrasing, renaming, and format changes |
| Domain portfolio | Separates contest coding, repository engineering, tool use, and debugging |
| Model-side disclosure | Makes cutoff-based claims more credible |
LiveCodeBench is best viewed as one layer in that stack. It gives the community a better public instrument, not a final theory of capability measurement.
10. How to read LiveCodeBench scores
A LiveCodeBench score should be interpreted with more care than a static benchmark score. The minimal responsible reading includes four qualifiers.
First, identify the benchmark release and time window. LiveCodeBench has had multiple releases and leaderboard windows. A score from an August 2024–January 2025 window is not directly comparable to a score from an August 2024–May 2025 or later window unless the problem set and prompting protocol are aligned. Official leaderboard snippets explicitly refer to selected current time windows, such as an August 1, 2024 to May 1, 2025 window with 454 selected problems. LiveCodeBench
Second, distinguish pass@1, pass@k, and reasoning-budget settings. A model allowed many samples or a large thinking budget is solving a different evaluation problem from a model producing one short answer. Reasoning models can convert extra inference compute into higher contest-solving accuracy.
Third, check whether the benchmark is measuring code generation or another task. LiveCodeBench includes self-repair, code execution, and test-output prediction, and those tasks can rank models differently.
Fourth, interpret frontier-model scores as moving targets. DeepSeek, OpenAI, Anthropic, Google, xAI, Qwen, and others update models and inference settings frequently. Recent leaderboard analyses are useful snapshots, not stable facts about a model family’s permanent capability. Reuters’ reporting on DeepSeek-R1-0528, for example, described its LiveCodeBench placement at the time of release; later leaderboard states can change as new models and windows appear. Reuters
11. LiveCodeBench versus related contamination-resistant approaches
LiveCodeBench is one answer to contamination, but not the only one.
| Approach | Representative idea | Strength | Weakness |
|---|---|---|---|
| Exact decontamination | Remove training examples that match benchmark strings | Simple and auditable | Misses paraphrases, translations, and semantic duplicates |
| Fuzzy decontamination | Remove near-matches by n-gram or embedding similarity | Catches more leakage | Thresholds are brittle and can over- or under-filter |
| Membership inference | Detect whether examples were likely in training data | Tests exposure more directly | Requires assumptions about model probabilities and access |
| Dynamic public benchmarks | Add recent dated examples over time | Reduces stale-item saturation | Public items eventually leak |
| Private holdouts | Keep items secret until evaluation | Stronger anti-overfitting | Less transparent and harder to reproduce |
| Perturbation-based audits | Rephrase, rename, or transform benchmark items | Tests robustness of score signals | May alter difficulty or distribution |
| Time-gated benchmarks | Compare pre- and post-cutoff performance | Makes contamination temporally testable | Depends on reliable dates and stable item difficulty |
The strongest benchmark regimes combine these approaches. LiveCodeBench is especially valuable because it sits at the intersection of dynamic public benchmarking and executable correctness. It is less strong as a secrecy mechanism. Once a LiveCodeBench release is public, its contents can enter the data ecosystem just like HumanEval did.
12. The open question: is time-gating the right long-term pattern?
The open question is not whether time-gating is useful. It is useful. The question is whether public, time-gated benchmarks can remain central as models, agents, and training pipelines become more adaptive.
There are two plausible futures.
In the optimistic future, time-gated benchmarks become standard infrastructure. Every benchmark item carries publication metadata. Leaderboards report scores by release window. Providers disclose enough training and post-training provenance for cutoff analysis to matter. Public rolling benchmarks are paired with private heldouts and perturbation audits. Under this regime, LiveCodeBench is a template: not perfect, but structurally honest.
In the pessimistic future, live benchmarks decay faster than they can be refreshed. Public items are scraped into training data, benchmark-specific fine-tuning becomes routine, and models learn to recognize the benchmark distribution. Time-gated leaderboards become another optimization target, and the community returns to the same problem with newer dates.
The likely future is mixed. LiveCodeBench-style time gating will remain valuable, but only as part of a benchmark governance system. The benchmark’s deepest lesson is not “use LeetCode problems.” It is “make contamination measurable, report time explicitly, and stop treating a single public scalar score as a clean capability estimate.”
Selected reference map
| Source cluster | Descriptive reference role |
|---|---|
| LiveCodeBench paper and official site | Primary source for benchmark design, task formats, temporal analysis, and HumanEval comparison arXiv+2arXiv+2 |
| LiveCodeBench methods sections | Primary source for data collection, release dates, testing, prompts, pass@1, and platform composition arXiv |
| LiveCodeBench contamination and limitations sections | Primary source for DeepSeek/GPT-4o temporal drops, HumanEval overfitting comparison, and benchmark-scope caveats arXiv+2arXiv+2 |
| HumanEval and EvalPlus | Primary/reference sources for functional-correctness evaluation and test-suite insufficiency in static code benchmarks arXiv |
| Dynamic and contamination-resistant benchmarking literature | Sources for decontamination limits, temporal detection, WIKIMIA, and LiveBench-style updating arXiv+2arXiv+2 |
| Frontier model trajectory | DeepSeek, OpenAI, Anthropic, Reuters, and official leaderboard sources for 2024–2026 model-performance context Reuters+4arXiv+4api-docs.deepseek.com+4 |
| Temporal-signal critique | 2026 evidence that post-cutoff performance decay can be fragile under benchmark transformations arXiv |
Companion entries
Core theory: Benchmark Contamination, Time-Gated Evaluation, Dynamic Benchmarks, Evaluation Provenance, Capability Measurement
Code evaluation: HumanEval, EvalPlus, MBPP, APPS, SWE-bench, Competitive Programming Benchmarks, Functional Correctness
Model trajectories: GPT-4, OpenAI o-series, DeepSeek-V3, DeepSeek-R1, Claude Reasoning Models, Inference-Time Scaling
Benchmark practice: Private Holdout Sets, Leaderboard Governance, Pass@k Metrics, Perturbation Audits, Evaluation Harness Design
Counterarguments and limits: Benchmark Saturation, Public Benchmark Decay, Temporal Signal Fragility, Contest Coding versus Software Engineering, Contamination Detection Is Not Capability Measurement