MMLU-CF
MMLU-CF is Zhao et al.’s contamination-resistant variant of MMLU, first released as a 2024 preprint and later published in ACL 2025. Its core claim is methodological: a broad multiple-choice benchmark should not merely measure model performance, but should also make benchmark leakage harder, more detectable, and less rewarding. The result is best understood as part of a wider shift from static public benchmark sets toward private, time-gated, and leakage-audited evaluation protocols.
Coverage note: verified through May 18, 2026.
Why MMLU-CF exists
MMLU became one of the canonical general-knowledge and reasoning benchmarks for large language models because it compressed many domains into a single, easily reported score. The original benchmark covers 57 subjects, ranging from elementary mathematics and U.S. history to computer science, law, and medicine; its paper framed the task as a test of broad multitask accuracy rather than as a narrow exam in one specialty. arXiv
That success created the benchmark’s central failure mode. Once a static benchmark is public, heavily used, mirrored into datasets, and optimized against by model developers, its test set is no longer an uncontaminated external measurement instrument. MMLU’s public availability made it useful for reproducibility and debugging, but also made it vulnerable to Benchmark Contamination: direct inclusion in training data, indirect paraphrase exposure, synthetic-data leakage, leaderboard overfitting, and prompt/template-specific artifacts. Zhao et al. explicitly position MMLU-CF against this failure mode, arguing that MMLU-style benchmarks need contamination resistance built into the benchmark design rather than handled only by post-hoc filtering. aclanthology.org
MMLU-CF is therefore not just “MMLU with different questions.” Its main contribution is an evaluation design: a private holdout test set, a public validation set released only after calibration, item transformations meant to reduce memorization payoff, and explicit leakage detection based on whether model outputs reveal access to hidden or formerly hidden item structure. Zhao et al. report that frontier models score substantially lower on MMLU-CF than on original MMLU, and that model rankings change in ways consistent with the original benchmark having accumulated contamination and saturation pressure. arXiv
The design: contamination resistance as a first-class requirement
MMLU-CF’s design can be summarized in four layers:
-
a broad MMLU-like question pool;
-
aggressive construction and cleaning;
-
decontamination transformations at the item level;
-
a split protocol that keeps the test set private while using the public validation set as a calibrated reference.
The important point is that these layers are not independent. The private split protects against future leakage; the public split allows developers to tune prompts and reproduce evaluation settings; the decontamination rules reduce gains from memorized surface form; and the validation–test relationship gives the benchmark maintainers a way to detect suspicious divergence after public release.
Question sourcing and filtering
Zhao et al. describe MMLU-CF as built from a very large source universe: more than 200 billion webpages, from which roughly 2.7 million candidate questions across more than 3,000 domains and 14 broad fields were collected before cleaning and filtering. The official dataset card describes a construction pipeline that includes MCQ collection, data cleaning, difficulty sampling, multiple LLM-based quality checks, and contamination-free processing, ending with a 10,000-question public validation set and a 10,000-question closed-source test set. aclanthology.org
The benchmark is deliberately MMLU-like in that it remains a multiple-choice test of broad knowledge and reasoning. It is not a new interactive agent benchmark, not a long-horizon task benchmark, and not a replacement for domain-specific exams such as code, math proof, tool use, or scientific reasoning suites. Its intended comparison class is the MMLU family: broad, multi-domain, mostly text-based, multiple-choice evaluation.
Public validation and private test
The most consequential design choice is the split between a public validation set and a private test set. MMLU-CF’s validation set is publicly available; the test set is closed-source, and model developers must request evaluation through the project workflow. The official repository states that validation can be evaluated through OpenCompass, while test-set evaluation requires submission through GitHub issues or temporary API access for API-only models. GitHub
This creates a deliberate asymmetry. Public validation supports reproducible prompt development and local sanity checks. Private testing preserves an uncontaminated holdout for final reporting. In practice, that makes MMLU-CF closer to an exam-administration protocol than to a fully open benchmark download.
The “time-gated” aspect matters here. The validation set was released publicly after the authors had already characterized its relationship to the private test set. The paper’s appendix reports that before validation release, validation and test scores were closely aligned: about 60% of evaluated models had a validation–test gap below 0.5 percentage points, and about 96% had a gap below 1 point. After public release, a widening gap can therefore function as an empirical warning sign that the public validation set has leaked or been overfit while the private test set has not. GitHub
This is a subtle but important methodological move. The public split is not merely a convenience set; it is a calibrated instrument. Once it becomes public, its future divergence from the private split becomes evidence about contamination.
Three item-level decontamination rules
Zhao et al. define three decontamination rules intended to reduce the value of memorized benchmark instances:
| Rule | Mechanism | Contamination failure targeted |
|---|---|---|
| Rephrase the question | Change the surface wording while preserving semantics | Exact or near-exact memorization of question text |
| Shuffle answer choices | Randomize option order | Memorization of answer position, e.g., “choice C” |
| Randomly replace choices with “None of the other choices” | Remove or alter some original distractor structure | Memorization of the full answer-option set |
The paper’s construction section explicitly lists these three rules and links them to known leakage pathways in public multiple-choice benchmarks. Rephrasing targets question-text memorization; shuffling targets answer-position artifacts; distractor replacement targets cases where a model recognizes the entire option set rather than solving the item. aclanthology.org
The rules are not a magic guarantee. A model could still learn the underlying facts, or even memorize a semantically equivalent version of the item. But that distinction is exactly the point: MMLU-CF tries to make benchmark-specific memorization less useful while preserving legitimate knowledge and reasoning. In other words, it does not try to prevent models from knowing medicine, law, history, or mathematics. It tries to prevent them from gaining credit merely because they have seen the exam.
Leakage detection by choice recovery and score gaps
MMLU-CF also includes explicit leakage detection. Zhao et al. test whether models can recover original multiple-choice structures in ways that imply exposure. In one reported experiment, they sample 1,000 cases from MMLU and MMLU-CF and examine 40 models. They report that on MMLU, a subset of models shows statistically significant signs of contamination, with outputs matching original choices at suspicious rates; after applying the decontamination rules, most models fall below a 1% suspicious matching rate, and on MMLU-CF all evaluated models remain below 0.2%. aclanthology.org
This connects MMLU-CF to a broader literature on Training Data Leakage Detection. Deng et al.’s Testset Slot Guessing work found that GPT-4 and ChatGPT could guess missing MMLU answer options at exact-match rates of 57% and 52%, respectively, when wrong answers or unlikely words were masked from benchmark items. That result is especially relevant because it shows that contamination can appear not only as high final accuracy, but also as unusual knowledge of the benchmark item’s form. aclanthology.org
Other contamination-detection papers support the same general concern. Yang et al. argue that n-gram overlap is insufficient because paraphrased or translated benchmark items can still contaminate training data; they show that models can overfit rephrased samples and that contamination can appear in both web corpora and synthetic data pipelines. Golchin and Surdeanu’s Data Contamination Quiz similarly treats exact benchmark wording as a detectable signal: if a model preferentially identifies original benchmark instances over perturbed versions, that suggests prior exposure. arXiv
MMLU-CF’s distinctive contribution is to internalize these lessons into the benchmark itself. It does not merely ask, after the fact, “Was MMLU contaminated?” It asks, before deployment, “How should an MMLU-like benchmark be structured so that contamination is harder to exploit and easier to detect?”
Empirical results: lower scores and rank disruption
The headline empirical result is that models score meaningfully lower on MMLU-CF than on original MMLU. Zhao et al. report GPT-4o at 73.4% in 5-shot and 71.9% in 0-shot on the MMLU-CF test set, compared with substantially higher original-MMLU reporting. arXiv
A selection from Zhao et al.’s reported frontier-model comparison illustrates the pattern:
| Model | Original MMLU | MMLU-CF | Drop |
|---|---|---|---|
| OpenAI o1 | 92.3 | 80.3 | -12.0 |
| DeepSeek-R1 | 90.8 | 76.3 | -13.5 |
| DeepSeek-V3 | 88.5 | 73.9 | -14.6 |
| GPT-4o | 88.0 | 73.4 | -14.6 |
| OpenAI o3-mini | 86.9 | 73.4 | -13.5 |
| GPT-4 Turbo | 86.5 | 70.4 | -16.1 |
| Llama-3.3-70B | 86.3 | 68.8 | -17.5 |
Zhao et al. report that the top three models remain relatively stable, but that ranking changes become more pronounced below the top tier; some models drop sharply, while others rise relative to their original-MMLU ordering. This is exactly what one would expect if original MMLU were partly saturated and partly contaminated: top models still perform well because they genuinely know a lot, but mid-tier ordering becomes less trustworthy when public-test familiarity and benchmark-specific optimization are mixed into the score. aclanthology.org
The decontamination ablation is also important. Zhao et al. apply the same three rules to MMLU and MMLU-CF items and report drops for GPT-4o, GPT-3.5, and Llama-3.1-8B. On original MMLU, GPT-4o falls from 88.0 to 79.8 after decontamination-style transformations; GPT-3.5 falls from 71.4 to 62.1; Llama-3.1-8B falls from 68.1 to 59.1. On MMLU-CF, the same transformations still reduce scores, but the contamination-detection experiment suggests much less direct benchmark leakage. aclanthology.org
The interpretation should be careful. A score drop under rephrasing, choice shuffling, and distractor replacement does not prove contamination by itself. It can also reveal brittle reasoning, prompt sensitivity, poor calibration, or dependence on superficial answer-choice patterns. But when combined with choice-recovery evidence and validation–test monitoring, the pattern strengthens the case that original MMLU scores contain a mixture of real capability, benchmark familiarity, and item-specific artifacts.
Methodological contribution
MMLU-CF is best read as a benchmark-methodology paper, not merely as a new leaderboard. Its contribution is the design pattern.
From post-hoc decontamination to contamination-aware construction
Most benchmark contamination work begins after a benchmark already exists. Researchers ask whether training corpora contain benchmark items, whether models can regurgitate answers, whether n-gram filters miss paraphrases, or whether performance drops under perturbation. Those are necessary audits, but they are reactive.
MMLU-CF moves the contamination problem upstream. It treats contamination resistance as a benchmark requirement from the start. That means the benchmark is designed around private holdouts, calibrated public splits, leakage diagnostics, and item transformations. It is not simply “cleaner MMLU”; it is a proposal for how broad LLM benchmarks should be built when public benchmark exposure is assumed to be inevitable. aclanthology.org
This makes MMLU-CF one of the first broad, MMLU-family multiple-choice benchmarks designed from the start around contamination resistance. It was not the first contamination-aware benchmark overall: contemporaneous work such as LiveBench, LiveCodeBench, and LatestEval also uses freshness, dynamic updates, recent sources, or continuously collected tasks to reduce contamination risk. But MMLU-CF is notable because it applies those concerns to the MMLU-style general-knowledge format rather than replacing that format with code contests, live tasks, or reading-comprehension freshness tests. arXiv+2arXiv+2
The benchmark as a measurement institution
The private-test design turns MMLU-CF into a lightweight measurement institution. A fully public benchmark is a static artifact: anyone can run it, but anyone can also train on it. A private benchmark is closer to a governed process: someone must administer the test, decide which models are eligible, prevent repeated-query overfitting, publish results, and maintain trust.
This is not a free upgrade. Private tests create governance burdens and reduce independent reproducibility. TRUCE and related private-benchmarking work emphasize the same tension: if open benchmarks are leaked, private evaluation becomes attractive; but private evaluation introduces new trust, access, and verification problems. arXiv
MMLU-CF’s public-validation/private-test split is a compromise. It preserves enough openness for prompt development and local evaluation while withholding the final measurement set. That compromise is likely to become more common, but it also makes benchmark reporting less frictionless than the original MMLU model.
Relationship to MMLU, MMLU-Pro, and MMLU-Redux
The MMLU family now contains several variants that are sometimes treated as substitutes. They should not be. Each variant addresses a different failure mode.
| Benchmark | Primary failure mode addressed | Main mechanism | What it is good for | Main limitation |
|---|---|---|---|---|
| MMLU | Need for broad multitask knowledge evaluation | 57 public subject tests, four-choice format | Historical comparability; broad capability trend lines | Public static test; contamination risk; saturation; item-quality concerns |
| MMLU-Pro | Saturation, shallow reasoning, answer-choice brittleness | Harder questions, more reasoning-focused filtering, 10 answer options, removal of trivial/noisy questions | More discriminative general-knowledge/reasoning evaluation | Not fundamentally a private or contamination-free protocol |
| MMLU-Redux | Incorrect or ambiguous MMLU items | Manual reannotation of MMLU questions across all 57 subjects | Auditing label quality and correcting benchmark errors | Does not solve public-test contamination; subset comparability issues |
| MMLU-CF | Training-data leakage and public benchmark contamination | Private test set, public validation set, decontamination rules, leakage diagnostics | Contamination-resistant MMLU-like evaluation | Requires centralized/gated test evaluation; still multiple-choice and text-only |
Original MMLU’s strength is also its weakness: it is public, widely known, easy to run, and historically entrenched. That makes it valuable for cross-paper comparison, but weak as a fresh test of frontier models trained after the benchmark became common. arXiv
MMLU-Pro addresses a different problem: original MMLU had become too easy and too sensitive to prompt artifacts for frontier models. The MMLU-Pro paper reports a more challenging, reasoning-focused benchmark, expands answer options from four to ten, removes trivial and noisy questions, and reports 16–33 percentage-point drops relative to MMLU while reducing prompt sensitivity. arXiv
MMLU-Redux addresses still another problem: some original MMLU questions are wrong, ambiguous, or poorly annotated. The MMLU-Redux paper manually reannotates 5,700 questions across all 57 MMLU subjects and estimates that 6.49% of MMLU questions contain errors, with some subjects much worse than average. The benchmark-quality issue is real, but it is not the same as contamination. A perfectly private benchmark can still contain bad labels; a perfectly labeled benchmark can still leak. arXiv
MMLU-CF addresses leakage. It does not primarily claim to be harder in the same way as MMLU-Pro, nor does it primarily claim to repair MMLU’s original labels in the same way as MMLU-Redux. Its purpose is to make broad MMLU-style evaluation less vulnerable to the fact that public benchmarks now circulate through pretraining corpora, fine-tuning sets, synthetic-data pipelines, model-selection loops, and leaderboard culture.
Which MMLU variant should practitioners report?
The wrong answer is: “report the highest one.” The right answer depends on the claim being made.
For historical comparability, report original MMLU, but label it as historical and high-contamination-risk. Original MMLU remains useful because many papers and model cards report it, but a new model’s high MMLU score should not be treated as strong evidence of uncontaminated capability unless the training-data and evaluation pipeline have been audited. The benchmark’s public status is not a minor caveat; it is central to interpreting the number.
For more discriminative broad reasoning, report MMLU-Pro. MMLU-Pro is better suited when the goal is to separate strong models on harder questions and reduce gains from trivial answer-choice patterns. It is especially useful when original MMLU scores are clustered near the top.
For label-quality analysis, report MMLU-Redux or use MMLU-Redux-style corrections. MMLU-Redux is most relevant when a paper discusses benchmark noise, annotation errors, or whether small score differences on original MMLU are meaningful. It is not a contamination-free replacement.
For contamination-resistant claims, report MMLU-CF test results where possible. If only the public validation set is used, the result should be labeled “MMLU-CF validation,” not treated as equivalent to private-test performance. The private test set is the contamination-resistant measurement instrument; the validation set is a public calibration and development split.
A practical reporting template is:
| Claim in paper or model card | Recommended MMLU-family metric |
|---|---|
| “Comparable to prior model reports” | Original MMLU, with prompt, shot count, harness, and contamination caveats |
| “Stronger broad reasoning under harder MCQ conditions” | MMLU-Pro |
| “Performance after correcting MMLU item-quality problems” | MMLU-Redux or a clearly specified Redux-based subset |
| “Broad MMLU-like performance with reduced leakage risk” | MMLU-CF private test |
| “Prompt/debugging sanity check before private submission” | MMLU-CF public validation |
The most defensible modern practice is to report at least two MMLU-family numbers: one for historical comparability and one for the specific failure mode the paper cares about. A frontier-model release that reports only original MMLU is leaving too much ambiguity. A paper that reports only MMLU-CF may be harder to compare with older work. A paper that reports MMLU, MMLU-Pro, and MMLU-CF together can separate three questions: “How does this compare historically?”, “How does it perform on harder MMLU-like reasoning?”, and “How much survives a contamination-resistant protocol?”
The cross-paper comparability problem
MMLU variants are not score-compatible.
An 88% on original MMLU and a 73% on MMLU-CF are not two measurements on the same ruler. They differ in item pools, exposure history, public/private status, answer-choice transformations, evaluation access, and sometimes prompt conventions. Similarly, MMLU-Pro’s 10-choice format changes random baseline probability and answer-selection behavior relative to MMLU’s four-choice format. MMLU-Redux’s manually corrected subset changes the distribution of valid questions. arXiv+2arXiv+2
This creates a serious comparability problem for papers, leaderboards, and model cards. A model evaluated on original MMLU in one paper, MMLU-Pro in another, and MMLU-CF validation in a third cannot be ranked by placing those numbers in a single column. The only safe comparisons are within the same benchmark variant, using the same split, prompt style, shot count, scoring convention, and evaluation harness.
The problem is worse for private benchmarks because independent replication is limited. MMLU-CF’s private test set protects against contamination, but the cost is that external researchers cannot freely rerun every model under every condition. That is a reasonable tradeoff for contamination resistance, but it means benchmark maintainers need transparent evaluation procedures, versioned prompts, model snapshot identifiers, and public result logs.
The cleanest reporting norm would be:
| Field | What to report |
|---|---|
| Benchmark variant | MMLU, MMLU-Pro, MMLU-Redux, MMLU-CF validation, or MMLU-CF test |
| Split | dev, validation, test, private test, or corrected subset |
| Shot count | 0-shot, 5-shot, chain-of-thought, or other |
| Prompt template | Exact template or evaluation harness identifier |
| Model snapshot | Model version, date, checkpoint, or API release |
| Contamination caveat | Public/static, private/gated, dynamic, audited, or unknown |
| Score uncertainty | Confidence intervals or bootstrap intervals where available |
Without these fields, cross-paper comparisons should be treated as approximate at best. The more public, older, and saturated the benchmark, the larger the interpretive uncertainty.
Relationship to contamination-detection literature
MMLU-CF sits in a cluster of recent work arguing that benchmark contamination is not a rare edge case but a structural problem for LLM evaluation.
Deng et al.’s work is directly relevant because it tests whether models can infer hidden benchmark content. Their Testset Slot Guessing method masks parts of benchmark items and asks whether a model can recover missing elements. On MMLU, they report exact-match rates for missing answer options that are too high to dismiss as ordinary problem-solving. This supports the idea that models may know the exam artifact, not just the subject matter. aclanthology.org
Yang et al.’s rephrased-sample contamination work attacks the assumption that exact string matching is sufficient. If a benchmark item is paraphrased, translated, embedded in synthetic data, or transformed into a similar training example, simple n-gram decontamination can miss it. This is particularly damaging for public benchmarks because modern data pipelines are not just web scrapes; they include instruction tuning, synthetic reasoning traces, retrieval augmentation, and benchmark-adjacent educational content. arXiv
Golchin and Surdeanu’s Data Contamination Quiz adds another useful perspective: a model may reveal exposure by distinguishing original benchmark instances from perturbed variants. This is close in spirit to MMLU-CF’s concern with surface-form memorization and option-set recovery. arXiv
Private-benchmarking work such as TRUCE generalizes the governance problem. If public benchmarks leak, one solution is to keep test sets private and evaluate models through controlled systems. But that introduces new questions: who administers the benchmark, how are results verified, how are repeated submissions handled, and how can the community trust a closed test without seeing the answers? arXiv
MMLU-CF’s answer is pragmatic rather than absolute. It does not solve every trust problem in private evaluation. It provides a concrete MMLU-like protocol that is more contamination-aware than the original benchmark while still giving practitioners a public validation split.
What MMLU-CF does not solve
MMLU-CF reduces one class of failure. It does not make multiple-choice evaluation a complete theory of intelligence.
First, MMLU-CF remains a multiple-choice benchmark. Multiple-choice tasks are efficient and easy to score, but they compress reasoning into answer selection. They can reward elimination heuristics, prior probabilities over answer styles, and calibration tricks that do not transfer cleanly to open-ended work.
Second, MMLU-CF is still mostly a text benchmark. It does not directly measure multimodal reasoning, tool use, long-horizon planning, software engineering, interactive debugging, embodied action, or scientific discovery. A model can improve on MMLU-CF without becoming a better agent in production.
Third, contamination resistance is not the same as construct validity. A benchmark can be private, fresh, and uncontaminated while still measuring the wrong thing. MMLU-CF is valuable because it makes one measurement problem less severe, not because it settles what “general intelligence” should mean.
Fourth, private tests can become bottlenecks. Evaluation access depends on maintainers, submission procedures, and accepted model interfaces. Zhao et al.’s repository describes workflows for Hugging Face models and API models, including temporary API-key submission for closed models; that is workable, but less frictionless than downloading a dataset and running a local script. GitHub
Finally, the authors themselves acknowledge limitations. The paper notes that some errors may remain, that the public validation set can become contaminated once released, and that the benchmark focuses on multiple-choice language tasks rather than broader modalities. aclanthology.org
Will contamination-free benchmark designs become the norm?
The evidence points toward partial normalization, not total replacement.
The case for normalization is strong. Frontier model developers now train on enormous web, code, synthetic, and instruction-tuning corpora. Public benchmarks are easy to ingest accidentally and easy to optimize against deliberately. Contamination-detection studies have repeatedly shown that exact-match filtering is too weak, that models can reveal benchmark exposure through unusual recovery behavior, and that benchmark artifacts can circulate through synthetic data. aclanthology.org+2arXiv+2
The benchmark ecosystem is already adapting. LiveBench emphasizes frequently updated questions from recent sources; LiveCodeBench continuously collects new programming problems to reduce contamination; LatestEval uses recent texts to construct fresh reading-comprehension evaluations. These are not identical to MMLU-CF, but they share the same premise: static public test sets are no longer sufficient for high-stakes frontier-model claims. arXiv+2arXiv+2
The case against full replacement is also strong. Public benchmarks are useful precisely because they are public. They allow local debugging, independent replication, educational use, ablation studies, prompt comparisons, and low-friction evaluation across many models. Private benchmarks protect measurement validity but weaken transparency. They can also be gamed by repeated submissions unless governance is strict.
The likely outcome is a layered evaluation stack:
| Layer | Role |
|---|---|
| Public historical benchmarks | Trend continuity and comparison with older work |
| Harder public variants | Better discrimination among current models |
| Corrected public subsets | Item-quality audits and cleaner historical interpretation |
| Private or time-gated benchmarks | Higher-trust claims about uncontaminated performance |
| Dynamic/live benchmarks | Freshness against rapidly evolving training corpora |
| Task-specific evaluations | Construct validity for coding, math, agents, science, safety, and deployment |
In that stack, MMLU-CF’s role is not to delete MMLU. It is to show how a familiar broad benchmark format can be rebuilt for an era in which public benchmark exposure is assumed. Its long-term importance may be less about the exact 10,000 private questions and more about the norm it pushes: benchmark designers should specify not only what a task measures, but how the task resists leakage, how leakage would be detected, and what public/private split is appropriate for the benchmark’s intended use.
References
[Zhao et al. — MMLU-CF: contamination-free MMLU variant]
Primary paper and proceedings record for MMLU-CF, including the closed-source test set, public validation set, three decontamination rules, and reported model results. arXiv+2aclanthology.org+2
[Microsoft — MMLU-CF repository and dataset documentation]
Official implementation and dataset documentation, including validation-set release, OpenCompass support, and private test-set evaluation workflow. GitHub
[Hendrycks et al. — Measuring Massive Multitask Language Understanding]
Original MMLU paper defining the 57-subject broad multitask evaluation setting. arXiv
[Wang et al. — MMLU-Pro]
MMLU-Pro paper introducing a harder, more reasoning-focused MMLU variant with 10 answer options and reduced prompt sensitivity. arXiv
[Gema et al. — MMLU-Redux]
Manual reannotation study estimating MMLU item-error rates and proposing a corrected subset for benchmark-quality analysis. arXiv
[Deng et al. — Testset Slot Guessing]
Contamination-detection work showing that models can recover missing MMLU answer-option content at suspiciously high rates. aclanthology.org
[Yang et al. — Rethinking Benchmark and Contamination for LLMs]
Study arguing that n-gram overlap is insufficient for contamination detection because paraphrases and translations can still leak benchmark information. arXiv
[Golchin and Surdeanu — Data Contamination Quiz]
Detection method using perturbed benchmark instances to test whether a model recognizes original benchmark wording. arXiv
[TRUCE — private benchmarking for LLM evaluation]
Private-evaluation framework discussing the trust and infrastructure problems that arise when public benchmarks are contaminated. arXiv
[Ishida et al. — publishing benchmarks without giving answers away]
Benchmark-publication proposal addressing the tension between public benchmark release and future contamination risk. arXiv
Companion entries
Core theory: Benchmark Contamination, Training Data Leakage Detection, Memorization vs Generalization, Static vs Dynamic Benchmarks, Private Benchmarking, Construct Validity
MMLU family: MMLU, MMLU-Pro, MMLU-Redux, MMLU-CF, Multiple-Choice Evaluation, Benchmark Saturation
Practice: Evaluation Harnesses, OpenCompass, Model Cards, Leaderboard Governance, Benchmark Reporting Standards, Contamination Audits
Counterarguments: Closed Benchmark Governance, Eval Overfitting, Goodhart’s Law, Public Reproducibility, Benchmark Validity Limits