MMLU: Anatomy of a Saturated Benchmark
MMLU became the canonical broad-knowledge benchmark because it compressed a large slice of academic and professional question-answering into a simple, reproducible multiple-choice test. Its very success made it less decisive: once frontier systems clustered around the high-80s to low-90s, the remaining score gap became a mixture of real capability, prompt dependence, contamination risk, item noise, and leaderboard overfitting.
Coverage note: verified through May 11, 2026.
1. Why MMLU mattered
Massive Multitask Language Understanding, or MMLU, was introduced by Hendrycks et al. as a broad test of language-model knowledge and problem-solving across 57 subjects, ranging from elementary mathematics and U.S. history to computer science, law, medicine, morality, and other academic or professional domains. The original paper framed the benchmark as a stress test for “massive multitask” generality: a model could not do well merely by mastering one narrow format or one school subject. It had to answer many types of questions, most in a terse multiple-choice format, across domains that humans usually learn through years of formal education. arXiv
MMLU’s importance came from three design choices. First, it was broad enough to feel like a proxy for general intellectual coverage. Second, it was easy to evaluate: four-way multiple choice, scored by accuracy. Third, it arrived at a time when large language models were beginning to look broadly capable but still failed many structured academic tasks. In the original results, smaller GPT-3 models were close to random-choice performance, while the largest GPT-3 model was far below the later frontier-model regime. That made MMLU a useful “capability frontier” benchmark rather than merely a release checklist. arXiv
The later history is the familiar benchmark arc: release, adoption, leaderboard pressure, saturation, critique, and successor benchmarks. MMLU is now less interesting as a single-number proof of model intelligence than as a case study in Benchmark Saturation, Goodhart’s Law, and Evaluation Drift.
2. The original design: broad, simple, and deliberately unforgiving
Hendrycks et al.’s MMLU design used 57 tasks grouped across broad academic and professional areas. The benchmark included humanities, social sciences, STEM, and applied professional subjects. Its questions were multiple-choice and were intended to span difficulty from undergraduate-level material to expert-level knowledge. The paper’s abstract emphasized that high accuracy would require both world knowledge and problem-solving ability, and it noted that the best models at the time remained substantially below expert-level performance. arXiv
| Design axis | MMLU choice | Consequence |
|---|---|---|
| Subject breadth | 57 subjects across academic and professional domains | A single score summarizes broad coverage, but can hide subject-level weaknesses. |
| Format | Mostly four-option multiple-choice questions | Evaluation is cheap and reproducible, but vulnerable to option-order effects, distractor artifacts, and answer-format sensitivity. |
| Difficulty | Undergraduate through expert-level questions | The benchmark initially separated weak, mid-tier, and frontier models; later it lost dynamic range near the top. |
| Scoring | Accuracy over fixed questions | Easy to compare, but a one-point difference can become hard to interpret near saturation. |
| Public availability | Open benchmark and widely integrated into evaluation harnesses | Accelerated adoption, but increased contamination and leaderboard overfitting risk. |
The elegance of this design was also its long-term weakness. A four-way multiple-choice benchmark has a random baseline of 25%, a limited number of distractors, and a fixed public item pool. At first, these properties made MMLU easy to run. Later, they made it easy to optimize against, memorize accidentally, or game through prompt and scoring choices. Multiple-Choice Evaluation is not intrinsically flawed, but at the frontier it becomes a fragile measurement instrument.
3. Performance trajectory: from near-random to frontier clustering
The often-repeated story is “GPT-3 scored around 25–30%; frontier models score around 88–95%.” That shorthand needs correction. In the original MMLU paper, smaller GPT-3-family models were near the 25% random baseline, but GPT-3 175B reached about 43.9% in a few-shot setting, with zero-shot performance lower. The broader point still holds: early large language models were far below expert-level performance, while later frontier systems moved into the high-80s and beyond. arXiv
| Approximate era | Representative MMLU result | Interpretation |
|---|---|---|
| 2020–2021: original MMLU period | Smaller GPT-3 variants near random; GPT-3 175B roughly 43.9% few-shot | MMLU clearly exposed broad knowledge and reasoning limitations in early LLMs. arXiv |
| 2022: compute-optimal scaling period | Chinchilla reported 67.5% average accuracy on MMLU | Scaling and training-data efficiency moved MMLU from “hard frontier” toward “core benchmark.” arXiv |
| 2023: GPT-4 period | GPT-3.5: 70.0%; GPT-4: 86.4% in the GPT-4 technical report | GPT-4 turned MMLU into a headline benchmark for broad professional and academic competence. arXiv |
| 2024: clustered frontier period | GPT-4o and other frontier models clustered around the high-80s on common MMLU settings | Small MMLU differences became less diagnostic, especially across different prompting and scoring setups. ar5iv |
| 2025–2026: post-saturation framing | HLE authors describe frontier LLMs as exceeding 90% on MMLU | MMLU increasingly functions as a saturated broad-knowledge check rather than a frontier discriminator. Nature |
These numbers are not perfectly apples-to-apples. Reported MMLU scores vary by prompting style, few-shot examples, chain-of-thought usage, answer-extraction method, model snapshot, and evaluation harness. A score difference that looks crisp in a table may reflect an experimental protocol difference rather than a stable capability difference. This is one reason Evaluation Harnesses matter so much once a benchmark becomes crowded near the ceiling.
4. What “saturation” means here
A benchmark is saturated when additional progress on the benchmark no longer gives a clear measurement of the underlying capability the benchmark was meant to track. Saturation does not mean the underlying task is solved, that the model understands everything in the benchmark, or that the remaining wrong answers are meaningless. It means that the benchmark’s signal-to-noise ratio has degraded for the population of systems being compared.
For MMLU, saturation has three layers.
First, there is score compression. Once several frontier models sit around the high-80s or low-90s, the benchmark has little room left to separate them. MMLU-Pro’s authors explicitly describe this clustering problem: GPT-4-class models were separated by only small margins on MMLU, while the same models separated more clearly on MMLU-Pro. They also estimate that original MMLU had much less remaining headroom than MMLU-Pro. ar5iv
Second, there is noise amplification. Near the ceiling, a 1–2 percentage-point gain can come from genuine capability, better prompting, better answer parsing, accidental benchmark exposure, or quirks in a subset of items. MMLU-Redux later estimated that a nontrivial fraction of original MMLU questions contain errors, and its authors found that a large share of the Virology subset they analyzed had problems. That does not mean MMLU is useless; it means tiny deltas at the top are unsafe to interpret as clean capability gains. ACL Anthology
Third, there is Goodhart pressure. Once MMLU became a standard leaderboard target, model developers had incentives to optimize data mixtures, prompts, training curricula, and release narratives around it. The benchmark’s public status made reproducibility easy, but also made it likely that MMLU-like questions, explanations, answer keys, or paraphrases would appear in training corpora or synthetic-data pipelines. Goodhart’s Law is not a metaphor here; the measure became a target.
The remaining gap from 90% to 100% is therefore mixed. Some of it is real: specialized domains, multi-step calculations, ambiguous professional judgments, and brittle reasoning still expose limitations. Some of it is measurement error: bad labels, ambiguous wording, formatting artifacts, and option-order effects. Some of it is epistemic uncertainty: with closed-source training data and proprietary evaluation recipes, outsiders often cannot distinguish clean competence from partial exposure.
5. Methodology critique I: contamination
Data Contamination is the central critique of any public fixed benchmark used to evaluate models trained on internet-scale corpora. Contamination can mean direct exposure to test questions and labels, exposure to test questions without labels, exposure to paraphrases, exposure to explanations, or exposure to synthetic derivatives of benchmark material. The issue is especially hard for closed-source models, where the training corpus and filtering pipeline are not fully inspectable. A survey of benchmark contamination methods distinguishes open-data and closed-data settings, input-only and input-plus-label contamination, and white-box, gray-box, and black-box detection regimes. arXiv
Simple string matching is not enough. Work on rephrased benchmark samples argues that n-gram overlap and embedding similarity can miss contamination when benchmark items are paraphrased or translated, and it shows that contaminated training can dramatically inflate benchmark performance. The paper explicitly evaluates this problem on benchmarks including MMLU, GSM8K, and HumanEval, and argues for stronger decontamination protocols and fresh exams. arXiv
Black-box contamination detection is also imperfect but informative. Deng et al. proposed a retrieval-based system and a “Testset Slot Guessing” method for modern benchmarks, including MMLU. In that setup, the model is asked to reconstruct masked benchmark content, such as a missing answer option. The authors report that ChatGPT and GPT-4 could exactly match missing MMLU options at rates high enough to raise contamination concerns, though such tests are evidence of exposure risk rather than a complete proof of training-set leakage. ACL Anthology
OpenAI’s GPT-4 technical report included contamination analysis and reported a low estimated contamination rate for MMLU, with GPT-4 scoring 86.4% and GPT-3.5 scoring 70.0% under the reported evaluation. But the report’s own methodology also illustrates the challenge: contamination estimates depend on sampling, matching rules, and what counts as meaningful overlap. A benchmark can be “mostly clean” under one criterion while still being vulnerable to paraphrased exposure or synthetic-data leakage. arXiv
The strongest conclusion is not “MMLU scores are fake.” It is narrower and more important: for frontier systems trained on huge, opaque corpora, a public fixed benchmark cannot provide high-confidence evidence of clean generalization on its own. MMLU still measures something, but the contamination-adjusted interpretation is weaker than the leaderboard interpretation.
6. Methodology critique II: item quality
Item quality matters more near saturation. If a model scores 45%, a handful of ambiguous or mislabeled questions will not change the qualitative conclusion. If several models score 88–91%, item defects become part of the ranking mechanism.
MMLU-Redux directly targets this problem. Its authors describe errors in MMLU, report that 57% of an analyzed Virology subset contained errors, and introduce MMLU-Redux as a manually re-annotated subset of 5,700 questions across all 57 subjects. A related abstract reports an estimated 6.49% error rate in MMLU. Even if that estimate is debated, it is large enough to make fine-grained frontier comparisons suspect. ACL Anthology
This point is often misunderstood. A 6.49% estimated item-error rate does not mechanically imply that the maximum meaningful score is 93.51%. A model can answer mislabeled questions in ways that interact oddly with the official answer key, and errors may be unevenly distributed across subjects. The correct conclusion is subtler: near the ceiling, benchmark accuracy is not a pure measure of model capability. It is a composite of model competence, item validity, label correctness, ambiguity tolerance, and scoring convention.
Item quality also intersects with domain expertise. A benchmark covering law, medicine, virology, physics, and moral reasoning cannot be audited like a single-purpose arithmetic dataset. Each subject requires its own experts, and even experts may disagree about ambiguous wording, outdated facts, or professional edge cases. That makes MMLU impressive as a broad benchmark but fragile as a fine-grained frontier ruler.
7. Methodology critique III: format dependence
MMLU is not merely a test of knowledge; it is a test of knowledge expressed through a particular prompt, answer-choice format, and scoring procedure. Format dependence becomes a serious problem when models are close together.
MMLU-Pro’s authors argue that original MMLU is sensitive to prompt and scoring choices, and they report that MMLU-Pro reduces prompt sensitivity from roughly 4–5% to about 2%. They also find that chain-of-thought prompting helps substantially more on MMLU-Pro than on original MMLU, suggesting that original MMLU often rewards answer selection more than explicit multi-step reasoning. arXiv
Other work finds that multiple-choice leaderboard rankings can shift under seemingly minor perturbations. “When Benchmarks are Targets” reports that for multiple-choice benchmarks such as MMLU, changes like answer-order perturbations or answer-selection methods can move models several positions in a ranking. Separate work on answer-order effects finds that changing answer labels can reduce accuracy across evaluated models, with sensitivity varying by model. arXiv
This does not make multiple-choice evaluation invalid. Humans are also affected by question wording and answer framing. But it does mean that a saturated multiple-choice benchmark should not be treated as a precise psychometric instrument unless the protocol is locked, blinded, and replicated. For MMLU, many public comparisons are not.
8. Methodology critique IV: four choices, shallow distractors, and shortcut learning
Original MMLU’s four-choice format has a 25% random baseline. That is convenient, but it makes distractor quality central. If distractors are weak, a model can eliminate bad options without fully solving the problem. If answer choices contain stylistic artifacts, the model can exploit them. If the correct answer is semantically more plausible than the distractors, the task may measure recognition rather than robust reasoning.
MMLU-Pro was designed partly as a response to this problem. It expands the number of choices from four to ten, removes trivial or noisy questions, incorporates harder college-level exam problems, and uses expert review. Its authors report that model accuracy drops by 16–33 percentage points relative to original MMLU and that MMLU-Pro better separates frontier systems. arXiv
The ten-choice design does not magically solve evaluation. More options can create new ambiguity, and harder questions can be noisier. But it does reduce the chance baseline, weakens simple elimination strategies, and creates more room between strong models. It is best understood as MMLU hardening, not as a clean replacement for all evaluation needs.
9. Variants and successors
MMLU’s successors split into three families: repair, hardening, and replacement. Repair tries to fix the original benchmark. Hardening preserves the broad academic idea but makes the test harder and less format-sensitive. Replacement builds new tasks aimed at expert reasoning, contamination resistance, or frontier-level difficulty.
| Benchmark | Type | What it changes | What it reveals |
|---|---|---|---|
| MMLU-Redux | Repair | Re-annotates a subset of MMLU and analyzes item errors | Original MMLU contains enough item noise that fine-grained frontier rankings are risky. ACL Anthology |
| MMLU-Pro | Hardening | Uses 14 domains, more than 12,000 questions, ten answer options, harder problems, and expert review | Frontier models that cluster on MMLU separate more clearly on a harder variant. arXiv |
| MMLU-CF | Contamination-control variant | Builds a contamination-free validation/test structure with a closed-source test set and decontamination rules | Public MMLU-style evaluation needs stronger test secrecy and contamination discipline. arXiv |
| GPQA | Expert-depth successor | Uses 448 expert-written multiple-choice questions in biology, physics, and chemistry | Expert-level scientific QA remains difficult even when broad MMLU is saturated. arXiv |
| Humanity’s Last Exam | Frontier replacement | Uses 2,500 difficult multimodal and text questions across many subjects, with a public release and private held-out test set | Broad closed-ended expert knowledge can still expose large gaps in frontier models, but item validity remains contested. Nature |
MMLU-Pro
MMLU-Pro is the most direct successor. It keeps the broad academic frame but targets MMLU’s weaknesses: too few distractors, too much shortcutting, noisy or trivial questions, and prompt sensitivity. The paper describes MMLU-Pro as spanning 14 domains with over 12,000 questions, expanding answer choices from four to ten, and using harder questions plus expert review. It reports that GPT-4o scores 72.6% on MMLU-Pro, while GPT-4-Turbo scores 63.7%, a much larger separation than on original MMLU. ar5iv
The important lesson is not that MMLU-Pro is “the new MMLU” forever. It is that benchmark difficulty and discriminative power must be renewed as models improve. A benchmark that usefully separated GPT-3-class models may be too coarse for GPT-4-class models.
MMLU-Redux
MMLU-Redux asks a different question: before making MMLU harder, how much of the original benchmark is trustworthy? Its contribution is not primarily a harder leaderboard but an audit. The headline finding is that MMLU contains meaningful annotation and item-quality problems, with the Virology subset especially problematic in the authors’ analysis. ACL Anthology
For practitioners, MMLU-Redux is a warning about overinterpreting decimal-place leaderboard gains. If a model improves from 89.1% to 89.8%, the item-quality question is not optional. It is central.
GPQA
GPQA is narrower and deeper than MMLU. It contains 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. The authors report that experts with or working toward PhDs achieved 65% accuracy, or 74% when discounting clear mistakes identified retrospectively, while highly skilled non-experts achieved 34% despite spending more than 30 minutes per question with web access. The strongest GPT-4 baseline in the paper scored 39%. arXiv
GPQA’s value is that it moves from broad academic recognition toward expert scientific reasoning. Its weakness is also obvious: it is much narrower than MMLU. It is not a universal benchmark for general intelligence. It is a sharper instrument for a narrower slice of competence.
Humanity’s Last Exam
Humanity’s Last Exam, or HLE, is a more explicit post-saturation benchmark. Its authors argue that popular benchmarks have not kept pace with model progress and that frontier LLMs now exceed 90% on MMLU. HLE contains 2,500 challenging questions across many subjects, including multimodal items, multiple-choice questions with five or more options, and exact-match questions. The stated goal is to test the limit of closed-ended, verifiable academic knowledge. Nature
HLE also illustrates the difficulty of successor benchmarks. Harder questions are not automatically better questions. If a benchmark is built from extremely difficult expert items, item validity, ambiguity, and grading become harder to audit. Nature’s HLE page notes targeted expert-review disagreement in parts of the dataset, and external critiques have challenged subsets of HLE’s question quality. The evidence here is contested and subset-specific, but it shows that “harder than MMLU” does not remove the measurement problem; it relocates it. Nature
The HLE authors themselves are careful not to equate high HLE accuracy with AGI. They describe high HLE performance as evidence of expert-level performance on closed-ended, verifiable questions, not proof of autonomous research ability. That distinction should also be applied retroactively to MMLU: a high MMLU score is evidence about benchmarked academic QA, not a general certificate of understanding. agi.safe.ai
10. Does MMLU still discriminate between frontier models?
For frontier models, mostly no. MMLU can still catch severe regressions, weak pretraining, poor instruction tuning, or subject-level holes. But it is no longer a reliable way to rank frontier systems separated by one or two points. At that level, measurement noise, prompt choice, item quality, and contamination uncertainty can dominate the interpretation.
For non-frontier models, yes. MMLU remains useful when comparing small models, open-weight models, quantized variants, distilled systems, or domain-adapted checkpoints. A model scoring 45% and a model scoring 70% are meaningfully different. A model scoring 70% and a model scoring 86% are also meaningfully different. The saturation critique is local to the frontier regime, not a universal dismissal.
| Practitioner use case | Is MMLU still useful? | Better interpretation |
|---|---|---|
| Release smoke test | Yes | A hygiene check for broad academic competence and obvious regressions. |
| Comparing small or mid-tier models | Yes | Large MMLU gaps still carry useful signal. |
| Comparing frontier models within 1–2 points | Weakly | Treat as suggestive at most; require harder and cleaner evals. |
| Claiming expert reasoning ability | No | Use domain-specific expert evals, GPQA-like tests, tool-use tasks, and human audits. |
| Measuring production usefulness | No, not alone | Combine with task-specific evals, reliability tests, latency/cost measures, and failure-mode analysis. |
| Tracking contamination-resistant progress | Weakly | Prefer private, fresh, or contamination-controlled evaluations. |
The best operational stance is to treat MMLU as a hygiene check. A frontier model that performs poorly on MMLU has a problem. A frontier model that performs very well on MMLU has merely passed a baseline expectation. That distinction is crucial.
11. The remaining gap: capability or noise?
The remaining MMLU gap is not one thing. It decomposes into at least six components.
Residual knowledge gaps. Some MMLU questions require domain knowledge that a model may not possess or may not retrieve reliably from its parameters. This is especially true in specialized professional or scientific subdomains.
Reasoning failures. Some questions require multi-step reasoning, calculation, or careful elimination. Original MMLU is not a pure reasoning benchmark, but it contains enough reasoning-dependent items that failures can still be meaningful.
Ambiguous or flawed items. MMLU-Redux shows that a nontrivial number of questions have annotation or quality problems. These errors matter disproportionately near the top. ACL Anthology
Format artifacts. Answer order, prompt template, scoring method, and answer extraction can change performance. A model can know the answer but fail the format, or exploit the format without robustly knowing the answer. arXiv
Contamination or benchmark familiarity. Public benchmark items can appear in pretraining data, evaluation examples, blog posts, synthetic datasets, or benchmark discussions. The contamination literature shows that this risk is real and hard to eliminate with string matching alone. arXiv
Statistical and reporting variance. Small score differences can arise from model snapshot differences, decoding settings, few-shot examples, chain-of-thought policy, or evaluation-harness implementation. Once scores cluster, these details stop being boring implementation details and become part of the result.
The saturated-benchmark mistake is to collapse all six components into a single story: “model A is smarter than model B.” That may be true when the gap is large. It is not justified when the gap is small and the benchmark is saturated.
12. The broader lifecycle of benchmarks
MMLU’s trajectory is now a standard lifecycle for LLM evaluation.
| Lifecycle stage | What happens | MMLU example |
|---|---|---|
| Release | A new benchmark captures an unsolved capability | MMLU tests broad academic and professional QA across 57 subjects. |
| Adoption | The benchmark becomes easy to compare and cite | MMLU becomes a standard table entry in model releases and leaderboards. |
| Optimization | Developers tune models, prompts, and evaluation pipelines around it | MMLU becomes part of the implicit target function for frontier model development. |
| Saturation | Top models cluster near the ceiling | GPT-4-class systems reach high-80s; later papers describe frontier systems above 90%. |
| Critique | Researchers identify contamination, item-quality, and format issues | MMLU-Redux, contamination studies, and prompt-sensitivity analyses challenge fine-grained interpretation. |
| Successor | Harder, cleaner, or narrower benchmarks emerge | MMLU-Pro, GPQA, MMLU-CF, and HLE target the post-MMLU regime. |
This lifecycle is not a failure of MMLU. It is evidence that MMLU succeeded. A benchmark that never saturates may be too hard, too noisy, too obscure, or too irrelevant to become a useful development target. A benchmark that saturates quickly after broad adoption has probably measured something important enough for the field to optimize.
The danger is not saturation itself. The danger is continuing to use a saturated benchmark as though it were still frontier-discriminative. That is how benchmarks become misleading: not because they stop measuring anything, but because people keep asking them to measure too much.
13. What MMLU taught the field
MMLU taught four durable lessons.
First, broad academic coverage matters. Before MMLU, many evaluations were too narrow to expose generality failures. MMLU made it harder for a model to look strong by specializing in one dataset family.
Second, simple evaluation scales socially. MMLU became influential partly because it was easy to run, easy to explain, and easy to put in a table. Benchmark adoption is not only a scientific property; it is an infrastructural property.
Third, public fixed benchmarks are temporary instruments. Once a benchmark becomes famous, it becomes entangled with training data, synthetic data, prompt engineering, and release incentives. Benchmark Half-Life should be treated as a design constraint, not an afterthought.
Fourth, evaluation must move from single-score leaderboards toward portfolios. A frontier evaluation stack needs broad hygiene checks like MMLU, harder variants like MMLU-Pro, expert-depth tests like GPQA, contamination-controlled or private tests like MMLU-CF-style designs, and task-specific production evals. No single benchmark can carry the evidential load.
14. Interpretive bottom line
MMLU should now be read historically and diagnostically. Historically, it marked the point where broad academic QA became a central axis of LLM evaluation. Diagnostically, it remains useful for detecting weak general models and broad regressions. But as a frontier-ranking tool, it is saturated.
The right question is no longer “which frontier model has the highest MMLU?” The better question is: “Given that strong models now pass MMLU, what harder, cleaner, more domain-relevant, and less-contaminated evaluations expose the failures that still matter?” That question leads directly to MMLU-Pro, MMLU-Redux, GPQA, HLE, and private operational evals.
MMLU’s afterlife is therefore not obsolescence. It is demotion: from frontier benchmark to hygiene check, from headline number to historical baseline, from proof of capability to one row in a serious evaluation portfolio.
Selected evidence base
Hendrycks et al., “Measuring Massive Multitask Language Understanding.” Original MMLU source: 57 tasks, broad academic/professional coverage, multiple-choice format, and early evidence that GPT-3-era systems were far below expert-level performance. arXiv
OpenAI, “GPT-4 Technical Report.” Key source for GPT-4-era MMLU results, including GPT-3.5 at 70.0% and GPT-4 at 86.4% under the reported evaluation. arXiv
Wang et al., “MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark.” Primary source for the MMLU-Pro critique and design: ten choices, harder questions, expert review, lower prompt sensitivity, and greater separation among frontier models. arXiv
Gema et al., “Are We Done with MMLU?” / MMLU-Redux. Primary item-quality audit showing meaningful error rates and introducing a re-annotated subset across MMLU subjects. ACL Anthology
Deng et al., “Investigating Data Contamination in Modern Benchmarks for Large Language Models.” Primary black-box contamination-detection paper using retrieval and Testset Slot Guessing, including results on MMLU. ACL Anthology
“Rethinking Benchmark and Contamination for Language Models with Rephrased Samples.” Primary source for the claim that string or n-gram overlap is insufficient when contaminated examples are paraphrased or translated. arXiv
Rein et al., “GPQA: A Graduate-Level Google-Proof Q&A Benchmark.” Primary successor benchmark emphasizing expert-written science questions and expert/non-expert/model performance gaps. arXiv
Phan et al. and collaborators, “Humanity’s Last Exam.” Primary post-saturation benchmark arguing that popular benchmarks such as MMLU no longer keep pace with frontier models and proposing a harder multimodal/exact-answer evaluation. Nature
Companion entries
Core theory: Benchmark Saturation, Goodhart’s Law, Evaluation Drift, Benchmark Half-Life, Capability Elicitation, Measurement Noise
Benchmark lineage: MMLU, MMLU-Pro, MMLU-Redux, MMLU-CF, GPQA, Humanity’s Last Exam
Methodology: Data Contamination, Prompt Sensitivity, Multiple-Choice Evaluation, Evaluation Harnesses, Held-Out Test Sets, Item Response Theory for LLMs
Practice: Model Release Gating, Regression Testing for LLMs, Domain-Specific Evals, Private Benchmark Design, Production Evaluation Portfolios
Counterarguments: Benchmark Skepticism, Leaderboard Overfitting, Synthetic Benchmark Fragility, Expert Benchmark Validity, Contamination Detection Limits