MMLU-Pro: The Less-Saturated Successor

MMLU-Pro is best understood as a methodological repair of MMLU, not merely a harder leaderboard: Wang et al. kept MMLU’s broad academic and professional ambition while replacing weak measurement elements—four-option recognition, easy or exhausted items, thin distractors, and noisy labels—with a filtered, refreshed, ten-option, reasoning-weighted benchmark. Its initial value was discriminative headroom: frontier models that bunched together on MMLU spread out on MMLU-Pro, and chain-of-thought prompting mattered much more. By 2026, however, public leaderboards already show frontier systems pushing into roughly the 90% range, making MMLU-Pro’s long-term value depend on version discipline, contamination controls, and complementary variants such as MMLU-Pro+, MMLU-CF, and MMLU-Redux.

Coverage note: verified through May 12, 2026.

Why MMLU-Pro exists

The original MMLU benchmark, introduced by Hendrycks et al., was designed to measure broad multitask knowledge and problem-solving across 57 subjects, including STEM, humanities, social sciences, and professional domains. Its test split contains 14,042 questions, each with four answer choices, making random guessing worth 25% before any language understanding enters the picture. At release, MMLU still had substantial headroom: the benchmark card reports GPT-3 at 43.9%, UnifiedQA at 48.9%, and a random baseline at 25%. arXiv

MMLU’s success created the conditions for its decline as a frontier-model discriminator. Once a benchmark becomes canonical, it becomes part of model-development culture: papers report it, data pipelines screen against it imperfectly, prompt templates optimize for it, and leaderboard differences shrink. Wang et al.’s MMLU-Pro was created to recover some of the measurement pressure MMLU originally had: the benchmark adds reasoning-heavy questions, expands choices from four to up to ten, removes trivial or noisy items, and verifies questions more aggressively. The authors report that MMLU-Pro caused a 16–33 percentage-point drop relative to MMLU across evaluated models, while reducing prompt sensitivity from roughly 4–5% on MMLU to around 2% on MMLU-Pro. NeurIPS Papers

One subtle but important correction: MMLU-Pro does not preserve MMLU’s original 57-subject reporting scheme one-to-one. It preserves MMLU’s broad academic and professional coverage goal, but merges the subject space into 14 broader domains and adds a large number of refreshed items from sources beyond MMLU. The Hugging Face benchmark card lists 12,032 total questions, with 6,810 originating from MMLU and 5,222 newly added; the domains include biology, business, chemistry, computer science, economics, engineering, health, history, law, math, philosophy, physics, psychology, and “other.” Hugging Face

Design at a glance

Benchmark	Main purpose	Subject structure	Answer format	Approximate scale	Contamination / quality posture
MMLU	Broad multitask academic and professional knowledge	57 subjects	4-choice multiple choice	14,042 test questions	Historically important, but now heavily exposed and label-quality issues have been documented
MMLU-Pro	Less-saturated successor with more reasoning pressure	14 broad domains derived from and extending MMLU coverage	Mostly 10-choice multiple choice; average 9.47 options after review	12,032 questions	Filters easy items, adds refreshed items, augments distractors, and performs expert review
MMLU-Pro+	Shortcut-learning and multi-answer reasoning stress test	Built on MMLU-Pro	Multiple correct answers plus crafted distractor categories	Derived from MMLU-Pro	Tests anchoring, shortcut selection, and ability to identify correct answer pairs
MMLU-CF	Contamination-free MMLU-style evaluation	Broad MMLU-like disciplinary coverage	Multiple choice	10k validation + 10k closed test	Uses decontamination rules, public validation, and closed-source test split
MMLU-Redux	Label-quality repair for MMLU	Original MMLU subject space	4-choice multiple choice with reannotations	5,700 manually reannotated questions in Redux 2.0	Audits and corrects wrong labels, ambiguous questions, and invalid options

The core measurement change is simple: MMLU-Pro makes the same general family of task harder to solve by shallow recognition. A four-option question often lets a model succeed by eliminating two implausible distractors; a ten-option question with harder distractors forces finer discrimination. The MMLU-Pro paper reports that 83% of retained questions have ten options, and the remaining 17% have fewer because expert review removed problematic choices; the average retained option count is 9.47. arXiv

That change also alters the random baseline. MMLU’s nominal random-choice baseline is 25%; MMLU-Pro’s practical random baseline is near 10%, though not exactly 10% because not every question retains ten choices after review. This matters because multiple-choice benchmarks partly measure answer-space search: a model that is only weakly calibrated can look better when the answer space is small, especially if distractors are stylistically or semantically weak. arXiv

Construction methodology

1. Subject consolidation and item filtering

MMLU-Pro begins with MMLU’s subject matter but changes the reporting and item-selection geometry. Wang et al. merge the original 57 MMLU subjects into 14 broader categories, then run an initial filtering pass over the original MMLU items using eight models. Questions answered correctly by more than four of those models were treated as too easy and removed. Across the source set used in this filtering pass, 5,886 of 13,937 questions were filtered out, or 42.23%. arXiv

This is the first major methodological contribution: MMLU-Pro explicitly removes low-information items. In benchmark terms, an item that nearly all current models answer correctly contributes little to ranking frontier systems, even if it remains useful for testing weaker models. Filtering such items restores some discriminative capacity without needing to invent an entirely new task family.

2. Item refresh from external sources

MMLU-Pro is not merely “MMLU with more answer choices.” After filtering, the authors add new questions from the STEM Website, TheoremQA, and SciBench, while retaining a large subset of filtered-and-verified MMLU questions. The reported final source composition is 6,810 MMLU-derived questions, 4,083 STEM Website questions, 598 TheoremQA questions, and 541 SciBench questions, totaling 12,032. arXiv

The refresh matters for two reasons. First, it replaces stale or overexposed items with questions less likely to have become implicit training targets. Second, it shifts the benchmark toward reasoning-heavy material, especially in math, science, and theorem-style questions. This does not make MMLU-Pro contamination-proof—its dataset is public—but it does make the benchmark less dependent on the exact aging MMLU item pool.

3. Harder distractors and ten-option expansion

The most visible change is option augmentation. MMLU-Pro uses GPT-4-Turbo to expand four-option questions toward ten options by generating additional plausible distractors; these are then subjected to review. For questions imported from sources without the same multiple-choice structure, GPT-4-Turbo was also used to extract or construct candidate multiple-choice forms. arXiv

The design goal is not merely a lower chance baseline. Ten options with weak distractors would be cosmetic. The intended measurement pressure comes from “hard distractors”: options close enough to the correct answer that a model must reason through the underlying domain content rather than rely on answer-style artifacts. This is particularly relevant for modern instruction-tuned models, which often learn the surface texture of benchmark answers.

4. Expert verification and false-negative control

Wang et al. use a two-stage expert-review process. In the first phase, experts verify question correctness, remove questions unsuitable for multiple-choice evaluation, remove non-textual or under-specified items, and check whether the answer key is accurate. In the second phase, Gemini-1.5-Pro is used to flag possible false negatives—cases where an allegedly incorrect option may also be valid—and human experts then review those flagged items. arXiv

The error counts reported by the authors illustrate why this matters. In the reviewed source pools, they found hundreds of incorrect answers and many more false-negative options, especially in MMLU-derived and newly generated-option settings. The table in the paper reports, for example, 350 incorrect-answer issues and 1,953 false-negative-option issues in the MMLU-derived pool, as well as 483 incorrect-answer issues and 293 false-negative-option issues in the STEM Website-derived pool. arXiv

This is a crucial point for practitioners: harder benchmarks often become worse benchmarks if they merely add adversarial or ambiguous questions. MMLU-Pro’s construction methodology tries to separate “hard because it requires reasoning” from “hard because the label or wording is broken.” The separation is imperfect, but the explicit expert-verification layer is part of why MMLU-Pro is more than a difficulty hack.

What MMLU-Pro measures better than MMLU

MMLU-Pro rebuilds MMLU’s discriminative power along three axes: answer-space pressure, reasoning depth, and prompt robustness.

First, expanding from four options to mostly ten options reduces the value of shallow elimination. In a four-choice setting, a model may perform well by recognizing that two choices are implausible and then picking between the remaining two using memorized associations. In a ten-choice setting with harder distractors, the model has to maintain more distinctions at once. That does not transform multiple choice into open-ended reasoning, but it raises the floor for superficial success. arXiv

Second, MMLU-Pro makes chain-of-thought prompting more consequential. Wang et al. report that GPT-4o improved from 53.5% direct-answer accuracy to 72.6% with chain-of-thought on MMLU-Pro, a 19.1-point gain; on MMLU, the same comparison was 87.2% to 88.7%, only a 1.5-point gain. GPT-4-Turbo showed a similar pattern: roughly no gain on MMLU, but a 15.3-point gain on MMLU-Pro. arXiv

Third, MMLU-Pro appears less prompt-fragile than MMLU under the paper’s evaluation. Wang et al. evaluated 24 prompt variants and found that MMLU scores could vary by roughly 4–5% and peak near 10.98% prompt-induced variation, while MMLU-Pro generally stayed around 2% variation with a reported maximum of 3.74%. arXiv

The important methodological claim is therefore not “MMLU-Pro is harder.” Hardness alone is cheap. The better claim is: MMLU-Pro restores ranking resolution among models that MMLU had begun to compress, and it does so while making reasoning-oriented prompting more useful and prompt-template artifacts less dominant.

Empirical trajectory across frontier models

The 2024 baseline: restored separation

In the MMLU-Pro paper, GPT-4o scored 72.6%, Gemini-1.5-Pro scored 69.0%, Claude-3-Opus scored 68.5%, GPT-4-Turbo scored 63.7%, Gemini-1.5-Flash scored 59.1%, Claude-3-Sonnet scored 56.8%, and Llama-3-70B-Instruct scored 56.2%. The same paper emphasizes that MMLU-Pro widened model gaps that were narrow on MMLU; for example, GPT-4o and GPT-4-Turbo were separated by roughly 9 points on MMLU-Pro but about 1 point on MMLU. arXiv

The GitHub leaderboard associated with the benchmark later listed Claude-3.5-Sonnet at 76.12%, GPT-4o at 72.55%, Gemini-1.5-Pro at 69.03%, Claude-3-Opus at 68.45%, and GPT-4-Turbo at 63.71%. These numbers made the initial practical story clear: MMLU-Pro did not merely lower every model’s score uniformly; it separated models that had looked nearly tied on MMLU. GitHub

The 2026 picture: approaching saturation again

By May 2026, public MMLU-Pro leaderboards show a different picture. One leaderboard drawing from Artificial Analysis data lists top MMLU-Pro scores around 89–90%, with Gemini 3 Pro Preview at 89.8%, Claude Opus 4.5 Thinking at 89.5%, and other frontier systems close behind; the same page reports 218 tracked models, an average score of 74.9%, and a standard deviation of 11.5. Price Per Token

A separate Vals.ai leaderboard, updated May 4, 2026, reports an even higher top score: Gemini 3.1 Pro Preview at 90.99%, followed by Gemini 3 Pro at 90.10% and Claude Opus 4.7 at 89.87%. Vals explicitly characterizes MMLU-Pro as “largely saturated” in its benchmark description, while also stating that reported results use 5-shot chain-of-thought, matching the original MMLU-Pro evaluation style. vals.ai

These leaderboards should be treated as trajectory evidence, not as a single controlled experiment. Model snapshots change, providers expose different inference modes, “thinking” models may use different reasoning budgets, and public leaderboards often mix internal runs, submitted results, and third-party harnesses. The right conclusion is not that every frontier model is now reliably at 90% under one fixed protocol; it is that MMLU-Pro’s original headroom has already narrowed substantially at the top.

The methodology contribution: rebuilding discriminative power

MMLU-Pro’s main contribution is methodological rather than taxonomic. It shows how to extend a saturated benchmark while preserving continuity with the older one:

Saturation problem in MMLU	MMLU-Pro intervention	Measurement effect
Many items too easy for frontier models	Filter items solved by most of eight reference models	Removes low-information questions
Four answer choices enable shallow elimination	Expand toward ten options	Reduces random baseline and cheap elimination
Weak distractors reveal the answer	Generate harder distractors, then review	Forces finer semantic discrimination
Aging public item pool	Add refreshed questions from external sources	Reduces dependence on stale MMLU items
Label errors and false negatives	Expert review plus model-assisted false-negative detection	Improves validity of hard questions
Prompt-template sensitivity	Evaluate multiple prompt variants	Tests whether score differences survive prompt changes

This is a useful pattern for benchmark maintenance. Static benchmarks do not fail only because models improve; they also fail because their item distribution becomes culturally and technically overexposed. MMLU-Pro responds by refreshing the item pool, increasing item difficulty, and auditing labels. That combination is more durable than simply adding “hard” questions.

The paper’s error analysis also matters. Wang et al. report that among GPT-4o’s MMLU-Pro errors, 39% involved reasoning errors, 35% involved domain-expertise gaps, and 12% involved computational errors. That distribution supports the claim that MMLU-Pro is measuring more than memorized factual recall, though it remains a multiple-choice benchmark and therefore cannot fully test open-ended synthesis, research planning, or creative problem-solving. arXiv

Variants and related refreshes

MMLU-Pro+

MMLU-Pro+ is a direct descendant of MMLU-Pro aimed at shortcut learning and higher-order reasoning. Rather than assuming exactly one correct answer, MMLU-Pro+ introduces questions with multiple correct answers and crafted distractor categories, then evaluates whether a model can distinguish fully correct, partially misleading, and completely false answer pairs. arXiv

The MMLU-Pro+ paper describes three answer-pair categories: True Positive Pairs, Partial False Positive Pairs, and Complete False Pairs. It reports modifying 3,718 questions to include true positive pairs, alongside 2,029 complete false-positive and 2,124 partial false-positive constructions. The stated goal is to reduce shortcut learning by forcing the model to recognize when more than one option is valid and when a plausible option contains a subtle error. ar5iv

MMLU-Pro+ also introduces metrics beyond ordinary accuracy. Shortcut Selection Ratio is intended to capture anchoring on the original or most obvious correct answer, while Correct Pair Identification Ratio measures the ability to identify valid pairs. In its evaluation, the paper reports average scores such as 68.3 for O1-preview, 67.9 for Sonnet-3.5, 62.6 for Llama-405B, and 60.6 for GPT-4o, with model-specific differences in shortcut behavior. ar5iv

The practical interpretation: MMLU-Pro+ is not simply “MMLU-Pro but harder.” It asks a different behavioral question: can the model avoid single-answer anchoring when a domain problem admits multiple valid answers or subtly invalid alternatives? That makes it especially relevant for Shortcut Learning in LLMs and Multiple-Answer Evaluation, but less directly comparable to standard MMLU or MMLU-Pro accuracy.

MMLU-CF

MMLU-CF addresses a different failure mode: contamination. The ACL 2025 paper presents it as a contamination-free and challenging multiple-choice benchmark, designed to prevent both unintentional and malicious benchmark leakage. It sources questions from more than 200 billion webpages, applies three decontamination rules, provides a public validation set, and keeps the test set closed-source. ACL Anthology

The Hugging Face card describes MMLU-CF as having 10,000 validation and 10,000 test questions, with a pipeline including MCQ collection, cleaning, difficulty sampling, LLM checking by GPT-4o, Gemini, and Claude, and contamination-free processing. It explicitly frames MMLU-CF as a response to the fact that MMLU’s breadth and reasoning coverage do not, by themselves, prevent contamination. Hugging Face

This makes MMLU-CF important but operationally different. A closed test set improves contamination resistance, but reduces fully open reproducibility. It is most appropriate when the evaluation claim is about leakage-resistant generalization rather than easy-to-reproduce open benchmarking.

MMLU-Redux

MMLU-Redux is not a successor in the same sense as MMLU-Pro. It is a label-quality repair effort for MMLU. The NAACL 2025 paper “Are We Done with MMLU?” reports numerous ground-truth errors, including the striking finding that 57% of analyzed virology questions contained errors, and introduces MMLU-Redux as a manually reannotated subset across MMLU subjects. ACL Anthology

The MMLU-Redux 2.0 dataset card lists 5,700 rows and preserves fields such as the original answer, corrected answer, error type, and annotator rationale. Its taxonomy includes bad question clarity, bad options clarity, no correct answer, multiple correct answers, and wrong ground truth. Hugging Face

MMLU-Redux is valuable because benchmark saturation and benchmark invalidity are different problems. MMLU-Pro addresses saturation by increasing difficulty and refreshing the item pool; MMLU-Redux addresses reliability by auditing whether original MMLU labels and options were valid. A serious evaluation program should care about both.

Which MMLU variant should practitioners report?

There is no single best MMLU-family number anymore. The right benchmark depends on the claim being made.

Evaluation claim	Best primary report	Useful secondary report	Main caveat
“This model is comparable to older papers.”	Original MMLU	MMLU-Redux where available	Historical comparability comes at the cost of saturation and possible label errors
“This model improves broad academic/professional reasoning.”	MMLU-Pro	Original MMLU for continuity	Must specify prompt, shots, chain-of-thought policy, split, and scoring harness
“This model is robust against benchmark leakage.”	MMLU-CF	MMLU-Pro	Closed test improves contamination resistance but complicates independent reproduction
“This model avoids shortcut learning and single-answer anchoring.”	MMLU-Pro+	MMLU-Pro	Multi-answer setup is not directly comparable to standard accuracy
“This model’s MMLU score is label-reliable.”	MMLU-Redux	Original MMLU	Redux measures corrected subsets, not the full historical benchmark

For frontier-model reporting, the most defensible default is a small panel rather than one score: original MMLU for historical continuity, MMLU-Pro for current broad reasoning pressure, and either MMLU-CF or another hidden/contamination-controlled benchmark for leakage-sensitive claims. If the paper is specifically about shortcut learning, add MMLU-Pro+; if the paper is about benchmark validity or label noise, add MMLU-Redux.

The reporting standard should include: benchmark name and version, number of shots, whether chain-of-thought was used or hidden, prompt template, answer-extraction method, model snapshot, decoding parameters, option ordering, evaluation harness, date of evaluation, and whether any decontamination was performed. Without those details, MMLU-family numbers are often not comparable across papers.

The cross-paper comparability problem

MMLU-Pro improves the benchmark, but it does not eliminate evaluation-harness variance. The original MMLU-Pro paper uses a 5-shot chain-of-thought prompt and an answer-extraction procedure, with fallback behavior when an answer cannot be extracted. The Hugging Face card notes that leaderboard results are normally run under 5-shot settings, but that some Gemini submissions were run 0-shot. arXiv

That detail alone can invalidate naive comparisons. If one paper reports 5-shot chain-of-thought MMLU-Pro and another reports 0-shot direct-answer MMLU-Pro, the numbers are measuring different protocols. The gap is not trivial: Wang et al. report large CoT gains on MMLU-Pro for several frontier models. arXiv

The problem is amplified by reasoning models. A “thinking” mode with a large hidden reasoning budget may be closer to test-time search than ordinary next-token answering. If a leaderboard mixes ordinary chat models, reasoning models, and models with provider-specific inference scaffolding, the benchmark still reports a single score, but the intervention being measured is no longer just pretraining or post-training quality.

The practical rule: never cite “MMLU-Pro = X%” without the protocol. A rigorous paper should say something like: “MMLU-Pro, 5-shot CoT, official prompt, official evaluation script, model snapshot dated YYYY-MM-DD, temperature 0, evaluated on full test split, no tool use, no retrieval.” Without that, cross-paper comparability is weak.

Will MMLU-Pro follow MMLU’s saturation arc?

The evidence points in both directions, but the stronger bet is that public static MMLU-Pro will saturate for frontier systems.

The pro-saturation evidence is already visible. In 2024, GPT-4o’s MMLU-Pro score around 72.6% left meaningful headroom. By May 2026, public leaderboards report top scores near 90–91%, and at least one leaderboard describes the benchmark as largely saturated. arXiv+2Price Per Token+2

The anti-saturation evidence is that MMLU-Pro still separates many non-frontier systems. The Artificial Analysis-derived leaderboard reports 218 tracked models with an average of 74.9 and standard deviation of 11.5, which means the benchmark still has broad practical ranking value outside the very top cluster. Price Per Token

The deeper issue is that saturation is not binary. A benchmark can be saturated for frontier closed models, useful for open-model regression testing, informative for domain-level error analysis, and inadequate for contamination-resistant claims—all at the same time. That is likely where MMLU-Pro is heading.

The long-term role of MMLU-Pro should therefore be as part of an evaluation stack, not as the successor-to-end-all-successors. Its strongest use is to maintain continuity with MMLU while restoring some reasoning pressure. Its weakest use is as a sole headline number for frontier intelligence.

Takeaways

MMLU-Pro is an important benchmark because it demonstrates a disciplined way to repair a saturated benchmark while preserving enough continuity for historical comparison. It filters easy items, refreshes the item pool, increases the answer-space difficulty, adds harder distractors, and performs expert verification. That combination makes it more valuable than a benchmark that merely adds adversarial questions.

Its empirical history also contains a warning. MMLU-Pro restored separation in 2024, but by 2026 top public scores were already approaching the low 90s. The benchmark may remain useful for broad model comparison, domain analysis, and regression testing, but it is unlikely to remain a definitive frontier-capability measure unless paired with contamination controls, hidden test sets, dynamic item generation, multi-answer reasoning tests, and open-ended evaluations.

The best practitioner stance is version pluralism with explicit protocols: report MMLU for historical continuity, MMLU-Pro for less-saturated broad reasoning, MMLU-CF for contamination-sensitive claims, MMLU-Pro+ for shortcut and multi-answer behavior, and MMLU-Redux when label reliability is central.