AIME: American Invitational Mathematics Examination as AI Benchmark

What AIME is

The American Invitational Mathematics Examination is the second stage in the Mathematical Association of America’s American Mathematics Competitions pipeline, after the AMC 10/12 and before the USAMO/USAJMO. The official MAA description defines AIME as a 15-question, 3-hour examination for students who excelled on the AMC 10 or AMC 12; each answer is an integer from 0 to 999, and AIME problems are “much more difficult” than AMC 10/12 problems. Top-scoring AIME participants advance toward the USAMO or USAJMO selection path. Mathematical Association of America

For human competitors, AIME is not an olympiad proof exam. It is a short-answer contest. There is no partial credit, no essay proof to grade, and no multiple-choice answer set to exploit. This makes it unusually attractive as an AI benchmark: it has the flavor of olympiad-style high-school mathematics, but the grading interface is closer to a deterministic programmatic check. AoPS describes the same format—15 questions, 3 hours, answers from 000 to 999—and notes that calculators are not permitted; it also summarizes the curricular range as arithmetic, algebra, counting, geometry, number theory, probability, and other secondary-school topics. Art of Problem Solving

The public archive is unusually rich. AoPS maintains an AIME Problems and Solutions page listing AIME exams from 1983 through recent years, with separate AIME I and AIME II pages for years after the two-version structure became standard. That archive is valuable for reproducibility, but it is also the main reason older AIME years are unsafe as uncontaminated AI benchmarks. Art of Problem Solving

In AI evaluation, “AIME” usually means one of several related but non-identical test sets:

Benchmark name in reports	Typical contents	Consequence
AIME 2024	Often AIME I + AIME II, 30 problems total; sometimes only one 15-problem exam	Good historical comparison point, but now contamination-prone.
AIME 2025	Often AIME I + AIME II, 30 problems total	Cleaner for models with pre-February-2025 training cutoffs, but no longer automatically clean after public release.
AIME single exam	15 problems	Very high variance: each problem is 6.67 percentage points.
AIME I + II	30 problems	Better, but still small: each problem is 3.33 percentage points.
AIME with tools	Same problems, but model may use Python/CAS/search-like tools	Not comparable to closed-book no-tool AIME unless separately labeled.
AIME with high test-time compute	Multiple samples, self-consistency, reranking, or long “thinking” budgets	Measures model-plus-inference-procedure, not only base model capability.

The benchmark should therefore be treated as a family of related evaluations, not a single stable number.

Why AIME became a reasoning-model benchmark

AIME became salient in AI because it sits at a sweet spot between Benchmark Saturation and impossible grading. GSM8K and much of the MATH dataset became too easy for frontier models; OpenAI’s o1 announcement explicitly framed AIME as a harder differentiator after noting that recent frontier models were already performing so well on MATH and GSM8K that those benchmarks no longer separated them effectively. In that same report, OpenAI used the 2024 AIME to show a large gap between GPT-4o and o1-style reasoning. OpenAI

The attraction is methodological. AIME has:

Property	Why AI evaluators like it	Hidden caveat
Integer answers from 0–999	Exact ground truth; easy answer extraction; no subjective grading.	A correct final integer can hide invalid reasoning, lucky enumeration, or tool-driven brute force.
Expert-authored contest problems	Problems are written and vetted for mathematically talented students.	Contest style is narrow; it does not measure proof-writing or research mathematics.
Annual fresh exams	New years can be evaluated soon after release, reducing pretraining contamination.	Once published online, problems enter the training and tuning ecosystem.
High but not research-level difficulty	Hard enough to separate non-reasoning and reasoning models.	Frontier systems are now near ceiling under generous compute.
Public archives	Transparent and reproducible.	Publicness creates contamination risk for old years.

AIME is especially convenient for evaluating Test-Time Compute. A model can be asked to solve each problem once; it can be allowed to sample many solutions and take a majority vote; it can use an external verifier; it can call Python; or it can perform search over possible answers. Those variants are not equivalent, but AIME’s exact-answer format makes all of them easy to score.

Empirical trajectory: from GPT-4-class baselines to reasoning models

A careful score history needs one caveat: there is no single canonical primary-source “GPT-4 on AIME” number from the original GPT-4 technical report comparable to later o1/o3/Gemini/Claude AIME tables. The clean primary-source anchor for the pre-reasoning era is GPT-4o, which OpenAI reported at about 12% on 2024 AIME in the o1 launch article, and later tabulated at 13.1% on AIME ’24; OpenAI’s same appendix gives GPT-4o mini at 8.6%, which is close to the commonly quoted “single-digit GPT-4-class” baseline. OpenAI

The headline trajectory is still clear: single-digit or low-teens accuracy for non-reasoning GPT-4-class systems, then 70%+ for first-generation reasoning models, then high-80s to 90%+ for later reasoning models, and near-perfect results when heavy test-time compute or tools are allowed.

Period / source	Model or system	AIME variant	Reported performance	Evaluation interpretation
Pre-reasoning GPT-4-class baseline	GPT-4o	2024 AIME	OpenAI reported 1.8/15, about 12%; later appendix lists GPT-4o at 13.1% on AIME ’24	Low-teens closed-book baseline; use this rather than an unsourced canonical “GPT-4” number. OpenAI
Pre-reasoning small/fast baseline	GPT-4o mini	AIME ’24	8.6%	The closest official OpenAI table entry to the “~8%” shorthand. OpenAI
First o1 release	o1	2024 AIME	74% single sample; 83% with consensus over 64 samples; 93% with reranking over 1,000 samples	Demonstrates the importance of inference-time sampling and selection. OpenAI
o3 preview, high compute	o3	AIME 2024	Reported by InfoQ as 96.7%, up from 83.3% for o1; The Verge described it as missing one question	Strong but not fully primary-source-clean in accessible documentation; useful as historical preview, not a standardized public eval. InfoQ
OpenAI o3-mini high	o3-mini high	AIME ’24	87.3% in OpenAI’s GPT-4.1 appendix	Later OpenAI table, more reproducible than preview rhetoric. OpenAI
Gemini 2.5 family	Gemini 2.5 Flash / Gemini 2.5 Pro	AIME 2025	72.0% / 88.0% single attempt	Google’s report emphasizes pass@1, no majority voting or parallel test-time compute unless specified. Google Cloud Storage
Cross-lab 2025 table	Gemini 2.5 Pro, o3 high, o4-mini high, Claude 4 Sonnet/Opus	AIME 2025	Gemini 2.5 Pro 88.0%; o3 high 88.9%; o4-mini high 92.7%; Claude 4 Sonnet 70.5%; Claude 4 Opus 75.5%	Useful same-table comparison, but still vendor-reported and dependent on prompting/effort settings. Google Cloud Storage
Tool-enabled OpenAI result	o4-mini with Python interpreter	AIME 2025	99.5% pass@1 and 100% consensus@8	Measures model + Python tool use; should not be mixed with no-tool pass@1. OpenAI
Anthropic later system card	Claude Sonnet 4.6	AIME 2025	95.6% without tools, averaged over 10 trials at adaptive thinking / max effort	Anthropic explicitly cautioned that contamination may have inflated the result. Anthropic

Two points matter more than any single number.

First, AIME scores are compute-sensitive. OpenAI’s o1 result moved from 74% to 83% to 93% as the evaluation moved from one sample, to consensus over 64 samples, to reranking 1,000 samples. That is not a minor prompting detail; it changes the benchmark’s meaning from “can the model solve this once?” to “does the model’s sample distribution contain a correct solution discoverable by search or selection?” OpenAI

Second, AIME scores are now contamination-sensitive. A model released after AIME 2024 has been public for months or years may have seen the problems directly, seen solutions, been trained on AoPS-like material, or been tuned against benchmark leaderboards. MathArena’s 2025 analysis says AIME 2024 is significantly contaminated by most leading LLMs and therefore unsuitable for evaluating true capability in those cases. arXiv

What the benchmark actually measures

AIME measures short-form mathematical problem solving under an answer-only interface. It is useful evidence for Mathematical Reasoning Benchmarks, but it is not a complete proxy for mathematical intelligence.

A model that performs well on AIME likely has some mix of the following capabilities:

Capability	AIME evidence
Algebraic manipulation	Many problems require nontrivial symbolic transformation.
Counting and probability	Combinatorics problems often require casework and inclusion-exclusion.
Synthetic geometry	Diagram-free geometry problems test spatial and theorem recall.
Number theory	Modular arithmetic, divisibility, and Diophantine constraints appear frequently.
Search discipline	Later problems often punish local greedy reasoning.
Error correction	Long reasoning traces must catch arithmetic and conceptual slips.
Answer extraction	The model must output a final integer, often with leading-zero awareness.

But AIME does not directly measure:

Not measured well	Why
Proof-writing rigor	The official output is a number, not a proof.
Research mathematics	AIME is secondary-school competition math, not graduate or frontier research.
Long-horizon theorem development	Problems are designed for a 3-hour contest, not weeks of abstraction.
Tool-use engineering	No-tool AIME says little about integrating CAS, Lean, Python, or search tools in real workflows.
Robustness outside contest style	AIME’s style is distinctive and trainable.

This distinction matters because frontier models may soon solve AIME while still failing at proof generation, formal verification, or mathematical research. MathArena makes the same criticism of final-answer benchmarks: single-answer tasks can miss proof-writing capability and may allow pattern recognition or brute-force enumeration to masquerade as reasoning. arXiv

Methodology strengths

1. Clean ground truth

AIME’s strongest benchmark property is its answer format. The MAA specifies that each answer is an integer between 0 and 999; AoPS similarly describes answers from 000 to 999. This gives evaluators a deterministic equality check rather than a subjective rubric. Mathematical Association of America

For AI systems, that means:

no LLM judge is required;
no proof grader is required;
false positives from persuasive prose are reduced;
evaluation code can simply parse a boxed integer or final numeric answer.

This is why AIME is easier to operationalize than proof contests such as USAMO or IMO. MAA’s own competition sequence contrasts AIME with USAMO/USAJMO, which are six-question, two-day, 9-hour essay/proof examinations. Mathematical Association of America

2. Expert authorship and time-tested difficulty

AIME is not an ad hoc benchmark generated after the fact for AI. It is a long-standing human contest with a stable role in the U.S. mathematical competition ladder. That gives it a kind of external validity many synthetic benchmarks lack: the problems are written to challenge mathematically strong students, not to expose quirks in one model family.

This also makes AIME psychologically legible. A result like “12/15 on AIME” means something to people familiar with contest math. It is not merely a benchmark percentage.

3. Low ambiguity in scoring

Many AI benchmarks fail because answer grading is messy. MATH-style problems often have algebraically equivalent expressions, alternate forms, radicals, intervals, or sets of values. OpenAI’s PRM800K repository, built around MATH problems, explicitly notes that answer grading is difficult and that even conservative grading logic can reject correct answers or admit incorrect ones. GitHub

AIME avoids much of this. The output space is small and canonical: an integer in [0, 999]. That does not remove all extraction errors—models can put the answer in prose, produce multiple candidate answers, or omit leading-zero formatting—but it dramatically simplifies the evaluation.

4. Fresh annual tests

Unlike static datasets, AIME creates new problems every year. If a model’s training cutoff predates the contest and the model is evaluated quickly after problem release, AIME can serve as a transparent live benchmark. MathArena’s core proposal is built around this idea: recurring math competitions provide high-quality new problems, and evaluating as soon as they are released can effectively eliminate contamination risk. arXiv

This is the strongest argument for AIME’s continued use: not old AIME as a static benchmark, but fresh AIME as a live eval.

Methodology weaknesses

1. Small N

AIME is statistically small. One exam has 15 problems; AIME I + II gives 30. On a 15-problem exam, one problem changes the score by 6.67 percentage points. On a 30-problem combined set, one problem changes the score by 3.33 percentage points.

A rough binomial standard error illustrates the problem. If a model scores 90% on 30 problems, the standard error is approximately:

0.9(1−0.9)30≈0.055\sqrt{\frac{0.9(1-0.9)}{30}} \approx 0.055300.9(1−0.9)≈0.055 That is about 5.5 percentage points, with a rough 95% interval near ±11 percentage points before considering correlated errors, prompt variance, or item heterogeneity. This does not mean AIME is useless; it means fine-grained leaderboard ordering is often fake precision.

2. Narrow style

AIME problems are contest problems. They reward compact tricks, clever transformations, and exact computation. That is a real kind of mathematical reasoning, but it is not the whole domain.

A model can become very good at AIME by learning:

common contest problem schemas;
modular arithmetic templates;
geometry theorem patterns;
AoPS-style solution moves;
answer-space search habits;
self-consistency over integer outputs.

That may transfer to other mathematical tasks, but the transfer is empirical, not guaranteed. FrontierMath was created precisely because high-school and early-undergraduate benchmarks were saturating and did not test deeper theoretical understanding, creative insight, or specialized expertise. arXiv

3. Item-quality and year-quality variance

AIME is curated, but not every year has the same difficulty distribution. One year’s problem 15 may be much harder than another’s; one version may contain a geometry problem especially hostile to text-only models; another may contain a problem especially easy to brute force with Python.

This matters when the benchmark is only 15 or 30 items. A model that is 3 problems better on one year may not be 10 percentage points “more intelligent”; it may simply match that year’s item mix better.

4. Contamination

AIME’s public archive is both a strength and a weakness. It gives reproducibility, but public problems can be absorbed into pretraining corpora, supervised fine-tuning datasets, reinforcement learning environments, prompt libraries, and leaderboard-driven tuning.

MathArena identifies contamination as a central issue for AIME 2024, saying that many commonly used math benchmarks are publicly available online and that AIME 2024 is significantly contaminated by most leading LLMs. It also distinguishes direct pretraining contamination from benchmark use in hyperparameter tuning or model selection, both of which can inflate scores. arXiv

Anthropic’s Claude Sonnet 4.6 system card is a useful example of responsible reporting: Anthropic reported 95.6% on AIME 2025 without tools, but explicitly warned that contamination may have inflated the score. Anthropic

5. Compute confounding

AIME is highly sensitive to sampling budget. The same base model may perform very differently under:

Setting	Meaning
pass@1	One attempt per problem; closest to “single-shot” ability.
avg@k	Average over several independent attempts.
majority / consensus@k	Sample k answers and choose the most common or most internally supported answer.
best-of-n with verifier	Generate many candidates and use a learned or symbolic verifier.
tool-integrated reasoning	Allow Python, CAS, or other tools.
high “thinking” budget	Allow long hidden or visible reasoning traces.

OpenAI’s o1 AIME result is the canonical example: 74% single-sample, 83% consensus over 64, and 93% with reranking over 1,000 samples. These are all valid measurements, but they answer different questions. OpenAI

Relationship to MATH-500 and FrontierMath

AIME, MATH-500, and FrontierMath occupy different difficulty bands and serve different evaluation purposes.

Benchmark	Source / construction	Typical difficulty band	Grading	Best use	Main failure mode
MATH / MATH-500	MATH is a 12,500-problem competition-math dataset with step-by-step solutions; MATH-500 is the 500-problem held-out subset popularized by OpenAI’s “Let’s Verify Step by Step” / PRM800K split. arXiv+2Hugging Face+2	High-school competition through early undergraduate contest-style math	Symbolic / expression answer grading	Broad math-reasoning regression test; process-supervision and verifier research	Saturation and messy answer equivalence.
AIME	MAA contest; 15-question, 3-hour, integer-answer invitational exam. Mathematical Association of America	Hard high-school contest math	Exact integer match	Transparent live benchmark for reasoning models; strong pass@1 sanity check	Small N, narrow style, contamination after publication.
FrontierMath	Hundreds of original unpublished advanced math problems authored and peer-reviewed by expert mathematicians; includes tiers through research-level math. Epoch AI	Undergraduate to advanced graduate and research-level mathematics	Automated verification, often definitive computable answers	Measuring frontier mathematical capability beyond AIME/MATH saturation	Private-access and reproducibility tradeoffs; extreme difficulty.

OpenAI’s o1 article explicitly says MATH and GSM8K no longer differentiated frontier models well, which explains the move toward AIME. FrontierMath’s paper makes the next step explicit: once high-school and early-undergraduate benchmarks saturate, evaluation must move toward advanced mathematical domains requiring deeper theoretical understanding and specialized expertise. OpenAI

The difficulty relationship is therefore not “which benchmark is better?” but “which failure regime do you want to study?”

Research question	Better benchmark
Can a model solve broad competition-style math problems with standard answer grading?	MATH-500
Can a model solve hard, fresh, answer-only high-school contest problems?	AIME
Can a model approach expert mathematical research difficulty?	FrontierMath
Can a model write rigorous proofs?	AIME is insufficient; use proof-based or formal-verification benchmarks.
Can a model exploit test-time compute?	AIME and MATH-500 both work; AIME’s integer answer makes this especially clean.
Can a model generalize beyond public training data?	Fresh live competitions or unpublished/private benchmarks.

Practitioner guidance: which AIME years to use

The core practitioner question is not “AIME 2024 or AIME 2025?” It is:

Was the problem set plausibly outside the model’s pretraining, post-training, benchmark tuning, and prompt-development loop?

A training cutoff alone is necessary but not sufficient. A model can have a pre-February-2025 knowledge cutoff and still be tuned after AIME 2025 became public, or selected because it performed well on AIME 2025. Conversely, a model with a later stated knowledge cutoff may have undergone decontamination, but that claim is hard to verify externally.

Decision table

Evaluation goal	Recommended AIME use	Avoid
Historical comparison with published o1/o3/Gemini/Claude tables	Use AIME 2024 and/or 2025, but label it historical and contamination-prone.	Treating old public scores as clean capability estimates.
Clean eval of a newly released model	Use the newest AIME year that occurred after the model’s final training and tuning cutoff; evaluate as soon as possible after release.	Using a year that was public before model post-training.
Model-selection inside an engineering org	Use a private copy of the newest available year or fresh contest problems; rotate quickly.	Reusing the same AIME set until teams overfit to it.
Frontier-model ranking	Use AIME only as one component; pair with FrontierMath, live competitions, proof tasks, and tool-use evals.	Ranking frontier systems by a 15- or 30-problem AIME score alone.
Open-source reproducibility	Use public AIME years, but separate “public reproducible” from “uncontaminated.”	Claiming public archival AIME is contamination-free.
Tool-integrated math agents	Use AIME with tools, but report tool policy, timeout, retries, and total samples.	Mixing tool and no-tool results in one leaderboard.

Practical reporting checklist

A defensible AIME report should include:

Field	Why it matters
Exact year and version	AIME I, AIME II, or both; 15 vs 30 problems changes variance.
Model version	Model names are unstable; date-stamped model IDs matter.
Training / knowledge cutoff	Needed for contamination analysis.
Release date and eval date	Public-problem exposure time matters.
Prompt	AIME is sensitive to instruction format and answer extraction.
Tool policy	Python/no Python is a different task.
Sampling policy	pass@1, consensus@k, best-of-n, reranking, or verifier use.
Thinking / effort setting	Modern reasoning models expose low/medium/high effort or hidden budgets.
Answer parser	Boxed answer extraction, final integer extraction, manual correction policy.
Per-problem results	Aggregate percentages hide systematic failures.
Confidence intervals or item counts	A 30-problem benchmark cannot justify tiny ranking differences.

Vals AI’s public AIME page illustrates why these details matter. It archived AIME because performance saturated; it also notes that AIME questions are publicly available, that older 2024 questions can show higher model performance than newer 2025 questions, and that the benchmark’s low size motivated running models multiple times to reduce variance. Vals AI

Contamination-resistant use is a moving window

AIME is not inherently contamination-resistant. Fresh AIME is contamination-resistant for a short window. Old AIME is a public dataset.

This distinction should be part of every article, benchmark card, and model report that cites AIME.

Problem set	Cleanliness status
AIME 1983–2023	Public for years; assume contaminated for frontier LLMs.
AIME 2024	Valuable historical o1/o3 comparison set, but MathArena reports strong contamination signs. arXiv
AIME 2025	Cleaner for models whose training and post-training ended before public release; no longer automatically clean for later models.
Newest available AIME year	Best public option if evaluated quickly and before it becomes a tuning target.
Future unreleased AIME	Not available; do not use leaked or embargoed contest materials.
Private contest-style problems	Cleaner but less reproducible; needs trusted governance.

The cleanest protocol is “live AIME”: evaluate models immediately after the contest and before the problems are widely incorporated into training or tuning pipelines. MathArena generalizes this approach across newly released math competitions, arguing that recurring contests can provide transparent, reproducible, continuously updated evaluation while reducing contamination. arXiv

AIME and the ceiling problem

AIME’s benchmark status is now under pressure. It is still useful, but no longer sufficient at the frontier.

The saturation literature is direct. MathArena states that GSM8K and MATH have largely saturated, and that even harder competitions such as AIME 2024 are close to saturation; it also confirms contamination concerns for AIME 2024. arXiv More general benchmark-saturation work argues that LLM/LRM benchmarks often saturate quickly and that it is frequently unclear whether high scores reflect genuine capability, contamination, or both. arXiv

The “edge of human comprehension” line of work frames the broader problem: math benchmarks have moved from grade-school arithmetic to MATH, then to AIME, then to FrontierMath, with each generation requiring more expertise to create and verify. As frontier models improve, humans may struggle to generate tasks that are simultaneously hard, correct, uncontaminated, and cheaply verifiable. arXiv

This creates a sharp open question:

Does AIME still measure frontier mathematical reasoning, or has it become a mid-tier regression test plus contamination detector?

The honest answer is conditional.

AIME remains useful when

Context	Why AIME still helps
Comparing non-frontier models	Many models still score far below ceiling.
Measuring cost-performance	Smaller or cheaper models separate clearly on AIME.
Testing reasoning-mode deltas	Base vs reasoning vs high-effort modes often differ sharply.
Running live evaluations	Fresh annual problems can still be informative.
Debugging answer extraction	Integer outputs expose final-answer discipline.
Public reproducibility	Anyone can inspect the problems and solutions.

AIME becomes weak when

Context	Why AIME fails
Ranking top frontier models	Scores cluster near 90–100%, and small N dominates.
Using old public years	Contamination risk overwhelms capability inference.
Mixing compute regimes	pass@1 and best-of-1000 are different tasks.
Claiming proof ability	AIME only requires final answers.
Inferring research math ability	FrontierMath-level tasks are qualitatively harder.
Reporting tiny deltas	A 1–3 percentage-point gap on 30 items is not meaningful.

The right future role for AIME is probably not “the frontier math benchmark.” It is a transparent, interpretable, medium-hard benchmark in a larger suite. For frontier work, it should be paired with:

fresh contest streams such as MathArena;
advanced private or semi-private tasks such as FrontierMath;
proof-generation benchmarks;
formal-verification tasks in Lean or related systems;
tool-integrated math-agent evaluations;
adversarial or critique-based evaluation for tasks beyond easy ground truth.

Interpretive rules for AIME scores

AIME scores are most informative when interpreted with discipline.

Rule 1: Always ask “which AIME?”

AIME 2024, AIME 2025, AIME I, AIME II, 15 problems, 30 problems, no tools, tools, high effort, consensus, and reranking are all different settings.

Rule 2: Treat old public years as contaminated unless proven otherwise

AoPS archives are public. Model developers know AIME matters. Public benchmark status changes training incentives.

Rule 3: Do not compare pass@1 with consensus or reranking

OpenAI’s o1 result shows why: 74%, 83%, and 93% can all be true for the same model family under different inference procedures. OpenAI

Rule 4: Report item counts, not only percentages

“27/30” is clearer than “90%.” “14/15” is clearer than “93.3%.” Percentages can imply statistical precision the benchmark does not have.

Rule 5: Separate model capability from system capability

A no-tool model solving AIME is different from a Python-enabled agent solving AIME. OpenAI’s o4-mini result with Python—99.5% pass@1 and 100% consensus@8—is impressive, but it is a tool-integrated result and should be labeled as such. OpenAI

Rule 6: Use AIME as evidence, not proof

AIME success is evidence of strong contest-math reasoning. It is not proof of general reasoning, proof-writing ability, or research-level mathematical competence.

Bottom line

AIME became a benchmark because it is almost perfectly shaped for early reasoning-model evaluation: difficult enough to expose weak reasoning, simple enough to grade exactly, and culturally trusted because it predates AI benchmarking by decades. The same properties that made it successful also make it vulnerable: it is small, public, stylistically narrow, and now close to saturated for frontier reasoning systems.

The benchmark’s best future use is as a transparent live or near-live contest-math probe. Its worst use is as a static leaderboard where old public AIME sets are treated as clean evidence that one frontier model is mathematically smarter than another by a few percentage points.

Companion entries

Core theory: Mathematical Reasoning Benchmarks, Reasoning Models, Benchmark Saturation, Benchmark Contamination, Test-Time Compute, Chain of Thought

Benchmark family: MATH-500, FrontierMath, GSM8K, OlympiadBench, MathArena, Live Benchmarks

Evaluation practice: Pass@1, Self-Consistency, Best-of-N Sampling, Verifier Models, Tool-Integrated Reasoning, Evaluation Variance

Mathematics context: American Mathematics Competitions, Olympiad Math, USAMO, Proof-Based Evaluation, Formal Verification in Lean

Counterarguments and caveats: Contamination-Resistant Evaluation, Leaderboard Overfitting, Ceiling Effects, Answer-Only Benchmarks, Private Benchmarks and Reproducibility