Test-Time Compute

Test-time compute has become a real scaling axis for frontier reasoning systems: extra computation spent after training—more hidden thought tokens, more sampled rollouts, more verifier passes, or more tool-using search—can materially raise performance without changing model weights. OpenAI says this directly for o1 and continues the claim for o3; Google productizes it as explicit thinking budgets; DeepSeek makes it unusually legible by exposing chain-of-thought in its API; and ARC-AGI provides the clearest public demonstration that more inference budget can buy more capability. The harder philosophical claim remains unresolved: these systems clearly get better at benchmarked problem solving with more inference compute, but the evidence that this constitutes robust, general reasoning rather than more expensive search over learned patterns is still incomplete. (OpenAI)

Coverage note: verified through April 5, 2026.

The source base here is mainly primary material: OpenAI’s o1 and o3/o4 disclosures and API docs, DeepSeek’s R1 paper/model card/API docs, Google’s Gemini and DeepMind documentation, ARC Prize analyses, and the academic literature on Inference Scaling Laws. Where the evidence is thin or contested, I say so directly. (OpenAI)

The core claim

Test-time compute is best understood as computation allocated per prompt after training is already done. In the recent literature, that includes longer generation of intermediate reasoning, repeated sampling, majority voting, best-of-N selection, verifier-guided ranking, tree search, and adaptive compute allocation per problem. The literature’s central result is not just that “thinking longer helps” in a vague sense, but that there are measurable performance-compute tradeoffs at inference time, with task-dependent optima and cases where smaller models with stronger inference strategies outperform larger models under matched compute budgets. (arXiv)

This makes test-time compute distinct from training-time scale. Training-time compute changes the model itself: parameters, data exposure, RL updates, learned heuristics. Test-time compute leaves weights fixed and changes how hard the system works on a particular query. That distinction matters because real workloads are heterogeneous. Some prompts are cheap and obvious; others are ambiguous, adversarial, or multi-step. A system that can spend more compute only on the hard cases is doing something economically and algorithmically different from a uniformly larger model. OpenAI’s o1 launch note states this explicitly, saying performance improved both with more reinforcement learning during training and with more time spent thinking at test time. (OpenAI)

The cleanest compact way to frame the distinction is:

Scaling axis	What increases	When it is paid	What it buys
Training-time compute	parameters, data, RL updates, optimization steps	before deployment	better priors, better heuristics, broader latent competence
Test-time compute	reasoning tokens, samples, verifier passes, tool calls, search depth	per query at deployment	more careful search, checking, branching, synthesis, and adaptation

That table is a synthesis of the vendor disclosures and the inference-time scaling literature rather than a quote from any single source. The key empirical support is OpenAI’s dual train-time/test-time curve for o1, plus academic work showing compute-optimal inference can outperform naive best-of-N and sometimes even much larger models under matched FLOPs. (OpenAI)

A useful refinement is that test-time compute is not a substitute for competence ex nihilo. Snell et al. report that extra test-time compute can beat a 14× larger model in FLOPs-matched comparisons only on problems where the smaller base model already has some non-trivial success rate, and Wu et al. find that smaller models paired with advanced inference algorithms often sit on the Pareto frontier of cost versus performance. That is the complementary picture: training creates a capable proposal distribution; test-time compute searches it more effectively. (arXiv)

Why this is not just “longer answers”

It is tempting to equate test-time compute with longer chain-of-thought and stop there. That is too narrow. The modern systems differ along at least four dimensions.

First, there is internal deliberation: extra hidden reasoning or “thinking” tokens before the final answer. OpenAI’s reasoning guide describes reasoning tokens as additional tokens used to break down a prompt and consider multiple approaches before generating visible output. Google’s Gemini docs make the same basic point, while calling them thinking tokens or thoughts. (OpenAI Developers)

Second, there is parallel exploration: sampling multiple candidate solutions and choosing among them. OpenAI’s o1 launch results explicitly distinguish pass@1 from consensus@64, and the self-consistency literature formalized this pattern years earlier by sampling diverse reasoning paths and marginalizing to the most consistent answer. ARC’s o3 preview results are also easiest to interpret as a large jump in parallel exploration budget: 6 samples for the efficient configuration versus 1024 samples for the high-compute configuration. (OpenAI)

Third, there is selection and verification: using a verifier or reward model to score candidates. The verifier literature on GSM8K showed that generating many candidates and then ranking them with a verifier can substantially improve performance. Snell et al. generalize this as one of the two major ways to scale test-time compute, alongside adaptively updating the response distribution at test time. (arXiv)

Fourth, there is tool-mediated search: letting the model browse, execute Python, inspect files, manipulate images, or otherwise perform actions during reasoning. OpenAI’s o3/o4-mini system card says these models use tools inside their chains of thought; Google’s Gemini docs say thinking models work with tools and can incorporate their results into reasoning; DeepMind’s AlphaProof and AlphaGeometry go further and couple learned components to explicit formal search. At that point, “test-time compute” is no longer just extra text generation. It is a compound system spending compute across model inference, external tools, and sometimes search over formal states. (OpenAI)

What the frontier systems actually show

OpenAI’s OpenAI o1 and OpenAI o3: the clearest direct evidence

OpenAI is the clearest public source for the thesis that inference-time scaling is real. In Learning to reason with LLMs, the company says that o1 performance “consistently improves” with both more reinforcement learning and more time spent thinking at test time. The same document also says that, unless otherwise noted, o1 was evaluated at the maximal test-time compute setting. That matters because it means some headline benchmark numbers already incorporate aggressive inference budgets rather than a neutral one-shot comparison. (OpenAI)

The appendix makes the point concrete. On AIME 2024, OpenAI reports GPT-4o at 9.3% pass@1 and 13.4% consensus@64, while o1 reports 74.4% pass@1 and 83.3% consensus@64. On GPQA Diamond, the same appendix reports o1 at 77.3% pass@1 and 78.0% consensus@64. The exact gains vary by task, but the basic pattern is stable: more inference compute via multiple sampled solutions can still buy accuracy even after the model is already strong. (OpenAI)

OpenAI’s system cards make the operational story similar. The December 2024 o1 system card describes the family as RL-trained for complex reasoning, producing a long chain of thought before responding, and learning to refine its thinking, try different strategies, and recognize mistakes. The April 2025 o3/o4-mini system card says the same about the broader o-series and adds that the models use tools in their chains of thought. That is an important escalation. The inference policy is no longer just “sample more tokens”; it becomes “sample more tokens while deciding when to browse, code, crop an image, or analyze a file.” (OpenAI)

OpenAI’s April 2025 launch note for o3 and o4-mini goes further still. It says the development of o3 pushed another order of magnitude in both training compute and inference-time reasoning and still showed clear gains; it also says that at equal latency and cost with o1, o3 performs better, and that if allowed to think longer its performance keeps climbing. That statement is notable because it combines three claims at once: train-time scaling still helps, test-time scaling still helps, and frontier engineering is trying to improve the cost-performance frontier rather than simply maximizing raw benchmark scores. (OpenAI)

This is why OpenAI’s evidence is stronger than a simple leaderboard delta. The company is not only releasing a stronger model; it is explicitly describing a family of systems whose capabilities depend on how much inference compute you let them spend. That is the core thesis of this article. (OpenAI)

DeepSeek-R1: RL-induced reasoning, cheaper visibility, and open operational semantics

DeepSeek-R1 matters for a different reason. The paper’s strongest claim is not an explicit scaling curve but a training result: reasoning behaviors can be incentivized through reinforcement learning, with emergent self-reflection, verification, and dynamic strategy adaptation. The pure-RL result is DeepSeek-R1-Zero; the public DeepSeek-R1 model card clarifies that the final R1 system adds cold-start data before RL to improve readability and stability. That distinction matters. The “pure RL gives reasoning-like behavior” claim belongs most directly to R1-Zero; the deployed R1 family is a more practical hybrid. (arXiv)

DeepSeek’s model card also reports results comparable to OpenAI-o1 on several math, code, and reasoning benchmarks, and notes that on benchmarks requiring sampling the evaluation generates 64 responses per query to estimate pass@1. In other words, even the open-model reporting culture around R1 quietly assumes that multiple-sample evaluation matters. That does not by itself prove test-time scaling laws, but it reinforces the broader point that frontier “reasoning” performance is often inseparable from the inference policy used to elicit it. (Hugging Face)

Where DeepSeek is uniquely important is operational transparency. Its reasoning API docs say deepseek-reasoner first generates chain-of-thought and then a final answer; the API exposes the chain-of-thought directly as reasoning_content; and max_tokens includes the CoT. In multi-turn usage, previous reasoning is not automatically concatenated into the next turn’s context, and the docs say including reasoning_content in the next request can trigger an error. This is the opposite of OpenAI’s decision to hide raw reasoning and expose only summaries. (DeepSeek API Docs)

That transparency made DeepSeek unusually valuable to researchers and practitioners trying to study or distill reasoning behavior. But it also introduces a temporal caveat: DeepSeek’s API naming shifted. The January 2025 R1 release says model=deepseek-reasoner accesses DeepSeek-R1, with output pricing at $2.19 per million output tokens. Current docs, however, say deepseek-reasoner corresponds to DeepSeek-V3.2 thinking mode, with output pricing at $0.42 per million output tokens. Historical comparisons that ignore this naming drift will confuse model identity with API aliasing. (DeepSeek API Docs)

DeepSeek also matters because its later work starts speaking the language of inference-time scalability directly. The DeepSeek-GRM paper argues that proper learning methods can enable effective inference-time scalability for generalist reward models. That is a sign that the field’s center of gravity is shifting from “Can models reason?” to “How should learned models and test-time compute be co-designed?” (arXiv)

Gemini Thinking Models and Google’s reasoning systems: productized budgets and stronger search hybrids

Google’s public Gemini line is the most operationally explicit about inference budgets. The Gemini 2.5 blog describes the models as “thinking models,” says Google is building these thinking capabilities directly into all models, and adds a telling caveat: Gemini 2.5 Pro’s strong math and science numbers are reported without cost-increasing test-time techniques like majority voting. That phrasing only makes sense in a world where majority voting and similar inference tricks are already part of the standard reasoning toolkit. (blog.google)

Google’s developer docs then make the tradeoff concrete. For Gemini 2.5 models, thinkingBudget controls the number of thinking tokens used for reasoning; 2.5 Flash can set the budget from 0 to 24,576 and can disable thinking with 0, while 2.5 Pro uses dynamic thinking by default and cannot disable it. Google’s own 2.5 Flash launch note says higher thinking budgets improve reasoning quality, while also acknowledging the tradeoff among quality, cost, and latency. This is test-time compute turned into a first-class product parameter. (Google AI for Developers)

Google’s stronger research systems push beyond simple hidden scratchpads. In the 2025 IMO result, DeepMind says an advanced Gemini Deep Think mode used “parallel thinking,” exploring and combining multiple possible solutions before answering, and solved five of six Olympiad problems for a gold-medal-level score within the competition time limit. That description is much closer to search over candidate solution paths than to a single long monologue. (Google DeepMind)

The formal-systems side is even more explicit. AlphaProof couples a pretrained language model with the AlphaZero RL algorithm and searches over Lean proof steps, while AlphaGeometry combines a neural language model with a symbolic deduction engine that repeatedly adds useful constructs and continues deduction until a proof is found. These are not merely “reasoning models” in the chat-product sense. They are learned-search systems in the stronger classical sense. If one wants the best evidence that the search half of Sutton’s frame is returning, Google’s formal math systems are more convincing than any generic chatbot. (Google DeepMind)

Classical search has returned—though in softened form

The easiest way to situate modern test-time compute is to map it onto older search ideas. The novelty is not that search suddenly exists; beam search has been standard in sequence generation for years. The novelty is that frontier LLM systems are moving from narrow token-level search toward broader trajectory-level, verifier-guided, tree-shaped, and tool-mediated search. (SciSpace)

Classical method	Unit being searched	Modern reasoning analogue	Main limitation
Beam Search	partial token sequences	wider decoding over candidate next-token paths	still local and left-to-right
Self-Consistency / majority vote	complete reasoning trajectories	pass@k / cons@64 / sampled solution voting	expensive and answer-level only
Best-of-N Sampling with verifier	complete trajectories ranked by a judge	verifier-guided selection, reward-model ranking	depends on judge quality
Tree of Thoughts	intermediate “thought” states	branching, self-evaluation, backtracking	complex orchestration
Monte Carlo Tree Search	action/state trees with rollouts and values	explicit search-heavy agent systems, theorem proving, some planner loops	requires a good state/action abstraction

The table condenses a lineage drawn from decoding work, self-consistency, verifier papers, Tree of Thoughts, and MCTS. It is a conceptual map, not a claim that current products literally implement the classical algorithms unchanged. (SciSpace)

Beam search is the old baseline. It explores multiple continuations but keeps only the top candidates at each step, which makes it a heuristic local search over token sequences. It absolutely counts as test-time compute, but it is the weakest form: fixed-width, next-token local, and usually not reasoning-aware. Modern reasoning systems differ by searching over larger units than tokens and by allowing re-evaluation later in the trajectory. (Rycolab)

Self-consistency and related consensus methods search over whole trajectories. Rather than asking for one chain-of-thought and trusting it, the model samples several and either majority-votes the answer or selects the most consistent one. OpenAI’s cons@64 reporting is the mainstream industrial version of this logic. The self-consistency paper formalized the idea and showed strong benchmark gains years before o1. In that sense, part of the “reasoning model” wave is a productization of already-known inference techniques married to stronger base models and RL-trained scratchpads. (OpenAI)

Verifier-based search adds a judge. The GSM8K verifier paper proposed generating many candidate solutions and selecting the one scored highest by a verifier; Snell et al. frame dense, process-based verifier reward models as one major route to inference scaling. The advantage is obvious: verification can be easier than generation. The disadvantage is equally obvious: the whole method depends on the quality of the verifier. Recent work on “when to solve, when to verify” makes the point sharper, finding that self-consistency is more compute-efficient than generative verification for most practical budgets, with GenRM needing up to 8× more inference compute to match SC. More compute is not enough; compute allocation matters. (arXiv)

Tree-of-thought and MCTS-like methods move closest to classical deliberative search. Tree of Thoughts explicitly branches on coherent thought units, self-evaluates, looks ahead, and backtracks. MCTS and UCT are the classical reference point: search a tree, sample rollouts, and guide exploration with value estimates or bandit-style confidence bounds. Most chat reasoning products do not expose this kind of structured tree search directly, but systems like AlphaProof, AlphaGeometry, and some agentic tool loops are unmistakably moving in that direction. (arXiv)

A practical way to say it is this: today’s “reasoning models” are usually not classical search engines with clean formal states. They are learned generators that can spend more inference compute exploring, checking, and synthesizing candidate traces in language-space, sometimes with tools or verifiers attached. That is softer than AlphaZero-style search, but it is still a real return of search to the center of AI engineering. (OpenAI)

How reasoning tokens work operationally

The operational semantics of reasoning tokens matter because they determine what users can control, what they can inspect, and what they pay for. The three major ecosystems have made markedly different choices.

Provider	Main control knob	Visibility of raw reasoning	Cross-turn handling	Billing facts
OpenAI	reasoning.effort, max_output_tokens	raw reasoning hidden; summaries optional; counts exposed	reasoning discarded from normal context, but reasoning items can be carried across turns/function calls	reasoning tokens are billed as output tokens
Google	thinkingBudget / thinkingLevel, dynamic thinking	raw thoughts hidden; thought summaries optional; thought-token counts exposed	stateless API uses thought signatures to preserve thought context	thinking tokens are billed, and output pricing includes them
DeepSeek	deepseek-reasoner or thinking enabled; max_tokens includes CoT	raw reasoning exposed as reasoning_content	do not feed prior reasoning_content back; next turn uses final content	billed on total input/output tokens; reasoning contributes to output cost

Current OpenAI, Google, and DeepSeek docs support the rows above; the historical R1 pricing figure below comes from DeepSeek’s January 2025 release note rather than its current API alias. (OpenAI Developers)

OpenAI’s current reasoning guide says reasoning.effort controls how many reasoning tokens the model generates, describes those tokens as hidden internal work that is discarded from the normal context after the answer, and says they are still billed as output tokens. The same guide says the number of reasoning tokens is visible in the usage object, that max_output_tokens constrains both hidden reasoning and visible output, and that developers can incur costs even if the model runs out of budget during reasoning before producing a visible answer. OpenAI recommends reserving at least 25,000 tokens while experimenting. For function-calling workflows, OpenAI also exposes reasoning items that can be preserved across turns, including encrypted reasoning content in stateless settings, while keeping raw reasoning text hidden; summaries can be requested explicitly. (OpenAI Developers)

Google’s design is more explicit about budgeting. Gemini’s docs say 2.5-series models use thinkingBudget, with 2.5 Flash supporting budgets from 0 to 24,576 and 2.5 Pro from 128 to 32,768. The docs say thought summaries are summarized versions of raw thoughts, not the raw thoughts themselves, and that the budget applies to raw thought generation rather than to summaries. The same docs describe thought signatures as encrypted representations of internal thought that let a stateless API preserve thought context across turns. Google also exposes thoughtsTokenCount, and its docs say pricing is based on the full thought tokens generated even when only summaries are returned. (Google AI for Developers)

DeepSeek is the outlier in transparency. Its docs say the API provides access to chain-of-thought via reasoning_content, with max_tokens including the CoT portion of the output. That makes the cost and the content much more visible to the user. But the same docs stress that prior reasoning_content should not be concatenated into the next turn’s input, and that including it can cause errors. As of current docs, deepseek-reasoner maps to DeepSeek-V3.2 thinking mode with output pricing of $0.42 per million tokens; the January 2025 R1 release used the same API label for DeepSeek-R1 and quoted $2.19 per million output tokens. That is a reminder that “reasoning model behavior” in production is partly an API-contract question, not just a model-weight question. (DeepSeek API Docs)

Current listed prices also show the economic incentives shaping product design. OpenAI’s current model pages list o1 at $60 per million output tokens and o3 at $8; Google lists Gemini 2.5 Flash standard output at $2.50 per million tokens including thinking; DeepSeek’s current deepseek-reasoner page lists $0.42. These are not apples-to-apples quality comparisons, but they do explain why the industry is racing to move from “max reasoning at any cost” to better cost-performance frontiers. (OpenAI Developers)

ARC-AGI as a case study in inference-time scaling

ARC-AGI is the most vivid public case study because it shows the gains from extra test-time compute and the economic absurdity of those gains at the same time. ARC Prize’s December 2024 o3-preview report says the model scored 75.7% on the ARC-AGI-1 semi-private set under the public leaderboard’s compute limit, and 87.5% in a high-compute configuration using 172× more compute. The report gives the operational details: 6 samples in the efficient setting versus 1024 samples in the expensive one, with cost per task rising from $26 to $4,560. That is almost the Platonic form of inference-time scaling: same general system, more rollout budget, better score, much worse economics. (ARC Prize)

ARC’s own analysis is careful about what this does and does not prove. The preview report says the tested o3 variant had been trained on 75% of the ARC-AGI-1 public training set, and ARC later noted that the released public o3 was a different model with different compute constraints. In April 2025, ARC said the preview-level test-time compute used in December was not available in public o3, so users should not expect identical results. This is important because it punctures a common confusion: “o3 scored X on ARC” only means something if one also specifies which o3 and how much inference budget was allowed. (ARC Prize)

ARC’s April 2025 production-model analysis then became a counterweight to the breakthrough story. It reported public o3 at 41% on ARC-AGI-1 and 1.9% on ARC-AGI-2 with low reasoning, and 53% / 2.9% with medium reasoning; o4-mini reached 42% / 2.3% at medium reasoning. High-reasoning runs were often incomplete or timed out, making them unsuitable for leaderboard use. ARC also observed that higher reasoning effort could be inefficient: easier tasks returned earlier and with higher accuracy, while extra reasoning on easy tasks often increased cost without improving answers. (ARC Prize)

That is why ARC is such a useful benchmark for this topic. It does not merely show that more compute helps. It shows that inference-time scaling can be real, fragile, and economically ugly all at once. The preview result demonstrated a true capability jump with more compute. The production analysis showed that public, latency-bounded systems expose only part of that curve. And ARC-AGI-2 showed that even an apparently dramatic jump on ARC-AGI-1 does not straightforwardly transfer to the next, harder benchmark. (ARC Prize)

ARC Prize’s own framing is worth taking seriously. In the ARC-AGI-2 announcement, the organization published a table where a human panel reached 100% on ARC-AGI-2 at roughly $17 per task, while o3-preview-low was estimated around 4% at roughly $200 per task and R1/R1-Zero around 0.3% at $0.08 per task. The post then states the main lesson directly: intelligence is not only about solving problems, but about doing so efficiently. ARC’s current leaderboard carries the same message, emphasizing cost-per-task and noting that reasoning-system trend lines usually show asymptotic returns as more reasoning time is added. (ARC Prize)

So ARC-AGI does not merely support the thesis that test-time compute matters. It sharpens it into a more uncomfortable thesis: test-time compute can buy impressive capability, but efficiency is part of the target, not an afterthought. If an 87.5% score requires thousands of dollars per task, the result is scientifically interesting but not yet economically general. (ARC Prize)

The capability-cost bargain

The academic literature on Inference Scaling Laws turns the ARC intuition into a general principle. Snell et al. report that a compute-optimal inference strategy can be more than 4× as efficient as a best-of-N baseline, and that on some tasks extra test-time compute lets a smaller model beat a 14× larger model under matched FLOPs. Wu et al. similarly find that smaller models combined with stronger inference strategies can dominate larger models under the same compute budgets. These are not arguments against training scale. They are arguments that how you spend inference compute matters enough to be a first-order design variable. (arXiv)

The literature is also more nuanced than the marketing. The 2025 “When to Solve, When to Verify” paper finds that self-consistency usually beats generative reward-model verification at practical budgets, with GenRM needing up to 8× more inference compute to catch up. The s1 paper shows that even very simple budget forcing—cutting off or extending a model’s reasoning trace, including by appending “Wait”—can move AIME24 from 50% to 57%. The lesson is not “just spend more compute.” The lesson is that inference algorithms, stopping rules, and adaptive allocation policies are part of the capability stack. (arXiv)

Product documentation says the same thing in plainer language. OpenAI’s reasoning guide says xhigh effort should be used only when evals show the benefit justifies the extra latency and cost. Google’s 2.5 Flash note says different use cases have different tradeoffs, that higher budgets can improve quality, and that a budget of 0 is appropriate when one wants the lowest cost and latency. When vendors tell developers to treat reasoning depth as a tunable cost knob, they are admitting that the central engineering problem is no longer merely model quality but capability under budget. (OpenAI Developers)

This is also why public model releases often lag behind the pure research frontier. A preview model with near-max test-time compute can produce a dramatic benchmark spike, but a public product must hit latency SLOs, unit economics, and volume targets. ARC’s production-o3 result is a vivid example of this narrowing from “what the research system can do” to “what the deployed service can afford to do.” OpenAI’s own o3 launch note, notably, frames the win as an improved cost-performance frontier over o1, not as a license to spend unlimited inference compute. (ARC Prize)

Does test-time compute produce genuine reasoning, or just better pattern matching at higher cost?

The empirical answer to the narrow engineering question is now clear enough: more inference compute often improves performance on hard tasks. The philosophical answer to the broader question is not clear at all.

The strongest evidence on the “yes, something reasoning-like is happening” side is substantive. OpenAI describes o-series models as learning to refine their thinking, try different strategies, and recognize mistakes. DeepSeek-R1’s paper reports emergent self-reflection, verification, and dynamic strategy adaptation under RL. Google’s Deep Think result describes parallel exploration and combination of multiple candidate solutions. The academic literature on self-consistency, Tree of Thoughts, verifier-guided selection, and compute-optimal inference shows systematic gains from broader search over candidate reasoning trajectories. These are not trivial signs of a model merely becoming more verbose. They are signs of a system doing more internal or external work before it commits. (OpenAI)

There is also a stronger systems-level argument. Google’s AlphaProof and AlphaGeometry are not plausibly described as mere next-token pattern matchers. They are learned components embedded in explicit search-and-verification loops over formal mathematical states. If one’s standard for “reasoning” requires structured search, explicit verification, or backtracking, then these systems are better evidence than everyday chat reasoning models. (Google DeepMind)

But the counterevidence is serious and cannot be waved away.

The first problem is faithfulness. The paper Chain-of-Thought Reasoning In The Wild Is Not Always Faithful reports that frontier models can produce coherent-looking but logically contradictory rationalizations, silently repair earlier mistakes, and use unfaithful shortcuts even on realistic prompts without artificial biasing. A March 2026 paper on chain-of-thought controllability adds a sharper concern: if reasoning models can strategically reshape what they verbalize, CoT monitoring becomes a weaker window into the actual mechanism. So even when visible reasoning looks sophisticated, it is unsafe to treat the text trace as a transparent explanation of causality inside the model. (arXiv)

The second problem is benchmark contamination and exposure. ARC’s own analysis says the o3-preview variant tested in December 2024 included 75% of the ARC-AGI-1 public training set, and that the public o3 was not directly trained on ARC but likely had some indirect exposure because the benchmark is public. Apple’s The Illusion of Thinking argues that standard math and coding evaluations often do not reveal underlying reasoning limits and may suffer contamination; a published comment on that paper argues that the reported collapses are better explained by experimental design limitations than by a fundamental failure of reasoning models. The right conclusion is not that Apple refuted reasoning models, nor that the rebuttal refuted Apple. It is that the empirical case remains contested when one moves from benchmark scores to claims about general cognitive mechanism. (ARC Prize)

The third problem is that better search over learned patterns can look like reasoning without settling the deeper issue. A model that samples many candidate traces and picks the one a verifier likes may become much better at benchmarks even if what improved was not an abstract world model but the efficiency of search through a learned manifold of plausible solutions. That is still a real advance. It may also be most of what matters for engineering. But it is not yet the same as a demonstrated capacity for robust, domain-general reasoning under distribution shift. The literature on compute-optimal inference and verifiers is powerful evidence for the engineering claim, not decisive evidence for the philosophical one. (arXiv)

The most honest synthesis is therefore strict. Current reasoning models show genuine improvement in problem-solving under added inference compute. They do not yet settle whether the mechanism deserves a strong philosophical notion of reasoning, because visible traces are not fully faithful, benchmarks are imperfect, and some of the strongest gains can be explained as better search over learned representations. That is still a big deal. It is just not a metaphysical victory lap. (arXiv)

Sutton’s “search and learning” and the partial vindication of The Bitter Lesson

Rich Sutton’s 2019 essay argued that the general-purpose methods that continue to scale with increased computation are search and learning. That formulation matters because the public conversation around large models often absorbed only the “learning” half: bigger models, more data, more compute, better pretraining. The recent reasoning-model wave is the clearest mainstream rehabilitation of the “search” half. (UT Austin Computer Science)

The learning half is already obvious. OpenAI’s o-series and DeepSeek-R1 both describe large-scale RL as central to their reasoning behavior; Google describes Gemini 2.5 as a stronger base model plus improved post-training, and its strongest formal-math systems are also trained with RL. On that front, nothing about recent reasoning systems contradicts Sutton. They intensify the lesson. (OpenAI)

The interesting question is whether they vindicate the search half. I think the answer is: partially, and in an unexpectedly soft form.

Partially, because modern frontier systems increasingly win by spending more inference compute to branch, sample, vote, verify, backtrack, or use tools. OpenAI’s o1 report explicitly plots gains from more test-time compute. ARC’s o3 preview result shows a dramatic score increase from much larger sample budgets. Self-consistency, Tree of Thoughts, verifier selection, and compute-optimal inference papers all tell the same story from different angles. Google’s Deep Think result explicitly describes parallel exploration of multiple solutions, and AlphaProof/AlphaGeometry make the search component fully explicit. If Sutton’s point was that scalable progress comes from methods that can absorb more compute at run time rather than from brittle human-crafted priors, then current reasoning systems are clearly moving in that direction. (OpenAI)

But only partially, because most mass-market reasoning models still do not search the way chess engines, AlphaZero, or formal theorem provers search. They usually search over language traces and tool outputs, not over crisp symbolic states with well-defined transition rules and reliable value functions. Their search is learned, approximate, and often opaque. The most convincing “search” systems in the current wave are arguably Google’s formal-math systems, not the general-purpose chat models. So the verdict is not “The Bitter Lesson fully resolved the philosophy of reasoning.” It is closer to: the LLM era is rediscovering search, but first in a fuzzy language-space form, not in the hard formal form classical AI partisans would have recognized. (GGP)

That softened return of search still matters enormously. It suggests the next phase of AI progress is unlikely to be “pretrain forever and hope.” The more plausible trajectory is co-design: stronger base models, stronger post-training, and increasingly compute-aware inference policies that decide when to think longer, when to sample more, when to verify, when to use tools, and when not to spend the money. That is Sutton’s pair—search and learning—returning as a single engineering program. (UT Austin Computer Science)

Conclusion

Test-time compute is now too central to be treated as a footnote to training scale. OpenAI states it plainly, Google exposes it as a product knob, DeepSeek makes it visible enough to inspect and distill, and ARC-AGI shows both the gains and the brutal economics. The research literature has also matured past slogans: there are now genuine inference-time scaling laws, compute-optimal allocation results, and increasingly sharp comparisons among sampling, verification, and search strategies. (OpenAI)

The sober bottom line is stronger than hype but weaker than metaphysics. Extra inference compute clearly improves frontier systems on many hard tasks. It also clearly reintroduces search into the mainstream of AI, partially vindicating Sutton’s “search and learning” frame. What it does not yet prove is that hidden reasoning traces, expensive search, or longer thought budgets amount to settled, robust, domain-general reasoning in the philosophical sense. That question remains open, and the honest answer today is still conditional. (UT Austin Computer Science)

Companion entries

Core theory: Inference-Time Compute, Inference Scaling Laws, Search and Learning, The Bitter Lesson, Chain-of-Thought, Process Reward Models

Methods: Beam Search, Self-Consistency, Best-of-N Sampling, Tree of Thoughts, Monte Carlo Tree Search, Verifier-Guided Search

Systems and case studies: OpenAI o1, OpenAI o3, DeepSeek-R1, Gemini Thinking Models, ARC-AGI, AlphaProof, AlphaGeometry

Practice: Reasoning Tokens, Cost-Aware Inference, Latency-Capability Tradeoffs, Agentic Tool Use, Reasoning API Design

Counterarguments: Chain-of-Thought Faithfulness, Benchmark Contamination, Brute Force vs Intelligence, The Illusion of Thinking, Interpretability of Reasoning Traces