Tree of Thoughts: Deliberate Reasoning Through Search

Tree of Thoughts (ToT) is a prompting-and-search framework that treats language-model reasoning as explicit state-space search over semantically meaningful intermediate steps, rather than as a single left-to-right chain. Its central contribution is not “more chain-of-thought,” but a modular inference loop: generate candidate thoughts, evaluate partial states, keep or discard branches, and optionally backtrack. ToT is best understood today as one concrete algorithm inside the broader Test-Time Compute family: powerful on tasks with brittle intermediate commitments, expensive compared with ordinary Chain-of-Thought Prompting, and increasingly entangled with the question of whether modern reasoning models internalize search inside hidden reasoning tokens.

Coverage note: verified through May 6, 2026.

Why Tree of Thoughts mattered

Yao et al.’s 2023 paper, “Tree of Thoughts: Deliberate Problem Solving with Large Language Models,” introduced ToT as a way to move beyond the dominant prompting pattern of the time: ask a model to produce a single chain of reasoning, perhaps sample many such chains, and choose an answer. The paper’s framing was simple but consequential: if a problem naturally involves exploration, lookahead, and revision, then an LLM should not be forced to solve it through one irreversible decoding trajectory. Instead, the reasoning process can be externalized as a search procedure over partial solutions, where the language model supplies both candidate next steps and judgments about which partial states are promising. NeurIPS Proceedings

The “thought” in ToT is deliberately coarser than a token. In the Game of 24 task, a thought is an arithmetic step; in creative writing, it is a high-level plan; in mini crosswords, it is a possible word placement. This makes ToT a bridge between Prompt Engineering and classical State-Space Search: the search state is expressed in natural language, but the control structure looks like beam search, breadth-first search, depth-first search, or a future extension such as A* or Monte Carlo Tree Search. Yao et al. explicitly described ToT as maintaining an active tree of thoughts, using self-evaluation to decide where to search, and allowing lookahead and backtracking rather than committing to the first sampled path. NeurIPS Proceedings

The paper’s headline result was striking: on the Game of 24, standard chain-of-thought prompting with GPT-4 solved only 4% of the selected hard problems, while ToT with breadth five solved 74%. But the broader lesson is more nuanced. ToT was much more expensive than ordinary prompting, its gains were task-dependent, and its best empirical case was not “all reasoning,” but tasks where local mistakes poison later reasoning and where partial states can be evaluated before a final answer is produced. NeurIPS Proceedings

The lineage: from CoT to self-consistency to ToT

ToT sits in a clear historical sequence: Chain-of-Thought Prompting made intermediate reasoning explicit; Self-Consistency sampled multiple reasoning paths and voted over final answers; ToT turned intermediate reasoning into a searchable object.

Chain-of-thought prompting, popularized by Wei et al. in 2022, showed that large language models could improve on arithmetic, commonsense, and symbolic reasoning tasks when prompted to generate intermediate reasoning steps before answering. Google’s accompanying research communication emphasized that the effect was broad and emerged strongly at sufficient model scale. arXiv

Self-consistency, introduced by Wang et al., replaced greedy decoding with multiple sampled reasoning chains and then selected the most consistent final answer. It improved performance across reasoning benchmarks such as GSM8K, SVAMP, AQuA, StrategyQA, and ARC by marginalizing over sampled paths rather than trusting a single one. arXiv

ToT extends that idea in two ways. First, it samples and evaluates partial reasoning states, not only completed chains. Second, it allows the search controller to abandon, rank, or revisit branches before producing a final answer. Yao et al. explicitly contrasted ToT with self-consistency: self-consistency samples independent full chains and votes only at the end, while ToT can inspect and select among intermediate states during the search. NeurIPS Proceedings+2NeurIPS Proceedings+2

Method	Reasoning topology	Where selection happens	Typical selection signal	Main strength	Main weakness
Input-output prompting	No explicit reasoning path	Final answer only	Model likelihood or one sample	Cheap and simple	No deliberate decomposition
Chain-of-Thought Prompting	One path	During left-to-right decoding, implicitly	Token probabilities	Exposes intermediate reasoning	Early errors are hard to recover from
Self-Consistency	Many independent paths	Final answer aggregation	Majority vote or answer agreement	Robust to unlucky samples	No local repair; weak for open-ended outputs
Tree of Thoughts	Tree of partial states	At each search depth	Value function, vote, verifier, heuristic	Enables lookahead, pruning, backtracking	Expensive; task-specific; evaluator may be unreliable

The important conceptual move is that ToT changes the unit of inference from “sample a completion” to “search a problem space.” This makes it natural to compare ToT not only to prompting methods, but also to classical AI search, program synthesis, theorem proving, and agent planning.

Algorithmic structure

Yao et al. formalize ToT as search over a tree whose nodes are partial reasoning states. Given an input problem x, a state at depth i can be written as:

s_i = [x, z_1, z_2, ..., z_i]

where each z_j is a thought: a meaningful intermediate step. The algorithm repeatedly performs three operations:

Thought generation: propose possible next thoughts from the current state.
State evaluation: estimate how promising each resulting partial state is.
Search control: choose which states to expand, retain, prune, or backtrack from.

In Yao et al.’s formulation, the language model can serve both as a generator and as an evaluator. Generation can be done by sampling independent completions or by prompting the model to propose a list of next steps. Evaluation can be done independently for each state, such as rating a Game of 24 partial equation as “sure,” “maybe,” or “impossible,” or comparatively, by asking the model to vote among candidate states. NeurIPS Proceedings

A simplified ToT loop looks like this:

frontier = {initial_state}for depth in 1..T:    candidates = []    for state in frontier:        thoughts = generate_next_thoughts(state)        candidates += [state + thought for thought in thoughts]    scores = evaluate(candidates)    frontier = select_top_states(candidates, scores, budget)return best_final_state(frontier)

This is not one algorithm so much as a template. Yao et al. instantiate it with breadth-first search for Game of 24 and creative writing, and depth-first search with pruning and backtracking for mini crosswords. Their BFS version generates candidate children, evaluates them, keeps the top b states, and proceeds to the next depth. Their DFS version sorts candidates by value, prunes states below a threshold, and backtracks when a branch fails. NeurIPS Proceedings

The key modular components are:

Component	Role in ToT	Typical implementation
State representation	Encodes the problem and partial solution	Natural-language transcript, equation trace, board state, tool log
Thought generator	Produces candidate next steps	LLM sampling, proposal prompt, tool call, domain heuristic
Value function	Scores partial states	LLM-as-judge, verifier, reward model, unit test, symbolic checker
Search policy	Decides what to expand	BFS, DFS, beam search, MCTS, best-first search
Termination criterion	Decides when to stop	Fixed depth, solved state, budget exhaustion, verifier success

This modularity is why ToT became influential beyond its original benchmarks. It gave practitioners a vocabulary for replacing “one big prompt” with an inference-time control loop: decompose, branch, evaluate, select, and continue. That same control loop later appears in Verifier-Guided Decoding, Language Agent Tree Search, Process Reward Models, and agentic workflows that combine language models with external tools.

What counts as a “thought”?

A frequent misunderstanding is that ToT searches over arbitrary sentences. The paper’s practical implementations are much more structured. The “thought” is task-designed.

In Game of 24, the input is four numbers and the goal is to form an expression equal to 24. A thought is an intermediate arithmetic operation, such as combining two numbers and carrying forward the remaining numbers. This structure makes partial states easy to inspect: a branch can be judged impossible if the remaining numbers are unlikely to reach 24. NeurIPS Proceedings

In creative writing, the task asks the model to produce a passage whose four paragraphs end with four given sentences. A thought is a plan: a high-level narrative outline that makes the later passage more coherent. Here, the evaluator is less objective; Yao et al. used GPT-4 and human judgments, but the paper acknowledges that automatic evaluation is noisy. NeurIPS Proceedings

In mini crosswords, a thought is a word fill for a clue. The search state is the partially filled grid, and backtracking matters because an early plausible word can block later intersections. The value function estimates whether remaining clues are still solvable, and DFS can prune branches judged impossible. NeurIPS Proceedings

This task dependence is not incidental; it is the method. ToT works best when “thought” is a semantically meaningful action with an evaluable partial consequence. If the thought unit is too small, search degenerates into token-level decoding. If it is too large, ToT becomes ordinary best-of-N sampling. The craft lies in choosing the granularity.

The Yao et al. experiments

Yao et al. evaluated ToT on three tasks intended to stress different reasoning behaviors: arithmetic search, open-ended planning, and combinatorial constraint solving. The experiments used GPT-4 for the main results, with sampling temperature 0.7. The authors also tested GPT-3.5 variants in an appendix, where ToT still helped but remained far below GPT-4+ToT on the hardest task. NeurIPS Proceedings

Game of 24

The Game of 24 is the cleanest demonstration. The model receives four numbers and must combine them with arithmetic operations to make 24. Yao et al. selected 100 hard instances from a dataset of four-number games, evaluated success by exact validity, and compared input-output prompting, chain-of-thought prompting, self-consistency, iterative refinement, and ToT. NeurIPS Proceedings

The result was dramatic:

Method	Game of 24 success
Input-output prompting	7.3%
Chain-of-thought	4.0%
Self-consistency over 100 CoT samples	9.0%
Best of 100 input-output samples	33.0%
Best of 100 CoT samples	49.0%
ToT, breadth 1	45.0%
ToT, breadth 5	74.0%

The paper’s qualitative diagnosis is important. Roughly 60% of sampled CoT paths failed after the first step, which means the model often made an early arithmetic commitment that could not be repaired downstream. ToT’s advantage came from evaluating partial arithmetic states before committing to them, not from merely generating more text. NeurIPS Proceedings

This is the canonical ToT use case: a small search space, brittle early decisions, cheap partial-state evaluation, and a clear final verifier. The algorithm is not magic; it is giving the model a scaffold that resembles how a human might try multiple arithmetic decompositions instead of writing one solution straight through.

Creative writing

The creative-writing task was intentionally less formal. The model had to write a four-paragraph passage ending each paragraph with one of four specified sentences. The ToT version generated five plans, voted on the best plan, then generated five passages from that plan and voted again. NeurIPS Proceedings

The reported GPT-4 coherence scores were:

Method	GPT-4 coherence score
Input-output prompting	6.19
Chain-of-thought	6.93
ToT	7.56

Human comparison also favored ToT: in 100 pairwise comparisons, humans preferred ToT over CoT 41 times, preferred CoT over ToT 21 times, and judged 38 pairs similar. NeurIPS Proceedings

This result is suggestive rather than decisive. Unlike Game of 24, there is no exact ground truth. The paper itself notes that automatic metrics can be noisy, and iterative refinement also improved performance. The stronger lesson is that ToT can be used as a planning scaffold for open-ended generation, but the evidence is thinner because the value function is less objective. NeurIPS Proceedings

Mini crosswords

The mini-crossword task tested backtracking. Each puzzle had five horizontal and five vertical clues, and a candidate answer had to satisfy intersecting constraints. Yao et al. used DFS-style ToT: choose a clue, propose fills, evaluate whether the resulting grid remains plausible, prune impossible states, and backtrack when necessary. NeurIPS Proceedings

The reported results were:

Method	Letter success	Word success	Game success
Input-output prompting	38.7%	14.0%	0%
Chain-of-thought	40.6%	15.6%	1%
ToT	78.0%	60.0%	20%
ToT + best state oracle	82.4%	67.5%	35%
ToT without pruning	65.4%	41.5%	5%
ToT without backtracking	54.6%	20.0%	5%

The ablations show why this task mattered. Removing pruning or backtracking substantially hurt performance, which supports the claim that ToT’s gain was not merely from asking GPT-4 to “think harder.” It came from a search procedure that could revise earlier choices. The oracle “best state” result also exposes evaluator weakness: some correct or promising states were not selected by the model’s value judgments. NeurIPS Proceedings

Cost geometry: why ToT is expensive

ToT buys accuracy with inference-time computation. The cost is not a small prompt tweak; it is a multiplicative search procedure.

At a high level, the cost depends on:

depth × frontier width × candidates per state × evaluation samples per candidate

The exact count depends on batching and prompt design. A generator may propose several thoughts in one call, and an evaluator may score multiple candidates in one prompt. But the basic geometry remains: every branch you entertain can create more branches to evaluate, and every evaluation may itself be an LLM call.

Yao et al.’s cost tables make the trade-off concrete. On Game of 24, ToT used roughly the same order of generated tokens as 100 chain-of-thought samples, but achieved higher accuracy. The paper reports the following approximate costs for one problem under its GPT-4 pricing assumptions:

Method	Generated tokens	Prompt tokens	Approx. cost	Success
Best of 100 input-output samples	1.8k	1.0k	$0.13	33%
Best of 100 CoT samples	6.7k	2.2k	$0.47	49%
ToT	5.5k	1.4k	$0.74	74%

For creative writing, ToT was also several times more expensive than input-output or CoT prompting: approximately $0.32 per problem for ToT versus $0.06–$0.07 for the baselines in the paper’s setup. Yao et al. summarize the broader cost range bluntly: ToT can require 5–100× more generated tokens than CoT depending on the search configuration, and efficiency remains an open problem. NeurIPS Proceedings

The cost geometry has several implications.

First, ToT is most attractive when each successful answer is valuable enough to justify expensive search: hard math, code repair, formal reasoning, planning with tools, high-stakes analysis, or tasks with automatic verifiers. It is much less attractive for low-value, latency-sensitive, or easy tasks.

Second, the value function is the economic bottleneck. A weak evaluator wastes compute by expanding bad branches or pruning good ones. The crossword oracle result in Yao et al. is a concrete example: the search sometimes contained better states than the algorithm selected. NeurIPS Proceedings

Third, ToT competes with simpler test-time compute baselines. Before deploying ToT, one should compare it with best-of-N sampling, self-consistency, verifier reranking, iterative refinement, and a stronger base model. In Game of 24, ToT beat best-of-100 CoT, but that does not imply it will beat simpler scaling baselines on every task. NeurIPS Proceedings

ToT as test-time compute

ToT is one specific algorithm in the larger class of Test-Time Compute methods: techniques that spend more computation at inference time to improve output quality. This family includes best-of-N sampling, self-consistency, beam search, verifier-guided search, process reward models, tool-augmented planning, iterative refinement, and explicit tree search.

Snell et al.’s later work on scaling test-time compute studied how to allocate a fixed inference budget across strategies such as searching against process-based verifier reward models and adaptively updating the model’s distribution at test time. Their main message aligns with the ToT experience: extra inference compute can improve outputs, but the optimal strategy depends on problem difficulty, verifier quality, and the base model’s competence. arXiv

The “Large Language Monkeys” line of work reached a related conclusion from a simpler direction: repeated sampling can scale surprisingly far, especially in domains like coding and formal proof where automatic verifiers can turn coverage into performance. But the same work also notes that selection by majority vote or reward model can plateau when verifiers are weak or the domain lacks reliable automatic checking. arXiv

This places ToT in a useful but bounded category. It is not synonymous with test-time compute; it is a structured way to spend test-time compute when partial states are meaningful and evaluable.

Test-time method	Uses branching?	Evaluates partial states?	Needs task-specific structure?	Typical use
Best-of-N	Yes, over full answers	No	Low	Cheap parallel sampling
Self-consistency	Yes, over full reasoning chains	No	Low to medium	Closed-answer reasoning
Iterative refinement	Usually linear	Sometimes	Medium	Writing, code repair, critique loops
Beam search over thoughts	Yes	Yes	Medium to high	Structured reasoning
ToT	Yes	Yes	High	Search problems with evaluable partial states
MCTS-style language-agent search	Yes	Yes, with backpropagation	High	Tool use, planning, interactive environments

The practical question is therefore not “Should we use ToT?” but “Given a fixed inference budget, where should computation be spent: more samples, deeper chains, explicit tree search, verifier calls, tool interactions, or a stronger reasoning model?”

Open-source ecosystem

The official ToT implementation from Princeton Language and Learning Lab is available as the Tree of Thoughts repository. It includes code, prompts, model outputs, and task scripts for Game of 24, creative writing, and crosswords. The repository exposes a BFS solver with parameters such as backend model, generation mode, evaluation mode, number of generated samples, number of evaluation samples, and number of selected states. It also notes that reproductions can vary because of randomness: a rerun reported 69% on Game of 24 versus the paper’s original 74%. GitHub

The official repository is also honest about task-specific work. To add a new task, one needs a task class, task data, and task-specific prompts for generation and evaluation. This is a crucial engineering point: ToT is not a drop-in universal reasoning switch. It is an orchestration pattern that must be adapted to a domain’s state representation, branching actions, and evaluator. GitHub

LangChain has exposed ToT-related abstractions in its experimental ecosystem, including a ToTChain in older/reference API material. The langchain-experimental package is explicitly positioned as experimental code, which is appropriate for ToT-style patterns because they often involve arbitrary model calls, tools, and task-specific control loops rather than stable one-call APIs. Scikit-learn+1

LlamaIndex’s ecosystem is more visible around Language Agent Tree Search than around a direct Yao-style ToT clone. The llama-index-agent-lats package implements LATS-style tree search for agents, and the official LATS repository points to both LangChain/LangGraph and LlamaIndex implementations. This reflects a broader trend: the production ecosystem absorbed ToT less as a fixed algorithm and more as a family resemblance among tree-search, reflection, verifier, and agent-planning loops. PyPI

Custom implementations remain common because the hard part is not writing the tree loop. The hard part is designing the thought unit, constraining the generator, building or prompting the evaluator, deciding a search budget, and preventing the model from rewarding fluent but invalid states. For many tasks, a few dozen lines of domain-specific search code plus a verifier are more reliable than a generic ToT framework.

Follow-up extensions

ToT quickly inspired variants that changed the search structure, reduced the number of calls, added validators, or moved toward agentic planning.

Graph of Thoughts generalizes the tree into an arbitrary graph of language-model thoughts. In GoT, thoughts can be combined, transformed, or refined through graph operations rather than expanded only as a parent-child tree. Besta et al. report improvements over ToT on several tasks while also reducing costs in some settings, but the method inherits the same central dependency: the graph operations and scoring functions must fit the task. arXiv

Algorithm of Thoughts tries to compress some of the benefit of search into fewer model calls by using algorithmic examples in context. Sel et al. frame AoT as a way to guide the model through algorithmic reasoning trajectories while reducing the overhead of multi-query tree methods. This is a different bet: instead of externalizing search broadly at inference time, teach the model a more efficient reasoning procedure through prompting. arXiv

Self-evaluation-guided beam search applies a similar idea to stepwise reasoning: sample intermediate steps, use model self-evaluation to score them, and keep a beam of promising candidates. This is close in spirit to ToT but narrower as an algorithmic pattern. arXiv

Boosting of Thoughts explores ensembles of reasoning trees and uses trial-and-error reasoning experiences as prompt material. Multi-agent ToT variants add independent reasoners and validator agents. These directions reflect the same underlying hypothesis: model performance improves when inference is structured as repeated proposal, critique, selection, and revision rather than one-shot generation. arXiv

The follow-up literature is uneven. Some papers provide useful abstractions; others are thin wrappers around prompting patterns. The durable contribution is not a branded acronym, but the recognition that language-model inference can be organized as explicit search over natural-language states.

Relationship to MCTS-style search

ToT is related to Monte Carlo Tree Search, but it is not MCTS by default.

Classical MCTS repeatedly selects a promising node, expands it, simulates or evaluates outcomes, and backpropagates value estimates through the tree. Yao et al.’s paper used simpler BFS and DFS variants and explicitly left extensions such as A* and MCTS for future work. NeurIPS Proceedings

Later work moved closer to MCTS. Reasoning with Language Model is Planning with World Model introduced RAP, which uses an LLM as both a world model and an agent, and incorporates MCTS-style planning to build reasoning trees under task-specific rewards. The key shift is that the model is not merely generating next thoughts; it is also simulating state transitions and planning over them. arXiv

Language Agent Tree Search further integrated MCTS with language agents, using language-model value functions, self-reflection, and environment feedback across tasks such as programming, web navigation, question answering, and math. This is more agentic than the original ToT paper: actions may affect an external environment, observations can be incorporated, and value can be propagated through a search tree. arXiv

The relationship can be summarized this way:

System	Search object	Evaluator	Backtracking?	Environment?	Closest classical analogy
Yao et al. ToT	Natural-language thoughts	LLM value/vote, sometimes task heuristic	Yes	Usually no	Beam/BFS/DFS search
RAP	Reasoning states and actions	Task reward, LLM world model	Yes	Simulated	MCTS planning
LATS	Agent states, actions, reflections	LM value, environment feedback	Yes	Yes	MCTS for language agents
AlphaProof-style systems	Formal proof states	Formal verifier + learned policy	Yes	Formal environment	AlphaZero-like search/RL

The stronger the verifier and the clearer the environment transition, the more search begins to look like classical planning or theorem proving. The weaker the verifier, the more it resembles expensive prompt sampling with an unreliable judge.

Relationship to OpenAI o1 and modern reasoning models

OpenAI’s o1 series changed the conversation around ToT because it made Reasoning Tokens and inference-time deliberation part of mainstream model design. OpenAI’s public materials describe o1 as trained with large-scale reinforcement learning to reason using chain-of-thought, with performance improving both from more RL training compute and from more time spent thinking at test time. OpenAI also describes the o1 models as designed to spend more time reasoning before answering, especially on hard science, coding, and math tasks. OpenAI

That does not mean o1 is “using Tree of Thoughts” in the Yao et al. sense. Public OpenAI materials do not disclose an external tree controller that enumerates visible branches, scores partial states, and backtracks. The safer interpretation is that o1-style models internalize some forms of deliberation through training and hidden inference-time computation. In other words, they compete with explicit ToT by moving part of the search process inside the model’s learned reasoning policy. OpenAI

Google’s reasoning-model literature points in a similar direction. Gemini 2.5 was introduced as a “thinking model,” and Deep Think was later described as an enhanced reasoning mode using extended and parallel thinking, novel reinforcement learning, and longer inference-time exploration of hypotheses. DeepMind’s 2026 discussion of Gemini Deep Think also emphasizes inference-time scaling on formal reasoning benchmarks. Google DeepMind+3blog.google+3blog.google+3

The important distinction is architectural visibility. ToT is an externalized search algorithm: the developer can inspect branches, swap evaluators, impose symbolic constraints, and allocate budget explicitly. o1/Gemini-style reasoning is largely internalized: the model may spend hidden compute before answering, but the developer usually sees the final answer and perhaps a summary, not the complete search frontier. This difference matters for debugging, safety, cost accounting, and tool integration.

There is also a third lineage: formal or semi-formal reasoning systems such as DeepMind’s AlphaProof, which couples language models with formal proof environments and AlphaZero-style reinforcement learning. These systems are closer to classical search and verification than generic natural-language ToT, because a formal environment can certify proof steps. Google DeepMind

The open question: does explicit ToT remain useful?

The live question is whether explicit ToT-style search remains useful as frontier models internalize more deliberation. The answer is probably task-dependent.

Explicit ToT remains compelling when:

Condition	Why it favors explicit ToT
Partial states are objectively checkable	Verifiers can prune bad branches better than the model alone
Early decisions are irreversible	Search can preserve alternatives before committing
The search space is small but tricky	Branching is affordable and valuable
Tool use or environment feedback matters	External state can guide expansion
Auditability matters	Branches, scores, and failures can be inspected
The base model is weaker than the task demands	External scaffolding compensates for weak internal planning

Explicit ToT is less compelling when:

Condition	Why it weakens ToT
The task is easy for the base model	Extra branching wastes compute
The evaluator is weaker than the generator	Search confidently selects bad states
Outputs are subjective and hard to score	Voting may reward bland or judge-friendly text
Latency dominates	Multi-call search is too slow
A reasoning model already solves the task within budget	Internal deliberation may be cheaper and simpler
Thought granularity is unclear	The tree structure becomes arbitrary

This is not a philosophical question about whether models “really reason.” It is an engineering question about cost, reliability, and control. Given a fixed budget, does explicit search over visible states outperform hidden reasoning tokens, best-of-N, self-consistency, verifier reranking, or a larger model? The answer must be measured per domain.

Design heuristics for using ToT today

A practical ToT implementation should begin with the state representation, not the prompt. The designer should ask: what is a partial solution, what are legal next steps, and how can a bad partial solution be detected early?

A minimal design checklist looks like this:

Design choice	Good sign	Warning sign
Thought granularity	A thought changes the problem state meaningfully	Thoughts are vague paragraphs with no checkable consequence
Generator	Produces diverse legal candidates	Produces fluent but invalid variants
Evaluator	Correlates with final success	Rewards plausibility or verbosity
Search depth	Matches natural decomposition	Arbitrary number of “thinking steps”
Branching factor	Captures real alternatives	Explodes without improving coverage
Verifier	Cheap and reliable	Another unconstrained LLM judgment
Budget	Tuned against baselines	Chosen because “more thinking” sounds better

The strongest ToT systems usually combine LLM judgment with non-LLM verification. In arithmetic, use symbolic checking. In coding, run tests. In theorem proving, use a proof assistant. In planning, use environment feedback. LLM-only value functions are useful when no verifier exists, but they are also where ToT is most vulnerable to self-reinforcing errors.

Limitations and failure modes

The most serious limitation is evaluator reliability. If the value function cannot distinguish promising from doomed states, ToT can spend more compute while becoming more confidently wrong. Yao et al.’s crossword oracle analysis illustrates this: better states were sometimes present in the search tree but not selected by the evaluator. NeurIPS Proceedings

A second limitation is benchmark selectivity. Yao et al. were explicit that ToT is not needed for every NLP task. Many ordinary language tasks do not require exploration, backtracking, or planning over partial states; for those tasks, CoT or ordinary prompting may be sufficient and cheaper. NeurIPS Proceedings

A third limitation is prompt and task engineering burden. ToT requires task-specific prompts for proposing thoughts, evaluating states, formatting intermediate solutions, and deciding when a state is terminal. The official repository’s instructions for adding new tasks make this clear: new tasks require custom task files and prompts, not just a universal ToT toggle. GitHub

A fourth limitation is cost opacity in modern deployments. With API-based reasoning models, some search-like computation may happen inside the model as hidden reasoning tokens or parallel inference. External ToT adds visible orchestration but may duplicate internal deliberation. The right comparison is not ToT versus no reasoning; it is explicit ToT versus the best available internal-reasoning, sampling, and verifier-guided alternatives under the same latency and cost budget. OpenAI

The durable idea

The durable idea of Tree of Thoughts is that LLM inference can be treated as deliberate search over semantic states. This shifts attention away from prompts as static strings and toward inference as an algorithmic process.

The 2023 ToT paper was early evidence that search structure can matter as much as model scale or prompt wording. On Game of 24 and mini crosswords, its gains came from preserving alternatives, evaluating partial states, and backtracking. On creative writing, its gains were more ambiguous but still illustrated the value of planning before generation. NeurIPS Proceedings+2NeurIPS Proceedings+2

In the current reasoning-model era, ToT is no longer the frontier story by itself. OpenAI o1, Gemini Deep Think, verifier-guided inference, MCTS-style agents, and formal proof systems all point toward a broader principle: intelligence at inference time is increasingly a budgeted computation, not a single sample. ToT remains valuable when developers need visible, controllable, inspectable search over partial solutions. It becomes less compelling when the model’s internal reasoning, a verifier-guided sampler, or a simpler best-of-N strategy achieves the same reliability more cheaply.