By Murat Akdeniz in ai-engineering — 06 Jun 2026

Why Left-to-Right Reasoning Is Holding LLMs Back: Inside Tree-of-Thoughts

I used to think Chain-of-Thought prompting was the ceiling of what structured reasoning could look like in language models—just prompt the model to think step by step and hope it lands somewhere useful. But the more I pushed LLMs on genuinely hard problems, the more I realized that a single linear chain is a fragile bet: one bad step cascades into a completely wrong answer, and there's no mechanism to go back. That frustration led me to Tree-of-Thoughts, a framework that doesn't just ask the model to reason—it asks the model to search. Instead of committing to the first reasoning path that comes to mind, ToT decomposes a problem, generates multiple candidate thoughts at each step, estimates how promising each partial solution is, and then runs classic search algorithms like BFS or DFS over the resulting tree. It's a fundamentally different paradigm, and honestly, once you see it, you can't unsee how limited vanilla CoT really is.

The Problem with Left-to-Right Decoding

When I look at how standard large language models generate text, the constraint is baked into the architecture itself. Autoregressive decoding forces the model to commit to one token at a time, moving strictly from left to right with no option to revise. Once a token is emitted, it becomes part of the immutable context for every subsequent prediction. This creates a fundamental bottleneck: the model cannot engage in genuine strategic planning because it has no mechanism to explore alternative paths or backtrack from early mistakes. Every decision is final, and the probability mass collapses onto a single trajectory before the model can assess whether that trajectory is actually sound.

Chain-of-Thought prompting was a genuine step forward because it exposed intermediate reasoning steps in a readable linear sequence—question, then explicit reasoning steps, then the final answer. I can see why this helped; it gave the model more surface area to work through logic explicitly rather than compressing everything into a single output token. But when I examine the structure closely, it is still fundamentally a single trajectory. The model generates one chain, and if any intermediate step contains a logical flaw, that error propagates uncontrollably through the rest of the generation. There is no branching, no evaluation of competing hypotheses, and no way to recover once the reasoning veers off track.

Why Chain-of-Thought Still Falls Short

The limitation is architectural, not just superficial. In standard decoding and Chain-of-Thought alike, the model produces a single linear chain where each new step depends entirely on the cumulative history of previous steps. Because there is no lookahead mechanism and no ability to spawn parallel reasoning threads, the model doubles down on its initial choices rather than stress-testing them against alternatives. I find this particularly evident when analyzing complex multi-step problems: a single misinterpreted clause in step two can cascade into a completely wrong conclusion five sentences later, and the model never pauses to ask whether the foundation was solid.

The Error Propagation Trap

In practice, this means a single hallucinated number, a misapplied formula, or a subtly wrong assumption in an intermediate step can poison every subsequent calculation. Because the architecture lacks backtracking capabilities, the error compounds silently. I notice that the entire generation is only as strong as its weakest intermediate link, yet the system provides no safety net for catching those weak links mid-flight. The result is a brittle reasoning process that behaves more like a chain reaction than a deliberate search for the correct answer.

From Tokens to Thoughts

Tree-of-Thoughts, introduced by researchers at Princeton and DeepMind in late 2023 (Yao et al., 2023), directly attacks this limitation by changing the fundamental unit of inference. Instead of treating reasoning as a single linear chain of next-token predictions, the framework reframes it as search over a tree of semantic thoughts. Each node in this tree represents a coherent chunk of text corresponding to an intermediate reasoning step, rather than merely one additional token. This shift matters because it elevates the unit of reasoning from the token level to the thought level, giving the model structural freedom to explore multiple reasoning paths simultaneously.

I see this as a fundamental change in perspective. Rather than being locked into one irreversible sequence, the model can generate several candidate thoughts at each decision point, evaluate them against a value function, and deliberately choose which branches to pursue or prune. The tree structure introduces exactly what left-to-right decoding removed: the ability to reconsider, compare alternatives, and backtrack when a branch proves unpromising. It replaces the monolithic chain with a navigable search space where reasoning becomes an active process of exploration rather than a passive, irreversible slide toward the next token.

The Four-Step ToT Pipeline: Decompose, Generate, Estimate, Search

When I examine the Tree-of-Thoughts framework, the first thing that strikes me is how deliberately it replaces a single reasoning chain with a navigable search space. The entire pipeline is model-agnostic and operates without fine-tuning or specialized code—every stage is mediated purely through structured prompting. This design choice matters because it means ToT isn't a model architecture tweak; it's a reasoning protocol that can be layered on top of any capable LLM. The framework organizes reasoning into four concrete operations over a tree of text states, and each operation addresses a specific weakness of left-to-right generation.

Decomposition into Coherent Thought Units

The pipeline begins by breaking the problem into intermediate reasoning steps that map directly onto tree levels. I see this as the structural foundation: instead of asking the model to solve everything in one pass, the task is segmented so that each depth in the tree corresponds to a meaningful thought unit. These units are not arbitrary token boundaries; they are coherent chunks derived from the task's inherent structure. For example, in a multi-step math problem, one level might represent setting up equations, while the next handles substitution. This decomposition forces the model to externalize its reasoning incrementally, which makes later evaluation possible.

Candidate Generation and Branching Factor

Once the levels are defined, the LLM generates multiple thought candidates at each step. I find it notable that these candidates can be produced in two distinct modes:

Independent generation: Each candidate is sampled fresh, increasing diversity at the cost of coherence with prior context.
Sequential conditioning: Each new candidate is conditioned on previous thoughts, preserving logical threads while still expanding the frontier.

This flexibility is what creates the explicit branching factor at each tree depth. Rather than committing to a single hypothesis, the model explores a set of parallel continuations. A node at depth k might spawn three or four distinct partial solutions, each representing a different angle on the problem. The branching factor becomes a tunable parameter that directly trades off computational cost against search coverage.

Value Estimation as a Heuristic Engine

Here is where ToT gets particularly clever: the framework repurposes the LLM itself as a heuristic evaluator. Through value estimation prompts, the model scores how promising each partial solution is with respect to reaching a correct final answer. I view this as the framework's built-in guidance system. Instead of blindly expanding every branch to completion, the LLM acts as a critic that can flag dead ends early. These prompts follow a consistent pattern:

Assess the current progress toward the goal.
Identify logical errors or contradictions in the partial state.
Estimate the proximity to a correct final answer.

The output is essentially a learned heuristic distilled into a text-based scoring function.

Search Algorithms for Exploration and Pruning

With decomposition, generation, and evaluation in place, the final step applies classic search algorithms—specifically Breadth-First Search (BFS) or Depth-First Search (DFS)—to navigate the tree. What makes this work is that the LLM's value estimates directly inform the search policy. Promising branches get expanded; suboptimal ones are pruned. I notice that this pruning is aggressive: if a partial reasoning state scores poorly, the entire subtree beneath it can be discarded without further token generation. This dramatically reduces the effective search space compared to exhaustive enumeration.

The marriage of traditional AI search with modern LLM heuristics creates a hybrid that feels both old-school and novel. Every stage—from carving up the problem to cutting off bad branches—happens through carefully engineered prompts. There is no gradient descent, no parameter updates, and no dependency on a specific model architecture. When I look at the whole system, it reads like a software engineer's approach to reasoning: decompose the problem, generate alternatives, write a test heuristic, and run a search algorithm.

Candidate Generation: Branching Beyond Single Paths

I view candidate generation as the architectural pivot that separates Tree-of-Thoughts from every linear reasoning method. At each node, the model refuses to commit to a single next step. Instead, it deliberately spawns multiple thought candidates, creating an explicit branching factor at every depth of the tree. This transforms the inference process from a straight line into a combinatorial search space where each partial solution can evolve in several directions simultaneously.

How Candidates Are Produced

The mechanics here are more flexible than simple repeated sampling. In practice, candidates can be generated through two distinct modes:

Independent generation: Each candidate is produced in isolation, branching from the same parent state without knowledge of its siblings. This tends to maximize diversity across the frontier and prevents early candidates from biasing later ones.
Sequential conditioning: Later candidates are generated with explicit awareness of earlier thoughts in the same step. This can help systematically cover the solution space by nudging the model toward complementary rather than redundant proposals.

Both approaches share the same goal: proposing several plausible continuations from the current partial solution rather than gambling everything on one decoded sequence. The choice between them is itself a tunable decision that shapes the tree's topology.

The Game of 24 in Practice

The benchmark from Yao et al., 2023 crystallizes why this matters. In the Game of 24, the model must reach the target number using basic arithmetic operations on four given digits. At an intermediate step—say, after reducing the problem to three remaining numbers—the framework does not lock into one arithmetic operation. Instead, it might generate multiple operations as parallel branches: one path attempts addition first, another tries multiplication, a third explores subtraction. Each branch carries its own partial equation forward, and each represents a distinct hypothesis about how to reach 24. Watching this unfold, it becomes clear that ToT treats reasoning as a search problem, not a monologue. The model is effectively building a proof tree where every node is a candidate lemma.

Modularity and the Separation from CoT

What I find particularly powerful is that the generation strategy is treated as a modular component. The framework can swap prompting templates to elicit different styles of candidates depending on the problem structure. For mathematical puzzles, a template might demand strict formal operations; for open-ended planning, it might request diverse strategic angles. This adaptability means the branching mechanism isn't hard-coded—it is a configurable search primitive that engineers can tune without retraining the base model.

This branching is exactly what fundamentally separates ToT from Chain-of-Thought. CoT produces one chain: a single left-to-right sequence where the model must commit to the first idea that surfaces during decoding. If that initial thought is a dead end, the entire reasoning path collapses. ToT produces a search tree with multiple viable paths, enabling the model to explore alternatives before settling on a final answer. It replaces the pressure of "first thought, best thought" with the discipline of structured deliberation. When I compare the two side by side, the shift feels like moving from greedy single-threaded execution to a parallelized search algorithm—and that reframe changes everything about how we can scale complex reasoning without simply scaling model size.

Value Estimation: The LLM as Its Own Judge

The value estimation layer is what separates Tree-of-Thoughts from an unguided brute-force expansion of reasoning paths. Without it, the framework would simply generate candidate states in bulk and hope that one eventually lands on the correct answer. I see this step as the critical pivot from random exploration to a directed search: the LLM actively judges its own partial solutions before committing additional computation to them. This judgment happens through value estimation prompts—carefully engineered instructions that ask the model to score, rank, or classify the quality of an intermediate reasoning state relative to the final goal.

How the LLM Evaluates Its Own Candidates

The evaluation mechanism is surprisingly flexible and does not require training a separate reward model. Instead, the authors implement the value function directly through prompting. In practice, this can take two forms:

Classifier prompting: The model is asked to output a discrete label—such as "sure/likely/impossible"—that reflects how promising a given reasoning step appears. This turns the LLM into a zero-shot classifier for its own intermediate outputs.
Majority vote aggregation: When the model's confidence is noisy, the framework can run multiple independent evaluations of the same state and aggregate the results. I find this approach particularly robust because it smooths out sporadic hallucinations or inconsistent judgments without adding external infrastructure.

Both methods leverage the same underlying model that generated the candidates, which means the evaluator shares the same domain knowledge and linguistic patterns as the proposer.

The Internal Feedback Loop

What stands out to me is the dual role the LLM assumes. It is no longer just a next-token predictor spitting out the most probable continuation; it becomes both the generator of reasoning branches and the critic that scrutinizes them. This creates a tight internal feedback loop: the model proposes a partial solution, pauses to assess its own progress, and then uses that assessment to decide where to search next.

This loop enables two concrete operations that define deliberate problem-solving:

Guided exploration: Value estimates direct the search budget toward branches that scored highly, ensuring promising lines of reasoning receive deeper investigation.
Aggressive pruning: Suboptimal or nonsensical branches are discarded early, before the framework wastes inference tokens expanding dead-end paths.

When I look at standard chain-of-thought reasoning, the model simply narrates its way forward without ever questioning whether a particular step is worth pursuing. ToT's value estimation breaks that reflexive habit. By forcing the model to judge its own intermediate states, the architecture mimics how a human solver might backtrack from a flawed assumption rather than doubling down on it.

Modularity Across Domains

Another strength I notice is that these evaluation prompts are entirely modular. There is no universal value function hard-coded into the framework; instead, the scoring criteria live in the prompt itself. This means researchers can tailor the evaluation rubric per task—whether that involves mathematical correctness, logical consistency, or creative constraints—without retraining the base model. That adaptability is what makes the same underlying ToT structure work across everything from arithmetic puzzles to creative writing tasks, giving the framework genuine domain flexibility.

BFS, DFS, and the Search Algorithms Behind ToT

When I look at how Tree-of-Thoughts actually operates under the hood, it becomes clear that the framework treats reasoning as a search problem rather than a linear sequence. The LLM generates candidate thoughts that branch outward, and classic graph search algorithms navigate this structure. What makes this work is the LLM's own value estimates—heuristic scores that judge whether a partial reasoning path is worth pursuing. These estimates act as the guiding signal for pruning and expansion.

How BFS and DFS Handle the Tree

The paper specifies two primary search strategies, and the choice between them changes how the model allocates its inference budget.

Breadth-First Search (BFS): At every depth level, the algorithm expands all candidate thoughts before moving any single branch deeper. I see this as the right fit when the problem rewards early breadth—when you need to survey many competing hypotheses at once before committing to one. For example, if the first few reasoning steps contain a wide range of plausible directions, BFS prevents the model from overcommitting to a bad path too early.
Depth-First Search (DFS): Here, the algorithm picks the most promising branch and pursues it aggressively until it either solves the problem or hits a dead end, then backtracks. This is computationally efficient when one promising path is likely to lead straight to the solution. Instead of spreading queries thin across dozens of shallow branches, DFS goes deep on a single hypothesis.

Both strategies rely on the same pruning mechanism. The LLM assigns value estimates to intermediate thoughts, and the search algorithm uses these scores to discard suboptimal branches immediately. This keeps the query budget focused on high-probability reasoning chains rather than wasting tokens on obvious dead ends.

Extending the Search with MCTS

The modular design of ToT means you can swap in more sophisticated algorithms. Pairing the framework with Monte Carlo Tree Search (MCTS) introduces capabilities that BFS and DFS simply do not have.

Backpropagation: Unlike static depth-first or breadth-first passes, MCTS revisits earlier decisions after deeper exploration. If a branch looked mediocre at depth two but reveals strong downstream potential at depth five, that information flows back up to update the value of the original choice.
Simulated rollouts: MCTS runs lightweight future simulations to estimate which branches deserve expansion. Rather than fully generating every possible continuation, it samples rollouts to approximate the expected reward of a path.
External reward signals: While basic ToT relies on the model's self-assessment, MCTS can incorporate external validators or reward models. This breaks the circular dependency of an LLM grading its own homework.

Inside MCTS, the Upper Confidence Bound (UCB) rule handles the exploration-exploitation trade-off. It mathematically balances doubling down on high-value branches against probing lesser-known alternatives that might outperform the current leader.

The Compute Trade-Off

What strikes me most is that this entire architecture is modular—the search algorithm is a swappable component tuned to the problem structure. Need broad coverage? Use BFS. Need targeted depth? Use DFS. Need adaptive, self-correcting search? Plug in MCTS. However, this flexibility comes at a clear cost. Every branch expansion, every value estimate, and every MCTS rollout requires additional LLM queries. Compared to zero-shot or chain-of-thought prompting, ToT's tree exploration is compute-heavy. The gains in reasoning accuracy are real, but they scale directly with inference budget, and that is a trade-off engineers have to measure carefully.

Backtracking, Look-Ahead, and Global Choice-Making

Standard autoregressive generation forces a model to commit to every token sequentially; once a token is emitted, the rest of the sequence must build on top of it, making early errors nearly impossible to recover from. Tree-of-Thoughts (ToT) breaks this lock by representing reasoning as an explicit search tree over textual states rather than a single linear chain. When I examine this architecture, I see that a solitary decoding path becomes a branching structure where multiple reasoning trajectories can coexist at the same time. This structural shift alone enables three capabilities—look-ahead, backtracking, and global choice-making—that are fundamentally incompatible with strict left-to-right generation.

Look-Ahead: Assessing Downstream Paths Without Full Generation

Evaluating alternatives before commitment: Because the framework maintains multiple candidate states as discrete nodes in a tree, the model can inspect future reasoning directions without fully generating every intermediate token. I find this particularly powerful because it replaces blind autoregressive commitment with informed selection; the system can score partial trajectories based on their estimated promise.
Assessing what lies downstream: In standard generation, the model must literally write out a sequence to know where it leads. ToT decouples evaluation from full generation, allowing the model to judge what lies downstream simply by examining textual state nodes. It is similar to reading trail markers before choosing a route rather than bushwhacking through every possible path.

Backtracking: Recovering from Dead Ends

Returning to earlier branches: When a standard LLM produces a flawed intermediate step, the error tends to cascade because the model has no native mechanism to undo its previous output. ToT introduces explicit backtracking: when progress stalls or a branch proves unpromising, the system can return to earlier decision points and explore different reasoning directions.
Human-like problem recovery: When I look at this mechanism, it clearly mirrors how humans approach complex problems—we consider options, pursue one, hit a wall, and then go back to try another route. The tree structure makes this return operation explicit and algorithmic rather than hoping the model can implicitly recover through more generation.

Global Choice-Making: Strategy Switches Across the Entire Tree

Escaping local next-token traps: Standard autoregressive models optimize strictly for the immediate next word given the prefix, with no ability to reconsider earlier strategic choices. ToT enables a broader view: when local decisions are insufficient, the model can abandon an entire subtree and pivot to a completely different reasoning approach.
Selecting globally distributed alternatives: Instead of being trapped in a single chain, the system can switch strategies entirely by selecting from among globally distributed alternatives across the tree. I see this as the most significant architectural departure because the decision is made globally, not greedily, allowing the model to prefer a stronger candidate on a distant branch over a weaker local continuation.

These three mechanisms collectively position ToT as a practical bridge between symbolic planning and modern LLMs. I notice that by integrating ideas from classical search with neural text generation, the framework makes reasoning more deliberate and interpretable. The entire process remains language-based throughout; every node in the tree is a human-readable string, so deliberation steps are transparent rather than buried in latent representations. This interpretability matters because it allows us to audit not just the final answer, but the exact path taken through the search space. For engineering teams building systems that require verifiable reasoning, that visibility is a significant practical advantage over black-box autoregressive chains.

Chain-of-Thought as a Special Case of ToT

I see Chain-of-Thought prompting as essentially a stripped-down Tree-of-Thoughts where the tree never actually forks. In standard CoT, the model walks a straight line: question, then a sequence of reasoning steps, then the final answer. There is no divergence, no exploration of alternative routes, and certainly no mechanism for backtracking if a step turns out to be weak. When I map this onto the ToT framework, it becomes clear that CoT is simply a configuration where the branching factor is locked to one at every depth level. Because only a single candidate exists at each node, the value estimation function either disappears entirely or trivially rubber-stamps the lone option. The search algorithm, rather than exploring a space, collapses into a deterministic, single-path traversal from root to leaf.

How the Architectures Stack Up

When I compare the three approaches side by side, the hierarchy becomes obvious:

CoT: One linear chain. No branching, no mid-generation evaluation, and no recovery from bad steps. The search space is a single path.
CoT-Self-Consistency: Multiple independent linear chains generated in parallel. Evaluation happens only after every chain is complete, selecting the majority or highest-confidence final answer. Weak paths survive until the very end.
ToT: Active branching at each reasoning step with proactive value estimation. Partial solutions are scored before they are fully developed, and low-quality branches are pruned immediately.

CoT-Self-Consistency moves slightly closer to the ToT philosophy, but it still misses the core architectural shift. Because evaluation happens post-hoc, only after every chain is fully materialized, weak reasoning paths are allowed to grow to completion before being discarded. This is computationally expensive and structurally blind. The model never pauses at an intermediate step to ask whether the current partial solution is worth expanding, which means errors and hallucinations can compound freely within each chain until the final selection stage.

Tree-of-Thoughts generalizes both approaches by treating reasoning as an active search process rather than a passive generation task. In ToT, I can branch at every step, spawning multiple candidate continuations from any given node. The framework introduces an evaluator that interrogates partial solutions before they are fully developed. If a branch scores poorly, it gets pruned early, saving inference budget and preventing error accumulation. This is a fundamental departure from CoT and CoT-SC, where the model either commits to a single path or runs multiple isolated paths to completion without intermediate quality control.

What strikes me most about the ToT framework is its modularity. The initial problem decomposition, the choice of search algorithm—whether breadth-first, depth-first, or beam search—and the design of the evaluation prompts all function as independent, swappable components. Because these pieces operate independently, I can tune the branching factor or swap the evaluator without rewriting the entire pipeline. CoT does not compete with this architecture; it simply represents the simplest possible configuration, where branching, evaluation, and search are all minimized to their most trivial forms. Viewing CoT as a degenerate case of a broader search-based paradigm makes the relationship explicit: linear reasoning is not an alternative to tree-based reasoning, but rather its most constrained baseline, nested inside a more expressive system.

Recursive Prompting and the Modular Design of ToT

When I look at how Tree-of-Thoughts actually operates under the hood, what stands out immediately is that it treats the LLM not as a single-shot oracle but as a recursive engine locked inside a tight control loop. The model generates candidate thoughts, evaluates their promise, searches across the expanding tree, and then loops back to do it again. This means the same model wears two hats simultaneously: it acts as the proposer that spawns intermediate states and as the evaluator that judges whether those states are worth pursuing. That iterative cycle is a structural break from one-shot prompting, where the model emits an answer in a single forward pass and leaves you to discover hallucinations only at the very end. In ToT, the reasoning is deliberately staged, and every intermediate node is subject to scrutiny before the search commits to a path.

The Four Levers of Modularity

What makes ToT genuinely flexible, in my view, is that the framework is built from four independent dials that you can tune to match the problem at hand. None of them are hard-coded:

Thought size: This controls the granularity of each intermediate step. You might want atomic single-token decisions for a math proof, or chunky high-level planning stages for a creative writing task. The granularity directly affects branching factor and search depth.
Generation strategy: This defines how the model produces candidate next-steps. Whether you ask for diverse sampling or targeted refinement changes how aggressively the tree branches and how much overlap exists between sibling nodes.
Evaluation prompts: These determine how partial states are scored. The prompt design here directly shapes the model's ability to judge its own unfinished reasoning, effectively turning the critic and the actor into the same underlying weights.
Search algorithm: You can plug in BFS, DFS, or even MCTS depending on whether you need exhaustive coverage, selective depth, or stochastic exploration with backpropagated value estimates.

Because these components are decoupled, ToT is not a rigid recipe you copy-paste from a paper. It is an adaptable scaffolding that you reshape per task, and that adaptability is what separates it from monolithic reasoning pipelines.

Prompt Engineering as Control Logic

I find the most elegant part of this architecture to be where the control logic lives. The entire multi-step loop—decomposition, branching, evaluation, and search—is implemented purely through prompting. The LLM is orchestrated to generate a sub-question, fetch or compute evidence, fold that evidence back into the context window, and then continue planning. External information access becomes a first-class citizen of the reasoning algorithm rather than an after-the-fact retrieval patch you bolt on after the model finishes.

Compared to standard one-shot prompting, this structure exposes failure points earlier. When a candidate thought scores poorly during evaluation, you catch the derailment mid-flight instead of discovering it buried inside a finished paragraph. It also supports deliberate exploration: the model can backtrack from dead-end branches because the tree maintains explicit state, something a linear chain-of-thought simply cannot do without regenerating everything from scratch.

The fact that all of this control logic—decomposition, branching, evaluation, and search—lives entirely inside the prompt layer means the approach is model-agnostic. You do not need fine-tuning, adapter layers, or architectural surgery. You just need a capable base model and carefully structured prompts that force it to reason about its own reasoning. That separation of control from weights is, to me, the strongest signal that recursive prompting is a genuine structural shift rather than just a clever prompt trick.