PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought

Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, Louis Castricato, Jan-Philipp Franken, Nick Haber, Chelsea Finn

arXiv:2501.04682v1cs.AIcs.CL

TL;DR

Traditional CoT does not fully represent the reasoning underlying a solution, and whether LLMs can implement complex search strategies autoregressively remains an open question. Meta-CoT models reasoning through search, verification, and iterative refinement; observed behaviors and experiments support the hypothesis that internal search matters for complex-task performance.

  • Problem

    Traditional CoT does not fully represent reasoning’s underlying data-generating process, and whether LLMs can autoregressively implement complex search strategies remains open.

    Source: §4.2.4 · p. 20, §9 · p. 51

  • Method

    Meta-CoT frames reasoning through search, verification, and iterative refinement, and proposes a concrete training pipeline for developing these capabilities.

    Source: §9 · p. 51

  • Results

    As maze complexity rises, models’ performance falls off relative to models that explicitly search, while observed behaviors and exploration/backtracking experiments support internal search’s importance for complex tasks.

    Source: §4.2.1 · p. 18, §9 · p. 51

  • Takeaways & Limitations

    The authors regard Meta-CoT as a promising path toward more robust and generalizable reasoning in LLMs.

    Source: §9 · p. 51

    Public evaluation of search scaling on realistically challenging LLM reasoning tasks remains limited, and research is constrained by a lack of data and scalable infrastructure.

    Source: §8.3.3 · p. 49

Abstract

from arXiv · show

We propose a novel framework, Meta Chain-of-Thought (Meta-CoT), which extends traditional Chain-of-Thought (CoT) by explicitly modeling the underlying reasoning required to arrive at a particular CoT. We present empirical evidence from state-of-the-art models exhibiting behaviors consistent with in-context search, and explore methods for producing Meta-CoT via process supervision, synthetic data generation, and search algorithms. Finally, we outline a concrete pipeline for training a model to produce Meta-CoTs, incorporating instruction tuning with linearized search traces and reinforcement learning post-training. Finally, we discuss open research questions, including scaling laws, verifier roles, and the potential for discovering novel reasoning algorithms. This work provides a theoretical and practical roadmap to enable Meta-CoT in LLMs, paving the way for more powerful and human-like reasoning in artificial intelligence.

1. Introduction

The introduction argues that complex reasoning requires more than a final Chain-of-Thought trace: Meta-CoT explicitly models the latent exploration and verification behind it. It presents a research roadmap spanning theory, evidence of search-like behavior, training methods, and open questions.

1.1. Motivation

  • The introduction describes language models as learning implicit meaning and reasoning in their activations to predict the next token, under the principle “compression is intelligence”.It contrasts this account with the view that sequence-to-sequence models merely fit correlations between sequential words.Source: §1.1 · p. 4
  • Even deterministic math can challenge powerful LLMs, while step-by-step CoT can improve their capabilities; current models still reliably solve only problems of limited complexity.The paper focuses on the complexity hypothesis: CoT can allocate arbitrarily large amounts of compute in theory and enable new representational complexity, but practical reliability remains limited (Merrill & Sabharwal, 2023).Source: §1.1 · pp. 4–5

1.2. Outline

  • Meta-CoT extends CoT by modeling the latent, nonlinear process of exploration and verification that the authors argue is needed for advanced reasoning.The paper frames this as a response to the limitations of current LLMs on complex reasoning tasks.Source: §1.2 · p. 6
  • o1 and DeepSeek-R1 exhibit behaviors consistent with internalized in-context search, and the paper explores process supervision and search-generated training data.The proposed data-generation methods include Monte Carlo Tree Search (MCTS) and A*.Source: §1.2 · p. 6
  • The proposed end-to-end pipeline combines instruction tuning on linearized search traces with reinforcement learning post-training, while raising questions about scaling, verifiers, and meta-RL.The paper also presents Big MATH, an effort to aggregate over 1,000,000 high-quality, verifiable math problems.Source: §1.2 · p. 6

2. Meta Chain-Of-Thought

Meta-CoT models complex problem solving as latent reasoning that generates a solution jointly, rather than as a left-to-right sequence of solution steps. The section argues that this missing process helps explain why classical CoT struggles on advanced reasoning tasks, while o1’s performance and token patterns are consistent with more extensive inference-time reasoning.

2.1. Deriving The Meta-CoT Process

  • Meta-CoT treats complex solutions as jointly generated from latent thoughts, unlike simple problems whose solution steps can follow a learnable left-to-right process.The authors define the latent process q → z1 → . . . → zK as Meta-CoT, conditioning the joint answer and CoT on that process.Source: §2.1 · pp. 7–8

2.2. Why Does (Classical) CoT Fail?

  • o1 significantly outperforms standard reasoning models across HARP difficulty levels, with a widening performance gap and more generated tokens on harder problems.The authors hypothesize that these longer outputs approximate more extensive Meta-CoT; on level 1 problems, o1 generates a comparable number of tokens to human-written solutions.Source: Figure 1 · p. 8, §2.2 · p. 9
  • Classical CoT can fail on advanced tasks because training corpora contain solutions but generally omit the latent process that derives them, leaving hard solution-level distributions highly complex.The windmill solution illustrates the mismatch: deriving it requires geometric exploration and inductive reasoning, and its utility is difficult to see before analyzing the line’s dynamics.Source: §2.1 · p. 7, §2.2 · p. 9
  • The distinction is not absolute: constant-depth transformers may internalize part of the Meta-CoT process, as reflected in gradually increasing reasoning lengths from Levels 2–6.This qualification accompanies the authors’ interpretation of o1’s increasing inference compute on more complex problems.Source: §2.2 · p. 9

3. Towards Deliberate Reasoning With Language Models - Search

The section argues that advanced reasoning depends on search processes underrepresented in text corpora, and presents evidence that inference-time compute, verification, and training can improve or internalize that search. It frames search as guided exploration of reasoning paths, while noting that evidence for scaling in LLMs remains limited.

3. Towards Deliberate Reasoning With Language Models - Search

  • The section’s premise is that generation can be harder than verification, so solutions in text corpora may reflect extended search processes that the corpora do not represent.The authors frame the generator-verifier gap as an assumption supported by empirical evidence from prior literature, not as a result they prove here.Source: §3 · pp. 9–10

3.1. Inference-Time Compute: Search

  • Pass@64 reaches close to 85% accuracy for the final 8B checkpoint, while majority@64 beats greedy decoding using only 15% of its training compute.The experiments show gains from increased sampling and training, including majority voting without a ground-truth verifier; the authors call the study preliminary and say further work is needed.Source: Figure 2 · p. 10, §3.1 · p. 11

3.2. Inference-Time Compute: Verification

  • Verifier-guided best-of-N improves with more online sampling, and explicitly trained verifiers outperform self-consistency or majority voting.The framework scores candidate reasoning processes with vθ(q, S) and ranks solutions; the cited verifier results are from Zhang et al. (2024a).Source: Figure 3 · p. 11, §3.2 · p. 12

3.3. From Best-of-N To General Search

  • Tree-search methods can terminate unproductive or incorrect attempts and reset to promising previously visited states; they formulate reasoning as tree search guided by an intermediate heuristic, potentially improving efficiency over parallel sampling.On the Game of 24 toy reasoning task, tree-structured search showed nearly 4 times increased efficiency in inference budget compared with parallel sampling.Source: §3.3 · p. 13

3.4. Is Search (Inference Time Compute) A Fundamental Capability Shift?

  • Post-training can make search learnable: zero-shot performance can match or exceed best-of-N, while MCTS fine-tuning improves policy performance and gains further from inference-time search at both iterations.The authors describe limited but promising evidence for train/test-compute scaling in LLMs; Feng et al. (2024) do not ablate model size, data scaling, or inference-time search scaling.Source: §3.4 · pp. 14–15, Table 1 · p. 14

4. Towards Meta-CoT Reasoning · 4.1. Bootstrapping Meta-CoT · 4.2. Empirical Examples Of Internalizing Search

The section proposes training models to internalize search as Meta-CoT, then surveys evidence that models can learn search, exploration, and backtracking, while noting that complex search in language domains remains an open question.

4. Towards Meta-CoT Reasoning

  • Internalizing search is proposed to improve efficiency by reusing visited reasoning in context and to enable RL-driven discovery of novel approaches; search components may be verbalized or modeled implicitly.The authors argue this could extend problem-solving beyond symbolic tree-search approaches, while explicit certainty or progress estimates may add capacity or aid interpretability (Gandhi et al., 2024).Source: §4 · p. 15, §4.2 · p. 16

4.1.1. Self-Taught Reasoner

4.1.2. Meta-STaR

  • Meta-STaR uses a base policy and search to generate verified search traces, then trains an autoregressive model to reproduce the linearized search sequentially as Meta-CoT.A solution may be verified through formal methods such as AlphaProof or other outcome checks.Source: §4.1.2 · p. 16

4.2.1. Small-Scale Empirical Results on Internalizing Search

  • Search-augmented maze models improve with training data and model size, and outperform direct-plan models increasingly as maze complexity grows.At low complexity, direct plans perform comparably to smaller search-augmented models; the gap widens as maze size increases.Source: Figure 8 · p. 17, §4.2.1 · p. 18

4.2.2. In-context Exploration For LLMs

  • In-context exploration raises pass@1 by nearly 6–7% from zero-shot to saturation, and autoregressive generation scales better than independent parallel sampling.The results indicate that models learn some degree of in-context exploration and self-correction.Source: §4.2.2 · p. 19, Figure 10 · p. 20

4.2.3. Using variable Compute

  • Variable-compute training elicits more solutions on harder problems—averaging 2.45 for Level 1 and 5.84 for Level 5—while the best run improves 2% over its LLaMa 3.1 8B Instruct initialization.The authors attribute the modest gain to mismatch between RL post-training, off-policy SFT, and limited MATH training data; regular SFT on the RL-post-trained model worsened performance.Source: §4.2.3 · p. 20

4.2.4. Backtracking in LLMs

  • Training with 50% incorrect steps raises hard-math accuracy from 78% to 94% versus correct-only training, though this result is limited to 124M-parameter models.The approach teaches models to mark an incorrect step with [BACK] and then correct it.Source: §4.2.4 · p. 21
  • For safety-oriented backtracking, SFT improves safety over safe-answer-only training, but the effect is weak until downstream RL substantially strengthens it.Zhang et al. (2024b) combine Meta-CoT and CoT objectives, mask unsafe completions, and reset to the initial state rather than correcting one logical step.Source: §4.2.4 · p. 21

4.3. Synthetic Meta-CoT Via Search

This section explores generating synthetic Meta-CoT through full-scale in-context tree search, producing linearized search traces that capture how a solution is found. It compares MCTS with a more efficient best-first A* variant.

4.3. Synthetic Meta-CoT Via Search

  • The section explores synthetic training data for realistic Meta-CoT involving full-scale in-context tree search, using MCTS and A* variants.Both search algorithms require heuristic state estimates, supplied by pure Monte-Carlo rollouts following Silver et al. (2018).Source: §4.3 · pp. 21–22
  • On the OpenAI (2024) math problem, Gemini 1.5 Pro (Reid et al., 2024) achieved Pass@128 of 6.25% (8/128 correct), the only frontier model without advanced reasoning then showing non-zero performance.For state values, the experiments sampled 128 completions from each partial solution and averaged success rates under ground-truth outcome supervision.Source: §4.3 · pp. 21–22

4.3.1. Monte-Carlo Tree Search

  • MCTS produced excessive backtracking and repetitions, even from high-value states, which the authors attribute to its exploration bonus.A single tree could consume up to 20 million inference tokens (about $100) and take up to half an hour; the search trace linearizes actions, with value estimates following reasoning steps.Source: §4.3.1 · p. 22, Figure 12 · p. 22

4.3.2. A* search

  • A* best-first search found a ground-truth-verified solution with more consistent reasoning flow and less backtracking concentrated around key steps.It prioritizes the highest-valued partial solution from a max-priority frontier, expands candidate next steps up to a depth limit, and continues until the ground-truth verifier confirms a solution.Source: §4.3.2 · pp. 22–23

4.4. Do Advanced Reasoning Systems Implement In-Context Search?

Advanced reasoning traces exhibit behaviors consistent with search, but the authors do not claim these models execute tree search at test time. They hypothesize that search examples may have appeared in training and leave the effects of explicit verification unresolved.

Do Advanced Reasoning Systems Implement In-Context Search?

  • o1 exhibits abrupt logical transitions, semantic backtracking to the same points, and repeated reasoning steps, resembling synthetic MCTS-style traces (OpenAI, 2024).The authors suggest branch changes may explain abrupt flow and repetitive logic, but do not claim o1 performs tree search at test time.Source: §4.4 · p. 23
  • The authors hypothesize that search examples were present in models’ training data, possibly during initialization, because outputs can resemble search traces; they do not establish test-time tree search.They point to synthetic traces generated with MCTS-like search as a comparison for observed behaviors.Source: §4.4 · p. 23
  • DeepSeek-R1 combines self-evaluation with smoother reasoning and fewer abrupt backtracks than o1; self-criticism or a generative verifier could explain its evaluation steps.The authors note that explicit CoT verification might expand computational capacity and improve self-verification, but this remains an open empirical question.Source: §4.4 · pp. 23, 26, Figure 14 · p. 24
  • Intermediate traces from o1, DeepSeek-R1, and Gemini 2.0 show inconsistent flow and repetition; DeepSeek-R1 and Gemini also verify outputs, with Gemini making an incorrect verification (OpenAI, 2024; DeepSeek, 2024).The traces also show backtracking, where a model returns from a bad state to a previously visited step.Source: Figure 14 · p. 24
  • Gemini 2.0 and Qwen QwQ appear to favor revision: they attempt full solutions, then may restart or generate alternative solutions rather than branch at individual steps (Qu et al., 2024; Anonymous, 2024; Kumar et al., 2024; Team, 2024; Chen et al., 2024).In the example, Gemini solves the problem before restarting and generating a new solution from scratch.Source: §4.4 · p. 26

5. Process Supervision

Process supervision uses learned process reward models (PRMs) to evaluate partial reasoning chains, enabling search to backtrack without repeatedly performing expensive outcome-based rollouts. The section describes PRM training and scaling, and explains why methods requiring verifiable answers do not cover open-ended problems.

5. Process Supervision

  • Process supervision lets search backtrack from suboptimal paths, while amortizing evaluation into a learned model avoids outcome-based MCTS rollouts that can require up to 20 million inference tokens per example.Those rollouts require ground-truth answers and are computationally inefficient.Source: §5 · p. 26

5.1. Learning Process Reward Models

  • Parameterized PRMs score a question and partial solution with a scalar value; labels can come from costly human step evaluations or Monte Carlo value estimates based on outcome verification.The latter approach amortizes Monte Carlo state-value estimates into a learned function, but requires problems with ground-truth answers.Source: §5.1 · p. 26

5.2. PRM Quality And Its Effect On Search

  • As training data grows, PRM prediction error falls and full-solution verification and search guidance improve, though trained PRMs remain far below an oracle PRM.The experiment uses 16 Monte Carlo rollouts per step and 7,086 training questions, with evaluation on MATH-500. Setlur et al. (2024b) report log-linear test-time search accuracy gains with training data, while Anonymous (2024) find oracle-verifier search orders of magnitude more efficient than search with noisy learned PRMs.Source: §5.2 · p. 27, Figure 16 · p. 28

5.3. Verifiable Versus Open-Ended Problems

  • Monte Carlo-trained value functions require verifiable solutions, limiting their reach in proof and scientific-derivation problems; human evaluations of valid reasoning steps could support more general verification.The passage notes that automated proof assistance is largely limited to mathematics and does not transfer to science or broader problem-solving.Source: §5.3 · p. 27

6. Meta Reinforcement Learning - Learning How To Think

The section frames reasoning as meta-RL over tasks with unknown prompt-specific rewards, where policies must learn to explore and adapt in context. Evidence suggests in-context search can improve efficiency, but whether it yields robust new reasoning capabilities remains uncertain.

6. Meta Reinforcement Learning - Learning How To Think

  • The unknown prompt-specific reward makes reasoning a partially observable problem, motivating meta-RL policies that adapt across tasks; training on reference solutions can also create train-test shift.Ghosh et al. (2021) show standard RL can have arbitrarily low test-time return relative to Bayes-optimal behavior. When reference data lacks coverage, the shift motivates on-policy RL that trains the sampling process for task adaptation.Source: §6 · pp. 28–30

6.1. Meta-RL In Small Domains

  • In Countdown, RL post-training improves SoS accuracy and efficiency over SFT, but does not outperform the symbolic search system that generated its instruction data.The paper leaves open whether this reflects limits of the environment, pretraining data, or scaling.Source: §6.1 · pp. 31–32, Figure 20 · p. 31

6.2. Meta-RL In Language Model Reasoning

  • RLEF uses exploration episodes with compiler feedback followed by a final hidden-test episode, and RL enables iterative code refinement even with unrelated feedback, while SFT alone does not induce in-context exploration.Performance continues to improve with additional revisions under random feedback, though it remains below performance with ground-truth feedback.Source: §6.2 · pp. 32–33, Figure 22 · p. 33

6.3. Efficiency Or Super-Intelligence?

  • In-context RL can improve search efficiency, but evidence for novel capabilities remains limited: at higher token budgets majority voting can outperform in-context search, and RL gains may saturate.The cited code-generation results show only about a 5% advantage for the 70B RL-tuned model over the base model at k=100, while the base model continues scaling.Source: §6.3 · p. 34, Figure 23 · p. 34
  • Current evidence supports efficiency gains from in-context search, but provides only weak evidence that it discovers methods solving problems beyond existing search capabilities.In Countdown, RL-tuned SoS solves up to 4% of problems that symbolic strategies do not, versus about 1% for base SFT; code results at 70B and k=100 are only about 5% above the base model. At low token limits, in-context search scales more efficiently than majority voting, while majority voting can outperform it at higher limits.Source: §6.3 · pp. 34–35, Figure 23 · p. 34

6.4. Can System 2 Reasoning Emerge From Pure RL?

  • Standard RL does not ensure System 2 exploration: one-turn training performs worse than the base model at higher inference budgets, while three-turn training improves scaling beyond its training budget.The section links greedy collapse in RL2 to inadequate in-context exploration and describes E-RL2 training before RL2 annealing as a remedy.Source: Figure 22 · p. 33, §6.4 · p. 35

6.4.1. Inducing Meta-Reasoning In LLMs

  • Explicit meta-cognitive prompts increase expressed regret—from under 0.5% with baseline or CoT to 25.67% for Llama 3.1 70B with Think & Verify—but expression varies by model.Llama models reach 15–25% under Think & Verify, whereas GPT-4o and Claude 3.5 Sonnet show 1–4%.Source: §6.4.1 · p. 38
  • More elaborate prompting increases token use without reliably improving accuracy, and self-correction or backtracking is associated with more often incorrect final answers.Think & Verify can require more than double the token budget for equivalent accuracy on the easiest problems; the authors caution that models may imitate reasoning style rather than improve reasoning substance.Source: Figure 25 · p. 37, §6.4.1 · p. 39

7. Putting It All Together - A Pipeline for System 2 Reasoning

The proposed System 2 pipeline pairs instruction tuning with RL post-training: synthetic search traces teach Meta-CoT behaviors, while RL is expected to be a key contributor to performance. The section also outlines verifier-free latent-reasoning training and discount-based controls for balancing inference compute against accuracy.

7.1. Instruction Tuning

  • The pipeline uses instruction tuning on synthetic in-context search traces to teach Meta-CoT backtracking and branching, followed by verifiable solutions.The authors hypothesize that the exact instruction-tuning objective matters less than RL post-training, an open empirical question.Source: §7.1 · p. 39

7.2. Post-Training With RL

  • RL post-training uses verifiable solution rewards and a reference-policy constraint to stabilize long-horizon Meta-CoT, though credit assignment remains challenging.Proposed options include MCTS-based branching, step-level DPO with tree-search exploration, and branching on-policy methods; DPO may scale better with off-policy data.Source: §7.2 · p. 40

7.2.1. Q* or q-STaR?

  • q-STaR treats Meta-CoT as latent reasoning trained from question-solution pairs, removing final-answer verification while still needing verifiable outcomes to bootstrap its traces.The summarization model receives supervised feedback rather than on-policy rewards, which may be suboptimal; Qu et al. (2024) report lower loss with in-context exploration than direct answer prediction, while q-STaR optimizes the full latent process rather than a fixed token-level budget as in Quiet-STaR.Source: §7.2.1 · p. 42

7.2.2. Discount Rates

  • Discounting can trade inference compute against certainty, producing a hump-shaped accuracy frontier, while prompts conditioned on discount rates may control search behavior at inference.Schultz et al. (2024) found that models trained on MCTS traces with different parameters could reproduce qualitative behaviors when conditioned on those parameters.Source: Figure 27 · p. 43, §7.2.2 · p. 43

8. Going Forward

The section identifies data, infrastructure, and algorithmic exploration as bottlenecks to open reasoning research, then outlines dataset construction and research directions for improving verification, search, and tool use.

8.1. The "Big MATH" Project

  • Big MATH aims to aggregate over 1,000,000 verifiable math problems because GSM8k is largely saturated and MATH contains only 12,000 problems.The authors require uniquely verifiable answers, open-ended formulations, and closed-form solutions, while noting that existing datasets have too few challenging examples and may contain incorrect labels. In NuminaMath, roughly 860,000 entries include more than 42,500 duplicates; 89.7% have one boxed solution, 2.6% none, and 7.7% multiple.Source: §8.1 · pp. 44–45

8.1.1. Data Sourcing; 8.1.2. Data Filtering

  • The data pipeline combines established datasets and manually curated competition problems, then applies base and strict filters to yield training and Meta-RL data.Base filtering removes duplicates, non-English items, hyperlinks, and MATH500 test examples; strict filtering further removes multi-part, proof, multiple-choice, and yes/no or true/false problems, as well as near-duplicates.Source: §8.1.1 · p. 45, §8.1.2 · pp. 45–46

8.2. Infrastructure

  • Shared CUDA memory lets inference and training run asynchronously, improving throughput by 40% over three-step asynchronous training but constraining GPU parallelism and inference speed.The authors note that dedicated GPU clusters may achieve higher overall throughput when inference compute greatly exceeds training compute, as in MCTS.Source: §8.2 · p. 47, Figure 28 · p. 47

8.3.1. Open-Ended Verification And CoT Faithfulness

  • Current models can produce unfaithful or incomplete CoTs despite correct answers, and rewarding full reasoning on open-ended problems remains unresolved.The authors propose exploring generative verifiers that assess solutions using reference proofs, relevant materials, or tools, while noting that such work needs human-annotated open-ended problems.Source: §8.3.1 · p. 48

8.3.2. Process Guidance And The Verifier Gap

  • Process reward models still lag pure Monte Carlo methods in search efficiency, and a gap remains between verifier-guided best-of-N and oracle pass@N.The authors identify verifier scaling, CoT-based evaluation, and the learnability of complex search traces as open questions.Source: §8.3.2 · pp. 48–49, Figure 29 · p. 48

8.3.3. Scaling Laws For Reasoning And Search

  • Public research lacks thorough evaluations of joint policy and verifier scaling, search strategies, instruction-tuning versus RL, and whether in-context search improves the compute-accuracy curve.The authors note that realistic applications appear to require substantial on-policy RL, while evidence for emergent capabilities in current models remains weak.Source: §8.3.3 · p. 49

8.3.4. Meta-Search/Search 2

  • Meta-Search or Search2 could add classical search over an advanced reasoning model, potentially improving efficiency with a critic conditioned on prior solution attempts.In-context search is limited by context length and sequential generation; the cited multi-turn value-function approach shows improved scaling over regular value-function training, but its benefits remain an open empirical question.Source: §8.3.4 · p. 50, Figure 30 · p. 50

8.3.5. Reasoning with External Tools

  • At 20–23 samples, tool-integrated reasoning nearly doubles CoT accuracy despite using 25% as much training data, with both majority vote and Best-of-N outperforming CoT.The authors identify scaling search strategies, verifier performance, and the trade-offs between external tools and internal reasoning as priorities for further study.Source: Figure 31 · p. 50, §8.3.5 · p. 51

9. Conclusion

The conclusion presents Meta-CoT as a fuller account of LLM reasoning that incorporates search, verification, and iterative refinement, and frames its training pipeline as a promising but still unvalidated path.

9. Conclusion

  • Meta-CoT models reasoning with search, verification, and iterative refinement because traditional Chain-of-Thought does not fully represent the data-generative process on reasoning problems.The paper presents this framework as a way to understand and enhance LLM reasoning capabilities.Source: §9 · p. 51
  • Observed state-of-the-art behaviors and experiments on in-context exploration and backtracking support the hypothesis that internal search is crucial for complex-task performance.The authors propose a concrete training pipeline for enhanced Meta-CoT capabilities and call for future work to validate its efficacy.Source: §9 · p. 51
  • Open research questions include reasoning and search scaling laws, process supervision and verification, instruction tuning versus reinforcement learning, and whether meta-RL with Meta-CoT can discover novel search algorithms.The introduced “Big MATH” dataset aims to support research by providing a large-scale resource for training reasoning models.Source: §9 · p. 51

A. Prompting

The section investigates whether carefully designed prompting can induce reasoning resembling that of RL-based models, evaluating five instruction sets through token patterns, reasoning-like behavior, and math performance.

A. Prompting

  • The study asks whether carefully constructed prompting protocols can induce reasoning capabilities similar to RL-based models like O1.Source: A. Prompting · p. 62
  • Five increasingly sophisticated instruction sets are evaluated by token-generation patterns, reasoning-like behaviors, and their correlation with math problem-solving performance.Source: A. Prompting · p. 62
  • Complex instructions such as Think (3-shot) use substantially more input tokens, primarily due to demonstrations and explicit capability descriptions.Table 4 reports input token counts using OpenAI’s GPT-4o tokenizer, excluding problem tokens.Source: Table 4 · p. 62

B. Regret Analysis

The section reports that models and prompting strategies differ in self-correction and error acknowledgment, with detailed regret-expression statistics presented in Table 5.

B. Regret Analysis

  • Table 5 summarizes regret-expression statistics and shows variation in self-correction and error acknowledgment across model scales and prompting strategies.Source: B. Regret Analysis · p. 63

C. Different Instruction Tuning Objectives

The section outlines instruction-tuning alternatives for Meta-CoT, from standard supervised cloning to objectives and data strategies tailored to search traces.

C. Different Instruction Tuning Objectives

D. MCTS Details

The MCTS procedure selects a leaf, expands it when nonterminal, and backs up visit and value statistics along its path to the root.

Selection

  • Selection follows UCT, choosing child steps by maximizing U(S_t, s); UCT is preferred to PUCT because logical-step likelihoods may be skewed.The search starts at the question-initialized root and continues until a leaf is reached; UCT uses value, visit-count, and exploration terms.Source: D. MCTS Details · p. 64

Expansion

  • Expansion stops at terminal leaves; otherwise, it samples b actions from the policy, adds them as children, and initializes their values with a value function instead of simulation.The sampled actions are conditioned on the partial solution reached at the selected leaf.Source: D. MCTS Details · p. 64

Backup

  • Backup propagates statistics from the expanded node toward the root, incrementing each ancestor’s visit count and updating its value.Source: D. MCTS Details · p. 64

E. Chains-Of-Thought

This section uses synthetic chains-of-thought to show stepwise reasoning and final answers in model comparisons and a tool-augmented example.

E. Chains-Of-Thought

  • The section presents Gemini-1.5 A* and MCTS traces alongside DeepSeek-R1, DeepSeek-O1, and Gemini-2 traces on identical planning tasks.Each trace shows step-by-step internal reasoning and a final solution.Source: E. Chains-Of-Thought · p. 64
  • A separate example shows a smaller SFT-tuned Llama3.1-8B model using tool-augmented reasoning with dynamic programming on a challenging problem.Source: E. Chains-Of-Thought · p. 64
  • The jury-seating trace ultimately states that there are 1,024 ways to seat the jury.Earlier counts in the trace are explicitly rejected before this final answer.Source: E. Chains-Of-Thought · p. 70

Trace - Reasoning with External Tools

The trace works through the 12-seat jury problem using Python-based counting approaches and gives 1024 as its answer. It notes problems with the initial dynamic-programming setup and revises the initialization and calculation approach.

Trace - Reasoning with External Tools

  • The trace concludes that there are 1024 possible jury seating arrangements under the stated rules.Nikolai starts in the next seat clockwise; each later member sits in their designated seat or the next available seat.Source: Trace - Reasoning with External Tools · pp. 74, 76
  • The initial approach models the count with dynamic programming over available seats and Nikolai’s current seat, then acknowledges that its setup overlooked a crucial detail.The trace specifically identifies the initial state as an issue.Source: Trace - Reasoning with External Tools · pp. 72, 75
  • The revised approach emphasizes correctly initializing the base case and using an iterative calculation before returning the same answer.The trace describes the iterative approach as simplifying the calculation.Source: Trace - Reasoning with External Tools · pp. 75–76

O1 official math CoT

The leads describe solutions to an even-degree polynomial problem that derive the additional real solutions as ±1/n!. They use a degree argument and factorization of a polynomial with known roots to determine the remaining roots.

A* Generated Solution Trace

  • The A* generated solution concludes that the other real solutions are ±1/n!.It reaches this result by forcing the residual quadratic to be x^2 − 1/(n!)^2.Source: O1 official math CoT · pp. 79–80
  • The A* trace factors r(x) into the known roots ±1,…,±n and a monic quadratic, then uses r(0) = −1 to determine its constant term.The constant term is −1/(n!)^2; evenness of r forces the linear coefficient to zero.Source: O1 official math CoT · pp. 79–80

MCTS Generated Solution Trace

  • For q(x)=p(1/x)−x², q(k)=0 for every integer k with 1≤|k|≤n; the related polynomial r(x) has roots at those k and degree 2n+2.Source: O1 official math CoT · pp. 81–82

O1 official math CoT

  • The coefficient of x in r(x) is zero, so a_(2n-1) = 0.Source: O1 official math CoT · p. 86
  • The other real solutions are x = ±1/n!.The remaining factor is -x^2 + 1/(n!)^2 because n is even.Source: O1 official math CoT · pp. 88–89
  • The transformed degree-2n+2 polynomial has roots ±1, …, ±n and factors into those linear terms times a quadratic.Its known roots come from the given condition, and its degree leaves two additional roots.Source: O1 official math CoT · pp. 88, 89, 92
  • Matching coefficients gives A = -1, B = 0, and, since n is even, C = 1/(n!)^2.The coefficient of x^(2n+1) forces B = 0, while the constant term determines C.Source: O1 official math CoT · pp. 86, 87, 89
  • The reasoning transforms the equation into a degree-2n+2 polynomial with the known roots and two additional roots, then factors it to determine the remaining pair.A negative sign is required in the factorization to match the leading coefficient.Source: O1 official math CoT · p. 97
  • The constant term yields a = 1/(n!)², since the product 1 · 4 · 9 · … · n² equals (n!)² and n is even.This gives the additional roots ±√a = ±1/n!.Source: O1 official math CoT · pp. 97–99
  • The additional real solutions to p(x) = x² are x = ±1/n!, beyond the known solutions x = ±1, ±2, …, ±n.The derivation gives a = 1/(n!)², using that n is even.Source: O1 official math CoT · pp. 98–99

Related papers

Loading 2501.04682v1…