Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future

Motivation

• CoT prompting significantly boosts LLM reasoning, especially on complex tasks (Wei et al., 2022b; Cobbe et al., 2021; Geva et al., 2021).Source: §1 · p. 1

Background

• CoT breaks intricate problems into linked, manageable reasoning steps and exposes a trajectory that can aid users’ understanding of decisions and trust in answers.The paper illustrates this stepwise problem-solving in Figure 1.Source: §1 · p. 1, Figure 1 · p. 1

Survey scope

• The authors identify a lack of systematic review and define their broader survey scope as generalized chain-of-thought (XoT), the gradual stepwise unraveling of complex problems.Source: §1 · p. 1

Contributions

• The paper presents itself as the first comprehensive XoT survey, introducing a taxonomy, discussing frontiers and future research, and releasing resources publicly.The authors also identify challenges and aim to facilitate research in the community.Source: §1 · p. 2

Survey organization

• The survey proceeds from background and benchmarks to advanced methods, frontier research, challenges and future directions, and open questions.Source: §1 · p. 2

2.1 Background

• As pre-training scales, language models gain capabilities including in-context learning and CoT reasoning, while pre-training followed by prompting increasingly replaces fine-tuning (Brown et al., 2020; Scao et al., 2022; Touvron et al., 2023b; Zhao et al., 2023b; Wei et al., 2022a; Wei et al., 2022b; Qiu et al., 2020).Source: §2.1 · p. 2

2.2 Preliminary

• Few-shot standard prompting supplies instructions and question-answer demonstrations for direct answer prediction, whereas CoT prompting adds rationales and generates a reasoning trajectory before the answer.The preliminary defines question Q, prompt T, probabilistic language model pLM, and prediction A, following Qiao et al. (2023).Source: §2.2 · p. 2

2.3 Advantages of CoT Reasoning

• CoT reasoning breaks complex problems into steps, exposes interpretable reasoning traces, and enables fine-grained user interaction with model execution.The paper presents these benefits as supporting reasoning, transparency, and collaboration, including autonomous-agent development.Source: §2.3 · p. 2

Benchmarks

• The benchmark overview spans five reasoning categories: mathematical, commonsense, symbolic, logical, and multimodal.The section directs readers to Appendix B for benchmark details.Source: §3 · p. 2

Mathematical Reasoning

• Mathematical reasoning benchmarks commonly assess LLMs’ general reasoning ability (Patel et al., 2021; Cobbe et al., 2021; Hendrycks et al., 2021b; Mishra et al., 2022a).The paper frames mathematical reasoning as foundational to human intelligence and relevant to problem-solving, decision-making, and world comprehension.Source: §3 · p. 2

Commonsense Reasoning

• Commonsense reasoning benchmarks assess language models’ world-comprehension capacity (Talmor et al., 2019, 2021; Geva et al., 2021).The paper describes commonsense reasoning as essential to daily-life interaction and perception of the world.Source: §3 · p. 2

Symbolic Reasoning

• Symbolic reasoning benchmarks test language models’ ability to simulate atomic operations while disentangling semantics (Wei et al., 2022b; Srivastava et al., 2022; Suzgun et al., 2023).Source: §3 · p. 3

Logical Reasoning

• Logical reasoning benchmarks target capacities underlying rational thinking, robust problem-solving, and interpretable decision-making (Liu et al., 2020; Yu et al., 2020; Tafjord et al., 2021; Han et al., 2022).Source: §3 · p. 3

Multi-modal Reasoning

• Multimodal reasoning benchmarks concern integrating textual thought with sensory input, including visual scenes and sounds, for more comprehensive understanding (Zellers et al., 2019; Park et al., 2020; Xiao et al., 2021; Lu et al., 2022; Chen et al., 2023c).Source: §3 · p. 3

4.1 XoT Prompt Construction

• XoT prompt construction ranges from manual rationale annotation to automatic prompting and semi-automatic methods that combine human supervision with automated rationale generation and demonstration selection.Manual prompting can improve performance but costs labor and may transfer poorly; automatic prompting avoids annotation costs but can be unstable, while semi-automatic prompting balances performance and cost. Automatic methods include instructions and selection of generated demonstrations; semi-automatic methods bootstrap rationales or optimize demonstration choice.Source: §4.1.1 · p. 3, §4.1.2 · pp. 3–4, §4.1.4 · p. 4

4.2 XoT Topological Variants

• Chain, tree, and graph topologies expand reasoning beyond vanilla chains, enabling structured execution, broader exploration, or subproblem aggregation, but complex structures can limit generalization.Code and formal-logic chains decouple thought generation from execution, reducing inconsistency errors and enabling higher-level algorithmic reasoning (Gao et al., 2023; Chen et al., 2022a; Olausson et al., 2023; Pan et al., 2023; Ye et al., 2023a; Sel et al., 2023). Trees support exploration and backtracking (Long, 2023; Yao et al., 2023b), but require explicit decomposition and state transitions; graphs model aggregation and self-verification (Besta et al., 2023; Lei et al., 2023a), yet specialized decomposition can impair generalization.Source: §4.2 · pp. 4–5

4.3.1 Verify and Refine

• Verification and refinement target faithful reasoning errors through model feedback, logical checks, and external verification, although self-feedback may fail beyond a model’s capabilities and feedback signals remain limited.Methods include iterative self-refinement (Madaan et al., 2023), step-level feedback (Li et al., 2023g), external feedback (Gou et al., 2024a; Nathani et al., 2023), and deductive or abductive checks (Ling et al., 2023; Wu et al., 2024; Xue et al., 2023; Weng et al., 2022; Jiang et al., 2023b).Source: §4.3.1 · pp. 5–6

4.3.2 Question Decomposition

• Question decomposition tackles complex problems through subquestions or smaller components, but decomposition and bottom-up aggregation depend on proper problem division and can become counterproductive when misaligned.Approaches include top-down sequential decomposition, iterative decomposition, task-specific modules, DAG-based reasoning, and bottom-up aggregation (Zhou et al., 2023b; Dua et al., 2022; Khot et al., 2023; Huang et al., 2024b; Qi et al., 2023; Zhang et al., 2024).Source: §4.3.2 · p. 6

4.3.3 Knowledge Enhancement

• Knowledge enhancement mitigates factual errors by eliciting or organizing internal knowledge and retrieving external knowledge, with verification and iterative subquestion retrieval supporting accuracy on multi-hop tasks.Methods prompt parametric knowledge (Dhuliawala et al., 2023; Ji et al., 2023; Zheng et al., 2024), build memory from reasoning traces, or retrieve and verify external information (Li et al., 2023f; Wang et al., 2023e; Wang et al., 2023c).Source: §4.3.3 · pp. 6–7

4.3.4 Self-Ensemble

• Self-ensemble improves performance by ranking or voting across sampled answers, with step-level and heterogeneous variants addressing limitations of answer-only voting and sampling diversity, but increasing inference costs.Variants include answer ranking (Cobbe et al., 2021; Hu et al., 2024a), majority or complexity-based voting (Wang et al., 2023m; Fu et al., 2023a), and ensembles using varied instructions, XoT methods, languages, or multi-agent debate (Naik et al., 2023; Liu et al., 2023e; Qin et al., 2023; Liang et al., 2023; Du et al., 2023; Wang et al., 2023b).Source: §4.3.4 · p. 7

4.3.5 Efficient Reasoning

• Efficient reasoning methods reduce latency or human and inference costs through parallel decomposition, draft-based skipping, speculative decoding, selective annotation, and adaptive sampling.Further work on efficiency is needed to promote wider application of LLM reasoning.Source: §4.3.5 · p. 7

5.1 Tool Use

• Tool-use methods extend LLMs’ capabilities and enable external interaction, but multiple invocations and query errors remain difficult to handle.ReAct and Reflexion combine reasoning with action; ART selects tools and demonstrations, while MM-REACT adds vision experts. Tool invocation gathers information, and XoT supports elicitation, tracking, and action refinement.Source: §5.1 · pp. 7–8

5.2 Planning

• Planning methods decompose complex goals and track execution, using either code or formal languages, while search algorithms dynamically explore possible actions.Python plans can be refined from execution feedback; PDDL supports decomposition and specialized planning models. Tree-of-Thought uses DFS or BFS, RAP and LATS use Monte Carlo Tree Search, and ToolChain* uses heuristic A*; planning, reasoning, memory, and tools underpin autonomous agents.Source: §5.2 · p. 8

5.3 Distillation of Reasoning Capabilities

• Distillation aims to transfer step-by-step reasoning to smaller models, which can gain reasoning ability through CoT fine-tuning and varied chain-generation methods.Approaches include self-distillation from unlabeled data and self-loop bootstrapping, filtering high-quality chains, sampling multiple chains, contrastive and counterfactual objectives, and LoRA mixture-of-experts distillation for unseen-task generalization.Source: §5.3 · p. 8
• Preference optimization further improves small-model reasoning by training on ranked or sampled reasoning traces and optimizing with PPO or DPO.DialCoT uses multi-round reasoning dialogues; other approaches train reward models on automatically generated data, while another samples and scores trajectories with Monte Carlo Tree Search to create preference data online.Source: §5.3 · p. 8
• Distillation also transfers reasoning in code, mixed natural-language and code formats, tables, and retrieval-augmented settings, but task-specific gains may downgrade general performance.This shared paradigm trains smaller models on chains generated by stronger models, despite tradeoffs among multidimensional capabilities.Source: §5.3 · p. 8

6.1 Multi-modal Reasoning

• Multimodal CoT research extends beyond plain text, but limited vision-language model performance in multi-step reasoning leaves feature integration, VLLM use, and video reasoning open.SciQA (Lu et al., 2022) and CURE (Chen et al., 2023c) emphasize multimodal reasoning; methods combine vision and language features (Zhang et al., 2023i; Wang et al., 2023g), incorporate graph structures (Yao et al., 2023d,a), or convert images to captions (Yang et al., 2023b; Zheng et al., 2023b).Source: §6.1 · p. 9

6.2 Faithful Reasoning

• Faithful reasoning still requires better detection of unfaithful outputs and accurate feedback for correction.Retrieval augmentation addresses factual errors but requires appropriate timing and retrieval accuracy; inconsistencies are harder to identify, and proposed approaches include deductive logic, post-processing, critic-based methods, neural-symbolic reasoning, and question decomposition (Trivedi et al., 2023; Zhao et al., 2023a; Paul et al., 2024b; Jiang et al., 2023b; Xue et al., 2023; Ling et al., 2023; He et al., 2023a; Lei et al., 2023b; Madaan et al., 2023; Nathani et al., 2023; Chen et al., 2022a; Olausson et al., 2023; Radhakrishnan et al., 2023). Zhang et al. (2023c) and Lanham et al. (2023) empirically investigate factors influencing faithfulness.Source: §6.2 · p. 9

6.3 Theoretical Perspective

• The mechanisms behind CoT and ICL remain unclear, leaving open how emergence arises and how CoT improves reasoning over standard few-shot prompting.Existing empirical and theoretical work examines CoT’s effects, feature-based mechanisms such as information flow and attention, and emergence (Wang et al., 2023a; Madaan and Yazdanbakhsh, 2022; Tang et al., 2023; Li et al., 2023h; Feng et al., 2023; Merrill and Sabharwal, 2023; Prystawski et al., 2023; Wu et al., 2023b; Tutunov et al., 2023; Hou et al., 2023; Wang et al., 2023f; Schaeffer et al., 2023; Zhou et al., 2023c), but exploration remains surface-level.Source: §6.3 · p. 9

7 Discussion

• How chain-of-thought reasoning ability emerges through large-scale pre-training is an open question.The paper directs readers to Appendix A.2 for the discussion.Source: §7 · p. 9
• Providing accurate feedback on a model’s reasoning and decision-making remains an open question.The paper directs readers to Appendix A.2 for the discussion.Source: §7 · p. 9
• The implications of chain-of-thought reasoning for LLM-powered autonomous agents and AGI remain an open question.The paper directs readers to Appendix A.2 for the discussion.Source: §7 · p. 9

• The survey categorizes advanced methods and reviews frontier research, challenges, future directions, and open questions in generalized chain-of-thought reasoning.Source: §8 · p. 9
• The authors describe the paper as the first systematic survey dedicated to CoT reasoning.Source: §8 · p. 9
• The authors hope the survey will facilitate further research in the area.Source: §8 · p. 9

Limitations

• Because of page limits, the survey summarizes each method briefly rather than providing exhaustive technical details.Source: Limitations · p. 10
• The literature review focuses mainly on ∗ACL, NeurIPS, ICLR, ICML, COLING, and arXiv, so important work from other venues may be missing.Source: Limitations · p. 10
• The benchmarks section lists widely used datasets rather than a complete set; Guo et al. (2023) is cited for more complete benchmarks.Source: Limitations · p. 10
• There is no definitive conclusion on open questions, and the authors plan to update their views and add overlooked work as discussions evolve.Source: Limitations · p. 10

A.1 Related Survey

• Related surveys cover contemporary LLMs (Zhao et al., 2023b), early PLMs, domain-specific reasoning (Lu et al., 2023c; Talmor et al., 2019; Yang et al., 2023c), and hallucinations (Huang et al., 2023b; Zhang et al., 2023e).Source: §A.1 · p. 26
• Other related work surveys in-context learning, natural-language reasoning, prompt tuning (Liu et al., 2023d), prompting and reasoning strategies (Yu et al., 2023c; Huang and Chang, 2023), and the shift toward autonomous agents (Zhang et al., 2023g).Source: §A.1 · p. 26
• The paper presents the first comprehensive survey and systematic investigation of generalized chain-of-thought (XoT) reasoning in the LLM era.The authors hope it will serve as an overview that facilitates future research.Source: Limitations · p. 10, §A.1 · p. 26

A.2 Further Discussion

• Whether CoT ability originates in code pre-training remains open, though recent code-rich models and programming-language rationales suggest a possible link (Fu and Khot, 2022; Gao et al., 2023; Chen et al., 2022a).Earlier GPT-3 davinci and OPT models generally lacked CoT and used little or nonspecialized code data, unlike recent models such as GPT-3.5 and LLaMA2, which incorporate specialized code data.Source: §A.2 · p. 26
• Code mixed during pre-training enhances general reasoning, whereas code added during instruction fine-tuning confers task-specific reasoning.Source: §A.2 · p. 27
• Feedback methods use signals from models themselves, other models, external environments, or reinforcement learning, but dependability, capability limits, and cross-scenario applicability remain questions (Madaan et al., 2023; Shinn et al., 2023; Paul et al., 2024a; Nathani et al., 2023; Gou et al., 2024a; Uesato et al., 2022; Lightman et al., 2024; Ma et al., 2023).Researchers also question whether self-feedback is reliable and whether feedback from another language model fundamentally differs from self-feedback (Huang et al., 2024a; Jiang et al., 2024).Source: §A.2 · p. 27
• No fully satisfying approach currently provides precise feedback on intermediate reasoning, leaving accurate feedback signals an open research need.Source: §A.2 · p. 27
• Text-based tool interaction supports planning and decision-making research, while LLMs serve as controllers that reason and delegate execution to specialized modules (Schick et al., 2023; Shen et al., 2023a; Qin et al., 2024; Shinn et al., 2023; Yao et al., 2023b; Zhuang et al., 2024; Wang et al., 2023h; Xi et al., 2023; Zhang et al., 2023g; Yang et al., 2023a).These developments are presented as preliminary progress toward nascent AGI (s35.p6#47a891).Source: §A.2 · p. 27
• Whether LLMs can evolve into AGI remains debated, including concerns about autoregressive modeling and limited memory, although LLM-powered AI is moving toward greater generality (Bubeck et al., 2023).Source: §A.2 · p. 27

A.3 Early Attempts and Efforts in Specific Domains

• Before CoT was named, researchers used rationales to improve reasoning; subsequent work tested CoT prompting and explored multilingual reasoning (Wei et al., 2022b; Marasovic et al., 2022; Rajani et al., 2019a,b; Dua et al., 2020; Lampinen et al., 2022; Ye and Durrett, 2022; Arora et al., 2023; Shi et al., 2023).Source: §A.3 · p. 27
• Domain-focused CoT work spans translation, sentiment analysis, embeddings, summarization, arithmetic, tabular reasoning, and backdoor attacks, alongside benchmarks, resources, and reasoning-oriented pre-training (He et al., 2023b; Fei et al., 2023; Zhang et al., 2023a; Wang et al., 2023n; Lee and Kim, 2023; Chen, 2023; Ziqi and Lu, 2023; Xiang et al., 2024; Katz et al., 2022; Zhang et al., 2022a; Lewkowycz et al., 2022; Zhao et al., 2022).Source: §A.3 · p. 27

A.4 Empirical Results

• Table 2 summarizes GPT-series XoT performance in mathematics, commonsense, and symbolic reasoning, but checkpoint and setup differences make its comparisons only rough trends.The reported results are collected mainly from the corresponding papers, with some drawn from other papers as baselines.Source: §A.4 · p. 28

B.1 Mathematical Reasoning

• Mathematical benchmarks range from simple arithmetic to reasoning-process annotation, complex and table-based questions, competition problems, and reading comprehension.Examples include natural-language reasoning labels (Ling et al., 2017), program-form labels (Amini et al., 2019), and table-content reasoning (Zhu et al., 2021; Chen et al., 2021, 2022b).Source: §B.1 · p. 28

B.2 Commonsense Reasoning

• Commonsense benchmarks cover general understanding, event-temporal reasoning, and verification, targeting the challenge of acquiring and understanding shared world knowledge.Examples include commonsense understanding (Talmor et al., 2019, 2021; Bhakthavatsalam et al., 2021), temporal reasoning (Rashkin et al., 2018; Zhou et al., 2019), and verification (Wang et al., 2019).Source: §B.2 · p. 28

B.3 Symbolic Reasoning

• Symbolic benchmarks test simple operations that challenge LLMs, including last-letter concatenation, coin flips, reverse lists, state tracking, and object counting.Last-letter concatenation, coin flip, and reverse-list tasks are associated with Wei et al. (2022b); BigBench and BigBench-Hard include state tracking and object counting.Source: §B.3 · p. 28

B.4 Logical Reasoning

• Logical reasoning benchmarks address deduction from general premises, induction from special cases, and abduction as explanation of observed phenomena.The passage associates deductive reasoning with Liu et al. (2020), Yu et al. (2020), Tafjord et al. (2021), Han et al. (2022), and Hong et al. (2023); induction with Yang et al. (2024b); and abduction with Saparov and He (2023).Source: §B.4 · p. 28

B.5 Multi-modal Reasoning

• Multi-modal reasoning includes visual benchmarks, with ScienceQA commonly used and annotated for reasoning, while video reasoning adds temporal information and is more challenging.The passage identifies visual benchmarks including Zellers et al. (2019), Park et al. (2020), Dong et al. (2022), and Lu et al. (2022), and video benchmarks including Lei et al. (2020), Yi et al. (2020), and Wu et al. (2021).Source: §B.5 · p. 28

B.6 Comprehensive Benchmarks

• Comprehensive benchmarks evaluate holistic general reasoning or target specific abilities through multi-task logical and temporal reasoning benchmarks.The cited examples include Srivastava et al. (2022), Suzgun et al. (2023), Hendrycks et al. (2021a), Huang et al. (2023d), and Liang et al. (2022) for general reasoning; Luo et al. (2023) and Liu et al. (2023b) for logical reasoning; and Chu et al. (2023) and Wang and Zhao (2023) for temporal reasoning.Source: §B.6 · p. 28
• Table 2 gives an overall empirical insight into XoT methods on mathematical, commonsense, and symbolic reasoning benchmarks, but differing experimental setups make their performances not directly comparable.The table covers commonly used benchmarks; its caption cautions against direct comparisons across methods because their experimental setups vary.Source: Table 2 · p. 30