PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Step Reasoning with Large Language Models, a Survey

Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, Thomas Back

arXiv:2407.11511v3cs.AIcs.CLcs.LG

TL;DR

LLMs perform well on language tasks but struggle with reasoning. This survey organizes multi-step reasoning methods by generation, evaluation, and control, and finds approaches extending to logic reasoning alongside gains over standard prompting.

  • Problem

    Simple prompting performs poorly on reasoning tasks, including grade-school math problems, despite LLMs' strong language-task performance.

    Source: §1 · p. 2, §5 · p. 26

  • Method

    The survey organizes multi-step reasoning approaches with a taxonomy of reasoning-step generation, evaluation, and control.

    Source: §1 · p. 2

  • Results

    Multi-step approaches now perform logic reasoning; for large models such as GPT-3, reasoning approaches improve 20-50% points over standard prompting methods.

    Source: §1 · p. 2, §5 · p. 26

  • Takeaways & Limitations

    The surveyed multi-step reasoning capability extends beyond grade-school math to logic reasoning.

    Source: §1 · p. 2

    Reasoning, including math word problems, was among the capabilities LLMs could not achieve until recently.

    Source: §4.3 · p. 22

Abstract

from arXiv · show

Large language models (LLMs) with billions of parameters exhibit in-context learning abilities, enabling few-shot learning on tasks that the model was not specifically trained for. Traditional models achieve breakthrough performance on language tasks, but do not perform well on basic reasoning benchmarks. However, a new in-context learning approach, Chain-of-thought, has demonstrated strong multi-step reasoning abilities on these benchmarks. The research on LLM reasoning abilities started with the question whether LLMs can solve grade school math word problems, and has expanded to other tasks in the past few years. This article reviews the field of multi-step reasoning with LLMs. We propose a taxonomy that identifies different ways to generate, evaluate, and control multi-step reasoning. We provide an in-depth coverage of core approaches and open problems, and we propose a research agenda for the near future. We find that multi-step reasoning approaches have progressed beyond math word problems, and can now successfully solve challenges in logic, combinatorial games, and robotics, sometimes by first generating code that is then executed by external tools. Many studies in multi-step methods use reinforcement learning for finetuning, external optimization loops, in-context reinforcement learning, and self-reflection.

1 Introduction

LLMs handle language tasks well but struggle with reasoning, while chain-of-thought prompting has improved multi-step reasoning and math-word benchmark performance. This survey organizes the field by how methods generate, evaluate, and control reasoning steps, and proposes a research agenda.

1 Introduction

  • Reasoning tasks such as grade-school math word problems are more difficult for LLMs than language-generation tasks.Source: §1 · p. 2
  • Adding an instruction to think step by step can elicit intermediate reasoning, and math-word benchmark performance increased markedly.Source: §1 · p. 2
  • Multi-step approaches now perform logic reasoning, planning, combinatorial games, and robotic actions.Source: §1 · p. 2
  • Some methods prompt LLMs to generate code for external tools, while studies also use reinforcement learning, optimization loops, and self-reflection.Source: §1 · p. 2
  • The survey proposes a taxonomy of reasoning-step generation, evaluation, and control, alongside a research agenda.Source: §1 · p. 2

2 Background, Scope and Selection of Papers

This section frames the survey’s focus on prompt-based, multi-step LLM reasoning, particularly in the in-context learning setting, against persistent limits in LLM reasoning. It traces the benchmarks’ expansion from math word problems to broader domains and notes that differing experimental designs complicate comparisons across studies.

2 Background, Scope and Selection of Papers

  • LLMs have shown basic reasoning errors and limits in transferring base-10 arithmetic to base-9 problems and in analogical reasoning [10] [178] [83].Source: §2 · p. 2

2.1 Scope

  • Chain-of-thought prompting with multi-step examples improved performance on GSM8k, and the surveyed work now spans combinatorial games, puzzles, robotics, logic, and other reasoning fields [31] [173].The survey focuses on prompt-based multi-step reasoning methods inspired by Chain-of-thought.Source: §2.1 · p. 3
  • The survey selected recent papers with benchmark experiments, using reasoning and large language model search terms and a focus on Chain-of-thought references; it also includes influential external-algorithm and finetuning work.The selection began with math-word-problem benchmarks and broadened to code, games, puzzles, robot movement, and webpage navigation.Source: §2.1 · p. 3

2.2 Language Model Training Pipeline

  • The training pipeline includes autoregressive pretraining and task-specific finetuning, while inference uses prompts without further parameter changes and is where surveyed complex-reasoning methods focus.Source: §2.2 · pp. 3–4

2.3 In-Context Learning

  • In-context learning uses a prompt with demonstration context and a query at inference time, adapting answers through the examples and the model’s existing knowledge without changing parameters.Source: §2.3 · p. 4
  • Because in-context learning is sensitive to prompt formulation, the survey covers advanced prompt-engineering methods [131] [172].Source: §2.3 · p. 5

2.4 Multi-step Reasoning Benchmarks

  • GSM8K [31], the benchmark most associated with LLM multi-step reasoning, contains 8,500 human-written problems; models struggled before Chain-of-thought.Source: §2.4 · p. 5
  • On basic math word-problem benchmarks, Chain-of-thought performs well against GPT-3 175B on some datasets, but leaves room for improvement [173].Source: §2.4 · p. 5
  • Beyond math word problems, benchmarks span advanced mathematics, code comprehension, robotics [142] [2], puzzles, games [185, 40], and webpage navigation.Source: §2.4 · p. 5
  • The survey focuses on multi-step reasoning, reports results as surveyed papers did, and cautions that experimental differences complicate direct comparisons, though general conclusions can guide approach choice [70].Source: §2.4 · p. 6

3 Step Generation, Evaluation and Control

The survey organizes multi-step reasoning as a three-stage pipeline: generate reasoning steps, evaluate them, and control their search. Its taxonomy classifies approaches by how they generate the prompts that elicit step-by-step reasoning.

3 Step Generation, Evaluation and Control

  • The taxonomy is based on generating reasoning steps, evaluating them, and controlling the search over generated steps.The survey presents these as a general three-stage pipeline for in-context reasoning approaches.Source: §3 · p. 6
  • The taxonomy presents approaches under the subsections on generation, evaluation, and control, with categories also diagrammed in Figure 1 and Table 2.The figure shows approaches in their main category, although some approaches contribute innovations to two categories and appear twice.Source: §3 · p. 6
  • Generation prompts instruct the LLM to produce reasoning steps by splitting a problem into substeps.Prompts may ask the model to fill in blanks step by step.Source: §3 · p. 6
  • The survey distinguishes handcrafted, external knowledge-based, and model-generated prompts for eliciting step-by-step reasoning.These approaches have different advantages and disadvantages.Source: §3 · p. 6

1. Generation

External knowledge is identified as a prompt source for generation.

1. Generation

  • External knowledge is identified as a source for prompts.The passage names this category “Prompt using external knowledge.”Source: §1 · p. 7

3. Control

The survey organizes multi-step LLM reasoning around generating steps, evaluating them, and controlling their execution. It distinguishes simple single-chain approaches from evaluation mechanisms, ensembles, search, and reinforcement learning.

3. Control

  • Evaluation can rely on model self-assessment, external programs such as interpreters or compilers, or external models that validate steps.An external physics model can assess whether proposed robotic actions are physically possible.Source: §3 · p. 8
  • Control ranges from greedy execution of one chain to ensembles that evaluate and combine steps, tree search, and reinforcement-learning policies.More intricate problems may use a separate prompt or external algorithm to execute, evaluate, and combine substeps.Source: §3 · p. 8
  • The taxonomy groups methods by step generation, evaluation, and control, with Table 2 organizing approaches and their reported benchmark results across these categories.The table’s result symbols distinguish absolute scores from improvements over a baseline.Source: §3 · pp. 8, 10, Table 2 · p. 9
  • The Codex-to-Program-aided-language subgroup focuses on formal languages, generating code or math equations, typically in Python, to formalize reasoning steps; approaches achieve between 61% and 99% accuracy on various mathematics tasks and can use interpreters and debuggers to evaluate correctness.Source: §3 · p. 9
  • Reported benchmark results vary across approaches and tasks, while the models and baselines also differ; many approaches achieve high performance in their respective domains.Some results, such as creative-writing evaluations, are reported qualitatively.Source: §3 · p. 10

3.1 Generation of Steps

The survey traces step generation from explicit intermediate reasoning to hand-written, externally informed, and model-generated prompts. Across these approaches, prompting can improve reasoning performance, while generated prompts offer alternatives to manually written examples.

3.1 Generation of Steps

  • Scratchpads train a transformer with supervised learning to emit intermediate computation steps, illustrating multi-step computation beyond prompt-based learning.The survey describes experiments involving addition, polynomial evaluation, and Python code execution.Source: §3.1 · p. 10

3.1.1 Hand-written Prompt.

  • GSM8K accuracy rose from 16% to 47% with Chain-of-Thought prompting, the largest accuracy increase reported for the approach.The method prompts the model to reproduce question information in multiple intermediate reasoning steps before answering.Source: §3.1.1 · p. 10

3.1.2 Prompt using External Knowledge.

  • Self-ask poses elicitive follow-up questions and can use a search engine to answer intermediate prompts; on three multi-hop benchmarks, it performs a few points better than vanilla Chain-of-Thought.A separate analysis finds that GPT-3’s single-hop performance improves faster than its multi-hop performance as model size increases, without compositional-reasoning improvement.Source: §3.1.2 · p. 12

3.1.3 Model-Generated Prompt.

  • Auto-Chain-of-Thought builds dataset-specific questions and diverse demonstrations, performing a few percentage points better than hand-written Chain-of-Thought on ten GPT-3 benchmarks.The approach clusters questions and samples model-generated examples for the in-context prompt.Source: §3.1.3 · p. 12
  • Complexity-based prompting finds that the most complex reasoning prompts perform best, improving GSM8K by 7 points, MathQA by 6 points, and Big Bench Hard Penguins by 3 percentage points.The approach studies reasoning-chain complexity, measured by number of steps.Source: §3.1.3 · p. 12

3.2 Evaluation of Steps

This section surveys approaches to evaluating reasoning steps and validating reasoning methods to reduce error accumulation in multi-step chains. It covers self-assessment, executable formal languages, external models, dataset augmentation, and robotic tasks that use ground-truth data or environment feedback to evaluate or correct reasoning.

3.2.1 Self-Assessment.

  • Self-consistency further improves Chain-of-thought performance by 10–20 percentage points; self-verification improves results by a few percentage points on arithmetic and general reasoning tasks (GSM8K 65%, AQuA 48%, SVAMP 77%).Source: §3.2.1 · p. 13

3.2.2 Tool-based Validation.

  • Self-debugging [22] uses code generation, execution, and explanation to find mistakes, improving text-to-SQL Spider by 9 points and C++ to Python Transcoder by 12 percentage points.It investigates execution results and can explain generated code in natural language.Source: §3.2.2 · p. 14
  • MathPrompter [64] compares multiple algebraic expressions or Python functions for one problem, raising MultiArith accuracy from 78.7% to 92.5% on GPT-3 175B.The comparison raises confidence in the result, similarly to self-consistency and self-verification.Source: §3.2.2 · p. 14
  • Program-of-thought [19] and Program-aided-language [45] generate Python for interpreter evaluation; Program-aided-language reports 72% on GSM8K and 93% on Penguins, outperforming much larger models.Program-aided-language uses generic prompts and was tested on more benchmarks than Program-of-thought.Source: §3.2.2 · p. 15

3.2.3 External Model Validation.

  • Refiner reaches 78% accuracy on GSM8K, and Self-improvement reaches 82% on GSM8K with similarly high scores on question-answering and adversarial benchmarks.Refiner finetunes a generator using a critic’s feedback on reasoning errors; Self-improvement augments training data with LLM-generated rationales.Source: §3.2.3 · p. 15
  • Self-correction [174] finetunes a separate corrector to rectify a generator’s hypotheses, reducing toxicity to 0% in program synthesis.Unlike Refiner, it finetunes the corrector rather than the generator.Source: §3.2.3 · p. 15
  • Self-taught-reasoner augments training data with rationales and ground-truth answers for problems the model failed to solve, then finetunes on the combined dataset.It performs comparably at 72% to finetuning a 30-times-larger model on CommonsenseQA.Source: §3.2.3 · p. 15
  • Say-can [2] combines prompt-based reasoning with an affordance value function learned through temporal-difference reinforcement learning, achieving 31% success on 101 real-world robotic kitchen tasks.The language model supplies high-level task semantics while the learned function grounds actions in what the environment permits.Source: §3.2.3 · p. 16
  • Inner-monologue [61] combines prompt-based planning with scripted feedback modules, achieving 90% accuracy on simulated and real tabletop rearrangement tasks and 60% on the kitchen environment.Its inputs include InstructGPT descriptions, object recognition and success-detection modules, task-progress descriptions, and language-conditioned pick-and-place primitives similar to CLIPort [143].Source: §3.2.3 · p. 16
  • Chain-of-tools [139] [140] proposes a plan-execute-observe pipeline to ground reasoning about tool behavior.Source: §3.2.3 · p. 16

3.3 Control of Steps

This section surveys methods for controlling multi-step reasoning, from greedy selection and ensembles to search and reinforcement-learning controllers that manage branching. It also covers reinforcement-learning approaches including in-context exploration, game-playing agents, value-function training, and replacing supervised fine-tuning, along with reported outcomes.

3.3.1 Greedy Selection.

  • Least-to-most prompting decomposes complex problems into sequential subproblems and achieves 99% accuracy on a compositional generalization benchmark.Answers to earlier subproblems can serve as a curriculum for solving later ones.Source: §3.3.1 · p. 16

3.3.2 Ensemble Strategy.

  • Ensemble strategies generate and evaluate multiple possible reasoning paths, then report a consensus result instead of following a single path.Examples include Self-consistency, Self-verification [176], and Chain-of-experts.Source: §3.3.2 · p. 16

3.3.3 Reinforcement Learning.

  • Beam-search-for-reasoning [181] surpasses few-shot baselines by 6.34% on GSM8K, 9.56% on AQuA, and 5.46% on StrategyQA.It searches promising paths using self-evaluation to guide exploration and score reasoning steps.Source: §3.3.3 · p. 17
  • Progressive-hint-prompting [195] iteratively feeds prior answers back as hints, reaching 95% on grade-school math and 91% on simple math word problems with GPT-4.It repeats question-and-answer prompting until the answer stabilizes.Source: §3.3.3 · p. 18
  • Self-refine [95] iteratively uses LLM-generated feedback to refine outputs, typically improving base models by around 30 percentage points across varied tasks.The approach has been applied to dialogue, code, math reasoning, sentiment reversal, acronym generation, and constrained generation.Source: §3.3.3 · p. 18
  • ReAct combines LLM reasoning with action-plan generation and uses LLMs to improve reinforcement-learning agent policies; it outperforms imitation and reinforcement-learning methods on Alfworld and WebShop.Its reinforcement-learning framework uses Chain-of-thought prompt-learning, Wikipedia as an external knowledge source, and finetuning.Source: §3.3.3 · pp. 18–19
  • Reflexion [141] uses actor, evaluator, and reflector models to learn from failures, with reported improvements of 10–20 percentage points on decision-making, reasoning, and coding tasks.The reflector supplies verbal reinforcement cues to help the actor improve.Source: §3.3.3 · p. 19
  • AoT [135] applies in-context reinforcement learning to general LLMs and achieves results comparable to Tree-of-thoughts on GSM8K, StrategyQA, and Crosswords.The approach targets models including GPT-4, Claude, and Gemini 1.5.Source: §3.3.3 · p. 19
  • In games, LLMs struggle to execute generated algorithms correctly and to use rather than merely mention game concepts [108] [129] [30] [159].These are described as the knowing-doing gap and a distinction between using and mentioning concepts.Source: §3.3.3 · p. 19
  • Voyager [166] uses iterative code prompting, self-verification, and a skill library to explore Minecraft, acquiring tools 15 times faster than the baseline.Its goal is to discover diverse items through novelty search [39].Source: §3.3.3 · p. 20
  • 85% accuracy on GSM8K and 81% on Alfworld were achieved by jointly training an LLM policy and value function through Bellman-equation optimization [167].Source: §3.3.3 · p. 20
  • Du et al. [38] and Guo et al. [50] replace supervised fine-tuning with reinforcement learning integrated with reasoning, reporting large training-pipeline efficiency gains in DeepSeek r1 and Kimi.Source: §3.3.3 · p. 20

4 Discussion

The discussion matches reasoning approaches to application needs while examining comparison challenges, faithfulness, and scaling. It closes by identifying conditions and limitations that shape current LLM reasoning.

4.1 Matching Methods and Applications

  • Direct benchmark comparisons are difficult because studies use different measurement setups, but the survey offers guidance for matching methods to applications.It also notes reported applications in logistics, finance, and medicine.Source: §4.1 · pp. 20–21
  • For simple single-sequence reasoning, Chain-of-thought is typical; verification such as Self-consistency is used when error rates are too high, while state-heavy tasks may call for external control.Tree-of-thoughts or ReAct can manage state externally; complex problems may instead use generated programs executed by tools.Source: §4.1 · pp. 20–21

4.2 Hallucination, Faithfulness and Scaling

  • The discussion frames LLM reasoning as error-prone and highlights hallucination as a concern when reasoning is not properly grounded.Mechanistic interpretability is discussed as a way to study individual representations and expected behavior.Source: §4.2 · p. 21, §4.2.1 · p. 21
  • Chain-of-thought can reason to a degree but struggles with proof trees that have wide branching; reasoning-step order matters, and longer prompts correlate positively with reasoning ability.Prompts should be relevant and coherent, with steps in the correct order.Source: §4.2 · p. 21

4.2.1 Faithfulness.

  • Chain-of-thought explanations may rationalize wrong answers, and studies report varying degrees of post-hoc reasoning, with larger models tending to be less faithful.Faithful-chain-of-thought proposes translating queries into formal symbolic language before problem solving.Source: §4.2.1 · p. 21

4.2.2 Scaling.

  • Reasoning can be distilled to smaller models through teacher-student finetuning, and prompt distillation has also been studied for retrieval, recommendation, embodied agents, and graph reasoning.Distillation can work especially well with more explicit instructions.Source: §4.2.2 · p. 22

4.3 Limitations: What LLMs Can and Cannot do

The survey describes reasoning capabilities that extend beyond grade-school mathematics while emphasizing that current systems remain limited. It connects stronger performance to symbolic search and external control, and notes that metacognitive reasoning remains early-stage.

4.3.1 What Can LLMs Do?

  • Prompt-based learning can solve grade-school math benchmarks and extend to robotics, autonomous agents, games, robotic movement gaming, and Python code generation.The survey says this can occur at inference time without expensive parameter training.Source: §4.3.1 · pp. 22–23
  • The survey’s generate-evaluate-control taxonomy describes the literature, where higher-performance reasoning combines a symbolic searcher or learner with a connectionist LLM.The LLM grounds symbolic reasoning in language, while symbolic methods help create prompts for LLM reasoning.Source: §4.3.1 · p. 22

4.3.2 What Can LLMs Not Do?

  • Auto-GPT or Auto-CoT prompts are prone to feedback-loop problems and need careful evaluation, verification, and grounding in the real world.The survey also identifies environmental, bias, ethical, legal, and copyright concerns associated with LLM training data and demands.Source: §4.3.2 · p. 23
  • Some symbolic-planning studies show planning failures and argue that LLMs are better used to improve heuristic elements of traditional planners such as PDDL.The survey also notes that LLMs can hallucinate, or invent facts that look right but are not.Source: §4.3.2 · p. 23
  • True human-like self-reflection remains largely beyond current LLMs; reflective behavior is written into prompts or control algorithms, and true self-reflection remains future work.The survey says it knows of no LLM that controls how many reasoning steps to take or stops once an answer is good enough.Source: §4.3.2 · p. 23

4.3.3 Reasoning towards Metacognition.

  • On 7,000 GSM8K instances, a model found 500 skill names that were clustered into 22; using those names in Chain-of-thought-like prompts solved more problems than standard prompt baselines.Examples of the skill names include multiplication-and-addition, basic-arithmetic, subtraction, and algebra.Source: §4.3.3 · p. 24
  • A skill-exemplar repository trained on GPT-4 also benefits GPT-3 when its skill names are used in prompts, while metacognitive reasoning remains in its early stages.The survey notes that LLMs struggle with analogies and analogical reasoning.Source: §4.3.3 · p. 24

4.4 LLMs that Generate Computer code

The survey treats code generation as one mode of LLM use and links it to formal-language reasoning and external tools. Generated code can manage state, while tool-assisted agentic approaches often yield good performance.

4.4 LLMs that Generate Computer code

  • LLMs can generate formal languages such as PDDL, logic, math equations, and Python, and agentic use with tools such as interpreters or debuggers often yields good performance.The survey distinguishes direct use from algorithmic or Chain-of-thought prompting, which adds examples of steps to take.Source: §4.4 · p. 24
  • For code generation, a prompt specifies a problem and the LLM generates code such as Python or PDDL to address the knowing-doing gap.The code may use variables to manage state and can be executed directly or checked and improved by a programmer.Source: §4.4 · p. 24
  • The survey expects LLM code generation and prompt engineering as natural-language programming to grow, and notes that test-time scaling can trade off model size and training time.It also describes training transformers so search and reinforcement-learning algorithms can be expressed.Source: §4.4 · p. 24

4.5 Research Agenda

The research agenda identifies practical and foundational directions for multi-step LLM reasoning, spanning tool integration, evaluation, and unresolved challenges. It highlights open questions about reasoning control, model scale, faithfulness, and broader settings.

Research Agenda

  • The agenda asks whether reasoning control can be incorporated into a static prompt or in-context reinforcement learning, since general LLMs struggle with ICRL.Source: §4.5 · p. 25
  • Integrating LLM reasoning with software tools and grounding agents in their environments are practical directions, with grounding still a challenge.The agenda points to software-engineering tools and RAG that can help find information when reasoning is uncertain.Source: §4.5 · p. 25
  • The agenda calls for more difficult and diverse benchmarks, including specialized legal, scientific, and medical reasoning contexts.It notes that standardized benchmarks have a limited set of problems, while real-world reasoning also occurs in challenging contexts.Source: §4.5 · pp. 25–26
  • Theoretical understanding remains incomplete, particularly for faithfulness and how multi-step in-context reasoning works.The agenda says more insight into implicit reasoning is needed.Source: §4.5 · p. 25
  • Smaller language models offer efficiency and accessibility advantages, but may have less reasoning power, leaving a trade-off for future work.The agenda notes that smaller models can also be finetuned and may have a smaller environmental footprint.Source: §4.5 · p. 25
  • Longer-term questions include combining symbolic and connectionist computation with multimodal world models grounded through perception–action loops.Source: §4.5 · p. 25
  • The agenda asks how reasoning can represent, adapt, and audit diverse cultural norms and ethical values while respecting context-dependent constraints.Source: §4.5 · p. 25
  • Reasoning in languages beyond English remains an open direction, especially for low-resource languages and specialized domains.The agenda notes that existing multilingual benchmarks mainly cover QA and NLI.Source: §4.5 · p. 25

5 Conclusion

Chain-of-thought reasoning has expanded from grade-school math to a broader set of tasks, while the survey organizes methods by how they generate, evaluate, and control reasoning steps. It reports gains over standard prompting alongside open limitations and directions for future work.

Conclusion

  • 20–50% points: reasoning approaches improve over standard prompting methods on benchmarks with older LLMs such as GPT-3.Source: §5 · p. 26
  • Reformulating problems in Python or another formal language can reduce error rates, while formal languages and code serve as intermediate reasoning chains.Source: §5 · p. 26
  • Reasoning approaches have expanded beyond math, and performance in robotic action has been achieved by grounding answers in environmental and physical constraints.Source: §5 · p. 26
  • The survey categorizes Chain-of-thought approaches by how they generate, evaluate, and control reasoning steps.Inference-time reasoning methods are also used for LLM finetuning.Source: §5 · p. 26
  • Reinforcement learning with verifiable rewards and GRPO use inference-time reasoning results for finetuning, while external search or reinforcement learning can control reasoning-space size.Source: §5 · p. 26
  • The field lacks theoretical understanding in areas such as faithfulness, and LLMs hallucinate, suffer from bias, and pose ethical and societal dangers.Self-verification can reduce error accumulation, but reflection is not error-free.Source: §5 · p. 26
  • Reasoning skills that work in large models can often be distilled to smaller models.Source: §5 · p. 26
  • The research agenda highlights reinforcement learning for finetuning and self-reflection, code-generating agents, multimodal world models, value-aligned reasoning, and small language models.Source: §5 · p. 26

Related papers

Loading 2407.11511v3…