PaperlayerView exact revision on arXiv

Source-linked AI summary

To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning

Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, Greg Durrett

arXiv:2409.12183v3cs.CLcs.AIcs.LG

TL;DR

The paper asks when CoT’s extra prompting-based reasoning is useful and studies this through a literature meta-analysis and experiments across datasets, models, and prompts. It finds that CoT mainly benefits math and formal reasoning, often through symbolic execution, while tool-augmented symbolic solvers perform better on these tasks. The authors therefore support selective CoT use and approaches beyond prompt-based CoT.

  • Problem

    The paper examines which kinds of tasks benefit from CoT, given its widespread use and the limited evidence beyond mathematical reasoning.

  • Method

    The authors conduct a meta-analysis of CoT studies and evaluate 20 datasets across 14 models, then separate planning from execution and compare CoT with symbolic-solver pipelines.

  • Results

    CoT predominantly helps on math and formal logic, with much of its gain arising from symbolic execution, while tool-augmented approaches rarely are outperformed.

  • Takeaways & Limitations

    CoT can be used selectively to maintain performance at lower inference cost, while future methods should better leverage intermediate computation beyond prompt-based CoT.

  • Takeaways & Limitations

    The experiments do not cover long-horizon planning except for BiGGen Bench, and the paper does not address dataset contamination directly.

Abstract

from arXiv · show

Chain-of-thought (CoT) via prompting is the de facto method for eliciting reasoning capabilities from large language models (LLMs). But for what kinds of tasks is this extra ``thinking'' really helpful? To analyze this, we conducted a quantitative meta-analysis covering over 100 papers using CoT and ran our own evaluations of 20 datasets across 14 models. Our results show that CoT gives strong performance benefits primarily on tasks involving math or logic, with much smaller gains on other types of tasks. On MMLU, directly generating the answer without CoT leads to almost identical accuracy as CoT unless the question or model's response contains an equals sign, indicating symbolic operations and reasoning. Following this finding, we analyze the behavior of CoT on these problems by separating planning and execution and comparing against tool-augmented LLMs. Much of CoT's gain comes from improving symbolic execution, but it underperforms relative to using a symbolic solver. Our results indicate that CoT can be applied selectively, maintaining performance while saving inference costs. Furthermore, they suggest a need to move beyond prompt-based CoT to new paradigms that better leverage intermediate computation across the whole range of LLM applications.

1 INTRODUCTION

The paper evaluates where prompt-based CoT helps and finds substantial benefits mainly for mathematical, logical, algorithmic, and symbolic reasoning tasks. It further separates planning from execution and compares CoT with tool-augmented approaches.

  • Scope and approach: The study combines a literature meta-analysis with experiments across 20 datasets and 14 contemporary LLMs.The experiments cover zero-shot and few-shot prompting settings.
  • Main finding: CoT gives its strongest benefits on problems requiring mathematical, logical, or algorithmic reasoning.Large gains outside these areas are uncommon and often involve symbolic reasoning components.
  • Main finding: 95% of MMLU’s total CoT performance gain is attributed to questions or generated outputs containing “=”.The equals sign is associated with symbolic operations and reasoning in the MMLU analysis.
  • Planning and execution: CoT primarily improves execution involving computation and symbolic manipulation, rather than replacing the need for planning.The authors distinguish planning, such as parsing a problem into equations, from execution, which builds intermediate outputs toward a solution.
  • Planning and execution: Generating a solution plan and using an external symbolic solver outperforms using CoT for both planning and execution.This comparison is reported for the mathematical and formal logical reasoning tasks studied.
  • Implications: CoT can be applied selectively because more efficient prompting strategies may preserve performance at lower inference cost.The paper also motivates approaches based on search, interacting agents, or more heavily fine-tuned models.

2 BACKGROUND: CHAIN-OF-THOUGHT

This section defines direct answering and CoT as alternative prompting strategies, distinguishes symbolic from non-symbolic reasoning, and describes the paper’s task framing. It focuses the analysis on single-prompt approaches while treating tool augmentation separately.

  • Task formulation: Tasks are framed as questions q over a vocabulary Σ with answers a drawn from a question-dependent label set L(q).Labels can be booleans, integers, classification labels, or problem-dependent entity names.
  • Prompting strategies: Direct answering outputs only the answer, whereas CoT produces a longer sequence containing intermediate tokens before or around the answer.Both approaches require parsing and detokenizing the generated output, with CoT requiring extra answer-location handling.
  • Prompting strategies: The prompts denote direct answering and chain-of-thought strategies as I_direct and I_cot.CoT prompts may say “think step by step,” while direct prompts may say “immediately generate the answer.”
  • Symbolic reasoning: A symbolic problem can be grounded in a natural, well-agreed formal system such as mathematics, first-order logic, or planning languages.The paper represents this grounding as a symbolic expression S = f(q).
  • Symbolic reasoning: Non-symbolic problems lack a natural, well-agreed framework for solving them, even when they could be formalized in principle.Commonsense questions such as where to hold a cup to catch river water exemplify this category.
  • Central claim: The paper’s central claim is that CoT helps most on symbolic and semi-symbolic tasks, with little improvement or possible harm on non-symbolic tasks.This challenges the view that CoT broadly improves all reasoning problems.
  • Scope: The analysis focuses on single-prompt approaches and handles tool-augmented methods in a separate section.Decomposed prompting and multi-agent debate are excluded from the analysis.

3 RESULTS FROM THE LITERATURE

The literature meta-analysis finds that CoT’s large gains are concentrated in symbolic, mathematical, and logical reasoning categories. Non-math outliers often contain related algorithmic, deductive, or symbolic components, while most other categories show minimal gains.

  • Meta-analysis design: 110 papers contributed 1,218 experimental comparisons after screening 4,642 papers from ICLR, EACL, and NAACL 2024.The comparisons were grouped by task and dataset type.
  • Aggregate results: Average CoT improvements were 14.2, 12.3, and 6.9 for symbolic reasoning, math, and logical reasoning, respectively.For the top three categories, average performance was 56.9 with CoT versus 45.5 without CoT.
  • Aggregate results: For other categories, average performance was 56.8 with CoT versus 56.1 without CoT.The paper does not regard this small improvement as a victory for CoT, especially because CoT uses more computation.
  • Outliers: BIG-bench Hard was the largest non-math outlier, but many of its tasks require algorithmic, arithmetic, or logical reasoning.Navigate relies on counting steps, while Temporal requires deductive reasoning.
  • Outliers: Other apparent outliers include Legal Argument Reasoning, MMLU-Moral Scenarios, ScienceQA, a proprietary dialogue dataset, Commitment Bank, and verbalized confidence.The authors note that some gains are difficult to interpret or do not generalize across related datasets.
  • Overall pattern: The majority of reported CoT benefits in the NLP and ML literature comes from math or math-related tasks.The paper treats the non-math gains as exceptions to the overall trend.

4 RESULTS FROM EXPERIMENTS

The experiments evaluate CoT across varied datasets, models, and prompting settings, finding its benefits concentrated in mathematical and symbolic reasoning rather than most non-symbolic tasks.

  • Experimental setup: The evaluation spans English models and datasets covering commonsense, knowledge, symbolic, mathematical, and soft reasoning across multiple answer formats.The study includes zero-shot and few-shot prompting strategies, with most datasets using multiple-choice or short-answer formats.
  • Where CoT helps: Zero-shot CoT improves performance mainly on datasets requiring mathematics or formal logic, including MATH, GSM8K, ContextHub, and MuSR.The reported strongest cases are mathematical datasets and, to a lesser degree, formal-logic datasets.
  • Where CoT does not help: Non-symbolic datasets involving commonsense, language understanding, or reading comprehension show little to no separation between zero-shot CoT and direct answering.The cited examples include CSQA, PIQA, SiQA, WinoGrande, AGI LSAT, ARC-Easy, and ARC-Challenge.
  • Magnitude of gains: MATH and GSM8K show gains as large as 41.6% and 66.9%, respectively, while semi-symbolic datasets show moderate gains.ContextHub and MuSR require applying logical rules to reach answers from natural-language statements.
  • Answer format and task structure: CoT does not improve MuSiQue overall despite its explicit multi-hop structure and non-multiple-choice short-form question answering.Answer-span equivalence is judged with GPT-4o because MuSiQue answers can be paraphrased.
  • MMLU analysis: For MMLU and MMLU Pro, most of the CoT gain comes from questions or generated responses containing an equals sign, identifying math-related slices.The datasets are divided into math and non-math bins using the presence of “=” in the question or model response.

5 STRENGTHS AND WEAKNESSES OF COT AT FORMAL REASONING

The paper separates symbolic reasoning into planning and execution to explain CoT’s gains. CoT improves execution over direct answering but is usually surpassed by external symbolic solvers.

  • 5 STRENGTHS AND WEAKNESSES OF COT AT FORMAL REASONING: Symbolic reasoning is analyzed as planning a formal specification and executing it with a language model or external solver.Planning extracts variables and relations; execution derives the final answer from the resulting plan.
  • 5 STRENGTHS AND WEAKNESSES OF COT AT FORMAL REASONING: On MMLU and MMLU Pro, most CoT gains come from questions or generated responses containing an equals sign.The authors use equals signs as an indicator of math-related questions.
  • 5.1 SETTINGS EVALUATED: The evaluation compares direct answering, CoT, planned direct solving, planned CoT solving, and planned tool solving.The planned variants first generate a formal solution specification and then vary how it is executed.
  • 5.1 SETTINGS EVALUATED: For math datasets, formal plans are Python programs, whereas logical reasoning plans are first-order logic specifications.The plans are designed to be executable by symbolic solvers.
  • 5.1 SETTINGS EVALUATED: The experiments evaluate math and logical reasoning datasets, including GSM8K-Hard to address possible GSM8K data contamination.The compared tool solvers use a Python interpreter for math and an SMT solver for logical reasoning.
  • 5.2 EVALUATION RESULTS: Tracking execution with CoT or planned CoT solving gives stronger accuracy than direct answering on many datasets and models.The benefit is especially pronounced on math-heavy datasets, while having only a plan often does not explain most of the gain.
  • 5.2 EVALUATION RESULTS: Plan + Tool Solver dominates CoT and Plan + CoT Solver in most settings.The results attribute this gap to language models’ limitations in executing and tracking symbolic steps compared with symbolic solvers.
  • 5.2 EVALUATION RESULTS: CoT outperforms direct answering mainly when solving requires substantial tracing and computation.The authors characterize CoT as a poor but universal approximation to symbolic solvers and recommend pairing models with solvers when possible.

6 DISCUSSION AND RELATED WORK

The discussion reassesses the broad use of CoT beyond multi-step reasoning and notes unresolved issues around computation and evaluation scope.

  • 6 DISCUSSION AND RELATED WORK: CoT’s application has broadened from mathematical reasoning to tasks that do not canonically require multiple steps.The authors suggest this contributed to a prevailing expectation that deliberation should help on any reasoning task.
  • 6 DISCUSSION AND RELATED WORK: Additional LLM calls can improve reasoning, but these approaches require significantly more computation.Careful benchmarking sometimes finds naive techniques as effective as iterative ones.
  • 6 DISCUSSION AND RELATED WORK: The experiments do not cover long-horizon planning except in BiGGen Bench, and possible data contamination is addressed through multiple models, datasets, and meta-analysis.Further discussion of planning and contamination appears in Appendix I.

7 CONCLUSION

The paper characterizes when prompt-based CoT helps and how its gains compare with tool augmentation. It finds the strongest benefits in math and formal logic, while motivating methods beyond prompt-only CoT.

  • 7 CONCLUSION: The paper combines a literature meta-analysis with experiments across models, datasets, and prompts to characterize prompt-based CoT.The conclusion summarizes CoT’s performance across these evaluation dimensions.
  • 7 CONCLUSION: CoT predominantly helps on math and formal logic, largely through tracing intermediate problem-solving steps.It rarely outperforms tool-augmented approaches on the same problems.
  • 7 CONCLUSION: The authors propose moving beyond prompt-based CoT toward paradigms such as search, interacting agents, or better fine-tuned models.This proposal is framed as a way to seek improvements across a wider range of NLP tasks.

REPRODUCIBILITY

The paper supports reproducibility by releasing prompts, model outputs, sampling parameters, and the materials underlying its literature meta-analysis.

  • REPRODUCIBILITY: The authors release prompts for every dataset on Huggingface, including per-model outputs and sampling parameters.These materials accompany the experiments described in the paper.
  • REPRODUCIBILITY: They also release the full list of papers included in the meta-analysis together with extracted materials.The passage indicates that the release covers the meta-analysis process and its extracted data.

C EXPANDED EXPERIMENTAL DETAILS

The experiments evaluate contemporary instruction-tuned models across reasoning categories, datasets, prompting settings, and multiple CoT prompt formulations. Different CoT prompts produce little average variation in performance.

  • Meta-analysis: The meta-analysis groups literature comparisons by task categories to evaluate CoT against direct answering.The category definitions are provided in Table 2, while the paper also releases extracted comparisons and annotations.
  • Prompt variation: Four common zero-shot CoT prompting strategies are tested on seven datasets with Llama 3.1 8B.This experiment specifically measures whether prompt wording changes performance.
  • Experimental setup: The study evaluates contemporary instruction-tuned models on Commonsense, Knowledge, Soft Reasoning, Symbolic, and Mathematical datasets.The dataset suite records answer formats and few-shot example counts, while the model suite focuses on contemporary instruction-tuned systems.
  • Prompt variation: There is very little average performance variation between the tested CoT prompts, and no prompt provides a consistent gain over the others.The tested variants include prompting phrases intended to elicit reasoning, including “Let’s think step by step.”

E FEW-SHOT EXPERIMENTS

Few-shot CoT adds demonstrations of reasoning traces, while few-shot direct prompting demonstrates answers. Across settings, few-shot prompting generally preserves which datasets benefit from CoT, with changes mainly in improvement magnitude.

  • Prompt construction: Few-shot direct prompts encode answer demonstrations, whereas few-shot CoT prompts include reasoning traces ending in the correct answer.Both prompt types append demonstrations to the question before generating the target response.
  • Results: Few-shot prompting can change the magnitude of CoT improvements, particularly on symbolic datasets.The figure summary states that this magnitude shift does not generally alter the qualitative zero-shot-versus-few-shot pattern.
  • Results: Few-shot CoT largely preserves the zero-shot pattern of which datasets benefit from CoT.The qualitative relationship between CoT and direct answering generally remains unchanged across the two settings.
  • Results: Only MuSR Team Allocation begins improving with few-shot CoT, which the authors attribute to clearer examples of its complex final derivation step.The authors characterize this dataset-level change as an exception.

F.1 FULL ZERO-SHOT RESULTS

The full zero-shot results compare direct answering and CoT across reasoning categories, datasets, and models. Results include answer-extraction reliability and answer-position measurements alongside accuracy.

  • Accuracy results: Zero-shot results report direct-answer and CoT accuracies across reasoning categories and all evaluated datasets.The tables include category-level and dataset-level comparisons, including examples such as Arc Challenge and MuSR.
  • Dataset results: MuSR Murder Mysteries has 50.0 accuracy for both direct answering and CoT with Llama 2 7B.This is one of the reported zero-shot dataset-model results.
  • Dataset results: MuSR Object Placements shows higher CoT than direct-answer accuracy for every listed model.Reported pairs include 55.5 versus 41.0 for Mistral 7B and 66.8 versus 50.4 for Llama 3.1 8B.
  • Answer extraction: Unparseable response rates are generally low and remain at or below 15%.The weakest models struggle on some challenging datasets, while “-1” denotes runs omitted because of few-shot context-length limitations.
  • Answer span: Direct-answer prompts return an answer within the first 60 characters, whereas CoT completions are much longer.Answer-span position is used to assess whether prompts produce reasoning before the answer.

G ZOOM-IN: MMLU AND MMLU PRO

The MMLU and MMLU Pro analysis examines where CoT gains arise at the instance and category levels. An equals sign identifies questions or responses associated with symbolic reasoning and explains most of the observed improvement.

  • Category analysis: MMLU and MMLU Pro show CoT gains, but their broad coverage requires analysis by category to explain performance differences.The authors inspect category-level error reductions and report the top three benefiting slices for selected Llama 3.1 models.
  • Figures: The MMLU and MMLU Pro plots compare zero-shot direct and CoT performance across datasets and models, sorted by median delta.Datasets with substantial benefits are described as symbolic or semi-symbolic.
  • Category analysis: Six of the 12 top-benefiting MMLU and MMLU Pro slices explicitly contain “math” or “mathematics.”The authors further report that the remaining improving categories are mathematical in nature.
  • Instance-level analysis: The instance-level classifier uses whether the question or CoT contains an equals sign to identify likely symbolic-reasoning cases.The classifier can also consult the generated CoT to capture how the model solves the problem.
  • Instance-level analysis: The majority of CoT’s MMLU and MMLU Pro performance gain comes from questions containing an equals sign in the question or generated response.The paper compares overall CoT-minus-direct deltas between “With =” and “Without =” bins across multiple models.

H FULL RESULTS OF EVALUATIONS ON FORMAL REASONING DATASETS

The appendix reports evaluation details for formal-reasoning methods, including accuracy, unparseable responses, and CoT comparisons with solver-based approaches. It also notes that long-horizon planning is not covered thoroughly in the experiments.

  • H FULL RESULTS OF EVALUATIONS ON FORMAL REASONING DATASETS: Unparseable responses are failures of answer extraction or symbolic-solver execution, depending on the method.For tool-augmented prompting, the unparseable rate instead concerns failures to execute generated formal specifications.
  • H FULL RESULTS OF EVALUATIONS ON FORMAL REASONING DATASETS: All models have unparseable rates below 10% for methods other than tool-augmented prompting.Higher tool-augmentation rates arise when models generate Python or other formal specifications that do not follow the required language format.
  • H FULL RESULTS OF EVALUATIONS ON FORMAL REASONING DATASETS: A logistic-regression classifier over question and CoT-response pairs discovered the equals-sign feature but did not surpass that single feature.The classifier used tf-idf features and was trained on subsets of MMLU and MMLU Pro.
  • H FULL RESULTS OF EVALUATIONS ON FORMAL REASONING DATASETS: Tables 18 and 19 break down MMLU and MMLU Pro CoT deltas by whether questions or responses contain an equals sign.The tables distinguish gains associated with equals signs from gains without them.
  • H FULL RESULTS OF EVALUATIONS ON FORMAL REASONING DATASETS: Table 20 compares few-shot direct answer, few-shot CoT, Plan + Direct Solver, Plan + CoT Solver, and Plan + Tool Solver Solver.It reports accuracy and unparseable-response rates, with special handling for unparseable FOLIO and ContextHub responses.
  • H FULL RESULTS OF EVALUATIONS ON FORMAL REASONING DATASETS: Tool augmentation can remain on par with or outperform other methods despite higher unparseable rates.The issue is particularly severe for smaller models because format errors lead to execution failures.
  • I.1 LONG HORIZON PLANNING: The experiments do not cover long-horizon planning as thoroughly as other symbolic-reasoning tasks.BiGGen-Bench is an exception; the paper gives limited language-only planning and an existing debate over CoT-based planning methods as reasons for this scope choice.

I.2 DATASET CONTAMINATION

The study acknowledges possible benchmark contamination because it is unknown which benchmarks were included in language-model pretraining. It presents several reasons for retaining confidence in its general conclusions and releases prompts and responses with examples.

  • I.2 DATASET CONTAMINATION: The study’s contamination risk is that some benchmarks may have been explicitly included in language-model pretraining.If answers were memorized, direct answering could narrow the observed gap with CoT.
  • I.2 DATASET CONTAMINATION: The authors use multiple model scales, including smaller models with less capacity to memorize, to reduce contamination concerns.This is one of four reasons they give for trusting the general conclusions.
  • I.2 DATASET CONTAMINATION: Datasets with poor direct-answer performance, such as GSM8K-Hard, are unlikely to have been substantially memorized.The authors cite this as another reason their conclusions remain trustworthy.
  • I.2 DATASET CONTAMINATION: Recent datasets including MuSR and BiGGen Bench help defray the risk of benchmark contamination.The literature survey also contributes to the authors’ stated justification for trusting their conclusions.
  • J EXAMPLE PROMPTS: The appendix includes prompt-and-response examples for MATH, CommonsenseQA, and AGIEval LSAT AR.These examples include direct-answer and step-by-step CoT prompt formats.
  • J EXAMPLE PROMPTS: The MATH example requests an efficient, clear solution, while the CommonsenseQA example requires only a formatted answer.The AGIEval LSAT AR example explicitly requests step-by-step reasoning before the final answer.
Loading 2409.12183v3…