Source-linked AI summary
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, Huan Sun
TL;DR
The paper asks which demonstrated-rationale properties make Chain-of-Thought prompting effective and tests them through targeted ablations. Invalid reasoning retains over 80-90% of CoT performance, while relevance and correct ordering matter more. The findings suggest CoT demonstrations often elicit pretrained reasoning abilities rather than teach reasoning itself.
Problem
The study asks whether ground-truth rationale components are necessary and which aspects enable LLMs to reason properly under CoT prompting.
Method
The paper conducts ablation experiments that alter different rationale aspects across arithmetic reasoning and multi-hop factual question answering.
Results
Over 80-90% of CoT performance is achieved with completely invalid demonstrated reasoning, while relevance and correct reasoning order are key effectiveness factors.
Takeaways & Limitations
The findings suggest demonstrations may mainly elicit pretrained reasoning abilities by specifying an ordered, relevant step-by-step output format.
Takeaways & Limitations
Invalid-reasoning experiments rely on manually written rationales, and the paper notes that automatically synthesizing them is challenging for its informal tasks.
Abstract
from arXiv · showhide
Chain-of-Thought (CoT) prompting can dramatically improve the multi-step reasoning abilities of large language models (LLMs). CoT explicitly encourages the LLM to generate intermediate rationales for solving a problem, by providing a series of reasoning steps in the demonstrations. Despite its success, there is still little understanding of what makes CoT prompting effective and which aspects of the demonstrated reasoning steps contribute to its performance. In this paper, we show that CoT reasoning is possible even with invalid demonstrations - prompting with invalid reasoning steps can achieve over 80-90% of the performance obtained using CoT under various metrics, while still generating coherent lines of reasoning during inference. Further experiments show that other aspects of the rationales, such as being relevant to the query and correctly ordering the reasoning steps, are much more important for effective CoT reasoning. Overall, these findings both deepen our understanding of CoT prompting, and open up new questions regarding LLMs' capability to learn to reason in context.
1 Introduction
The paper investigates why Chain-of-Thought prompting works and finds that demonstration reasoning need not be valid for strong performance. Relevance, ordered reasoning steps, and an output format that elicits coherent reasoning appear more important.
- CoT prompting adds rationales to query–answer demonstrations, encouraging LLMs to generate intermediate reasoning before the final answer.
- The study uses ablations that deliberately alter different aspects of demonstrated rationales and measure resulting model-performance changes.
- Over 80-90% of CoT performance remains achievable with completely invalid demonstration reasoning, while inference still produces coherent reasoning toward answers.
- Rationale relevance to the query and correct ordering of reasoning steps are more important for effective CoT prompting than reasoning validity.
- The findings suggest demonstrations may mainly specify an ordered, relevant step-by-step output format that regularizes generation.
2 Background & Study Formulation
The paper frames CoT rationales as combinations of bridging objects and language templates, then asks whether their ground-truth content is necessary. It focuses on which rationale properties enable proper reasoning.
- CoT prompting supplements query–answer demonstrations with rationales that verbalize intermediate reasoning steps for complex tasks.
- Bridging objects are key objects the model traverses to make a successful final prediction.
- For arithmetic, bridging objects are numbers and equations; for factual QA, they are subject and object entities.
- Language templates complement bridging objects by supplying textual hints and relations or predicates that guide derivation.
- The study asks whether ground-truth bridging objects and language templates matter and which rationale aspects support proper reasoning.
3 Experimental Setup
The experiments test rationale ablations on arithmetic reasoning and multi-hop factual QA using established CoT benchmarks and several language models. Evaluation includes both final-answer performance and intermediate rationale quality.
- The study evaluates arithmetic reasoning and multi-hop factual question answering, selecting benchmarks where CoT substantially improves over standard prompting.
- GSM8K contributes 800 uniformly sampled test examples, while Bamboogle contributes all 125 test samples.
- The experiments edit original CoT exemplars slightly to make their structure more consistent and reduce redundancy for ablations.
- The main backbone is InstructGPT-175B text-davinci-002, with additional tests on text-davinci-003, PaLM, and Flan-PaLM using greedy decoding.
- Evaluation combines extrinsic final-answer metrics with intrinsic bridging-object Recall/F1 measures for rationale quality.
- Without instruction fine-tuning, original GPT-3 175B showed little CoT improvement over standard prompting in preliminary experiments.
4 How Much Does Valid Reasoning Matter?
The paper constructs deliberately invalid demonstration rationales and tests their effects across tasks, difficulty levels, metrics, and qualitative rationale behavior. Invalid demonstrations cause only modest degradation, and generated reasoning usually remains sound and pertinent.
- 4.1 Constructing Invalid Chain of Reasoning: Invalid rationales preserve premise facts but alter subsequent derivations so they no longer logically produce the answer.
- 4.1 Constructing Invalid Chain of Reasoning: The construction makes drastic changes to both bridging objects and language templates rather than applying minimal errors.
- 4.1 Constructing Invalid Chain of Reasoning: An arithmetic example replaces the required sum 32 + 42 = 74 with the irrelevant claim 42 - 32 = 10, then continues with unsupported calculations.
- 4.2 Results & Analysis: Over 90% of CoT performance is retained under intrinsic Inter. Recall/Inter.F1 evaluation when demonstrations contain invalid reasoning.
- 4.2 Results & Analysis: The performance drop remains consistent across GSM8K reasoning depths, and invalid reasoning sometimes succeeds where CoT fails: 62/196 GSM8K cases and 6/20 Bamboogle cases.
- 4.2 Results & Analysis: Generated rationales under invalid demonstrations are generally indistinguishable from CoT rationales, with similar error-type distributions.
5 What are the Key Aspects of Chain-of-Thoughts?
The study separates rationale relevance and coherence across bridging objects and language templates, finding both matter for CoT performance. Relevance is especially important for bridging objects, while coherence is especially important for language templates.
- Rationale properties: Invalid rationales can retain query relevance by preserving query information and using language templates about the same entities and relations.Examples retain numbers such as “32” and “42” while discussing Leah, her sister, and chocolates.
- Rationale properties: Each rationale step can remain coherent when later steps follow earlier steps, even if the underlying reasoning is invalid.The ordering of bridging objects and language templates can preserve an apparently coherent reasoning sequence.
- Ablation design: The experiments define relevance as correspondence to query components and coherence as maintaining the correct prerequisite order between steps.Relevance covers bridging objects and language templates; coherence requires later steps not to precede their prerequisites.
- Ablation design: The ablations independently manipulate bridging objects and language templates, including irrelevant substitutions and randomly shuffled component orderings.The design uses four component-by-aspect settings plus broader no-relevance settings, with results reported in Table 2.
- Main findings: Relevance and coherence are key for CoT performance, as most ablations produce large drops that approach or underperform standard prompting.The comparison uses multiple ablation settings against CoT and standard prompting.
- Main findings: 15 out of 20 GSM8K examples under no relevance generated irrelevant rationales, and this setting underperformed standard prompting.The finding supports relevance as necessary for keeping generation directed toward the query.
- Component-specific findings: 39.2 vs. 26.2 Inter. F1 shows incoherent bridging objects outperform irrelevant bridging objects on GSM8K, indicating relevance matters more than ordering there.Bridging-object coverage is below 60% without relevance versus around 80% in other settings.
- Component-specific findings: 14 out of 20 examined examples had incoherent language templates, which negatively affected reasoning.Unlike bridging-object coherence, language-template coherence substantially affects CoT performance.
6 Discussion
The discussion asks whether CoT demonstrations teach reasoning or mainly elicit abilities acquired during pretraining. The authors argue that current CoT success does not by itself establish strong few-shot learning of reasoning.
- Do LLMs learn to reason from CoT demonstrations?: High performance with invalid reasoning demonstrations suggests that what LLMs learn from CoT examples about how to reason may be limited.The discussion treats the result as evidence requiring a distinction between learning reasoning and eliciting existing abilities.
- Do LLMs learn to reason from CoT demonstrations?: Stronger models with more prior task knowledge suffer less from ablations, and none of the tested ablations significantly affects Flan-PaLM.Flan-PaLM was trained on arithmetic reasoning and factual QA in CoT fashion.
- Benchmarking few-shot reasoning: The findings suggest demonstrations may primarily elicit pretrained reasoning skills by specifying an output space or step-by-step format.The authors connect this interpretation to benchmarking the prior knowledge models already possess about evaluated tasks.
7 Related Work
The paper positions itself among CoT variants, analyses of rationale components, and broader studies of in-context learning. Its focus is empirical understanding of the original CoT method in multi-step reasoning.
- CoT prompting: Subsequent CoT work samples diverse reasoning paths, automatically constructs exemplars, and proposes other prompting variants, whereas this paper studies original CoT prompting.The experiments retain the original method’s settings and few-shot exemplars, while noting possible relevance to later variants.
- Rationale analysis: Prior rationale analyses changed limited elements such as incorrect calculations or entities, often leaving counterfactual rationales correct because the query was changed.This paper broadens the examination of rationale aspects through systematic ablations.
- In-context learning: General in-context learning studies find that ground-truth input-label mappings can matter little, while label space, format, and text distribution remain important.This work extends that line of inquiry to sequence-generation tasks requiring multi-step reasoning.
8 Conclusion
The paper concludes that reasoning validity contributes only a small portion of CoT performance, while query relevance and ordered reasoning steps are key. These findings deepen understanding of CoT and raise questions about in-context reasoning.
- Conclusion: Reasoning validity in prompting examples matters only a small portion to performance.The conclusion summarizes the invalid-rationale ablation findings.
- Conclusion: Relevance to the input query and following the order of reasoning steps are key to CoT effectiveness.These are the paper’s principal rationale-level factors identified across its ablations.
- Conclusion: The findings deepen understanding of CoT prompting and open questions about LLMs’ capability to learn reasoning in context.The conclusion frames the contribution as both explanatory and question-generating.
Limitations
The study’s ablations do not cover many symbolic reasoning tasks because their rationales are highly templated, while informal tasks make systematic invalid-reasoning analysis difficult. Its intrinsic rationale evaluation also depends on costly ground-truth bridging-object annotations and may not generalize broadly.
- Task coverage: Highly template-based symbolic tasks offer little variation in rationale steps, making the ablation studies difficult to conduct.For Last letter concatenation, most steps share the same language template and have no meaningful order.
- Invalid reasoning: The invalid-reasoning analysis is challenging because the evaluated tasks are informal and their original rationales are human-written.
- Evaluation: The intrinsic evaluation uses bridging-object correctness, which may not measure rationale quality generally and requires ground-truth objects that are costly to annotate.The paper identifies reference-free intrinsic evaluation as a future direction.
A.1 Chain of Thought Exemplars
The appendix details prompt construction, intrinsic evaluation, and additional model results. Across models, invalid reasoning often has limited impact, whereas relevance and coherence matter more on challenging tasks, with weaker effects as models possess more task knowledge.
- Prompt construction: The authors edit released CoT exemplars to standardize structure, shift premise steps earlier, and expand language templates, with only slight performance changes.Full prompts for the original and edited settings are provided in the appendix tables.
- Evaluation: Intrinsic evaluation measures Recall/F1 of bridging objects, alongside answer accuracy for GSM8K and answer F1 for Bamboogle.The appendix reports intrinsic and extrinsic results across the tested settings and models.
- Additional results: Under text-davinci-003, invalid reasoning generally only marginally harms performance and can outperform CoT on GSM8K under intrinsic evaluation.
- Additional results: On challenging GSM8K, irrelevant or incoherent rationales substantially hurt performance, while Bamboogle mainly retains a need for relevance.
- Additional results: Models with more prior task knowledge suffer less from rationale ablations; Flan-PaLM shows no significant performance impact from any ablation.The paper also reports similar overall observations for non-instruction-tuned PaLM, with some Bamboogle inconsistencies.