Source-linked AI summary
The Impact of Reasoning Step Length on Large Language Models
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, Mengnan Du
TL;DR
The paper addresses the lack of systematic understanding of how reasoning-step length affects Chain-of-Thought performance. Through experiments that expand, compress, and alter rationale steps, it finds that longer chains generally improve reasoning, while compression causes substantial performance declines and even incorrect rationales can remain effective when their length is maintained.
Problem
The precise mechanics of why Chain-of-Thought and its variations work effectively remain unsettled, leaving improvements reliant on trial and error.
Method
The study manipulates reasoning steps in standardized few-shot Chain-of-Thought demonstrations and measures model performance across datasets, including experiments with compressed and incorrect rationales.
Results
Longer reasoning chains improve performance across datasets, compression produces a notable decline toward zero-shot performance, and favorable outcomes can persist with incorrect rationales when inference length is maintained.
Takeaways & Limitations
Reasoning-chain length is a key factor for refining Chain-of-Thought strategies, including when the chain contains misleading information.
Takeaways & Limitations
The study does not deeply analyze the mechanisms behind the benefits of increased reasoning steps and is limited to certain datasets and models such as GPT-3.5 and GPT-4.
Abstract
from arXiv · showhide
Chain of Thought (CoT) is significant in improving the reasoning abilities of large language models (LLMs). However, the correlation between the effectiveness of CoT and the length of reasoning steps in prompts remains largely unknown. To shed light on this, we have conducted several empirical experiments to explore the relations. Specifically, we design experiments that expand and compress the rationale reasoning steps within CoT demonstrations while keeping all other factors constant. We have the following key findings. First, the results indicate that lengthening the reasoning steps in prompts, even without adding new information into the prompt, considerably enhances LLMs' reasoning abilities across multiple datasets. Alternatively, shortening the reasoning steps, even while preserving the key information, significantly diminishes the reasoning abilities of models. This finding highlights the importance of the number of steps in CoT prompts and provides practical guidance to make better use of LLMs' potential in complex problem-solving scenarios. Second, we also investigated the relationship between the performance of CoT and the rationales used in demonstrations. Surprisingly, the result shows that even incorrect rationales can yield favorable outcomes if they maintain the requisite length of inference. Third, we observed that the advantages of increasing reasoning steps are task-dependent: simpler tasks require fewer steps, whereas complex tasks gain significantly from longer inference sequences. The code is available at https://github.com/MingyuJ666/The-Impact-of-Reasoning-Step-Length-on-Large-Language-Models
1 Introduction
The paper examines how reasoning-step length affects CoT effectiveness, addressing limited understanding of CoT mechanics. Experiments report that longer reasoning chains improve performance, including when added steps introduce no new information.
- CoT improves multi-step problem solving through logical, sequential reasoning chains.
- The field lacks consensus on how CoT works, leaving systematic improvement methods underdeveloped.
- The study varies reasoning-step length while controlling other factors and avoiding additional knowledge.
- Longer reasoning steps improve LLM reasoning across datasets, whereas shorter chains significantly reduce performance despite preserving key information.
- Increasing zero-shot CoT steps through an expanded prompt noticeably improves reasoning, particularly on mathematical datasets.
2 Related Works
Related work frames CoT as a prompting strategy that improves reasoning through intermediate steps and connects it to automated demonstrations, branching reasoning, semantic analysis, and theoretical accounts.
- In-context learning places input-output examples directly in prompts, enabling language models to perform tasks without parameter updates.
- CoT prompting adds logical reasoning steps to demonstrations, improving reasoning capabilities and producing articulated intermediate sequences.
- Tree of Thoughts extends CoT by exploring multiple intermediate reasoning paths for more deliberate decisions.
- Prior studies examine how prompt components, semantics, demonstration selection, and question variation affect CoT reasoning.
- Theoretical work characterizes CoT as multi-step computation and studies its sufficiency for complex questions and dynamic planning.
- Empirical research reports that CoT improvements scale with the number of intermediate steps and evaluates robustness to perturbations.
3 Analyzing Methods
The method isolates reasoning-step length by expanding or compressing CoT rationales while holding other factors constant. It applies standardized prompting strategies across zero-shot and few-shot settings.
- The experiments systematically vary only the number of reasoning steps without introducing new reasoning information.
- Zero-Shot-CoT adds a phrase such as “Let’s think step by step,” while Few-Shot-CoT supplies reasoning examples in the prompt.
- The zero-shot prompt is extended to require more steps because additional example reasoning cannot be inserted.
- Reasoning chains are expanded through repeated question reading, state repetition, word interpretation, self-verification, and equation construction.
- Table 1 illustrates the Think About The Word strategy, which asks the model to interpret selected words in context.
- Each added strategy is treated as one additional reasoning step for quantitative analysis.
4 Experimental Results
The experiments examine how reasoning-step length, rationale correctness, compression, model size, and question content relate to CoT performance. Across the reported studies, longer reasoning chains generally improve accuracy, while compression reduces performance.
- Experimental setup: The study evaluates CoT across eight datasets and three models, using controlled experiments on reasoning-step length and prompt variants.The experimental setup includes MultiArith, GSM8K, AQuA, SingleEq, SAVMP, Letter, Coin, and Strategyqa, evaluated with text-davinci-002, GPT-3.5-turbo-1106, and GPT-4.
- Steps and accuracy: Additional reasoning steps improve LLM reasoning ability across all datasets, with accuracy showing a linear relationship to CoT complexity through up to six added steps.The relationship is measured with GPT-3.5-turbo-1106 after standardizing the thought-chain process.
- Rationale correctness: Changing one rationale step to an incorrect answer has task-dependent effects: arithmetic performance changes minimally, whereas logic chains can fragment.The wrong-rationale experiment preserves the overall chain structure and uses each dataset’s optimal step count.
- Reasoning-step compression: Compressing rationale steps in Auto-CoT and Few-Shot-CoT causes a notable performance decline, reaching approximately the zero-shot level.The compression experiment randomly merges consecutive rationale sentences while aiming to preserve information.
- Model size: On GSM8K, the model with the weakest initial performance receives the strongest boost, while GPT-4 shows the highest tolerance with no performance decrease.The comparison examines text-davinci-002, GPT-3.5-turbo-1106, and GPT-4 at their peak-performing average inference-step counts.
- Question content: A preliminary case study suggests that reasoning-step length, rather than the nature of rationale questions, predominantly influences model reasoning ability.The investigation uses MultiArith and GSM8K with Auto-CoT and Few-Shot-CoT.
5 Conclusions and Future Work
The paper reports that longer reasoning chains improve LLM performance, even with misleading information, and proposes further analysis of long versus short inference steps.
- Longer reasoning chains improve model performance, even when they contain misleading information.The paper concludes that chain length is more crucial than factual accuracy for effective problem-solving.
- The study provides experimental analysis of how manipulating CoT reasoning steps affects model performance.
- The authors propose analyzing long and short reasoning steps through visualization of activation patterns.This planned analysis is intended to examine whether longer inferential steps correlate with broader neuronal engagement.
6 Limitation
The study is limited by its experimental focus and restricted coverage of mechanisms, datasets, and models.
- The study does not deeply analyze why increasing reasoning steps improves performance.The authors identify missing theoretical and explainability analyses of LLM internal workings.
- The evaluation is limited to certain datasets and models, including GPT-3.5 and GPT-4.The authors note that more diverse tasks and newer models could reveal different trends.
- Further research could examine the underlying mechanisms through theoretical or explainability analysis.
A Appendix
The appendix references Tables 6, 7, and 8.
- Table 6 is referenced in the appendix.
- Table 7 is referenced in the appendix.
- Table 8 is referenced in the appendix.