PaperlayerView exact revision on arXiv

Source-linked AI summary

When More is Less: Understanding Chain-of-Thought Length in LLMs

Yuyang Wu, Yifei Wang, Ziyu Ye, Tianqi Du, Stefanie Jegelka, Yisen Wang

arXiv:2502.07266v3cs.AIcs.CLcs.LG

TL;DR

The paper asks whether reasoning performance consistently improves as Chain-of-Thought processes grow longer. It combines real-world observations, controlled synthetic experiments, and theoretical analysis to characterize optimal CoT length and its scaling. The results show an inverted U-shaped accuracy relationship, with longer optimal CoTs for harder tasks, shorter ones for more capable models, and practical gains from adaptive length calibration.

  • Problem

    The paper investigates whether longer Chain-of-Thought reasoning consistently improves performance, given evidence that concise CoTs can also be effective.

  • Method

    The paper combines real-world observations, controlled synthetic arithmetic experiments, and theoretical analysis to study optimal CoT length and simplicity bias.

  • Results

    CoT accuracy typically follows an inverted U-shaped curve; harder tasks favor longer optimal CoTs, while more capable models favor shorter ones, with RL also shifting toward shorter CoTs.

  • Takeaways & Limitations

    Training with optimal-length CoTs and filtering excessively long CoTs at inference can improve reasoning performance through adaptive CoT calibration.

  • Takeaways & Limitations

    Real-world CoTs contain uncontrolled variables and varied pretraining, so the paper uses synthetic arithmetic tasks for more precise mechanistic analysis.

Abstract

from arXiv · show

Large Language Models (LLMs) employ Chain-of-Thought (CoT) reasoning to deconstruct complex problems. While longer CoTs are often presumed superior, this paper challenges that notion, arguing that longer is not always better. Drawing on combined evidence from real-world observations, controlled experiments, and theoretical analysis, we demonstrate that task accuracy typically follows an inverted U-shaped curve with CoT length, where performance initially improves but eventually decreases as the number of CoT steps increases. With controlled experiments, we further uncover the scaling behaviors of the optimal CoT length: it increases with task difficulty but decreases with model capability, exposing an inherent simplicity bias where more capable models favor shorter, more efficient CoT reasoning. This bias is also evident in Reinforcement Learning (RL) training, where models gravitate towards shorter CoTs as their accuracy improves. To have a deep understanding of these dynamics, we establish a simple theoretical model that formally proves these phenomena, including the optimal length's scaling laws and the emergence of simplicity bias during RL. Guided by this framework, we demonstrate significant practical benefits from training with optimally-lengthed CoTs and employing length-aware filtering at inference. These findings offer both a principled understanding of the "overthinking" phenomenon and multiple practical guidelines for CoT calibration, enabling LLMs to achieve optimal reasoning performance with adaptive CoTs tailored to task complexity and model capability.

1 Introduction

The paper challenges the assumption that longer Chain-of-Thought reasoning is always better, showing that accuracy typically peaks at an intermediate length. Optimal length depends on task difficulty and model capability, while theory and experiments motivate adaptive calibration.

  • CoT accuracy typically follows an inverted U-shaped curve: both excessively short and excessively long reasoning can reduce performance.Short CoTs leave steps overly complex, whereas long CoTs accumulate errors.
  • Harder tasks tend to require longer optimal CoTs, while more capable models tend to perform best with shorter CoTs.This pattern is described as an inherent simplicity bias in LLM reasoning.
  • During reinforcement learning, models can shift toward shorter CoTs as reasoning accuracy improves.The observed behavior is interpreted as learning more efficient reasoning paths.
  • A controlled synthetic arithmetic study formally characterizes optimal CoT length, its scaling laws, and simplicity bias.The analysis models the trade-off between easier subtasks from decomposition and exponentially accumulating errors.
  • Training on optimal-length CoTs and filtering excessively long CoTs at inference can improve reasoning performance.These findings support adaptive CoT calibration rather than assuming a universally desirable length.

2 Optimal CoT Length and Simplicity Bias in Real-World LLMs

Real-world evaluations show that optimal CoT length varies systematically with both model capability and task difficulty. Excessively long reasoning can substantially underperform the optimal length, while RL training may favor shorter outputs.

  • Evaluation setup: The evaluation samples 60 solutions per question from Qwen2.5 Instruct models on Level 5 MATH problems, defining CoT length by intermediate reasoning steps.The optimal length is the length producing the highest average accuracy.
  • Model capability: The optimal length decreases from 14 steps for the 1.5B model to 4 steps for the 72B model.This indicates that stronger models can consolidate reasoning into fewer steps.
  • Task difficulty: p = 1 × 10−8 ≪0.05 marks a statistically significant positive correlation between task difficulty and optimal CoT length.Difficulty is approximated as 1 − accuracy across sampled solutions.
  • Excessive length: For a 72B model, the accuracy gap between optimal and longest CoTs can reach 40%.The gap grows larger as models become more capable, illustrating the cost of failing to adapt length.
  • Reinforcement learning: As GRPO training converges, average response length can decrease, leaving RL-trained models with shorter CoTs than the base model.The experiment uses Qwen2.5-7B-Instruct on LeetCode-2K.

3 A Controlled Study of CoT Length in Arithmetic Tasks

The controlled arithmetic study varies CoT length, per-step computation, task difficulty, and model capability to characterize optimal reasoning structures. It finds inverted U-shaped performance, longer optimal CoTs for harder tasks, shorter optimal CoTs for stronger models, and RL convergence toward optimal lengths.

  • Experimental Setup: Synthetic arithmetic tasks vary CoT length and per-step operator count while holding total task difficulty fixed.For problems with T operators, CoT length is N ≈ T/t, where t is the approximately constant number of operators processed per step.
  • Experimental Setup: The experiments use arithmetic addition problems and GPT-2 models with varying depths to isolate the effects of task difficulty and model capability.Depth is varied while other hyperparameters are kept fixed, and CoT solutions are automatically synthesized with different lengths.
  • Scaling Laws: Accuracy follows an inverted U-shaped curve with CoT length: both underthinking and overthinking reduce performance.Long CoTs accumulate errors, whereas short CoTs require more complex steps; adaptive lengths better match problem demands.
  • Scaling Laws: Harder tasks favor longer CoTs and larger per-step computation, indicating that both reasoning length and subtask complexity should adapt to task difficulty.The optimal number of steps increases for harder tasks, while the optimal operators-per-step value also tends to rise.
  • Scaling Laws: The optimal CoT length decreases as model capability increases, because stronger models handle more complex subtasks per step.The authors advocate training each model with CoT data whose complexity matches its capabilities.
  • RL Training: During RL on a task with T = 24, sampled CoTs increasingly favor the optimal length N*=5, which achieves 96% peak accuracy.Outcome-based PPO rewards shift the policy toward the CoT structure with the highest task success.

4 Theoretical Analysis: Why an Optimal CoT Length Exists

The paper models CoT accuracy as a trade-off between easier subtasks from decomposition and accumulating errors from additional steps. This yields an optimal length that increases with task difficulty, decreases with model capability, and can emerge automatically through RL.

  • Theoretical formulation: The theoretical framework models CoT as N subtasks over a task with T operators, estimating final accuracy from sub-question and sub-answer correctness.The framework treats T as a proxy for task difficulty and N as the number of reasoning steps.
  • Optimal length: Under linear error assumptions, accuracy initially increases and then decreases with N, producing an inverted U-shaped curve and an optimal CoT length.The model assumes sub-answer error decreases with more steps but accumulates across the chain.
  • Scaling laws: The optimal number of reasoning steps increases monotonically with task difficulty, so harder tasks require longer CoTs for peak performance.The theory also predicts that the optimal number of operators per step increases with task difficulty.
  • Scaling laws: The optimal number of reasoning steps decreases monotonically with model capability, reflecting a simplicity bias toward fewer steps.This theoretical prediction matches the paper’s empirical characterization of more capable models using shorter optimal CoTs.
  • RL simplicity bias: RL converges to the available CoT length with the highest expected reward, thereby automatically discovering the optimal length and typically shifting toward shorter reasoning.The RL analysis reduces the setting to a stateless bandit with binary rewards.
  • Practical implications: Training on optimal-length CoTs can let a smaller 6-layer model outperform a larger 9-layer model trained on mixed-length data, with the gap widening as task difficulty increases.Length-Filtered Vote also consistently outperforms vanilla majority vote on GPQA as sample counts increase.

5 Practical Applications of Optimal CoT Length

The paper applies its optimal-length analysis to both training and inference. Optimal-length training data improves reasoning performance, while length-aware voting filters sampled answers using CoT length and answer uncertainty.

  • Training with optimal-length data: Training on CoT solutions known to be optimal for model size and task difficulty is compared with training on randomly mixed CoT lengths.The testing setup allows models to freely choose their CoT strategy.
  • Training with optimal-length data: Optimal-length training significantly outperforms mixed-length training, allowing a 6-layer model to outperform a 9-layer model trained on randomly chosen lengths.The authors present this as a proof-of-concept for the influence of training-data CoT length.
  • Length-aware inference: Length-Filtered Vote improves majority voting by weighting or selecting answers generated from CoTs within a proper length range.The method groups sampled answers by CoT length before voting.
  • Length-aware inference: On 100 GPQA questions, filtered voting consistently outperforms vanilla majority vote at different sample counts and degrades little as samples increase.The result supports accounting for CoT length during inference.

6 Related Work

The related work situates CoT among structured reasoning methods, overthinking studies, simplicity-bias research, and theoretical analyses of reasoning. The paper’s stated novelty is a unified account of optimal length, scaling behavior, and simplicity bias.

  • Chain-of-Thought reasoning: CoT research uses intermediate steps for task decomposition, with variants including least-to-most prompting, Tree of Thoughts, and divide-and-conquer methods.These approaches treat CoT as a framework for decomposing tasks and solving subtasks.
  • Overthinking in CoT reasoning: Recent work on overthinking studies excessively long CoTs and mitigation strategies, while this paper emphasizes the broader inverted-U relationship between length and accuracy.The paper also addresses how optimal length scales with task difficulty and model capability.
  • Paper positioning: The paper claims a systematic explanation of overthinking supported by controlled experiments and theoretical analysis, together with principled CoT-design guidelines.Its distinctive focus is the joint behavior of length, model capability, and task difficulty.
  • Simplicity bias: The paper connects CoT simplicity bias to broader machine-learning findings that favor simpler functions or lower-complexity solutions.It extends this discussion to generated reasoning paths.
  • Theoretical understanding: Theoretical CoT studies analyze computational advantages, error correction, information gain, and learning stability from different formal perspectives.These lines of work provide theoretical context for formalizing CoT effectiveness.

7 Conclusion

The paper concludes that CoT length should be calibrated rather than maximized: optimal length balances task decomposition against error accumulation. It links this optimum to model capability and task difficulty, while noting that precise real-world estimation remains unresolved.

  • 7 Conclusion: CoT accuracy typically follows an inverted U-shaped curve, so excessively long or short reasoning can reduce performance.Longer chains accumulate errors, whereas shorter chains leave subtasks overly complex.
  • 7 Conclusion: More capable models favor shorter effective reasoning paths, while harder tasks generally require longer optimal CoTs.The paper frames this pattern as a simplicity bias and supports aligning CoT length with task difficulty and capability.
  • 7 Conclusion: Accurately estimating optimal CoT length in diverse real-world settings remains challenging, despite theoretical and empirical heuristics for coarse approximation.The paper leaves more precise estimation and adaptive selection methods for future work.
  • B.1 Problem Formulation: The controlled study uses synthetic addition tasks with variable reasoning-step counts to isolate how CoT structure affects performance.Problems are represented as computation trees, and solutions vary in step granularity while preserving the underlying arithmetic task.
  • B.1 Problem Formulation: The synthetic design varies CoT length and subtask complexity while controlling total task difficulty, enabling systematic comparisons across reasoning structures.Each step processes an approximately constant number of operators within a solution, and alternative valid solutions can use different granularities.
  • 7 Conclusion: The study uses in-context examples to induce varied solution lengths and counts reasoning steps rather than tokens as its primary CoT-length measure.This choice reflects reasoning decomposition more directly than raw output length.

C.2 More Experimental Results

Supplementary experiments extend the relationship between task difficulty and optimal CoT length across models and datasets. They report consistent trends on MATH and similar behavior on MMLU STEM.

  • C.2 More Experimental Results: All evaluated models show a significant correlation between task difficulty and optimal CoT length on MATH datasets.The supplementary evaluations use Qwen2.5 and Llama3.1 Instruct model series.
  • C.2 More Experimental Results: MMLU STEM experiments with Qwen2.5 Instruct models exhibit trends similar to those observed on MATH.The results are reported in Figures 8 and 9 under the same experimental settings.

D Supplementary Details on Real World Experiment for RL Simplicity Bias

The supplementary RL experiments use Qwen2.5-7B-Instruct with GRPO and R1-like prompting on LeetCode-2K. The setup evaluates how training affects CoT simplicity bias.

  • D Supplementary Details on Real World Experiment for RL Simplicity Bias: The RL study uses Qwen2.5-7B-Instruct, Group Relative Policy Optimization with R1-like prompting, and LeetCode-2K.The passage identifies the base model, training method, prompting style, and dataset for Figure 1b.
  • D Supplementary Details on Real World Experiment for RL Simplicity Bias: Table 1 provides the hyperparameter settings for the real-world RL experiments with Qwen2.5 Instruct models.The supplied passage identifies the table’s scope but does not enumerate its settings.

E Additional Synthetic Experiment Details

The supplementary synthetic analysis specifies assumptions under which the paper’s optimal-length conclusions generalize. It connects subtask error, model capability, and task difficulty to the resulting scaling behavior.

  • E Additional Synthetic Experiment Details: The general analysis assumes subtask error decreases with more decomposition, is convex in decomposition, decreases with model capability, and increases with task difficulty.These assumptions encode easier subtasks from finer decomposition, diminishing decomposition benefits, stronger models, and harder total tasks.
  • E Additional Synthetic Experiment Details: Under these assumptions, the general accuracy function retains an optimal CoT length when its maximum occurs above one step.The theorem states that this optimum has a lower bound related to model capability and task difficulty.
  • E Additional Synthetic Experiment Details: The theorem characterizes optimal length through the inverse of the subtask error function with respect to the number of reasoning steps.The inverse-function formulation links the optimum to how decomposition changes subtask error.
  • E Additional Synthetic Experiment Details: The generalized results preserve the qualitative scaling laws: stronger models favor shorter optimal CoTs, while harder tasks favor longer ones.The paper presents these conclusions as supporting the empirically observed simplicity bias and inverted U-shaped performance.

F.2 Random Error

The random-error analysis preserves the trade-off behind an optimal CoT length: decomposition simplifies subtasks, but accumulated errors eventually reduce final accuracy. Under the paper’s assumptions, accuracy therefore rises and then falls with the number of reasoning steps.

  • Random-error model: The model assumes identical sub-question and sub-answer error rates, then extends the analysis with stochastic error-rate variables.The paper notes that identical-error assumptions are unrealistic and introduces Beta-distributed sub-question and sub-answer error rates as a variant.
  • Stochastic extension: The stochastic extension shows that the fundamental trade-off leading to an optimal CoT length persists despite dependent, variable subtask errors.The paper explicitly states that stochasticity does not remove the initial-increase and eventual-decrease pattern.
  • Random-error model: The final-accuracy analysis models each subtask as containing t = T/N operators and assumes equal difficulty across subtasks.This setup makes the trade-off between decomposition and error accumulation analytically tractable.
  • Accuracy trade-off: The theoretical accuracy function A(N) initially increases and then decreases as CoT steps N grow, yielding an inverted U-shaped relationship.The same pattern is obtained for the general accuracy function under the stated noise and subtask-error assumptions.
  • Optimal length: The optimal CoT length is characterized analytically using the smaller real branch W−1 of the Lambert W function.The solution selects W−1 because the relevant solution satisfies Z < −1.

G.3 Proof of Corollary 4.4

The analysis proves how optimal CoT length scales with task difficulty and model capability, and explains RL selection of reasoning lengths as optimization over a stationary bandit. Harder tasks favor more steps, stronger models favor fewer steps, and sufficiently small-step gradient ascent converges to an optimal length.

  • Scaling laws: The optimal CoT length N* increases monotonically with task difficulty T, so harder tasks require more reasoning steps for optimal performance.The same corollary also states that the optimal operators per step increase with task difficulty.
  • Scaling laws: The optimal number of operators per step t* = T/N*(M, T) increases monotonically with task difficulty T.The paper gives t* = M(1 + 1/Z) and relates this result to the envelope curve.
  • Scaling laws: The optimal CoT length N* decreases monotonically with model capability M, reflecting a simplicity bias toward fewer reasoning steps.This scaling law is stated as the third conclusion of Corollary 4.4.
  • RL convergence: With sufficiently small step size, gradient ascent converges to a deterministic policy that places all probability on a CoT length maximizing A(N).When multiple lengths tie for the maximum, the limit may select any maximizing arm; mixtures over sub-optimal arms are unstable.
  • RL convergence: RL length selection is modeled as a k-armed stochastic bandit whose stationary reward for action Ni is the probability A(Ni) of a correct final answer.A(Ni) does not depend on time or the agent’s past actions, making the bandit stationary.
Loading 2502.07266v3…