PaperlayerView exact revision on arXiv

Source-linked AI summary

What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoT

Yunzhen Feng, Julia Kempe, Cheng Zhang, Parag Jain, Anthony Hartshorn

arXiv:2509.19284v1cs.LG

TL;DR

The paper asks what characterizes effective CoTs beyond length and review, and evaluates these properties across ten models on math and scientific reasoning. It introduces graph-based FSF and finds that lower failure is more informative than longer or more heavily reviewed traces, with interventions supporting its practical relevance.

  • Problem

    It remains unclear whether longer CoTs and increased review improve correctness, or whether token-level measures obscure deeper structural properties of reasoning.

  • Method

    The study evaluates length, Review Ratio, and graph-based Failed-Step Fraction across ten reasoning models and uses reranking and failed-branch editing as causal tests.

  • Results

    Shorter CoTs and lower Review Ratio are associated with higher accuracy, while FSF more consistently predicts correctness and FSF-based selection yields the largest gains.

  • Takeaways & Limitations

    Effective CoTs are characterized by fewer failed branches, supporting structure-aware test-time scaling rather than indiscriminate generation of longer CoTs.

  • Takeaways & Limitations

    The analysis assumes that a CoT reflects the model’s reasoning, while CoT faithfulness is beyond the study’s scope.

Abstract

from arXiv · show

Large reasoning models (LRMs) spend substantial test-time compute on long chain-of-thought (CoT) traces, but what *characterizes* an effective CoT remains unclear. While prior work reports gains from lengthening CoTs and increasing review (revisiting earlier steps) via appended *wait* tokens, recent studies suggest that shorter thinking can outperform longer traces. We therefore conduct a systematic evaluation across ten LRMs on math and scientific reasoning. Contrary to the "longer-is-better" narrative, we find that both naive CoT lengthening and increased review are associated with *lower* accuracy. As CoT unfolds step by step, token-level metrics can conflate verbosity with process quality. We introduce a graph view of CoT to extract structure and identify a single statistic-the *Failed-Step Fraction (FSF)*, the fraction of steps in abandoned branches-that consistently outpredicts length and review ratio for correctness across models. To probe causality, we design two interventions. First, we rank candidate CoTs by each metric at test time, where FSF yields the largest pass@1 gains; second, we edit CoTs to remove failed branches, which significantly improves accuracy, indicating that failed branches bias subsequent reasoning. Taken together, these results characterize effective CoTs as those that *fail less* and support *structure-aware* test-time scaling over indiscriminately generating long CoT.

1 Introduction

The paper reexamines whether longer CoTs and more review improve reasoning, finding that effective reasoning is better characterized by structural quality than token-level quantity.

  • The study evaluates length, review behavior, and structural properties to identify what characterizes effective reasoning.It analyzes ten reasoning models across math and scientific reasoning tasks.
  • Shorter reasoning traces and lower Review Ratio are associated with higher accuracy within the same question across models and datasets.
  • FSF-based reranking produces the largest and most consistent test-time gains, reaching up to 10% accuracy improvement on AIME.Length- and Review Ratio–based selection provide smaller benefits.
  • Removing failed branches substantially increases accuracy, indicating that failed attempts bias subsequent reasoning and exploration.The editing intervention suggests models do not fully unsee earlier mistakes when backtracking.
  • Across ten models on math and scientific reasoning, FSF more consistently predicts correctness than CoT Length or Review Ratio.Lower FSF correlates with higher accuracy across difficulty strata and models.

2 Related Work

Prior work links test-time scaling to longer CoTs and review, but conflicting findings motivate broader analysis of when length helps and how reasoning structure should be represented.

  • Test-time scaling research uses long step-by-step CoTs and appended or learned continuation prompts to increase generation and review.S1 appends wait tokens, while follow-up work uses learned “continue thinking” prompts.
  • Recent studies report that suppressing wait can preserve accuracy, continually adding wait can degrade performance, and longer CoTs are not always better.
  • These mixed results and the limited model coverage of some studies motivate a systematic study across ten models.
  • Prior graph approaches represent reasoning steps and dependencies, but extracting faithful graphs from existing CoTs remains challenging.Methods include prompt scaffolds, hidden-embedding clustering, and connecting nodes in visitation order.
  • Verification, backtracking, and other cognitive behaviors have been studied as important for reasoning, but they are difficult to measure reliably and often use synthetic tasks.

3 Framework

The framework evaluates length, review, and graph structure across math and scientific reasoning, defining Failed-Step Fraction as the fraction of graph nodes in failed or abandoned attempts.

  • The framework asks whether increasing CoT length or Review improves accuracy and which structural properties underlie their effects.
  • The study uses HARP for mathematical reasoning and GPQA-Diamond for scientific reasoning, with human-labeled difficulty levels.HARP contains 5,409 math questions, with 50 sampled from each of six difficulty levels.
  • CoT Length is measured in characters, while Review Ratio is computed from character-level chunks labeled progress or review.Progress advances the active reasoning frontier; review checks, restates, deletes, or rewinds existing material without advancing it.
  • The reasoning-graph procedure converts each CoT into graph nodes and edges, with nodes labeled as successful or failed attempts.The extracted graphs generally match the natural-language traces, and the procedure compiled without error in 100% of cases.
  • Failed-Step Fraction is the fraction of reasoning nodes marked as failed or abandoned, providing a graph-level measure of structural failure.

4 Correlation Analysis

Across math and scientific reasoning, shorter CoTs, lower Review Ratio, and lower Failed-Step Fraction generally correlate with higher accuracy after controlling for question-level confounders. FSF is the strongest and most consistent predictor across models, datasets, and difficulty levels.

  • Overall Conditional Correlations: The analysis controls for question-level heterogeneity using conditional correlations and corroborates significant associations with a Bayesian GLMM.The conditional test residualizes each metric and correctness by question-level means, while the GLMM uses question-level random intercepts.
  • Overall Conditional Correlations: Shorter CoTs and lower Review Ratio generally correlate with higher accuracy within the same question.These negative associations are observed across datasets and most models, with a notable positive Review Ratio trend for Claude 3.7 on math reasoning.
  • Overall Conditional Correlations: FSF correlates significantly with accuracy for every model on both math and scientific reasoning tasks.Its association is more consistent than those of Length or Review Ratio, including for Claude, which uniquely benefits from higher Review Ratio.
  • Conditional Correlations by Difficulty Level: Metric-accuracy correlations are most consistently significant on harder questions, while easier items show mixed patterns.On HARP, the clearest effects occur at levels 4, 5, and 6; GPQA shows consistent patterns on its Hard Undergraduate and Hard Graduate splits.
  • Conditional Correlations by Difficulty Level: FSF is the strongest and most consistent predictor of accuracy across models, datasets, and difficulty levels.When significant correlations emerge, FSF is consistently negative and more prevalent than Length or Review Ratio.
  • Overall Conditional Correlations: Additional graph metrics and stylistic features show weaker or model-dependent relationships with accuracy than FSF.Other graph metrics are mostly significant only on math tasks, while stylistic features do not correlate consistently across models.

5 From Correlation to Causality

Two interventions test whether CoT metrics are causally useful: reranking candidate traces at test time and editing failed branches before continuation. FSF is the strongest selector, while removing failed branches substantially improves accuracy, supporting structure-aware scaling over indiscriminate lengthening.

  • 5.1 Test-time selection: Test-time selection reranks a fixed set of 64 candidates by each metric and evaluates the top-ranked trace with pass@1.The compared selectors are FSF, Length, Review Ratio, and random selection; uncertainty is estimated by bootstrap resampling.
  • 5.1 Test-time selection: FSF is the strongest selector across models and datasets, with roughly 5–13% AIME 2025 gains over random selection and significant gains on GPQA-Diamond.FSF also improves math accuracy by up to 12% when Claude 3.7 generates, estimates, and selects its own candidates.
  • 5.2 Modifying the CoT: Removing failed branches from incorrect CoTs increases continuation accuracy by roughly 8–14% at either the first or last failed point.A concise summary of the failed branch also helps, but less than removing it entirely; Table 1 reports the edited-CoT comparison.
  • 5.2 Modifying the CoT: The editing results indicate that long failed branches bias subsequent exploration even after backtracking, so models do not fully unsee past mistakes.The intervention compares original prefixes, reduced prefixes, and prefixes augmented with summaries of failed branches.
  • Conclusion: Together, the interventions support quality-aware test-time scaling through structure-aware selection and targeted branch pruning or summarization rather than indiscriminate CoT lengthening.The paper characterizes failed branches as a direct target for context management.

6 Discussion

The study finds that shorter CoTs and less review are associated with higher accuracy, while FSF is the strongest predictor across models and datasets. Causal interventions further show that selecting or editing away failed branches improves performance, though CoT faithfulness remains outside the study’s scope.

  • 6 Discussion: Lower FSF reliably correlates with higher accuracy across all 10 models on math and scientific reasoning.FSF is presented as the strongest predictor in the study, outperforming token-level Length and Review Ratio.
  • 6 Discussion: The paper argues for structure-aware test-time scaling rather than indiscriminately increasing token count.The proposed direction is to manage failure propagation, while the study does not analyze CoT faithfulness or how training induces low-FSF reasoning.
  • 6 Discussion: The analysis assumes that a CoT reflects the model’s reasoning, while assessing CoT faithfulness is left for future work.Reported correlations are measured at test time, and the relationship between training and test-time behavior remains unresolved.

B.2 Evaluation

The evaluation combines answer parsing, review and motivation annotation, conditional correlations, and GLMM analysis. The GLMM results largely match the conditional-correlation findings, providing a second line of evidence for the reported associations.

  • B.2 Evaluation: Review annotations label chunks as progress or review, with review defined as checking or rewinding existing material without advancing the reasoning frontier.The annotation pipeline uses the current chunk together with five preceding and five following chunks as context.
  • B.2 Evaluation: The review annotation pipeline has low type I error and treats missed review labels as a lower bound for review.The authors compare model annotations with human labels using traces from DeepSeek R1 and Qwen 3 235B.
  • B.2 Evaluation: Motivation is scored from clear, semi-clear, and unclear review rationales, weighted 1.0, 0.5, and 0.0 respectively.The score averages these character-level values over Review spans.
  • B.2 Evaluation: The GLMM estimates a metric’s association with correctness while modeling question-level heterogeneity through a question-specific random intercept.The model uses a logistic link and derives Wald-style p-values from the posterior mean and standard deviation of β1.
  • B.2 Evaluation: GLMM coefficients largely match the conditional-correlation findings in sign and significance.The agreement is reported as a second line of evidence supporting the correlation results.

C.4 Other Metrics

The paper evaluates several review-position, review-frequency, motivation, and graph-depth metrics. Across models, many effects are unstable, while math shows stronger correlations and Review-Chunk Fraction is weaker than FSF.

  • C.4 Other Metrics: The study evaluates Review Centroid, Review Chunk Fraction, Review→Progress Switch Count, Motivation Score, First Failed-Step Depth, and Reasoning Depth.Figure 8 reports correlations for these metrics, with significance indicated by the figure’s color encoding.
  • C.4 Other Metrics: Review-Chunk Fraction has weaker and more unstable associations with accuracy than FSF.This comparison suggests that graph-level metrics capture predictive structure more reliably than review frequency alone.
  • C.4 Other Metrics: Correlations are stronger and more frequent in math reasoning than in general scientific reasoning.The pattern is reported among the additional metrics evaluated in Figure 8.
  • C.4 Other Metrics: Review position often behaves as a model-specific stylistic feature rather than a general predictor of accuracy.The reported conditional correlations for Review Centroid and related measures are not consistent across models.
  • C.4 Other Metrics: Model-level generation styles can bias cross-model comparisons, motivating within-model correlation estimates replicated across models.The authors report additional graph-based, entropy, and progressiveness analyses in appendices.

D Graph Metrics

The paper converts CoTs into reasoning graphs containing successful and failed steps, then extracts structural features describing failures, branching, paths, and information flow. Failed-Step Fraction is the central failure-density measure used to characterize reasoning structure.

  • D Graph Metrics: The graph representation preserves all reasoning attempts, branching new attempts from their starting ancestor rather than from dead ends.Nodes represent distinct reasoning steps, with successful and failed attempts encoded by separate colors.
  • D Graph Metrics: The extracted feature set includes information propagation, cross-reference density, path coverage, endpoint reachability, and error depth.These features measure how information moves through the graph and how directly it connects the problem to the answer.
  • D Graph Metrics: Failed-Step Fraction measures the proportion of graph nodes marked as failed steps.It is the paper’s central graph-level measure of the density of failed attempts.
  • D Graph Metrics: Reasoning Depth is the shortest path length from the problem node to the answer node, while Orphaned Steps capture isolated nodes.The graph also records total node count as Total Steps.
  • D Graph Metrics: Additional structural features quantify branching tendency, failed children, recovery distance, successful branching, and flow coherence.These metrics describe how failures are corrected and how many nodes participate in problem-to-answer paths.

D.2 Extra Graphical Results

Extra graphical metrics show weaker and less generalizable correlations with correctness than FSF, with the clearest effects concentrated in mathematical reasoning.

  • Several extra graphical metrics correlate significantly with accuracy, but their effects are weaker than FSF and sparse in scientific reasoning.

E.1 Test-time Selection

Test-time selection on HARP reproduces earlier gains for several models, while Qwen behaves anomalously; FSF-based selection is generally the strongest supported strategy.

  • On HARP, 180 disjoint questions receive 64 generated CoTs each, with one candidate selected under each metric.The sample includes 60 questions from each of Levels 4, 5, and 6.
  • Claude 3.7, Grok 3 mini, and Deepseek R1 reproduce the earlier selection improvements, whereas Qwen reaches 0 accuracy under minimum-length or minimum-review selection.The passage attributes Qwen’s anomalous behavior to likely train–evaluation contamination.
  • The intervention requires extracting graph steps, locating failed-branch starts, and mapping nodes back to exact reasoning text before branch removal.The extraction procedure uses node identifiers and the first 20 words of each step to support precise matching.
  • FSF-based selection consistently identifies higher-quality generations at test time when excluding Qwen.The figure reports bootstrap-standard-deviation error bars and flags Qwen’s results as anomalous.
  • Failed branches are traced from the most recent successful node where an alternative path diverged, identifying the point at which the branch should be removed.

F.1 Progressiveness and Entropy.

Progressiveness and entropy track how confidence evolves as reasoning is truncated, but these measures do not distinguish correct from incorrect final answers reliably. Shorter CoTs instead show higher accuracy across difficulty levels.

  • The analysis estimates answer distributions from eight sampled continuations at multiple prefix truncations after prompting the model to conclude.
  • Answer entropy measures confidence and information accumulation, while progressiveness summarizes information gain through an area-under-the-curve calculation.
  • Models become confident even when wrong: entropy declines similarly across truncation conditions, with low terminal entropy regardless of correctness.Because of this behavior, progressiveness and answer entropy are excluded from the correlation analyses.
  • Short CoTs achieve higher accuracy than long CoTs across all difficulty levels and truncation rates.The comparison partitions each question’s traces into short and long groups by length.
  • Model-level plots compare average Length, Review Ratio, and Failed-Step Fraction with accuracy, but no uniformly consistent cross-model pattern appears.FSF provides the clearest cross-model signal, especially on GPQA-Diamond, where lower FSF tends to accompany higher accuracy.
Loading 2509.19284v1…