PaperlayerView exact revision on arXiv

Source-linked AI summary

Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning

Kevin Du, Alexander Hoyle, Laura Ruis, Acyr Locatelli

arXiv:2609.04194v1cs.CLcs.LG

TL;DR

The paper asks whether the text of CoT reasoning steps encodes their functional importance, a question made consequential by widespread use of judges, critics, and process reward models. It operationalizes importance as advantage estimated with Monte Carlo rollouts and evaluates whether judges can decode it. Judges recover importance only partially: performance is stronger for incorrect responses than correct ones, supporting caution against equating legibility with interpretability.

  • Problem

    The paper examines whether reasoning-step importance is encoded in CoT text, given the use of judges, critics, and process reward models to analyze and improve reasoning traces.

  • Method

    The paper defines step importance as advantage—the change in expected reward from including a step—and estimates it with Monte Carlo rollouts before evaluating text-based judges and critics.

  • Results

    Across models, datasets, and prompting strategies, judges beat a prevalence baseline but remain below a noise ceiling; fine-tuned critics perform much better on incorrect than correct responses.

  • Takeaways & Limitations

    Step importance is only partially recoverable from CoT text, cautioning against treating reasoning-trace legibility as interpretability.

  • Takeaways & Limitations

    The analysis primarily uses non-thinking models, and thinking-model responses longer than 600 steps are filtered, leaving 81% of responses.

Abstract

from arXiv · show

Reasoning traces from chain-of-thought models appear to offer a legible window into how a model arrives at its answer. A growing body of work treats them as such, using LLM judges to diagnose errors, evaluate faithfulness, and provide step-level supervision via process reward models and generative critics. These practices rely on the text of a reasoning step carrying information about its functional role. But does the text actually encode information about which reasoning steps matter? We operationalize the importance of a reasoning step as its advantage: the change in expected reward, e.g., producing the correct final answer, from including that step, estimated via Monte Carlo rollouts. Basing ground truth on these estimates, we evaluate whether LLM judges can identify high-advantage steps and find that sufficiently capable LLMs can outperform a prevalence baseline but fall well short of a noise ceiling. Fine-tuning a model as a step-level critic yields strong improvement for incorrect responses but remains distant from ceiling for correct responses, suggesting that step importance is only partially recoverable from the text of the reasoning trace. Our findings contribute to a growing body of chain-of-thought faithfulness work that cautions against treating the legibility of reasoning traces as interpretability, especially with implications for process reward modeling.

1 Introduction

The paper asks whether reasoning-step importance is measurable and recoverable from CoT text, motivated by the growing use of judges and critics to interpret reasoning traces. It operationalizes importance as advantage and finds that judges recover it only partially, especially for correct responses.

  • Semantically similar self-checking steps can differ sharply in impact, with one example raising correct-answer probability from ≈52% to ≈94%.
  • The paper asks whether a reasoning step’s importance can be appropriately measured and decoded from its text alone.
  • Advantage operationalizes step importance as how much including a step improves the likelihood of the final answer.
  • Out-of-the-box judges improve with scale over a prevalence baseline but remain well below a noise ceiling.
  • Fine-tuned critics identify consequential steps reasonably well in incorrect responses but poorly in correct responses.
  • The findings caution against treating legible CoT text as interpretability and encourage deeper measurement of reasoning-step importance.

2 Analyzing Language Model Reasoning with Judges

CoT is widely treated as an interpretable record of model reasoning, but prior faithfulness tests and this paper’s framing question whether step text reveals its functional importance.

  • Reasoning steps are commonly segmented and classified as interpretable units, including cognitive behaviors and judge-assigned categories.
  • Prior perturbation-based studies find that CoTs may omit influential prompt hints, raising doubts about whether traces faithfully represent reasoning.
  • Existing faithfulness methods usually assess complete responses rather than the function of individual reasoning steps toward the output.

3 Operationalizing Importance as Advantage

The paper models a reasoning trace as a sequential decision process and defines a step’s importance through its advantage under a reward function.

  • The framework characterizes how critical a sampled step was for reaching the final or correct answer and complements faithfulness measures.
  • A reasoning trace consists of contiguous generated substrings, while each state concatenates the prompt with preceding steps.
  • Importance can target correctness or matching the original final answer, with analyses ordinarily using self-advantage.
  • Advantage is the expected reward after committing to a step minus the expected reward from the preceding state.
  • Monte Carlo rollouts estimate value before a step and Q-value after committing to it, yielding an empirical advantage estimate.

4 Identifying Consequential Reasoning Steps

A step is consequential when its advantage exceeds a specified effect size, and the paper detects such changes using changepoints in the value trajectory.

  • A step is consequential when |Aπ(st, at)| exceeds δ; the illustrative procedure uses δ = 0.1 and 95% posterior confidence.
  • The method models value trajectories as piecewise constant and uses PELT with an exact binomial cost to detect changepoints.
  • Detected changepoints are labeled consequential when Beta-posterior segment means imply a sufficiently large jump in advantage.
  • The procedure labels a completion consequential if at least one of its steps is consequential.
  • Independent per-step z-tests produce many false positives across long responses, while multiplicity correction reduces power as CoTs grow longer.

5 Characterizing Advantage As Importance Empirically

The paper uses self-advantage to characterize which reasoning steps affect model behavior, finding that consequential steps vary by reasoning style, task difficulty, and answer trajectory. Cue-based prompts can also produce mostly uninformative traces.

  • Qualitative Examples: Consequential steps are more common with thinking mode off, while active computation and uncertainty management are most consequential in correct thinking-model responses.In incorrect responses, multiple step types steer behavior, though final answer emission remains prominent.
  • Comparing Reasoning Patterns for Model Scale and Thinking Mode: Performance gains from thinking and scale primarily reflect answers already high-value before reasoning begins, with high-throughout responses rising from 24% to 61% with thinking and 25% to 39% across sizes.These increases absorb mass almost entirely from never-high responses, rather than from mid-trajectory recovery.
  • Comparing Reasoning Patterns for Model Scale and Thinking Mode: Incorrect answers are rarely locked in from the start: only 1–3% of non-thinking and 12.4% of thinking responses are high throughout, while thinking reaches wrong answers gradually 59% versus suddenly 24%.Non-thinking models more often reach incorrect answers suddenly, at 48–56%, than gradually, at 39–46%.
  • Against Dataset Difficulty: Harder datasets usually contain more consequential reasoning, whereas easier GSM8K and MATH500 responses have fewer consequential steps than AIME responses.The paper reports that AIME datasets almost always have at least one consequential reasoning step.
  • Self-Advantage Complements Cue-Based Faithfulness Tests: Cue prompts mostly produce uninformative traces with no consequential self-advantage, unlike uncued prompts, which typically produce at least one consequential step.Self-advantage therefore provides a per-step complement to binary cue-based faithfulness judgments.

6 Can Step-Advantage be Predicted From the Text Alone?

The paper tests whether step advantage can be inferred from reasoning text using out-of-the-box judges and fine-tuned critics. Judges improve with scale but remain weak, while critics perform much better on incorrect responses than on correct ones.

  • Results: Fine-tuned critics reach PR-AUC 0.28–0.30 ID and 0.18–0.32 OOD on incorrect responses, roughly half the conservative noise ceiling of ≈0.6.These results represent a substantial improvement over out-of-the-box judges.
  • Results: Out-of-the-box judges improve with scale above prevalence, but the best judge remains 9× below the ID noise ceiling and 6× below the OOD ceiling.The evaluation uses both in-distribution and out-of-distribution math datasets.
  • Results: Critic scaling is largely flat, with the 1.7B critic approximately matching Qwen3.6-27B on ID.This differs from the scale-related improvement observed for out-of-the-box judges.
  • Results: On correct responses, critics attain only 0.065–0.10 PR-AUC, or 10–20% of the corresponding 0.51–0.64 ceiling.The text therefore contains much less recoverable information about consequential steps within correct solutions.
  • Results: At a 0.5% inspection budget, critics reach precision 0.55–0.60 ID and 0.55–0.72 OOD for incorrect responses, but only 0.08–0.16 ID and 0.18–0.29 OOD for correct responses.Precision on incorrect responses matches conservative ceilings, whereas correct-response precision remains far below ceiling.

7 Discussion

Viewing step importance as advantage connects interpretability of reasoning traces to reinforcement-learning credit assignment and process reward modeling. However, text-based critics recover importance asymmetrically: they perform well on incorrect responses but struggle on correct ones.

  • Advantage connects identifying consequential reasoning steps to reinforcement-learning credit assignment and process reward modeling.
  • Critics struggle to decode step importance for correct responses while performing well for incorrect responses.The asymmetry may partly reflect easy proxy signals in incorrect responses, such as consequential final-answer emission steps.
  • Correct-response steps are especially important because they may mark the model’s genuine discovery of how to solve a problem.

8 Related Work

The paper distinguishes advantage from counterfactual necessity, token-ablation measures, KL-based distributional shifts, and prior value-estimation methods. Its advantage-based formulation targets how a step changes expected reward under the model’s policy.

  • Advantage measures how a reasoning step shifts the model’s trajectory from baseline expectation, whereas counterfactual necessity measures response change when the step is omitted.
  • Token-masking and token-replacement methods measure behavioral changes after interventions but cannot capture complex interactions between tokens in a sequence.
  • KL-based resampling importance measures overall answer-distribution shifts without indicating whether a step moves toward or away from the target answer.
  • Prior process-supervision work uses a mathematically equivalent potential measure but differs in rollout count and completer-model choice.

9 Conclusion

The paper defines consequential reasoning steps through advantage estimated from prefix rollouts and identifies them with a changepoint pipeline. Validation shows useful but imperfect labels, with gradual ramps as the main failure mode and spurious labels largely filtered out.

  • The pipeline converts per-prefix Monte Carlo value estimates into per-step consequential labels using PELT segmentation, effect-size filtering, and localization checks.
  • The input series uses 50 rollouts per prefix, with value-level changes between consecutive prefixes corresponding to intervening-step advantage.
  • The Beta-posterior filter requires a 95% credible interval excluding zero and at least 95% posterior probability that the jump exceeds δ = 0.1.
  • 16.2 expected false discoveries occurred among 7,563 labeled consequential steps, an expected false-discovery rate of approximately 0.2% under the posterior audit.
  • 10.6–15.2% of fitted segments showed within-segment drift versus a nominal 5% rate, while ramp-like trajectories were flagged in only 2.2–13.0% of series.
  • Linear ramps are the main failure mode because the pipeline may wrongly attribute gradual change to one step, whereas temporary excursions are usually labeled at both edges.

A.3 Comparison with Per-Step Hypothesis Testing

The changepoint pipeline identifies consequential steps more reliably than per-step hypothesis tests, which trade substantially higher false-label rates for lower power. On real traces, the pipeline’s labels replicate strongly and its surplus over z-test flags is largely attributable to sampling noise.

  • Simulated power–error frontier: Power reached 0.46–0.54 / 0.73–0.77 / 0.96 for jump sizes 0.15/0.2/0.3, while false-label rates remained 1.3–3.7 × 10−4.These are the deployed pipeline’s simulated power ranges across four configurations.
  • Simulated power–error frontier: At α = 0.05, the per-step z-test achieved only 0.18–0.19 / 0.38–0.42 / 0.81–0.82 power while falsely labeling 14–25% of flat series.The strongest per-step comparison was dominated in every configuration.
  • Split-half replication on real data: The changepoint labels replicate on independent rollout halves, while the uncorrected z-test flags 3–10× as many steps.The validation compares each method under its own selection criterion.
  • Where the label sets agree and disagree: Although 86–93% of consequential steps have a nearby z-flag, only 10–35% of z-flags lie near consequential steps, and surplus flags are mostly sampling noise.Noise-only simulations predict 1.17× fewer in-segment flags than observed.
  • Consistency across configurations: Across non-thinking models, validation statistics remain in narrow bands, while the pipeline is strongest for thinking traces and per-step testing deteriorates with longer traces.The thinking configuration reaches power 0.54/0.77/0.96 at a 1.3 × 10−4 false-label rate, whereas the z-test falsely flags 25% of null traces.

F.2 Stratified Analysis

Stratified analyses compare consequential responses across cue conditions, judging strategies, outcomes, generators, and critics. Consequential-step detection is stronger for incorrect responses than correct ones, with fine-tuned critics approaching ceiling only in the former setting.

  • Cue-stratified responses: Without a cue, 58% of responses contained a consequential step versus 15% with a cue, and consequential steps comprised 5.0% versus 1.4% of steps.The comparison covers 100 examples under BASE and CUE prompting.
  • Cue-resistant responses: The five examples resisting the cue showed nearly indistinguishable BASE and CUE reasoning, but the small sample prevents strong conclusions.The authors present this as a possibility requiring further investigation.
  • Outcome-stratified critic evaluation: Fine-tuned critics dominate judges on incorrect responses, with head precision ≈0.5–0.6 versus ≤0.19, while correct-response curves remain near prevalence.On correct responses, head precision is ≤0.10 in-distribution and ≤0.15 out-of-distribution.
  • Generator-scale replication: On Qwen3-8B non-thinking generations, the largest critics at a 0.5% budget reached the conservative ceiling on incorrect responses: 0.63–0.64 versus 0.57 for ID.The broader pattern reproduces strong incorrect-response performance and weak correct-response performance relative to ceiling.

H.3 Per-Dataset Breakdown

Critic self-advantage performance varies by source dataset: contest datasets are easier than GSM8K and MATH500. The breakdown is reported with trace-bootstrap confidence intervals.

  • Per-dataset breakdown: Self-advantage PR-AUC is ≈0.25–0.30 on AIME 24/25/26 and AMC 23, versus ≈0.10–0.11 on GSM8K and MATH500.The contest datasets include both in-distribution AIME datasets and out-of-distribution AMC 23.

I Thinking-Model Judge and Critic Results

On thinking-mode traces, consequential steps are rarer and absolute evaluation levels are lower than in non-thinking settings. Value-derived judges remain near chance, while think-trained critics recover self-advantage above chance but remain below ceiling.

  • Thinking-trace evaluation scope: Consequential steps occur at ≈0.5% prevalence for self-advantage and ≈0.2% for correctness-based advantage, making absolute scores incomparable to non-thinking arms.The corresponding split-half ceilings are approximately 0.3 PR-AUC.
  • Judge and critic performance: Value-derived judging reaches PR-AUC 0.006 ID and 0.004 OOD versus prevalences of 0.005 and 0.004, while think-trained critics reach 0.037–0.066 ID.Critics decode self-advantage at 8–14× prevalence but remain further below their ceiling than non-thinking critics.
  • Study limitation: A 32B critic was not trained in this arm because of compute constraints.This limits the available scale comparison for thinking-mode critics.

J Results: Correctness-Based Advantage

Correctness-based step importance is substantially harder to decode than self-advantage: critics remain far below the noise ceiling, especially for correct responses. The leave-one-out advantage is introduced as a probability-scaled alternative but is not empirically tested because comparing alternative reasoning steps is difficult.

  • Correctness-Based Advantage: 0.028–0.092 population-level critic PR-AUC on gen-1.7B versus a 0.60 conservative ceiling shows large decoding headroom under correctness-based reward.Precision at a 0.5% inspection budget reaches only 0.07–0.21 against ceilings of 0.57–0.62, and full curves compress toward the prevalence floor.
  • Correctness-Based Advantage: Correctness-based outcome stratification is prevalence-sensitive, so each PR-AUC must be compared with its own chance line rather than across correct and incorrect groups.On gen-1.7B, correct-response prevalence is 0.017–0.020, versus 0.003–0.012 for incorrect responses.
  • Advantage vs Leave-One-Out Advantage: The leave-one-out advantage scales ordinary advantage by the inverse probability of sampling any other reasoning step.The construction uses the pushforward distribution over the next reasoning step, π_ϕ(a | s), and conditions on not choosing a.
  • Advantage vs Leave-One-Out Advantage: Empirical evaluation of leave-one-out advantage was omitted because judging alternative reasoning steps beyond exact string matching raises difficult empirical questions.The paper presents a proof for the measure but does not test it further.
Loading 2609.04194v1…