PaperlayerView exact revision on arXiv

Source-linked AI summary

Mechanistic Evidence for Faithfulness Decay in Chain-of-Thought Reasoning

Donald Ye, Max Loffgren, Om Kotadia, Linus Wong, Jonas Rohweder

arXiv:2602.11201v2cs.CL

TL;DR

The paper addresses whether CoT explanations faithfully reflect the processes producing model predictions or instead rationalize them after the fact. It introduces NLDD, which measures confidence changes after corrupting reasoning steps and normalizes them for cross-model comparison, then combines it with structural diagnostics across three model families. The main result is a consistent reasoning horizon at 70–85% of chain length, showing that accuracy alone does not establish functional dependence on CoT.

  • Problem

    CoT explanations may be post-hoc rationalizations, while existing faithfulness evaluations miss the graded causal contributions of individual reasoning steps.

  • Method

    NLDD measures confidence changes when reasoning steps are corrupted and normalizes logit degradation against intrinsic output variability, complemented by RSA, linear probes, and TAS.

  • Results

    A reasoning horizon consistently occurs at 70–85% of chain length, beyond which reasoning tokens contribute negligible or harmful causal influence across three models and three reasoning tasks.

  • Takeaways & Limitations

    Accuracy alone does not reveal whether models functionally depend on their generated reasoning, because some models achieve high accuracy despite negative NLDD.

  • Takeaways & Limitations

    The study evaluates decoder-only models on three tasks with 100 samples each, including two synthetic tasks, leaving generalization to larger models, other architectures, and open-ended tasks untested.

Abstract

from arXiv · show

Chain-of-Thought (CoT) explanations are widely used to interpret how language models solve complex problems, yet it remains unclear whether these step-by-step explanations reflect how the model actually reaches its answer, or merely post-hoc justifications. We propose Normalized Logit Difference Decay (NLDD), a metric that measures whether individual reasoning steps are faithful to the model's decision-making process. Our approach corrupts individual reasoning steps from the explanation and measures how much the model's confidence in its answer drops, to determine if a step is truly important. By standardizing these measurements, NLDD enables rigorous cross-model comparison across different architectures. Testing three model families across syntactic, logical, and arithmetic tasks, we discover a consistent Reasoning Horizon (k*) at 70--85% of chain length, beyond which reasoning tokens have little or negative effect on the final answer. We also find that models can encode correct internal representations while completely failing the task. These results show that accuracy alone does not reveal whether a model actually reasons through its chain. NLDD offers a way to measure when CoT matters.

1 INTRODUCTION

CoT explanations may be post-hoc rationalizations rather than faithful accounts of model predictions. NLDD addresses this gap by measuring step-level causal reliance in normalized logit space, supported by structural diagnostics and cross-architecture evaluation.

  • Motivation: CoT faithfulness remains uncertain because explanations can reflect plausible post-hoc rationalizations rather than the processes producing predictions.This concern is especially relevant when models rely on spurious correlations or pretrained priors despite generating apparently correct derivations.
  • Problem: Existing interventions reveal whether models rely on CoT but often treat faithfulness as binary, missing graded contributions from individual reasoning steps.The limitation motivates measuring causal effects at the step level rather than only observing whether the final answer changes.
  • Approach: NLDD measures confidence changes after corrupting reasoning steps and normalizes logit degradation against intrinsic output variability for cross-model comparison.A 50% NLDD drop means the corrupted chain produces half the standardized logit margin of the clean chain.
  • Approach: Structural diagnostics combine RSA, linear probes, and TAS to track representational alignment, solution-relevant information, and geometric drift after counterfactual interventions.These diagnostics complement causal measurements by probing internal geometry while corrupted reasoning remains surface-coherent.
  • Evaluation: The evaluation spans DeepSeek-Coder-6.7B-Instruct, Llama-3.1-8B-Instruct, and Gemma-2-9B-Instruct to distinguish architecture-invariant patterns from architectural idiosyncrasies.Gemma-2 is included to test whether NLDD remains interpretable under logit soft-capping.

2 RELATED WORK

Prior work evaluates CoT faithfulness through behavioral interventions, process-level benchmarks, causal analyses, circuit methods, and geometric interpretations. These approaches motivate combining causal measurements with representation-level diagnostics.

  • Behavioral and process-level evaluation: Early faithfulness studies modified reasoning steps and tracked output changes, while newer benchmarks detect process-level errors in generated reasoning.These methods established behavioral and fine-grained process-level perspectives on CoT faithfulness.
  • Causal and mechanistic analysis: Formal causal approaches estimate indirect effects or enforce alignment with predefined causal graphs, while circuit methods isolate attention heads for logical operations.Sparse autoencoders provide another mechanistic route for analyzing internal representations.
  • Performance horizons and geometry: Related work examines performance cliffs and geometric model states, motivating NLDD alongside RSA to study causal dependence and representational distinctiveness.The underlying mechanism of performance degradation with sequential dependencies remains unclear in the cited work.

3 MECHANISTIC FRAMEWORK AND EXPERIMENTAL DESIGN

The framework combines counterfactual corruption of individual CoT steps with behavioral, representational, and geometric analyses to measure faithfulness across reasoning tasks and models. It identifies the reasoning horizon as the step with peak causal contribution before faithfulness decay.

  • Mechanistic framework: The framework corrupts individual reasoning steps, truncates subsequent steps, and analyzes output confidence alongside internal representations.This dual design tests both whether steps causally influence predictions and how reasoning trajectories change internally.
  • Experimental design: Three instruction-tuned decoder-only transformers are evaluated on syntactic, logical, and arithmetic reasoning benchmarks.The benchmarks include Dyck-n, PrOntoQA, and GSM8K, with explicit intermediate reasoning steps for localized corruption.
  • NLDD: NLDD measures graded confidence degradation after step corruption in normalized logit space, enabling comparisons across reasoning steps and models.Confidence is normalized using output variability, while negative NLDD indicates that corruption increases the answer margin.
  • Representational and geometric analyses: RSA compares clean and corrupted representational geometry, while TAS measures the directness of hidden-state trajectories through latent space.RSA uses middle-layer token trajectories, and TAS is the ratio of straight-line displacement to cumulative path length.
  • Reasoning horizon detection: The reasoning horizon k* is the corruption position with maximum mean NLDD, marking the transition from active reasoning to post-hoc formatting.Across the truncation design, steps beyond k* form a theoretical pruning zone with NLDD below 20% of peak values, and convergent NLDD, RSA, and TAS degradation supports the transition.

4 RESULTS

Across models and tasks, NLDD separates faithful reasoning from anti-faithful behavior and reveals a consistent horizon beyond which additional tokens contribute little or harmful causal influence. Internal representations can remain stable even when they no longer affect the output.

  • Faithfulness regimes: Llama and DeepSeek show positive NLDD, while Gemma reaches 99.0% accuracy on PrOntoQA with NLDD of -52.5%.Gemma’s correct predictions can coexist with reasoning traces that increase, rather than decrease, confidence when corrupted.
  • Mapping Gap: Gemma’s Dyck-n representations encode stack-depth information at 82.0% probe accuracy, yet complete-CoT task accuracy is 0.0%.This Mapping Gap separates internally available task structure from its use during final decoding.
  • Representational dynamics: RSA is 0.422 for Dyck-n and 0.254 for PrOntoQA, showing that representational stability does not ensure causal utilization.PrOntoQA retains near-perfect accuracy despite lower representational stability and anti-faithful behavior.
  • Representational dynamics: TAS decays systematically in Llama, whereas Gemma and DeepSeek show marginal variance between the first and last reasoning steps.For Gemma, rigid trajectories align with anti-faithfulness and little computational effect from generated reasoning tokens.
  • Reasoning horizon: RSA remains stable beyond k*, indicating that models can continue tracking task logic internally without using those representations to inform the output.In Gemma’s PrOntoQA, anti-faithfulness deepens beyond k*=11, so later tokens interfere with a pre-computed answer.

5 DISCUSSION

The discussion presents NLDD and k* as diagnostics that expose causal dependence and locate where reasoning influence decays. Together, the findings show that accuracy, representational consistency, and CoT surface coherence can fail to reveal whether reasoning drives the answer.

  • NLDD as a faithfulness metric: NLDD measures whether corrupting reasoning changes output confidence, revealing causal dependence that accuracy alone can miss.Gemma achieves 99% accuracy on PrOntoQA while exhibiting negative NLDD.
  • Reasoning horizon: The consistent 70–85% horizon across tasks suggests a general property of CoT computation rather than a task-specific artifact.The horizon identifies where causal influence decays across chains with different lengths.
  • Reasoning horizon: At k*, later tokens can theoretically be pruned because they contribute negligible causal influence, with NLDD providing a causal basis for model-task-specific truncation.The proposed procedure locates k* by computing NLDD across chain positions rather than using length-based heuristics.
  • Anti-faithful behavior: Gemma’s NLDD drops from −22.9 to −120.2 beyond k*, indicating that extended reasoning can interfere with pre-computed answers.This pattern makes rigid step-by-step generation potentially counterproductive for some architectures.
  • Representational limits: Stable RSA beyond the horizon shows that internal geometric consistency can persist after causal influence decays.Representational analysis alone therefore cannot establish reasoning quality or causal faithfulness.
  • Cross-model comparison: Table 1 compares Accuracy (%) with Faithfulness (NLDD %), showing positive NLDD for Llama and DeepSeek and negative NLDD for Gemma on logical tasks despite high accuracy.The table’s comparison makes accuracy–faithfulness dissociations directly visible across tasks and models.
  • Cross-task comparison: Table 2 summarizes reasoning horizons across GSM8K, Dyck-n, and PrOntoQA, whose chains contain 8, 12, and 16 total reasoning steps.These task-specific chain lengths contextualize the relative horizon positions reported across models.

6 CONCLUSION

The paper introduces NLDD to measure whether generated reasoning causally influences predictions rather than serving as post-hoc rationalization. Across three models and three tasks, it identifies architecture-dependent faithfulness, a Mapping Gap, and a reasoning horizon at 70–85% of chain length.

  • Contribution: NLDD quantifies the causal influence of reasoning chains on model predictions beyond accuracy-based evaluation.It distinguishes genuine dependence on generated reasoning from post-hoc rationalization.
  • Findings: Across three models and three reasoning tasks, architectures show different faithfulness patterns, including positive CoT dependence and high accuracy with negative NLDD.The findings also include a Mapping Gap between encoded task structure and its utilization, plus a 70–85% reasoning horizon.

7 LIMITATIONS

The study's limitations concern causal interpretation, normalization assumptions, evaluation scope, and decoding conditions. Several implementation and generalization boundaries remain open.

  • Causal intervention: Truncating chains measures forward causal dependence but conflates weak reliance on a step with earlier steps already being sufficient.A replacement-based intervention could distinguish these effects.
  • Method assumptions: The logit normalization assumes architectural differences affect margins and variance proportionally, an assumption supported empirically but not guaranteed.The TAS and RSA diagnostics also use a single layer at 50% depth.
  • Evaluation scope: The evaluation covers decoder-only models on three tasks with 100 samples each, including two synthetic tasks.Generalization to larger models, other architectures, or open-ended tasks remains untested.
  • Evaluation conditions: Greedy decoding with fixed prompts may limit conclusions because stochastic decoding could shift horizon locations.The step-based analysis also assumes an explicit chain structure; free-form CoT would require different segmentation.

A.3 ROBUSTNESS OF NLDD NORMALIZATION

The appendix explains why NLDD normalization is intended to remain comparable across architectures and output distributions. It favors a global logit-space normalization over alternatives requiring model-specific calibration.

  • Architectural robustness: Logit soft-capping can rescale margins and normalization constants proportionally, preserving comparable standardized NLDD values across architectures.Gemma-2 is the motivating example.
  • Vocabulary robustness: Vocabulary-size differences affect absolute entropy and variability, but same-distribution standardization preserves relative margin-to-variability ratios.This is intended to keep NLDD focused on causal sensitivity rather than vocabulary artifacts.
  • Normalization choice: NLDD uses logit-space normalization via S because probability-based alternatives introduce model-specific tuning or additional calibration.The chosen approach is presented as simple and architecture-agnostic.
  • Global calibration: Global normalization preserves native confidence calibration, whereas per-input scaling could inflate noise in low-confidence samples.S is derived from a clean calibration set and fixed globally.
  • Model coverage: The evaluation spans three decoder-only transformer models from 6.7B to 9B parameters: DeepSeek-Coder, Llama-3.1, and Gemma-2.The models provide specialization, dense-baseline, and output-rescaling conditions.
  • Model controls: All evaluated models are instruction-tuned variants, helping isolate architectural and pretraining effects from stylistic competence.The variants were fine-tuned with SFT and RLHF.

A.5 IMPLEMENTATION DETAILS

The implementation uses shared configurations across experiments to isolate causal dependence on intermediate reasoning while reducing stochastic, architectural, and surface-level confounds.

  • Shared setup: Implementation choices cover inference configuration, counterfactual construction, representation extraction, and diagnostic metrics.These choices are summarized for all experiments.
  • Comparability: Shared settings across models support comparability by minimizing confounds from stochastic decoding, logit rescaling, and surface-level perturbations.Reported results use these fixed configurations unless otherwise noted.

A.6 STATISTICAL METHODOLOGY

The statistical methodology specifies filtering, uncertainty estimation, and reproducible confidence intervals for the reported faithfulness metrics.

  • Filtering: The analysis excludes samples with negligible clean-answer confidence, |LDclean| < 10^-6, to avoid numerical instability in standardized scores.The filter affected a negligible fraction of the dataset.
  • Uncertainty estimation: 95% confidence intervals for NLDD and RSA estimates use BCa bootstrap with B = 10, 000 resamples and fixed random seed 42.The BCa method adjusts for skewness and estimator bias.

B EXTENDED EXPERIMENTAL RESULTS

This section presents extended faithfulness-degradation results across models, with figures organized by model and Table 6 reporting NLDD values and uncertainty. The results include cases where corruption increases answer confidence, indicating an anti-faithful regime.

  • Results: Extended NLDD, RSA, TAS, accuracy, and probability-delta results are reported across all evaluated models.Figures are grouped by model for readability.
  • Results: Table 6 reports full faithfulness-degradation results with 95% BCa bootstrap confidence intervals based on B = 10, 000.The table covers all models.
  • Results: Negative NLDD values identify an anti-faithful regime in which corruption paradoxically increases answer confidence.This interpretation is specified for the full faithfulness-degradation results.
  • Results: Figures 3–8 show NLDD, RSA, and TAS as functions of corruption step index across tasks for LLaMA-3.1-8B, DeepSeek-Coder-6.7B, and Gemma-2-9B.The corresponding robustness diagnostics are presented in Figures 4, 6, and 8.
Loading 2602.11201v2…