PaperlayerView exact revision on arXiv

Source-linked AI summary

How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding

Xi Chen, Aske Plaat, Niki van Stein

arXiv:2507.22928v1cs.CLcs.AI

TL;DR

Whether CoT’s intermediate reasoning reflects a model’s true internal decisions remains unclear. The paper combines SAE feature extraction with activation patching to test CoT faithfulness, finding stronger effects in larger models than in small ones.

  • Problem

    It remains unclear whether CoT’s intermediate steps reflect the model’s true internal decision-making or serve as plausible surface scaffolding.

    Source: Introduction · p. 1

  • Method

    The study combines sparse autoencoders and activation patching to extract and exchange features between CoT and NoCoT conditions.

    Source: Introduction · p. 1

  • Results

    CoT induces more semantically coherent, causally effective, and sparser internal features, with effects strongest in larger models and minimal in small models.

    Source: Introduction · p. 1, Discussion · p. 7, Conclusion · p. 8

  • Takeaways & Limitations

    The findings suggest CoT can reshape internal reasoning processes in larger models, offering insight into mechanisms underlying LLM reasoning.

    Source: Conclusion · p. 8

    Patching targets only the final token’s residual activation and does not trace causal effects through the reasoning process.

    Source: Limitation and Future Work · p. 7

Abstract

from arXiv · show

Chain-of-thought (CoT) prompting boosts Large Language Models accuracy on multi-step tasks, yet whether the generated "thoughts" reflect the true internal reasoning process is unresolved. We present the first feature-level causal study of CoT faithfulness. Combining sparse autoencoders with activation patching, we extract monosemantic features from Pythia-70M and Pythia-2.8B while they tackle GSM8K math problems under CoT and plain (noCoT) prompting. Swapping a small set of CoT-reasoning features into a noCoT run raises answer log-probabilities significantly in the 2.8B model, but has no reliable effect in 70M, revealing a clear scale threshold. CoT also leads to significantly higher activation sparsity and feature interpretability scores in the larger model, signalling more modular internal computation. For example, the model's confidence in generating correct answers improves from 1.2 to 4.3. We introduce patch-curves and random-feature patching baselines, showing that useful CoT information is not only present in the top-K patches but widely distributed. Overall, our results indicate that CoT can induce more interpretable internal structures in high-capacity LLMs, validating its role as a structured prompting method.

Introduction

CoT improves multi-step task performance, but whether its reasoning reflects models’ internal decision-making remains unresolved. This study combines sparse autoencoders and activation patching to test CoT faithfulness at the feature level.

Introduction

  • LLMs’ internal decision-making remains difficult to understand, and CoT’s intermediate steps may be surface scaffolding rather than faithful reflections of that process.The paper identifies little feature-level, causally grounded analysis of faithfulness in multi-step math word problems.Source: Introduction · p. 1
  • The study combines sparse autoencoders with activation patching to extract and swap features between CoT and NoCoT conditions.Selected features also receive lightweight natural-language interpretations.Source: Introduction · p. 1
  • The authors introduce a log-probability-based procedure to assess feature-level causal impacts in multi-step mathematical reasoning.The workflow also includes activation patching, feature interpretation, and activation sparsity analysis.Source: Introduction · p. 1, Figure 1 · p. 2
  • CoT induces sparser and more causally effective internal features, enhancing faithful reasoning only in sufficiently large models.The paper reports that the effect is minimal in small models.Source: Introduction · p. 1

Related Work

Prior work studies CoT performance, SAE-derived features, and causal approaches to faithfulness, but feature-specific causal mechanisms remain difficult to isolate. This paper combines SAE feature extraction and activation patching to examine CoT in multi-step math reasoning.

Related Work

  • CoT can produce coherent reasoning, but models may reach correct answers despite incorrect intermediate steps (Yee et al. 2024).GSM8K is used as a challenging benchmark because answers often span multiple tokens and require precision and logical coherence (Cobbe et al. 2021).Source: Related Work · p. 2
  • SAEs decompose dense activations into interpretable, causally manipulable features, while prior work reports feature interventions can alter outputs (Cunningham et al. 2023).The paper frames feature-level interventions as more precise than neuron- or layer-level changes.Source: Related Work · p. 2
  • Faithfulness concerns whether explanations reflect true decision-making, and prior approaches include counterfactual interventions and causal mediation (Agarwal, Tanneru, and Lakkaraju 2024; Atanasova et al. 2023; Siegel et al. 2024; Paul et al. 2024).Matton et al. (2025) study interventions at the semantic-concept level.Source: Related Work · p. 2
  • Layer- or component-level interventions make it difficult to pinpoint responsible features, while backup circuits further complicate attribution (Dutta et al. 2024).The paper positions its feature-level analysis as addressing this challenge.Source: Related Work · pp. 2–3
  • Feature-level work enables finer tracking and control, but interventions can produce interpretability illusions when output changes do not originate from intended features (Geiger et al. 2024; Marks et al. 2024; Makelov, Lange, and Nanda 2023).This study applies SAE features, activation patching, and Top-K patch curves to multi-step mathematical reasoning.Source: Related Work · pp. 2–3

Methodology

The methodology combines sparse autoencoders, feature activation patching, structural analysis, and semantic interpretation to examine CoT reasoning faithfulness. It compares CoT and NoCoT features and internal activations across two Pythia models on GSM8K.

Methodology

  • The framework extracts SAE features, patches them between CoT and NoCoT runs, compares activation sparsity, and evaluates feature explanations.The four components are feature extraction, causal intervention, structural analysis, and semantic interpretation.Source: Methodology · p. 3

Feature Extraction

  • Separate SAEs encode CoT and NoCoT residual activations as sparse feature vectors, trained with reconstruction loss plus an L1 penalty.The study applies SAE-based feature extraction to CoT prompting, building on prior work producing semantically meaningful features.Source: Feature Extraction · p. 3

Causal Intervention

  • For matched problems, selected CoT feature values replace NoCoT values; the patched representation is decoded, forwarded, and evaluated by the change in correct-answer log-probability.An increase in confidence after inserting CoT features is treated as evidence that the features play a key causal role in reasoning.Source: Causal Intervention · p. 3
  • Patch curves add features ranked by CoT–NoCoT activation difference and compare their effects with Random-K patches to test whether causal effects are concentrated or distributed.The number of patched features varies across steps, with the random baseline sampling from the full feature set.Source: Causal Intervention · p. 3

Structural Analysis

  • Structural analysis compares CoT and NoCoT activation sparsity, defined as the proportion of hidden-state units inactive or close to zero.The study examines how prompting strategies influence internal activation structure.Source: Structural Analysis · p. 4
  • For efficient computation, the sequence is divided into non-overlapping chunks, sparsity is computed per chunk, and results are aggregated into a dataset-wide distribution.Chunking is described as a technical computation that does not change the global sparsity definition.Source: Structural Analysis · p. 4

Semantic Interpretation

  • Feature explanations are generated from highly activating text snippets and evaluated by correlating predicted with true activation sequences; score distributions are compared across prompting conditions.The comparative analysis examines semantic consistency of features under CoT and NoCoT prompting.Source: Semantic Interpretation · p. 4

Experiment Setup

  • Experiments compare Pythia-70M and Pythia-2.8B on GSM8K training problems with fixed few-shot CoT or question-only NoCoT prompts, using layer-2 final-token residual activations.SAEs are trained separately by model and prompt setting; patching compares Top-K features with uniformly sampled Random-K controls.Source: Experiment Setup · p. 4

Results

CoT features transfer causally to improve outputs in Pythia-2.8B, while effects in Pythia-70M are unreliable. CoT also yields more interpretable and sparser internal features, especially at larger scale.

Effect of CoT on Feature Interpretability

  • Pythia-2.8B explanation scores average 0.056 under CoT versus –0.013 under NoCoT, while the 70M difference is slight and statistically nonsignificant.In 70M, averages are 0.018 under CoT and 0.016 under NoCoT (t = 0.082, p = 0.935); in 2.8B, the difference is significant (t = 2.96, p = 0.004).Source: Effect of CoT on Feature… · p. 5, Figure 2 · p. 5, Table 1 · p. 6

Causal Effects of CoT Features via Activation Patching

  • CoT-to-NoCoT patching improves Pythia-2.8B output log-probabilities, but effects in Pythia-70M are variable and often harmful.At dictionary ratio 8, the 2.8B curve exceeds +3.2 at K=2 and then stabilizes; 70M shows persistent declines.Source: Figure 4 · p. 5, Causal Effects of CoT Features via… · p. 5
  • Randomly sampled CoT features often outperform Top-K features in Pythia-2.8B, with correct-answer confidence improving from 1.2 to 4.3.The reported benefit is distributed across moderately activated features rather than concentrated in the strongest activations; random and Top-K patching do not consistently help 70M.Source: Causal Effects of CoT Features via… · pp. 5–6

Activation Sparsity under CoT and NoCoT

  • CoT makes residual activations sparser in both models, with a markedly stronger effect in Pythia-2.8B.Under CoT, most neurons are near zero while a small subset is strongly activated.Source: Figure 7 · p. 6, Activation Sparsity under CoT and NoCoT · p. 6
  • CoT SAE features tend to activate fewer neurons than NoCoT features, with particularly structured sparsity and feature-level variability in Pythia-2.8B.This structure is offered as an explanation for why random feature patching can outperform selecting top activations.Source: Activation Sparsity under CoT and NoCoT · pp. 6–7

Discussion

The discussion reports scale-dependent gains in feature interpretability and causal influence, alongside sparser CoT activations. It cautions that the analysis is limited in causal tracing, interpretation, model scale, and SAE coverage.

Discussion

  • CoT features show higher explanation scores and semantic consistency in Pythia-2.8B, while the effect is small in 70M.The authors frame CoT as an effective structural prompt in larger models, but not as sufficient for logically faithful reasoning chains.Source: Discussion · p. 7
  • Random CoT-feature patches significantly boost output log-probabilities in the large model, but fail to improve and sometimes degrade performance in the small model.The discussion describes this as a scale-dependent effect of activation patching.Source: Discussion · p. 7
  • CoT induces sparser activations especially in 2.8B, which the authors link to focused semantic allocation and effective random feature patching.Both residual-stream and SAE-feature activations are suppressed except for a small subset.Source: Discussion · p. 7

Limitation and Future Work

  • Patching targets only the final-token residual activation, and results are restricted to Pythia-2.8B and smaller models, limiting process tracing and generalization.The authors also note indirect LLM-based interpretation and possible SAE biases; they suggest token- or path-level analysis, grounded explanations, and broader model coverage.Source: Limitation and Future Work · p. 7

Conclusion

The study combines sparse autoencoding, activation patching, and automated feature interpretation to examine CoT faithfulness. CoT effects on internal features are stronger in larger models and minimal in small ones.

Conclusion

  • CoT prompts, especially in larger models, induce more semantically coherent, causally effective, and sparser internal features, while effects are minimal in small models.The authors conclude that CoT reshapes internal reasoning processes as well as outputs.Source: Conclusion · p. 8

Appendix to: How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding

The appendix supplies additional experimental detail and reproducibility materials for the main paper, which was shortened due to space restrictions.

Appendix

  • The appendix provides six pages of additional results detail, code and configuration links, and enlarged versions of figures, while repeating some main-paper passages.The authors say these materials support reproducibility and readability.Source: Appendix to: How does Chain of Thought… · p. 9

Details

The experiments compare frozen Pythia-70M and Pythia-2.8B on GSM8K under CoT and NoCoT prompts, using matched data and SAE feature patching to evaluate reasoning-related activations.

Details

  • The study compares frozen Pythia-70M and Pythia-2.8B models using publicly available Pythia v0 weights.The models share a vocabulary and tokenizer and were pretrained on the Pile.Source: Details · p. 9
  • GSM8K training data supplies problems and ground-truth answers for activation collection and evaluation; its test split remains hidden.CoT uses a fixed few-shot prompt with three worked examples, while NoCoT presents only the current problem.Source: Details · p. 9
  • Both reasoning conditions use identical training data, differing only in input formatting, to support comparisons of activations and learned features.The model receives question text, not ground-truth answers; labels are reserved for evaluation.Source: Details · pp. 9–10
  • Separate SAEs are trained at dictionary ratios 4 and 8, and patching tests both top-K features and uniformly sampled Random-K features.The study varies K across 2, 4, 8, 16, 32, 64, and 128 for patch curves, with up to 1,000 problem pairs per condition.Source: Details · p. 10

Causal Effects of CoT Features via Activation Patching

Feature patching shows that CoT-activated features transfer reliably from CoT to NoCoT in Pythia-2.8B, but not Pythia-70M. In the larger model, useful information is distributed across features rather than concentrated in the highest-activation subset.

Causal Effects of CoT Features via Activation Patching

  • In Pythia-70M, CoT→NoCoT patching declines after K > 4 under ratio 4 and remains below zero under ratio 8.Under ratio 8, the minimum drop is about –3, versus –6 to –8 under ratio 4.Source: Causal Effects of CoT Features via… · p. 10
  • At K = 2, CoT→NoCoT patching gains over +2.5 log-prob under ratio 4 and surpasses +3.2 under ratio 8 in Pythia-2.8B.The ratio-4 curve later settles near +1.8; the ratio-8 curve stabilizes at approximately +2.4.Source: Causal Effects of CoT Features via… · p. 11
  • CoT→NoCoT patching improves performance in Pythia-2.8B across dictionary ratios, while Pythia-70M shows no stable benefit and can decline.The cross-ratio results point to scale-sensitive transfer of CoT-derived features.Source: Causal Effects of CoT Features via… · pp. 10, 12, Figure 12 · p. 12
  • Randomly selected CoT features often outperform Top-K features in Pythia-2.8B, indicating that useful information is spread across many moderately activated features.Top-K selection may emphasize local activation peaks while missing supportive features.Source: Causal Effects of CoT Features via… · p. 12
  • Higher dictionary ratio 8 produces smoother, more cumulatively stable performance changes than ratio 4, where gains are more abrupt and susceptible to outliers.The patch-curve analysis characterizes cumulative contributions as K increases.Source: Causal Effects of CoT Features via… · p. 12

Activation Sparsity under CoT and NoCoT

CoT prompts yield sparser activations in both models, with a stronger effect in Pythia-2.8B. In that model, feature-level activation patterns suggest structured sparsity and distributed CoT information.

Activation Sparsity under CoT and NoCoT

  • Pythia-2.8B combines globally sparse CoT activations with greater variation in neuron counts across features, which the authors interpret as structured sparsity.Some features use only a few neurons, while more complex features recruit broader neuronal collaboration.Source: Activation Sparsity under CoT and NoCoT · pp. 13–14
  • The authors argue that CoT-induced structured sparsity may mitigate superposition by increasing feature separation and interpretability.They describe this disentanglement as an emergent byproduct of semantic prompting, not an explicit training objective.Source: Activation Sparsity under CoT and NoCoT · p. 13
  • CoT produces significantly sparser residual activations than NoCoT in both models, with the effect more pronounced in Pythia-2.8B.Under CoT, most residual activations are near zero, with a small subset strongly activated.Source: Activation Sparsity under CoT and NoCoT · p. 12, Figure 14 · p. 14
  • CoT-related information in Pythia-2.8B is spread across feature combinations, helping explain why Random-K patching can outperform Top-K.Random sampling can capture moderately activated supporting features that Top-K selection misses.Source: Activation Sparsity under CoT and NoCoT · pp. 13–14
  • Under CoT, each SAE feature tends to activate fewer neurons than under NoCoT in both models, especially Pythia-2.8B.In the larger model, many features are supported by only a handful of neurons.Source: Activation Sparsity under CoT and NoCoT · p. 13, Figure 16 · p. 15

Related papers

Loading 2507.22928v1…