Source-linked AI summary
Mechanistic Interpretability of Chain-of-Thought Reasoning via Sequential Activation Patching
Murat Dura, Serkan Öztürk, Selma Tekir
TL;DR
The paper addresses the limited understanding of where CoT-related causal effects arise across multi-token reasoning trajectories. It introduces sequential, POS-guided activation patching with joint-head controls and ablations, finding evidence that coordinated attention-head activity supports CoT-conditioned answer computation but also overlapping auxiliary functions.
Problem
CoT improves reasoning, but the internal locations and causal roles of its effects across generated reasoning trajectories remain poorly understood.
Method
The study sequentially patches CoT attention-head activations into No-CoT executions across token positions, aggregates effects by POS, and evaluates joint heads with cross-question, random, and zero-ablation controls.
Results
The experiments provide evidence that coordinated attention-head activity moves No-CoT executions toward CoT behavior and supports final-answer generation across overlapping mechanisms.
Takeaways & Limitations
CoT-related computation in the evaluated model is better characterized as a distributed reasoning-support sub-circuit than as a localized single-head or pure reasoning-only circuit.
Takeaways & Limitations
Head rankings may vary with token-grouping choices and the fixed top-k selection budget, whose sensitivity is not analyzed.
Abstract
from arXiv · showhide
Large Language Models (LLMs) demonstrate remarkable problem-solving capabilities when guided by Chain-of-Thought (CoT) prompting, yet the internal mechanisms underlying these improvements remain poorly understood. In this work, we investigate where CoT-related causal effects emerge across the generated reasoning trajectory and which attention heads carry signals that contribute to final-answer computation. Because CoT reasoning unfolds over multiple generated tokens, standard activation patching at a single static token position is insufficient to characterize these temporally distributed effects. To address this limitation, we introduce a sequential activation patching framework that traces CoT-conditioned attention-head activations across token positions and aggregates their effects using Part-of-Speech-guided analysis. We further introduce Sequential Multi-Head Patching to evaluate the joint contribution of distributed head sets, together with cross-question and random activation controls. Targeted zero-ablation experiments show that the identified heads are functionally important for successful answer generation and affect several overlapping mechanisms, including reasoning-trajectory maintenance, answer anchoring, exemplar-target separation, and numerical generation. Overall, our results provide evidence for distributed reasoning-support sub-circuits associated with CoT-conditioned computation.
1 Introduction
The paper asks where CoT-related causal effects arise across generated reasoning trajectories and which attention heads contribute to final-answer computation. It introduces sequential interventions to trace these distributed effects and reports evidence for coordinated CoT-supporting components.
- CoT improves complex and mathematical multi-step problem solving, but the internal mechanisms underlying this improvement remain poorly understood.
- Prior studies show that generated rationales can be misleading and that the causal role of intermediate reasoning steps varies across tasks, models, and evaluation settings.
- Standard activation patching at one static token position is insufficient for reasoning that unfolds across multiple generated tokens.
- The study uses sequential activation patching to trace which generated token positions and attention heads carry signals distinguishing CoT from No-CoT computation.
- Sequential patching moves No-CoT executions toward CoT behavior, while targeted ablation disrupts answer accuracy and auxiliary mechanisms including anchoring, exemplar–target separation, answer-slot binding, and numerical generation.
2 Related Work
Related work frames CoT as behaviorally useful but not necessarily faithful, while mechanistic interpretability uses causal interventions to study internal components underlying model behavior.
- CoT improves mathematical reasoning, but whether internal computation aligns with behavioral explanations remains a longstanding faithfulness challenge.
- Generated rationales may reflect post-hoc or encoded reasoning, and models can bypass their own rationales when producing final answers.
- Mechanistic interpretability studies internal components that causally influence behavior using activation patching and related analyses.
- Methodological choices, especially corruption strategies and evaluation metrics, strongly shape activation-patching interpretations.
3 Methodology
The methodology contrasts clean CoT and corrupted No-CoT executions, sequentially patches attention-head activations across reasoning tokens, and aggregates effects to identify distributed components. It evaluates interventions with distributional and logit-level metrics, controls, and joint-head selection.
- 3.1 Experimental Setup: The evaluation uses SVAMP mathematical word problems with controlled one-shot CoT prompting and standardized answer formatting.
- 3.2 Corruption Method: The clean run uses CoT and the corrupted run uses No-CoT, isolating the presence of reasoning traces as the intervention contrast.
- 3.3 Evaluation Metrics: JSD measures distributional movement toward CoT, while MRR measures recovery of the decision margin favoring the correct answer.
- 3.3 Sequential Activation Patching: Sequential patching injects clean-run attention-head outputs into the corrupted run across generated reasoning tokens and measures resulting shifts.
- 3.4 POS-Guided Aggregation: POS-guided aggregation reduces noisy token-specific effects by grouping heterogeneous reasoning tokens into linguistic categories and layer-by-head analyses.
- 3.5 Sequential Multi-Head Patching: Because isolated single-head interventions lacked meaningful effects, the method selects and merges top heads across POS categories into broader intervention sets.
- 3.6 Controls and Ablation: Cross-question and random-activation patching provide controls for problem-specific information and arbitrary perturbations, while zero ablation tests functional importance.
4 Experimental Results
Across patching and ablation experiments, targeted sequential multi-head interventions most strongly shifted executions toward CoT-like behavior, while selected-head ablations substantially degraded accuracy across prompting settings.
- Patching Results: Sequential Multi-Head Patching produced the strongest improvement, moving patched executions closer to the clean CoT distribution and recovering part of the CoT decision margin.The intervention jointly injected structured, POS-guided activations from the same question.
- Patching Results: Cross-question patching also shifted executions toward the CoT reference, but more weakly than same-question patching.Its remaining improvement suggests that selected heads encode partially reusable reasoning-support patterns.
- Patching Results: Random-Activation Sequential Multi-Head Patching failed to reproduce targeted-patching improvements, supporting an association with structured CoT-related signals.The control used randomly generated activation tensors rather than actual CoT activations.
- Zero Ablation Results: 65.63% baseline CoT accuracy fell to 0.00% after zero-ablating JSD-selected heads, compared with 37.50% after randomly selecting heads.The larger targeted degradation provides causal evidence that these heads help maintain the CoT reasoning trajectory.
- Zero Ablation Results: In No-CoT prompting, targeted ablations reduced accuracy to 6.25%–12.50% versus 15.60% for random selection; in Equation prompting, accuracy fell to 6.25%–18.75% versus 34.40%.The results indicate contributions beyond explicit CoT, including direct-answer reasoning and arithmetic computation, while limited instances and k = 3 per POS category reduce separation between conditions.
5 Discussion and Conclusion
The discussion presents CoT as an internal control signal that steers computation toward reasoning-compatible trajectories, with effects distributed across coordinated attention-head activity rather than localized to one head.
- Discussion and Conclusion: CoT-conditioned activations causally contribute to final-answer computation in the evaluated setting, although rationale text may remain only partially faithful.The findings distinguish useful internal control from fully faithful textual explanation.
- Discussion and Conclusion: Same-question patching recovers more strongly than cross-question patching, indicating instance-sensitive computation alongside partially reusable reasoning-support patterns.Cross-question gains suggest the heads are not merely copying problem-specific operands or entities.
- Discussion and Conclusion: Joint multi-head interventions and zero-ablation effects support a distributed reasoning-support sub-circuit rather than a localized single-head mechanism.Ablations affect answer accuracy, anchoring, exemplar–target separation, answer-slot binding, and numerical generation.
6 Limitations
The paper identifies several scope and interpretability limits: results depend on token grouping and fixed head-selection policies, use a narrow empirical setting, and do not isolate individual head functions. The proposed PEFT direction therefore remains contingent on disentangling overlapping computational roles.
- Methodological scope: POS-based token grouping and a fixed top-k budget may change head rankings, but sensitivity to alternative groupings and selection policies is untested.The authors specifically leave stability under alternative selection policies open.
- Empirical scope: Experiments use mathematical word problems and one relatively small base model, leaving generalization to other models, architectures, and reasoning tasks unresolved.The intervention metrics also evaluate only the first token of the final-answer span, not the complete generated answer sequence.
- Interpretation: The identified heads form an overlapping reasoning-support sub-circuit rather than a pure reasoning-only circuit, also supporting grounding, binding, formatting, and numerical generation.The paper cannot yet isolate which individual heads support abstraction, calculation, or other subtasks.
- Future work: Head-specific adapters or LoRA updates are proposed as a targeted PEFT direction, but overlapping functions must be disentangled before treating heads as isolated reasoning modules.This limitation directly constrains the proposed mechanistically informed fine-tuning strategy.
7 Ethics Statement
The authors disclose using generative AI tools to improve language quality and assist with figure and table design, with subsequent author review and revision for technical accuracy.
- Disclosure: Generative AI tools assisted with linguistic editing and the design and generation of figures and tables.The authors state that all AI-assisted outputs were reviewed, edited, and revised.
A.1 Sequential Multi-Head Cross Patching
Sequential Multi-Head Cross Patching tests whether POS-guided attention-head activations transfer from a source question to a distinct target question. The procedure selects and deduplicates heads, jointly patches their source CoT activations into the target No-CoT run, and compares the resulting distributional metrics with random-activation controls.
- Cross-question design: Cross-question patching uses CoT activations from a distinct source question QB and injects them into the target question QA’s No-CoT execution.This design decouples problem-specific content from the broader CoT mechanism.
- Control logic: Cross-question recovery tests whether selected activations transfer across instances rather than encoding only local problem-specific content.Recovery despite semantic mismatch supports transfer of at least part of the selected activation structure.
- Head selection: For each POS category, candidate heads are ranked by mean patching performance and the top-k heads are selected using lower JSD or higher MRR.The procedure records activations across token positions, layers, and heads before POS-guided aggregation.
- Random control: Random-activation controls generate and rank random head interventions, then jointly patch the top-n random activations for comparison.This control addresses distributional shifts caused by arbitrary hidden-state perturbations.
- Deduplication: Duplicate heads are assigned exclusively to the POS category with the better metric value, producing a deduplicated global head set for joint intervention.The deduplicated set is then used to extract the corresponding source CoT activations.
- Joint evaluation: The final intervention jointly patches the selected source activations into the target No-CoT run and computes JSDcross and MRRcross.The algorithm returns the assignments, deduplicated head set, and both cross-patching metrics.
B Evaluation Metrics Details
The paper evaluates causal effects of attention-head interventions with complementary distributional and decision-level metrics. JSD measures closeness to clean CoT behavior, while MRR measures recovery of the CoT–No-CoT decision-margin difference.
- Jensen–Shannon Divergence (JSD): Jensen–Shannon Divergence (JSD) compares patched and clean CoT next-token probability distributions.It is symmetric and smoothed relative to KL divergence.
- Jensen–Shannon Divergence (JSD): JSD ranges from 0 to ln 2 ≈0.693, with lower values indicating closer alignment to clean CoT distributions.JSD = 0 means the evaluated patched and clean distributions are identical.
- Margin Recovery Ratio (MRR): The logit margin M subtracts the highest-scoring alternative token’s logit from the correct answer token’s logit.The target token is t∗, and z(t) denotes the logit assigned to token t.
- Margin Recovery Ratio (MRR): MRR compares patched, clean CoT, and No-CoT margins to quantify how much of the CoT–No-CoT margin difference patching recovers.The ratio is defined only when MCoT ≠ MNoCoT.
- Margin Recovery Ratio (MRR): MRR = 0 indicates no recovery, 0 < MRR < 1 partial recovery, MRR = 1 matching CoT, and MRR > 1 exceeding the CoT margin.MRR < 0 indicates movement opposite to the clean CoT reference relative to No-CoT.
C.1 Head-Selection Overlap and Stability Analysis
The analysis tests whether selected attention-head sets are stable, overlapping, and concentrated in particular model layers. Targeted selection yields more concentrated and mutually overlapping head sets than random selection, with strongest concentration from middle to late layers.
- Overlap Metrics: Head-set agreement is measured with exact-head Jaccard similarity and minimum-overlap coverage, then repeated after collapsing heads into layers.Layer-level metrics distinguish agreement on identical heads from coarser agreement on model layers.
- Overlap Results: 56.4% exact-head Jaccard and 78.7% minimum-overlap coverage link JSD- and MRR-based same-question Sequential Multi-Head selections.Their layer-level agreement is higher, reaching 69.9% Jaccard and 91.0% minimum-overlap coverage.
- Overlap Results: 44.2% exact-head Jaccard and 68.0% minimum-overlap coverage connect Cross Multi-Head MRR with Sequential Multi-Head MRR.Cross-question patching therefore partially overlaps with the same reasoning-support structure while losing some instance-specific alignment.
- Overlap Results: Random-Activation JSD and MRR selections share only 7.5% exact-head Jaccard and 15.3% minimum-overlap coverage, despite 42.4% layer-level Jaccard.Random patching can perturb similar layers without consistently recovering the same specific heads.
- Selection Concentration: Across 32 questions, targeted selections cover 89–99 unique heads, versus 175–186 for random selections.The result indicates that targeted procedures repeatedly identify a more concentrated subset of heads.
- Layer Distribution: Targeted methods select approximately 70–76% of heads from middle layers and only 2–5% from early layers.Sequential Multi-Head selections also include roughly 25% late-layer heads, making the pattern middle-to-late rather than purely middle-layer.
- Layer Distribution: Random selections have weaker middle-layer concentration at approximately 60–64% and more early-layer selections at 22–25%.This profile is consistent with random patching producing a more diffuse selection pattern.
D Qualitative Analysis of Zero-Ablation Failures
Selected-head ablation produces larger accuracy drops than matched random-head ablation and disrupts multiple mechanisms involved in answer generation. Across CoT, No-CoT, and direct-equation settings, failures affect reasoning structure, answer binding, exemplar separation, and numerical formation.
- Selected-head ablation generally causes larger accuracy drops than ablating the same number of randomly selected heads.
- CoT prompting: Under CoT, ablation causes final-answer anchor loss, exemplar–target contamination, and corrupted or repetitive arithmetic trajectories.Random-head ablation more often preserves the broad reasoning-trace structure, even when the final answer is incorrect.
- No-CoT prompting: Under No-CoT, ablation disrupts answer-position binding by copying exemplar answers or selecting salient but incorrect target quantities.The final-answer anchor is already present in the prompt in this condition.
- Direct-equation prompting: Under direct-equation prompting, ablation produces operand copying, malformed numeric continuations, and incorrect arithmetic outputs.These failures occur after both natural-language context and explicit CoT reasoning are removed.
- Together, the examples implicate a distributed reasoning-support structure spanning CoT maintenance, exemplar–target separation, answer-slot binding, and numerical result formation.