Source-linked AI summary
Thought Anchors: Which LLM Reasoning Steps Matter?
Paul C. Bogdan, Uzay Macar, Neel Nanda, Arthur Conmy
TL;DR
Interpretability methods designed for single forward passes provide limited insight into multi-token reasoning traces. The paper analyzes reasoning at the sentence level using counterfactual resampling, attention patterns, and causal links, finding that planning and uncertainty-management sentences often anchor reasoning and align with focused downstream attention. These methods support analysis of reasoning structure, difficulty, and reliability while remaining preliminary and computationally costly.
Problem
Existing interpretability work has limited methods for understanding the multi-token computational steps in autoregressive reasoning traces.
Method
The paper combines semantically filtered counterfactual sentence resampling, receiver-head attention analysis, and sentence masking to measure influence and causal dependencies.
Results
Planning and uncertainty-management sentences consistently show higher counterfactual importance and focused receiver-head attention, supporting their functional relevance in reasoning traces.
Takeaways & Limitations
Sentence-level analysis provides a principled toolkit for interpreting reasoning traces and studying their structure, difficulty, and potential reliability.
Takeaways & Limitations
The work is preliminary: resampling is computationally costly, receiver-head analyses are position-confounded, and attention suppression requires processing out-of-distribution information.
Abstract
from arXiv · showhide
Current frontier large-language models rely on reasoning to achieve state-of-the-art performance. Many existing interpretability are limited in this area, as standard methods have been designed to study single forward passes of a model rather than the multi-token computational steps that unfold during reasoning. We argue that analyzing reasoning traces at the sentence level is a promising approach to understanding reasoning processes. We introduce a black-box method that measures each sentence's counterfactual importance by repeatedly sampling replacement sentences from the model, filtering for semantically different ones, and continuing the chain of thought from that point onwards to quantify the sentence's impact on the distribution of final answers. We discover that certain sentences can have an outsized impact on the trajectory of the reasoning trace and final answer. We term these sentences \textit{thought anchors}. These are generally planning or uncertainty management sentences, and specialized attention heads consistently attend from subsequent sentences to thought anchors. We further show that examining sentence-sentence causal links within a reasoning trace gives insight into a model's behavior. Such information can be used to predict a problem's difficulty and the extent different question domains involve sequential or diffuse reasoning. As a proof-of-concept, we demonstrate that our techniques together provide a practical toolkit for analyzing reasoning models by conducting a detailed case study of how the model solves a difficult math problem, finding that our techniques yield a consistent picture of the reasoning trace's structure. We provide an open-source tool (thought-anchors.com) for visualizing the outputs of our methods on further problems. The convergence across our methods shows the potential of sentence-level analysis for a deeper understanding of reasoning models.
1 INTRODUCTION
Existing interpretability methods are poorly suited to multi-token reasoning traces, motivating sentence-level analysis of how reasoning steps depend on one another. The paper introduces thought anchors and complementary methods for measuring their influence and causal structure.
- Traditional mechanistic interpretability typically analyzes a single forward pass, whereas autoregressive reasoning unfolds across multiple computational steps.
- Sentences offer an intermediate abstraction level: they are more coherent than tokens and less likely than paragraphs to conflate distinct reasoning steps.
- Prior work identifies distinct chain-of-thought functions, but approaches for mapping the resulting higher-level reasoning structure remain limited.
- Thought anchors are critical reasoning steps that guide the trajectory of reasoning.
- The paper combines black-box resampling, attention-based evidence, and sentence masking to measure importance and causal dependencies within reasoning traces.
- These sentence-level measures are presented as a foundation for interpreting reasoning traces and supporting debugging, unreliability analysis, and reliability-enhancement research.
2 QUANTIFYING SENTENCE IMPORTANCE
The paper defines sentence importance by counterfactual influence on downstream reasoning and final answers, using resampling to preserve the continuation rather than forcing an early answer. Sentence-level analysis offers high resolution at lower cost than token-level resampling.
- Sentence importance is framed as how including or excluding a sentence changes subsequent steps and the model’s final output.
- Forced-answer importance can miss earlier influential steps when a necessary sentence is reliably produced only late in the reasoning trace.
- For each sentence, the resampling method generates alternative continuations with and without that sentence, then compares their final-answer distributions.
- In the case study, Sentence 13 sharply increases expected accuracy after sentences 6–12 show a steady decline.
- Sentence 13 redirects the solution from an incorrect 20-bit shortcut to decimal conversion and the correct 19-bit answer, a pivotal role missed by forced-answer importance.
- Sentence-level resampling mirrors token-level patterns at a fraction of the cost, while paragraph-level resampling provides less resolution.
3 CONSISTENT PATTERNS IN SENTENCE IMPORTANCE
The paper formalizes sentence importance with resampling-based distributional measures and finds that planning and uncertainty-management sentences have the greatest counterfactual influence. This differs from forced-answer importance, which emphasizes active computation.
- The analysis categorizes reasoning sentences by functions including problem setup, plan generation, fact retrieval, active computation, and uncertainty management.
- Resampling importance compares final-answer distributions across conditions, while counterfactual importance conditions on semantically dissimilar replacement sentences.
- Plan-generation and uncertainty-management sentences show higher counterfactual importance than fact-retrieval or active-computation sentences.
- Forced-answer importance instead implicates active computation as producing the greatest distributional shifts, unlike counterfactual importance.
4 THE MECHANISTIC ROOTS OF IMPORTANCE
The paper examines whether specialized attention heads mechanistically focus on important sentences. It identifies receiver heads whose attention narrows toward particular sentences and aligns with the sentence categories showing high counterfactual importance.
- The analysis tests whether downstream attention focuses on important sentences, while acknowledging that attention weights do not necessarily imply causal links.
- Sentence-level attention matrices are summarized by downstream attention concentration and kurtosis, revealing heads that narrow attention toward particular sentences.
- Heads that narrow attention toward specific sentences are called receiver heads, and their kurtosis scores show strong split-half reliability (r = .84).
- Among the 16 highest-kurtosis heads, mean sentence-attention correlation is r = .56, exceeding the average correlation among arbitrary heads (r = .35).
- Receiver-head ablation reduces accuracy more than random self-attention ablation, and reasoning models narrow attention more than base models.
- Receiver heads attend most to plan-generation and uncertainty-management sentences, while active-computation sentences receive relatively little attention (ts > 4.0, ps < .001).
5 SENTENCE-SENTENCE CAUSAL LINKS
The sentence-masking method measures how suppressing a sentence changes future-token logits, enabling causal links within a reasoning trace. In a case study, these links aligned with receiver-head-identified important sentences and exposed interpretable computational structure.
- 5.1 APPROACH: Masking a sentence suppresses subsequent attention to it, and its effect on future sentences is measured through average token-logit KL divergence.The score is normalized by subtracting the future sentence’s average causal effect from prior sentences; suppression is mostly equivalent to omitting the sentence.
- 5.2 CASE STUDY: The masking graph identified local maxima that closely aligned with sentences implicated as important by receiver heads.The case study focused on three such local maxima in the correct-answer chain of thought.
- 5.2 CASE STUDY: Sentence 12’s verification decision led from an initial “20 bits” answer to a different “19 bits” solution at Sentence 43.Most intermediate text between these sentences performed arithmetic.
- 5.2 CASE STUDY: Other causal links captured discrepancy checking and the synthesis of prior statements, forming an interpretable scaffold of pursuit, self-correction, and synthesis.The released interface visualizes important sentences as larger nodes and causal connections as dashed gray lines.
6 SYSTEMATIC DIFFERENCES IN SENTENCE-SENTENCE RELATIONSHIPS
Sentence-to-sentence causal distance tracks reasoning difficulty and differs across domains. More accurate reasoning is associated with stronger close-range links and weaker long-range links, especially in mathematics-related subjects.
- 6 INTRODUCTION: The causal graphs were motivated as a way to connect pivotal moments, such as decisions and discrepancies, with downstream reasoning behavior.The authors hypothesized that nearby links reflect coherent sequential reasoning while distant links reflect uncertainty and backtracking.
- 6.1 METHOD: The analysis compared causal-link strength by sentence distance, using mean attention-suppression effects for each distance k up to m/2.Restricting distances to m/2 reduced noise by retaining adequately populated matrix subdiagonals.
- 6.2 RESULTS: Questions with higher average accuracy elicited chains with stronger close-range causal links and weaker long-range links.Within-subject correlations connected causal-link distance with mean reasoning accuracy across ten chains of thought.
- 6.2 RESULTS: r = .44 for close links and r = −.54 for long links related subjects’ average accuracy to their reasoning-trace structure.Both correlations were statistically significant at p < .001.
- 6.2 RESULTS: Mathematics and physics showed the strongest accuracy and stronger close-range but weaker long-range connections than other domains.Domain differences were significant with two-sample t-test |t|s > 10 and ps < .001.
7 RELATED WORK
Prior work questions whether chain-of-thought text faithfully reflects underlying computation, while other studies identify disproportionately important reasoning steps. This work positions sentence-level causal analysis as a principled complement.
- Reasoning advances and unfaithfulness in LLMs: CoT text has been questioned as unfaithful, but the paper reports that sentence types differ systematically in resampling and receiver-head importance.Plan-generation and uncertainty-management sentences were especially important in both analyses.
- Importance of individual steps: Earlier methods identified important reasoning steps using Shapley values, ROSCOE metrics, gradient scores, and resampling at fork tokens.The paper presents its approach as a framework for understanding how traces are constructed around key sentences.
8 DISCUSSION AND LIMITATIONS
The paper frames thought anchors as a principled unit for decomposing reasoning traces and reports convergent evidence that CoT text is mechanistically relevant. It also identifies substantial computational and methodological limitations.
- Discussion: Thought anchors are sentences with outsized importance for final responses, specific future sentences, and downstream reasoning trajectories.The paper also examines attentional mechanisms associated with these important sentences.
- Discussion: Plan-generation and uncertainty-management sentences show higher counterfactual importance and more focused receiver-head attention, supporting CoT text’s mechanistic relevance.This links what a sentence says with its measured causal and attentional significance.
- Limitations: 100 resamples per sentence yielded final-answer accuracy confidence intervals of at worst ±10%, but the resampling approach is computationally costly.The authors suggest fewer or adaptive resamples for aggregate analyses to reduce cost.
- Limitations: The analyses remain preliminary because downstream sentences may be overdetermined, error correction is not formally examined, and receiver-head results are position-confounded.Attention suppression also effectively requires the model to process out-of-distribution information.
- Discussion: Convergence across the three methods suggests potential value for future interpretability techniques.The authors describe this as an advance on prior work while acknowledging the stated limitations.
9 REPRODUCIBILITY STATEMENT
The work supports reproducibility through public code, implementation details, experimental specifications, and an interactive visualization tool. Additional validation shows that the reported importance patterns persist across smoothing choices and generalize from R1-Distill-Qwen-14B to R1-Distill-Llama-8B.
- Reproducibility materials: Public code, scripts, experimental specifications, and an interactive visualization tool support reproducing and exploring the analyses.The materials include black-box resampling, receiver-head analysis, and attention-suppression experiments.
- Reproducibility materials: The supplementary materials include a full case study transcript, sentence-category distributions, receiver-head ablations with 128/256/512 heads, cross-model validation, and pseudocode for sentence-to-sentence importance.
- Robustness checks: Replacing the ϵ = 10^-9 floor with α = 1.0 Laplace smoothing or α = 0.5 Jeffreys-prior smoothing preserves the reported category-importance patterns.The same relationships remain after computing KL divergence with smoothed final-answer distributions.
- Cross-model validation: 100 rollouts for 20 correct and 20 incorrect base solutions replicated the resampling methodology on R1-Distill-Llama-8B using the same question set and sampling parameters.The replication used temperature = 0.6 and top-p = 0.95, matching the R1-Distill-Qwen-14B experiments.
- Cross-model validation: R1-Distill-Llama-8B showed similar accuracy fluctuations at critical reasoning transitions and similar sentence-category effects to R1-Distill-Qwen-14B.Plan-generation and uncertainty-management sentences had higher counterfactual importance than active-computation and fact-retrieval sentences in the cross-model comparison.
- Cross-model validation: The cross-model consistency supports the claim that reasoning traces are organized around high-level sentences rather than only low-level computational steps.The authors interpret the stable counterfactual-importance patterns as evidence that the framework generalizes beyond specific model implementations.
B.2 ATTENTION AGGREGATION
Receiver-head analyses identify sentence categories that attract downstream attention, while case-study analyses connect these patterns to computational chunks and sentence dependencies. The attention scores have position-related limitations, including dilution and proximity effects.
- Attention patterns: R1-Distill-Llama-8B shows receiver-head patterns broadly consistent with R1-Distill-Qwen-14B, but without the latter’s upward trend toward later layers.The Llama histogram has a right tail, indicating that some heads focus on subsets of sentences.
- Attention patterns: Plan generation, uncertainty management, and self-checking sentences receive heightened receiver-head attention, significantly exceeding fact retrieval and active computation.Four paired tests found p ≤ .01 for the relevant category comparisons.
- Case study: The case study segments reasoning into computational chunks for decimal conversion, binary conversion, discrepancy detection, and verification.The trace ultimately resolves the initial 20-bit proposal in favor of a 19-bit answer.
- Case study: Attention-suppression maxima identify dependencies linking the initial proposal, discrepancy, resolution, and verification sentences, complementing receiver-head attention.Reported links include 12 →43, 43 →65, 12 →66, 32 →46, and 32 →59.
- Position effects: Receiver-head scores are shaped by sentence position because later sentences face more competing broadcast sources, while proximity increases attention from nearby future sentences.The analysis omits the four immediately proximal sentences but does not fully remove proximity effects.
- Sentence distribution: Active computation is the largest category in the dataset at 32.7%, followed by fact retrieval at 20.1%, plan generation at 15.5%, and uncertainty management at 14.0%.Problem setup and final answer emission are rarer and concentrated near the beginning and end, respectively.
H ADDITIONAL RESAMPLING RESULTS
Expanded resampling results across all eight sentence categories preserve the main counterfactual-importance pattern. Category position and importance differ across problem setup, self-checking, and final answer emission.
- Additional resampling results: Problem setup sentences occur near trace beginnings at mean normalized position ≈0.1 and show moderate-high counterfactual importance.This extends the five-category analysis to the full eight-category taxonomy.
- Additional resampling results: Self-checking sentences tend to occur in the second half of traces and show lower counterfactual importance.The expanded view includes categories omitted from the main five-category analysis.
- Additional resampling results: Final answer emission sentences appear late at mean normalized position ≈0.9 and show the lowest counterfactual importance.The five-category patterns remain consistent when all eight categories are examined.
I COUNTERFACTUAL VERSUS RESAMPLING IMPORTANCE
Resampling behavior varies by sentence category, creating systematic differences between ordinary resampling importance and counterfactual importance. The analysis also identifies receiver heads that concentrate attention on particular reasoning sentences, especially in later layers and in the reasoning model.
- Uncertainty management and plan generation sentences produce semantically divergent alternatives more often than active computation and problem setup sentences.
- Resampling usually preserves uncertainty management and active computation categories, while plan generation and fact retrieval often transition into varied categories.
- Counterfactual and resampling importance vary substantially across sentences and categories because counterfactual importance filters for semantically different replacements.
- Counterfactual importance can become high-variance when fewer than 10 semantically divergent resamples are available.
- Receiver heads are more common in later layers and consistently narrow attention toward particular sentences across responses.
- The reasoning model directs greater receiver-head attention toward high-scoring sentences than the base model, although this difference was not observed with R1-Distill-Llama-8B.
L.2 RESULTS AND DISCUSSION
The paper evaluates receiver-head ablations and sentence-level causal dependencies in reasoning traces. Receiver heads matter most under extensive ablation, while causal matrices reveal directed influences among planning, computation, uncertainty-management, and checking sentences.
- At baseline, model accuracy was 64.1% [95% CI: 56.0%, 72.1%], and ablating 128 or 256 heads produced no significant or only small accuracy reductions.
- With 512 heads ablated, receiver-head targeting reduced accuracy to 28% versus 37% for control heads, a significant difference (t[31] = 2.55, p = .02).
- Excluding responses without final answers preserved the receiver-head effect: 29% accuracy versus 39% for controls (t[31] = 2.66, p = .02).
- The causal graph masks or removes each source sentence and measures average token-level log-KL divergence in subsequent target sentences.
- In the case study, planning sentence 12 increased the probability of subordinate planning sentence 16 by 0.39, while computation and uncertainty-management sentences propagated effects across later checking steps.
- The attention-suppression and resampling matrices were positively correlated for 90% of traces (mean r = .20), rising to mean r = .34 for sentence pairs fewer than five sentences apart.