PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training

Yein Park, Minbyul Jeong, Jaewoo Kang

arXiv:2509.25758v2cs.AI

TL;DR

How post-training changes reasoning models internally remains unclear, despite concerns about overthinking and differing training paradigms. The paper uses circuit analysis and ablation to compare those changes, finding distinct patterns across distillation/SFT, GRPO, and think-on/off settings. The authors identify scope limits in the model families studied and in prompt-based circuits.

  • Problem

    The mechanisms by which post-training enhances reasoning remain unclear, as do differences among training paradigms and whether they add skills or amplify latent capabilities.

    Source: §1 · pp. 1–2

  • Method

    The paper applies circuit analysis to compare post-training changes and uses ablation to investigate the roles of emergent attention-head circuits.

    Source: §2 · p. 3, §3 · p. 3

  • Results

    Distillation and SFT add reasoning heads, GRPO dynamically configures heads in response to reward, and think-on/off modes use different head patterns rather than dedicated thinking heads.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The authors propose attention-head-informed training policies that target activation, shape reward around plan quality and calculation reliability, and use per-head influence estimates.

    Source: §7 · p. 9

    The implementation was validated only on Qwen models and one Llama model, and prompt-based circuits require precise setup and may be vulnerable to polysemanticity.

    Source: §7 · p. 9

Abstract

from arXiv · show

The remarkable capabilities of modern large reasoning models are largely unlocked through post-training techniques such as supervised fine-tuning (SFT) and reinforcement learning (RL). However, the architectural mechanisms behind such improvements remain largely opaque. In this work, we use circuit analysis to demonstrate that post-training for complex reasoning sparks the emergence of novel, functionally specialized attention heads. These heads collectively support structured reasoning and computation. Our comparative analysis across various model families reveals that these emergent heads evolve differently under different training regimes. Distillation and SFT foster a cumulative addition of stable reasoning heads. In contrast, group relative policy optimization (GRPO) operates in a dynamic search mode: relatively few attention heads are iteratively activated, evaluated, and pruned, with their survival closely tracking fluctuations in the task reward signal. Furthermore, we find that controllable "think on/off" models do not possess dedicated "thinking" heads. Instead, turning off explicit reasoning triggers a broader-but less efficient-set of compensatory heads. Through ablation and qualitative analyses, we connect these circuit-level dynamics to a crucial performance trade-off: strengthened heads enable sophisticated problem-solving strategies for difficult problems but can also introduce "over-thinking" failure modes, such as calculation errors or logical loops on simpler tasks. These findings connect circuit-level dynamics to macro-level performance, identifying an inherent tension where complex reasoning comes at the cost of elementary computations. More broadly, our work points to future directions for training policy design, emphasizing the need to balance the development of effective reasoning strategies with the assurance of reliable, flawless execution.

1 Introduction

The introduction asks how post-training changes reasoning models internally, given unclear mechanisms and concerns about overthinking and differing training paradigms. It uses circuit analysis to compare those changes and reports distinct patterns across distillation/SFT, GRPO, and think on/off models.

1 Introduction

  • Large reasoning models use explicit chains of thought and are enhanced by post-training methods such as SFT and RL, as well as additional test-time compute.Source: §1 · p. 1
  • The mechanisms behind post-training's reasoning gains remain unclear, alongside debate over whether training adds new problem-solving skills or amplifies latent capabilities.The introduction also describes overthinking on simple tasks and the lack of granular understanding of training paradigms as motivations for the analysis.Source: §1 · pp. 1–2
  • Circuit analysis traces specialized attention-head groups after post-training to investigate how internal mechanisms relate to reasoning performance.Figure 1 depicts internal computations at checkpoints and newly activated heads influencing checkpoint performance.Source: §1 · p. 2, Figure 1 · p. 2
  • Distillation and SFT add many reasoning heads in different layer ranges, while GRPO dynamically searches among heads in step with task-reward fluctuations.Distillation heads are mostly early-to-mid layer; SFT heads are mid-to-late layer. GRPO makes targeted edits to existing knowledge and pathways rather than building entirely new ones.Source: §1 · p. 2
  • Think-on mode has no exclusive reasoning heads, whereas think-off activates many compensatory heads that can temporarily help when disabled or scaled down but matter for robust problem solving at higher sampling coverage.The introduction also notes that post-trained models can overthink simple tasks with long, costly reasoning chains.Source: §1 · pp. 1–2

2 Preliminary

The preliminary section represents transformer computation as a graph of model components and defines circuits as behavior-focused subgraphs. It describes edge attribution with integrated gradients and a faithfulness check, alongside a complementary sparse-feature analysis.

2 Preliminary

  • A transformer circuit is a directed acyclic graph whose nodes include attention heads, MLP modules, input embeddings, and output logits; edges encode contributions to later residual-stream inputs.Source: §2 · pp. 2–3
  • A circuit is a behavior-focused subgraph, and EAP-IG ranks edges using integrated gradients before selecting a sparse set and checking whether non-circuit-edge ablation preserves task performance.The output signal is a divergence between clean and interpolated logits, typically KL divergence; the path integral is approximated with Riemann-sum steps.Source: §2 · p. 3
  • The edge score uses differences between clean and corrupted activations and averages gradients along the straight-line path between them.Source: §2 · p. 3
  • The analysis also uses Sparse Feature Circuits as a complementary approach.Source: §2 · p. 3

3 Identifying Emergent Attention Heads with Circuits

This section compares baseline and post-trained reasoning circuits to identify emergent heads, then uses ablation and activation scaling to assess their functional roles. Its broader framing is that post-training reshapes internal architecture by strengthening specific attention heads.

3 Identifying Emergent Attention Heads with Circuits

  • The experiment uses circuit analysis and ablation as a proxy for more complex patching experiments to identify and validate emergent heads' causal roles.Source: §3 · p. 3
  • For each task, the researchers map baseline and post-trained circuits using paired clean and corrupted prompts, with clean prompts sampled to elicit reasoning behavior.Baseline and reasoning-model prompts use different answer content; representative circuit examples appear in Figures 15–18.Source: §3 · p. 3
  • Emergent heads are those active in a post-trained circuit but absent from the baseline circuit; comparisons use Qwen families and an additional Llama model trained with SFT or GRPO.The approach also uses an importance-based analysis as qualitative support for the emergence-based distinction.Source: §3 · p. 3
  • Ablation validation disables emergent heads by zeroing their outputs and compares benchmark performance with the intact post-trained model.Table 1 reports pass@1 at temperature 0.6 and 32k context after zeroing roughly 5–10 selected heads; scaling experiments also vary baseline reasoning-head activations and measure performance quantitatively and qualitatively.Source: §3 · p. 4, Table 1 · p. 4
  • The section frames post-training as reshaping internal architecture by strengthening specific attention heads, rather than merely fine-tuning parameters.Source: §3 · p. 4

4 In-Depth Analysis on SFT & Distillation

Distillation and SFT add emergent reasoning heads, but their layer distributions and functional effects vary. Ablations and qualitative analyses show that these heads can aid difficult reasoning while also contributing to performance trade-offs and overcomplicated solutions.

4.1 Distillation heads strongly affect performance

  • Distillation adds substantial, consistently activated heads to existing circuitry; about two-thirds of baseline attention-head nodes and all baseline MLP nodes remain active.Complementary sparse-feature circuit analysis also finds increased Layer-0 attention importance and stronger mid-to-late residual/MLP computations after distillation.Source: §4.1 · p. 4
  • Ablating emergent heads in the distilled model lowers AIME’24 pass@1 from 30.0 to 26.6, while head importance varies across models.TriviaQA-head ablation can collapse scores to zero in the 1.5B model, whereas ablating base-model heads can improve overall benchmark performance in the 7B model.Source: §4.1 · p. 4
  • The effect of head ablation reverses across model families: Qwen2.5-Math is more sensitive to base-model-head ablation than reasoning-head ablation, unlike DeepSeekR1-Distill.This asymmetry indicates that the identified functional units are model-specific rather than a universal shared pool.Source: §4.1 · pp. 4–5

4.2 SFT introduces a large number of attention heads in middle-to-late layers

  • SFT activates many additional heads, about half by step 100, with most in middle-to-late layers and most persisting to the final checkpoint.Figure 2 tracks emergent-head cohorts and activation frequency across checkpoints.Source: §4.2 · p. 5, Figure 2 · p. 5
  • Ablating about 10 SFT-emergent heads sharply lowers every benchmark, often near zero; scaling them 1.3× improves MATH but lowers AMC slightly and AIME’24 substantially.The scaling result is a task-specific trade-off rather than a uniform improvement.Source: §4.2 · p. 5
  • After SFT, overcomplicated solutions and strategy-selection problems introduce more errors than they resolve, producing net degradation on previously solved items.The reported patterns include long substitutions or theory-first detours replacing short algebraic manipulations.Source: §4.2 · p. 5
  • Llama-3.2 also accumulates emergent heads under SFT, but they spread across early-to-mid layers rather than Qwen2.5-Math’s middle-to-late layers.The authors hypothesize that this distributional difference reflects differing baseline mathematical capability.Source: §4.2 · p. 5

5 In-Depth Analysis on GRPO

GRPO dynamically explores and prunes reasoning heads in response to reward trends, producing a small, targeted head set. Its effects vary by task and checkpoint, with reasoning gains sometimes accompanied by losses in numeracy, flexibility, or benchmark performance.

5 In-Depth Analysis on GRPO

  • GRPO head activation fluctuates with reward accuracy, pruning heads that fail to sustain positive reward impact and yielding a small final set with limited overlap with SFT heads.The authors describe this as an iterative search for an effective circuit configuration.Source: §5 · p. 6
  • Activating, retaining, or pruning heads reflects an explore–exploit trade-off, while GRPO primarily re-optimizes existing knowledge and pathways rather than building new ones.The authors characterize these as minimal but potentially high-impact functional edits.Source: §5 · p. 6
  • Halving the 100-step GSM8K heads changes AIME’24 from 13.3 to 3.3, MATH from 56 to 63, and AMC from 38.5 to 42.1.The cited intervention illustrates that suppressing the same heads can hurt one benchmark while helping others.Source: §5 · p. 7
  • With GSM8K training, AIME and AMC rise from 13.3 and 38.6 to about 20 and 43, and MATH from 56 to about 67 at 100–200 steps; later checkpoints lose some gains.With OpenR1-Math-220k training, AIME varies sharply across checkpoints, including function-calling loops.Source: §5 · p. 7
  • GRPO can improve multi-step reasoning and problem structuring while degrading basic numeracy, execution stability, and simple programmatic verification.Mid-to-late checkpoints show signs of overfitting and partial forgetting in core algebra and geometry.Source: §5 · p. 7
  • Llama-3.2 GRPO activates many more heads after checkpoint 500 across early-to-mid layers, unlike Qwen2.5-Math’s more compact pattern.The authors hypothesize that this reflects weaker base capability, while noting the interpretation is correlational rather than causal.Source: §5 · p. 7

6 In-Depth Analysis on Think On/Off

In Qwen3-8B, think-on uses an optimized pathway rather than exclusive reasoning heads, while think-off recruits a broader set of heads. Interventions reveal a trade-off between efficient low-sample reasoning and broader solution exploration as sampling coverage increases.

6 In-Depth Analysis on Think On/Off

  • Think-on circuits largely overlap with think-off circuits; disabling the <think> mode instead activates many more heads, consistent with compensatory pathway recruitment.The authors interpret the pattern as think-on selecting efficient pathways from a broader attention-head pool.Source: §6 · p. 8
  • Ablating or halving think-off-specific heads can improve performance, especially on difficult AIME benchmarks, but halving them lowers AIME’24 from 30 to 20.In some cases, including GPQA and AIME’25, scaling down outperforms ablation.Source: §6 · p. 8
  • As sampling coverage increases, the baseline think-off model maintains a slight pass@k advantage, while ablated and scaled-down models discover fewer novel solutions.The authors hypothesize that the larger active head set supports exploration of diverse reasoning pathways.Source: §6 · pp. 8–9, Figure 5 · p. 9
  • Ablation initially improves success@k at k ≤2, but that edge fades as coverage rises; scaled-down heads consistently underperform, while baseline exploration benefits at higher k.The reported pattern assigns the think-off heads a dual role: potential noise at low sampling and value for robust problem-solving at larger sampling.Source: §6 · p. 9

7 Conclusion, Limitation, and Future Work

Post-training paradigms reshape reasoning circuits differently, and their head dynamics align with a trade-off between solving difficult problems and avoiding errors on easy ones. The findings are limited by model coverage and prompt-based circuit analysis, and motivate more targeted, reliability-aware training policies.

7 Conclusion, Limitation, and Future Work

  • Distillation and SFT add sustained reasoning pathways, whereas GRPO configures heads through reward-guided appearance and pruning.Think-on mode activates task-relevant heads; think-off mode compensates with more diverse attention heads.Source: §7 · p. 9
  • Deeper, more structured plans accompany better performance on hard problems, while over-reasoning or arithmetic slips can regress on easy items.Source: §7 · p. 9
  • Validation covers the Qwen series and one Llama model, while prompt-based circuits require precise setup and may be vulnerable to polysemanticity.Source: §7 · p. 9
  • The findings motivate training policies that target head activation, shape rewards for both plan quality and calculation reliability, and use per-head influence estimates.Source: §7 · p. 9

A.1.1 Supervised Fine-Tuning (SFT) & Distillation

The section describes SFT and knowledge distillation as ways to adapt a pretrained model to reasoning tasks, including training a smaller model on a teacher’s generated reasoning.

A.1.1 Supervised Fine-Tuning (SFT) & Distillation

  • SFT adapts a pretrained model using curated input-output examples.Source: §A.1.1 · p. 16
  • In the described distillation setup, a teacher generates step-by-step reasoning traces that a smaller student model learns to mimic.Source: §A.1.1 · p. 16
  • The study compares distilled DeepSeek-R1 versions of Qwen2.5 Math with SFT on sampled OpenR1-Math-220k data.Source: §A.1.1 · p. 16

A.1.2 Reinforcement Learning with Verifiable Rewards (RLVR)

RL trains models through interaction and reward signals; RLVR uses verifiable outcomes to reinforce successful reasoning paths. The study uses GRPO for RLVR in mathematical reasoning.

A.1.3 Reinforcement Learning with Verifiable Rewards (RLVR)

  • RL trains a model by interacting with an environment and receiving reward signals.Source: §A.1.2 · p. 16
  • RLVR uses automatically verifiable final answers to reward correct reasoning paths without constraining training to a single gold path.Source: §A.1.2 · p. 16
  • The study adopts GRPO, a memory-efficient and stable PPO variant, to implement RLVR for mathematical reasoning.Source: §A.1.2 · p. 16

A.1.3 Mechanistic Interpretability for LLM Reasoning

Prior work has studied reasoning circuits and specialized heads, while feature-based circuit methods offer a more precise alternative with practical costs. This study analyzes native heads, accepting some residual polysemanticity.

A.1.3 Mechanistic Interpretability for LLM Reasoning

  • Prior circuit studies span object identification, factual and temporal knowledge, and chain-of-thought reasoning, while head analyses associate specialized heads with verification and backtracking.Source: §A.1.3 · p. 16
  • Feature-based circuit approaches seek more precise editing but require substantial extra training and currently cover only a few architectures.Source: §A.1.3 · p. 16
  • The study analyzes native attention heads for computational efficiency and transferability, accepting residual polysemanticity; its math-only setting finds emergent heads unlike entity-tracking fine-tuning [Prakash et al. (2024)].Source: §A.1.3 · p. 17

A.2 Detail of EAP-IG Calculation

This section details the EAP-IG and sparse-feature analyses, experimental and circuit-construction setups, measures, GRPO formulation, and evaluation metrics. It also presents response and circuit examples across SFT, GRPO, and think-on/off models, contrasting successful and unsuccessful outcomes and patterns of head activation and connectivity.

A.2 Detail of EAP-IG Calculation

  • EAP-IG interpolates across the full token-embedding sequence, scores destination-position residual inputs, aggregates absolute edge scores across examples, then selects top-n edges and tests faithfulness by ablating non-circuit edges.For next-token objectives, scoring uses the evaluated logit position; for sequence-level objectives, it averages supervised positions.Source: §A.2 · p. 17

A.3 Detail of Sparse Feature Circuit Analysis

  • The SAE analysis finds rising Layer 0 Attention importance and progressive mid-to-late Residual Stream importance, with later MLP gains less dominant; its block-level granularity cannot identify specialized heads.Figure 12 presents component-level importance; the authors retain head-level EAP-IG as their primary framework.Source: §A.3 · p. 18

A.4 Experimental Setup

  • The experiments compare Qwen math-instruct baselines, DeepSeek-R1 distilled Qwen models, and Qwen3-8B think-on/off, with Llama-3.2-1B added for re-implementation.Training uses OpenR1-Math-220k and GSM8K; evaluation spans AIME’24/’25, AMC, GPQA, MATH-500, and TriviaQA. GRPO checkpoints are saved every 100 steps for temporal circuit analysis.Source: §A.4 · pp. 18–19

A.5 Circuit Construction Setup

  • Circuit construction uses EAP-IG with ig-step 100 and top-n 5000, then simplifies circuits at τ = 0.1; response samples from baseline and reasoning models provide construction prompts.Figures 15–18 show examples, including Llama3.2 1B.Source: §A.5 · pp. 19–20

A.6 Detail of Effect and Importance Measure

  • Head importance sums absolute EAP-IG scores from each head’s outgoing edges and is globally normalized; a symmetric effect measure compares pre- and post-training importance within thresholded circuits.The effect lies in [−1, 1], with positive values indicating increased post-training importance and negative values indicating a decrease.Source: §A.6 · pp. 20–21

A.7 Detail of Ablation Setup and Interpretation

  • Ablation is a task-level head-removal sensitivity test: zero ablation removes the head’s entire residual-stream write, and large performance drops indicate necessity under complete removal, not a distribution-faithful mechanistic effect.Mean ablation preserves a reference mean but was often weakly discriminative in open-ended math tasks because constructing a faithful reference distribution is difficult; the authors leave counterfactual, resample, and optimal ablation for future work.Source: §A.7 · pp. 21–22

A.8 Detail of GRPO Formulation

  • In the outcome-reward GRPO variant, a scalar reward per response yields a group-normalized, value-free advantage shared across its tokens; clipped updates and KL regularization constrain policy changes.The implementation uses OpenR1 with Math-220k to compare the base model with its reasoning-trained version.Source: §A.8 · p. 22

A.9 Detail of Evaluation

  • Pass@k estimates the chance of at least one correct answer among k attempts, whereas success@k measures finding a correct answer within the first k sequential samples.The paired metrics distinguish theoretical capability from practical generation efficiency.Source: §A.9 · pp. 22–23

A.10.1 Example of SFT

  • The baseline repeats the same equation without reaching a conclusion, whereas the SFT response works through the walk-time problem and gives 204 minutes.The SFT response parses the shared coffee-shop time and finishes with \boxed{204}.Source: §A.10 · p. 23, §A.10.1 · p. 24

A.10.2 Example of GRPO

  • For the walk-time problem, the GRPO response derives the needed relationships and gives a final answer of 204 minutes.It includes the 24-minute coffee-shop stop in its final total.Source: §A.10.2 · p. 25

A.11 Example of Qualitative Analysis: Newly Incorrect

  • On the complex-number problem, the baseline correctly calculates the largest possible real part as 540 using Python.The response presents the result as \boxed{540}.Source: §A.11 · p. 26

A.11.1 Example of SFT

  • The SFT response initially computes the complex-number answer correctly, but continues calculating, goes wrong, and leaves the output incomplete.Its response begins by parametrizing z and deriving the expression to maximize, then rechecks its calculation after stating 540.Source: §A.11.1 · p. 27

A.11.2 Example of GRPO

  • On the complex-number problem, the GRPO response makes a calculation mistake and reports 1134 as the largest possible real part.The response derives a real-part expression before giving that result.Source: §A.11.2 · p. 28
  • In Qwen-2.5-Math-1.5B AIME circuits, SFT activates more attention heads, while GRPO shows more complexly connected specific nodes associated with internalized high-level mathematical reasoning.The figure presents these as example circuits after post-training.Source: Figure 16 · p. 36
  • In Qwen3-8B AIME circuits, think-off activates more attention heads, while think-on has more complexly connected specific nodes associated with internalized high-level mathematical reasoning.The caption notes that the think-on connectivity resembles the GRPO circuit example.Source: Figure 17 · p. 37
  • The appendix also shows Llama 3.2 AIME circuits after SFT and GRPO using the OpenR1-Math-220k dataset.The caption does not state a comparative outcome for these circuits.Source: Figure 18 · p. 37

Related papers

Loading 2509.25758v2…