Source-linked AI summary
Attention as a Routing Graph: Live Circuit Extraction from a Single Forward Pass
Ash Manvi, Samreena Tajreen
TL;DR
Circuit discovery for language models often relies on repeated interventions, while raw attention alone is too weak to explain behavior. This paper extracts a small answer-directed routing graph from one forward pass and tests it against random edges. On induction and IOI, ablating the extracted edges produces larger score drops than equal-sized random sets, but the result is a partial sketch rather than a complete circuit atlas.
Problem
Established circuit-finding methods use repeated interventions, while raw attention alone is too weak as an explanation.
Source: §1 · p. 1Method
The method reads attention as a routing graph, keeps a small answer-directed edge set, and tests its ablation against random edge sets of the same size.
Source: §2 · pp. 2–3Results
In all six task-model cells, ablating the extracted edges lowers task scores more than equal-sized random ablations, with Wilcoxon tests reporting p<10^-10.
Source: §3.1 · p. 3Takeaways & Limitations
The checked routing sketch carries causal signal on induction and IOI, while remaining an incomplete account of the circuits.
Source: §3.3 · p. 4, §5 · p. 6The method is a candidate-route ranking signal, not a complete attribution method or circuit discovery engine; fixed k and hop depth may under-cover longer prompts.
Source: §4 · pp. 5–6
Abstract
from arXiv · showhide
Finding circuits in language models usually means running many careful interventions. We try something simpler: treat attention as a routing map from one forward pass, keep a small set of routes that point toward the answer, and ask whether those routes actually matter. They often do. On induction and IOI (tasks where the "right" circuit is already known), ablating our extracted edges hurts the model much more than ablating a random set of the same size. We evaluate n=100 prompts per cell on GPT-2 Small, GPT-2 Medium, and Pythia-410M, with paired gap tests and bootstrap confidence intervals. The extract step costs one forward; a head-by-head patch sweep costs about two orders of magnitude more. We are not claiming a complete circuit atlas. We are claiming a cheap sketch that carries real causal signal on known tasks, with clear failure modes when it does not. Code and evaluation artifacts are at https://github.com/Aquinf03/live-circuit-routing.
1 Introduction
Prior circuit-mapping methods for well-studied behaviors rely on repeated interventions, while raw attention alone is too weak to serve as an explanation. The paper asks whether a small answer-directed graph from one forward pass can beat chance under ablation, yielding a fast but explicitly partial sketch.
1 Introduction
- Repeated interventions make established circuit-finding methods costly, while raw attention alone is too weak as an explanation.The paper frames this as a gap between trusting attention and paying for a full offline search.Source: §1 · p. 1
- The central test is whether a small subgraph extracted from one forward pass hurts performance more than chance when ablated.The subgraph is directed toward the answer.Source: §1 · p. 1, Figure 1 · p. 2
- A positive result would provide a fast, partly causal sketch without replacing heavier circuit-finding methods.The paper presents the sketch as limited rather than a complete circuit account.Source: §1 · p. 1
2 What we do
The method turns attention into a head-identified routing graph, walks backward from the answer to retain a small set of edges, and checks that set against random controls. Its claim test is intentionally short: routing, a short walk, and a fair control.
2 What we do
- The pipeline scores each edge by attention weight times value-vector magnitude, then retains top incoming routes while walking backward from the answer.The graph keeps head identity, ignores the BOS sink, and skips self-loops; the default walk uses k=15 and four hops.Source: §2 · p. 2
- The edge score is a cheap ranking signal that accounts for both attention weight and the transformed value, unlike raw attention alone.The paper uses it to rank discrete routes rather than to provide a complete attribution analysis.Source: §2 · p. 2
- The verification step ablates the retained edges, renormalizes remaining attention, and compares the task-score drop with ten random edge sets of equal size.The score is the target logit for induction and the name logit-difference for IOI.Source: §2 · p. 3
- The paper treats the extracted subgraph plus its random-control check—not a complete circuit atlas—as the claim test.The pipeline is presented as routing, a short backward walk, and a fair control.Source: §2 · p. 3
3 What we find
The evaluation uses 100 clean prompts per cell across GPT-2 Small, GPT-2 Medium, and Pythia-410M, with paired gap tests. The extracted subgraphs beat random controls across induction and IOI, while results also show the method’s cost, partial overlap with known heads, and failure modes.
3.1 The sketch beats random
- Across all six cells, extracted-edge ablation beats random (Wilcoxon p<10^-10); GPT-2 Small induction drops the target logit 4.42 versus 0.09 (gap +4.33).Every cell’s 95% bootstrap confidence interval for the gap excludes zero.Source: §3.1 · p. 3, Table 1 · p. 4
3.2 It is also cheap
- Building the sketch takes one forward; verification takes about a dozen, versus 145 for a head sweep and roughly 100× the CPU speed of head-patching.The head sweep is a lower bound on the cost of full edge search.Source: Figure 3 · p. 4, §3.2 · p. 4
3.3 It overlaps known heads, but not all of them
- The induction walk often selects L4H11, L5H5, and L5H8, but later named heads appear only sometimes, so strong ablation gaps do not imply full circuit recovery.Figure 4 marks known induction-related heads when they dominate a route.Source: §3.3 · p. 4, Figure 4 · p. 5
3.4 When it fails
- Failures concentrate on Medium induction (19/100), Pythia induction (11/100), and Medium IOI (12/100); distractors, short hop budgets, and attention mass not equaling causation are residual modes.Small induction, Small IOI, and Pythia IOI each have 0/100 failures.Source: §3.4 · p. 5
3.5 What we varied
- On GPT-2 Small induction, k=5 is unreliable, k≥10 is fine, and the default k=15 is stable; the result also needs enough graph coverage and the right layers.Early layers alone recover most of the gap, while changing the prompt template or adding filler sentences does not kill it.Source: §3.5 · p. 5
4 What we are not saying
The method is a routing sketch with a causal check, not a complete discovery engine. Its evidence is limited to small open models and two synthetic tasks, and its random-edge control removes less attention mass than the extracted set.
4 What we are not saying
- The score ranks candidate routes and the claim rests on the ablation gap, not complete attribution; heavier circuit tools remain relevant when completeness is needed.The paper does not name features or train sparse autoencoders.Source: §4 · p. 5
- Evaluation covers small open models and two synthetic tasks; transfer to larger models or messier behavior remains open.The verification zeros and renormalizes attention, while the control samples edges uniformly.Source: §4 · p. 5
- Extracted edges remove more attention mass than same-sized uniform random sets, so part of the measured gap may reflect mass rather than routing; fixed k and hop depth may under-cover longer prompts.The size of the mass-matched control’s effect is unmeasured.Source: §4 · p. 6
5 Conclusion
The paper concludes that an answer-directed routing sketch carries useful signal on induction and IOI at a fraction of head-patching’s cost, without claiming a complete circuit atlas. Code, configs, and experiment artifacts are public.
5 Conclusion
- On induction and IOI, extracted routes carry useful signal across several models at a fraction of head-patching’s cost, while remaining a partial sketch with failure modes.The authors frame the sketch and its check as useful, not as a finished circuit atlas.Source: §5 · p. 6
- Code, configs, and experiment run artifacts are publicly available.The repository is https://github.com/Aquinf03/live-circuit-routing.Source: §5 · p. 6