Source-linked AI summary
Attribution Patching Outperforms Automated Circuit Discovery
Aaquib Syed, Can Rager, Arthur Conmy
TL;DR
The paper addresses the difficulty of scaling automated circuit discovery and introduces Edge Attribution Patching, which uses a linear approximation of activation patching to rank computational-graph edges. EAP outperforms existing methods overall while requiring only two forward passes and one backward pass, though its applicability depends on nonzero metric gradients and its approximation can be inaccurate.
Problem
Existing circuit-discovery methods use activation patching, while scalable explanations of widely deployed large models remain limited.
Method
Edge Attribution Patching linearly approximates activation patching, scores all computational-graph edges, and retains the top k edges.
Results
EAP outperforms previous circuit-discovery methods overall while using a constant number of forward and backward passes instead of ACDC’s exponentially scaling forward passes.
Takeaways & Limitations
Future circuit-discovery experiments can run EAP first and then apply ACDC to the prepruned subnetwork to improve identification of relevant edges.
Takeaways & Limitations
Attribution patching can fail when the metric’s gradient is the zero vector, and its scores may only weakly approximate activation-patching effects.
Abstract
from arXiv · showhide
Automated interpretability research has recently attracted attention as a potential research direction that could scale explanations of neural network behavior to large models. Existing automated circuit discovery work applies activation patching to identify subnetworks responsible for solving specific tasks (circuits). In this work, we show that a simple method based on attribution patching outperforms all existing methods while requiring just two forward passes and a backward pass. We apply a linear approximation to activation patching to estimate the importance of each edge in the computational subgraph. Using this approximation, we prune the least important edges of the network. We survey the performance and limitations of this method, finding that averaged over all tasks our method has greater AUC from circuit recovery than other methods.
1 Introduction
The introduction motivates scaling mechanistic interpretability to large models and presents automated circuit discovery as a way to automate circuit identification. The paper introduces Edge Attribution Patching and benchmarks it against existing methods while examining its limitations.
- Mechanistic interpretability attributes model behaviors to components and seeks subnetworks responsible for specific tasks.
- Manual mechanistic interpretability has generally remained limited to relatively small models because end-to-end circuit analysis has not scaled.
- Scaled interpretability matters because large models are widely deployed, yet their mechanisms and failure modes remain poorly understood.
- Automated Circuit Discovery prunes computational-graph edges that do not affect a task, using activation patching to estimate edge importance.
- The paper introduces Edge Attribution Patching, benchmarks it against existing circuit-discovery methods, and analyzes its limitations.
2 Related Work
Related work frames automated circuit discovery as finding task-relevant computational subgraphs and situates the paper within activation-patching and transformer-circuit interpretability.
- Automated Circuit Discovery finds the important subgraph of a model’s computational graph for performance on a particular task.
- Existing circuit-discovery algorithms include efficient heuristics and gradient-descent-based methods, while differing from compression methods in their circuit-oriented goal.
- Activation patching analyzes individual components through targeted manipulations of activations during a forward pass.
- Transformer circuits model attention heads and MLPs as nodes that read and write information through the residual stream.
3 Edge Attribution Patching
Edge Attribution Patching (EAP) estimates edge importance through a first-order attribution approximation, then retains the highest-scoring edges to form task-specific circuits. It reduces circuit-discovery computation to two forward passes and one backward pass, but fails when the metric gradient is zero.
- 3.1 Activation Patching: EAP identifies relevant model components by treating language models as directed acyclic graphs and targeting small subgraphs that retain task performance.The method measures edge importance through activation interventions and compares attribution patching with activation patching.
- 3.1 Activation Patching: Activation patching measures an edge’s counterfactual importance by replacing its activation with one from another forward pass and evaluating the resulting metric change.The metric may represent loss or logit difference, and do-notation emphasizes that the manipulation is causal.
- 3.2 Attribution Patching: Attribution patching linearly approximates activation-patching effects with a first-order Taylor expansion, estimating scores for many edges using two forward passes and one backward pass.The approximation uses the activation difference between corrupted and clean inputs and the gradient with respect to the clean activation.
- 3.3 Edge Attribution Patching: EAP computes absolute attribution scores across clean-corrupted input pairs, sorts all edges by score, and retains the top k edges as the circuit.The reported score is |∆eL|, averaged over the input pairs.
- 3.3 Edge Attribution Patching: Attribution patching does not work when the metric gradient is the zero vector; task-specific metrics avoid the zero-gradient issue associated with KL divergence.KL divergence is zero without ablations, making that point a global minimum with zero gradients.
4 Results
The evaluation compares EAP with ACDC and other circuit-discovery methods on three task circuits, using prior identified subnetworks as ground truth. EAP generally achieves stronger circuit recovery than ACDC while requiring substantially less computation, though KL-based ACDC wins on one plotted task.
- 4.1 Edge Attribution Patching vs Activation Patching vs ACDC: EAP, ACDC, and reference methods are evaluated on Indirect Object Identification, Docstring, and Greater-Than circuits using previously identified subgraphs as ground truth.The comparison also includes ACDC with task-specific metrics, ACDC with KL divergence, and repeated activation patching for Docstring.
- 4.1 Edge Attribution Patching vs Activation Patching vs ACDC: EAP has the maximal AUC in Figure 2a–2b, while ACDC with KL Divergence outperforms EAP in Figure 2c.The curves compare circuit-recovery performance across EAP, ACDC variants, and, for Docstring, activation patching.
- 4.1 Edge Attribution Patching vs Activation Patching vs ACDC: EAP is competitive with ACDC for circuit recovery while using a constant number of forward and backward passes instead of ACDC’s exponentially scaling forward-pass requirement.The paper concludes that this computational difference makes EAP outperform previous circuit-discovery methods overall.
- 4.2 Validating EAP Attribution Scores: For IOI, attribution scores for circuit edges tend to spread farther from zero, with only 6 non-circuit edges outside [−0.25, 0.25].This distribution provides qualitative separation between edges inside and outside the known IOI circuit.
5 Limitations
The paper finds that attribution patching can substantially overestimate activation-patching effects, yet combining EAP with ACDC can improve circuit identification. It also notes that interpretability metrics do not precisely measure meaningful human understanding.
- 5 Limitations: The authors investigate whether EAP’s success reflects accurate approximations and whether ACDC remains useful, finding no and yes, respectively.The Docstring task serves as the case study because of its small model size.
- 5.1 How faithful are Attribution Patching’s approximations?: R2 = 0.27 indicates a fairly weak correlation between attribution-patching and activation-patching scores.The best-fit gradient is 0.531, suggesting attribution patching estimates activation-patching effects as roughly twice as important.
- 5.1 How faithful are Attribution Patching’s approximations?: A concave attribution-score curve causes the linear approximation to overestimate an edge’s activation-patching effect.This behavior was also observed for other outlier edges in the corresponding ellipse.
- 5.2 Is there any further use for ACDC?: EAP followed by ACDC shows increased performance compared with EAP alone on the Docstring task.The evaluation varied EAP and ACDC pruning thresholds independently and compared TPR and FPR against EAP’s ROC curve.
- 5 Limitations: Interpretability metrics do not precisely capture meaningful human understanding because known circuits are unavailable for unfamiliar tasks and human circuits are imperfect.These limitations make recovered human circuits an imperfect evaluation target.
6 Conclusion
The conclusion reports that EAP outperforms ACDC in circuit identification while being substantially faster, and that combining EAP with ACDC can further improve relevant-edge identification.
- 6 Conclusion: EAP outperforms ACDC in identifying circuits while being substantially faster to compute.The conclusion presents this as evidence from the paper’s evaluation.
- 6 Conclusion: EAP’s advantage is notable because it approximates the activation-patching method used by ACDC.The conclusion describes this relationship as surprising.
- 6 Conclusion: Running ACDC on EAP’s prepruned subnetwork can improve identification of relevant edges.The authors therefore suggest using EAP first and ACDC second in future circuit-discovery experiments.
7 Author Contributions
The authors describe contributions spanning method development, empirical benchmarking, and analysis of Edge Attribution Patching’s limitations.
- 7 Author Contributions: The authors contributed to the paper’s figures, experiments, and code.All authors worked on these components.
- 7 Author Contributions: Aaquib Syed and Can Rager proposed combining ACDC with attribution-patching methods and implemented initial prototypes.
- 7 Author Contributions: Arthur Conmy advised attributing edges rather than nodes, and Aaquib Syed made the first findings that this outperformed Automatic Circuit Discovery.
A EAP Subnetworks
The figures present EAP-derived subnetworks, attribution-score distributions, and performance comparisons for combining EAP with ACDC.
- Figure 6 shows the subnetworks produced by EAP at different pruning thresholds.
- Figure 7 compares attribution-score distributions for the Docstring and Greater-Than tasks.
- Figure 8 reports Youden’s-J statistics for combining EAP and ACDC on the Docstring task.
D Further failures of attribution patching approximation
Attribution patching can fail when its linear approximation poorly matches the activation-patching curve, particularly for embedding-originating edges in the Docstring task.
- Embedding-originating edges produce misleading attribution-patching results in further Docstring-task cases.The affected edges originate from positional or token embeddings.
- Concave activation-patching curves make the linear fit used by attribution patching poor.
E Edges Roles in IOI
The IOI analysis relates edge-role attribution scores to the functional roles of attention heads and explains why name-mover classes receive directional scores.
- Name-mover edges have largely negative attribution scores, while negative name-mover edges have largely positive scores in IOI.Ablating name movers lowers the logit difference, whereas ablating negative name movers improves it.
- Previous-token, duplicate-token, induction, and S-inhibition edges have attribution scores centered around zero.
- Edge roles are defined by the roles of their origin nodes in the IOI circuit.
- ACDC iterates over residual edges and edges connecting the residual stream with query, key, and value calculations.
- Gradients for all relevant edge-attribution terms can be computed efficiently in one backward pass.For residual edges, the gradient with respect to the endpoint suffices because the endpoint input sums incoming edges.