Source-linked AI summary
Towards Automated Circuit Discovery for Mechanistic Interpretability
Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, Adrià Garriga-Alonso
TL;DR
Mechanistic interpretability has produced detailed circuit explanations, but extracting circuits remains labor-intensive and difficult to scale. This paper systematizes the workflow and automates circuit discovery with ACDC and related algorithms. Across prior tasks, ACDC shows competitive but setting-sensitive recovery, while its scope remains limited to extraction rather than functional interpretation.
Problem
Manual inspection makes extracting circuits difficult to scale to larger models, more behaviors, and complicated behaviors composed of many sub-circuits.
Method
The paper systematizes the mechanistic interpretability workflow and introduces ACDC, SP, and HISP-based methods to extract behavior-related computational subgraphs.
Results
ACDC has competitive AUC performance but is sensitive to corrupted distributions and can fail in some settings.
Takeaways & Limitations
ACDC can automate part of interpretability work and may help scale circuit research to behaviors in larger models.
Takeaways & Limitations
The work automates circuit extraction before functional interpretation, which remains for researchers to perform.
Abstract
from arXiv · showhide
Through considerable effort and intuition, several recent works have reverse-engineered nontrivial behaviors of transformer models. This paper systematizes the mechanistic interpretability process they followed. First, researchers choose a metric and dataset that elicit the desired model behavior. Then, they apply activation patching to find which abstract neural network units are involved in the behavior. By varying the dataset, metric, and units under investigation, researchers can understand the functionality of each component. We automate one of the process' steps: to identify the circuit that implements the specified behavior in the model's computational graph. We propose several algorithms and reproduce previous interpretability results to validate them. For example, the ACDC algorithm rediscovered 5/5 of the component types in a circuit in GPT-2 Small that computes the Greater-Than operation. ACDC selected 68 of the 32,000 edges in GPT-2 Small, all of which were manually found by previous work. Our code is available at https://github.com/ArthurConmy/Automatic-Circuit-Discovery.
1 Introduction
Mechanistic interpretability seeks to explain model behavior through human-understandable algorithms, but extracting functional circuits still depends heavily on manual inspection. This work systematizes that workflow and automates circuit extraction with ACDC and related methods.
- Mechanistic interpretability reverse-engineers model components into human-understandable algorithms and represents models as computational graphs.
- Manual circuit extraction limits scaling mechanistic interpretability to larger models, more behaviors, and behaviors composed of many sub-circuits.
- The paper systematizes the workflow used in prior mechanistic interpretability research.
- ACDC automates identifying a behavior-implementing subgraph, while SP and HISP are adapted for the same circuit-discovery task.
- The paper introduces quantitative metrics for evaluating circuit-extraction success and reports ablation and qualitative studies.
2 The Mechanistic Interpretability Workflow
Prior mechanistic interpretability studies follow a recurring workflow: elicit and measure a behavior, define a computational graph, and iteratively patch components to isolate a circuit. ACDC automates the patching-based circuit-isolation step before researchers interpret component functions.
- Step 1: Choose a behavior: Researchers create prompts that elicit a defined behavior and choose a metric measuring task performance.
- Step 2: Define the graph: Researchers choose a computational-graph granularity, ranging from heads and MLPs to neurons, token positions, or query, key, and value activations.
- Step 3: Patch components: They perform iterative patching experiments to remove unnecessary components and connections from the model.
- Iterative investigation: Researchers vary datasets, metrics, and granularity until they are satisfied with the explanation of circuit components.
- Step 4: Explain components: After isolating a subgraph, researchers formulate and test hypotheses about the functions implemented by its nodes.
- ACDC workflow: Figure 2 depicts ACDC's recursive process of measuring patched connection effects, removing connections below threshold τ, and returning a subgraph.
3 Automating circuit discovery (Step 3)
ACDC automates circuit discovery by recursively pruning a computational graph using activation patching and a performance threshold. The paper also adapts SP and HISP to extract subnetworks for comparison.
- ACDC: ACDC starts from the output and traverses toward inputs, removing incoming edges that do not reduce performance on the selected metric.
- ACDC: The algorithm receives a computational graph, task dataset, and threshold τ, and returns a subgraph H ⊆ G.
- Evaluation: For each candidate subgraph, omitted edges are replaced with activations from corrupted inputs, and the resulting predictions are evaluated using average KL divergence.
- Implementation: ACDC's parent-iteration order is a hyperparameter, and one experiment found that this order affected results.
- Comparison methods: SP learns a component mask using an accuracy-and-sparsity objective, while HISP ranks heads and retains only the top k.
4 Evaluating Subgraph Recovery Algorithms
The evaluation compares circuit-recovery methods using ROC-based metrics and stand-alone subgraph properties, finding that performance depends strongly on distributions and metrics. ACDC is competitive overall but is not robust, while lower-KL, smaller circuits provide evidence against including irrelevant components.
- The evaluation asks whether methods recover the underlying algorithmic subgraph and avoid including components unrelated to the elicited behavior.
- Circuit recovery is evaluated as edge classification using ROC curves, with true-positive and false-positive rates corresponding to recovery and exclusion objectives.
- ACDC has competitive AUC performance against gradient-based methods, but results are sensitive to corrupted distributions and fail in some settings.
- ACDC achieves higher AUC than both other methods on IOI, Greater-Than, and tracr-reverse tasks, making it optimal overall in the reported comparison.
- A single optimized metric can miss harmful negative components, while practitioner-reported ground-truth circuits may contain extraneous edges or omit important ones.
- Lower KL divergence with fewer edges indicates a better subgraph because it preserves behavior while reducing components that may not participate in the task.
5 Related work
Related work situates automated circuit discovery within mechanistic interpretability, pruning, causal interpretation, and computational-subgraph research. ACDC differs from conventional pruning because it targets task-specific explanatory circuits rather than compression or faster inference.
- Mechanistic interpretability reverse-engineers model components into human-understandable algorithms and represents circuits as functional computational subgraphs.
- Neural-network pruning generally seeks sparse connectivity for faster inference or reduced storage, rather than interpretability.
- ACDC seeks compressed networks that reflect task-specific computational circuits and generally slows forward passes instead of accelerating them.
- Related research connects circuit analysis with causal explanations and computational-path views of residual and transformer models.
6 Conclusion
The paper systematizes mechanistic interpretability and introduces ACDC to automate circuit discovery, while showing that current methods remain sensitive to metrics, datasets, and hyperparameters. ACDC can automate part of interpretability work, but does not yet fully automate the workflow or support societally important exact-behavior applications.
- The workflow identifies a behavior with a metric and dataset, uses activation patching to locate involved units, and iterates variations until the model’s algorithm is understood.
- ACDC systematically conducts the activation patching experiments needed to find a behavior-relevant circuit of abstract units.
- ACDC and SP recover most of the compositional circuit implementing a language-model behavior, while ACDC with zero activations fully recovers toy-model circuits.
- ACDC and SP can miss circuit components, including negative name mover heads, so they do not fully automate the activation-patching step.
- Performance varies with hyperparameter and metric choices, and current methods show sensitivity to corrupted distributions and non-robust behavior in some settings.
- ACDC remains a research tool and is not mature enough for applications where determining a model’s exact behavior is societally important.
C.3 Alternatives to minimizing a metric
The paper compares alternative objectives and component-importance adaptations for automated circuit discovery. It finds KL divergence preferable to task-specific metrics because alternatives can be unstable or over-optimized.
- Alternative objectives: Matching the model’s performance can be unstable, producing subgraph logit differences from 1.5 to 5.0 versus the model’s 3.55.This can cause inconsistent component selection and dependence on parent-iteration order.
- Alternative objectives: Only including components causing small performance changes focuses on local metric changes rather than the base model’s metric value.The criterion is |F(Hnew) − F(H)| < τ.
- Metric comparison: Using task-specific metrics made ACDC less effective than KL divergence and worse than logit difference on IOI under random ablations.The comparison used ROC curves.
- Metric comparison: KL divergence was the least flawed metric tested and recovered several circuits previously found with other metrics.It also avoids specifying output labels for next-token-prediction tasks.
- Method adaptations: The study adapts Subnetwork Probing and HISP by removing or generalizing components such as probes, activation types, and internal model components.The adaptations support comparisons with ACDC using corrupted activations and internal query, key, value, or MLP outputs.
E Experimental study of algorithm design
The experimental study varies metrics, activation patching, and evaluation granularity to test circuit-recovery design choices. Results are mixed: zero-patching can excel on specific circuits, while metric choice strongly affects robustness.
- Activation patching: ACDC with zero-patching perfectly detects the tracr circuits in the reported experiments.The result appears in the edge-wise ROC analyses using zero activations.
- Overall findings: The design-choice results present a mixed picture of whether ACDC or SP is better overall.Tables 2 and 3 summarize the comparisons with AUCs for the tested choices.
- Metric choice: Using task-specific metrics made ACDC better on Docstring but worse on Greater-Than and IOI than minimizing KL divergence.The authors interpret this variation as evidence that ACDC is not robust.
- Metric choice: The authors prefer KL divergence because task-specific metrics can be over-optimized, causing recovered circuits to perform the task more than the original model.Figure 8 compares the task-specific metric achieved by the methods.
- Evaluation granularity: Node-level ROC results were broadly similar to edge-level ROC results.Node evaluation is included because SP and HISP operate at the node level, whereas ACDC operates at the edge level.
F.1 Further details on the IOI experiments
The IOI experiments compare recovered subgraphs with a hand-specified GPT-2 Small circuit at multiple granularities and thresholds. ACDC recovers sufficient IOI components, but lower thresholds introduce extraneous heads and still miss some negative components.
- Experimental setup: The IOI dataset contained 50 examples from one template, with corrupted examples drawn from the ABC dataset.The main experiment did not split each attention head into separate query, key, and value calculations.
- Recovered circuit: All nine heads recovered in the main ACDC IOI figure belong to the published IOI circuit and span three head classes.Those heads are sufficient to complete the IOI task.
- Circuit representation: The low-level IOI implementation contains 1041 edges after expanding group connections across query, key, value, and MLP components.The figure represents each group-level connection as all pairwise member connections.
- Reference circuit: The ground-truth IOI circuit has logit difference 3.24 versus the model’s 4.11 and KL divergence 0.44 from the original model.These values characterize the reference circuit used for comparison.
- Threshold effects: Lowering the ACDC threshold recovered Previous Token and Negative Name Mover Heads, but also many heads not documented in the IOI paper.The comparison indicates KL divergence performs better than logit-difference maximization for finding Negative Name Movers, though recovery remains non-optimal.
G Greater-Than task: details and qualitative evidence
The docstring experiments compare ACDC circuits with manual interpretations while varying the optimized metric and threshold. ACDC recovers the central manually identified components, with specificity and performance depending on the setup.
- ACDC is applied to the docstring prompts using a randomized corrupted dataset that randomizes variable names in definitions and docstrings.
- ACDC recovers all five manually identified overlapping heads and additionally identifies head 1.0, later added manually to improve performance.
- 258 edges: the lower-threshold KL-divergence run includes head 1.2, which is absent from the smaller circuit at τ = 0.095.
- 93% fewer edges: the τ = 0.067 logit-difference run recovers the relevant manual heads, including 1.2, while achieving better scores on all metrics than the manual head-based circuit.
- ACDC operates on attention heads and query, key, and value nodes, whereas the manual analysis separately considers head outputs at each token position.
H.2 Additional docstring experiments
Additional experiments examine ACDC on tracr programs and compare corrupted activations with zero activations. At neuron level, ACDC recovers compact or perfect circuits, while zero activations perform substantially worse for the docstring-style experiment.
- 177 edges: with zero activations at τ = 0.067, ACDC produces a larger and worse-performing circuit than the 98-edge circuit found with corrupted activations.The zero-activation circuit has KL divergence 3.35 and logit difference −2.895.
- The tracr proportion experiment evaluates the L2 distance between the correct and recovered lists of proportions.
- At neuron level, ACDC adds no unused nodes and produces a more compact visualization than the complete residual-stream state view for the tracr proportion task.
- The tracr-reverse experiment evaluates the L2 distance between one-hot vectors for the recovered list and the true reversed list.
- For tracr-reverse, neuron-level ACDC recovers a perfect graph containing the minimal components required to reverse lists.
J Induction task: details and qualitative evidence
The induction experiments apply ACDC and related methods to a small attention-only transformer on filtered OpenWebText sequences containing repeated-token induction patterns. The setup emphasizes controlled metrics, graph size, and implementation details affecting search results.
- The evaluation uses 40 sequences of 300 tokens and measures KL divergence only at the second B tokens in A, B, …, A, B induction patterns.
- ACDC and other methods are compared using both zero activations and corrupted activations.
- The induction model is a 2-layer, 8-head-per-layer attention-only transformer trained on OpenWebText.
- The computational graph contains 305 edges, although Figure 4 displays only subgraphs with at most 120 edges.
- Increasing head-index order within the reverse-layer traversal was important for achieving better results in Figure 4.
L Reset Network Experiments
Reset-network and OR-gate experiments probe whether circuit-discovery methods recover genuine task structure and whether their node-based search handles nonlinear interactions. The results expose both a useful sanity check and a specific limitation.
- Reset networks permute Q, K, and V head dimensions and MLP biases, disrupting the subject model’s functionality without changing many other model facts.
- Because reset networks do not exhibit the original behavior, successful recovery of a behavior-performing circuit should not be possible and would indicate hallucinated components.
- The three circuit-discovery methods do not identify both inputs to OR gates inside neural networks.
- ACDC recovers only one input to the toy OR gate, while HISP recovers neither and SP recovers one plus several additional nodes.
- ACDC can still be computationally slow because realistic models may contain many edges to search over.
- ACDC and Causal Scrubbing are complementary: ACDC searches broadly over a small hypothesis class, whereas Causal Scrubbing tests hypotheses more methodically and can specify node information.