Source-linked AI summary
Localizing Model Behavior with Path Patching
Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, Aryaman Arora
TL;DR
Neural-network behavior localization is often qualitative and lacks a consensus evaluation method. The paper introduces path patching, which expresses localization hypotheses over paths and quantitatively tests them, then refines induction-head explanations and characterizes a GPT-2 behavior. Its results support more targeted, sparser abstractions, while the method remains distribution- and task-dependent.
Problem
Existing neural-network localization work is often qualitative and ad hoc, while state-of-the-art networks remain difficult to understand and lack robust out-of-distribution guarantees.
Method
Path patching represents important and unimportant paths in a computational graph, removes unimportant-path contributions, and compares the resulting abstraction with the original network.
Results
The method refines an explanation of induction heads and characterizes a GPT-2 behavior, while enabling more precisely targeted interventions and a sparser abstraction than causal tracing and zero ablation.
Takeaways & Limitations
Path patching provides an expressive, principled, and sufficiently efficient formalism for quantitatively testing localization claims on real models.
Takeaways & Limitations
AUE measures sufficiency on the tested distribution, does not establish downstream-task performance, and path patching cannot definitively prove hypotheses over all possible inputs.
Abstract
from arXiv · showhide
Localizing behaviors of neural networks to a subset of the network's components or a subset of interactions between components is a natural first step towards analyzing network mechanisms and possible failure modes. Existing work is often qualitative and ad-hoc, and there is no consensus on the appropriate way to evaluate localization claims. We introduce path patching, a technique for expressing and quantitatively testing a natural class of hypotheses expressing that behaviors are localized to a set of paths. We refine an explanation of induction heads, characterize a behavior of GPT-2, and open source a framework for efficiently running similar experiments.
1 INTRODUCTION
The paper introduces path patching to formalize and quantitatively test claims that behaviors are localized to particular paths, extending analysis beyond qualitative component-level explanations. It applies the method to induction heads and a GPT-2 behavior while providing an open-source framework for experiments.
- Motivation: The framework addresses a need for more rigorous behavioral analysis because state-of-the-art networks remain difficult to understand and existing explanations are often approximate.Compact causal abstractions may be easier to reason about, but their compactness and behavioral fidelity trade off.
- Contribution: Path patching generalizes prior interventions to hypotheses containing any number of input-to-output paths in arbitrary computational graphs.The framework treats paths as more expressive than sets of nodes such as neurons or attention heads.
- Method: The method formalizes localization claims by designating important paths, removing unimportant-path contributions, and comparing the resulting abstraction with the original network.When a hypothesis is rejected, attribution identifies discrepancy sources for iterative refinement.
- Applications: The authors use path patching to test and iteratively refine hypotheses about induction heads in an attention-only transformer.This application examines whether specific pathways mediate induction behavior.
- Applications: The paper also formalizes, tests, and refines a hypothesis about a behavior of GPT-2.These experiments demonstrate the framework on a state-of-the-art language model behavior.
- Contribution: The authors open-source a framework for running path patching experiments.The framework is intended to support similar localization analyses.
2 METHODOLOGY
The methodology formalizes localization as testing whether selected nodes or paths mediate a behavior on a specified input distribution. Path patching constructs counterfactual abstractions, measures their unexplained effects, and supports quantitative comparison while preserving path-specific interactions.
- 2.1 LOCALIZATION: Localization identifies which network components matter for a chosen behavior, while path-based descriptions are more expressive than sets of nodes.The framework represents neurons, attention heads, subspaces, compositions, blocks, and layer ranges uniformly as paths in a computational graph.
- 2.2 CHOOSING THE DATASET: A behavior is defined by input-output pairs on a specific dataset, so localization evidence is explicitly scoped to that distribution.An approximation accurate on one dataset need not transfer to another distribution.
- 2.3 PATH PATCHING WITH NODES AS MEDIATORS: Path patching replaces hypothesized unimportant nodes or paths with their values under a counterfactual input while retaining the reference computation elsewhere.For node mediation, GH(xr, xc) evaluates G on xr except that unimportant nodes receive their values from xc; path mediation uses Treeify(G) to isolate each path.
- 2.3 PATH PATCHING WITH NODES AS MEDIATORS: The average unexplained effect measures the dissimilarity between the original and patched outputs over a joint distribution, with zero as the ideal hypothesis value.Because exact zero is too stringent for modern neural networks, lower AUE is preferred when hypotheses share a dataset and path count.
- 2.4 PATH PATCHING WITH PATHS AS MEDIATORS: Treeify(G) copies shared subtrees so each input copy corresponds one-to-one with a path, allowing a counterfactual value to alter only the selected path.The resulting graph can independently assign xc to the input associated with an unimportant path.
- 2.5 METRICS: Multiple unimportant paths reuse the same counterfactual input by default, and including labels and loss permits unexplained effects to be measured as changes in loss.Loss-based recovered proportions can be misleadingly high when the network is usually correct with high confidence.
3 RESULTS ON INDUCTION
The paper applies path patching to GPT-2’s induction behavior, testing progressively richer hypotheses about which paths through attention heads mediate prediction. The minimal hypothesis fails to capture most behavior, while adding positional and longer-context paths yields better approximations and reveals a distinct parroting heuristic.
- Method: Path patching quantitatively compares hypotheses by replacing contributions along selected paths and measuring how closely the resulting abstraction matches the original model.A hypothesis specifies important paths; contributions from unimportant paths are removed to form an approximate abstraction.
- Experimental setup: GPT-2’s induction behavior is studied in a two-layer attention-only transformer, with head 0.0 identified as a previous-token head and heads 1.5 and 1.6 as induction heads.Experiments use 100K held-out OpenWebText examples, with cross-entropy loss measured at each token.
- Initial hypotheses: The initial hypotheses separately test direct value, direct query, and previous-token-head routes, while All-Initial takes their union of 15 paths.The PTH-K hypothesis restricts the induction heads’ key input to the previous-token head, whereas Direct-V and Direct-Q restrict value and query inputs through skip connections.
- First refinement: positional hypotheses: Positional-K was not significantly improved, indicating that information earlier than j −1 remains useful for the induction mechanism.Inspection suggests some layer-0 heads attend to multiple recent tokens, supporting a longer recent-tokens pattern.
- Second refinement: long induction: K = 3 is the shortest window that performs well for the long positional query-and-key hypothesis, although increasing induction context still leaves room for improvement.The long hypothesis adds key paths from previous tokens beginning at j −K and query paths beginning at i −K + 1.
- Third refinement: repeating entities: Head 1.5 appears to combine induction with parroting, a heuristic favoring previously seen tokens; adding the relevant paths raises All-Final’s explained proportion to 73.5%.All-Final improves by 18.3% over All-Long and by 45.3% over All-Initial, while narrower datasets distinguish induction from repeating-token behavior.
- Overall result: Overall, the very sparse All-Initial hypothesis fails to capture most behavior, whereas a modest number of additional paths produces a substantially better approximation.The refined paths help identify where the minimal mechanistic story is insufficient.
4 PATH PATCHING VS CAUSAL TRACING AND ZERO ABLATION ON GPT-2
On GPT-2, path patching compares targeted path interventions with causal tracing and zero ablation, identifying a smaller set of heads while testing specific behavioral hypotheses. The experiments also show that heads 5.6 and 6.1 affect numerical completions under corruption but not when patched.
- The GPT-2 small experiment runs one path-patching intervention per attention head across 100 reference-counterfactual pairs, with all paths through one head marked unimportant.GPT-2 small contains 144 attention heads in this experiment.
- Path patching identifies fewer influential heads than input corruption, although the paper cannot determine which plot better reflects the true mechanism.Path patching keeps counterfactual inputs closer to the reference distribution, whereas corrupted prompt numbers may be off distribution.
- The method’s targeted dataset, whose counterfactual prompt numbers remain numbers, excludes heads 5.6 and 6.1 from consideration for this behavior.This illustrates how path patching can test narrowly specified localization claims.
- For GPT-2 small, Figure 9 encodes each head’s mean KL divergence after ablation or patching, with darker colors indicating larger divergence.Casual tracing samples noise from N(0, 0.2).
- Corrupting heads 5.6 and 6.1 decreases numerical-completion probability, while patching those heads has no effect from an 88.1% baseline.The result is evidence consistent with these heads recognizing that the token before the hyphen is a number.
5 GREEDILY BUILDING HYPOTHESES
The paper explores a fast greedy baseline for constructing path-patching hypotheses, while noting that hypothesis generation remains labor intensive and greedy selection is not optimal. GPT-2 small results illustrate that a few heads can recover most of the loss.
- Hypothesis generation remains labor intensive because the induction analysis relied on visual inspection and domain knowledge.The authors propose greedy head addition as a quick starting point for later refinement.
- Including 8 of GPT-2 small’s 144 heads as important recovers 98.0% of the loss, compared with 72.8% for only heads 9.1 and 8.8.The figure tests the hypothesis that all paths through other heads are unimportant.
- GPT-2 XL contains 1200 heads arranged as 48 × 25, providing the corresponding larger-model comparison.The supplied passage identifies the model’s head count but does not report the comparison’s numerical outcome.
6 RELATED WORK
The paper situates path patching among causal-abstraction and mediation methods, emphasizing its path-level representation and its distinctions from node-based interventions and alternative faithfulness metrics.
- Path patching is a simpler special case of causal scrubbing and is more computationally efficient because it needs only reference and counterfactual samples.Causal scrubbing generally uses more distinct samples and makes claims about node equivalence classes.
- Unlike Wang et al.’s GPT-2 circuit analysis, path patching uses path hypotheses and avoids faithfulness metrics based on differences of expectations that can cancel across examples.Wang et al. identify a 26-head circuit for indirect-object identification on a synthetic dataset.
- Vig et al. intervene on individual neurons or attention heads by altering important nodes, whereas path patching intervenes on unimportant paths while holding important paths constant.Vig et al. apply causal mediation analysis to Transformer language models.
- Geiger et al.’s interchange interventions operate on semantically specified nodes, while path patching represents localization claims over paths.The supplied passage contrasts the methods’ intervention targets and levels of detail.
- Geiger et al. (2021) identify cliques where interchange-intervention hypotheses hold fully, trading approximate-abstraction measurement for a restricted input set.Their method considers examples as graph vertices and adds bidirectional edges when the hypothesis holds.
- Finlayson et al. measure relative probabilities of correct and incorrect verb tenses, so their method need not explain heads affecting other logits or both logits equally.Their activation-patching study concerns subject-verb agreement in GPT-2 and other transformers.
7 DISCUSSION
The discussion presents path patching as an efficient, expressive localization framework while qualifying what its metrics and demonstrations establish. Its evidence is distribution-specific, tests sufficiency rather than completeness, and does not yet establish downstream-task value or scaling to the largest models.
- 7 DISCUSSION: The induction analysis finds prefix matching and copying relatively separate, with head 1.5 copying both with and without prefix matching.The authors caution that induction heads may perform additional behaviors not characterized in the paper.
- 7 DISCUSSION: Path patching provides more targeted interventions and a sparser abstraction than causal tracing and zero ablation.The discussion frames this as a comparative advantage of path-level intervention.
- 7.1 LIMITATIONS: AUE measures sufficiency rather than completeness, so multiple compact path sets can achieve low AUE when their outputs are sufficient to mediate the behavior.The authors recommend interpreting AUE-based metrics as sufficiency measures rather than completeness measures.
- 7.1 LIMITATIONS: The work does not establish how AUE relates to specific downstream applications, and the authors expect it to be incomplete as a predictor of abstraction quality.This limitation concerns applications such as adversarial-example generation and other downstream uses.
- 7.1 LIMITATIONS: Path patching makes no claims outside the tested distribution because approximately zero or cancelling paths may depend on distribution-specific properties.Low AUE on one distribution therefore does not imply low AUE on a wider distribution.
- 7.1 LIMITATIONS: Path patching cannot reject all false hypotheses: cancellation-sensitive metrics and finite sampling can hide unexplained effects, especially for rare inputs.The paper recommends metrics such as average KL divergence and notes that naive sampling may miss rare, high-impact cases.
- 7.1 LIMITATIONS: Path patching alone cannot definitively prove hypotheses because verification would require testing all possible inputs, which is infeasible for large networks.The authors view it as complementary to mechanistic analysis and other interpretability techniques.
- 7.1 LIMITATIONS: The demonstrations are narrow, and it remains unknown whether path patching scales from the tested 1.5B-parameter setting to the largest models.The authors call for experiments across more diverse tasks and models.
A WHY REUSE THE SAME COUNTERFACTUAL INPUT?
Reusing the same counterfactual input makes path-patching rewrites preserve unimportance, represent additive and canceling interactions correctly, and reduce computation. Distinct counterfactual inputs can instead create off-distribution mixtures that produce unusual downstream behavior.
- Using the same counterfactual input lets rewritten nodes together preserve the original node's unimportance.This guarantee depends on feeding both new nodes the same x_c.
- Shared counterfactual inputs correctly represent behaviors formed by combining contributions from distributed components.Residual networks often behave like additive ensembles, with dropout encouraging gradual degradation when contributions are removed.
- Reusing inputs allows paths that approximately cancel over the reference domain to be excluded together from an explanation.The simplification is restricted to D_r and need not hold on arbitrary inputs.
- Distinct counterfactual inputs can make an averaging node output an off-distribution mixture, potentially causing unusual later behavior.When three functions compute f, reuse yields f(x_c), whereas distinct inputs yield (f(x_c1) + f(x_c2) + f(x_c3))/3.
- For one graph, tests of x → f0 → Y, x → Y, and both together all classify the paths as unimportant.
- Caching repeated subexpressions makes reuse computationally efficient, especially because residual-network path counts grow exponentially with depth.For the tested hypothesis, f0(x_c) is computed once rather than separately for distinct counterfactual inputs.
B IDENTIFYING PREVIOUS TOKEN AND INDUCTION HEADS
The paper identifies previous-token and induction heads from their attention patterns. Head 0.0 matches the previous-token pattern, while heads 1.5 and 1.6 show clear induction patterns.
- Identifying induction heads: Induction heads should attend to earlier occurrences of token sequences, producing short diagonal patterns in attention heatmaps.The heatmap's i-th row uses token i as the query, while column j records attention to token j.
- Identifying the previous-token head: Head 0.0 is identified as a previous-token head because it reliably attends to the immediately preceding token.Its attention plot shows a diagonal line immediately below the main diagonal.
- Identifying induction heads: Heads 1.5 and 1.6 exhibit clear induction patterns in the layer 1 attention plots.
C DEFINING THE INDUCTION AND UNCOMMON REPEAT SUBSETS
The induction subset follows the subset used by Chan et al. (2022), while the uncommon repeat subset selects previously seen tokens outside the validation set's 200 most common tokens.
- Induction subset: The induction subset is the same subset used by Chan et al. (2022).
- Induction subset: A full description of how the induction subset was selected is provided in the referenced external document.
- Uncommon repeat subset: The uncommon repeat subset includes tokens that previously occurred in context but are not among the validation set's 200 most common tokens.The results were robust to varying the number of filtered tokens.
D REPEATED ATTENTION TO PROPER NOUNS
The paper examines head 1.5's attention pattern on repeated proper nouns, including examples such as “Brazil” and “UFC.”
- The figure describes path-patching attribution when head 1.5 is patched.
- Head 1.5 repeatedly attends to proper nouns such as “Brazil” and “UFC” in the illustrated example.Attention paid to the first token, “[BEGIN],” is omitted from the display.
E MODEL REWRITES
Model rewrites refine the computation graph while preserving the original behavior, enabling more fine-grained localization of which components or activation parts matter.
- Rewriting the graph preserves G's behavior on all inputs while expanding the space of paths available for localization.
- Dividing attention layers into heads and token vectors into slices supports localization to particular heads or tokens.
- The residual rewrite generalizes subspaces and mean ablation by decomposing activations into separately testable parts.
- Projection-based decomposition tests the importance of a subspace projection and its remainder individually.
- Mean-based decomposition treats the constant mean as unimportant by definition, isolating whether the deviation is important.