PaperlayerView exact revision on arXiv

Source-linked AI summary

Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms

Michael Hanna, Sandro Pezzelle, Yonatan Belinkov

arXiv:2403.17806v2cs.LGcs.CL

TL;DR

Circuit finding in language models needs scalable edge attribution, but overlap with known circuits does not establish that a circuit faithfully preserves model behavior. The paper introduces EAP-IG and finds it more faithful than EAP on many tasks, concluding that faithfulness—not overlap—should guide circuit comparisons.

  • Problem

    Edge-level causal interventions scale poorly, while high overlap with known circuits does not establish that removing non-circuit edges preserves task behavior.

  • Method

    The paper introduces EAP-IG, which combines gradient-based edge attribution with integrated gradients, and evaluates faithfulness alongside EAP and activation-patching circuits.

  • Results

    EAP-IG circuits are more faithful than EAP circuits on many tasks, although activation-patching circuits sometimes outperform both.

  • Takeaways & Limitations

    When comparing mechanisms across tasks or models, researchers should measure cross-task faithfulness separately from overlap.

  • Takeaways & Limitations

    Completeness remains difficult to test, and overlap is an imperfect completeness metric because manually found ground-truth circuits may be unreliable.

Abstract

from arXiv · show

Many recent language model (LM) interpretability studies have adopted the circuits framework, which aims to find the minimal computational subgraph, or circuit, that explains LM behavior on a given task. Most studies determine which edges belong in a LM's circuit by performing causal interventions on each edge independently, but this scales poorly with model size. Edge attribution patching (EAP), gradient-based approximation to interventions, has emerged as a scalable but imperfect solution to this problem. In this paper, we introduce a new method - EAP with integrated gradients (EAP-IG) - that aims to better maintain a core property of circuits: faithfulness. A circuit is faithful if all model edges outside the circuit can be ablated without changing the model's performance on the task; faithfulness is what justifies studying circuits, rather than the full model. Our experiments demonstrate that circuits found using EAP are less faithful than those found using EAP-IG, even though both have high node overlap with circuits found previously using causal interventions. We conclude more generally that when using circuits to compare the mechanisms models use to solve tasks, faithfulness, not overlap, is what should be measured.

1 Introduction

Circuit finding in language models is costly because edge-level causal interventions scale poorly, motivating EAP as a scalable approximation. This paper instead emphasizes faithfulness—the requirement that removing non-circuit edges preserves task behavior—and introduces EAP-IG to improve it.

  • 32,491 edges in GPT-2 small illustrate why testing every edge with causal interventions becomes increasingly expensive as models grow.
  • EAP approximates edge interventions using gradients in two forward passes and one backward pass, and its circuits show high overlap with manually found circuits.
  • Faithfulness requires ablating every edge outside a circuit without changing task behavior, because unfaithful simplified models can yield misleading mechanistic conclusions.
  • EAP-IG applies integrated gradients to circuit finding, and the paper reports that it finds more faithful circuits than EAP.
  • Across the study, EAP-IG is more faithful than EAP on many tasks, while activation-patching circuits sometimes outperform both and overlap does not always imply faithfulness.

2 The Circuits Framework and Circuit Finding

The circuits framework represents model behavior as a task-specific subgraph connecting inputs to logits, found by scoring components or edges and testing whether non-circuit edges can be corrupted without changing behavior. The section describes both causal intervention methods and scalable EAP, while noting that the chosen faithfulness test is not universal.

  • Circuits: A circuit is a subgraph of a transformer’s computational graph, whose nodes are attention heads or MLPs and whose edges connect node outputs toward logits.
  • Circuits: Faithfulness is tested by running clean inputs while replacing non-circuit edge activations with activations from corresponding corrupted inputs, then checking whether task performance remains similar.
  • Circuit Finding: The paper’s activation-patching faithfulness test is not universal, with mean ablations and zero ablations providing alternative intervention choices.
  • Circuit Finding: Activation patching adds an edge when replacing its clean activation with a corrupted one changes the task metric beyond threshold τ, but this intervention-based process scales poorly.
  • Circuit Finding: EAP scores edges using gradients to approximate loss changes, requiring one corrupted-input forward pass plus one clean-input forward and backward pass before selecting top-scoring edges.

3 EAP with Integrated Gradients

EAP-IG combines EAP with integrated gradients to address zero-gradient failures by evaluating gradients along a path between clean and baseline inputs. It retains a runtime similar to EAP in practice and supports losses such as KL divergence.

  • Integrated gradients addresses cases where a zero gradient at the clean input hides an activation’s influence despite a substantial activation difference.
  • EAP-IG combines EAP with integrated gradients, using gradients along a straight-line path between baseline and input embeddings to score circuit edges.
  • Unlike a contemporaneous node-output interpolation method whose cost scales with model depth, EAP-IG achieves similar performance with lower stated runtime scaling.
  • EAP-IG enables losses such as KL divergence, whereas EAP’s clean-input gradient for the unpatched KL divergence is zero.
  • EAP-IG uses m gradient steps and is theoretically m times slower than EAP, but the authors select m = 5 in practice.

4 Evaluating Edge Attribution Faithfulness

Across six GPT-2 small tasks, the study evaluates EAP, EAP-IG, and activation-patching circuits by faithfulness rather than overlap. EAP-IG generally matches or exceeds EAP, but performance varies by task and remains below activation patching in some cases.

  • Results: EAP-IG circuits’ faithfulness matches or surpasses EAP circuits across the evaluated tasks, although activation patching remains stronger on some tasks.The comparison uses circuits selected through greedy searches over edge scores and evaluates faithfulness at different circuit sizes.
  • Results: On IOI, activation-patching circuits exceed 0.8 faithfulness while EAP and EAP-IG circuits plateau near 0.6 under logit difference.This gap does not appear with KL divergence in the same way.
  • Results: On SVA, EAP produces completely unfaithful circuits through 1000 edges, partly because pruning removes parentless heads; EAP-IG retains 916 edges at that size.EAP-IG-KL retains 970 edges, compared with 807 for EAP.
  • Results: In Hypernymy, EAP-IG-KL reaches a fully faithful circuit by 2000 edges while EAP reaches only about halfway, but this may omit negatively contributing heads.The authors therefore identify remaining room for improvement beyond EAP-IG.
  • Why Does EAP-IG Work?: EAP-IG edge scores correlate more consistently with activation-patching scores, but their ranking overlap advantage appears mainly among the top fewer than 50 edges.The result suggests EAP-IG prioritizes performance-critical edges without being globally more accurate in edge rankings.

5 Overlap and Faithfulness

The paper compares circuit overlap with cross-task faithfulness using circuits reaching at least 85% within-task faithfulness. Overlap correlates strongly with faithfulness overall, but moderate overlap can correspond to sharply different cross-task behavior.

  • Method: The study computes node and edge Jaccard overlap and directional cross-task faithfulness using the smallest circuits achieving at least 85% within-task faithfulness.Additional Greater-Than variants and the reverse Country-Capital task provide task pairs expected to share mechanisms.
  • Circuit overlap: Node and edge overlap are high among Greater-Than variants and the Capital-Country/Country-Capital pair, while baseline edge overlap is often only 0.05–0.15 IoU.Hypernymy also has high node overlap with the country-retrieval tasks, but not similarly high edge overlap.
  • Cross-task faithfulness: Cross-task faithfulness is asymmetric: Hypernymy circuits transfer well to Capital-Country, whereas Capital-Country circuits transfer poorly to Hypernymy.The asymmetry may reflect the Hypernymy circuit’s larger size, which can support faithfulness across more tasks.
  • Overlap and faithfulness: Faithfulness correlates with node overlap at 0.86 and edge overlap at 0.83, with combined overlap explaining R2 = 0.74.Both correlations are highly significant at p < 0.001.
  • Overlap and faithfulness: Moderate overlap does not reliably predict cross-task faithfulness: the same 0.58 node overlap yields faithfulness of 0.61 in one direction and 0.2 in the reverse.Greater-Than variants provide a contrasting case with roughly 0.64 overlap and high faithfulness in both directions.

6 Discussion

The discussion argues that faithfulness should be evaluated separately from overlap when comparing circuits, while noting that completeness remains difficult to test.

  • EAP circuits have poor faithfulness compared with EAP-IG circuits, while zero or full overlap predicts low or high cross-task faithfulness but moderate overlap predicts little.
  • Completeness: Testing completeness is challenging because random-ablation evaluations are costly and manually found circuits may be unreliable ground truths.
  • Bridging the gap between overlap and faithfulness: Overlap and cross-task faithfulness can diverge because faithfulness depends on which specific nodes or edges overlap, not overlap in general.
  • Bridging the gap between overlap and faithfulness: Weighted graph edit distance could incorporate edge importance, but applying it would require published circuits to report edge scores.
  • Best practices for circuits: For comparisons across tasks and models, measuring cross-task faithfulness and characterizing component behavior can support sounder conclusions about model mechanisms.

A Dataset and Metric Details

The study uses task-specific metrics and evaluates whether EAP-IG retrieves manually found circuit structure and maintains faithfulness across step counts.

  • The study uses logit difference for single-answer tasks and probability difference for multi-answer tasks such as Greater-Than.
  • Probability difference sums probabilities assigned to correct and incorrect options, whereas logit difference compares logits for correct and corrupted answers.
  • All EAP-IG circuits with more than two integration steps have similarly high faithfulness across tasks, while two steps remain unfaithful for Greater-Than.
  • EAP-IG retrieves nodes and edges at least as well as EAP, dominating EAP on both node and edge precision and recall for Greater-Than.
  • For IOI, EAP and EAP-IG have similar edge precision and recall, while EAP-IG initially has better node precision and recall before slipping slightly at high recall.

C EAP-IG Requires Few Steps

EAP-IG generally achieves high faithfulness with few interpolation steps, while variants trade speed, implementation assumptions, and performance across tasks.

  • C EAP-IG Requires Few Steps: All EAP-IG circuits with more than 2 steps show similarly high faithfulness across tasks, motivating the use of 5 steps.At 2 steps, Greater-Than remains unfaithful and IOI retains EAP’s faithfulness spike.
  • C EAP-IG Requires Few Steps: The authors hypothesize that few steps suffice because clean and corrupted activations are closer than standard integrated-gradients baselines such as zero inputs.They also note that accurate approximation of activation scores may not be necessary for faithfulness.
  • D.1 EAP-IG Variants and Baselines: The tested EAP-IG variants differ in whether gradients are interpolated across inputs, node outputs, or current and corrupted activations.
  • D.1 EAP-IG Variants and Baselines: Feature-level activation-space EAP-IG requires separate passes across parallel sublayers, scaling worse for component and edge circuits than input-space EAP-IG.

D.2 Results

The authors use a greedy score-based procedure to construct connected circuits, selecting high-magnitude edges while preserving links to the logits.

  • D.2 Results: The greedy algorithm starts from logits and repeatedly adds the highest-magnitude scored edge whose child is already in the circuit.
  • D.2 Results: The method is designed to avoid the parent- or childless nodes produced by naive top-n edge selection.
  • D.2 Results: The implementation initializes the circuit with logits and updates its node and edge sets over n greedy selections.
  • D.2 Results: The procedure guarantees that selected circuits contain no childless nodes, while finding the exact maximum-flow subgraph is likely NP-hard.
  • D.2 Results: Edges are selected by absolute score, retaining components that may hurt task performance but remain important to the model’s task mechanism.

F Approximation Error and Kendall Correlations

EAP-IG does not consistently approximate activation-patching scores more accurately or rank edges more similarly than EAP, despite its faithfulness gains elsewhere.

  • F Approximation Error and Kendall Correlations: EAP-IG generally has slightly smaller score-approximation errors than EAP, but the difference is much smaller than the errors themselves.
  • F Approximation Error and Kendall Correlations: EAP-IG’s edge-ranking correlations with activation patching are significantly worse than EAP’s correlations.
  • F Approximation Error and Kendall Correlations: The figures compare EAP and EAP-IG approximation error and Kendall edge-ranking correlation against activation-patching ground truth.

G Within-Task Overlap and Faithfulness

Within-task overlap is an unreliable proxy for faithfulness: circuits can overlap strongly while differing substantially in whether they preserve task behavior.

  • G Within-Task Overlap and Faithfulness: An EAP IOI circuit containing at least 90% of the manually found circuit’s nodes achieves 0% faithfulness, while the analogous Greater-Than circuit reaches only 51%.
  • G Within-Task Overlap and Faithfulness: On SVA, EAP overlaps heavily with activation patching but performs much worse, and at 250 edges EAP-IG and activation patching remain faithful despite lower overlap than EAP.
  • G Within-Task Overlap and Faithfulness: The SVA discrepancy may reflect a crucial missing edge from the inputs to MLP 0 in EAP’s circuit.
  • G Within-Task Overlap and Faithfulness: The overlap experiments use selected EAP-IG-KL circuits for Greater-Than Price, Greater-Than Sequence, and Country-Capital after confirming their faithfulness.
  • J Asymmetric Overlap Measures: Node and edge recall sometimes reproduce faithfulness trends, including inter-Greater-Than similarity and Hypernymy’s broad circuit coverage, but also introduce many errors.

K Replications in Larger Models

Experiments in GPT-2 XL and Pythia-2.8B confirm that EAP-IG generally finds more faithful circuits than EAP, although some task and testing limitations remain.

  • K Replications in Larger Models: Activation patching circuits are omitted in larger models because of computational constraints.The larger-model experiments test GPT-2 XL and Pythia-2.8B using EAP, EAP-IG, and EAP-IG-KL.
  • K Replications in Larger Models: EAP-IG circuits’ faithfulness equals or surpasses EAP circuits’ in GPT-2 XL.
  • K Replications in Larger Models: Faithfulness computation becomes more expensive as the circuit’s number of nodes increases.The paper identifies more efficient faithfulness algorithms as future work.
  • K Replications in Larger Models: EAP-IG outperforms EAP in faithfulness whenever discernable differences appear in GPT-2 XL and Pythia-2.8B.These results confirm the earlier findings in larger models.
  • K Replications in Larger Models: EAP-IG circuits’ faithfulness equals or surpasses EAP circuits’ in Pythia-2.8B.
Loading 2403.17806v2…