Source-linked AI summary
Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, Jacob Steinhardt
TL;DR
Mechanistic interpretability seeks to explain model behavior through internal components, but prior work often addresses either simple behaviors in small models or complex behaviors only broadly. This paper reverse-engineers GPT-2 small’s indirect object identification using causal interventions and related analyses, identifying a 26-head, 7-class circuit whose validation supports the explanation while exposing remaining gaps.
Problem
Mechanistic interpretability lacks detailed explanations spanning a natural behavior in a language model while remaining grounded in internal components.
Method
The authors trace components from logits using path patching, supplement this with complementary analyses, and validate the resulting circuit using faithfulness, completeness, and minimality.
Results
The paper identifies a circuit of 26 attention heads grouped into 7 categories that completes the bulk of indirect object identification in GPT-2 small.
Takeaways & Limitations
The investigation provides a detailed example of reverse-engineering transformer behavior and identifies redundant, unexpected, and negatively contributing internal structures.
Takeaways & Limitations
The explanation’s validation is incomplete: the circuit fails the most challenging tests, while zero ablation is noisy and layer-normalization scaling complicates some measurements.
Abstract
from arXiv · showhide
Research in mechanistic interpretability seeks to explain behaviors of machine learning models in terms of their internal components. However, most previous work either focuses on simple behaviors in small models, or describes complicated behaviors in larger models with broad strokes. In this work, we bridge this gap by presenting an explanation for how GPT-2 small performs a natural language task called indirect object identification (IOI). Our explanation encompasses 26 attention heads grouped into 7 main classes, which we discovered using a combination of interpretability approaches relying on causal interventions. To our knowledge, this investigation is the largest end-to-end attempt at reverse-engineering a natural behavior "in the wild" in a language model. We evaluate the reliability of our explanation using three quantitative criteria--faithfulness, completeness and minimality. Though these criteria support our explanation, they also point to remaining gaps in our understanding. Our work provides evidence that a mechanistic understanding of large ML models is feasible, opening opportunities to scale our understanding to both larger models and more complex tasks.
1 INTRODUCTION
This paper mechanistically explains how GPT-2 small performs indirect object identification by isolating and validating a human-understandable circuit. The investigation combines causal interventions with complementary interpretability analyses and examines both the circuit’s structure and remaining validation gaps.
- The work aims to make GPT-2 small’s computation human-understandable because transformer models remain difficult to understand despite deployment in high-impact settings.Mechanistic interpretability is presented as a way to discover, understand, and verify algorithms implemented by model weights.
- The circuit is discovered by tracing important components back from the logits with path patching, supplemented by embedding projections, attention-pattern analysis, and activation patching.
- GPT-2 small performs indirect object identification by predicting the name that is not the subject of the last clause.For example, in “When Mary and John went to the store, John gave a drink to”, the expected completion is “Mary”.
- The authors identify a circuit of 26 attention heads, grouped into 7 categories, that completes the bulk of IOI.The heads route information between name tokens, the end position, and the output.
- The analysis reveals redundant heads, unexpected uses of induction-head structures, and heads that reliably write opposite to the correct answer.Backup Name-Mover Heads copy names correctly only when regular Name-Mover Heads are ablated.
- The study validates the circuit with faithfulness, completeness, and minimality criteria, but it fails the most challenging tests.The circuit improves significantly over a naïve faithful circuit while leaving gaps in the explanation.
2 BACKGROUND
The paper defines IOI, describes the relevant GPT-2 small architecture and circuit abstraction, and specifies knockout methods for isolating circuit behavior. It also replaces noisy zero ablation with mean ablation using a modified reference distribution.
- Task description: IOI requires predicting the indirect object at the final token of templates containing an initial clause and a main clause.The dataset uses 15 templates with random single-token names, places, and items.
- Transformer architecture: GPT-2 small is a decoder-only transformer with 12 layers and 12 attention heads per layer.
- Circuits and knockouts: A circuit is a subgraph of the model’s computational graph responsible for a behavior, with nodes as model components and edges as their interactions.
- Circuits and knockouts: The circuit function is measured by knocking out all model nodes outside the circuit and taking the resulting logits.
- Circuits and knockouts: Zero ablation is noisy because replacing removed nodes with 0 is arbitrary and later nodes may rely on average activation as an implicit bias.
- Circuits and knockouts: Mean ablation replaces nodes with their average activation over a reference distribution, removing varying task information while preserving constant information.All knockouts use pABC, which contains three unrelated names while preserving the grammatical structure of IOI templates.
- Circuits and knockouts: Template-matched averaging preserves grammatical information when a node is knocked out because token positions have different grammatical roles across templates.
3 DISCOVERING THE CIRCUIT
The authors trace GPT-2 small’s IOI circuit backward from the logits, identifying head classes that detect duplicated subjects, inhibit subject attention, and move the indirect object to the output. Causal interventions and activation analyses support these roles while revealing redundancy and remaining validation gaps.
- Related and redundant components: Backup Name Mover Heads provide redundant output behavior, activating when regular Name Mover Heads are ablated, while Negative Name Mover Heads write opposite name directions.These redundancies complicate efforts to identify complete mechanisms because ablations can reveal different model structure.
- Scope and caveats: The investigation focuses on attention heads and does not intervene on MLPs, layer norms, or embedding matrices, leaving the role of MLPs for future work.Individually knocking out most MLPs preserved task performance, but knocking out all MLPs after the first layer prevented the task.
- Which heads directly affect the output?: Path patching identifies heads that directly influence the IOI logit difference, including positive Name Mover Heads and negatively influencing Negative Name Mover Heads.Heads 9.6, 9.9, and 10.0 contribute positively, whereas 10.7 and 11.10 contribute negatively.
- Which heads directly affect the output?: Name Mover Heads attend strongly to the indirect-object token and copy attended names into the output representation.Their average attention to the IO token is 0.59, and attention probability correlates with output in the corresponding name direction at ρ > 0.81 over N = 500.
- Which heads affect the Name Mover Heads’ attention?: S-Inhibition Heads modify Name Mover queries so those heads attend less to subject occurrences and preferentially select the indirect object.The four S-Inhibition Heads attend from END to S2 and write token and position signals that suppress subject attention.
- Scope and caveats: Additional parameter-level checks were still needed to validate whether Duplicate Token and Induction Heads implement their proposed collision-detection, copying, and key-composition mechanisms.The omitted checks concern interactions between QK matrices and token embeddings, OV matrices and positional embeddings, and the parameter-level analysis of Induction Heads.
4 EXPERIMENTAL VALIDATION
The authors evaluate the IOI circuit with faithfulness, completeness, and minimality criteria, compare it with a naive circuit, and use circuit knowledge to construct adversarial examples. The circuit performs well on faithfulness and minimality, but completeness testing reveals gaps and the adversarial analysis has important limitations.
- Faithfulness: The circuit’s faithfulness gap is 0.46, only 13% of the full model’s logit difference, so it achieves 87% of model performance.
- Completeness: Completeness requires circuit and model performance to remain similar under knockouts of arbitrary subsets of components.
- Completeness: 3.09, or 87% of the original logit difference, was the largest incompleteness score found by greedy subset optimization.Random sampling and class-based knockouts suggested completeness, but greedy optimization found high-scoring, usually uninterpretable subsets.
- Minimality: Every circuit head had a nontrivial impact of at least 1% of the original logit difference, although some heads contributed only modestly.
- Baseline comparison: The naive circuit matched the full circuit’s faithfulness score of 0.1 but was more easily shown to be incomplete.
- Adversarial examples: Duplicating both subject and indirect-object tokens reduced the logit difference and made the model predict the subject 23% of the time.The examples were inspired by the circuit’s duplicate-detection account, but they were simple enough to find without that understanding.
5 DISCUSSION
The discussion presents the validated circuit as a detailed account of GPT-2 small’s IOI behavior while highlighting pointer communication, compensatory and negatively contributing heads, unresolved mechanisms, and the challenge of scaling beyond GPT-2 small.
- The study isolates, understands, and validates an attention-head circuit for indirect-object identification in GPT-2 small.
- Some attention heads communicate by sharing information locations through pointers rather than copying the information itself.
- Other heads compensate for lost functions, while some contribute negatively to next-token prediction; early results suggest negative contributions beyond IOI.
- The analysis leaves several components unresolved, and GPT-2 small is orders of magnitude smaller than state-of-the-art transformer language models.
- GPT-2 medium also has a sparse set of heads directly influencing logits, but not all attend to IO and S as in GPT-2 small.
- The authors view narrowly defined, highly detailed analyses as useful starting points for understanding broader model behavior.
A DISENTANGLING TOKEN AND POSITIONAL SIGNAL IN THE OUTPUT OF S-INHIBITION HEADS
The authors disentangle token and positional signals emitted by S-Inhibition Heads using counterfactual datasets and activation patching. The signals independently affect the IOI logit difference, with positional information contributing more strongly and appearing to depend on relative rather than absolute position.
- S-Inhibition Heads cause Name Mover Heads to avoid the subject by transmitting token and positional signals.
- Counterfactual datasets vary or invert token and positional signals, allowing each signal’s isolated effect on the final logit difference to be measured by patching.
- The logit difference is approximated by 2.31Spos + 0.99Stok, with mean error 7% relative to the baseline logit difference.
- When both signals are inverted, the model favors the subject as strongly as it originally favored the indirect object, and Name Movers copy S1 instead.
- Path patching isolates a sender head’s influence on receiver nodes by freezing other heads, recomputing intervening components, and measuring the resulting logit difference.
- The positional signal appears to depend on the relative position of S1 and S2, since changing absolute positions leaves the logit difference nearly unchanged at 3.56 versus 3.57.
C DIRECT EFFECT ON S-INHIBITION HEADS’ KEYS
Path-patching experiments identified heads that influence S-Inhibition Heads through their keys, including three newly recognized positive contributors. Attention patterns characterize these contributors as fuzzy Duplicate Token and fuzzy Induction Heads.
- Duplicate Token Heads 3.0 and 0.1 also influence S-Inhibition Heads’ values, but their key-mediated effect is reversed.
- Three heads—5.9, 5.8, and 0.10—positively influence logit difference through S-Inhibition Heads’ keys.
- Head 0.10 attends fuzzily from S2 to S1 and, on Open Web Text, attends to short-range duplicates.
- Heads 5.9 and 5.8 attend to S1+1 with less interpretable patterns, so they are classified as fuzzy Induction Heads.For 5.8, S1+1 receives the highest attention after the start token, but its absolute attention is below 0.1.
- The authors hypothesize that these heads amplify positional signals written by other Induction and Duplicate Token Heads.
D IDENTIFICATION OF PREVIOUS TOKEN HEADS
The section uses path patching to trace heads affecting Induction Heads at S1+1, the repeated-pattern position associated with the [B] token. It also reports that query-side interventions at S2 found no significant effects.
- Induction Heads recognize [A] [B] ... [A] patterns through key composition with Previous Token Heads, with S2 as the repeated token in IOI.
- Path patching at S1+1 identifies two main heads that decrease logit difference and therefore contribute positively to it.
- Path patching found no significant heads influencing Induction Heads’ queries at S2.This contrasts with an induction mechanism described as relying on query composition.
- The IOI evaluation uses one-token names, places, and objects drawn from controlled lists to preserve sequence alignment.Names come from 100 English first names, while places and objects come from handmade lists of 20 common words.
- The dataset includes templates following both BABA and ABBA name-order patterns.
F BACKUP NAME MOVER HEADS
Knocking out the three main Name Mover Heads reveals backup heads that compensate for their lost function. These heads exhibit diverse behaviors, but their exact selection has little effect on the circuit’s behavior.
- After the three main Name Mover Heads are knocked out, the logit difference is only 5% lower because other heads compensate for the lost function.Negative Name Mover Heads become less negative, and head 10.7 even has a positive effect after knockout.
- Eight positively influencing heads not assigned to other groups were selected as Backup Name Mover Heads using a 2% logit-difference effect threshold.
- The backup heads divide into four behavioral categories based on attention patterns and output directions.
- Four heads behave similarly to Name Mover Heads, while two attend equally to S1 and IO and write both tokens.
- Head 11.2 attends more to S1 and writes preferentially toward WU[S], whereas head 9.7 attends to S2 and writes negatively.
- The backup heads were not thoroughly investigated, but they have low individual importance and their exact selection does not significantly change circuit behavior.
G GPT-2 SMALL FULL ARCHITECTURE
GPT-2 small combines token embeddings, positional information, attention heads, layer normalization, and one-hidden-layer MLPs in a unidirectional transformer architecture. The forward pass combines these elements across the model’s layers.
- GPT-2 small is parameterized by input length N, vocabulary length V, residual-stream dimension d, layer count L, head count H, and MLP hidden dimension D.
- Layer normalization computes the mean and deviation from the mean across the d components at each sequence position, followed by a learned linear transformation M.
- Each MLP has one hidden layer of dimension D, uses the GeLU nonlinearity, and receives the layer-normalized residual stream.
- Attention patterns are computed with WQK-based scores, softmax-normalized separately for each token position, and applied unidirectionally.
- Algorithm 2 specifies how the architecture’s components are combined during GPT-2 small’s forward pass.
H VALIDATION OF THE INDUCTION MECHANISM ON SEQUENCES OF RANDOM TOKENS
Validation on repeated random-token sequences supports the roles of Previous Token and Induction Heads, including prefix matching and copying. Other circuit heads also contribute to next-token prediction, suggesting the mechanism extends beyond IOI.
- The evaluation uses duplicated 100-token random sequences and attention-based scores for previous-token and induction behavior.Each sequence is duplicated to form AA, enabling attention-pattern tests on repeated contexts.
- 4.11 and 2.2 have the highest previous token scores, validating their role outside pIOI.
- 5.5 and 6.9 rank among the five heads with the highest induction scores, validating their prefix-matching property.
- 5.5 and 6.9 are among the 20 heads contributing most to next-token prediction, validating their copying property.
- Most Negative, Backup, and regular Name Mover Heads write in the next-token direction on repeated random-token sequences.Negative Name Mover Heads contribute negatively, suggesting these heads support contextual next-token prediction beyond IOI.
I VALIDATION OF DUPLICATE TOKEN HEADS
Repeated random-token tests provide evidence that heads 3.0 and 0.1 detect duplicate tokens outside the IOI circuit. Fuzzy Duplicate Head 0.10 requires qualitative analysis because it does not appear in the repeated-token test.
- Duplicate-token scores measure attention from each second-half token to its corresponding occurrence in the first half.
- 3.0 and 0.1 rank among the three heads with the highest duplicate-token scores, supporting their role as Duplicate Token Heads.
- Fuzzy Duplicate Head 0.10 does not appear in the repeated random-token test.
- On Open Web Text, 0.10 attends strongly to the current token and to its previous occurrence when the duplicated current token is a name.
J ROLE OF MLPS IN THE TASK
MLP0 strongly affects IOI performance, while individually ablating later MLPs has little effect; ablating them collectively nevertheless disrupts the task. The circuit analysis therefore leaves MLP contributions incompletely characterized.
- The investigation focuses on attention heads and does not intervene on MLPs, layer norms, or embedding matrices.
- MLP0 reverses the sign of the logit difference when knocked out, whereas other MLP layers show little individual effect.
- Knocking out all MLP layers after the first makes the logit difference −1.1, similar to knocking out MLP0 alone.
- Adversarial examples add a context-independent duplicate occurrence of [A] to natural-sentence templates.
- The greedy-search sets combine nodes from different classes, making the circuit's incompleteness difficult to interpret.