Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

• The work aims to make GPT-2 small’s computation human-understandable because transformer models remain difficult to understand despite deployment in high-impact settings.Mechanistic interpretability is presented as a way to discover, understand, and verify algorithms implemented by model weights.Source: §1 · p. 1
• The circuit is discovered by tracing important components back from the logits with path patching, supplemented by embedding projections, attention-pattern analysis, and activation patching.Source: §1 · p. 1
• GPT-2 small performs indirect object identification by predicting the name that is not the subject of the last clause.For example, in “When Mary and John went to the store, John gave a drink to”, the expected completion is “Mary”.Source: §1 · p. 2
• The authors identify a circuit of 26 attention heads, grouped into 7 categories, that completes the bulk of IOI.The heads route information between name tokens, the end position, and the output.Source: §1 · p. 2
• The analysis reveals redundant heads, unexpected uses of induction-head structures, and heads that reliably write opposite to the correct answer.Backup Name-Mover Heads copy names correctly only when regular Name-Mover Heads are ablated.Source: §1 · p. 2
• The study validates the circuit with faithfulness, completeness, and minimality criteria, but it fails the most challenging tests.The circuit improves significantly over a naïve faithful circuit while leaving gaps in the explanation.Source: §1 · p. 2

Task description

• IOI requires predicting the indirect object at the final token of templates containing an initial clause and a main clause.The dataset uses 15 templates with random single-token names, places, and items.Source: §2 · p. 3

Transformer architecture

• GPT-2 small is a decoder-only transformer with 12 layers and 12 attention heads per layer.Source: §2 · p. 3

Circuits and knockouts

• A circuit is a subgraph of the model’s computational graph responsible for a behavior, with nodes as model components and edges as their interactions.Source: §2.1 · p. 3
• The circuit function is measured by knocking out all model nodes outside the circuit and taking the resulting logits.Source: §2.1 · p. 3
• Zero ablation is noisy because replacing removed nodes with 0 is arbitrary and later nodes may rely on average activation as an implicit bias.Source: §2.1 · p. 3
• Mean ablation replaces nodes with their average activation over a reference distribution, removing varying task information while preserving constant information.All knockouts use pABC, which contains three unrelated names while preserving the grammatical structure of IOI templates.Source: §2.1 · pp. 3–4
• Template-matched averaging preserves grammatical information when a node is knocked out because token positions have different grammatical roles across templates.Source: §2.1 · p. 4

Related and redundant components

• Backup Name Mover Heads provide redundant output behavior, activating when regular Name Mover Heads are ablated, while Negative Name Mover Heads write opposite name directions.These redundancies complicate efforts to identify complete mechanisms because ablations can reveal different model structure.Source: §3 · p. 5, §3.1 · p. 6

Scope and caveats

• The investigation focuses on attention heads and does not intervene on MLPs, layer norms, or embedding matrices, leaving the role of MLPs for future work.Individually knocking out most MLPs preserved task performance, but knocking out all MLPs after the first layer prevented the task.Source: §3 · p. 5

Which heads directly affect the output?

• Path patching identifies heads that directly influence the IOI logit difference, including positive Name Mover Heads and negatively influencing Negative Name Mover Heads.Heads 9.6, 9.9, and 10.0 contribute positively, whereas 10.7 and 11.10 contribute negatively.Source: §3.1 · pp. 5–6
• Name Mover Heads attend strongly to the indirect-object token and copy attended names into the output representation.Their average attention to the IO token is 0.59, and attention probability correlates with output in the corresponding name direction at ρ > 0.81 over N = 500.Source: §3.1 · pp. 5–6

Which heads affect the Name Mover Heads’ attention?

• S-Inhibition Heads modify Name Mover queries so those heads attend less to subject occurrences and preferentially select the indirect object.The four S-Inhibition Heads attend from END to S2 and write token and position signals that suppress subject attention.Source: §3.2 · pp. 7–8

Scope and caveats

• Additional parameter-level checks were still needed to validate whether Duplicate Token and Induction Heads implement their proposed collision-detection, copying, and key-composition mechanisms.The omitted checks concern interactions between QK matrices and token embeddings, OV matrices and positional embeddings, and the parameter-level analysis of Induction Heads.Source: §3.3 · p. 8

Faithfulness

• The circuit’s faithfulness gap is 0.46, only 13% of the full model’s logit difference, so it achieves 87% of model performance.Source: §4 · p. 9

Completeness

• Completeness requires circuit and model performance to remain similar under knockouts of arbitrary subsets of components.Source: §4.1 · pp. 9–10
• 3.09, or 87% of the original logit difference, was the largest incompleteness score found by greedy subset optimization.Random sampling and class-based knockouts suggested completeness, but greedy optimization found high-scoring, usually uninterpretable subsets.Source: §4.1 · p. 10

Minimality

• Every circuit head had a nontrivial impact of at least 1% of the original logit difference, although some heads contributed only modestly.Source: §4.2 · pp. 10–11

Baseline comparison

• The naive circuit matched the full circuit’s faithfulness score of 0.1 but was more easily shown to be incomplete.Source: §4 · p. 9, §4.3 · p. 11

Adversarial examples

• Duplicating both subject and indirect-object tokens reduced the logit difference and made the model predict the subject 23% of the time.The examples were inspired by the circuit’s duplicate-detection account, but they were simple enough to find without that understanding.Source: §4.4 · p. 11

• The study isolates, understands, and validates an attention-head circuit for indirect-object identification in GPT-2 small.Source: §5 · p. 12
• Some attention heads communicate by sharing information locations through pointers rather than copying the information itself.Source: §5 · p. 12
• Other heads compensate for lost functions, while some contribute negatively to next-token prediction; early results suggest negative contributions beyond IOI.Source: §5 · p. 12
• The analysis leaves several components unresolved, and GPT-2 small is orders of magnitude smaller than state-of-the-art transformer language models.Source: §5 · p. 12
• GPT-2 medium also has a sparse set of heads directly influencing logits, but not all attend to IO and S as in GPT-2 small.Source: §5 · p. 12
• The authors view narrowly defined, highly detailed analyses as useful starting points for understanding broader model behavior.Source: §5 · p. 12

• S-Inhibition Heads cause Name Mover Heads to avoid the subject by transmitting token and positional signals.Source: A DISENTANGLING TOKEN AND POSITIONAL… · p. 14
• Counterfactual datasets vary or invert token and positional signals, allowing each signal’s isolated effect on the final logit difference to be measured by patching.Source: A DISENTANGLING TOKEN AND POSITIONAL… · pp. 14–15
• The logit difference is approximated by 2.31Spos + 0.99Stok, with mean error 7% relative to the baseline logit difference.Source: A DISENTANGLING TOKEN AND POSITIONAL… · p. 15
• When both signals are inverted, the model favors the subject as strongly as it originally favored the indirect object, and Name Movers copy S1 instead.Source: A DISENTANGLING TOKEN AND POSITIONAL… · p. 15, Figure 10 · p. 15
• Path patching isolates a sender head’s influence on receiver nodes by freezing other heads, recomputing intervening components, and measuring the resulting logit difference.Source: Figure 11 · p. 16, B PATH PATCHING · p. 16, §1 · p. 16
• The positional signal appears to depend on the relative position of S1 and S2, since changing absolute positions leaves the logit difference nearly unchanged at 3.56 versus 3.57.Source: A DISENTANGLING TOKEN AND POSITIONAL… · p. 16

• Duplicate Token Heads 3.0 and 0.1 also influence S-Inhibition Heads’ values, but their key-mediated effect is reversed.Source: C DIRECT EFFECT ON S-INHIBITION HEADS’… · p. 17
• Three heads—5.9, 5.8, and 0.10—positively influence logit difference through S-Inhibition Heads’ keys.Source: C DIRECT EFFECT ON S-INHIBITION HEADS’… · p. 17
• Head 0.10 attends fuzzily from S2 to S1 and, on Open Web Text, attends to short-range duplicates.Source: C DIRECT EFFECT ON S-INHIBITION HEADS’… · p. 17
• Heads 5.9 and 5.8 attend to S1+1 with less interpretable patterns, so they are classified as fuzzy Induction Heads.For 5.8, S1+1 receives the highest attention after the start token, but its absolute attention is below 0.1.Source: C DIRECT EFFECT ON S-INHIBITION HEADS’… · p. 17
• The authors hypothesize that these heads amplify positional signals written by other Induction and Duplicate Token Heads.Source: C DIRECT EFFECT ON S-INHIBITION HEADS’… · p. 17

• Induction Heads recognize [A] [B] ... [A] patterns through key composition with Previous Token Heads, with S2 as the repeated token in IOI.Source: D IDENTIFICATION OF PREVIOUS TOKEN HEADS · p. 18
• Path patching at S1+1 identifies two main heads that decrease logit difference and therefore contribute positively to it.Source: D IDENTIFICATION OF PREVIOUS TOKEN HEADS · p. 18
• Path patching found no significant heads influencing Induction Heads’ queries at S2.This contrasts with an induction mechanism described as relying on query composition.Source: D IDENTIFICATION OF PREVIOUS TOKEN HEADS · p. 18
• The IOI evaluation uses one-token names, places, and objects drawn from controlled lists to preserve sequence alignment.Names come from 100 English first names, while places and objects come from handmade lists of 20 common words.Source: E IOI TEMPLATES · p. 18
• The dataset includes templates following both BABA and ABBA name-order patterns.Source: Figure 14 · p. 19, E IOI TEMPLATES · p. 19

• After the three main Name Mover Heads are knocked out, the logit difference is only 5% lower because other heads compensate for the lost function.Negative Name Mover Heads become less negative, and head 10.7 even has a positive effect after knockout.Source: F BACKUP NAME MOVER HEADS · p. 19
• Eight positively influencing heads not assigned to other groups were selected as Backup Name Mover Heads using a 2% logit-difference effect threshold.Source: F BACKUP NAME MOVER HEADS · p. 19
• The backup heads divide into four behavioral categories based on attention patterns and output directions.Source: F BACKUP NAME MOVER HEADS · p. 19
• Four heads behave similarly to Name Mover Heads, while two attend equally to S1 and IO and write both tokens.Source: F BACKUP NAME MOVER HEADS · p. 19
• Head 11.2 attends more to S1 and writes preferentially toward WU[S], whereas head 9.7 attends to S2 and writes negatively.Source: F BACKUP NAME MOVER HEADS · p. 19
• The backup heads were not thoroughly investigated, but they have low individual importance and their exact selection does not significantly change circuit behavior.Source: F BACKUP NAME MOVER HEADS · p. 19

• GPT-2 small is parameterized by input length N, vocabulary length V, residual-stream dimension d, layer count L, head count H, and MLP hidden dimension D.Source: G GPT-2 SMALL FULL ARCHITECTURE · p. 20
• Layer normalization computes the mean and deviation from the mean across the d components at each sequence position, followed by a learned linear transformation M.Source: G GPT-2 SMALL FULL ARCHITECTURE · p. 20
• Each MLP has one hidden layer of dimension D, uses the GeLU nonlinearity, and receives the layer-normalized residual stream.Source: G GPT-2 SMALL FULL ARCHITECTURE · p. 20
• Attention patterns are computed with WQK-based scores, softmax-normalized separately for each token position, and applied unidirectionally.Source: G GPT-2 SMALL FULL ARCHITECTURE · p. 20
• Algorithm 2 specifies how the architecture’s components are combined during GPT-2 small’s forward pass.Source: G GPT-2 SMALL FULL ARCHITECTURE · p. 20

• The evaluation uses duplicated 100-token random sequences and attention-based scores for previous-token and induction behavior.Each sequence is duplicated to form AA, enabling attention-pattern tests on repeated contexts.Source: H VALIDATION OF THE INDUCTION… · pp. 20, 22
• 4.11 and 2.2 have the highest previous token scores, validating their role outside pIOI.Source: H VALIDATION OF THE INDUCTION… · p. 22
• 5.5 and 6.9 rank among the five heads with the highest induction scores, validating their prefix-matching property.Source: H VALIDATION OF THE INDUCTION… · p. 22
• 5.5 and 6.9 are among the 20 heads contributing most to next-token prediction, validating their copying property.Source: H VALIDATION OF THE INDUCTION… · p. 22, Figure 17 · p. 22
• Most Negative, Backup, and regular Name Mover Heads write in the next-token direction on repeated random-token sequences.Negative Name Mover Heads contribute negatively, suggesting these heads support contextual next-token prediction beyond IOI.Source: H VALIDATION OF THE INDUCTION… · p. 22

• Duplicate-token scores measure attention from each second-half token to its corresponding occurrence in the first half.Source: I VALIDATION OF DUPLICATE TOKEN HEADS · p. 22
• 3.0 and 0.1 rank among the three heads with the highest duplicate-token scores, supporting their role as Duplicate Token Heads.Source: I VALIDATION OF DUPLICATE TOKEN HEADS · p. 22, Figure 18 · p. 23
• Fuzzy Duplicate Head 0.10 does not appear in the repeated random-token test.Source: I VALIDATION OF DUPLICATE TOKEN HEADS · p. 23
• On Open Web Text, 0.10 attends strongly to the current token and to its previous occurrence when the duplicated current token is a name.Source: I VALIDATION OF DUPLICATE TOKEN HEADS · p. 23

• The investigation focuses on attention heads and does not intervene on MLPs, layer norms, or embedding matrices.Source: J ROLE OF MLPS IN THE TASK · p. 23
• MLP0 reverses the sign of the logit difference when knocked out, whereas other MLP layers show little individual effect.Source: J ROLE OF MLPS IN THE TASK · p. 23, Figure 19 · p. 23
• Knocking out all MLP layers after the first makes the logit difference −1.1, similar to knocking out MLP0 alone.Source: J ROLE OF MLPS IN THE TASK · p. 23
• Adversarial examples add a context-independent duplicate occurrence of [A] to natural-sentence templates.Source: L TEMPLATE FOR ADVERSARIAL EXAMPLES · p. 25, Figure 21 · p. 25
• The greedy-search sets combine nodes from different classes, making the circuit's incompleteness difficult to interpret.Source: M GREEDY ALGORITHM · p. 25, Figure 22 · p. 25