PaperlayerView exact revision on arXiv

Source-linked AI summary

Circuit Component Reuse Across Tasks in Transformer Language Models

Jack Merullo, Carsten Eickhoff, Ellie Pavlick

arXiv:2310.08744v3cs.CLcs.LG

TL;DR

The paper asks whether circuit analysis can explain behavior beyond the task used to discover a circuit. It compares IOI and Colored Objects circuits and intervenes on the latter. The circuits share many in-circuit attention heads, and the intervention improves Colored Objects performance.

  • Problem

    It is unclear whether circuit analyses that explain a specific task can explain model behavior beyond that task or whether different tasks use task-specific circuits.

    Source: §1 · p. 1

  • Method

    The authors use path patching and related circuit analyses to compare IOI and Colored Objects on GPT2-Medium, then intervene on the Colored Objects circuit to make it behave more like IOI.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    Approximately 78% of the most-important in-circuit attention heads are shared between IOI and Colored Objects.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings provide evidence that a subcircuit for inhibition and copying can be reused across task domains.

    Source: §1 · p. 2, §4.5 · p. 7

    The authors lack an exact solution for quantifying circuit overlap.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

Recent work in mechanistic interpretability has shown that behaviors in language models can be successfully reverse-engineered through circuit analysis. A common criticism, however, is that each circuit is task-specific, and thus such analysis cannot contribute to understanding the models at a higher level. In this work, we present evidence that insights (both low-level findings about specific heads and higher-level findings about general algorithms) can indeed generalize across tasks. Specifically, we study the circuit discovered in Wang et al. (2022) for the Indirect Object Identification (IOI) task and 1.) show that it reproduces on a larger GPT2 model, and 2.) that it is mostly reused to solve a seemingly different task: Colored Objects (Ippolito & Callison-Burch, 2023). We provide evidence that the process underlying both tasks is functionally very similar, and contains about a 78% overlap in in-circuit attention heads. We further present a proof-of-concept intervention experiment, in which we adjust four attention heads in middle layers in order to 'repair' the Colored Objects circuit and make it behave like the IOI circuit. In doing so, we boost accuracy from 49.6% to 93.7% on the Colored Objects task and explain most sources of error. The intervention affects downstream attention heads in specific ways predicted by their interactions in the IOI circuit, indicating that this subcircuit behavior is invariant to the different task inputs. Overall, our results provide evidence that it may yet be possible to explain large language models' behavior in terms of a relatively small number of interpretable task-general algorithmic building blocks and computational components.

1 INTRODUCTION

This paper asks whether circuit analysis can explain model behavior beyond the specific task used to discover a circuit. Comparing IOI with Colored Objects, the authors find substantial component reuse and use it to predict an intervention that improves performance.

1 INTRODUCTION

  • The study addresses whether task-specific circuit analyses can explain model behavior beyond the task they were built to explain.The authors note that larger structures recovered through circuit analysis had not yet been shown to be repurposed across tasks.Source: §1 · p. 1
  • The authors compare the IOI circuit with Colored Objects, a seemingly different task that also requires selecting a token from contextual options.They use path patching on GPT2-Medium to compare the circuits.Source: §1 · p. 2
  • Approximately 78% of the most-important in-circuit attention heads are shared between IOI and Colored Objects.The authors also report increasing Colored Objects accuracy from 49.6% to 93.7% through interventions on inactive circuit parts.Source: §1 · p. 2
  • Interventions make the Colored Objects circuit behave more like the IOI circuit and produce downstream effects predicted by IOI circuit interactions.The authors interpret this as evidence that the inhibition-mover subcircuit is robust across changes in task inputs.Source: §1 · p. 2

2 EXPERIMENTAL SETUP

The study analyzes IOI and Colored Objects on GPT2-Medium, then uses path patching to identify attention heads with direct effects on predictions and trace the heads influencing them.

2 EXPERIMENTAL SETUP

  • The authors use GPT2-Medium to analyze both tasks, applying attention-pattern analysis, logit attribution, and path patching to explain components involved in predicting the answer.They describe the task examples and methods in the experimental setup.Source: §2 · p. 2

2.1 TASKS

  • IOI asks the model to predict an indirect-object name, while Colored Objects asks it to retrieve an object's color from context among alternatives.The modified Colored Objects task uses one in-context example and a test example; GPT2-Medium predicts a test color 100% of the time but achieves 49.6% accuracy.Source: §2.1 · pp. 2–3, Figure 1 · p. 3
  • The authors state that they do not have an exact solution for assigning a numerical similarity to two circuits.They discuss this difficulty when describing the tasks.Source: §2.1 · p. 2

2.2 PATH PATCHING

  • Path patching replaces activations from one input with those from another to identify attention heads with direct effects on prediction logits and work backward through influencing heads.The analysis patches paths between attention heads while allowing MLPs to be recomputed.Source: §2.2 · p. 3

3 THE INDIRECT OBJECT IDENTIFICATION (IOI) CIRCUIT

The IOI circuit is reproduced on GPT2-Medium and described as a sequence that detects duplicate names, inhibits attention to them, and copies the remaining indirect-object name. The reproduced circuit also includes a negative mover head that demotes the subject token.

3 THE INDIRECT OBJECT IDENTIFICATION (IOI) CIRCUIT

  • Duplicate-token and induction heads detect repeated subject names, inhibition heads steer mover heads away from them, and mover heads copy the remaining indirect-object name.This reproduces the IOI circuit on GPT2-Medium, with all components described in Wang et al. (2022) active in the larger model.Source: §3 · pp. 3–4
  • The authors argue that part of the IOI circuit implements a more general algorithm for copying from a list of potential options.The task is therefore not limited, in their account, to indirect object identification.Source: §3 · p. 4
  • The negative mover head 19.1 attends only to S2 and demotes its likelihood as the next prediction in GPT2-Medium.It is the most important head contributing to the logit difference in path patching; Wang et al. (2022) found that negative mover heads in GPT2-Small attend to all names.Source: §3 · p. 4

4 THE COLORED OBJECTS CIRCUIT

The Colored Objects and IOI circuits use similar duplication-guided copying, sharing 25/32 (78%) of their top-2%-important heads. Their key difference is whether duplication guides copying through positive content-gatherer signals or negative inhibition signals.

4 THE COLORED OBJECTS CIRCUIT

  • 25/32 (78%) of the top-2%-important circuit heads are shared between Colored Objects and IOI.The circuits differ by three inhibition heads and a negative mover head in IOI versus three content gatherer heads in Colored Objects.Source: §4 · p. 4

4.1 MOVER HEADS

  • Mover heads including 15.14 and 16.15 perform the final copy step in both circuits, though their relative importance differs by task.Head 18.5 is more important in IOI, while 15.14 is most important in Colored Objects.Source: §4.1 · p. 5

4.2 CONTENT GATHERER HEADS

  • Blocking attention at the [end] position to hypothesized question words reduced accuracy to 35%, versus 49.9%±1.0 average for three-seed random-head interventions using heads not predicted as content-gathering.Heads 11.6, 11.7, and 12.15 perform content gathering, influencing mover query vectors and attending primarily to obj2 and the token ‘color’ in the question.Source: §4.2 · p. 5

4.3 DUPLICATE/INDUCTION HEADS

  • Path patching shows duplicate-token and induction heads influence content-gatherer value vectors at obj2; induction head 9.3 and duplicate-token head 6.4 attend from obj2 to obj1 or obj1+1, helping the model find the final color token.Figure 2 reports that content-gatherer heads pay most attention to obj2 and the word color.Source: §4.3 · p. 5, Figure 2 · p. 5

4 THE COLORED OBJECTS CIRCUIT

  • Both tasks use duplication detection to guide mover heads toward the token to copy, despite their different task structures.In Colored Objects, content gatherers provide a positive signal about which token to copy; in IOI, inhibition heads provide a negative signal about which tokens to ignore.Source: §4 · p. 4, Figure 3 · p. 6

4.4 IOI COMPONENTS MISSING IN COLORED OBJECTS

  • Colored Objects’ inhibition heads attend to color tokens and the in-context label but spread a noisy signal rather than directing movers to ignore wrong answers.Patching another activation into these heads improves logit difference by around 1–3%, and the authors hypothesize that a correct signal could help performance.Source: Figure 4 · p. 6, §4.4 · p. 7

4.5 SUMMARY OF SIMILARITIES AND DIFFERENCES

  • The inhibition heads and negative mover head are the main component differences; intervening to make the circuit more IOI-like increases accuracy and changes downstream behavior as predicted.The authors use this evidence to argue that the model learns a modular inhibition-and-copying subcircuit usable across task domains.Source: §4.5 · p. 7, §4.4 · p. 7

5 ERROR ANALYSIS ON COLORED OBJECTS

The Colored Objects circuit lacks useful inhibition and negative-mover signals, leaving mover heads to choose among distractor colors. Artificially supplying those signals improves accuracy and produces downstream mover-head changes predicted by the IOI circuit.

5 ERROR ANALYSIS ON COLORED OBJECTS

  • 49.6% to 93.7%: intervening on inhibition-head and negative-mover attention raised accuracy on the full dataset.The caption attributes the increase to interventions on both sets of heads.Source: Figure 5 · p. 8
  • The intervention forces three inhibition heads and one negative mover head to attend from the end position to incorrect color options.The proposed signal counters the inactive inhibition and negative-mover heads that leave movers struggling to select the right color over distractors.Source: §5 · pp. 7–8
  • 81.5% with just inhibition-head interventions and 78.1% with just the negative mover head, compared with 49.7% before both interventions.The combined interventions introduced zero new mistakes.Source: §5 · p. 8
  • After inhibition-head intervention, mover attention to incorrect colors fell by 8.7% on average, correct-color attention stayed high or rose by 2.7%, and mover logit attribution increased about 3x.Changes in logit attribution correlated with original path-patching effects across non-intervened heads (Spearman 0.69, p < 0.01).Source: §5 · p. 8, Figure 5 · p. 8

6 RELATED WORK

Prior work has examined both reusable specialized functions and task-specific components, alongside increasingly causal approaches to studying attention-head contributions. The cited work situates circuit analysis among these methods for understanding model predictions.

6 RELATED WORK

  • Induction, previous-token, and duplicate-token heads have specialized functions that can also be reused in arbitrary tasks, while other components detect conceptual features or store factual associations.The passage cites Olsson et al. (2022), Gurnee et al. (2023), Meng et al. (2022), and Geva et al. (2021) for these examples.Source: §6 · p. 9
  • Attention-head contributions to predictions have been studied through attention analysis and causal interventions, including path patching and circuit-analysis techniques.The passage cites Voita et al. (2019), Vig (2019), Pearl, Vig et al. (2020), Jeoung & Diesner (2022), Wang et al. (2022), Goldowsky-Dill et al. (2023), Nanda et al. (2022), Conmy et al. (2023), and Elhage et al. (2021).Source: §6 · p. 9
  • The passage contrasts reusable computational functions with specialized components that activate conceptual features or store factual associations.Examples include French-language-detecting neurons and components storing factual associations.Source: §6 · p. 9

7 CONCLUSION AND FURTHER DISCUSSION

The discussion argues that circuit findings from a small model and toy task can inform analysis of a larger model, while proposing a possible bottleneck in inhibition-head activation. It also outlines path patching and defines the paper’s specialized terminology.

7.1 CONCLUSION

  • The GPT2-Small IOI analysis enabled accurate predictions about a different, more complex task on GPT2-Medium, though the tested scale remains relatively small.The authors present this as a proof of concept for applying small-model circuit analysis to larger models.Source: §7.1 · p. 9

7.2 PREDICTING MODEL BEHAVIORS

  • Understanding core algorithmic steps could help anticipate model behavior on new data; this work builds a higher-level account of a larger model from low-level analysis of a small model.The authors note that demonstrating this kind of prediction may not be trivial.Source: §7.2 · p. 9

7.3 WHY CAN’T GPT2 ACTIVATE THE INHIBITION HEADS ON ITS OWN?

  • The authors speculate that content-gathering signals may remain incomplete when inhibition heads are called, potentially explaining why scale helps or motivating a second-pass recurrent model.They also suggest deeper networks may implement inhibition heads redundantly.Source: §7.3 · p. 9

A EXPLANATION OF PATH PATCHING

  • Path patching compares two inputs with different expected next tokens, caches their activations, and uses interventions to identify components affecting predictions.The examples differ in which name is the indirect object, and activation patching is introduced as a preliminary to path patching.Source: A EXPLANATION OF PATH PATCHING · p. 12

A.1 ACTIVATION PATCHING

  • Activation patching replaces a head’s output on input B with its cached output from A, but the resulting logit change may include effects through downstream components.The procedure checks whether the replacement promotes A’s answer.Source: §A.1 · p. 12

A.2 PATH PATCHING

  • Path patching isolates a sender-to-receiver path by recomputing receiver activations while patching other downstream components, then measuring the resulting logit difference.The algorithm uses four forward passes; in practice, heads are first tested against logits and then against selected heads, with MLPs recomputed.Source: §A.2 · pp. 12–13

B GLOSSARY OF TERMS

  • The glossary defines circuit, residual stream, [end], patching to X vectors, dot with token embedding, and attention probability on a token as paper-specific analysis terms.These terms describe component paths, information flow, sequence position, targeted patching, token-direction output, and attention mass.Source: B GLOSSARY OF TERMS · p. 13

C PATH PATCHING RESULTS ON IOI

Path patching identifies a similar sequence of components across IOI and Colored Objects: mover heads affect logits, while upstream heads shape where they attend and what they write. In Colored Objects, content gatherers carry object information, and duplicate-token and induction heads provide signal to them.

C PATH PATCHING RESULTS ON IOI

  • In GPT2-Medium IOI, path patching reproduces the circuit and identifies mover heads as direct logit contributors, with query patching revealing inhibition heads and value patching revealing induction and duplicate-token heads.The IOI mover heads copy the attended indirect-object name into the residual stream.Source: C PATH PATCHING RESULTS ON IOI · p. 13, Figure 6 · p. 14, §C.1 · p. 15

C.2 THE NEGATIVE MOVER HEAD

  • The IOI negative mover head 19.1 demotes the subject token and is informed by a mostly disjoint query-input set: heads 16.0, 18.2, and 18.9.Head 18.9 tends to attend to test color-option tokens.Source: §C.2 · p. 15

RESULTS ON COLORED OBJECTS

  • Heads 15.14 and 16.15 contribute 28% and 13% of the logit difference, respectively, in Colored Objects; 19.15, 18.5, and 17.4 also contribute significantly.The negative mover head 19.1 contributes about 1%, differing from its role in IOI.Source: RESULTS ON COLORED OBJECTS · p. 15
  • In Colored Objects, the same mover heads copy attended color words, while path patching identifies inhibition heads 12.3, 13.4, and 13.14 as affecting their queries.The three principal mover heads in IOI are also important movers in Colored Objects.Source: RESULTS ON COLORED OBJECTS · p. 15, §C.3 · p. 15, Figure 11 · p. 19
  • GPT2-Medium’s ability to separate what the two mover-head types attend to may reflect scale, but the authors leave that explanation for future work.GPT2-Small attends to the IO token using both types of mover heads.Source: RESULTS ON COLORED OBJECTS · p. 20

D.2 CONTENT GATHERER HEADS

  • Content gatherers attend from the final position to the question, especially obj2; blocking attention to obj2 or ablating these heads reduces performance to around chance among three colors.Blocking attention to “color” does not reduce performance, but the authors cannot infer the importance of attending to that token because the prompt provides other cues.Source: §D.2 · p. 20, Figure 13 · p. 20, Figure 12 · p. 20

D.3 WHAT ARE THE INHIBITION HEADS DOING?

  • The Colored Objects inhibition heads show varied attention to prior answers and test color options, without clear selective inhibition of wrong answers; blocking obj2 also redistributes mover attention unevenly by answer position.When the correct color is the first or third option, blocking obj2 reduces its attention and increases attention to wrong options, unlike when it is the middle option.Source: Figure 14 · p. 21, Figure 15 · p. 22

D.4 WHAT IS THE NEGATIVE MOVER HEAD DOING?

  • The negative mover in Colored Objects has a congruent but less impactful role, directing some attention to color words and the in-context answer, while duplicate-token and induction heads provide the strongest signal to content gatherers.The attention to the previous answer would prevent simply repeating it.Source: Figure 16 · p. 22, E DUPLICATE TOKEN AND INDUCTION HEADS · p. 23

F ADDITIONAL SOURCES OF ERROR IN COLORED OBJECTS

The authors identify input-feature and position sensitivities as additional sources of Colored Objects errors.

F ADDITIONAL SOURCES OF ERROR IN COLORED OBJECTS

  • Head 19.15 can overreact to brown, and to a lesser extent orange, promoting those colors as answers even when brown appears in the in-context example.The authors describe this as an additional error source beyond circuit-level difficulty inhibiting incorrect choices.Source: F ADDITIONAL SOURCES OF ERROR IN… · pp. 23–24, Figure 19 · p. 24

F.1 BIAS TOWARDS POSITION IN A SEQUENCE

  • GPT2-Medium is sensitive to the order of colored objects in the input, showing a position bias in Colored Objects.The figure analysis notes that blocking obj2 affects mover attention differently depending on whether the correct answer is first, middle, or third.Source: §F.1 · p. 24

G ADDITIONAL DETAILS ON INTERVENTION RESULTS

The intervention changes mover-head contributions and attention patterns in ways consistent with the IOI circuit analysis.

G ADDITIONAL DETAILS ON INTERVENTION RESULTS

H DETAILS ON COLORED OBJECTS DATASET CONSTRUCTION

The appendix describes the modified Colored Objects dataset and compares circuit overlap in two additional tasks. Greater Than has low overlap with the main-task circuits, while the World Capitals passage describes a separate factual-recall analysis setup.

H DETAILS ON COLORED OBJECTS DATASET CONSTRUCTION

  • The modified Colored Objects task uses three differently colored objects, one in-context example, and a test question about one object; objects are drawn from 17 classes and eight colors.Each object and color appears only once per example, though repeats across the context and test examples are allowed.Source: H DETAILS ON COLORED OBJECTS DATASET… · p. 25

I CIRCUIT OVERLAP IN ADDITIONAL TASKS

  • The additional-task comparison examines Greater Than and World Capitals as tasks distinct from IOI and Colored Objects, with prior work motivating different expected circuit overlap.The Greater Than analysis uses 200 examples; the World Capitals analysis uses 200 pairs filtered to 47 single-token countries.Source: I CIRCUIT OVERLAP IN ADDITIONAL TASKS · p. 26

I.1 GREATER THAN CIRCUIT

  • In Greater Than, MLPs perform most of the numerical comparison, producing low overlap with IOI and Colored Objects, where MLPs play almost no part.The circuit uses earlier induction heads to attend from the final token to the start year's last two digits, and these feed later MLPs.Source: §I.1 · p. 26
  • Among the 18 Greater Than circuit components, Colored Objects overlaps on 4/18 (22.2%) and IOI on 3/18 (16.7%).Some shared induction heads contribute in all tasks because they match an induction pattern; mover head 14.14 plays a similar role in Colored Objects.Source: §I.1 · p. 26

I.2 WORLD CAPITALS CIRCUIT:

The World Capitals circuit builds a country representation and uses mover heads to promote its capital; its full circuit shares only a quarter of components with both IOI and Colored Objects.

I.2 WORLD CAPITALS CIRCUIT:

  • Mover heads attend to the country token and promote the capital, using country-position values shaped predominantly by early-layer MLPs.The summary identifies MLPs 0, 1, 6, and 9–14, plus heads 14.7, 15.0, 16.2, 19.15, 20.6, 20.12, and 21.12.Source: §I.2 · pp. 26, 28, Figure 24 · p. 27
  • 4/16=25% of full-circuit components overlap with both Colored Objects and IOI, although 4/7 mover heads are shared.The remaining circuit relies heavily on MLPs, reflecting different earlier processing.Source: §I.2 · p. 28

I.3 CHALLENGES IN QUANTIFYING CIRCUIT OVERLAP

  • The authors do not claim an exact measure of circuit similarity because circuit membership criteria remain poorly defined.They use the proportion of shared nodes and examine three processing steps, while noting that other analyses could yield different overlap estimates.Source: §I.3 · p. 28

J DO OTHER MODELS ALSO USE SIMILAR CIRCUITS FOR THESE TASKS?

Across GPT2-Large and GPT2-XL, overlap in the most important heads declines with model scale, but mover-head functions remain similar; the paper also limits its generality claims to the tasks studied.

J DO OTHER MODELS ALSO USE SIMILAR CIRCUITS FOR THESE TASKS?

  • As model scale increases, exact head overlap becomes less likely, while functional overlap remains: many identified heads are movers on both tasks.The authors attribute the reduced exact overlap to the greater number of paths through larger networks.Source: J DO OTHER MODELS ALSO USE SIMILAR… · pp. 28–29
  • 5 of the top 10 heads overlap in GPT2-Large, versus none in GPT2-XL; the shared Large heads are movers, while XL’s identified heads include four movers and one negative mover on both tasks.The figure’s colored bars show percent logit difference.Source: J DO OTHER MODELS ALSO USE SIMILAR… · p. 28, Figure 25 · p. 29

K HOW WAS THE COLORED OBJECTS TASK SELECTED?

  • Colored Objects was chosen because selecting an answer from distractors qualitatively resembles IOI, while preliminary number-prediction and capital tasks showed virtually no IOI overlap.The authors did not path-patch the full circuits of those other tasks.Source: K HOW WAS THE COLORED OBJECTS TASK… · p. 29
  • The study cannot establish how generally the mover circuit is used beyond the tasks examined.The authors identify broader circuit generality as an open problem.Source: K HOW WAS THE COLORED OBJECTS TASK… · p. 29
Loading 2310.08744v3…