PaperlayerView exact revision on arXiv

Source-linked AI summary

Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals

Francesco Ortu, Zhijing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, Bernhard Schölkopf

arXiv:2402.11655v2cs.CL

TL;DR

LLMs’ black-box nature limits scientific understanding, and models can face prompts where factual and counterfactual mechanisms compete. This paper traces that competition with logit inspection and attention modification, finding distinct signals across components and localized attention positions that control factual-mechanism strength.

  • Problem

    The black-box nature of LLMs obscures scientific understanding, including how models handle a prompt that pits factual knowledge against a counterfactual redefinition.

    Source: §1 · p. 1

  • Method

    The paper formulates competition between mechanisms and uses logit inspection and attention modification to trace their contributions and identify influential positions.

    Source: §1 · p. 2, §8 · p. 9

  • Results

    Factual attributes appear at subject positions early, counterfactual attributes at attribute positions, and attention writes both signals to the last position; localized attention changes strengthen factual responses.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Tracing competing mechanisms identifies model components and attention positions associated with which mechanism dominates the final prediction.

    Source: §8 · p. 9

    The study primarily uses GPT-2 small, which has far fewer parameters than current state-of-the-art LLMs, so its conclusions may not generalize to larger models.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Interpretability research aims to bridge the gap between empirical success and our scientific understanding of the inner workings of large language models (LLMs). However, most existing research focuses on analyzing a single mechanism, such as how models copy or recall factual knowledge. In this work, we propose a formulation of competition of mechanisms, which focuses on the interplay of multiple mechanisms instead of individual mechanisms and traces how one of them becomes dominant in the final prediction. We uncover how and where mechanisms compete within LLMs using two interpretability methods: logit inspection and attention modification. Our findings show traces of the mechanisms and their competition across various model components and reveal attention positions that effectively control the strength of certain mechanisms. Code: https://github.com/francescortu/comp-mech. Data: https://huggingface.co/datasets/francescortu/comp-mech.

1 Introduction

The paper reframes interpretability from identifying individual mechanisms to tracing how competing mechanisms shape a model’s final prediction. It studies factual recall against counterfactual adaptation and reports how their signals develop and interact across model components.

Motivation

  • Prior interpretability work largely studies individual mechanisms, leaving open how multiple mechanisms interact in LLM decision-making.The motivating example is a model choosing between factual recall about who developed the iPhone and a counterfactual redefinition.Source: §1 · p. 1

Approach

  • The proposed competition-of-mechanisms formulation traces each mechanism and how one becomes dominant in the final prediction.The study pits MLP-based factual recall against counterfactual in-context adaptation enabled by attention-head copying.Source: §1 · pp. 1–2

Methods

  • Logit inspection and attention modification examine contributions from layers and individual heads, identifying critical positions and components.The authors also locate attention-matrix positions that can substantially control factual-mechanism strength.Source: §1 · p. 2

Main findings

  • Early layers encode factual attributes at the subject and counterfactual attributes at the attribute position; attention writes both signals to the last position.Highly activated heads attend to the attribute position, while factual information often gains strength by suppressing the counterfactual attribute.Source: §1 · p. 2

2 Related Work on Interpretability

Interpretability research has examined both what LLM representations encode and the mechanisms that produce model behaviors. This paper situates its focus on competition between mechanisms within that work.

Interpreting the Representations

  • Representation studies examine encoded features such as syntax, verb tense, and factual knowledge, using probing and geometric approaches.The cited examples include work on part of speech, syntax, and factual knowledge.Source: §2 · p. 2

Interpreting the Mechanisms/Functions

  • Mechanistic interpretability studies learned functions, including induction-head copying and factual recall attributed to MLP layers.Other examples include modular addition and greater-than operations.Source: §2 · p. 2

The Single Mechanism of Copy

  • The copy mechanism passes a copied token toward next-token prediction by attending to it in attention heads.The cited work describes this as a foundation for studying more complex mechanisms.Source: §2 · p. 2

The Single Mechanism of Factual Knowledge Recall

  • Prior factual-recall work locates factual information in GPT-2 mid-layer MLP units and describes early MLP enrichment followed by late attention writing to the last position.The account distinguishes early subject-embedding enrichment from later selection and writing of factual information.Source: §2 · pp. 2–3

Interplay of Multiple Mechanisms

  • A related study examines capital-country prompts and finds that subject and attribute frequencies in pretraining can affect factual recall; this work uses broader prompts and localizes attention-map paths.The authors characterize their mechanistic analyses as additional findings beyond that study.Source: §2 · p. 3

3 Problem Setup

The setup studies autoregressive next-token prediction when factual knowledge competes with a prompt-supplied counterfactual. It constructs paired statements and asks the model to continue the factual statement after a redefinition.

Prediction setting

  • The model is evaluated in inference mode as an autoregressive next-token predictor conditioned on preceding context tokens.The setup follows the next-token prediction behavior of autoregressive LLMs.Source: §3 · p. 3

Task construction

  • The task pairs a factual statement with a corresponding counterfactual statement, then prompts the model to continue the factual statement after the counterfactual redefinition.The example redefines the iPhone developer as Google before asking who developed it.Source: §3 · p. 3

Mechanism tracking

  • The competing mechanisms are tracked through the factual and counterfactual answer tokens, denoted tfact and tcofa.These labels abstract away from each token’s exact position in its statement.Source: §3 · p. 3

4 Method and Background

The paper describes the residual stream and two interpretability tools for tracing model computations: projecting internal states into vocabulary logits and modifying selected attention entries. These methods support inspection of both layer-level and localized attention behavior.

Residual stream

  • Each layer updates the residual stream with attention and MLP outputs, and the final state is projected back into vocabulary space.The residual stream begins with token embeddings; attention and MLP blocks use layer-normalized inputs.Source: §4 · p. 3

Method 1: Logit Inspection

  • Logit inspection projects residual-stream states into vocabulary space to compare support for factual and counterfactual tokens.This projection is known as the Logit Lens, which can occasionally misrepresent vocabulary-item importance, especially in early layers.Source: §4 · pp. 3–4

Method 2: Attention Modification

  • Attention modification changes selected entries in a head’s attention matrix, scaling a target attention value while leaving other values unchanged.The selected entry represents a token attending to an earlier token, and the modification is applied after softmax.Source: §4 · p. 4

5 Experimental Setup

The experiments use COUNTERFACT2 examples to contrast factual and counterfactual attributes, selecting cases where GPT-2 correctly predicts the factual token. The setup also tests GPT-2 small and Pythia-6.9B, with LogitLens used to inspect residual streams.

5 Experimental Setup

  • Each COUNTERFACT2 instance relates a subject and relation to factual and counterfactual attribute tokens, then prompts the model to complete the factual statement after a counterfactual rewrite.The template is (“Redefine:”, subject, relation, counterfactual token, subject, relation, blank).Source: §5 · p. 4
  • 10K test samples were randomly selected from 219,180 examples with single-token attributes for which the model correctly predicts the factual token.The setup builds on COUNTERFACT2.Source: §5 · p. 4
  • GPT-2 small was chosen to align with prior interpretability studies, while supplemental Pythia-6.9B results assess generalizability across models with different architectures and scales.The paper describes similar results across the two models as strengthening the findings.Source: §5 · p. 4
  • GPT-2 small has 117M parameters, 12 layers, 12 attention heads per layer, and a 768-dimensional residual stream; Pythia-6.9B has 32 layers, 32 heads per layer, and dimension 4,096.The paper uses pretrained models from Hugging Face and LogitLens in TransformerLens to inspect residual streams.Source: §5 · p. 4

6 Results and Findings

The analyses trace how factual recall and counterfactual adaptation compete across positions, layers, blocks, and attention heads. They also show that localized attention edits and the similarity of the competing tokens affect which mechanism wins.

6.1 Macroscopic Inspection across Layers and Token Positions

  • GPT-2 identifies the counterfactual mechanism in 96% of 10K test examples; early factual logits are strongest at the subject, while late logits for both mechanisms are strongest at the last position.The counterfactual token is best encoded at the attribute position in early layers, unlike the factual token; attention moves information toward the last position.Source: §6.1 · p. 5, Finding 1: Information flows from… · pp. 5–6, Figure 2 · p. 5

Finding 1: Information flows from different tokens

  • Both mechanisms’ logits grow across layers, but stronger counterfactual-over-factual competition begins after layer five and becomes more prevalent later.The individual logits rise from below 1 in early layers to around 15 in the final layer.Source: Finding 1: Information flows from… · p. 6

6.2 Intermediate Inspection of Attention and MLP Blocks

  • The block comparison uses ∆cofa, the added tcofa logit minus the added tfact logit at the last sequence position; positive values favor counterfactual support.Attention-block contributions sum the contributions of their heads.Source: §6.2 · p. 6

Finding 1: The attention blocks play a larger role

6.3 Microscopic Inspection of Individual Attention Heads

  • A few heads dominate the competition: L9H6 and L9H9 account for 65% of Layer 9’s ∆cofa, while L10H7 and L11H10 explain nearly 70% of the network’s negative contribution.Jointly ablating L10H7 and L11H10 decreases GPT-2 small factual recall from 4.13% to 0.65%.Source: §6.3 · p. 7
  • Highly activated heads supporting either mechanism attend to the attribute position, but factual-supporting heads lower the tcofa logit rather than raise tfact.For L10H7 and L11H10, tcofa logits are -1.13 and -1.05, respectively, the lowest among the heads discussed.Source: §6.3 · pp. 7–8, Finding 2: Only late but not early… · p. 7

6.4 Intrinsic Intervention by Attention Modification

  • Attention modification raises factual-mechanism wins from 4% to 50% in GPT-2 and from 30% to 50% in Pythia.The edits multiply attention weights from the final token to the attribute position in two GPT-2 heads and three Pythia heads; α = 5 is selected for both models.Source: §6.4 · p. 8

Finding 1: Similar tokens confuse the model more

7 Discussion and Future Work

The findings connect factual prediction to suppressing counterfactual information, while a small number of specialized attention heads can be targeted to increase factual responses. The authors identify prompt structure, model scale, and dataset breadth as questions for future work.

7 Discussion and Future Work

  • Factual prediction mostly reflects suppression of the counterfactual token read from the attribute position, rather than promotion of the factual token from the subject position.This finding is consistent with Geva et al. (2023) on late attention blocks writing information to the last position when a counterfactual premise is added.Source: §7 · p. 9
  • Up-weighting only two or three entries in specialized attention heads substantially increases factual responses.The authors describe this as a distinctive contribution alongside prior findings that specialized heads suppress counterfactual information (McDougall et al., 2023).Source: §7 · p. 9
  • Compared with Yu et al. (2023), the authors report improving factual responses through focused attention-map modification, and suggest prompt breadth or framing as possible reasons for the difference.They also propose that selecting fewer, more specialized heads may contribute to the discrepancy.Source: §7 · p. 9
  • Future work will test how findings depend on prompt structure and whether suppressing counterfactuals promotes factual responses in larger models and more varied datasets.Source: §7 · p. 9

8 Conclusion

The paper formulates competition between mechanisms as an interpretive lens for LLMs handling multiple mechanisms, and uses interpretability tools to trace that competition and identify localized attention positions associated with factual-mechanism strength.

8 Conclusion

  • The study proposes competition of mechanisms to interpret cases where multiple mechanisms operate but only one leads to the correct answer.It applies logit inspection and attention modification to identify critical positions and components, and finds a few localized attention-map positions that largely control factual-mechanism strength.Source: §8 · p. 9

Limitations

The study’s conclusions are bounded by its model choice, interpretability approach, and prompt design. The authors identify larger models and more diverse prompts as directions for further study.

Limitations

  • The experiments primarily use GPT-2 small, which has far fewer parameters than current state-of-the-art LLMs; larger models may yield different or more generalizable conclusions.The authors also raise the possibility of a U-shaped curve in counterfactual-mechanism dominance as models grow.Source: Limitations · p. 9
  • The analysis is heavily grounded in embedding-space interpretability, and logit inspection can sometimes misrepresent vocabulary-item importance, especially in early layers.Source: Limitations · pp. 9–10
  • The prompts use a relatively simple structure to make counterfactual information traceable, leaving behavior under more diverse prompts unexplored.The authors note that following information flow in more varied prompts is challenging.Source: Limitations · p. 10

Ethical Considerations

The ethical discussion connects counterfactual prompts with the risk of eliciting sensitive factual information. The Pythia-6.9b appendix extends the analysis across layers, model components, and attention heads, including heads that focus on the attribute position.

Ethical Considerations

  • An inaccurate redefinition in a prompt may lead a model to inadvertently reveal sensitive factual information.The paper frames understanding these dynamics as relevant to improving LLM safety.Source: Ethical Considerations · p. 10

A Experiments for Pythia-6.9b

A.1 Macroscopic Inspection across Layers and Token Positions

  • Figure 7 presents factual-token and counterfactual-token logits across positions and layers, with the two tokens shown in separate panels.The left panel is factual and the right panel counterfactual.Source: §A.1 · p. 14, Figure 7 · p. 14

A.2 Intermediate Inspection of Attention and MLP Blocks

  • Figure 8 examines attention and MLP contributions to the final-position logit difference, with attention shaping early processing and MLP blocks contributing more to factual logits in concluding layers.Source: Figure 8 · p. 14, §A.2 · p. 14

A.3 Microscopic Inspection of Individual Attention Heads

  • Figure 9 distinguishes heads favoring factual versus counterfactual logits, while Figure 10 shows relevant heads consistently attending to the attribute position.Figure 9 uses blue for heads favoring tfact and red for those favoring tcofa.Source: §A.3 · p. 14, Figure 9 · p. 15, Figure 10 · p. 15

B.1 Ranks Analysis in the Last Position

The counterfactual token remains highly ranked in the projected logits across layers, while its rank trends upward as the factual token’s rank trends downward. Relevant attention heads show a similar pattern, including at the attribute-to-final-position edge, whether they favor the factual or counterfactual token.

B.1 Ranks Analysis in the Last Position

  • The counterfactual token ranks among the 20 most likely tokens in the first five layers and between 20th and 70th in the network’s last part.The figure also shows its rank rising while the factual token’s rank falls, though the counterfactual token’s rank remains lower.Source: §B.1 · p. 15, Figure 11 · p. 15

B.2 Attention Pattern of Relevant Attention Heads

  • Across relevant attention heads, attention patterns are similar regardless of whether a head favors the factual or counterfactual token.The attribute-to-final-position edge is a notable shared pattern.Source: §B.2 · p. 16, Figure 12 · p. 17
Loading 2402.11655v2…