PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Best Practices of Activation Patching in Language Models: Metrics and Methods

Fred Zhang, Neel Nanda

arXiv:2309.16042v2cs.LGcs.AIcs.CL

TL;DR

Activation patching is widely used for localization, but the literature lacks consensus on methodological details. The paper systematically varies corruption methods, evaluation metrics, and patching procedures across language-model localization and circuit-discovery tasks, finding disparate interpretability results and proposing best-practice recommendations.

  • Problem

    Activation patching has broad applications but little consensus on how to generate corrupted prompts or evaluate patching effects, motivating a systematic study of methodological sensitivity.

  • Method

    The paper empirically compares Gaussian noising and symmetric token replacement, probability and logit difference, and sliding-window patching across several localization and circuit-discovery tasks.

  • Results

    Variations in activation-patching methods can produce different interpretability results, including inconsistent localization and circuit-discovery outcomes across corruption methods and metrics.

  • Takeaways & Limitations

    The authors recommend STR when possible and logit difference as an evaluation metric because these choices preserve in-distribution behavior and can detect negative modules.

  • Takeaways & Limitations

    Probability may fail to detect negative model components because of its non-negative nature, and interventions that do not affect a head will not appear on any metric.

Abstract

from arXiv · show

Mechanistic interpretability seeks to understand the internal mechanisms of machine learning models, where localization -- identifying the important model components -- is a key step. Activation patching, also known as causal tracing or interchange intervention, is a standard technique for this task (Vig et al., 2020), but the literature contains many variants with little consensus on the choice of hyperparameters or methodology. In this work, we systematically examine the impact of methodological details in activation patching, including evaluation metrics and corruption methods. In several settings of localization and circuit discovery in language models, we find that varying these hyperparameters could lead to disparate interpretability results. Backed by empirical observations, we give conceptual arguments for why certain metrics or methods may be preferred. Finally, we provide recommendations for the best practices of activation patching going forwards.

1 INTRODUCTION

Activation patching is a standard localization tool, but inconsistent methodological choices can produce different interpretability results. This study systematically examines corruption methods, evaluation metrics, and sliding-window patching across several language-model tasks, then proposes recommendations.

  • Activation patching: Activation patching localizes important components by comparing a corrupted run with one where a selected clean activation is restored.The procedure uses clean, corrupted, and patched forward passes to assess whether a component affects the output.
  • Motivation: Prior work applies activation patching to factual recall and circuit analysis, but uses varied methodological details.Applications include identifying weights storing factual information and sub-networks implementing behaviors.
  • Study design: The study varies corruption methods, evaluation metrics, and sliding-window patching across factual recall and multiple circuit-discovery tasks.The tasks include IOI, greater-than, Python docstring completion, and basic arithmetic.
  • Findings: GN and STR can yield inconsistent localization and circuit-discovery outcomes, with GN potentially disrupting internal mechanisms by moving the model off distribution.The paper presents tentative evidence for this explanation in IOI circuit discovery.
  • Findings: Sliding-window patching produces more pronounced localization than patching individual layers and summing their effects.The paper also discusses the conceptual difference between these approaches.
  • Recommendations: The authors recommend STR for corruption and logit difference for evaluation because STR preserves in-distribution behavior and logit difference can detect negative modules.They argue that logit difference offers fine-grained control over localization outcomes.

2 BACKGROUND

Activation patching compares model behavior across clean, corrupted, and patched runs to estimate the importance of individual activations. The background defines alternative corruptions and metrics and introduces the factual-recall and IOI settings used in the study.

  • Activation patching: Activation patching caches clean activations, corrupts the prompt, then restores one component’s clean activation during a patched run.The patching effect measures how much the intervention restores performance relative to the corrupted run.
  • Corruption methods: GN adds Gaussian noise to key-token embeddings, whereas STR replaces key tokens with semantically related tokens of equal sequence length.The GN noise scale is defined as 3 times the standard deviation of token embeddings from the textset.
  • Corruption methods: STR produces in-distribution corrupted prompts, while GN does not define a corrupted answer token for evaluating logit difference.Under GN, the same alternative answer token used for comparison is supplied to the metric.
  • Metrics: Probability measures P(r), logit difference compares Logit(r)−Logit(r′), and KL divergence compares the clean output distribution with corrupted or patched distributions.The normalized logit-difference effect typically lies in [0, 1], with 1 representing fully restored performance and 0 the corrupted-run performance.
  • Problem settings: Factual-recall experiments test prompts requiring factual completions such as “The Eiffel Tower is in,” while IOI experiments predict an indirect-object name.The IOI setting uses single-token names and focuses on an attention-head circuit previously identified in GPT-2 small.

3 CORRUPTION METHODS

The study finds that corruption choices can substantially change activation-patching localization and circuit-discovery results. Gaussian noising may disrupt internal mechanisms through out-of-distribution inputs, while symmetric token replacement preserves in-distribution behavior but does not eliminate all discovery limitations.

  • Factual recall: In factual recall, GN produces a clear MLP-patching peak around layer 16, whereas STR makes that peak nonsalient regardless of the evaluation metric.
  • Factual recall: Across window sizes, the peak value under GN is 2×–5× higher than under STR.
  • IOI circuit discovery: For IOI circuit discovery, STR and GN detect different sets of important attention heads for any fixed metric.
  • IOI circuit discovery: The detected IOI heads remain within the known circuit, but discovery is incomplete, including critical misses such as Name Movers.The authors state that manual inspection and path patching are necessary for fully discovering the IOI circuit.
  • Cross-task validation: GN can produce highly noisy localization outcomes on Python docstring and greater-than tasks, while STR can produce stronger MLP concentration on some basic arithmetic tasks in GPT-J.
  • Evidence for OOD behavior: Under GN, Name Mover attention is divided between IO and S1, whereas STR preserves the attention pattern with their roles switched.On clean prompts, Name Movers assign an average of 0.58 attention probability to IO; under GN, the corresponding averages are 0.26 for IO and 0.21 for S1.

4 EVALUATION METRICS

The choice of evaluation metric changes which activations appear important. Probability can emphasize token-level effects but may miss negative components when corruption drives the correct-token probability near zero, whereas logit difference remains informative.

  • Metric-dependent localization: Metric choice changes patching effects across token positions, with probability assigning stronger importance to the last subject token than logit difference.This pattern holds under both STR and GN corruption and across additional window sizes and GPT-J 6B validation.
  • Metric-dependent localization: 4.33× in probability > 1.22× in logit difference for the last-to-middle subject-token effect ratio under STR corruption.
  • Metric-dependent localization: 1.74× in probability > 0.77× in logit difference for the corresponding ratio under GN corruption.
  • Probability limitation: Probability can fail to detect negative model components when corruption reduces the correct-token probability to near zero.Its non-negative nature limits the available patching-effect range in such settings.
  • Probability limitation: Under random-name corruption, the original indirect-object probability is 5e−4, so probability detects neither Negative Name Mover heads while logit difference still can.

5 SLIDING WINDOW PATCHING

Sliding-window patching jointly restores adjacent MLP layers, producing stronger and more concentrated localization than aggregating individual-layer effects. The authors caution that this amplification may reflect nonlinear joint effects rather than single-layer importance.

  • Method: Sliding-window patching jointly patches adjacent layers, whereas the comparison method sums effects from separate single-layer interventions.The aggregation assigns neighboring single-layer effects to the middle layer of each window.
  • Results: Sliding-window patching typically produces at least 20% more peak effect than summing single-layer patching effects across tested windows, metrics, and corruption methods.
  • Results: For window sizes 3, 5, and 10, sliding-window patching shows significant gaps over summation, while single-layer patching has a weak peak around layer 15.Increasing the window size appears to produce more pronounced concentration.
  • Interpretation: Sliding-window patching tends to amplify weak localization from single-layer patching, possibly because joint patching introduces nonlinear effects.The resulting localization plots therefore require careful interpretation.

6 DISCUSSION AND RECOMMENDATIONS

The discussion finds activation-patching outcomes sensitive to corruption choices, metrics, token selections, and layer-window design. It recommends in-distribution corruption, logit difference, testing alternative tokens, and treating sliding-window effects as joint rather than single-layer evidence.

  • Corruption methods: GN may place the model off distribution, potentially producing anomalous behavior and unreliable or illusory localization results.STR instead uses in-distribution counterfactual prompts and is recommended whenever possible; GN remains an alternative when STR is unsuitable.
  • Evaluation metrics: Logit difference is recommended because it controls for components that boost logits for all names and can detect negative modules that probability may miss.
  • Evaluation metrics: KL divergence tracks the full model output distribution and can also be reasonable for circuit discovery.
  • Sliding window patching: Sliding-window results should be interpreted as joint effects of the full window, not effects attributable to a single layer.The authors recommend trying single-layer patching first and using windows when individual layers have small effects.
  • Corruption tokens: Corrupting different tokens can change localization outcomes because interventions may trace different information within the model.Changing S1 and IO recovers nearly all three Name Mover heads, whereas corrupting S2 misses most of them.
  • Corruption tokens: Trying alternative corruptible tokens may support more exhaustive circuit discovery when the prompt permits that flexibility.

7 RELATED WORK

The related work situates activation patching within causal mediation and connects it to circuit analysis and broader mechanistic-interpretability research. These lines of work use internal model analysis to study computations, representations, and behavior.

  • Activation patching: Activation patching is presented as a causal-mediation-analysis variant used broadly in mechanistic interpretability.Related variants include causal tracing and path patching.
  • Circuit analysis: Circuit analysis provides post-hoc model interpretability and often requires manual researcher effort.
  • Mechanistic interpretability: Mechanistic interpretability aims to explain models’ internal computations and representations, with circuit analysis as one major direction.

8 CONCLUSION

The paper finds that activation-patching results vary substantially with methodological choices and recommends in-distribution STR corruption and logit difference metrics. Its scope is limited to decoder-only models up to 6B parameters, while several directions remain for future work.

  • Activation-patching variations can produce different interpretability results, motivating best-practice recommendations.
  • The experiments use decoder-only language models with at most 6B parameters, leaving other architectures and larger models untested.
  • The study tests overriding corrupted activations with clean activations, while the reverse patching direction remains for future comparison.
  • The authors provide tentative evidence that some corruption methods produce out-of-distribution behavior that may make interpretability claims unreliable.

C RESULTS ON ARITHMETIC REASONING IN GPT-J

In GPT-J arithmetic reasoning, the study compares corruption methods and patching metrics for MLP localization across addition, subtraction, and multiplication. STR produces sharper concentration for addition and subtraction, while multiplication yields nearly matching GN and STR results.

  • Experimental setup: The GPT-J experiments study addition, subtraction, and multiplication with up to three-digit arithmetic using 2-shot prompts.
  • Experimental setup: GN adds noise to X3 and Y3 embeddings, whereas STR replaces them with random integers from the same set to keep prompts in-distribution.
  • Experimental setup: The experiments patch single MLP-layer activations at the prompt’s last token and compare the authors’ metric with probability and logit difference.
  • Experimental results: 4× sharper concentration is observed with STR than GN for addition and subtraction under the probability and logit-difference metrics.
  • Experimental results: For multiplication, GN and STR produce nearly matching localization results, highlighting unpredictable sensitivity to corruption method.
  • Metric analysis: Under STR, a tiny corrupted-run probability can make the denominator amplify absolute differences between patched layers, producing pronounced peaks.

D RESULTS ON PYTHON DOCSTRING CIRCUIT

For Python docstring completion, localization depends on corruption method and evaluation metric. GN is generally noisy except with probability, while heads 3.0 and 3.6 are consistently highlighted across conditions.

  • Experimental setup: The study evaluates activation-patching variants on a pretrained 4-layer attention-only Transformer for Python docstring completion.
  • Experimental setup: The task uses random single-token English words and requires completing a docstring prompt with C_def; prior work reported 56% accuracy.
  • Experimental setup: Patching covers all attention heads and token positions, providing finer-grained localization of important heads and positions.
  • Experimental results: Across 200 instances, detections concentrate mainly at the C_def position and the prompt’s last token.
  • Experimental results: GN localization is mostly noisy except when probability is used, whereas heads 3.0 and 3.6 are consistently highlighted across metrics and methods.

E RESULTS ON THE GREATER-THAN CIRCUIT IN GPT-2 SMALL

In the GPT-2 small greater-than task, GN and STR localize substantially different attention heads. GN produces noisy, weakly localized results, whereas STR yields interpretable results overlapping prior circuit findings.

  • Experimental setup: The greater-than task asks GPT-2 small to complete a year prompt with an integer greater than the visible two-digit suffix.
  • Experimental setup: GN corrupts the YY token embedding, while STR replaces YY with 01; probability and logit-difference metrics aggregate outcomes above and below YY.
  • Experimental setup: The study patches attention-head outputs across all token positions.
  • Experimental results: GN and STR localize mostly disjoint head sets, with GN producing extremely noisy results inconsistent with prior findings.
  • Experimental results: STR identifies heads 6.9, 7.10, 8.11, and 9.1, which overlap heads discovered by prior, more sophisticated methods.
  • Experimental results: GN visualizations provide little localization, whereas STR plots are easily interpretable.

F WHICH TOKENS TO CORRUPT MATTERS

The token(s) selected for corruption substantially affect activation-patching localization because they determine which information the intervention traces. Experiments across IOI and factual-recall settings show that corruption choices, metrics, and corruption methods can produce different localization patterns.

  • IOI corruption choices: Corrupting S2 alone missed at least 2 of 3 Name Mover Heads, while all metric-and-method combinations missed heads 9.6 and 10.0.The Name Mover Heads directly contribute to the IO logit according to the cited prior circuit analysis.
  • IOI corruption choices: Corrupting S1 and IO enabled most metric-and-method combinations to detect all Name Mover Heads, while probability still missed the Negative Name Mover Head.The experiment used STR and GN corruption and averaged results across 500 sampled IOI sentences.
  • Which tokens to corrupt matters: Corrupting different tokens can greatly change localization outcomes, even when the task and patching procedure remain otherwise similar.The authors recommend trying multiple corruptions when several key tokens are available.
  • Factual recall and appendix results: In factual recall, single-layer patching showed weak or no middle-MLP peak, whereas GN produced a more pronounced concentration than other settings.Across token positions, probability highlighted the last subject token more strongly than logit difference; GPT-2 large and GPT-J showed related patterns.
Loading 2309.16042v2…