Source-linked AI summary
How to use and interpret activation patching
Stefan Heimersheim, Neel Nanda
TL;DR
Activation patching is widely used, but applying and interpreting it involves subtleties concerning circuit evidence and metric choice. The paper synthesizes practical advice and best practices, covering patching variants, interpretation, and metrics. Its examples show that denoising and noising can provide different evidence for circuit components, while prompt sensitivity, backup components, and metric design constrain interpretation.
Problem
Activation patching is widely used, but its application and interpretation involve subtleties about circuit evidence and metric choice.
Method
The paper synthesizes practical advice and best practices, reviewing patching variants, circuit interpretation, and metric choices and pitfalls.
Results
Denoising can identify sufficient components while noising identifies necessary components, and their results differ for circuits with AND or OR structure.
Takeaways & Limitations
Patching results should be interpreted in light of prompt choice, backup components, circuit structure, and the metric used.
Takeaways & Limitations
Patching is sensitive to the choice of corrupted prompts, backup components can weaken visibility of original components, and difference metrics can produce false positives.
Abstract
from arXiv · showhide
Activation patching is a popular mechanistic interpretability technique, but has many subtleties regarding how it is applied and how one may interpret the results. We provide a summary of advice and best practices, based on our experience using this technique in practice. We include an overview of the different ways to apply activation patching and a discussion on how to interpret the results. We focus on what evidence patching experiments provide about circuits, and on the choice of metric and associated pitfalls.
1 Introduction
Activation patching replaces selected internal activations with cached activations from another model run to study model behaviours and circuits. This introduction presents the basic workflow, its contrast with ablation, and the document’s focus on interpreting evidence and metrics.
- 1.1 What is activation patching?: Activation patching replaces internal activations during one model run with cached activations from a different input, then measures the output change.The technique is also called Interchange Intervention, Causal Tracing, Resample Ablation, or Causal Mediation Analysis.
- 1.2 How is this related to ablation?: Unlike ablation, which zeros activations, patching substitutes other activations to make more targeted and controlled manipulations.This enables experiments aimed at locating specific model behaviours and circuits.
- 1.3 An example: Changing the source prompt can isolate different information, such as language, landmark identity, factual recall, or which entity attributes are retrieved.For example, contrasting “The Colosseum is in” with “The Louvre is in” varies the landmark while controlling much of the surrounding prompt.
- 1.3 An example: A typical experiment selects two similar prompts, chooses activations to patch, caches activations from a source prompt, and inserts them into a destination run.Researchers then inspect output changes and repeat the procedure across activations of interest, such as MLP layers.
- 1.4 What is this document about: The document focuses on what patching experiments reveal about circuits, how to interpret those results, and how metric choices create pitfalls.It is intended as practical advice and complements broader introductions to mechanistic interpretability and activation patching.
2 What kind of patching experiments should you run?
Patching experiments should be chosen according to whether they explore candidate components or confirm a hypothesised circuit, while varying both patching direction and component granularity. Denoising and noising can yield different evidence, so practical workflows often progress from broad component scans to finer-grained and path patching analyses.
- Exploratory versus confirmatory patching: Exploratory patching sweeps components individually to identify model parts involved in a task and possible circuit membership.
- Exploratory versus confirmatory patching: Confirmatory patching changes many components together to test whether a hypothesised circuit covers the components required for task performance.Causal Scrubbing is cited as an example of circuit-verification patching.
- Choosing components and granularity: Patch granularity ranges from residual streams to MLPs, attention heads, individual neurons, and sparse autoencoder features.
- Choosing components and granularity: Path patching restricts an intervention to a single target component, enabling tests of direct connections versus mediation through another component.Default component patching can make mediation and direct amplification or calibration appear identical.
- Choosing components and granularity: A practical rule is to begin with low-granularity patching, increase granularity, and then use path patching; attribution patching can provide faster approximations.
- Noising and denoising: Denoising restores clean behaviour by inserting clean activations into a corrupted run, whereas noising tests which activations are necessary by inserting corrupt activations into a clean run.
- Noising and denoising: In an AND-gate example, denoising A or B has no effect while denoising C restores output; noising can affect A, B, and C, showing that direction changes which components are detected.The passages state that noising finds all circuit components in the first pass for this case, while denoising finds only one.
3 How do you interpret patching evidence?
Patching direction determines what evidence an experiment provides: denoising tests sufficiency, while noising tests necessity, and the two are not generally complements. The toy circuit shows how prompt choice, downstream mediation, backup behavior, and other caveats constrain interpretation.
- Patching directions: Denoising tests whether patched activations are sufficient to restore behavior, whereas noising tests whether they are necessary to maintain it.These directions can yield different results and neither generally implies the other.
- Stylized example: In the Nobel Peace Prize circuit, noising identifies the L0H0 output, L1N42 output, and Nobel and Peace embeddings as necessary components.The circuit requires the attention-head output and Peace embedding together, while L1N42 alone can restore behavior under denoising.
- Stylized example: Denoising finds only L1N42 because downstream patching can restore behavior using clean information already present in earlier layers.Thus, denoising a later component may bypass an important mediated component in the preceding layer.
- Prompt and path design: Varying corrupt prompts, such as changing Nobel or Peace separately, helps identify which components depend on which inputs.Path patching can then test precise interactions and distinguish direct connections from mediated or amplified effects.
- Scope and interpretation: Positive patching results identify activations handling the clean–corrupt prompt difference, so multiple corrupted prompt sets may be needed.Activation patching makes claims only for the tested prompt distributions, not for model behavior outside them.
- Scope and interpretation: Circuit analyses typically test sufficiency rather than minimality, while backup components can form OR-like redundancy and obscure the importance of original components.Backup behavior may be lossy; one cited example reports compensation of about 0.7*X for an original +X logit boost.
4 Metrics and common pitfalls
Metric choice can substantially change how exploratory patching results are interpreted. Continuous, roughly logit-linear metrics are generally more informative for partial effects, while discrete, nonlinear, or unspecific metrics can distort them.
- Metric selection: The metric matters more for exploratory patching than confirmatory patching, where a binary-like circuit verdict should make metrics broadly agree.The authors recommend implementing several metrics and comparing their results.
- Alternative metrics: KL divergence is useful when comparing the full model output, and model internals can themselves serve as task-specific metrics.Examples include attention paid to the indirect object, key neuron or SAE-feature activations, and probe projections.
- Metric selection: Continuous, roughly logit-linear metrics are usually preferable for exploratory patching because they support more accurate partial credit.The paper especially recommends logit difference or logprob difference among commonly used metrics.
- Metric selection: Logit difference compares selected correct and incorrect logits, filtering out components that affect all compared answers and often matching the mechanistic process.It is also mostly linear in the residual stream, supporting attribution to individual components.
- Common pitfalls: Logprob can saturate after the correct answer becomes the top guess, while also failing to control for components that boost correct and incorrect answers together.Raw logits have arbitrary baselines and can capture components contributing broadly across logits.
- Common pitfalls: Probability is exponential in logit space, so it can overemphasize effects near thresholds and suppress effects elsewhere.Its nonlinearity means the same logit change can produce very different percentage-point changes depending on the baseline, and it can saturate.
- Common pitfalls: Discrete metrics can hide substantial changes that do not cross thresholds or overstate a single contributor that happens to cross one.They are therefore generally recommended for confirmatory rather than exploratory patching.
5 Summary
Activation patching should be preferred to ablation in most situations, with careful attention to patching direction and prompt selection. Use multiple metrics, emphasizing continuous logit-based measures, and interpret results within the tested prompt distributions.
- Summary: Activation patching is generally preferable to ablation because it replaces activations with controlled alternatives rather than zeroing them out.Different corrupted prompts provide different information, so experiments should test a range of prompts.
- Summary: Denoising tests sufficiency for restoring behavior, while noising tests necessity for maintaining it; the directions are not symmetric.The interpretation of a result depends on which direction was patched.
- Summary: Discrete, overly sharp, or unintentionally sensitive metrics can mislead, so results should be evaluated with multiple metrics including a continuous, roughly logit-linear measure.Logit difference and logprob are given as examples of suitable metrics.