PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability

Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, Thomas Icard

arXiv:2301.04709v4cs.AI

TL;DR

Mechanistic interpretability seeks transparent algorithms that faithfully explain opaque low-level model mechanisms, but requires a precise way to relate such explanations to internal workings. The paper generalizes causal abstraction to arbitrary mechanism transformations, formalizes core interpretability concepts, and unifies many methods in this language. Its framework includes intervention algebras and exact transformations, while general interventionals may satisfy weaker algebraic constraints.

  • Problem

    Mechanistic interpretability needs a precise account of when transparent algorithms faithfully interpret known but opaque low-level model details without becoming just-so stories detached from internal workings.

  • Method

    The paper generalizes causal abstraction from hard and soft mechanism replacement to interventionals, functional mappings from old mechanisms to new ones, and develops intervention algebras and exact transformations.

  • Results

    The framework provides precise definitions for polysemantic neurons, the linear representation hypothesis, modular features, and graded faithfulness, while unifying a wide range of mechanistic interpretability methods.

  • Takeaways & Limitations

    Causal abstraction supplies a common theoretical language for comparing interpretability methods and relating high-level explanations to low-level model mechanisms.

  • Takeaways & Limitations

    The deterministic framework excludes probability, and general interventionals can fail to satisfy intervention-algebra properties such as left-annihilativity.

Abstract

from arXiv · show

Causal abstraction provides a theoretical foundation for mechanistic interpretability, the field concerned with providing intelligible algorithms that are faithful simplifications of the known, but opaque low-level details of black box AI models. Our contributions are (1) generalizing the theory of causal abstraction from mechanism replacement (i.e., hard and soft interventions) to arbitrary mechanism transformation (i.e., functionals from old mechanisms to new mechanisms), (2) providing a flexible, yet precise formalization for the core concepts of polysemantic neurons, the linear representation hypothesis, modular features, and graded faithfulness, and (3) unifying a variety of mechanistic interpretability methods in the common language of causal abstraction, namely, activation and path patching, causal mediation analysis, causal scrubbing, causal tracing, circuit analysis, concept erasure, sparse autoencoders, differential binary masking, distributed alignment search, and steering.

1 Introduction

Mechanistic interpretability seeks transparent algorithms that faithfully explain opaque low-level model mechanisms, avoiding explanations detached from internal workings. The paper proposes causal abstraction as a common language and theoretical foundation for this goal.

  • Low-level accounts in terms of vectors, activation functions, and weights are complete but typically fail to convey high-level principles that guide human action.
  • Causal explanations are especially apt because they elucidate counterfactual difference-making details of mechanisms underlying model behavior.
  • Mechanistic interpretability asks when a transparent algorithm over human-intelligible concepts faithfully interprets a black-box model’s opaque internal details.
  • Mechanistic interpretability methods need a common language to compare methodologies and avoid just-so stories divorced from model internals.
  • Causal abstraction mathematically analyzes systems at multiple detail levels by relating fine-grained microvariables to more abstract macrovariables.
  • The paper extends causal abstraction to arbitrary mechanism transformations and uses it to formalize interpretability concepts and unify many existing methods.

2 Causality and Abstraction

The paper develops a general theory of causal abstraction for deterministic neural-network models, extending mechanism replacement to broader mechanism transformations. It structures these transformations through intervention algebras and characterizes exact and lossy abstractions between causal models.

  • The theory defines interventionals as functional mappings from old mechanisms to new mechanisms, extending hard and soft interventions beyond complete mechanism replacement.
  • Intervention algebras impose modularity structure on otherwise unconstrained interventionals, allowing selected transformations to behave like hard interventions in a translated variable space.
  • Exact transformations characterize when one causal model’s mechanisms are realized by another, and their composition remains an exact transformation.
  • Bijective translations preserve all original details, whereas constructive causal abstractions merge microvariables into macrovariables while maintaining a precise description.

2.1 Deterministic Causal Models with Implicit Graphical Structure

This section formalizes deterministic causal models through signatures, mechanisms, settings, interventions, and implicitly induced graphical structure. Models may be acyclic or cyclic, and interventions are treated as mechanism transformations.

  • A signature specifies variables together with each variable’s non-empty range of possible values.
  • Partial settings assign values to selected variables, while total settings assign values to every variable; projections restrict settings to subsets.
  • A deterministic causal model consists of a signature and mechanisms assigning each variable a value as a function of all variable values.
  • Although mechanisms take total settings, variable dependence induces a causal ordering and its transitive closure captures indirect causation.
  • Acyclic models have a single solution, while abstraction operations can create cycles even from initially acyclic models.
  • Hard interventions replace selected mechanisms with constants, soft interventions replace them with functions, and interventionals transform mechanisms functionally.
  • The unrestricted class of all-variable interventionals contains every function mapping causal models with the same signature, but need not isolate natural model components.

2.2 Intervention Algebras

Intervention algebras capture the compositional structure of hard interventions and selected classes of soft or general interventions. Normal forms characterize their equivalence, while broader interventionals can violate the required algebraic properties.

  • An intervention algebra is an algebraic structure isomorphic to hard interventions under function composition.
  • Hard interventions are modular because operations on different variables commute, whereas repeated interventions on one variable are left-annihilative.
  • Sequences of atomic interventions have a unique normal form obtained by collapsing repeated equivalent elements and sorting equivalence classes.
  • Two intervention sequences are equivalent exactly when they have the same normal form.
  • The quotient of the free algebra by this equivalence relation forms an intervention algebra and is represented by hard interventions.
  • Closures of atomic soft interventions under composition also form intervention algebras because they satisfy commutativity and left-annihilativity.
  • General interventionals need not form intervention algebras: a mechanism-flipping interventional can compose with itself to produce the identity rather than itself.

2.3 Exact Transformation with Interventionals

Exact transformation formalizes when two causal models are consistent descriptions of the same causal situation under translations of settings and interventions. The framework includes lossy abstractions, bijective translations, alignments, and compositional decompositions.

  • Exact transformation: Exact transformation relates models through a setting translation τ and an interventional translation ω that preserve corresponding causal outcomes.The definition requires a commuting relationship for interventions in ω's domain.
  • Exact transformation: Exact transformations compose, so compatible transformations can be chained when their compositions are defined.
  • Bijective translation: A bijective translation changes representation without losing model detail and is itself an exact transformation with a canonical intervention mapping.The induced intervention algebra is isomorphic to hard interventions on the translated signature.
  • Bijective translation: The paper illustrates bijective translation with a rotated coordinate system that preserves Boolean conjunction while changing the mechanisms' representation.The output remains Z = X1 ∧ X2 after the rotated intermediate variables are un-rotated.
  • Constructive abstraction: An alignment partitions low-level variables into non-overlapping cells, maps each cell's settings to a high-level variable, and may forget residual variables.These cell-wise maps induce the setting and intervention translations used for constructive abstraction.
  • Constructive abstraction: Constructive abstraction holds when the high-level model is an exact transformation of the low-level model under the alignment-induced translations.Every alignment can be decomposed into marginalization, variable merge, and value merge.

2.4 Approximate Transformation

Approximate transformation relaxes all-or-nothing exactness by measuring how closely translated causal outcomes match across a chosen distribution of interventionals. The framework supports adaptable similarity measures and statistics, illustrated by a modulo-10 summation example.

  • Motivation: Approximate abstraction provides a graded alternative because exact transformation either holds or fails.
  • Definition: The metric combines a distance function Sim, an intervention distribution P, and a statistic S to quantify transformation quality in one number.Sim compares total settings, P specifies expected interventionals, and S summarizes the resulting random variable.
  • Definition: An η-approximate abstraction is obtained when the graded metric exceeds a chosen cutoff η.
  • Assumptions: Restricting intervention distributions to those assigning all input variables removes dependence on default input values.
  • Example: The framework can compare a full integer-sum model with a model restricted to multiples of 10 by translating values modulo 10.The example uses uniform hard interventions on both addends, absolute output difference as Sim, and expected value as S.

2.5 Interchange Interventions

Interchange interventions fix selected variables to values produced under source inputs while evaluating a base input, making them a central tool for causal abstraction analyses. The framework extends them recursively and to quantities distributed across multiple variables.

  • Interchange interventions: An interchange intervention evaluates a base input while fixing target variables to the values they would have under one or more source inputs.These interventions are used to test whether high- and low-level models exhibit corresponding input-output behavior.
  • Recursive interventions: Recursive interchange interventions fix variables to values generated by lower-depth interchange interventions.Depth 0 consists of ordinary interchange interventions, and depth m + 1 uses interventions of depth m.
  • Distributed interventions: Distributed interchange interventions use a bijective translation so interventions can target variables distributed across multiple causal variables.The translated model is intervened on, then mapped back to the original variable space.
  • Alignment: An alignment is sufficient to determine which low-level settings and interventions correspond to high-level variables and interventions.Only low-level values realized by some input are generally in the alignment's domain.
  • Limitation: Alignment construction can fail when identical low-level partial settings arise from inputs that imply different high-level values.In that case, the causal abstraction relationship does not hold for the alignment.
  • Evaluation: Interchange intervention accuracy is the proportion of interventions for which low- and high-level models have identical input-output behavior.Restricting interventions to a single identical base and source input makes this equivalent to input-output accuracy.

2.6 Example: Causal Abstraction in Mechanistic Interpretability

The paper demonstrates causal abstraction on hierarchical equality using a compositional tree algorithm and a fully connected feed-forward neural network that solve the same task. The symbolic model computes pairwise equalities before comparing those intermediate results.

  • Task: Hierarchical equality returns True when both input pairs are equal or both are unequal, and False otherwise.The illustrative object domain contains triangles, squares, and pentagons.
  • Motivation: The task is chosen because a tree-structured symbolic algorithm solves it compositionally and equality reasoning is relevant to relational-representation research.
  • Tree-structured algorithm: The tree algorithm computes equality for the first pair, equality for the second pair, and equality between those two results.
  • Tree-structured algorithm: The high-level causal model contains four object-valued input variables, two Boolean intermediate variables, and one Boolean output variable.The intermediate and output mechanisms compute equality over their respective domains.
  • Interventions: The default intervention-free setting is [D, D, D, D, True, True, True], while fixing X3, X4, and Y1 yields [D, D, △, □, False, False, True].
  • Neural network: The low-level model is a neural network with eight input neurons, twenty-four intermediate neurons, and two output neurons.

XOut

The paper illustrates causal abstraction by aligning a handcrafted fully connected network with a tree-structured algorithm and showing matching counterfactual behavior. It also demonstrates transformations and abstractions for hierarchical equality and bubble sort, including infinite-variable models and cyclic abstractions.

  • Hierarchical equality: The hierarchical equality network uses handcrafted weights to implement a tree-structured algorithm.The network labels inputs using real-valued intermediate activations and output logits.
  • Hierarchical equality: The alignment partitions network neurons into input, intermediate, and output groups corresponding to the algorithm’s variables.Input pairs encode shapes, while intermediate groups align with Y1 and Y2 and outputs align with OTrue and OFalse.
  • Hierarchical equality: Aligned interchange interventions produce equivalent counterfactual behavior in the low-level network and high-level algorithm.The example uses base input (D, D, △, □) and source input (□, D, △, △), with verification code provided in a companion notebook.
  • Hierarchical equality: The network-to-algorithm transformation marginalizes unaligned neurons, merges aligned variables, and maps continuous neural activity to symbolic values.This decomposition gives a causal-abstraction view of how the high-level algorithm is constructed from the low-level network.
  • Bubble sort: Causal abstraction can represent bubble sort with countably infinite variables and abstract its acyclic implementation into a cyclic equilibrium model.The model supports sequences of arbitrary length and arbitrarily many sorting iterations; abstraction can merge variables while retaining sorting histories or final sorted values.

3 A Common Language for Mechanistic Interpretability

The paper presents causal abstraction as a common language for defining mechanistic-interpretability concepts and comparing methods. It formalizes modular features and graded faithfulness, while showing how behavioral fidelity and integrated gradients fit into the framework with important scope differences.

  • Core concepts: Causal abstraction provides precise definitions for polysemantic neurons, linear representations, modular features, and graded faithfulness.The framework is intended to prevent mechanistic explanations from becoming stories disconnected from model internals.
  • Core concepts: Modular features are sets of variables forming an intervention algebra and accessed through a bijective translation.Individual activations, orthogonal directions, and model components can serve as separable components under this conception.
  • Core concepts: Rotation matrices suffice for mechanistic interpretation if the linear representation hypothesis is correct, whereas nonlinear translations may reveal nonlinearly accessible features.The proposal remains agnostic about the exact units of analysis and allows empirical falsification of particular decompositions.
  • Faithfulness: Approximate transformation makes faithfulness graded by varying state similarity, intervention distributions, and aggregation statistics.Exact transformations are binary, while approximate abstraction can be evaluated against a chosen cutoff.
  • Interpretability methods: LIME fidelity is an approximate-transformation metric with identity translation and intervention functions, but LIME does not capture internal causal dynamics.Its explanations match local input-output behavior without connecting the interpretable model to the uninterpretable model’s internal structure.
  • Interpretability methods: Integrated gradients can compute interchange interventions by using an interchange intervention as its baseline, although integration is inefficient for this purpose.This connects a gradient-based attribution method to causal-abstraction analysis through the completeness axiom.

XIn

The paper recasts several mechanistic interpretability techniques as causal-abstraction analyses of neural networks. These include mediation, patching, and ablation-based concept erasure, each represented through high-level models and intervention mappings.

  • Activation and Path Patching: Activation-patching terminology is broader than interchange intervention, sometimes including ablations or arbitrary transformations, so the paper distinguishes the terms.The paper uses activation patch broadly for hidden-vector interventions and interchange intervention for a specific causal-model operation.
  • Causal Mediation as Abstraction: Causal mediation analysis measures how changing inputs affects outputs through intermediate variables using total, direct, and indirect effects.The paper shows that mediation analysis is a special case of causal abstraction and is compatible with ablation and interchange interventions.
  • Causal Mediation as Abstraction: A hidden vector completely mediates input effects on outputs exactly when its indirect effect equals the total effect in the abstract model.The neural network is mapped to a three-variable high-level model whose input, output, and mediator variables correspond to the network’s inputs, outputs, and hidden vector.
  • Activation and Path Patching: Path patching is recursive interchange intervention analysis that targets connections between variables rather than variables themselves.Sender variables receive values from a source input, receiver variables are patched, and selected variables may remain fixed to the base-input values.
  • Concept Erasure: Concept erasure models ablation by introducing a binary high-level variable indicating whether a concept has been erased.The abstraction maps ordinary input interventions identically and maps the low-level ablation to setting the binary variable to 1; exactness requires the specified degraded behavior.

XIn

The paper formalizes concept erasure and sub-circuit analysis as causal abstractions with explicit behavioral criteria. It also identifies a scope limitation: ablation tests whether a concept is used, but not how it is used.

  • Concept Erasure: Ablation studies determine whether a concept is used by testing for degraded behavior, but they do not characterize how the concept is represented or used.The high-level model contains no variable encoding the concept or its possible values.
  • Sub-Circuit Analysis: Sub-circuit analysis seeks a minimal set of connections sufficient to preserve a specified behavior.The circuit is represented as a partial behavior function, and connections outside the selected sub-circuit are ablated.
  • Sub-Circuit Analysis: The high-level sub-circuit model uses a binary variable to indicate whether the selected connections have been severed.When severed, the model preserves the target behavior on its domain and returns an undefined value outside that domain.
  • Sub-Circuit Analysis: The abstraction is exact exactly when the selected sub-circuit preserves the specified behavior.This criterion translates circuit sufficiency into a causal-abstraction condition.

XIn

This section places causal scrubbing, modular feature learning, feature alignment, probing, and steering within the causal-abstraction framework, while identifying important scope limitations. It also connects several feature-learning methods to exact or approximate transformations.

  • Causal Scrubbing: Causal scrubbing tests whether a circuit is sufficient for a behavior by recursively intervening on every network connection.Connections in the circuit are replaced using source inputs selected to preserve corresponding high-level variables.
  • Causal Scrubbing: Causal scrubbing can be modeled as causal abstraction even though its high-level model only guides source-input sampling, not high-level interventions.The resulting formulation uses the three-variable causal model H∗.
  • Modular Features: Modular feature learning is formalized as constructing a bijective translation, with distributed interchange interventions enabling unified evaluation of feature-learning methods.The framework covers supervised methods with explicit alignment and unsupervised methods requiring a separate alignment step.
  • Unsupervised Methods: Sparse autoencoders separately learn encoder and decoder functions to represent n-dimensional activations in a sparse k-dimensional feature space where k ≫ n.They can be treated as bijective translations only when reconstruction is perfect; in practice, nonzero reconstruction loss makes them approximate transformations.
  • Feature Alignment: Differential binary masking aligns features with high-level variables by optimizing intervention-based binary masks, while sparse feature selection uses regularized linear probes.These alignment methods are especially relevant when unsupervised feature learning does not already provide correspondence to high-level variables.
  • Probing and Steering: Probing can reveal whether concepts are present in hidden vectors, but successful probing alone does not establish that those representations causally influence model behavior.Activation steering likewise can control hidden states without guaranteeing successful interchange-intervention effects, especially off the input-induced distribution.

4 Conclusion

The conclusion presents causal abstraction as a theoretical foundation for mechanistic interpretability. It argues that the framework clarifies core concepts and supports future investigation of algorithmic hypotheses about AI models’ internal reasoning.

  • 4 Conclusion: Causal abstraction provides a theoretical foundation that clarifies core mechanistic interpretability concepts.The conclusion frames this foundation as groundwork for future method development.
  • 4 Conclusion: The framework lays groundwork for methods investigating algorithmic hypotheses about the internal reasoning of AI models.This consequence is stated within the paper’s conclusion rather than as a broader empirical claim.
Loading 2301.04709v4…