PaperlayerView exact revision on arXiv

Source-linked AI summary

The Cognitive Revolution in Interpretability: From Explaining Behavior to Interpreting Representations and Algorithms

Adam Davies, Ashkan Khakzar

arXiv:2408.05859v1cs.AI

TL;DR

Deep learning has often treated model behavior as sufficient for understanding, while mechanistic interpretability also studies learned representations and internal algorithms. Drawing on cognitive science, this paper distinguishes those two interpretive aims and argues that causal analysis across them matters for faithful explanations of behavior.

  • Problem

    Deep learning often assumes behavior alone suffices to explain models, despite the interpretive questions posed by their learned representations and internal operations.

    Source: §1 · p. 2

  • Method

    The paper uses parallels with cognitive science to distinguish semantic interpretation of latent representations from algorithmic interpretation of operations over them.

    Source: §1 · p. 2

  • Results

    The paper presents semantic and algorithmic interpretation as distinct categories and highlights causal analysis across both as important for faithful explanations of model behavior.

    Source: §6 · p. 11

  • Takeaways & Limitations

    The taxonomy clarifies the distinct objects and assumptions of these approaches while considering whether they can be unified in a common framework.

    Source: §1 · p. 2, §6 · p. 11

    Because structural and causal probing require predefined latent properties, they can miss properties a model uses if those properties were not selected for probing.

    Source: §3.4 · p. 7

Abstract

from arXiv · show

Artificial neural networks have long been understood as "black boxes": though we know their computation graphs and learned parameters, the knowledge encoded by these weights and functions they perform are not inherently interpretable. As such, from the early days of deep learning, there have been efforts to explain these models' behavior and understand them internally; and recently, mechanistic interpretability (MI) has emerged as a distinct research area studying the features and implicit algorithms learned by foundation models such as large language models. In this work, we aim to ground MI in the context of cognitive science, which has long struggled with analogous questions in studying and explaining the behavior of "black box" intelligent systems like the human brain. We leverage several important ideas and developments in the history of cognitive science to disentangle divergent objectives in MI and indicate a clear path forward. First, we argue that current methods are ripe to facilitate a transition in deep learning interpretation echoing the "cognitive revolution" in 20th-century psychology that shifted the study of human psychology from pure behaviorism toward mental representations and processing. Second, we propose a taxonomy mirroring key parallels in computational neuroscience to describe two broad categories of MI research, semantic interpretation (what latent representations are learned and used) and algorithmic interpretation (what operations are performed over representations) to elucidate their divergent goals and objects of study. Finally, we elaborate the parallels and distinctions between various approaches in both categories, analyze the respective strengths and weaknesses of representative works, clarify underlying assumptions, outline key challenges, and discuss the possibility of unifying these modes of interpretation under a common framework.

1 Introduction

The paper frames mechanistic interpretability as a potential cognitive revolution in deep learning, shifting attention beyond behavior-only explanations toward learned representations and internal processing. It distinguishes semantic from algorithmic interpretation and examines their relationship.

  • Behaviorism explained human behavior through stimulus–response mechanisms, but gaps in language acquisition, problem-solving, and working memory helped spark the cognitive revolution.The shift centered psychological study on mental representation and information processing.Source: §1 · pp. 1–2
  • Mechanistic interpretability challenges deep learning’s behaviorist assumption by studying learned representations and internal algorithms, as well as how they relate to observed behavior [16] [17] [18].The paper describes this shift as a potential cognitive revolution in deep learning.Source: §1 · p. 2
  • Semantic interpretation asks what latent representations models learn and use.Source: §1 · p. 2
  • Algorithmic interpretation asks what operations and algorithms models implement over representations to produce behavior.Source: §1 · p. 2
  • The paper compares the two approaches’ assumptions, challenges, and conceptions of interpretation, drawing on cognitive-science history and considering a shared framework.Source: §1 · p. 2

2 Background

The paper traces a shift from post-hoc explanations of model outputs toward interpreting what foundation models learn and do internally. It uses Marr’s levels of analysis to distinguish representation-focused and algorithm-focused interpretation from analyses of the learning process.

2.1 A Brief History of Deep Learning Interpretation

  • Behavioral explanations and mechanistic interpretations are presented as distinct historical paradigms for explaining neural-network behavior.The paper situates these terms within a broader history of deep-learning interpretation.Source: §2.1 · p. 2
  • In the supervised-classification era, post-hoc behavioral explanations gained traction through feature attribution and counterfactual methods [25] [26] [27] [30] [34] [36].Feature attribution identifies input features relevant to an output; counterfactual explanations ask what must change in the input for a particular output.Source: §2.1 · p. 2
  • With foundation models, interpretation increasingly asks what is learned during pretraining and what internal algorithms are performed, distinguishing latent representations from operations and algorithms [37].The paper notes that probing can specify concepts in advance, whereas dictionary learning can dynamically discover them.Source: §2.1 · p. 3

2.2 Levels of Analysis

  • Marr’s framework separates computational goals, representations and algorithms, and their physical or software implementation.Source: §2.2 · p. 3
  • Earlier applications of Marr’s levels focused on learning-process choices, whereas recent interpretation methods study pretrained models’ representations and algorithms [49, 50].Without interpretation, application to a pretrained model is limited to treating embeddings as representations and its forward pass as the algorithm.Source: §2.2 · p. 4

3 Semantic Interpretation

Semantic interpretation asks what latent properties neural networks represent and how those properties contribute to behavior. Neuron-level methods investigate this by inspecting inputs that maximize a selected neuron's activation.

3 Semantic Interpretation

  • Semantic interpretation asks what latent properties neural networks learn and represent, and how those properties contribute to observed behavior.Examples include object categories or spatial relations in vision models and syntactic dependencies or lexical relations in language models.Source: §3 · p. 4

3.1 Optimization and Search

  • Neuron-level optimization and search inspects inputs that maximize an individual neuron's activation for recurring features.Methods search a predefined query set or generate optimized inputs; work extends from image models to language and multimodal models.Source: §3.1 · p. 4

3.1.1 Assumptions and Challenges

  • Single-neuron analysis is limited because representations can be distributed across neurons and multiple concepts can share one neuron.Some works analyze combinations of neurons, but the appropriate representation level and which Transformer components to examine remain open questions.Source: §3.1.1 · pp. 4–5
  • At billion-parameter scale, millions of functional neurons make exhaustive analysis intractable, and no generally accepted method identifies the most relevant neurons.Heuristics exist, but they cannot guarantee that the neurons most important to a research question have been targeted.Source: §3.1.1 · p. 5

3.2 Structural Probing

Structural probing trains auxiliary classifiers to predict discrete latent properties from model embeddings. The probe's architecture determines which representations it can detect, making that choice consequential.

3.2 Structural Probing

  • Structural probes predict discrete latent properties from model embeddings, with probe accuracy treated as evidence that a property is represented.The pipeline hypothesis used BERT layer probe accuracies to argue that surface, syntactic, and semantic properties are processed in that order.Source: §3.2 · p. 5

3.2.1 Assumptions and Challenges

  • Probe architecture must be selected in advance, and linear probes detect only properties encoded linearly.Which architecture to use is contentious, with associated arguments and limitations.Source: §3.2.1 · p. 5

Linear Probes

  • Linear probes assume properties are encoded in linear embedding subspaces, while simpler probes may better reflect embedding content than highly expressive ones.Expressive probes can memorize probe tasks irrespective of the model being probed.Source: Linear Probes · pp. 5–6

Nonlinear Probes

Nonlinear probes can detect properties encoded nonlinearly, but their expressivity can make it difficult to tell whether accuracy reflects the model or the probe. Probe results also depend on which properties are tested and do not establish that the model uses them.

Nonlinear Probes

  • Nonlinear probes can seek representations encoded nonlinearly, especially in intermediate layers, but their expressivity risks learning the task independently of the probed model.The paper illustrates this risk with an LLM trained to caption images from a frozen image encoder: the LLM learns the generative task, not the encoder.Source: Nonlinear Probes · p. 6
  • Researchers cannot know in advance which properties a model represents or uses, so selected probing properties may miss what matters to the model.Probe results may mislead without comparisons to randomized baselines, control tasks, or information gain computed using control functions.Source: Nonlinear Probes · p. 6
  • Predicting a property from a representation does not establish whether or how the model uses that property.Structural probing measures predictability, not use [12] [16] [17].Source: Nonlinear Probes · p. 6

3.3 Causal Probing

Causal probing intervenes on properties detected by structural probes and measures the resulting behavioral impact to study how models use them. Its results are constrained by representation-level choices, intervention selectivity, and the possibility of multiple equally effective explanations.

3.3 Causal Probing

  • Causal probing intervenes on probed representations and measures behavioral changes to study how properties are used by a model.Amnesic probing uses INLP to remove linearly predictable target-property information from LLM embeddings and measures language-modeling performance across a large text corpus [16].Source: §3.3 · p. 6

3.3.1 Assumptions and Challenges

  • Causal probing inherits the need to predefine representation levels and target properties, while most interventions operate at only one level.Adversarial approaches can target representations found by arbitrary probes but offer fewer guarantees against collateral damage to non-target properties [17].Source: §3.3.1 · p. 7
  • Linear interventions can be incomplete when properties removed from earlier layers remain recoverable by probes at later layers.More expressive interventions may improve completeness while increasing collateral damage and reducing selectivity, making behavioral changes harder to attribute to the target property [17].Source: §3.3.1 · p. 7
  • When removing different properties has the same behavioral impact, causal probing cannot determine which equally effective explanation is correct.The paper calls this ambiguity the Rashomon effect.Source: §3.3.1 · p. 7

3.4 Dictionary Learning

Dictionary learning replaces supervised prediction of preselected properties with unsupervised sparse feature decomposition, but shifts the annotation burden to interpreting and evaluating the learned features.

3.4.1 Assumptions and Challenges

  • Even perfect structural or causal probes can miss important task-relevant properties when those properties were not pre-specified for probing.This limits what supervised probing can establish about the properties a model represents and uses.Source: §3.4 · p. 7

3.4 Dictionary Learning

  • Dictionary learning decomposes embeddings into sparse feature combinations and reconstructs them, avoiding the need to predefine properties for supervised probes.Sparse auto-encoders implement this approach with an overcomplete linear basis and neural networks, enabling scalable feature learning.Source: §3.4 · pp. 7–8

3.4.1 Assumptions and Challenges

  • SAEs follow the superposition hypothesis: embeddings encode more features than neurons using near-orthogonal directions, with interference noise filtered by nonlinear activations.The account says this trade-off is favorable when features are sufficiently sparse.Source: §3.4.1 · p. 8
  • Unsupervised features still require interpretation from highly activating examples, often by human or LLM annotators, and interpretations must be redone for each model, layer, or SAE.At large scale, manual annotation is infeasible, while automated feature interpretation has shown reasonable alignment with human judgments.Source: §3.4.1 · p. 8
  • Feature interpretations are difficult to evaluate; causal interventions test whether features contribute to outputs but not whether other features contribute similarly or interventions are complete and selective.Automating interpretation and evaluation can compound errors across layers of LLM automation.Source: §3.4.1 · p. 9

4 Algorithmic Interpretation

Algorithmic interpretation studies operations over representations and the circuits that implement them; circuit discovery uses interventions but depends on choices and assumptions that constrain what it can find.

4 Algorithmic Interpretation

  • Algorithmic interpretation asks what operations a network performs over representations, how they relate to behavior, and what algorithm they implement in aggregate.These operations are typically described as circuits: network subgraphs that may compose into larger circuits.Source: §4 · p. 9

4.1 Circuit Discovery

  • Circuit discovery seeks circuits that faithfully describe model behaviors; an induction circuit is an end-to-end example that predicts a repeated token sequence.For an input shaped like “[A] [B] ... [A],” it determines whether to predict that [B] follows the second [A].Source: §4.1 · p. 9
  • Knockouts nullify a circuit and patching swaps in activations from a source sequence; both interventions test effects on model predictions.Mean ablation can be more theoretically sound and empirically less noisy than zero ablation when later circuits rely on activation values as implicit bias terms.Source: §4.1 · pp. 9–10

4.1.1 Assumptions and Challenges

  • Circuit results depend on the chosen atomic subgraph level and a one-operation-per-subgraph assumption; head-only analyses can miss MLP subcircuits, while larger subgraphs may contain redundant elements.Interventions can test whether a subgraph is involved, but cannot establish that no other subgraphs have similar effects.Source: §4.1.1 · p. 10
  • Real-world LLM circuit discovery has required pre-specifying an algorithm, and distinct circuit descriptions can explain the same behavior, sometimes with a model apparently performing both algorithms simultaneously.Pre-specification reduces computational complexity but makes discovery top-down and less likely to find intermediate subgraphs outside the specified algorithms.Source: §4.1.1 · p. 10

5 Towards Unifying Semantic and Algorithmic Interpretation

The paper describes causal abstraction as a possible common framework for semantic and algorithmic interpretation, while noting that its empirical demonstrations do not yet establish layer-to-layer algorithmic interpretation in real-world models.

5 Towards Unifying Semantic and Algorithmic Interpretation

  • Causal abstraction represents representations as nodes and operations over them as edges in an underlying graphical causal model.Faithfulness is measured through instance-level alignment under interchange interventions; a related framework extends interventions to concepts.Source: §5 · p. 11
  • Proposed empirical methods include learning linear rotational interventions over embedding spaces or replacing interchange interventions with gradient-based attacks against nonlinear causal probes.Source: §5 · p. 11
  • The cited demonstrations are limited to toy models and tasks or single-layer interventions, so layer-to-layer operations in real-world models have not yet been empirically demonstrated.Source: §5 · p. 11

6 Conclusion

The paper connects deep learning interpretation to cognitive science and distinguishes semantic interpretation of represented properties from algorithmic interpretation of operations over representations.

6 Conclusion

  • The paper relates deep learning interpretation to cognitive-science developments including Marr’s levels of analysis and the cognitive revolution.Source: §6 · p. 11
  • It surveys semantic and algorithmic interpretation, comparing their strengths, weaknesses, assumptions, challenges, and possible complementarity.Source: §6 · p. 11
  • The paper aims to provide a shared lexicon of goals, assumptions, and challenges to support discussion and further research toward more rigorous neural network interpretation.Source: §6 · p. 11

A.1 Interpretable Machine Learning

The section distinguishes conventional interpretability definitions from mechanistic interpretability’s focus on internal representations and mechanisms in self-supervised foundation models. It argues that output-focused post-hoc explanations do not capture this internal analysis.

A.1 Interpretable Machine Learning

  • This work focuses on mechanistic interpretability: the study of internal mechanisms and latent representations learned by self-supervised foundation models.It distinguishes this scope from interpretability as inherent transparency and input-output explanations of supervised models [24] [128].Source: §A.1 · pp. 18–19
  • Lipton divides interpretability into transparency, where operations are comprehensible, and post-hoc explanations of specific outputs from opaque models.Examples include decision trees and rule-based systems for transparency, and saliency maps for post-hoc explanation.Source: §A.1 · p. 19
  • Post-hoc explanation alone does not describe probing or circuit discovery, which study foundation models’ internal representations, operations, and algorithms rather than necessarily explaining particular outputs.The section notes that earlier post-hoc categories predate self-supervised foundation models and do not apply to these internal analysis techniques.Source: §A.1 · p. 19
Loading 2408.05859v1…