A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models

1 Introduction

• MI reverse-engineers model computations into human-understandable mechanisms, but its scalability, generalizability, and practical use for AI safety remain concerns.MI has been used to study component functions and explain model behaviors.Source: §1 · p. 5
• The survey organizes MI around research tasks and a Beginner’s Roadmap with actionable workflows to guide readers from problem formulation toward practical implementation.It discusses technique trade-offs and illustrates workflows through prior-work case studies.Source: §1 · p. 5
• The paper covers MI objects, techniques and evaluations, roadmap case studies, findings and applications, and current challenges and future directions.Its roadmap is organized around feature study, circuit study, and universality.Source: §1 · p. 5

2 Related Work

• Unlike technique-centric surveys, this survey organizes MI around practical objectives and workflows, linking research goals to technique selection and case studies.The three core tasks are feature study, circuit study, and universality study.Source: §2 · p. 6
• The survey details MI techniques, their limitations and advances, and their use in workflows, while some related surveys omit technical introductions or focus only on SAEs.Sharkey et al. (2025) and Dang et al. (2024) do not cover technical introductions; Shu et al. (2025) focuses on SAEs.Source: §2 · pp. 6–7
• It also surveys LM findings and practical applications, including knowledge editing, steering, and AI safety, which some related surveys address less extensively.Zhao et al. (2024) and Dang et al. (2024) primarily focus on explanation techniques; Ferrando et al. (2024) addresses practical applications to a lesser extent.Source: §2 · p. 7
• The survey concludes with discussion of MI’s current challenges and directions for future research.It presents these alongside broad technique and findings coverage.Source: §2 · p. 7

3 Background: Transformer-based LMs

• A transformer-based LM maps input tokens to a probability distribution over the vocabulary for predicting the next token, refining token representations layer by layer.The initial representation is obtained from an embedding lookup, with position embeddings and layer normalization omitted in this background account.Source: §3 · p. 7
• Each layer updates a token’s representation by adding the multi-head attention output and feed-forward output to its prior representation.The sequence of layer representations is called the residual stream.Source: §3 · p. 7
• Multi-head attention combines head outputs and projects them to the model dimension, while the feed-forward sublayer applies two linear transformations with a nonlinear function between them.The final-layer representation for the last input token is unembedded and passed through softmax to produce vocabulary probabilities.Source: §3 · p. 8

4 What is Mechanistic Interpretability?

• MI takes a bottom-up approach, decomposing models into components and elementary computations to build an understanding of the full model.The survey uses three objects of study to structure its account.Source: §4 · p. 8

4.1 Fundamental Objects of Study in MI

• The survey follows a three-part taxonomy of feature study, circuit study, and universality study.These are presented as the fundamental objects of study in MI.Source: §4.1 · p. 8

4.1.1 Features

• A feature is a human-interpretable input property encoded in LM activations, which MI seeks to decode from model representations.The paper’s example includes properties such as “animal” and “pet” encoded when processing “dog.”Source: §4.1.1 · p. 8

4.1.2 Circuits

• Circuits are computational pathways connecting features or transformer components that implement specific LM behaviors.An induction circuit example links a previous-token head to an induction head to detect and continue repeated subsequences [Elhage et al. (2021)].Source: §4.1.2 · pp. 8–9, Figure 2 · p. 9

4.1.3 Universality

• Universality asks whether similar features or circuits form across different LMs or tasks, affecting whether findings from toy or small models may transfer.If they are not universal, interpreting each model and task requires independent effort.Source: §4.1.3 · p. 9

4.2 An Overview of Beginner’s Roadmap to MI

• The roadmap gives feature, circuit, and universality studies step-by-step workflows, with a final evaluation step for all three categories.Feature study distinguishes targeted hypothesis generation and validation from open-ended observation and explanation; circuit study separates behavior and component interpretation.Source: §4.2 · pp. 9–10, Figure 3 · p. 10

4.3 How is MI Different From Other Sub-fields of Interpretability?

• The survey adopts a broad definition of MI as any work seeking to describe model internals, while noting blurred boundaries with other interpretability approaches.It urges greater integration of related non-MI research that studies similar topics.Source: §4.3 · pp. 10–11

• The section reviews techniques for studying MI’s fundamental objects, presenting each technique’s basic concepts, interpretation, and recent advancements.Source: §5 · p. 11

5.1.1 Basic Concepts

• Logit lens projects residual-stream activations into vocabulary space, extending the final-layer projection used for next-token prediction to intermediate layers.Source: §5.1.1 · p. 11
• Vocabulary projection applies the unembedding matrix to intermediate representations to produce logit distributions whose highest-scoring tokens indicate candidate next tokens.Source: §5.1.1 · p. 11
• The logit lens can also project attention-head and feed-forward sublayer outputs before they enter the residual stream to interpret their contributions to next-token prediction.Source: §5.1.1 · p. 11

Technical Advancements

• Table 2 summarizes the reviewed techniques’ basic concepts, interpretation approaches, and technical advancements.Source: Table 2 · p. 12

5.1.2 Technique-specific Interpretation

• Interpretation commonly relies on humans examining top tokens, while LM-generated explanations and classifications can automate evaluation; these explanations are correlational, not causal.Source: §5.1.2 · p. 13

5.1.3 Technical Advancements

• Vocabulary-projection advancements target three dimensions: reliability, decoding position, and expressivity beyond immediate next-token predictions.Source: §5.1.3 · p. 13
• Tuned-lens methods address unreliable early-layer projections by translating intermediate activations toward the final-layer representation before vocabulary projection.For BLOOM, the top-1 projected token in more than half of earlier layers is often the input token rather than a plausible continuation; translator losses differ across the proposed approaches.Source: §5.1.3 · p. 13
• Vocabulary projection has been applied to activations, model weights, and gradients, including individual attention-head outputs and parameter interactions.The backward lens projects FF-sub-layer gradients during training to investigate how new information is stored.Source: §5.1.3 · p. 14
• Extensions decode beyond immediate next-token predictions, including future tokens, relation-conditioned objects, and information revealed through spectral filtering.Logit Spectrology uses singular value decomposition and filtering to expose information overlooked by the standard logit lens.Source: §5.1.3 · p. 15

5.2.1 Basic Concepts

• Intervention methods treat transformer components as nodes in a computational or causal graph and alter intermediate representations to observe changes in model output.Components can be defined at different granularities, from a neuron or attention head to an entire layer.Source: §5.2.1 · p. 15
• Noising tests whether a component is necessary by replacing its contribution and observing whether the behavior changes, whereas denoising tests whether it can restore the behavior.Denoising first corrupts all components, then restores one component and measures how much it recovers the target prediction.Source: §5.2.1 · p. 15
• The usual procedure caches activations in clean and corrupt runs, then patches a component in a third run to remove or restore its contribution.A corrupt run may use a different prompt or, in some cases, continuous noise or a zero activation.Source: §5.2.1 · p. 16

5.2.2 Technique-specific Interpretation

• Logit difference is generally recommended because it can control for components that promote both candidate outputs; KL divergence instead compares full output distributions.Absolute probability may miss components that suppress the correct output, while KL divergence is a reasonable alternative for assessing patch effects.Source: §5.2.2 · p. 17

5.2.3 Technical Advancements

• Mean ablation can still create off-distribution activations when the activation distribution is nonlinear, while resampling uses a counterfactual activation from the same distribution.Zero and random-noise ablations can also be unreliable because their replacement values are arbitrary.Source: §5.2.3 · p. 17
• Localization ranges from node patching to edge-level path patching and rotated-subspace interventions, with DAS learning rotations and subspaces aligned to causal variables.Path patching restricts a component’s patched effect to paths feeding a selected component; DII intervenes on subspaces rather than raw neurons.Source: §5.2.3 · pp. 17, 19, 20, Figure 5 · p. 19
• ACDC iteratively removes low-impact edges, while EAP estimates edge importance with two forward passes and one backward pass; EAP-IG addresses zero-gradient concerns.ACDC requires independent inferences at each iteration and is described as not scalable to large models.Source: §5.2.3 · p. 20
• Causal scrubbing and interchange interventions test hypotheses about component roles, while Patchscope and related methods combine intervention with vocabulary projection to decode activations.Patchscope uses source and target prompts and models, with an inspection prompt designed to elicit the feature represented by a patched activation.Source: §5.2.3 · pp. 20–21

5.3.1 Basic Concepts

• SAEs map a d-dimensional activation into a larger sparse representation intended to make its features more interpretable and address superposition.The encoder produces sparse activations and the decoder reconstructs the original input activation.Source: §5.3.1 · p. 21
• SAE training balances reconstruction fidelity against sparsity, using an L1 coefficient to encourage a small number of active features.The decoder reconstructs the input from the encoder’s sparse activations, and its learned feature vectors are stored in the decoder weights.Source: §5.3.1 · pp. 21–22

5.3.2 Technique-specific Interpretation

• Low reconstruction loss indicates fidelity, but feature interpretability remains an open evaluation challenge, and increased sparsity does not guarantee interpretable features.Feature quality has been assessed by human raters or language models prompted to evaluate interpretability.Source: §5.3.2 · p. 22

5.3.3 Technical Advancements

• TopK, JumpReLU, and Gated SAEs address trade-offs in sparsity and reconstruction, including L1-related feature suppression; Matryoshka SAEs target feature splitting, absorption, and composition.Matryoshka SAEs train nested sub-SAEs of increasing dictionary sizes to represent features at multiple abstraction levels.Source: §5.3.3 · p. 23
• End-to-end SAEs optimize KL divergence between original and SAE-mediated output distributions to prioritize features that influence predictions over features that only reconstruct activations.This objective is presented as an approach to discovering functionally important features.Source: §5.3.3 · p. 23

5.4 Others

• Probing and visualization are presented as two additional techniques with a history in interpretable AI and machine learning.The survey introduces them as tools for interpreting language models.Source: §5.4 · p. 23

5.4.1 Probing

• Probing trains a shallow or linear classifier to predict whether a predefined feature is present in intermediate activations.Probe results indicate correlation between a feature and activations, not a causal relation.Source: §5.4.1 · p. 24

5.4.2 Visualization

• Visualization supports hypothesis generation, refinement, qualitative analysis, and validation, including inspection of attention patterns or neuron activations across text.The survey cautions that interpretation requires human effort and risks overgeneralization, so claims need further experimentation and analysis.Source: §5.4.2 · p. 24

6 A Beginner’s Roadmap to MI

• Feature study, circuit study, and universality form the roadmap’s three categories, each divided into subcategories with associated workflows and techniques.The categories correspond to the survey’s three fundamental objects of study.Source: §6 · pp. 24–25

6.1.1 General Workflow for Targeted Feature Study

• Targeted studies test whether a human-predefined feature is encoded; open-ended studies seek features beyond those human hypotheses.Targeted verification can use probes, vocabulary projection, or PatchScope; DII intervenes on a proposed causal subspace, while DAS learns a suitable subspace, and both require clean-counterfactual training pairs.Source: §6.1 · p. 25, §6.1.1 · p. 26

6.1.2 General Workflow for Open-ended Feature Study

• Open-ended study alternates between observing intermediate explanations and interpreting them, but vocabulary projection reveals only features used for next-token prediction.Neuron visualizations assume neurons correspond one-to-one with features, an assumption challenged by polysemantic neurons; SAEs aim for more monosemantic representations but require non-trivial computation and face a sparsity–reconstruction trade-off.Source: §6.1.2 · pp. 26–27

6.1.3 Example Interfaces for Making Observations in Open-Ended Feature Study

• Elhage et al.’s interface combines activation examples and vocabulary projections to support a human’s base-64 feature interpretation.Neuronpedia offers an open-source, general-purpose dashboard with multiple intermediate explanations; its auto-interpretations can assist annotation but may bias evaluators toward LM-generated explanations.Source: §6.1.3 · pp. 28–29, §6.1.2 · p. 28

6.1.4 Evaluation of Feature Study

• Feature-study evaluation separates faithfulness—whether a feature exists in the representation—from interpretability—how readily people understand its description.Faithfulness proxies include probe accuracy and SAE reconstruction loss; activation steering can measure a feature’s direct effect on next-token distributions, while interpretability is assessed by humans or LM-generated explanation scores.Source: §6.1.4 · p. 31

6.2 Circuit Study

• Behavior-focused circuit studies locate components responsible for a behavior and explain their roles; component-focused studies seek interpretations consistent across tasks.Behavior studies should target behaviors the model performs reliably, while comprehensive component interpretation may require analysis across multiple tasks.Source: §6.2 · p. 32

6.2.1 General Workflow for Circuit Study

• Circuit discovery selects a well-performing behavior and dataset, defines a computational graph, localizes nodes and edges, interprets components, and evaluates the circuit.Graph abstractions vary by whether components at different positions are merged, by node granularity, and by which connections count as edges.Source: §6.2.1 · pp. 32–33, Figure 14 · p. 34
• Interventions such as zero or mean ablation identify important nodes, while path patching and automated methods such as ACDC and EAP localize edges.Testing edges individually can be computationally intensive.Source: §6.2.1 · p. 34
• Interpretation generates and tests hypotheses about component functions, while circuit evaluation checks faithfulness, minimality, and completeness.Faithfulness compares the full model with the circuit alone; minimality tests whether circuit parts are necessary, and completeness tests whether relevant components are included.Source: §6.2.1 · pp. 34–35

6.3 Study of Universality

• Universality concerns similarity of features across models and whether similar circuits implement a behavior across models or tasks.Circuit universality is meaningful when the behavior under study is consistently observed across the models or tasks being compared.Source: §6.3 · p. 35, §6.3.1 · p. 35

6.3.1 General Workflow for Universality Study

• Universality studies vary models by size, training data, initialization, or architecture, or vary tasks and their settings.Feature similarity can be measured by correlating activation patterns on a shared dataset; circuit similarity is assessed by comparing functionally equivalent components implementing the same algorithm.Source: §6.3.1 · pp. 35–36
• Only 1–5% of neurons were universal across five GPT2-Small and GPT2-Medium models in Gurnee et al. (2023).The survey frames universality as a degree, not a binary outcome.Source: §6.3.1 · p. 36

7.1.1 Targeted Feature Study with Probing

• Gurnee et al. (2023) used sparse probes to test 100 predefined features across seven language models and localize features to neurons.They evaluated probes with F1, precision, and recall; high precision with low recall can indicate that the selected neurons encode a narrower feature.Source: §7.1.1 · pp. 36–37

7.1.2 Open-ended Feature Study with SAEs

• Bricken et al. (2023) used SAEs, activation and logit observations, and human and automatic explanations to study features in a one-layer toy model.Human evaluators used a 0–3 confidence rubric when labeling features.Source: §7.1.2 · p. 37

7.2 Case Study for Circuit Study

• Wang et al. (2022a) studied GPT-2 Small’s indirect object identification behavior using a synthesized dataset and a position-dependent graph of attention heads.The analysis excluded feed-forward layers, LayerNorms, and embedding matrices, and used path patching to localize important nodes and edges.Source: §7.2 · p. 37
• The circuit interpretation hypothesized that a head attending to the indirect object copied the attended name, then tested that account through attention and OV-matrix analyses.Attention to the indirect-object token correlated with increased logit output toward the correct name (ρ > 0.81).Source: §7.2 · pp. 37–38
• Copy score exceeded 95% for Name Mover Heads, compared with less than 20% for average heads, supporting their role in transferring name information.In GPT-2 Small’s indirect object identification circuit, 26 attention heads were localized; Name Mover Heads were examined for attending to and copying the correct name.Source: §7.2 · pp. 37–38

8.1 Feature Study

• LM neurons can respond to multiple unrelated features, motivating superposition accounts and sparse autoencoders, whose extracted features remain limited by reconstruction, coverage, and functional-use concerns.Feature analyses also rely heavily on the linear representation hypothesis, although reported multidimensional and nonlinear representations leave its scope unsettled.Source: §8.1 · pp. 39–40

8.2.1 Interpreting LM Behaviors

• Circuit studies identify circuits across model sizes and show component reuse across tasks, but discovery can generalize poorly to adversarial examples and scale remains a concern.Different algorithms may operate in tandem on the same task, with their interactions sensitive to training and architectural hyperparameters.Source: §8.2.1 · p. 40

8.2.2 Interpreting LM components

• Attention heads move information between tokens and exhibit specialized roles, while feed-forward sublayers support feature extraction, knowledge retrieval, and arithmetic computation.Earlier feed-forward layers tend to process shallow patterns and later layers more semantic ones; LayerNorm’s inference-time contributions remain unsettled despite evidence of additional functions.Source: §8.2.2 · pp. 41–42

8.3 Universality

• Feature universality remains largely unexplored, while SAE studies find similar features across models; circuit studies report reuse across tasks but also divergent circuits across initializations.The evidence therefore remains mixed on whether similar circuits reliably form across models and tasks.Source: §8.3.1 · p. 42, §8.3.2 · p. 42

8.4.1 In-Context Learning (ICL)

• In-context learning studies trace adaptation to induction-head mechanisms and task vectors derived from demonstrations, while reasoning studies identify information-transfer, computation, and parallel pathways.Findings include attention heads transferring operands and operators to the answer position before feed-forward modules calculate the answer, and multiple pathways computing answers in some multi-step tasks.Source: §8.4.1 · pp. 42–43, §8.4.2 · p. 43

8.4.3 Knowledge Mechanisms

• Knowledge studies locate storage across feed-forward and attention sublayers and describe multistep factual recall; other circuit work explains task algorithms and relates circuit formation to training phase changes.Some studies find that distinct factual-recall mechanisms combine constructively, while studies of grokking and other phase changes report correlations with developing features or circuits.Source: §8.4.3 · p. 43, §8.4.4 · p. 44, §8.5.1 · p. 44

8.5.2 Effects of Post-training

• Mechanistic findings have informed knowledge editing, generation steering, safety analysis, and downstream applications, while post-training studies suggest existing mechanisms can be modified or bypassed rather than erased.Reported applications include editing localized factual-knowledge components, monitoring safety-related latent directions, improving classifier generalization by removing spurious features, and self-supervised early-exit prediction.Source: §8.5.2 · p. 44, §8.6.1 · pp. 44–45, §8.6.2 · p. 45

8.7.1 Benchmarks for Feature Study Techniques

• Feature benchmarks report strong targeted-feature performance for DAS and strong overall SAE performance for Matryoshka SAEs, while circuit-localization benchmarks report strong results for ACDC and EAP-IG.MIB reports that SAE features are no more effective than individual neurons for open-ended feature isolation; its circuit evaluations also find EAP-IG outperforms edge patching, while synthetic benchmark fidelity remains a concern.Source: §8.7.1 · p. 45, §8.7.2 · p. 46

9.1 Advancement in MI Techniques

• Current MI techniques face scalability and generality limits from computational demands, human-dependent interpretation, and technique-specific weaknesses such as out-of-distribution interventions and correlational probes.The authors suggest more automated and efficient methods, combining complementary techniques, and addressing limitations of existing approaches.Source: §9.1 · p. 46

9.2 Practical Utility of MI Studies

• Most MI applications use toy tasks or simple behaviors, and most studies examine smaller LMs; the authors call for stronger emphasis on practical utility.Some studies demonstrate applications in safety, machine unlearning, knowledge editing, and scientific discovery, but these examples do not change the broader pattern described.Source: §9.2 · pp. 46–47

9.3 Standardized Benchmarks and Metrics

• Interpretability evaluation lacks ground truth and often relies on ad hoc approaches, while synthetic-model benchmarks such as Tracr may not generalize to naturally trained transformers.The authors call for standardized evaluation techniques and wider adoption.Source: §9.3 · p. 47

9.4 Automated Hypothesis Generation in MI Practices

• Formulating hypotheses about LM mechanisms can require laborious manual analysis of activation patterns or intervention results, creating a scalability bottleneck.The authors propose automated hypothesis generation, potentially with human oversight.Source: §9.4 · p. 47

9.5 Generalizing Beyond MI for More Intuitive Applications

• The authors argue that MI should extend beyond low-level features toward higher-level propositions, which can more directly represent a model’s safety stance.Binding vectors are described as a mechanism for encoding associations between entities and relations in activations.Source: §9.5 · p. 47

9.6 A New Paradigm of Human-AI Collaboration Driven by MI

• Feature-attribution explanations have often failed to improve human-AI team performance, while MI-based collaboration remains underexplored and tested in limited domains and benchmarks.SHIFT is presented as an example that removes spurious features with human evaluators and improves classifier generalization.Source: §9.6 · p. 48