Transcoders Find Interpretable LLM Feature Circuits

Jacob Dunefsky; Philippe Chlenski; Neel Nanda

Source-linked AI summary

Transcoders Find Interpretable LLM Feature Circuits

Jacob Dunefsky, Philippe Chlenski, Neel Nanda

arXiv:2406.11944v2Jun 17, 2024cs.LGcs.CL

TL;DR

MLP sublayers and dense SAE features make fine-grained, input-general circuit analysis difficult. The paper trains sparse MLP transcoders and introduces weights-based circuit analysis that factorizes connections into input-dependent and input-invariant terms. Transcoders perform on par with SAEs or better on interpretability, sparsity, and faithfulness, and support qualitative circuit case studies.

Problem
MLP sublayers and dense SAE features make fine-grained circuit analysis potentially infeasible and unable to characterize general MLP input-output behavior.
Method
The paper trains wide, sparsely activating MLP transcoders to approximate MLP sublayers and uses them for feature-level, weights-based circuit analysis.
Results
Transcoders are on par with SAEs or better on interpretability, sparsity, and faithfulness, and the method is applied to multiple circuit case studies.
Takeaways & Limitations
Transcoder circuits factorize into input-dependent and input-invariant components, enabling interpretable analysis of general MLP behavior and qualitative unknown-circuit investigations.
Takeaways & Limitations
Transcoders approximate the model and may lose key information; the method treats attention patterns as fixed and is demonstrated on only a few qualitative case studies.

Abstract

from arXiv · show

A key goal in mechanistic interpretability is circuit analysis: finding sparse subgraphs of models corresponding to specific behaviors or capabilities. However, MLP sublayers make fine-grained circuit analysis on transformer-based language models difficult. In particular, interpretable features -- such as those found by sparse autoencoders (SAEs) -- are typically linear combinations of extremely many neurons, each with its own nonlinearity to account for. Circuit analysis in this setting thus either yields intractably large circuits or fails to disentangle local and global behavior. To address this we explore transcoders, which seek to faithfully approximate a densely activating MLP layer with a wider, sparsely-activating MLP layer. We introduce a novel method for using transcoders to perform weights-based circuit analysis through MLP sublayers. The resulting circuits neatly factorize into input-dependent and input-invariant terms. We then successfully train transcoders on language models with 120M, 410M, and 1.4B parameters, and find them to perform at least on par with SAEs in terms of sparsity, faithfulness, and human-interpretability. Finally, we apply transcoders to reverse-engineer unknown circuits in the model, and we obtain novel insights regarding the "greater-than circuit" in GPT2-small. Our results suggest that transcoders can prove effective in decomposing model computations involving MLPs into interpretable circuits. Code is available at https://github.com/jacobdunefsky/transcoder_circuits/.

1 Introduction

Mechanistic interpretability seeks circuits that explain specific model behaviors, but MLPs and dense SAE features make fine-grained, input-general circuit analysis difficult. The paper introduces transcoders and a circuit-analysis method, then evaluates and applies them in case studies.

Circuit analysis aims to identify interpretable subcomputations responsible for specific model behaviors in otherwise opaque language models.
SAE features can combine many neurons and nonlinearities, making feature-to-feature circuit analysis through MLPs potentially infeasible.
SAE-based approximations cannot simultaneously describe an MLP’s input-dependent behavior on one input and its general input-output behavior across inputs.
Transcoders are wide, sparsely activating ReLU MLPs trained to approximate narrower MLP sublayers with an activation-sparsity penalty.
The paper introduces transcoder-based circuit analysis, evaluates faithfulness, sparsity, and interpretability against SAEs, and applies the method to detailed case studies.

2 Transformers preliminaries

Transformer computation maps token-position inputs through sequential layers whose attention and MLP sublayers update hidden states. Because each sublayer adds its output to the hidden state, the residual stream can be decomposed into prior sublayer outputs.

The model first maps input tokens and positions to embeddings, then applies a sequence of layers to update hidden states.
Attention heads produce outputs using preceding source-token states and a destination-token state.
The MLP sublayer produces an output that updates the layer’s hidden state.
Each attention or MLP sublayer adds its output to the current hidden state, enabling additive decomposition into previous sublayer outputs called the residual stream.

3 Transcoders

Transcoders approximate MLP outputs with sparse feature combinations and support scalable circuit analysis. Their feature attributions separate input-dependent activations from input-invariant weight-based connections, enabling greedy computational-path and subgraph construction.

3 Transcoders: Transcoders approximate MLP sublayer outputs with sparse linear combinations of feature decoder vectors, using a wider feature space than the model’s input dimension.
3 Transcoders: Each transcoder feature uses an encoder vector to determine activation strength and a decoder vector to contribute to the output.
3 Transcoders: Training balances MLP-output faithfulness against sparse feature activations through an L1-regularized loss.
3.2 Circuit analysis with transcoders: The circuit-analysis method identifies feature connections across layers and provides information about general MLP input-output behavior unavailable from SAEs.
3.2 Circuit analysis with transcoders: Feature-pair attribution factorizes into an input-dependent earlier-feature activation and an input-invariant decoder–encoder dot product.
3.2 Circuit analysis with transcoders: The algorithm recursively extends and greedily prunes computational paths, then merges top paths into a circuit; attention-mediated contributions can also be analyzed through OV circuits.

T f (l,i)

De-embeddings provide input-invariant information about how vocabulary tokens directly contribute to transcoder features. In GPT2-small, high-scoring tokens for one feature reveal a pattern of primarily Polish and broader European surnames.

A transcoder feature’s de-embedding vector measures each vocabulary token’s direct contribution to that feature’s activation.
The highest-scoring tokens for one GPT2-small feature included “oglu,” “owsky,” “zyk,” “chenko,” and “kowski,” forming a European-surname pattern.

4 Comparison with SAEs

Transcoders were evaluated against SAEs on interpretability, sparsity, and fidelity using matched model activations and standard evaluation procedures. Across GPT2-small, Pythia-410M, and Pythia-1.4B, transcoders achieved comparable or better sparsity-accuracy tradeoffs, though SAE hyperparameter searches were less exhaustive.

Evaluation setup: Transcoders and SAEs differ chiefly in their faithfulness targets: SAEs reconstruct inputs, whereas transcoders approximate MLP outputs.Both use sparse feature representations, but their loss functions measure fidelity to different quantities.
Results: Transcoders were comparable to or better than SAEs across interpretability, sparsity, and fidelity evaluations.The authors report no observed penalty relative to SAEs trained on MLP activations, while noting that alternative SAE hyperparameters might change the comparison.
Evaluation setup: Interpretability was assessed by human ratings of shuffled top-activating examples from 50 transcoder and 50 SAE features.The evaluator did not know whether each feature came from an SAE or transcoder.
Evaluation setup: Sparsity was measured by mean active features per token, while fidelity was evaluated by replacing the model MLP with the trained representation and measuring prediction loss.The evaluations used the same OpenWebText inputs and compared against unmodified and mean-ablated model bounds.
Evaluation setup: Transcoders were trained alongside SAEs on GPT2-small, Pythia-410M, and Pythia-1.4B using varied λ1 values controlling the fidelity-sparsity tradeoff.Transcoders used MLP-in and MLP-out activations, while SAEs were trained on MLP-out activations; the interpretability comparison used MLP-in SAEs.
Results: In all models, transcoders were equal to or better than SAEs on the sparsity-accuracy Pareto frontier.The apparent gap widened for larger models, but compute limits prevented exhaustive SAE hyperparameter sweeps.

5 Circuit analysis case studies

The case studies use transcoder circuits to infer feature semantics without directly inspecting prompts and to analyze the GPT2-small greater-than circuit. They recover interpretable token and contextual patterns, while exposing both sparse computational structure and a feature that harms performance.

5.1 Blind case studies: Nine blind GPT2-small case studies inferred individual transcoder-feature semantics from input-invariant and input-dependent circuit information.Restricted studies prohibited input-dependent MLP0 feature information to reduce prompt leakage.
5.1 Blind case studies: The recovered circuit linked tc8[355] to current-token semicolon features, previous-token years, and surname features from earlier layers.Input-invariant de-embeddings associated upstream features with semicolons, years, Eastern European surnames, and English surnames.
5.1 Blind case studies: Circuit analysis identified tc8[355] as responding to semicolons in parenthetical citations, later confirmed by top-activating examples.The circuit also connected to features associated with citation-related words and patterns.
5.2 Greater-than circuit: In the greater-than task, MLP10 received high attribution from earlier-layer features mediated by attention head 1 in layer 9, corroborating prior analysis.The task asks how the model predicts a year greater than a preceding year in prompts such as “The war lasted from 1737 to 17”.
5.2 Greater-than circuit: For fewer than 24 retained components, the transcoder outperformed the neuronal approach under the probability difference metric.Performance dropped sharply around 24 features because tc10[5315] contributed negatively; removing it was evaluated separately.
5.2 Greater-than circuit: The transcoder did not recover full original-model performance but required only a handful of features to recover most of it, whereas many neurons were needed.The features suggest an approximate computation in which year-range features boost logits for following years.

6 Related work

The paper situates transcoders within mechanistic circuit analysis and the broader use of SAEs for interpretable feature decompositions. It builds on causal, automated, and sparse-probing approaches while targeting MLP-specific analysis.

Circuit analysis: Mechanistic interpretability uses circuit analysis to identify interpretable subcomputations underlying model behaviors.Related approaches include causal methods, automated circuit discovery, and sparse probing.
Sparse autoencoders: Sparse autoencoders disentangle model activations into interpretable features, motivated in part by the theory of superposition.Prior work also developed tools for exploring SAE features and applied SAEs to attention sublayers.

7 Conclusion

The paper presents transcoders as a circuit-analysis tool for MLP sublayers that separates input-invariant from input-dependent information while retaining interpretability, sparsity, and faithfulness comparable to SAEs. It identifies several limitations and directions for future work, including attention-pattern analysis and more systematic case studies.

Conclusion: Transcoder-based circuit analysis disentangles input-invariant information from input-dependent information without sacrificing fidelity or interpretability relative to SAEs.The authors describe this as the only such approach they know that cleanly makes this separation for MLP sublayers.
Conclusion: Transcoders are presented as an improvement over other feature-level interpretability tools for analyzing MLPs, including SAEs on MLP outputs.
Future work: Future work includes comparing transcoder and SAE features, identifying feature classes transcoders struggle to learn, finding novel circuits, and scaling analysis to larger models.
Limitations: The method does not analyze how attention patterns are computed, instead treating them as fixed.The authors suggest extending transcoders to approximate the attention softmax.
Limitations: The circuit-analysis results cover only a few qualitative case studies, so more systematic analysis would strengthen the evidence.

Impact statement

The paper contributes a new circuit-analysis tool as foundational research in mechanistic interpretability. The authors expect its impact to arise indirectly through future applications such as understanding unexpected model behavior and controlling or steering models.

Impact statement: The paper presents transcoders as a new tool for circuit analysis within mechanistic interpretability.
Impact statement: The expected impact is indirect, through future applications such as understanding and debugging unexpected model behavior.
Impact statement: Future applications may also involve controlling and steering models to be more useful to users.

B Compute details

The paper reports GPU-intensive training for SAEs and transcoders, while case studies and activation-example computations used comparatively little compute. The supplied passages also describe the architectures, evaluation measures, and attribution assumptions used in the analysis.

Compute usage: Training the SAEs and transcoders used in the experiments was the most compute-intensive part of the research and involved GPUs.Section 4.2 models used an internal cluster with an A100 GPU containing 80 GB of VRAM; individual runs used approximately 16–60 GB.
Compute usage: GPT2-small transcoders were trained through a cloud provider with similar time and VRAM per run, including a hyperparameter sweep of approximately 200 unused runs.
Compute usage: Datasets were streamed during training, so no significant amount of storage was used.
Compute usage: Case studies used no more than 6 GB of VRAM and were de minimis compared with the training runs.The same comparison applies to computing top activating examples.
Evaluation: SAE evaluation measures sparsity through mean L0 and accuracy through the change in next-token-prediction cross-entropy loss after replacing model activations with reconstructions.
Attribution method: Transcoder attribution replaces an MLP with a sufficiently faithful transcoder, decomposes its output into individual features, and uses linearity to attribute downstream feature activations.

D.3 Attribution through attention heads

The method extends feature-vector attribution through attention heads by treating QK scores as fixed and decomposing OV-mediated contributions into earlier components. Iterative attribution propagation then yields computational paths and graphs while incorporating LayerNorm scaling as an approximation.

Attribution through attention heads: Earlier-layer components can affect attention contributions nonlinearly through both QK and OV circuits, so the analysis holds QK scores fixed.The resulting analysis is informative about OV-mediated information transfer but does not capture transcoder-feature contributions to QK composition.
Attribution through attention heads: Attention-head contributions to later transcoder features are decomposed through the OV circuit while treating QK scores as fixed.This isolates what information an attention head takes from source tokens from which tokens it selects.
Attribution through attention heads: The OV-mediated contribution of a previous component y to a transcoder feature is represented by a feature vector p′, scored as y · p′.The vector can be de-embedded and analyzed using the same techniques as transcoder features.
Recursing on a single computational path: The algorithm recursively propagates feature vectors backward, greedily retaining important computational paths and combining them into an attributed computational graph.Graph construction sums node and edge attributions across distinct paths to prevent double-counting.
Recursing on a single computational path: A propagated feature vector f is chosen to satisfy f · x′ = a′, allowing earlier-node attributions to be computed as f · y.This preserves the current node’s attribution while exploiting linearity in the residual stream.
Recursing on a single computational path: LayerNorm nonlinearities are approximated by scaling each feature vector with an empirically estimated pre- to post-LayerNorm norm ratio.The paper notes this approximation is supported by prior intuition, theory, and empirical results.

E Details on Section 4.2 SAE/transcoder training

This section specifies training and evaluation details for the SAE/transcoder comparison and describes the procedures used to construct interpretability and greater-than-circuit analyses.

SAE/transcoder training: SAEs model MLP output activations, whereas transcoders map post-LayerNorm MLP inputs to MLP outputs for cross-entropy evaluation.The comparison uses output-trained SAEs because both methods replace the original MLP output activations.
SAE/transcoder training: 32× model dimensionality determined the feature count for every SAE and transcoder.This corresponds to 24,576 features for GPT2-small, 32,768 for Pythia-410M, and 65,536 for Pythia-1.4B.
SAE/transcoder training: 60 million OpenWebText tokens were used for training, with batches of 4096 examples and 128-token contexts.Evaluation also used examples of length 128 tokens, and all models shared random seed 42.
Feature interpretation: The compared transcoder and SAE occupied similar L0-cross-entropy Pareto-frontier positions: 44.04 L0 and 3.35 nats versus 47.85 L0 and 3.36 nats.The pair used Pythia-410M layer 15, with the SAE trained on MLP inputs for this activation-example comparison.
Feature interpretation: Figure 6 shows examples of feature dashboards and human annotations spanning interpretable, maybe interpretable, uninterpretable, and context-free categories.Context-free features appeared to fire on one or two tokens regardless of surrounding context.
Greater-than circuit analysis: De-embedding scores for MLP10 features sum token scores across the top ten MLP0 features, weighted by their input-invariant attributions.The procedure first identifies the strongest MLP0-to-MLP10 connections through attention head 1 in layer 9.

H.1.1 Citation feature: tc8[355]

Circuit analysis identified tc8[355] as a likely semicolon-in-citations feature by tracing current- and previous-token contributions through interpretable transcoder features. Additional prompts supported this interpretation, while related case studies illustrate how the same analysis infers token and contextual roles.

11 prompts activated tc8[355] above a score of 11, including the first case-study input (5701, 37).
Current-token features: Top de-embeddings for current-token features tc0[9188] and tc0[16632] were semicolon variants, leading to the conclusion that the final token was a semicolon.
Previous-token features: Input-independent connections implicated surname components, open parentheses, and years among the earlier-token contributions analyzed in the citation context.
Final result: Top-activating prompts supported the final interpretation of tc8[355] as a semicolon-in-citations feature, although comma features were not identified by circuit analysis.