Source-linked AI summary
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models
Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, Aaron Mueller
TL;DR
The paper addresses the difficulty of explaining neural-network behaviors with scalable, interpretable units rather than polysemantic neurons or attention heads. It discovers causal circuits over sparse-autoencoder features, applies them to classifier editing, and scales discovery to thousands of behaviors. The authors report human-interpretable causal graphs across several settings and more effective sparse-feature ablation than a neuron-based method, while relying on trained SAEs and retaining an interpretability boundary around components they do not capture.
Problem
Existing interpretability circuits often use polysemantic, difficult-to-interpret components, while fine-grained methods face unit-selection and scalability challenges.
Method
The paper uses sparse autoencoders and linear causal-effect approximations to discover sparse feature circuits, then labels and edits features within those circuits.
Results
The authors discover human-interpretable causal graphs for subject-verb agreement, a classifier, and thousands of general token-prediction behaviors, with sparse-feature ablation more effective than a neuron-based method.
Takeaways & Limitations
Sparse feature circuits support mechanistic analysis and human-guided removal of features judged spurious in downstream classifier behavior.
Takeaways & Limitations
The method requires trained SAEs, incurs a large one-time training cost, and cannot interpret model components not captured by those SAEs.
Abstract
from arXiv · showhide
We introduce methods for discovering and applying sparse feature circuits. These are causally implicated subnetworks of human-interpretable features for explaining language model behaviors. Circuits identified in prior work consist of polysemantic and difficult-to-interpret units like attention heads or neurons, rendering them unsuitable for many downstream applications. In contrast, sparse feature circuits enable detailed understanding of unanticipated mechanisms. Because they are based on fine-grained units, sparse feature circuits are useful for downstream tasks: We introduce SHIFT, where we improve the generalization of a classifier by ablating features that a human judges to be task-irrelevant. Finally, we demonstrate an entirely unsupervised and scalable interpretability pipeline by discovering thousands of sparse feature circuits for automatically discovered model behaviors.
1 INTRODUCTION
The paper targets scalable, interpretable explanations of neural-network behaviors by replacing coarse or polysemantic units with sparse, human-interpretable feature circuits. It applies these circuits to classifier editing and to an unsupervised pipeline that discovers circuits for thousands of model behaviors.
- Interpretability methods must explain unanticipated neural-network behaviors at scale, but coarse components such as attention heads and MLPs are often polysemantic and difficult to interpret.
- The proposed approach uses sparse autoencoders to identify interpretable latent directions and linear approximations to efficiently find causally implicated features and feature connections.
- Sparse feature circuits represent model behaviors as interactions among fine-grained, human-interpretable units rather than neurons or other coarse components.
- SHIFT removes human-judged unintended features from classifier circuits, including gender when it is perfectly predictive of profession labels in a worst-case setting.
- The authors demonstrate an unsupervised pipeline that automatically discovers thousands of narrow language-model behaviors and their feature circuits.
2 FORMULATION
The formulation treats sparse-autoencoder features and errors as nodes in a model computation graph, then uses activation, gradient, and indirect-effect calculations to identify salient nodes and edges. The method supports contrastive-pair and single-input settings, with scalable linear approximations and a more accurate integrated-gradients alternative.
- Feature disentanglement with sparse autoencoders: Sparse autoencoders decompose activations into sparse feature sums plus an error term, and the method retains SAE errors in circuits rather than discarding them.
- Attributing causal effects with linear approximations: For contrastive inputs, indirect effects measure how setting a node to its patch value changes a metric computed on the clean input.
- Attributing causal effects with linear approximations: Linear attribution approximations enable indirect effects for many nodes in parallel, while integrated gradients provide a more expensive and accurate alternative.
- Attributing causal effects with linear approximations: For single-input settings, the method replaces patching with zero-ablation by measuring the metric change from setting a node to zero.
- Method overview: The pipeline caches activations, computes gradients and approximate indirect effects, then filters nodes and edges using thresholds.
3 SPARSE FEATURE CIRCUIT DISCOVERY
The method represents language-model computation as a graph of SAE features and errors, estimates causal effects efficiently, and evaluates sparse circuits on subject-verb agreement tasks. Feature circuits capture model behavior with far fewer nodes than neuron circuits while exposing interpretable agreement mechanisms.
- Method: The pipeline models SAE feature activations and reconstruction errors as nodes in the language model’s computation graph.Nodes correspond to feature activations or SAE errors at particular token positions.
- Method: Linear approximations estimate node and edge indirect effects across many components, then retain nodes and edges exceeding chosen thresholds.Integrated gradients improve accuracy over attribution patching for layer 0 MLP and early residual-stream components.
- Evaluation: The evaluation uses contrastive subject-verb agreement datasets whose paired inputs differ only in the grammatical number of the subject.Circuits are compared with neuron circuits for interpretability, faithfulness, and completeness on held-out data.
- Evaluation: The majority of performance is explained by only 100 Pythia-70M nodes or 500 Gemma-2-2B nodes, versus about 1500 or 50000 neurons for half the performance.Removing residual-stream SAE error nodes severely disrupts performance, while removing MLP and attention error nodes is less disruptive.
- Case study: A relative-clause case study identifies two agreement pathways: direct number tracking and number transport to clause boundaries.Gemma 2 additionally uses noun-phrase number trackers, and the circuit overlaps substantially with the prepositional-phrase agreement circuit.
4 APPLICATION: REMOVING UNINTENDED SIGNALS FROM A CLASSIFIER
SHIFT uses human inspection of sparse feature circuits to remove task-irrelevant signals from classifiers without disambiguating labels. In the Bias in Bios worst-case setting, it largely removes gender dependence while preserving or restoring profession performance through retraining.
- SHIFT: SHIFT has humans inspect a classifier’s feature circuit, ablate features judged task-irrelevant, and optionally retrain the resulting classifier.The procedure computes a circuit, evaluates feature relevance, removes selected features, and optionally fine-tunes on the original data.
- Experimental setup: The Bias in Bios evaluation uses ambiguous biographies where gender perfectly predicts training labels, then tests profession classification on balanced data.The balanced set contains equal combinations of gender and profession, separating the intended and unintended signals.
- Results: SHIFT almost completely removes both classifiers’ dependence on gender information.The reported result applies to both Pythia and Gemma models.
- Results: For Gemma but not Pythia, feature ablations damage performance; further training restores performance without reintroducing bias.Retraining is therefore used as an optional recovery step after ablation.
- Caveat: SHIFT depends on properly selected SAE features: random feature ablations have essentially no effect, while neuron-based alternatives are difficult to interpret and underperform.The neuron skyline also fails to match SHIFT because implicated neurons can support both spurious and ground-truth classification.
5 UNSUPERVISED CIRCUIT DISCOVERY AT SCALE
The paper builds a near-fully-automated pipeline that clusters corpus examples into model behaviors and discovers feature circuits for the resulting subcorpora. Circuit analysis exposes multiple mechanisms within apparently single behaviors, including induction, succession, and distinct uses of “to.”
- Pipeline: The pipeline starts from a large text corpus and aims to produce thousands of feature circuits for automatically discovered language-model behaviors.This replaces human-collected behavior datasets with an automated discovery process.
- Behavior discovery: Unsupervised clustering of context–next-token representations segments the corpus into subcorpora that often capture interpretable behaviors.Examples include predicting the next number in a sequence.
- Caveat: Evaluating the discovered clusters and circuits remains an important open problem, although the authors generally find them useful for understanding language-model mechanisms.The interface provides access to unlabeled clusters and circuits.
- Circuit discovery: Automatically discovered circuits include attention features implicated in succession and induction.These behaviors had previously been studied at the attention-head level using human-curated data.
- Circuit-level analysis: Circuit analysis reveals that clusters can combine distinct mechanisms rather than representing one mechanism.Cluster 475 uses two manners of predicting “to” as an object, while cluster 382 combines succession and narrow induction features.
6 RELATED WORK
The related work situates sparse feature circuits within causal interpretability, robustness to spurious correlations, and feature disentanglement. These areas provide the methodological and application context for the paper’s approach.
- Causal interpretability: Prior causal-interpretability work uses counterfactual interventions such as activation patching or path patching to study model mechanisms.The cited literature commonly analyzes coarse-grained model components.
- Robustness to spurious correlations: Robustness research includes worst-group optimization, data reweighting, and related methods for mitigating spurious correlations.These approaches form the broader context for removing unintended classifier signals.
- Feature disentanglement: Sparse autoencoders and earlier representation-learning methods belong to a broader literature on feature disentanglement.The paper positions its sparse-feature analysis alongside these approaches.
7 CONCLUSION
The paper discovers human-interpretable causal graphs for agreement, classification, and thousands of token-prediction behaviors, and edits model access by ablating features judged spurious. Feature ablation is reported as significantly more effective than a neuron-based method with an unfair advantage.
- Conclusion: The method discovers human-interpretable causal graphs for subject–verb agreement, a classifier, and thousands of general token-prediction tasks.These graphs are built from sparse features and represent causal model behavior.
- Conclusion: Humans can edit model behavior by ablating sparse features judged to be spurious.The conclusion frames feature access as an editable part of the model’s computation.
- Conclusion: Feature ablation is significantly more effective than a neuron-based ablation method that had an unfair advantage.This comparison is reported as part of the classifier-editing evaluation.
8 LIMITATIONS
The method depends on pretrained SAEs and cannot interpret model components those SAEs fail to capture. Its evaluation is also substantially qualitative, including feature labeling.
- The method requires access to SAEs for the model, whose training imposes a large one-time upfront compute cost.
- Components not captured by the SAEs remain uninterpretable after applying the method.
- Much of the evaluation is qualitative, making dictionary and circuit assessment without downstream tasks challenging.
- Feature labels may vary across annotators and depending on the task of interest.
REPRODUCIBILITY
The pipeline represents SAE features and errors as computation-graph nodes, estimates causal effects with gradient-based approximations, and aggregates effects according to data structure. SAE training uses sparse reconstruction objectives and practical procedures for efficient, stable optimization.
- Effect aggregation: For templatic data, effects are averaged across examples while preserving token positions as distinct feature nodes; for non-templatic data, effects are summed across positions before example-wise averaging.
- Efficient attribution: Linear approximations estimate effects for many components in parallel, reducing computation to two backward passes in the described Jacobian-vector-product procedure.
- SAE representation: SAEs reconstruct activations as sparse feature sums plus an error term, using feature activations computed by a ReLU encoder and reconstructions formed from decoder features.
- SAE training: SAE training minimizes reconstruction and sparsity losses on activation samples from The Pile, with dead-feature reinitialization, learning-rate warmup, and extended optimization.
B.1.3 EVALUATION
The evaluation reports quantitative SAE-quality metrics across submodules and layers, while noting that these metrics omit important qualitative properties such as feature interpretability. Gemma Scope SAEs introduce an additional BOS-feature handling boundary.
- Metrics: SAE quality is assessed using feature sparsity, feature activity, and cross-entropy reconstruction metrics.
- Metrics: Cross-entropy difference compares the model’s original loss with its loss after replacing an activation by its SAE reconstruction.
- Reported evaluations: The reported metrics are shown in the embedding, attention-by-layer, and MLP-by-layer evaluation tables.
- Gemma-2-2B SAEs: For Gemma-2-2B, the evaluation uses width-16384 Gemma Scope SAEs across attention, MLP, and residual vectors.
- Gemma-2-2B SAEs: BOS features can have high indirect effects but are excluded from annotation, SHIFT analysis, and the feature skyline because they are difficult to interpret.
C.1 SUBJECT-VERB AGREEMENT
The discovered agreement circuits reuse a layered pattern: early features detect subject number, while later features route or promote information predictive of the correct verb inflection. Across syntactic structures and models, the circuits also reveal shared and task-specific mechanisms.
- Circuit visualization: Circuit diagrams distinguish sparse features from SAE error terms and use node shading to indicate causal effect strength.Blue nodes and edges have positive indirect effects, whereas red nodes and edges are counterproductive for the target metric.
- Agreement across relative clauses: Pythia and Gemma circuits for agreement across relative clauses detect subject number, identify the distractor clause, and move subject information to the final position.Gemma additionally uses NP number trackers that remain active across noun-phrase positions while excluding opposite-number distractors.
- Agreement across distractor phrases: The prepositional-phrase and relative-clause circuits share over 85% of their features across both Pythia and Gemma.Many of the same features detect prepositions and relative clauses.
- Agreement circuit structure: Agreement circuits detect the subject’s grammatical number early, then use later features to predict the appropriate verb inflection.This pattern appears in simple agreement and agreement within relative clauses.
- Automatically discovered behaviors: The circuit for incrementing number sequences combines general succession features, token-specific induction features, and successor features that increment particular numbers.A strongly activating layer-5 residual feature increments “3” to “4” in induction-like lists.
- Automatically discovered behaviors: The infinitival-object circuit contains two mechanisms that influence the layer-5 output without fully intersecting.One responds to verbs, participles, or predicate adjectives; the other responds to nominal direct objects preceding infinitival complements.
D SAMPLE FEATURES
Sparse features in the Bias in Bios circuit are more interpretable than dense neurons, and their activation patterns support separating task-relevant profession signals from gender-related spurious signals.
- Sparse features: Sparse features with high indirect effects detect either profession-related terms, such as medicine and academia, or gender-related terms, such as “he” and female names.The distinction supports judging which features relate to the target profession classification task versus an unintended correlate.
- Neurons: Dense neurons with high indirect effects are difficult to interpret because their activation patterns do not reveal whether they support gender or profession prediction.The examples include neurons responding to sentence boundaries, capitals, or numbers, and neurons with mixed positive and negative activations.
E.1 CLASSIFIER TRAINING
The classifier-training and evaluation setup trains linear heads on pooled model activations, selects layers using held-out performance, and compares interpretability across sparse and dense representations. Human ratings favor sparse features, but annotator selection and semantic-labeling bias constrain that evidence.
- Classifier training: The Bias in Bios classifiers use mean-pooled residual-stream activations and logistic-regression heads trained with AdamW for one epoch.Baseline activations and labels come from the ambiguous set, while the oracle classifier uses the balanced set.
- Classifier training: Layer selection uses ambiguous-set test accuracy for the baseline, with Pythia using layer 4 and Gemma using layer 22 to expose worst generalization.The Gemma choice uses the balanced set only for this layer-selection decision.
- SHIFT evaluation: After SHIFT, only the linear classification head is retrained rather than the full language model.This isolates the classifier update from changes to the underlying model.
- Interpretability evaluation: Crowdworkers rate sparse features as significantly more interpretable than neurons, and circuit-selected features as more interpretable than randomly sampled features.The evaluation compares random and circuit-selected features and neurons on a 0–100 interpretability scale.
- Interpretability evaluation: The human evaluation interface presents top activations, ablation-affected token probabilities, example contexts, and separate interpretability and semantic-complexity ratings.Ratings are assigned on 0–100 scales.
- Evaluation caveats: The annotator pool came from an AI-alignment research Slack channel, and the authors note that this selection likely affects rating variance.Annotators also commonly emphasized semantic groupings instead of syntactic context.
- Causal-effect estimation: Integrated-gradients-based indirect-effect approximation improves accuracy over attribution patching for layer-0 MLP and early residual-stream components.Attribution patching generally performs well but underestimates true indirect effects in those components.