PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse Autoencoders Find Highly Interpretable Features in Language Models

Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, Lee Sharkey

arXiv:2309.08600v3cs.LGcs.CL

TL;DR

Polysemanticity and superposition make it difficult to identify concise, human-understandable features inside language models. The paper trains sparse autoencoders on internal activations to learn feature dictionaries, finding features that are more interpretable and that localize indirect-object-identification behavior more finely than prior decompositions. The method is scalable and unsupervised, but its reconstructions do not capture all activation information.

  • Problem

    Polysemantic neurons and superposition complicate the identification of concise, human-understandable units for analyzing neural-network computations.

  • Method

    The paper trains sparse autoencoders on language-model internal activations to learn sparsely activating, overcomplete feature dictionaries.

  • Results

    The learned features are more interpretable than comparable decompositions and require fewer patches or smaller edit magnitudes to reach the same indirect-object-identification model-editing effect as PCA.

  • Takeaways & Limitations

    Sparse autoencoders provide a scalable, unsupervised approach for disentangling language-model features and may support circuit mapping, targeted editing, and representation analysis.

  • Takeaways & Limitations

    The dictionaries do not capture all activation information: replacing layer-2 Pythia-70M residual activations with reconstructions raises Pile perplexity from 25 to 40.

Abstract

from arXiv · show

One of the roadblocks to a better understanding of neural networks' internals is \textit{polysemanticity}, where neurons appear to activate in multiple, semantically distinct contexts. Polysemanticity prevents us from identifying concise, human-understandable explanations for what neural networks are doing internally. One hypothesised cause of polysemanticity is \textit{superposition}, where neural networks represent more features than they have neurons by assigning features to an overcomplete set of directions in activation space, rather than to individual neurons. Here, we attempt to identify those directions, using sparse autoencoders to reconstruct the internal activations of a language model. These autoencoders learn sets of sparsely activating features that are more interpretable and monosemantic than directions identified by alternative approaches, where interpretability is measured by automated methods. Moreover, we show that with our learned set of features, we can pinpoint the features that are causally responsible for counterfactual behaviour on the indirect object identification task \citep{wang2022interpretability} to a finer degree than previous decompositions. This work indicates that it is possible to resolve superposition in language models using a scalable, unsupervised method. Our method may serve as a foundation for future mechanistic interpretability work, which we hope will enable greater model transparency and steerability.

1 INTRODUCTION

Mechanistic interpretability seeks to understand and modify neural-network computations, but polysemantic neurons and superposition make individual-neuron explanations inadequate. The paper uses sparse autoencoders to recover more interpretable feature directions from language-model activations and evaluates them against alternative decompositions.

  • Mechanistic interpretability aims to reverse engineer neural-network calculations so their internal processes can be understood and targeted.
  • Polysemantic neurons activate for several unrelated feature types, making them poor standalone units for analyzing network computations.
  • Superposition hypothesizes that models represent more distinct features than a layer has dimensions, using an overcomplete set of non-orthogonal directions.
  • The paper trains sparse autoencoders on sampled residual-stream, MLP, or attention-head activations, then interprets the learned feature dictionary with automated methods.
  • Sparse dictionary learning can recover features by representing each activation vector as a sparse linear combination of directions.
  • The learned features are evaluated for interpretability, causal localization, monosemanticity, predictable output effects, and fine-grained circuit detection.

2 TAKING FEATURES OUT OF SUPERPOSITION WITH SPARSE DICTIONARY LEARNING

The method treats language-model activations as sparse mixtures of unknown network features and trains an overcomplete sparse autoencoder to learn a corresponding dictionary. Its normalized weights define feature directions, while sparse hidden coefficients reconstruct the input under a reconstruction-plus-sparsity objective.

  • The method models internal activation vectors as sparse linear combinations of unknown ground-truth feature vectors and seeks approximate dictionary features for them.
  • The sparse autoencoder uses a single ReLU hidden layer with size dhid = Rdin, making the dictionary overcomplete relative to the activation dimension.
  • The autoencoder uses tied encoder and decoder weights, so the learned parameter matrix forms a feature dictionary whose rows are dictionary features.
  • The hidden coefficients reconstruct each input activation, while row-wise normalization prevents enlarging feature vectors from reducing the sparsity penalty.
  • Training minimizes reconstruction loss plus an ℓ1 penalty on hidden activations, encouraging reconstructions to use sparse combinations of dictionary features.

3 INTERPRETING DICTIONARY FEATURES

The paper measures feature interpretability with automated descriptions and compares sparse-autoencoder dictionaries with several baselines. The learned features score higher on average, although the advantage declines in later layers and automatic interpretation has important limitations.

  • Autointerpretability evaluates dictionary features at scale by using activating text samples to generate descriptions and measuring how well those descriptions predict other activations.
  • The comparison includes the default basis, random directions, PCA, and ICA, with nonnegative activations enforced for random directions and the residual-stream default basis.
  • Figure 2 reports average top-and-random autointerpretability scores for 150 learned and baseline features, with 95% confidence intervals.
  • Sparse-autoencoder dictionary features are far more interpretable by this measure than features found by comparable techniques.
  • The interpretability advantage declines through the model, becoming comparable to ICA in layer 4 and minimal in the final layer.
  • Later-layer results may reflect weaker sparse-autoencoder performance, greater feature complexity, or limitations of automated interpretation.

4 IDENTIFYING CAUSALLY-IMPORTANT DICTIONARY FEATURES FOR INDIRECT OBJECT IDENTIFICATION

The paper adapts activation patching to sparse dictionary features to localise features involved in indirect object identification. Sparse dictionaries achieve target-level model editing with fewer patches and smaller edits than PCA, while greater sparsity introduces a reconstruction trade-off.

  • 4.1 ADAPTING ACTIVATION PATCHING TO DICTIONARY FEATURES: Activation patching edits residual-stream representations along selected dictionary-feature directions and measures the resulting change using KL divergence from target predictions.The experiment transfers encoded feature activations from a counterfactual sentence to the base sentence at an intervention layer.
  • 4.1 ADAPTING ACTIVATION PATCHING TO DICTIONARY FEATURES: ACDC orders features by their effect on the KL metric, and the first k features in that ordering define each intervention subset.The ordering is computed separately for each decomposition and averaged over 50 IOI test examples.
  • 4.2 PRECISE LOCALISATION OF IOI DICTIONARY FEATURES: Sparse feature dictionaries reach a given KL divergence from the target with fewer patches and smaller edit magnitude than PCA components.Figure 3 evaluates these interventions on layer 11 of the Pythia-410M residual stream.
  • 4.2 PRECISE LOCALISATION OF IOI DICTIONARY FEATURES: Non-sparse dictionaries with α = 0 do not produce the same editing advantage as sparse dictionaries.The comparison uses the same activation-patching framework and decomposition-level intervention procedure.
  • 4.2 PRECISE LOCALISATION OF IOI DICTIONARY FEATURES: Larger sparsity coefficients α reduce overall reconstruction accuracy, appearing as a larger minimum KL divergence in the editing results.This establishes a trade-off between sparsity and reconstruction fidelity in the evaluated dictionaries.

5 CASE STUDIES

The case studies examine dictionary features through their activating inputs, output-logit effects, and relationships with intermediate features. They identify largely monosemantic features, including context-specific apostrophe features, and trace causal feature circuits across layers.

  • 5 CASE STUDIES: The case studies use input activations, output-logit changes under ablation, and intermediate-feature relationships to interpret individual dictionary features.These analyses target features that appear to correspond to single human-understandable explanations.
  • 5.1 INPUT: DICTIONARY FEATURES ARE HIGHLY MONOSEMANTIC: Dictionary features can activate on restricted token sets, including apostrophes, periods, the token “ the”, and newline characters.The apostrophe feature is more specific than a residual-stream dimension that represents different information across activation ranges.
  • 5.1 INPUT: DICTIONARY FEATURES ARE HIGHLY MONOSEMANTIC: The apostrophe feature is context-specific rather than active on every apostrophe, with other features responding to contexts such as “[I/We/They]’ll” and “[don/won/wouldn]’t”.Thus, multiple dictionary features can partition similar tokens by surrounding context.
  • 5.2 OUTPUT: DICTIONARY FEATURES HAVE PREDICTABLE EFFECTS: Ablating the apostrophe feature mainly reduces the logit for the following “s”, matching its interpretation as an apostrophe detector used for contractions.Figure 4 also shows effects across many logits, with a 0.1 threshold used for visual clarity.
  • 5.3 INTERMEDIATE FEATURES: DICTIONARY FEATURES ALLOW AUTOMATIC CIRCUIT DETECTION: Causal trees connect dictionary features across successive layers by ranking upstream features according to how much their ablation decreases a target feature’s activation.For the closing-parenthesis example, edge thickness represents the strength of these ablation-measured causal effects.

6 DISCUSSION

The paper presents sparse autoencoders as a scalable, unsupervised approach for learning interpretable features, while identifying limitations in reconstruction fidelity, layer coverage, and demonstrated task generalization.

  • Limitations and future work: The dictionaries do not achieve zero reconstruction loss, so they fail to capture all information in a layer’s activations.Replacing layer-2 residual-stream activations in Pythia-70M with reconstructions increases Pile perplexity from 25 to 40.
  • Limitations and future work: Current training methods are best suited to the residual stream and do not robustly learn overcomplete bases in intermediate MLP layers.The authors identify understanding attention heads and MLP computations as requiring further work.
  • Limitations and future work: The authors expect task-agnostically learned features to generalize to similar behaviours, but state that this expectation requires confirmation.The IOI results involve a relatively small number of features, motivating further tests of generalization.
  • Conclusion: Sparse autoencoders use unlabelled activations and substantially less compute than original model training to disentangle features from superposition.The authors suggest this approach could support mapping model circuits, targeted editing, and improved understanding of representations.
  • Motivation: The paper frames complete, human-understandable feature enumeration as an ambitious route toward safety guarantees against behaviours such as deception.The authors present their techniques as a step toward this ambition rather than as achieving it.
  • Autointerpretation protocol: The autointerpretability protocol samples activating text fragments, uses language models to generate and simulate explanations, and scores correlations with actual activations.The protocol uses GPT-4 for interpretation and GPT-3.5 for simulation because the latter API provides logprobs.

B SPARSE AUTOENCODER TRAINING AND HYPERPARAMETER SELECTION

The authors train sparse autoencoders on cached language-model activations and examine how sparsity settings affect reconstruction. Across hyperparameter sweeps, sparsity and reconstruction accuracy exhibit a smooth tradeoff.

  • Training setup: The training data consist of 5–50M cached activation vectors from Pile text, and autoencoders are trained for 1–3 epochs with Adam.The reported learning rate is 1e-3, with larger dictionaries receiving more training data.
  • Hyperparameter selection: Increasing the sparsity-loss hyperparameter α consistently produces a smooth tradeoff between sparsity and reconstruction accuracy.The authors report no clear bump or knee, which provides evidence against a uniquely correct sparse-basis decomposition, though more experiments are needed.
  • Hyperparameter selection: Figure 6 compares the average number of active features with the proportion of unexplained variance for the layer-0 MLP.The figure is used to inspect the sparsity–reconstruction tradeoff.
  • Hyperparameter selection: Figure 7 plots sparsity against unexplained reconstruction variance across α sweeps, with separate series corresponding to different training durations.The figure shows convergence behaviour for models trained for the epoch counts listed in its legend.

C FURTHER AUTOINTERPRETATION RESULTS

Higher interpretability scores are not restricted to overcomplete dictionaries: they also occur in dictionaries smaller than the underlying basis, although smaller dictionaries reconstruct activations less accurately.

  • Interpretability across dictionary sizes: Larger interpretability scores occur in both overcomplete and smaller-than-basis dictionaries.The result indicates that interpretability gains are not limited to dictionaries with more features than model dimensions.
  • Interpretability across dictionary sizes: Smaller dictionaries reconstruct activation vectors less accurately, while larger dictionaries can explain more overall variance when feature interpretability is similar.The comparison links dictionary size to reconstruction coverage rather than to a claim of uniformly better interpretability.

C.2 HIGH INTERPRETABILITY SCORES ARE NOT AN ARTEFACT OF TOP SCORING

Control analyses indicate that the higher interpretability of sparse-coding features is not explained by the use of top-activating fragments or by an uneven comparison with baseline directions. Residual-stream basis directions also show no special interpretability advantage, while MLP results remain limited by dead features.

  • Random-only evaluation: Random-only scores are lower than top-and-random scores, but sparse-coding features consistently outperform baselines, especially in early layers.Because sparse-coding features activate less on individual fragments, the random-only evaluation may underestimate their relative performance.
  • Comparison fairness: Appendix G finds that sensitivity to less than a full activation-space direction does not explain sparse coding’s improved performance.The concern was that sparse autoencoders and comparison directions were not being evaluated on equivalent activation regions.
  • Residual-stream basis: Residual-stream basis directions are no more interpretable than random directions.This result holds despite reported tendencies for transformers to store information in the residual-stream basis.
  • MLP limitations: MLP applications find many interpretable features but also many dead features that never activate across the corpus.A larger fraction of MLP features remains active in early layers, and separate encoder and decoder matrices retain more features.
  • MLP limitations: The authors are developing methods to find truly overcomplete bases in middle and later MLP layers.This identifies the current scope boundary for the MLP results.

C.4 INTERPRETABILITY SCORES CORRELATE WITH KURTOSIS AND SKEW OF ACTIVATION

Interpretability scores are associated with statistical properties of feature activations, particularly positive skew and kurtosis. This relationship helps explain why ICA performs best among the alternative decompositions considered, while residual-basis examples connect activation patterns to specific token features.

  • Activation moments: Interpretability correlates 0.19 with positive skew and 0.24 with kurtosis across residual-stream results.The correlations are reported across all layers and dictionary size ratios R ∈ {0.5, 1, 2, 4, 8}.
  • Activation moments: Heavy-tailed feature activations are expected when interference from other active features is approximately normally distributed.The proposed intuition is that interference follows a central-limit-theorem pattern, making heavy-tailed features more distinctive.
  • Alternative decompositions: ICA is the best-performing alternative because it maximizes the non-Gaussianity of its components.The paper links this property to the observed relationship between activation statistics and interpretability.
  • Feature examples: A residual-stream dimension associated with an apostrophe feature ranked tenth among dimensions read by that feature.The token activation histogram also shows that apostrophes are common in the upper activation range but explain little variance at middle-to-lower ranges.
  • Feature examples: The paper illustrates feature examples involving coding-context ‘If’, ‘Dis’ bigrams, and apostrophe contexts such as “I’ll” and “don’t”.These examples are presented as additional feature case studies and include residual-basis comparisons.

D.4 FAILED INTERPRETABILITY METHODS

The paper tests whether sparse coding’s interpretability gains arise from implementation or comparison artifacts. The analyses also expose practical limitations in MLP feature learning and report intervention results across residual-stream layers.

  • Dictionary capacity: Residual-stream dictionaries remain at least 4x overcomplete before features begin to drop out completely with suitable hyperparameters.For MLP layers, many dead features appear even with α = 0.
  • Top-K comparisons: Matching the average number of active sparse-coding features in PCA and ICA explains only a small fraction of the interpretability improvement.The control fixes the number of directions that can be active for a datapoint, addressing differences caused by sparse coding’s biased activation geometry.
  • Feature patching: Figure 17 evaluates divergence from a target output after varying the number and magnitude of patched feature edits across selected Pythia-410M residual-stream layers.The evaluated layers are 3, 7, 11, 15, 19, and 23 in a 24-layer model.
  • Top-K comparisons: Figure 18 compares interpretability across layers using sparse-coding directions and top-K PCA and ICA baselines.The comparison specifically tests whether unequal activation regions account for the reported score differences.
Loading 2309.08600v3…