PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse Autoencoders Do Not Find Canonical Units of Analysis

Patrick Leask, Bart Bussmann, Michael Pearce, Joseph Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, Neel Nanda

arXiv:2502.04878v1cs.LGcs.AI

TL;DR

Mechanistic interpretability seeks unique, complete, atomic units, and SAEs were proposed as a way to find them. The paper uses SAE stitching and meta-SAEs to test this proposal, finding that smaller SAEs miss information and larger latents often compose smaller features. It concludes that SAE width is task-dependent rather than universally canonical.

  • Problem

    The paper examines whether SAEs can identify a unique, complete, and atomic set of features for mechanistic interpretability.

  • Method

    The authors compare SAE sizes with SAE stitching and decompose SAE decoder directions using meta-SAEs.

  • Results

    Larger SAEs learn novel latents missing from smaller SAEs, while many larger latents decompose into interpretable combinations of smaller features.

  • Takeaways & Limitations

    SAE width should be chosen pragmatically for the interpretability task, while alternative approaches to canonical units should be explored.

  • Takeaways & Limitations

    The study covers only two LLMs and does not include very large SAEs; it also does not determine the dictionary size for a given task.

Abstract

from arXiv · show

A common goal of mechanistic interpretability is to decompose the activations of neural networks into features: interpretable properties of the input computed by the model. Sparse autoencoders (SAEs) are a popular method for finding these features in LLMs, and it has been postulated that they can be used to find a \textit{canonical} set of units: a unique and complete list of atomic features. We cast doubt on this belief using two novel techniques: SAE stitching to show they are incomplete, and meta-SAEs to show they are not atomic. SAE stitching involves inserting or swapping latents from a larger SAE into a smaller one. Latents from the larger SAE can be divided into two categories: \emph{novel latents}, which improve performance when added to the smaller SAE, indicating they capture novel information, and \emph{reconstruction latents}, which can replace corresponding latents in the smaller SAE that have similar behavior. The existence of novel features indicates incompleteness of smaller SAEs. Using meta-SAEs -- SAEs trained on the decoder matrix of another SAE -- we find that latents in SAEs often decompose into combinations of latents from a smaller SAE, showing that larger SAE latents are not atomic. The resulting decompositions are often interpretable; e.g. a latent representing ``Einstein'' decomposes into ``scientist'', ``Germany'', and ``famous person''. Even if SAEs do not find canonical units of analysis, they may still be useful tools. We suggest that future research should either pursue different approaches for identifying such units, or pragmatically choose the SAE size suited to their task. We provide an interactive dashboard to explore meta-SAEs: https://metasaes.streamlit.app/

1 INTRODUCTION

The paper questions whether SAEs recover canonical units that are unique, complete, and atomic. SAE stitching reveals missing features in smaller SAEs, while meta-SAEs show that larger SAE latents can be compositions of smaller features.

  • Canonical units should be unique, complete, and atomic components that collectively explain a network’s function.
  • SAEs were proposed as a way to recover canonical features by decomposing language-model activations into interpretable, monosemantic dictionaries.
  • Feature splitting and newly observed features suggested that SAE dictionary size changes which features are represented, but comparison methods were lacking.
  • SAE stitching swaps similar latent clusters across dictionary sizes and identifies both finer-grained reconstruction latents and entirely novel latents.Novel latents indicate that smaller SAEs miss information rather than merely approximating or coarsening existing features.
  • Some larger-SAE reconstruction latents combine multiple smaller-SAE latents, consistent with sparsity favoring composed features such as color-shape combinations.A larger SAE can represent a colored shape with one combined latent instead of separate color and shape latents.
  • Meta-SAEs decompose SAE decoder directions into interpretable meta-latents, showing that larger SAE latents are often not atomic.The paper gives an “Einstein” example decomposing into concepts including scientist, Germany, and prominent figures.
  • The findings suggest no single SAE width provides a unique, complete dictionary of atomic features for all mechanistic-interpretability tasks.The authors recommend pragmatically trying several widths and exploring alternative approaches.

2 SPARSE AUTOENCODERS

SAEs decompose language-model activations into sparse, nonnegative latent representations and reconstruct the original activations from learned decoder directions. Their training objective balances reconstruction quality against sparsity, with optional auxiliary losses.

  • Sparse dictionary learning seeks a sparse and overcomplete decomposition of a signal.
  • In language models, SAEs represent activations x ∈ R^n as sparse linear combinations of learned directions that are often interpretable and monosemantic.
  • An SAE uses an encoder to produce a sparse latent representation f(x) and a decoder to reconstruct the input activation.The activation function enforces non-negativity and sparsity; latent i is active when f_i(x) > 0.
  • SAEs are trained on language-model activations from a chosen site and a large text corpus using a reconstruction-and-sparsity loss.
  • Higher sparsity coefficients encourage sparser decompositions but increase reconstruction error.Some architectures additionally use an auxiliary loss to recycle inactive latents.

3 RELATED WORK

Related work frames SAEs as tools for recovering interpretable representations, while prior studies report mixed effects of dictionary size. SAE stitching adapts model-stitching ideas to compare latent representations across SAE sizes.

  • SAEs have been shown to recover sparse, monosemantic, and interpretable features from language-model activations.
  • Representation studies show that language models learn structured information such as board states, geography, and temporal features.
  • Prior dictionary-size studies disagree: larger SAEs can learn absent chemical-element features, while other work finds similar latents that split into finer features.
  • Model stitching swaps layers between networks to assess representation similarity through the performance cost of connecting them.
  • SAE stitching swaps connected latent subgraphs between differently sized SAEs based on cosine similarity.

4 SAE STITCHING

SAE stitching compares latent behavior across dictionary sizes by adding or swapping larger-SAE latents into smaller SAEs. It distinguishes novel latents that add information from reconstruction latents that replace similar features, showing that larger SAEs improve reconstruction through both mechanisms.

  • Stitching operation: SAE stitching transfers or replaces latents between SAEs of different dictionary sizes to compare their feature representations.The method uses decoder similarity and reconstruction performance to identify relationships across SAEs.
  • Novel and reconstruction latents: Larger SAEs contain both finer-grained versions of smaller-SAE latents and entirely novel latents that capture additional information.Novel latents improve reconstruction when added, whereas reconstruction latents correspond to similar features already represented in smaller SAEs.
  • Novel and reconstruction latents: A maximum decoder-cosine threshold classifies larger-SAE latents as novel or reconstruction latents, using 0.7 for GPT-2 SAEs and 0.4 for Gemma Scope SAEs.The threshold is a heuristic based on exploratory analysis of cosine similarity and reconstruction effects.
  • Novel and reconstruction latents: Both adding novel latents and swapping reconstruction latents improve reconstruction on average, allowing interpolation across SAE size, sparsity, and performance.The result is reported using average reconstruction MSE over input samples.
  • Novel and reconstruction latents: Adding novel latents increases average L0, while swapping reconstruction groups decreases average L0 because larger SAEs represent similar information more sparsely.The transitions compare dictionary sizes 768→1536→3072→6144→12288.
  • Novel and reconstruction latents: Some larger-SAE reconstruction latents match multiple smaller-SAE latents, indicating interpolation or composition rather than only feature splitting.The findings support the possibility that sparsity encourages composed features that are sparser without adding new information.

5 META-SAES

Meta-SAEs decompose the decoder directions of larger SAEs into sparse, interpretable meta-latents. Their components resemble latents from comparably sized standard SAEs, supporting the view that larger-SAE latents can be compositions of smaller-SAE features.

  • Meta-SAE method: Meta-SAEs are trained on the decoder directions of a standard SAE rather than directly on network activations.The decoder directions serve as the meta-SAE training data, and the learned decompositions are sparse and interpretable.
  • Meta-SAE decompositions: Meta-latents are often monosemantic, and their decompositions provide interpretable explanations of larger-SAE latents.The paper reports examples in which a larger latent is represented through multiple interpretable meta-latents.
  • Comparison to smaller SAE latents: Meta-SAE latents have decoder directions similar to those of standard SAEs trained directly on the same network activations.This similarity appears in comparisons across SAE dictionary sizes.
  • Experimental setup: A meta-SAE trained on 49,152 GPT-2 SAE decoder directions used 2,304 meta-latents, with approximately 4 active meta-latents per SAE latent.The meta-SAE was trained with BatchTopK for 2,000 epochs.
  • Comparison to smaller SAE latents: Replacing meta-SAE decoder directions with their most similar standard-SAE directions causes only a small decrease in reconstruction performance.This validation supports the correspondence between meta-latents and standard-SAE latents.
  • Comparison to smaller SAE latents: The results suggest that larger-SAE latents are composed of features learned less precisely by smaller SAEs rather than being entirely new atomic features.The conclusion follows from the similarity between meta-SAE latents and comparable standard-SAE latents.

6 CONCLUSION

The paper concludes that SAEs do not yield a unique, complete, and atomic canonical dictionary of features. Instead, SAE size changes the granularity and composition of learned features, so size selection should be task-dependent.

  • Conclusion: SAE stitching shows that smaller SAEs are incomplete because larger SAEs contain novel features they miss.These novel features capture additional information beyond imperfect approximations of existing features.
  • Conclusion: Meta-SAE experiments show that larger-SAE latents are often compositions of interpretable meta-latents rather than atomic features.The paper attributes this pattern to the sparsity penalty.
  • Conclusion: No single SAE width provides a unique, complete, and atomic dictionary that explains model behavior.The conclusion directly challenges the idea that SAEs discover canonical units of analysis.
  • Conclusion: SAEs of different sizes offer varying feature granularities and compositions, so practitioners should choose dictionary size pragmatically for their interpretability task.The paper names probing, unlearning, and steering as example tasks.
  • Limitations: The methods do not identify canonical units or determine the dictionary size appropriate for a given task.The study also covers only two LLMs and does not include very large SAEs.

A.1 GLOSSARY OF TERMS

The glossary defines core SAE concepts used to describe feature representations, sparsity, stitching, and comparisons across dictionary sizes.

  • Activation and sparsity: An active latent is counted when an SAE activation is nonzero, with L0 typically averaged across a batch.L0 measures the number of nonzero elements in the activation vector.
  • Core concepts: A canonical unit is a hypothetical LLM computation building block that is unique, complete, and atomic.The paper tests whether SAEs can discover such units.
  • Methods and representations: Dictionary size is the dimensionality of an SAE latent space and sets the maximum number of unique features it can learn.Feature splitting describes broad smaller-SAE latents becoming finer-grained in larger SAEs.
  • Methods and representations: Meta-latents are features learned by a meta-SAE trained on another SAE’s decoder directions.They are used to decompose larger-SAE latents.
  • Feature semantics: A monosemantic feature responds selectively to one coherent concept, unlike a polysemantic feature responding to multiple unrelated concepts.Polysemanticity is presented as a limitation of individual neurons or features.
  • Latent relationships: Novel latents have maximum cosine similarity below a threshold to every smaller-SAE feature and indicate previously unrepresented information.Reconstruction latents exceed the threshold and represent refined or specialized versions of existing features.
  • Methods and representations: SAE stitching transfers latents between different-sized SAEs based on decoder similarity to analyze feature relationships.Decoder directions are compared through cosine similarity.

A.2 SAE VARIANTS

The paper distinguishes SAE variants by how they enforce sparsity, trading flexibility, explicit sparsity control, and reconstruction behavior in different ways.

  • ReLU SAEs: ReLU SAEs use an L1-norm sparsity penalty as a differentiable approximation to the L0-norm.The penalty can suppress activations and harm reconstruction, while decoder normalization prevents reducing it through rescaling.
  • TopK SAEs: TopK SAEs retain only the k largest activations per sample, eliminating the need for an explicit sparsity penalty.This imposes a rigid number of active latents for each sample.
  • TopK SAEs: TopK SAEs use an auxiliary loss to recycle inactive latents and avoid dead latents.The reconstruction uses only the top-k_aux dead latents, typically 512, with a small coefficient α, usually 1/32.
  • BatchTopK SAEs: BatchTopK relaxes TopK sparsity from each sample to the batch level during training, retaining the top b ∗k activations per batch.At test time, it uses a training-estimated global threshold instead.
  • JumpReLU SAEs: JumpReLU replaces standard ReLU with a learned threshold that sets each latent to zero below θ.It combines L2 reconstruction error with an L0 sparsity penalty but does not provide explicit sparsity control without costly tuning.

A.3 EXAMPLE LATENTS

Comparing GPT2-1536 and GPT2-768 latents reveals both strong correspondences and genuinely novel larger-SAE features, whose absence is associated with worse reconstruction.

  • Similarity distributions: Many GPT2-1536 latents have high maximum decoder cosine similarity to GPT2-768 latents, alongside a long tail of novel latents.The histogram measures each larger-SAE latent against all latents in the smaller SAE.
  • Similarity distributions: Decoder cosine similarity is used because it is cheaper to compute and captures effects on reconstruction, while correlating highly with activation similarity in the relevant range.For decoder cosine similarity above 0.5, the coefficient of determination is 0.87.
  • Matched latents: Latents with decoder cosine similarity 0.99 across GPT2-1536 and GPT2-768 activate on the same inputs and boost similar logits.Neuronpedia dashboards visualize maximally activating tokens and the logits latents boost or suppress.
  • Novel latents: GPT2-1536 contains a “make sure” latent with no GPT2-768 counterpart; its nearest smaller-SAE latents have cosine similarity around 0.3.This provides an example of a larger-SAE feature with no close match in the smaller SAE.
  • Novel latents: The smaller SAE reconstructs considerably worse when the larger SAE’s novel latent is active, a pattern averaged across 657 such latents.Figure 11 compares reconstruction MSE on inputs where novel larger-SAE latents are active versus inactive.

A.4 LATENT FAMILIES

The paper visualizes latent families as connected subgraphs in a bipartite graph linking GPT2-768 and GPT2-1536 latents.

  • Latent families: Connected subgraphs represent families in the bipartite graph of GPT2-768 and GPT2-1536 latents.

A.5 OPEN SOURCE SAE WEIGHTS

The experiments compare SAE latents across dictionary sizes using stitching, reconstruction performance, sparse probing, and concept-removal evaluations. Larger SAEs add novel information but also produce composed features, while evaluation outcomes vary by task and are not consistently monotonic with SAE size.

  • SAE stitching: Swapping reconstruction latents generally increases MSE but almost always decreases L0, with smaller percentage effects in larger models.
  • SAE stitching: Adding GPT-1536 latents with cosine similarity ≤0.7 to GPT-768 reduces MSE by almost 10%, indicating that larger SAEs contain novel information.
  • SAE stitching: 94% of latents are reconstruction latents when comparing SAEs of the same size, using a cosine-similarity threshold above 0.7.
  • Gemma Scope SAEs: A cosine-similarity threshold of 0.4 distinguishes novel from reconstruction features in Gemma Scope SAEs and supports smooth interpolation between dictionary sizes.
  • Interpretability evaluations: Sparse-probe accuracy using a single top latent has a complex, dataset-dependent relationship with SAE dictionary size.
  • Interpretability evaluations: Targeted probe perturbation accuracy generally trends downward with SAE dictionary size but does not decrease monotonically.
Loading 2502.04878v1…