Source-linked AI summary
Sparse Autoencoders Trained on the Same Data Learn Different Features
Gonçalo Paulo, Nora Belrose
TL;DR
SAEs are often expected to uncover a unique set of model features, but evidence for seed invariance is limited. This paper compares independently initialized SAEs trained on the same models and data using matched encoder and decoder features. It finds substantial seed dependence, including only 30% shared features in Llama 3 8B, and concludes that SAEs provide useful decompositions rather than universal feature inventories.
Problem
The paper tests whether independently trained SAEs recover a unique, objective decomposition of a model into features that can be identified and enumerated.
Method
The paper uses Hungarian matching of latent features and compares encoder and decoder cosine similarities across independently trained SAEs.
Results
Only 30% of features are shared across seeds in Llama 3 8B, while overlap varies across model layers, architectures, and SAE configurations.
Takeaways & Limitations
SAE feature sets are pragmatically useful decompositions of activation space rather than universal sets of features truly used by the model.
Takeaways & Limitations
The authors found no evidence of feature absorption in their MLP SAEs, but acknowledge that their metric may not detect absorption reliably for MLP SAEs.
Abstract
from arXiv · showhide
Sparse autoencoders (SAEs) are a useful tool for uncovering human-interpretable features in the activations of large language models (LLMs). While some expect SAEs to find the true underlying features used by a model, our research shows that SAEs trained on the same model and data, differing only in the random seed used to initialize their weights, identify different sets of features. For example, in an SAE with 131K latents trained on a feedforward network in Llama 3 8B, only 30% of the features were shared across different seeds. We observed this phenomenon across multiple layers of three different LLMs, two datasets, and several SAE architectures. While ReLU SAEs trained with the L1 sparsity loss showed greater stability across seeds, SAEs using the state-of-the-art TopK activation function were more seed-dependent, even when controlling for the level of sparsity. Our results suggest that the set of features uncovered by an SAE should be viewed as a pragmatically useful decomposition of activation space, rather than an exhaustive and universal list of features "truly used" by the model.
1. Introduction
SAEs are intended to decompose neural activations into interpretable features, but this paper tests whether independently initialized SAEs recover a unique feature decomposition. It compares matched features across same-data training runs and finds shared and orphan features, with only 30% shared in Llama 3 8B.
- SAEs encode activations into a sparse higher-dimensional latent space and decode those latents back into the original activation space.
- The paper tests whether SAE features depend on random initialization despite identical training data and whether SAEs can enumerate a model’s features.
- The Hungarian algorithm matches features across SAEs, while encoder and decoder cosine similarities provide two measures of feature similarity.
- Matched feature similarities form high-similarity shared and low-similarity orphan modes, with shared features tending toward similar contexts and explanations.
- 30% of features are shared across seeds in Llama 3 8B, and orphan features are often interpretable, indicating that individual runs can miss interpretable features.
2. Related Work
Related work examines feature splitting, seed stability, and cross-model SAE alignment. These findings motivate comparing overlap across seeds while accounting for SAE architecture, scale, and initialization choices.
- Meta SAEs can decompose SAE features into more specific features, while larger SAEs may replace one small-SAE feature with multiple specific features.
- Figure 2 measures how the number of latents found only in a base SAE changes as the number of seeds increases.
- Prior work reported seed stability for ReLU SAEs with L1 sparsity, whereas TopK SAEs benefited from explicitly aligning two training seeds.
- JumpReLU and TopK latents have been reported to show higher feature-splitting rates than ReLU latents.
- Positive alignment results for Gemma SAEs may depend on using the same random seed to initialize every SAE.
3. Methods
The paper aligns independently trained SAEs by optimally matching their latent features and evaluates whether matched features are shared or orphaned. It also compares this matching with maximum cosine similarity and examines interpretability examples.
- The Hungarian algorithm maximizes average cosine similarity between matched encoder and decoder vectors to produce an overall alignment score.
- Two SAEs trained on the same Pythia 160M MLP and data but different seeds produce distinct high-similarity shared and low-similarity orphan modes.
- Encoder and decoder matchings usually agree when similarities are high and disagree when similarities are low.
- A latent is shared when the same counterpart is selected by both matchings and both cosine similarities are at least 0.7.
- Table 1 selects latent pairs across five alignment bins to show cases where explanations score well despite low cosine similarity.
- 42% of latents are shared across the two independently trained SAEs, implying that close to 60% could be seed dependent if each latent corresponds to a learned feature.
- Maximum cosine similarity differs little empirically from Hungarian matching, although it does not enforce a bijective correspondence.
4. Asymptotic Trend
Across nine independently trained SAEs, the fraction of latents found in only one SAE decreases slowly as more seeds are considered, reaching about 35% at k = 9. Overlap also depends on SAE size, active-latent count, training time, and latent firing frequency.
- Seed scaling: About 35% of latents are found in only one SAE when all nine independently trained SAEs are considered.The experiment uses pairwise Hungarian matchings and classifies a latent as unique when it is an orphan across all comparisons with the other SAEs.
- Seed scaling: The number of latents found in only one SAE decreases slowly as the number of seeds increases.A power law with an offset term fits the data significantly better than a power law without one.
- Hyperparameter trends: The fraction of aligned latents decreases as the total number of SAE latents increases.Figure 4 studies this dependence for Pythia-160M SAEs trained on the sixth-layer MLP output.
- Hyperparameter trends: Increasing TopK’s active-latent count k also decreases overlap between SAE seeds.Figure 4 varies k through the TopK activation function.
- Hyperparameter trends: Longer training increases alignment between SAE seeds.This trend is shown in the training-time panel of Figure 4.
- Firing frequency: Latents that fire most frequently are shared across all eight comparisons, whereas infrequently firing latents are often shared with none.Some orphan latents nevertheless have higher average firing rates than latents shared with all SAEs.
5. Are “Orphan” Latents Interpretable?
Orphan latents are not uniformly uninterpretable: shared latents tend to receive higher explanation scores, but some low-alignment or single-SAE latents remain highly interpretable. The analysis evaluates explanations using detection scoring on active and non-active sequences.
- Scoring setup: The average explanation score for the 32K SAEs is 0.72.Only 25% of explanations score below 0.62, and only 25% score above 0.8.
- Interpretability results: Shared latents across more SAE seeds have higher average explanation scores.The explanations are scored with detection scoring over 100 active and 100 non-active sequences.
- Interpretability results: A significant fraction of latents found in only one SAE have high-scoring explanations.Some latents with average cosine alignment below 0.7 also have high scores.
- Interpretability results: Most low-alignment latents have either low or average explanation scores, while some remain highly interpretable.Figure 6 compares score distributions conditioned on how many SAEs contain a latent and compares matched latents across seeds.
6. Ablations
Ablations show that overlap is lower for larger SAEs and higher with longer training, while architecture and model position also affect seed alignment. The Hungarian and maximum-cosine comparison methods produce strongly correlated results.
- Hyperparameters: Increasing the number of SAE latents decreases overlap, whereas longer training increases overlap between seeds.These trends are observed while varying SAE training hyperparameters.
- Interpretation caveat: The authors found no evidence of feature absorption in their MLP SAEs, but the metric may not be tuned to detect absorption there.The metric was used mostly on residual-stream SAEs.
- Architectures and models: Standard and Gated SAEs trained with an L1 loss have larger latent overlap than the TopK SAEs studied here.The overlap is much lower for the Llama 3 8B SAEs, which have more latents but the same expansion factor.
- Layers and positions: Overlap remains almost constant across middle layers but is lower in earlier layers and the last layer.MLP latents overlap more than residual-stream latents on SmolLM and GPT2, but not on Pythia.
- Robustness: Matched cosine similarity from the Hungarian algorithm is strongly correlated with maximum cosine similarity.This indicates that the reported results are not strongly dependent on the feature-comparison method.
7. Conclusion
The results argue that SAEs do not recover a universal feature set: different initializations can produce different features, and divergence increases with scale. The authors instead frame feature discovery as a compositional, hierarchical way of partitioning activation space.
- Conclusion: Different random initializations can lead SAEs to identify different feature sets rather than a universal set.The authors present this as evidence against universal SAE features.
- Conclusion: SAEs appear to diverge rather than converge as scale increases.The conclusion connects this trend to the observed lack of universality.
- Implication: Feature discovery is best viewed as a compositional process that partitions activation space into categories that can be subdivided hierarchically.This framing allows categories to be further cut into more specific categories.
- Mechanism: The lack of universality arises mathematically from the nonconvex SAE loss and its many local optima.The authors note that different local optima might nevertheless have been expected to share more features than observed.