PaperlayerView exact revision on arXiv

Source-linked AI summary

Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

Gleb Gerasimov, Timofei Rusalev, Nikita Balagansky, Daniil Laptev, Vadim Kurochkin, Daniil Gavrilov

arXiv:2606.12138v1cs.LGcs.AIcs.CL

TL;DR

SAE features can vary substantially across random seeds, raising whether instability reflects missing directions or different bases for shared structure. This paper measures feature stability across runs and finds that stable features carry most functional signal, while unstable features occupy reproducible low-rank subspaces and can support more seed-robust dictionaries.

  • Problem

    It is unclear whether seed-dependent SAE features represent absent directions or different bases for shared activation-space structure, affecting their status as reproducible interpretive units.

  • Method

    The paper estimates each feature’s cross-run reappearance probability using cosine-similarity matching, then compares stable and unstable features across function, token structure, explanations, and geometry.

  • Results

    Stable features carry most reconstruction- and prediction-relevant signal, whereas unstable features have weaker marginal impact and concentrate in reproducible lower-rank subspaces.

  • Takeaways & Limitations

    Unstable features are not simply noise: they reflect shared low-dimensional structure that different SAE seeds resolve through different bases, while pooled features can yield more seed-robust dictionaries.

  • Takeaways & Limitations

    The subspace analysis demonstrates a concrete mechanism rather than fully identifying real-LLM instability, and the no-tradeoff result is specific to aggregated feature-pool construction.

Abstract

from arXiv · show

Sparse autoencoders (SAEs) are widely used to interpret neural network representations, but their utility depends on whether the learned features are reproducible across training runs. We study this question through \emph{feature stability}: for each SAE feature, we estimate the probability that a similar feature reappears in an independently trained SAE. This yields a scalable per-feature signal that separates stable from unstable features. In a large-scale study across seeds, models, layers, dictionary sizes, and SAE variants, we find a pronounced functional asymmetry: stable features carry most of the reconstruction- and prediction-relevant signal, while unstable features have weak marginal impact and are dominated by low-frequency surface-form triggers in both activation statistics and automatic explanations. Geometrically, unstable features are individually non-reproducible but concentrate in reproducible lower-rank subspaces, suggesting that seed dependence often reflects basis ambiguity within a shared region of activation space rather than pure noise. A controlled synthetic model makes this mechanism explicit, showing that low-rank ground-truth features can be recovered at the subspace level while remaining non-identifiable as individual SAE latents across seeds. Finally, by pooling unique cross-seed features, we construct more stable SAEs while preserving explained variance in this setting. Together, these results show that unstable features are not merely failed or noisy latents: they have weak individual functional impact, but reflect reproducible low-dimensional structure that standard SAEs resolve differently across seeds.

1 Introduction

This section frames seed dependence as a reproducibility problem for SAE features and introduces feature stability to distinguish stable from unstable features. It argues that unstable features have weak individual functional impact while collectively reflecting reproducible lower-rank structure, without an observed stability–explained-variance trade-off in feature-pool construction.

  • Motivation: SAEs trained on identical activations with different random seeds can learn substantially different features, raising questions about feature reproducibility.The section distinguishes whether a missing feature reflects an absent direction or a different basis for the same activation-space region.
  • Method: Feature stability estimates the probability that a feature reappears in an independently trained SAE under cosine-similarity matching.The signal supports comparisons of functional impact, token structure, automatic explanations, and decoder-space geometry.
  • Functional asymmetry: Stable features carry most reconstruction- and prediction-relevant signal, whereas unstable features have weak marginal impact and are dominated by low-frequency surface-form triggers.This pattern appears in both activation statistics and automatic explanations.
  • Subspace recovery without feature identifiability: Unstable features are individually non-reproducible but collectively span reproducible lower-rank subspaces, a pattern also shown for low-rank ground-truth features in a controlled synthetic setting.The synthetic features are recovered at the subspace level while failing to align with individual SAE latents.
  • No stability–EV trade-off in feature-pool construction: Higher explained variance coincides with higher mean feature probability after tuning when SAEs are built from deduplicated unique cross-seed features.In this feature-pool construction setting, stability and reconstruction quality need not trade off.

2 Related Work

Prior work shows that SAE features vary across seeds, dictionary sizes, layers, and architectures, motivating methods for stabilizing or matching features. This work studies instability as seed-dependent basis choice within reproducible lower-rank subspaces, using reappearance probabilities and controlled comparisons.

  • Feature stability: Prior studies find that SAEs can learn different feature sets across random seeds, while features also vary with dictionary size and can be merged across dictionaries.Proposed mitigations include anchoring dictionary atoms to data-derived archetypes.
  • Contribution: This work estimates per-feature reappearance probabilities to study seed-dependent basis choices within reproducible lower-rank decoder subspaces.The analysis uses a controlled low-rank synthetic model and compares stable and unstable features by reconstruction impact, downstream effects, and automatic explanations.
  • Finding similar features across SAEs: Researchers have developed methods to track, match, or graph similar SAE features across layers and language-model architectures.These efforts include adjacent-layer tracking, data-free cross-layer matching, crosslayer feature-flow graphs, and cross-architecture correspondence.

3 Preliminaries

Sparse autoencoders represent hidden states as sparse, non-negative combinations of learned feature embeddings and are trained with reconstruction loss plus optional sparsity regularization. Feature stability is estimated by matching each feature to its most similar cross-seed counterpart using maximum cosine similarity, a scalable alternative to one-to-one assignment.

  • SAE formulation: SAEs represent hidden states as sparse linear combinations of feature embeddings, with non-negative activation magnitudes indicating feature importance.Sparsity is induced by the nonlinear activation function.
  • SAE formulation: SAEs are trained using reconstruction loss plus optional regularization controlling sparsity, scaled by α ≥0.
  • SAE variants: The experiments compare Vanilla ReLU+ℓ1, TopK, BatchTopK, HierarchicalTopK, and JumpReLU SAEs with different sparsity mechanisms.TopK variants retain largest activations, HierarchicalTopK spans multiple sparsity budgets, and JumpReLU learns per-feature thresholds.
  • Feature matching: Because feature ordering is arbitrary, cross-seed comparisons require matching corresponding learned features.Two different initializations can produce the same dictionaries with different feature indices.
  • Feature matching: The study matches each feature to its most similar counterpart by maximum cosine similarity, enabling scalable individual-feature stability estimates.Replacing this many-to-one argmax-cosine rule with one-to-one Hungarian matching yields nearly identical matched feature sets.

4 Methodology

The methodology measures feature reproducibility by matching ℓ2-normalized decoder vectors across independently seeded SAEs. It summarizes reappearance probabilities through endpoint stability and uses thresholded cross-seed deduplication to construct a unique feature pool.

  • Feature representation: Features are represented by decoder vectors that are ℓ2-normalized, making cosine similarity equal to their dot product.
  • Feature matching: Feature matching compares decoder columns across two SAEs using a cosine-similarity threshold θ, set to 0.7 unless otherwise stated.
  • Reappearance probability: For each anchor feature, reappearance probability is estimated by counting how many of N independently seeded comparison SAEs contain a sufficiently similar feature.The study trains N + 1 SAEs with identical data and hyperparameters, selecting one run as the anchor.
  • Endpoint stability: Endpoint stability labels features as unstable when their estimated reappearance probability is near zero or one, using ε = 0.05.The methodology targets the empirical distribution at F(ε) and F(1 −ε).
  • Unique feature pool: A unique feature pool is formed by greedily deduplicating decoder features from independently trained SAEs with the same θ = 0.7 threshold.The resulting pool U initializes new SAEs in the construction experiment.

5 Quantitative Analysis

Stable SAE features are more frequent, lexically diverse, and functionally important than unstable features, which are biased toward low-frequency surface-form triggers. Yet pooled high-probability features can yield seed-robust SAEs without an apparent explained-variance penalty.

  • Activation frequency and magnitude: Unstable features activate less frequently than stable features, with mean frequencies of 0.18% versus 0.44%, respectively.Stable features also exhibit a heavier high-magnitude activation tail.
  • Token diversity and automatic interpretability: Unstable features concentrate on punctuation, formatting, subword fragments, and brittle substrings, whereas stable features support broader lexical and conceptual patterns.Automatic explanations similarly favor surface-form triggers for unstable features and phrases, syntactic roles, constructions, and semantic groupings for stable features; stable features have 4.5× more perfect-score features.
  • Functional impact: Masking four times as many unstable features changes explained variance only slightly, while masking fewer stable features causes a substantially larger drop.This indicates that stable features account for most reconstruction-relevant variance.
  • Functional impact: Removing many unstable features produces only a modest next-token loss change, whereas masking fewer stable features causes substantially larger degradation.Overall, unstable features have limited functional impact while stable features capture most reconstruction- and prediction-relevant structure.
  • Seed-robust SAE construction: After brief tuning, SAEs built from the most-probable pooled features recover near-baseline explained variance, while least-probable dictionaries perform much worse.The most-probable construction improves feature stability without sacrificing downstream interpretability performance on SAEBench metrics.

6 Geometric Analysis

Unstable SAE features occupy lower-dimensional decoder subspaces than stable features, yet those subspaces are reproducible across seeds. The results support seed-dependent mixing or basis ambiguity within shared low-dimensional structure rather than changes in the underlying subspace.

  • Dimensionality: Unstable features have lower effective rank than stable features: ER/d ≈0.59–0.65 versus ≈0.80–0.81 over N = 96 seeds, a 20–27% reduction.This indicates that unstable decoder vectors concentrate in a lower effective-dimensional subspace.
  • Basis ambiguity: The geometric results indicate that unstable features are individually non-reproducible because seeds mix different bases within a shared low-dimensional decoder subspace.This explains how individual feature identities can vary while the spanned subspace remains reproducible.
  • Cross-seed reproducibility: Within-seed and cross-seed classifiers achieve similar F1 scores, saturating at approximately 0.73 for ε = 0 and 0.67 for ε = 0.1.Cross-seed transfer therefore separates unstable features from remaining decoder vectors nearly as well as within-seed classification.
  • Cross-seed reproducibility: A top-r singular subspace learned in one seed accurately approximates the corresponding stable and unstable feature subspaces in other seeds.Explained-variance comparisons include Uε and Sε for ε ∈ {0.01, 0.05} across N = 96 seeds.
  • Synthetic validation: In a synthetic model with d = 32, r = 2, and k = 8, low-rank features have reappearance probability ≈0.25 and worse one-to-one recovery, unlike near-perfect full-rank recovery.The learned low-rank block also has lower effective rank and non-random cross-seed subspace similarity.

7 Other Setups

Across models, layers, dictionary sizes, SAE variants, training budgets, and random-model controls, a substantial stable subset coexists with a non-trivial unstable subset. Instability persists with extended training and reflects both sparsity mechanisms and differences between trained and random-model activations.

  • Other setups: Overall, these ablations show that a non-trivial unstable subset consistently coexists with a large stable subset across alternative experimental setups.The tested setups vary the base model, layer, dictionary size, SAE nonlinearity, training budget, and model initialization.
  • Effect of sparsity mechanism and reconstruction quality: TopK and BatchTopK are nearly indistinguishable in stability and EV (≈0.892), implicating hard k-sparsity rather than per-token versus per-batch enforcement.Within TopK SAEs, increasing k improves EV but monotonically increases the unstable fraction and decreases the stable fraction.
  • More SAE training does not remove instability: The unstable fraction reaches a non-zero plateau (≈8% at 10B tokens), while the stable fraction continues increasing with diminishing returns.The unstable fraction decreases early in training before approaching the plateau.
  • Dead-salmon baseline: trained vs. random transformers: The stability metric separates trained from random GPT-2 activations: trained-model SAEs retain a large stable subset, whereas random-model stability collapses across a wide range of θ.In the random-model setting, instability dominates, unlike the trained-model setting.

8 Conclusion

SAE seed dependence is structured: stable features carry most reconstruction- and prediction-relevant signal, whereas unstable features have weaker marginal impact and favor low-frequency surface-form triggers. Unstable features nevertheless concentrate in reproducible lower-rank subspaces, enabling more seed-robust dictionaries through cross-seed feature pooling without sacrificing reconstruction quality.

  • Functional asymmetry: Feature stability separates SAE features into stable and unstable regimes, with stable features carrying most reconstruction- and prediction-relevant signal.Stability is measured via cross-run reappearance probability.
  • Functional asymmetry: Unstable features have weaker marginal impact and are biased toward low-frequency surface-form triggers, unlike stable features’ structural or compositional patterns.
  • Geometric interpretation: Unstable features rarely reappear individually across seeds but concentrate in reproducible lower-rank subspaces, reflecting different basis choices within shared activation-space regions.A controlled toy model shows that low-rank ground-truth features can be recovered at the subspace level while remaining non-identifiable as individual SAE latents.
  • Cross-seed reuse: Pooling high-probability features across runs and briefly post-training produces more seed-robust dictionaries without sacrificing reconstruction quality.

Limitations … C.4 Automatic interpretation for baseline vs. most-probable-feature SAE

The paper’s supplemental analyses establish methodological details, threshold robustness, and limitations, while showing that probability-based feature-pool construction can preserve reconstruction quality and remain competitive with standard SAEs. The analyses also compare baseline and most-probable-feature SAEs using automatic interpretation.

  • Limitations: Stable and unstable feature membership depends on matching threshold θ and endpoint cutoff ε, with qualitative trends robust but endpoint sizes conditional on these choices.The main experiments use θ = 0.7 and ε = 0.05.
  • Limitations: The low-rank subspace analysis provides evidence for a concrete seed-dependence mechanism rather than a complete explanation of instability in real LLM SAEs.The empirical results and synthetic model show that individually unstable features can arise through this mechanism.
  • A.1 SAE variants: SAE variants differ in sparsity implementation: Vanilla uses ReLU and L1 regularization, TopK and Batch TopK impose top-k sparsity, HierarchicalTopK uses nested budgets, and JumpReLU learns thresholds.TopK fixes sparsity by retaining top-k values, while Batch TopK applies the threshold across a batch.
  • A.2 SAE training setup: Training uses 1B FineWeb activations with shared sampling order across seeds, while an auxiliary loss trains features inactive for more than 20 consecutive batches to reconstruct residual error.The shared sampling seed makes initialization the source of run-to-run differences, and the auxiliary loss targets currently unexplained directions.
  • A.3 Decoder normalization: Decoder weights are normalized during training, and post-training normalization folds hidden-state statistics into the weights while preserving the SAE output.The procedure uses unit-norm decoder columns and rescales encoder weights and bias to counteract post-training normalization.
  • B.1 Evaluation tokens and activation collection: Quantitative statistics are computed on held-out evaluation token positions disjoint from training batches, with Ntok = Beval · T.The evaluation uses base-model activations and nonnegative SAE feature activations at the training location.
  • B.2 Cosine-threshold sensitivity across SAE families; B.3 Usage statistics: activation frequency and mean magnitude; B.4 Token entropy computation: Increasing cosine threshold θ makes matching stricter, reducing stable fractions and increasing unstable fractions while preserving family ordering: TopK and BatchTopK are nearly identical, JumpReLU intermediate, and Vanilla strongest.Usage statistics distinguish activation frequency from conditional mean magnitude; unstable features concentrate at lower frequencies, whereas stable features have a heavier high-magnitude tail.

D Additional Discussion on Stability-Inducing Methods · E Automatic Interpretation and Qualitative Feature Analysis

The paper discusses three routes to more stable SAE dictionaries: regularization or architectural constraints, reconstruction-metric reweighting, and pooling reproducible features across runs. It also supplements the stability analysis with automatic interpretation and explanation-text comparisons of stable and unstable features.

  • D Additional Discussion on Stability-Inducing Methods: Stability-inducing methods fall into regularization or architectural constraints, metric reweighting, and bagging-style feature pooling.These approaches target either the solution space, the reconstruction metric, or aggregation across independently trained SAEs.
  • D Additional Discussion on Stability-Inducing Methods: Vanilla ReLU+ℓ1 SAEs are substantially more stable than TopK SAEs at comparable sparsity, but have lower EV.JumpReLU’s ℓ1-style sparsity regularization may partly explain its improved stability relative to TopK.
  • D Additional Discussion on Stability-Inducing Methods: HierarchicalTopK acts as an implicit regularizer by requiring one dictionary to reconstruct well across nested sparsity budgets.This may reduce degeneracy in the fixed-k TopK objective and make seed-dependent basis choices less likely.
  • D Additional Discussion on Stability-Inducing Methods: Increasing the Mahalanobis/whitening interpolation parameter α can reduce instability but substantially worsens EV.Here α = 0 recovers MSE and α = 1 recovers regularized Mahalanobis loss; reweighting low-variance directions reduces alignment with standard reconstruction quality.
  • D Additional Discussion on Stability-Inducing Methods: Most-probable feature pooling deduplicates decoder features across runs, selects high-probability features, and briefly post-trains the resulting dictionary.The construction can substantially reduce instability while preserving high EV.
  • E Automatic Interpretation and Qualitative Feature Analysis: The qualitative appendix compares stable and unstable features using automatic feature interpretation and explanation-text analysis.These results provide the detailed qualitative evidence summarized in the token-entropy discussion of Section 5.
  • E Automatic Interpretation and Qualitative Feature Analysis: Post-training shifts some pooled features toward lower reappearance probabilities, indicating that lower-stability directions can remain reconstruction-useful rather than random artifacts.Figure 15 compares feature reappearance probabilities before and after post-training.

E.1 Qualitative results from automatic interpretation … F.1 Effective Rank and SVD of the Decoder Submatrices

Automatic interpretations distinguish stable from unstable SAE features: stable features receive higher detection scores and structural explanations, while unstable features are associated with surface-form triggers. Explanation text alone predicts stability with 0.88 accuracy, and the geometric appendix defines size-matched effective-rank and cross-seed SVD analyses for decoder subspaces.

  • E.1 Qualitative results from automatic interpretation: Stable features achieve higher automatic-interpretation detection scores, with substantially more explanations attaining perfect detection at score = 1.Detection score is evaluator accuracy on held-out contexts containing any activation of the feature.
  • E.1 Qualitative results from automatic interpretation: Unstable explanations emphasize surface form, whereas stable explanations describe contextual function, syntactic roles, phrases, constructions, or broader semantic groupings.This distinction corresponds to unstable lower token-entropy triggers and stable higher-entropy lexical or conceptual patterns.
  • E.4 Keyword analysis across stability bins: Among explanations with detection score > 0.7, substring appears in 38.9% of unstable versus 11.3% of stable explanations, while phrase rises from 4.1% to 32.0%.Keyword frequencies are computed as case-sensitive substring matches across stability bins, with confidence intervals reported separately.
  • E.2 Auto-interpretation SAEBench setup: The SAEBench evaluation uses Qwen/Qwen3-32B on 2M held-out FineWeb tokens, with 10 top-activation plus 5 importance-sampled explanation contexts and 2 top, 10 importance-sampled, and 2 random evaluation contexts.Contexts are sampled independently per feature, and generation is capped at 128 tokens.
  • E.5 LLM-only classification from explanation text: GPT-5 classifies stable versus unstable features from anonymized explanation text alone with accuracy 0.88 on a balanced test set.The model receives no activation traces, token examples, decoder vectors, or reappearance probabilities.
  • E.5 LLM-only classification from explanation text: The classification experiment is a lightweight diagnostic whose model- and prompt-specific evidence supports, rather than establishes, the surface-form versus structural distinction.The model’s qualitative summaries consistently describe unstable features through substrings, prefixes, capitalization, punctuation, and narrow triggers, versus stable features through compositional language and discourse roles.
  • F.1 Effective Rank and SVD of the Decoder Submatrices: The geometric appendix defines effective rank for decoder submatrices using size-matched feature sets and estimates mismatched sizes by averaging over B = 50 random subsamples.Effective rank is normalized by hidden-state dimension d = 768, and the appendix also defines explained variance and top-r SVD projectors.
  • F Additional Details for Geometric Analysis: The cross-seed SVD measure evaluates how well seed a’s top-r singular subspace explains the feature subspace learned in another seed.Figure 17 provides full F1 curves for the related linear-separability classifier, while Figure 18 compares full-rank and low-rank synthetic settings across subspace rank and sparsity.

F.2 Additional Results for the Controlled Low-Rank Synthetic Model … G.3 Automatic interpretation on trained vs. random transformers

Additional synthetic-model results show that low-rank features remain individually unstable but recoverable as reproducible subspaces, while residual-based clustering partially recovers their ground-truth structure. Across further SAE analyses, stability varies with depth and training duration, and automatic interpretation scores can be high even without cross-seed reproducibility.

  • F.2 Additional Results for the Controlled Low-Rank Synthetic Model: Across low-rank dimensions and sparsity levels, full-rank features remain stable and well recovered, whereas shared low-rank features are less stable as individual directions.These additional settings preserve the qualitative pattern reported in the main synthetic-model setting.
  • F.2 Additional Results for the Controlled Low-Rank Synthetic Model: The low-rank block has smaller effective rank than the full-rank block, yet its cross-seed subspace similarity remains above the random baseline.Thus, low-rank features are unstable as individual vectors but reconstruct a reproducible shared subspace.
  • F.2.1 Residual-Based Diagnostic for Low-Rank Feature Recovery: Residual-corrected activations cluster around ground-truth low-rank atoms, exactly in the idealized case and approximately as noisy samples under imperfect full-rank recovery.The diagnostic assumes each low-rank atom appears in at least one selected activation.
  • F.2.1 Residual-Based Diagnostic for Low-Rank Feature Recovery: In the d = 32, r = 2, k = 8 toy setting, k-means centroids from residual-corrected activations substantially improve low-rank decoder alignment while largely preserving the identified full-rank block.This is a diagnostic rather than a practical training method because it relies on the toy setup’s known atom count and selected examples.
  • G.1 Endpoint fractions across models, layers, and dictionary sizes: Stable features become more prevalent with layer depth and are much more likely than unstable features to have a next-layer decoder-cosine match at τlayer = 0.7.Figure 21 uses GPT–2 residual-stream activations, K = 64, F = 214, and 21 independently trained SAEs per layer.
  • G.2 Training-token sweeps: setup and measurement details: As training tokens increase from 10M to 10B, instability decreases early toward a non-zero plateau, while the stable fraction continues increasing with diminishing returns.The sweep keeps the main TopK architecture and hyperparameters fixed.
  • G.3 Automatic interpretation on trained vs. random transformers: High automatic detection scores occur for SAEs trained on randomly initialized GPT-2 activations, despite the absence of cross-seed reproducibility.This motivates stability as a complementary faithfulness signal.
Loading 2606.12138v1…