Source-linked AI summary
Diminishing Returns of Data in Sparse Autoencoders: Functional Behavior does not Guarantee Feature Reproducibility
Sambit Ghosh, Madelyn Mathai, R. Venkatesh Babu, Chirag Agarwal
TL;DR
The paper addresses limited evidence on whether SAE training needs the large activation budgets used in practice. It combines theory with experiments across architectures, models, and data budgets, finding that functional metrics can approach large-data results earlier than reproducible feature recovery does.
Problem
Method
Results
Takeaways & Limitations
Match SAE training scale to its intended use: smaller budgets can often preserve functional behavior, but assess reproducibility when individual latents are scientific objects.
Source: §7 · p. 9The study relies on SAEBench training and evaluation implementations, and its RAVEL and unlearning evaluations are supported only for Gemma models.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Sparse autoencoders (SAEs) are typically trained on hundreds of millions to billions of activation tokens, yet little work has tested whether this scale is necessary. We study how much activation data SAEs need by training four SAE architectures on three language models with budgets ranging from 5M tokens to a 500M-token reference. We measure two notions of convergence: functional behavior on standard evaluations and one-to-one reproducibility of learned features. Functional data requirements are strongly metric-dependent: reconstruction often approaches the full-data reference at substantially smaller budgets, whereas feature reproducibility continues to improve with data. For Gemma-2-2B BatchTopK, the fraction of strictly matched features rises from 2\% at 5M tokens to 29\% at 50M. Data selection also matters: random sampling better preserves the original activation distribution and generally favors reconstruction, whereas diversity-based sampling frequently improves AutoInterp. A theoretical analysis separates statistical consistency of subsampled training from the stronger identifiability and optimization conditions required for one-to-one feature recovery. Our results show that SAE training can often be made far more data-efficient when budgets are matched to the goal, and feature-level claims are backed by explicit reproducibility checks, paving the way toward cheaper and more reliable SAE research.
1 Introduction
The paper asks whether SAE training needs the large activation budgets common in practice, distinguishing functional performance from reproducibility of learned features. It combines a theoretical account of feature recovery with a broad empirical study of data requirements.
1 Introduction
- The paper asks how much activation data SAEs need and distinguishes functional behavior from feature convergence under one-to-one matching.It develops a theoretical framework for when additional data can lead to feature recovery, while noting that reducing data alone does not guarantee dictionary convergence.Source: §1 · p. 1
- Earlier work used Hungarian matching and encoder-decoder agreement to quantify feature correspondence across independently trained dictionaries.This study asks how such reproducibility depends on activation data quantity.Source: §1 · pp. 1–2
- Across three language models and four SAE architectures, the study finds functional metrics have distinct data-scaling behavior while one-to-one feature reproducibility continues to improve with more activation exposure.The authors also examine multiple configurations and activation budgets.Source: §1 · p. 2
- Small budgets can recover much of the 500M-token SAE’s reconstruction and several functional evaluations, while reproducible feature recovery becomes more reliable with larger budgets.The paper reports that this depends on the evaluation objective.Source: §1 · p. 2
2 Related Work
Prior work has expanded SAE architectures and evaluations while finding that learned features can vary across runs even when their spans remain similar. The paper identifies a lack of systematic tests of whether large activation budgets are necessary or selected subsets can suffice.
2 Related Work
- SAE evaluation has broadened beyond reconstruction to absorption, sparse probing, unlearning, and automated interpretability, compiled in SAEBench.Architectural variants include JumpReLU [14], TopK and BatchTopK [7], and Matryoshka SAEs.Source: §2 · p. 2
- SAE feature stability studies report seed-dependent feature differences [10] [11], with later work suggesting unstable features may carry little signal and cluster in reproducible low-rank subspaces [10].Lan et al. [11] find feature-spanned spaces remain similar, including across language models.Source: §2 · p. 2
- Prior data-selection work includes coreset methods and farthest point sampling [22] [23], while Yang et al. [27] prune instruction-tuning data rather than SAE training activations.This paper instead tests whether activation choice matters beyond activation quantity.Source: §2 · p. 2
- No prior work had systematically tested whether standard hundreds-of-millions-token training scales are necessary or whether selected activation subsets can match larger-budget SAEs on established metrics [28].This is the research gap the paper addresses.Source: §2 · p. 2
3 When Should Feature Matching Agree?
The theoretical account gives conditions under which full- and reduced-data SAEs recover the same feature system, and distinguishes that recovery from mere agreement between encoder and decoder assignments. In particular, matching assignments at finite budgets need not mean strongly aligned feature directions.
3 When Should Feature Matching Agree?
- Encoder and decoder one-to-one distances compare feature systems after permutation alignment, with Frobenius distance zero exactly when feature directions match.The encoder and decoder feature systems are denoted E(θ) and D(θ).Source: §3 · pp. 2–3
- Feature recovery under data reduction requires a common population objective, statistical and optimization consistency, and identifiable population feature systems separated from alternatives.Under these assumptions, full- and reduced-data training approach minimizers sharing one feature system up to permutation.Source: §3 · p. 3
- Encoder-decoder assignment agreement does not imply high encoder and decoder cosine similarity at a finite data budget.A shared assignment means both matchings independently select the same reference counterpart, even if the corresponding cosine similarities are below a fixed threshold.Source: §3 · p. 3
- The assumptions clarify why asymptotic feature-system preservation is stronger than finite-budget assignment agreement, motivating separate measures of counterpart agreement and strict shared-feature recovery.A unique optimal one-to-one encoder and decoder matching with a fixed margin is also assumed for the matching-agreement result.Source: §3 · p. 3
4 Experimental Setup
The experiments vary activation quantity and selection while following SAEBench configurations to keep reduced-data runs comparable with a large-data reference. They cover three language models, four SAE architectures, online FineWeb training, and standard functional and feature-reproducibility evaluations.
4 Experimental Setup
- Small-data experiments follow corresponding SAEBench training and evaluation configurations so their results are directly comparable with the large-data reference.The study varies activation quantity and activation selection.Source: §4 · p. 4
- The study evaluates three language models and four SAE architectures, holding model, activation site, dictionary width, sparsity, and optimization hyperparameters fixed within comparisons.The models are Gemma-2-2B, Pythia-160M, and Qwen3-8B; architectures are BatchTopK, Matryoshka BatchTopK, JumpReLU, and ReLU.Source: §4 · p. 4
- Online training streams FineWeb text through a frozen language model, buffers target-layer activations, and consumes selected activations once before refreshing the buffer.Training configuration and optimization steps are fixed while the retained activation fraction varies.Source: §4 · p. 4
- The reference uses 500M activation tokens, while reduced-data runs retain 1%, 5%, or 10% of activations from each fresh online buffer.The retained activations are consumed once rather than drawn from a fixed replay dataset.Source: §4 · p. 4
- Random sampling and farthest-point sampling compare value-independent selection with a strategy that promotes geometric diversity.Farthest-point sampling retains points far from those already selected.Source: §4 · p. 4
- Evaluation uses SAEBench functional metrics and one-to-one Hungarian matching of encoder and decoder features against the 500M reference.The evaluations include reconstruction, sparse probing, absorption, SCR, TPP, RAVEL, automated interpretability, and unlearning.Source: §4 · p. 4
5 Functional Data Requirements Are Metric Dependent
Functional data requirements vary by metric: reconstruction and several other evaluations can remain close to the 500M-token reference at smaller budgets, while feature-level and intervention metrics show varied, often non-monotonic responses. At matched budgets, random sampling and Farthest Point Sampling favor different evaluation properties.
Concept localization and feature alignment
- Sparse-probing accuracy remains broadly consistent across 5M–500M budgets, and reduced-data SAEs frequently outperform the 500M reference on feature absorption.Smaller budgets sometimes outperform the reference on sparse probing as well.Source: §5 · pp. 4–5
Reconstruction
- At 25M and 50M tokens, reconstruction is generally very close to the 500M reference, while 5M shows a modest gap.The comparison covers BatchTopK with 16k latents and standard ReLU with 65k latents on Gemma-2-2B layer 12 residual-stream activations.Source: §5 · p. 4, Figure 1 · p. 5, Figure 2 · p. 6
Automated interpretability
- AutoInterp varies moderately and non-monotonically with activation budget, with its ordering depending on SAE configuration and smaller-budget SAEs sometimes scoring higher.The reported configuration factors include sparsity and dictionary width.Source: §5 · p. 5
Intervention metrics
- SCR varies more with activation budget than reconstruction, but increasing data does not consistently improve intervention specificity.Reduced-data performance also depends on width and architecture.Source: §5 · p. 5
- TPP stays close to the 500M reference across dictionary sizes despite non-monotonic budget ordering, with small absolute deviations.The authors suggest this class-specific feature-separation measure requires relatively little additional data.Source: §5 · pp. 5–6
- RAVEL varies non-monotonically with budget and architecture, while unlearning varies substantially with budget, architecture, and sparsity.Reduced-data SAEs span a wide range relative to the 500M reference on unlearning.Source: §5 · p. 6
Data selection
- Across dictionary sizes and budgets, random sampling keeps reconstruction loss closer to the 500M reference than FPS, most visibly at small budgets.Random sampling preserves the activation stream’s frequency structure in expectation; FPS selects for geometric coverage.Source: §5 · pp. 6–7
- FPS outperforms random sampling on AutoInterp in 79% of matched comparisons, while random sampling favors SCR, TPP, and RAVEL and FPS can favor unlearning at some budgets.The selection strategies therefore favor different properties rather than producing uniformly better or worse SAEs; FPS is associated with higher-norm activations and lower reconstruction scores.Source: §5 · pp. 6–7
6 Functional Similarity Does Not Imply Feature Reproducibility
Functional similarity can emerge at smaller activation budgets than strong feature reproducibility. Feature correspondence improves with more data, while data-selection strategies also differ in how they preserve the activation distribution.
Data selection
- Random selection preserves the activation-norm distribution within sampling error, whereas FPS concentrates on higher-norm activations and shifts further as the subset shrinks.Table 1 reports random-selection KL ≤0.0003 bits; Δmean norm, KL, and mean-direction cosine describe distribution shift and alignment.Source: Table 1 · p. 8, §6 · p. 8
Counterpart Agreement Emerges Before Strong Feature Recovery
- At 5M tokens, encoder and decoder matchings select the same reference counterpart for 66.6% of Gemma-2-2B BatchTopK latents, but only 2.20% meet the strict criterion.The strict criterion requires both cosine similarities to exceed τ = 0.7.Source: §6 · p. 8
Feature Reproducibility Improves Systematically with Data
- 28.86% of Gemma-2-2B BatchTopK features meet the strict shared-feature criterion at 50M tokens, up from 2.20% at 5M.Encoder-decoder agreement rises from 66.61% to 83.47% over the same budgets.Source: §6 · p. 8, Table 2 · p. 8
Qualitative Examples Illustrate the Same Distinction
- Low- and medium-alignment matched ReLU features can have different semantic descriptions, unlike the highly aligned example, which captures the same pattern.The table presents representative 5M-to-500M Gemma-2-2B ReLU pairs across alignment levels.Source: §6 · pp. 8–9, Table 3 · p. 9
Functional Similarity Does Not Imply Feature Reproducibility
- By 25M–50M tokens, reconstruction and several functional metrics are close to 500M reference values in many regimes, while feature recovery remains incomplete.One-to-one feature correspondence strengthens consistently with exposure even when functional metrics do not improve monotonically.Source: Table 2 · p. 8, §6 · p. 9
7 Conclusion
The paper studies SAE data requirements through theory, functional evaluations, and feature reproducibility. It finds that functional behavior often approaches large-data references before individual features become strongly reproducible, so data needs depend on the objective.
Conclusion
- Across four architectures and three language models, many SAEBench metrics approach the large-data reference before the full activation budget, while strict feature correspondence continues to improve with data.The theoretical framework distinguishes convergence to the same population objective from the additional identifiability and optimization conditions needed for one-to-one feature recovery.Source: §7 · p. 9
- Training scale should match the intended use: smaller budgets can often preserve reconstruction and aggregate functional behavior, but individual-latent claims warrant explicit reproducibility checks.The authors advise against defaulting to the largest corpus for functional evaluation.Source: §7 · p. 9
Limitations
- The study relies on SAEBench implementations and inherits their assumptions and metric limitations; RAVEL and unlearning evaluations are supported only for Gemma models.The latter boundary prevents those evaluations for Qwen and Pythia in this study.Source: §7 · p. 9
A Experimental details and additional Functional Data-Scaling Results
The appendix specifies SAE training and activation-extraction settings, then lists functional-evaluation tables across activation budgets, selection strategies, models, and architectures.
Experimental setup
- Training used batch size 2048, dictionary widths 16,384 and 65,536, and a default full-data budget of 500M activation tokens.The setup also specifies warmup, decay, token filtering, and model-specific activation-extraction settings.Source: A Experimental details and additional… · p. 12
Trainer configurations
- The per-trainer configuration maps L1 sparsity penalties to trainer indices by dictionary width, while BatchTopK and JumpReLU use specified operating points.BatchTopK uses k values of 80 and 160; JumpReLU uses target activation counts of 80 and 160.Source: A Experimental details and additional… · p. 12
- Matryoshka BatchTopK uses k values of 80 and 160 at both dictionary widths, with auxiliary-loss and threshold-tracking settings specified.Threshold tracking begins after 1000 steps.Source: A Experimental details and additional… · p. 12
Functional evaluation tables
- The Gemma-2-2B ReLU (16k) table compares Random and FPS at 5M, 25M, and 50M tokens against corresponding 500M baselines.The supplied table description gives its scope but not cell values.Source: Table 4 · p. 13
B.1 Separate encoder and decoder feature alignment 1
The proof separates statistical consistency from conditions on optimization and feature identifiability when reasoning about feature recovery under reduced data.
Proof step
- The proof invokes statistical consistency as an assumption in deriving a statement about a1,N.The supplied passage does not state the full conclusion.Source: §B.1 · p. 25
Feature recovery conditions
- The text defines optimization error ηr,N through the empirical objective's excess over its infimum.The bound compares the empirical risk at the trained parameters with the best empirical risk.Source: §B.1 · p. 25
- Feature identifiability and separation imply a positive population excess-risk gap for encoder features that remain at least ϵ from the population encoder orbit.The gap is denoted ∆E(ϵ).Source: §B.1 · p. 25
Proof step
- A triangle-inequality step is applied separately in encoder and decoder feature spaces.The preceding text refers to p−→0.Source: §B.1 · p. 25
B.2 Proof of Agreement of encoder and decoder Hungarian matchings) 2
The proof links separate encoder and decoder alignment to shared-label feature agreement under stated assumptions. The accompanying results describe selection effects and increasingly strict feature-correspondence criteria.
Matching proof
- The proof says normalized encoder and decoder feature systems converge to their population systems after one-to-one alignment.The alignment argument assumes a shared labeling within each run and uses a permutation between runs; positive feature separation stabilizes the optimal Hungarian assignment with probability tending to one.Source: §B.2 · p. 25
Selection distribution shift
- Random selection is distribution-neutral at every budget, while FPS shifts subsets toward higher-norm activations, with divergence growing as subsets shrink.The stated pattern holds across Gemma-2-2B, Qwen3-8B, and Pythia-160M.Source: Table 24 · p. 26
Feature correspondence
- The feature-correspondence table reports encoder-decoder agreement and shared features under cosine-similarity thresholds for Qwen3-8B and Pythia-160M.Shared features additionally require both encoder and decoder cosine similarities to exceed τ.Source: Table 25 · p. 26
Statistical tests
- The paired-test table compares 500M with Random subsets and FPS with Random at matched budgets using paired two-sided t-tests across matched configurations.The supplied table description specifies the test design, not its outcomes.Source: Table 26 · p. 27
Functional results
- Random generally favors reconstruction and SCR, whereas FPS frequently improves AutoInterp.Reduced-data SAEs become increasingly comparable to the 500M reference as budgets grow, with several metrics reaching parity at small budgets.Source: §B.2 · p. 27