Source-linked AI summary
SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability
Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, Demian Till, Matthew Wearden, Arthur Conmy, Samuel Marks, Neel Nanda
TL;DR
SAEBench addresses the limited practical relevance of traditional unsupervised SAE proxy metrics by introducing a standardized benchmark across diverse evaluations. It evaluates over 200 SAEs and finds that proxy-metric gains do not reliably predict practical performance, with Matryoshka SAEs showing strong feature disentanglement despite slightly worse reconstruction-related results.
Problem
SAE progress is difficult to assess because prior work relies heavily on unsupervised proxy metrics whose relationship to interpretability and practical performance is unclear.
Method
SAEBench evaluates SAEs across eight metrics covering concept detection, interpretability, reconstruction, feature disentanglement, and practical tasks, using over 200 models.
Results
Proxy-metric gains do not reliably translate to practical performance, while Matryoshka SAEs perform strongly on feature disentanglement despite slightly worse reconstruction fidelity.
Takeaways & Limitations
Comprehensive evaluation is needed to compare SAE architectures, expose trade-offs, and guide choices for practical interpretability applications.
Takeaways & Limitations
The Unlearning evaluation is constrained because Gemma-2-2B achieved sufficient baseline performance on only one existing unlearning test set.
Abstract
from arXiv · showhide
Sparse autoencoders (SAEs) are a popular technique for interpreting language model activations, and there is extensive recent work on improving SAE effectiveness. However, most prior work evaluates progress using unsupervised proxy metrics with unclear practical relevance. We introduce SAEBench, a comprehensive evaluation suite that measures SAE performance across eight diverse metrics, spanning interpretability, feature disentanglement and practical applications like unlearning. To enable systematic comparison, we open-source a suite of over 200 SAEs across eight recently proposed SAE architectures and training algorithms. Our evaluation reveals that gains on proxy metrics do not reliably translate to better practical performance. For instance, while Matryoshka SAEs slightly underperform on existing proxy metrics, they substantially outperform other architectures on feature disentanglement metrics; moreover, this advantage grows with SAE scale. By providing a standardized framework for measuring progress in SAE development, SAEBench enables researchers to study scaling trends and make nuanced comparisons between different SAE architectures and training methodologies. Our interactive interface enables researchers to flexibly visualize relationships between metrics across hundreds of open-source SAEs at: www.neuronpedia.org/sae-bench
1. Introduction
Evaluating SAE effectiveness remains difficult because recent improvements are mainly measured with unsupervised proxy metrics whose relationship to interpretability is unclear. SAEBench addresses this gap by evaluating multiple practitioner-relevant properties.
- SAEs are widely used to find interpretable units in language models through dictionary learning, but measuring interpretability improvements remains challenging.
- Ground-truth labels for language models’ internal features are unavailable, so researchers often optimize sparsity and fidelity as unsupervised proxies.
- Reconstruction at a fixed sparsity can produce interpretable latents, yet sparsity may cause feature absorption or composition of independent latents.
- SAEBench evaluates concept detection, interpretability, feature disentanglement, and reconstruction to provide a broader picture of SAE quality.
1. SAEBench: a standardized suite of eight evaluations
SAEBench combines a broad evaluation suite with large-scale comparisons of SAE designs. Its analyses show that traditional proxy metrics can miss strong performance on concept detection and feature disentanglement.
- The benchmark introduces two novel metrics for feature disentanglement alongside established evaluations.
- The benchmark trains and evaluates over 200 SAEs spanning architectures, training methodologies, and widths.
- SAEBench’s nuanced evaluation reveals trends in architecture choice, scaling, and training dynamics that traditional metrics can miss.
- Matryoshka SAEs perform well on feature disentanglement and concept detection despite appearing worse on existing proxy metrics.
2. Related work
SAEs decompose language-model activations into sparse feature representations, while recent variants modify architecture, activation, or loss design. Existing evaluation has largely centered on sparsity and reconstruction, despite limited correspondence with interpretability.
- A basic SAE encodes activation x into a sparse higher-dimensional representation h and decodes it into reconstructed activation ˆx.
- The standard objective combines reconstruction error with an L1 sparsity penalty weighted by λ.
- Recent SAE variants change network structure, activation functions, or loss functions, including Gated, TopK, BatchTopK, JumpReLU, and Matryoshka SAEs.
- Most innovations optimize the sparsity-fidelity trade-off, but this metric may not directly measure desirable properties such as interpretability.
- Existing benchmarks target specific SAE properties, encouraging continued reliance on sparsity-fidelity optimization despite its known limitations.
3. SAEBench: A Comprehensive Benchmark
SAEBench is a unified, reproducible framework that evaluates SAEs across multiple capabilities and eight metrics, combining practical and diagnostic tests. It operationalizes interpretability, reconstruction, concept detection, and feature disentanglement through complementary procedures.
- SAEBench requires diverse, extensible, fast, automated, and deterministic evaluations for practical comparison of SAE architectures.
- The framework organizes evaluation around concept detection, interpretability, reconstruction, and feature disentanglement.
- Concept Detection: Concept detection uses Sparse Probing and Feature Absorption to assess whether individual latents correspond precisely to meaningful concepts.
- Interpretability: Interpretability uses an LLM judge that predicts latent activation on test sequences from descriptions generated from activating examples.
- Reconstruction: Reconstruction measures how faithfully an SAE preserves model behavior using Loss Recovered metrics.
- Feature Disentanglement: Feature disentanglement evaluates separation of independent concepts through Unlearning, Spurious Correlation Removal, and Targeted Probe Perturbation.
- Feature Disentanglement: RAVEL measures disentanglement by averaging Cause and Isolation metrics after targeted latent interventions.
- Concept Detection: Feature absorption describes sparsity-driven representations in which hierarchical concepts are split into a concept and a residual exception feature.
4. Results
Across architectures, Matryoshka Batch TopK excels on concept detection and feature disentanglement despite weaker sparsity-fidelity performance, while metric rankings and scaling trends vary substantially by task, width, and L0.
- 4.1. Comparing SAE Architectures: Matryoshka Batch TopK performs best on concept detection and feature disentanglement, leading on 5 of 8 metrics in the typical L0 range of 40-200.It leads on Absorption, RAVEL, Sparse Probing, SCR, and TPP, but underperforms TopK and BatchTopK on the sparsity-fidelity frontier.
- 4.1. Comparing SAE Architectures: The ReLU SAE is outperformed by other methods on 5 of 8 metrics, including loss recovered, absorption, SCR, RAVEL, and TPP.However, 65k-width ReLU SAEs with L0 > 200 perform best overall on 1-sparse probing and show comparable unlearning performance.
- 4.1. Comparing SAE Architectures: The sparsity-fidelity frontier does not reliably predict downstream-task performance, because architecture rankings differ across metrics and L0 regimes.Matryoshka combines middling frontier performance with strong SCR and absorption, while p-anneal and Gated SAE reverse their relative ranking across absorption and loss recovered.
- 4.2. Dictionary Size Scaling Dynamics: Automated Interpretability and Loss Recovered generally improve as dictionary size increases from 4k to 16k to 65k latents across architectures.These trends indicate better reconstruction and more interpretable individual latents at larger dictionary sizes.
- 4.2. Dictionary Size Scaling Dynamics: Matryoshka is the only architecture improving feature disentanglement with scale, while most architectures show inverse scaling on disentanglement and concept-detection metrics.Absorption worsens with dictionary size for every architecture except Matryoshka, and SCR generally declines for non-hierarchical architectures even when intervention size is controlled.
- 4.3. Task-Dependent Optimal Sparsity: Moderate L0 values of 50-150 offer a reasonable compromise across metrics, because no single sparsity level is optimal for all tasks.Higher L0 improves reconstruction fidelity, RAVEL, targeted probe perturbation, and reduces feature absorption, whereas moderate L0 can favor sparse probing and spurious-correlation removal.
5. Limitations
SAEBench’s supervised metrics are constrained by limited ground-truth concepts, qualitative interpretability, model coverage, and incompatible metric scales. These boundaries limit what the benchmark can measure and how its results can be combined.
- Ground-truth concepts cover only a small portion of the concepts encoded in language models, limiting supervised metrics’ coverage of SAE latents.This limited concept set can make some metrics relatively noisy.
- Automated metrics may miss nuanced interpretive insights obtained through manual investigation of SAE latents.
- The evaluation covers Gemma-2-2B and Pythia-160M but not all model architectures, scales, or layers.SAE behavior may vary across these settings, so broader generalization remains untested.
- Different users prioritize different SAE properties, while metric scales and noise levels make a single aggregate score arbitrary.Arbitrary weighting could obscure trade-offs between performance dimensions.
6. Conclusion
SAEBench extends SAE evaluation beyond sparsity and fidelity to capture multiple performance dimensions. Its results expose architecture- and scale-dependent trade-offs that traditional proxy metrics can miss, supporting broader evaluation for practical SAE development.
- SAEBench evaluates SAE performance across multiple dimensions rather than only the traditional sparsity-fidelity frontier.
- Matryoshka SAEs substantially outperform other architectures on feature disentanglement despite slightly worse reconstruction fidelity.
- Larger dictionaries generally improve reconstruction and per-feature interpretability but can degrade concept isolation in non-hierarchical architectures.
- Optimal sparsity varies by task, while moderate L0 values of 50-150 provide a reasonable compromise across most metrics.
- Comprehensive evaluation should include downstream task performance and feature disentanglement alongside sparsity-fidelity optimization.
- SAEBench provides a standardized suite intended to reveal hidden trade-offs and accelerate SAE development.
Impact Statement
The paper positions SAEBench as a tool for improving SAE evaluation and interpretability research. Its computational requirements are substantial and vary with evaluation and hardware choices.
- SAEBench aims to help researchers develop interpretable models, diagnose failure modes, and better understand model representations.
- Evaluations use an initial setup phase whose preprocessing can be reused across multiple SAE evaluations, followed by per-SAE runtimes.
- 107 minutes of setup and approximately 65 minutes per SAE are required to run all benchmarks on the reported configuration.Actual runtimes vary with dictionary size, base model, and GPU selection.
B. SAE Training Details
The benchmark standardizes SAE training and evaluates models with core reconstruction and sparsity metrics alongside interpretability, probing, and disentanglement tests.
- Training setup: All SAEs use directly comparable data, ordering, and fixed variables across layer, width, and type combinations.Targets include six L0 values, although sparsity-penalty SAEs may not exactly reach them.
- Training setup: Activation normalization uses a learned scalar folded into the weights after training, so inference does not require normalized activations.The scalar is estimated to give activations unit mean squared norm during training.
- Training setup: The decoder is initialized as the encoder transpose but is not tied during training, reducing dead-latent risk.Transpose initialization was important for avoiding latents that never activate.
- Evaluation suite: Evaluation combines core sparsity and reconstruction metrics with automated interpretability, concept probing, and RAVEL disentanglement tests.The probing suite covers 35 binary classification tasks from five datasets, while RAVEL tests selective interventions without unintended attribute changes.
- Evaluation suite: Automated interpretability generates feature explanations from highly activating sequences and scores them by predicting activations on shuffled test sequences.The test sets combine random, maximally activating, and importance-weighted sequences.
- Evaluation suite: Feature absorption is averaged over correctly classified vocabulary inputs filtered to tokens containing English letters and optional leading spaces.The metric is zero for inputs that do not satisfy the stated criteria.
E. Baseline Comparison: SAEs on Randomly Initialized vs. Fully Trained Models
Comparisons between SAEs trained on randomly initialized and fully trained models show stronger supervised and reconstruction results for trained models, but some metrics are not directly comparable across models.
- Supervised metrics: Fully trained-model SAEs significantly outperform random-model SAEs on most supervised metrics, especially sparse probing and SCR.Random-model SAEs perform close to or worse than directly probing residual-stream values.
- Automated interpretability: Fully trained-model SAEs achieve slightly higher automated interpretability scores, while both SAE sets outperform direct residual-stream readings.The evaluation uses detection-based auto-interpretability rather than the fuzzing method used in related work.
- Baseline comparison: SAEs consistently and significantly outperform MLP neurons, PCA components, and direct residual-stream readings in the cited baseline comparison.This result is reported for Figure 5's auto-interp comparison.
- Reconstruction: Fully trained-model SAEs achieve significantly higher KL divergence scores, indicating superior reconstruction quality despite higher absolute KL divergence values.Randomly initialized models produce essentially random predictions, leaving limited opportunity for meaningful reconstruction improvement.
- Targeted probe perturbation: Random-model SAEs obtain higher TPP scores, but cross-model comparison is potentially misleading because TPP measures within-model relative probe-accuracy changes.Random-model probes begin from substantially lower baseline accuracy.
- Scaling analysis: The benchmark’s scaling analyses compare width increases from 16k to 65k while separating aggregate trends from sparsity-specific behavior.Figure 6 varies L0 values, and Figure 7 examines SCR across intervention budgets.
H. Intervention Set Size Analysis
The analysis varies intervention or probing set size K to examine concept isolation and detection. K = 20 is used as a practical default for manual analysis, while relative architectural differences generally remain consistent from K = 5 to 50.
- SCR and TPP use K latents for zero ablation, whereas sparse probing selects K latents for probing.
- K = 20 is the main setting because it balances complete concept capture with practical human analysis.
- Relative architectural performance differences remain consistent for K from 5 to 50 but break down at K = 500.
- Figure 9 compares SCR scores across intervention set sizes K and architectures to assess removal of spurious correlations.
- Figure 10 compares TPP scores across intervention set sizes K, showing how ablated-latent count relates to concept isolation.
- Figure 11 compares sparse probing performance across numbers of probed latents K to show concept detection with additional latents.
I. Gemma-Scope Evaluation Results
The Gemma-Scope width series evaluates SAE scaling across Gemma-2-2B and Gemma-2-9B layers, revealing consistent metric-specific and layer-dependent trends. Reconstruction and automated interpretability improve with width, while several disentanglement metrics degrade; unlearning varies strongly by layer.
- The Gemma-Scope width series studies SAEs ranging from 16k to 1M latents across selected layers of Gemma-2-2B and Gemma-2-9B.
- SAEs on the final evaluated layers achieve unlearning scores near zero regardless of width.
- Loss Recovered and AutoInterp improve consistently as SAE width increases.
- Feature Absorption, SCR, and TPP scores degrade at larger widths.
- Unlearning is most effective at earlier layers and varies significantly by layer.
- Sparse Probing scores increase at later layers.
- These scaling patterns align with the main architecture comparison, preserving trade-offs between reconstruction fidelity and feature disentanglement at larger scales.
J. Further SAE Bench Evaluation Results
Further evaluations report scores for the SAEBench metrics across SAE suites spanning multiple dictionary sizes and two model families. The figures organize results by suite size for Gemma-2-2B and Pythia-160M.
- Figures 18–20 report scores for the 4K, 16K, and 65K Gemma-2-2B suites.
- Figures 21–23 report scores for the 4K, 16K, and 65K Pythia-160M suites.