PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing

Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, Neel Nanda

arXiv:2502.16681v1cs.LGcs.AI

TL;DR

SAEs lack direct ground-truth validation, motivating evaluation on whether they improve practical LLM activation probing beyond strong baselines. The paper tests SAE probes across difficult regimes and datasets, finding no consistent ensemble advantage and concluding that rigorous baseline comparisons are essential while leaving utility on other tasks open.

  • Problem

    SAEs lack a ground truth for validating their interpretations, while proxy metrics such as reconstruction loss may not align with mechanistic interpretability’s goal.

  • Method

    The paper evaluates SAE-based activation probes across data scarcity, class imbalance, label noise, and covariate shift using diverse datasets and baseline comparisons.

  • Results

    SAE probes provide no consistent advantage over baselines across multiple regimes and over 100 datasets, and improved non-SAE baselines match initially promising SAE results.

  • Takeaways & Limitations

    Interpretability methods should be evaluated on downstream tasks with rigorous, strong baselines rather than relying only on proxy metrics.

  • Takeaways & Limitations

    Probing performance is itself only a proxy for SAE utility, despite being viewed as more informative than reconstruction error or downstream cross-entropy loss.

Abstract

from arXiv · show

Sparse autoencoders (SAEs) are a popular method for interpreting concepts represented in large language model (LLM) activations. However, there is a lack of evidence regarding the validity of their interpretations due to the lack of a ground truth for the concepts used by an LLM, and a growing number of works have presented problems with current SAEs. One alternative source of evidence would be demonstrating that SAEs improve performance on downstream tasks beyond existing baselines. We test this by applying SAEs to the real-world task of LLM activation probing in four regimes: data scarcity, class imbalance, label noise, and covariate shift. Due to the difficulty of detecting concepts in these challenging settings, we hypothesize that SAEs' basis of interpretable, concept-level latents should provide a useful inductive bias. However, although SAEs occasionally perform better than baselines on individual datasets, we are unable to design ensemble methods combining SAEs with baselines that consistently outperform ensemble methods solely using baselines. Additionally, although SAEs initially appear promising for identifying spurious correlations, detecting poor dataset quality, and training multi-token probes, we are able to achieve similar results with simple non-SAE baselines as well. Though we cannot discount SAEs' utility on other tasks, our findings highlight the shortcomings of current SAEs and the need to rigorously evaluate interpretability methods on downstream tasks with strong baselines.

1. Introduction

The paper argues that SAE interpretations lack direct ground-truth validation and evaluates whether SAEs improve a practical activation-probing task. It motivates probing as useful and hypothesizes that interpretable SAE latents could help in difficult settings.

  • Motivation: SAEs lack a ground truth for determining whether they extract the interpretable concepts used by language models.Prior work often relies on reconstruction loss and related proxy metrics, which may not align with mechanistic interpretability’s goal.
  • Related work: Prior positive downstream results for SAEs are limited, often focusing on individual examples and baselines with varying rigor.The paper contrasts these results with a broader evaluation intended to assess comparative utility more systematically.
  • Motivation: The paper proposes evaluating SAE utility through activation probing, a practical task used to investigate representations, detect safety-relevant quantities, remove knowledge, and catch sleeper agents.The authors frame probing as a concrete downstream task for testing whether SAEs provide a competitive advantage.
  • Research question: The study examines whether SAE probes help across data scarcity, class imbalance, label noise, and other difficult probing settings.The motivation is that concept-level, interpretable SAE latents might provide a useful inductive bias when concepts are difficult to detect.
  • Research question: The paper also investigates whether SAE probes help identify multi-token concepts and dataset issues, but reports that improved baselines can achieve the same results.This motivates rigorous baseline design when evaluating interpretability techniques.

2. Methodology

The methodology evaluates probes on diverse binary classification datasets using model activations, SAE latent selection, multiple baseline classifiers, and validation-based model selection. It compares adding SAE probes to a baseline toolkit through the Quiver of Arrows framework.

  • Models and SAEs: The study uses Gemma-2-9B and Llama-3.1-8B activations with JumpReLU and TopK SAEs, respectively.Main-paper results use Gemma-2-9B, with core results replicated on Llama-3.1-8B in an appendix.
  • Classification datasets: The dataset collection contains 113 diverse binary classification tasks, including examples designed to challenge probes.Prompts range from 5 tokens to a left-truncated maximum of 1024 tokens, and targets are 0 or 1.
  • Activation probing: Probes predict binary targets from model hidden activations, primarily using the final token representation.The authors focus on the last token to emulate common probing studies while later considering multi-token probes.
  • SAE probing strategy: SAE probes encode activations into latent vectors, select the k latents with the largest average absolute class difference, and train a probe on those latents.The selected latent basis is intended to isolate a small number of SAE features associated with the target.
  • Evaluation: The evaluation uses AUC, selecting hyperparameters by validation AUC before measuring performance on a held-out test set.AUC assesses ranking quality independently of a classification threshold.
  • Baselines: Baseline methods include logistic regression, PCA regression, KNN, XGBoost, and MLP, while SAE probes use logistic regression.The comparison tests whether SAE probes add value beyond a varied set of conventional probing methods.
  • Quiver of Arrows: The Quiver of Arrows compares the best validation-selected method with and without SAE probes, measuring whether SAE access improves held-out test performance.This avoids selecting among methods using the held-out test set and provides a practitioner-oriented counterfactual.
  • SAE configuration: For a fixed SAE width, higher L0 and a larger latent basis k produce more performant probes.This relationship is reported as an experimental setup result rather than as the paper’s main cross-dataset conclusion.

3. Comparing Probing Techniques in Different Regimes

Across standard and challenging probing regimes, SAE probes do not consistently improve over baseline methods. They slightly reduce standard-condition performance, show no average gain under data scarcity, class imbalance, or label noise, and often generalize worse under covariate shift.

  • 3.1. Standard Conditions: Larger k leads to better SAE probe performance, while the study uses k = 16 for interpretability and k = 128 for performance.The preliminary investigation finds SAE width relatively unimportant, whereas larger L0 values improve probe performance.
  • 3.1. Standard Conditions: SAEs are selected as the arrow for 14/113 standard-condition tasks, yet adding them to the quiver slightly decreases performance.The experiments use layer 20, where baseline methods perform best under standard conditions.
  • 3.2. Data Scarcity, Class Imbalance, and Label Noise: The label-noise evaluation compares logistic regression directly with a width = 16k, k = 128 SAE probe because corrupted validation data is not representative of test data.The quiver method is used for data scarcity and class imbalance, but not label noise.
  • 3.2. Data Scarcity, Class Imbalance, and Label Noise: Across all parameter values in data scarcity, class imbalance, and label noise, SAEs show no meaningful average improvement over baseline quivers.This lack of improvement is not explained by SAEs being absent from the selected quivers: they are chosen for up to 40 datasets in each regime but underperform baselines.
  • 3.3. Covariate Shift: Under covariate shift, baseline logistic regression outperforms SAE probes when generalizing to out-of-distribution data.The covariate-shift evaluation uses eight OOD datasets, including altered language, syntactic changes, and extreme GLUE-X tasks.

4. Interpretability

The paper uses autointerpretability to examine why SAE probes fail, whether their latents reveal dataset problems, and whether these benefits exceed baseline methods. SAE analyses expose some spurious or misleading features, but comparable findings are often obtainable with non-SAE classifiers.

  • Probe Interpretability: Autointerpretability examines whether SAE probes fail because they use spurious correlations or because their underlying latents are not robust under distribution shift.The study focuses on pruning supposedly spurious latents and testing whether individual latents generalize across transformations.
  • Probe Interpretability: Pruning increased OOD AUC by 0.024 for 66 living-room, 0.052 for 90 glue qnli, and 0.077 for 7 hist fig ispolitician.The authors also report that 7 hist fig ispolitician requires all k = 8 latents, unlike the other two tasks where pruning helps.
  • Latent Interpretability: Autointerpreted latent descriptions sometimes fit tasks, but can also be incorrect, such as latent 50817’s misleading description despite 0.989 AUC for Italian concepts.Another example is latent 10555, which classified Obama-era headlines with 0.782 AUC while activating on numerical strings.
  • Latent Interpretability: SAE-based autointerpretability offers infrastructure and decomposability advantages, but the reported findings may also be obtained with baseline classifiers.Baseline classifiers can inspect maximally activating hidden-state examples, although SAE latents can expose multiple independent features of varying importance.
  • Detecting Dataset Quality Issues: In CoLA, latent 369585 identified apparently mislabeled grammatical examples, yet logistic regression also identified the same dataset errors.On examples where ensembled clean labels disagreed with original CoLA labels, latent 369585 outperformed the dense SAE probe, which outperformed the baseline.
  • Detecting Dataset Quality Issues: For human-versus-AI text, latent 105150 exposed punctuation and final-token differences, but logistic regression applied to Pile tokens reached the same spurious-feature conclusion.AI text more often ended in a period, while human text more often ended in a space.

5. Why Didn’t This Work: Illusions of SAE Probes

The apparent advantage of SAE probes can depend on weak or mismatched baselines and pooling choices. Stronger attention-pooled baselines substantially reduce the observed SAE win rate.

  • Cross-dataset comparison: SAE probes outperform baselines on only 2.2% of datasets, suggesting that single-dataset improvements may not generalize across the benchmark.The authors cannot determine whether Bricken et al.’s dataset belonged to this small subset.
  • Pooling choices: Max-pooled SAE probes win 19.6% of comparisons against last-token baselines, versus 2.2% for last-token SAE probes.The comparison changes because Bricken et al. use multi-token SAE probing but max-pool model dimensions for their baseline.
  • Pooling choices: Adding attention-pooled baselines and selecting between pooled and last-token strategies reduces the SAE probe win rate to 8.7%.This result motivates the paper’s concern that insufficiently strong baselines can create an illusion of SAE advantage.

6. Assessing Improvements in SAE Architectures

The paper tests whether newer SAE architectures improve probing performance by comparing eight architectures across probe sizes and sparsity levels. The results suggest only a possible slight improvement.

  • Architecture comparison: Eight SAE architectures released over the last two years are compared to assess whether architectural progress improves SAE probing.The study evaluates architectures using probes across regimes and datasets, including k = 16, k = 128, and standard-condition k = 1 probes.
  • Architecture comparison: Newer SAE architectures show a possible slight improvement in standard-condition probing performance.The comparison takes the maximum average test AUC across L0 values for each architecture because a single L0 value is noisy.

7. Conclusion

Across more than 100 datasets and multiple probing regimes, current SAE methods show no consistent advantage over strong baselines. The study treats this as a rigorously evaluated datapoint rather than a wholesale critique of SAEs, while noting that probing remains only a proxy for SAE utility.

  • Conclusion: SAE probes showed no improvement over traditional methods across multiple regimes and more than 100 datasets.The authors report that some failures involve distributional shifts and complex concepts.
  • Conclusion: Lower L0 SAEs produced worse sparse probes, while SAE latents struggled with distributional shifts and complex concepts.
  • Conclusion: The authors emphasize rigorous baseline design because previously reported SAE advantages were overturned when stronger baselines were considered.
  • Limitations: Probing performance is a more effective measure than reconstruction loss or downstream cross-entropy, but it remains a proxy for SAE utility.
  • Conclusion: Figure 12 shows only a slight probing uptick over baselines in standard conditions, with substantially greater spread in the data.
  • Limitations: Even a basis of true model representations might provide only a mild inductive-bias advantage over traditional linear probing.

9. Author Contributions

The supplied passages situate the work within research on probing, interpretability reliability, and SAE representations. They also identify prior positive and negative downstream findings that motivate broader evaluation.

  • Related work: Probing originated in computational neuroscience and became a tool for studying information representation in artificial neural networks.
  • Related work: Prior work found that plausible-looking saliency maps and interpretable neurons can fail to faithfully reflect general model or dataset representations.
  • SAE background: SAEs map model activations into a sparse, higher-dimensional latent space whose individual latents are hypothesized to represent monosemantic concepts.

B. Classification Datasets

The study uses a broad collection of binary classification datasets and evaluates probes with AUC-based validation and testing procedures. The supplied passages also note a small test-data leakage in selecting some SAE settings.

  • Evaluation metric: AUC summarizes binary-classifier performance across all classification thresholds using the area under the ROC curve.
  • Limitations: Selecting k and the baseline SAE using test AUC on normal datasets introduces slight test-data leakage.
  • Validation: Hyperparameters are selected using the method with maximal average validation AUC across held-out validation sets.

C.3. Probing Method Hyperparameter Details

The probing pipeline searches baseline hyperparameters and uses a validation-selected “quiver of arrows” to compare baseline methods with and without SAE probes. Additional analyses examine layers, SAE width, L0, and latent count.

  • Probing Method Hyperparameter Details: Baseline methods are evaluated through explicit hyperparameter searches, including regularization, PCA components, neighbors, estimators, tree depth, and MLP architecture.The supplied details specify ranges for C, PCA components, KNN neighbors, tree-based parameters, and MLP depth, width, learning rate, and weight decay.
  • C.4. Formal Discussion of Quiver of Arrows: The quiver of arrows selects the method with maximal validation AUC and compares its test AUC with alternative method sets.Adding SAE probes measures their marginal contribution to a practitioner’s available toolbox, while potentially increasing overfitting opportunities.
  • Layer performance: Layer 20 is best for baselines, and logistic regression most often achieves the highest test AUC under these conditions.
  • SAE width and L0: Higher L0 appears to improve probe performance, whereas SAE width is largely unimportant; three widths are tested: 16,000, 131,000, and 1,000,000.The width-performance trend is negative but has almost 0 slope.
  • Latent count: Probe performance increases with the number of SAE latents on almost all datasets, sometimes sharply and otherwise relatively smoothly.

E.1. Data Scarcity

SAE probes are often selected in data-scarce and severely imbalanced settings, but their advantage is inconsistent under label corruption. Selection reflects both probe performance and the paper’s tie-breaking rules.

  • Data scarcity: At small training fractions, the quiver mostly selects SAE methods because ties are explicitly broken in their favor.The procedure first prefers the smallest-width SAE, then the SAE with the largest k.
  • Evaluation setup: The evaluation compares quiver-based method selection across datasets, corruption ratios, positive-class ratios, and training-example counts.
  • Class imbalance: At more extreme class-imbalance ratios, the SAE method is generally selected from the quiver.
  • Label noise: Under label corruption, SAEs often achieve the highest test AUC, but per-dataset results show degradation as label noise increases.

F. Reproducing Core Results on Llama-3.1-8b

Replication on Llama-3.1-8B confirms that SAE probes do not improve over baselines across several challenging regimes. Additional analyses find that pruning, latent descriptions, and binarization provide limited or non-unique benefits.

  • Core replication: Llama-3.1-8B SAE probes underperform baseline probes in normal, data-scarce, class-imbalanced, and label-noise settings.
  • Pruning and OOD detection: Pruning improves two datasets but fails on one, and its gains are marginal relative to the in-distribution-to-OOD performance drop.The authors hypothesize that the underlying latents are not sufficiently expressive for robust OOD transfer.
  • Latent descriptions: GPT-4o-generated latent descriptions are usually task-relevant or visibly spurious, but some descriptions appear incomplete given task performance.
  • Dataset quality: SAE and baseline classifiers both identify apparently mislabeled CoLA examples, indicating that logistic regression can reproduce the SAE probe’s dataset-quality finding.The CoLA evaluation estimates that approximately 25% of labels are mislabeled by the LLM-majority-vote criterion.
  • SAE architectures: Later SAE architectures show a slight average probing-performance improvement, but substantial across-dataset variance remains.
  • Multi-token probes: Binarizing pooled SAE latents with threshold 1 produces worse multi-token probe performance than leaving them continuous.
Loading 2502.16681v1…