PaperlayerView exact revision on arXiv

Source-linked AI summary

Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry

Sai Sumedh R. Hindupur, Ekdeep Singh Lubana, Thomas Fel, Demba Ba

arXiv:2503.01822v2cs.LGcs.AI

TL;DR

SAEs are widely used to identify concepts in neural representations, but the paper asks whether their architectures bias which concepts they recover. It formulates SAEs as bilevel optimization problems, evaluates assumptions about concept geometry, and finds that architecture-specific designs can reveal concepts missed by standard variants.

  • Problem

    The paper investigates whether SAEs recover all concepts used by a model or instead favor concepts with particular organizational structures.

  • Method

    The paper formulates SAEs as bilevel optimization problems and evaluates architectures against heterogeneous intrinsic dimensionality and nonlinear separability across controlled, semi-synthetic, and realistic settings.

  • Results

    Standard SAEs miss concepts when their implicit geometric assumptions are violated, whereas SpaDE achieves higher F1 scores, lower latent co-occurrence, and adaptive sparsity on structured representations.

  • Takeaways & Limitations

    SAE architectures should be tailored to known properties of concept geometry rather than treated as universally interchangeable.

  • Takeaways & Limitations

    The paper does not claim SpaDE is optimal for all scenarios and identifies additional data properties, Euclidean separation assumptions, aggressive sparsity, and overlapping concepts as remaining limitations.

Abstract

from arXiv · show

Sparse Autoencoders (SAEs) are widely used to interpret neural networks by identifying meaningful concepts from their representations. However, do SAEs truly uncover all concepts a model relies on, or are they inherently biased toward certain kinds of concepts? We introduce a unified framework that recasts SAEs as solutions to a bilevel optimization problem, revealing a fundamental challenge: each SAE imposes structural assumptions about how concepts are encoded in model representations, which in turn shapes what it can and cannot detect. This means different SAEs are not interchangeable -- switching architectures can expose entirely new concepts or obscure existing ones. To systematically probe this effect, we evaluate SAEs across a spectrum of settings: from controlled toy models that isolate key variables, to semi-synthetic experiments on real model activations and finally to large-scale, naturalistic datasets. Across this progression, we examine two fundamental properties that real-world concepts often exhibit: heterogeneity in intrinsic dimensionality (some concepts are inherently low-dimensional, others are not) and nonlinear separability. We show that SAEs fail to recover concepts when these properties are ignored, and we design a new SAE that explicitly incorporates both, enabling the discovery of previously hidden concepts and reinforcing our theoretical insights. Our findings challenge the idea of a universal SAE and underscores the need for architecture-specific choices in model interpretability. Overall, we argue an SAE does not just reveal concepts -- it determines what can be seen at all.

1 Introduction

SAEs are used to identify concepts for interpreting and controlling neural networks, but their theoretical foundations may bias which concepts they recover. The paper frames this as a duality between SAE architecture and concept organization, then evaluates that claim across concept geometries.

  • Motivation: SAEs support concept discovery and can enable targeted interventions on model representations when recovered concepts faithfully reflect model computations.This motivates studying whether SAE-derived concepts are complete and reliable enough for interpretability and control.
  • Research gap: Different SAEs may achieve similar fidelity and sparsity while identifying qualitatively different concepts because their architectural biases differ.The paper connects this possibility to reported algorithmic instability and limited causal evidence.
  • Core contribution: The framework formalizes a duality between concept organization in representations and the receptive fields imposed by an SAE encoder.This duality motivates treating SAE architecture as a determinant of which concepts can be detected.
  • Empirical scope: Experiments test heterogeneous intrinsic dimensionality and nonlinear separability from controlled synthetic settings through real model activations and naturalistic data.The paper reports that SAEs failing to account for these properties systematically miss corresponding concepts.

2 Preliminaries

Sparse coding represents data with sparse combinations of learned dictionary atoms, while SAEs approximate this process with an encoder-decoder architecture. Their encoder nonlinearity and sparsity regularization distinguish different SAE variants.

  • Sparse coding: Sparse coding models data as a sparse combination of latent variables and jointly optimizes sparse codes with dictionary atoms.The sparse code is constrained through a sparsity-promoting regularizer, typically the ℓ1 norm.
  • Projection view: Projection-based SAE variants use architecture-specific constraint sets, including angular or sparse-subspace constraints represented by TopK.The projection set determines which activation patterns the encoder can represent.
  • Sparse autoencoders: An SAE computes a latent code from an input with an encoder and reconstructs the input from that code with a decoder.The encoder uses z = g(W^T x + b_e), followed by reconstruction with the learned dictionary and decoder bias.
  • Architectural variation: SAEs differ primarily in their encoder nonlinearity and sparsity regularizer, which determine how sparsity is imposed on latent codes.These choices are the basis for the unified architectural framework developed later.

3 Unified Framework for SAEs

The unified framework treats common SAEs as constrained bilevel optimization problems whose encoder architectures impose receptive-field structures. These structures encode implicit assumptions about concept geometry and therefore constrain concept discovery.

  • Unified framework: The framework analyzes ReLU, TopK, and JumpReLU SAEs as projection-based encoders with different constraint sets.The projection formulation makes architectural differences explicit and connects them to activation behavior and sparsity structure.
  • Bilevel optimization: In the bilevel formulation, the inner optimization associated with the encoder distinguishes different SAE architectures.The encoder is represented through a variational objective over an architecture-specific constraint set.
  • Bilevel optimization: Each SAE solves a constrained version of sparse dictionary learning, so its encoder restricts the search space and may exclude solutions available to unconstrained sparse coding.The constraint affects the quality and scope of the dictionary-learning solution.
  • Receptive fields: An SAE’s encoder determines the structure of latent receptive fields, defined as the input regions where individual latents are active.Monosemanticity requires these receptive fields to match the organization of concepts in the data.
  • Implicit assumptions: The framework concludes that SAE data assumptions should match the true organization of concepts in model representations.Otherwise, the encoder may identify some concepts while leaving others obscure.

4 Implicit SAE Assumptions and Data Properties

SAEs make implicit assumptions about concept geometry through their encoder architectures, favoring particular receptive-field structures. The paper identifies nonlinear separability and heterogeneous intrinsic dimensionality as important data properties and designs SpaDE to accommodate both.

  • Implicit assumptions: SAE encoder architectures implicitly assume specific concept structures, constraining which concepts their receptive fields can identify.These assumptions reflect the duality between encoder receptive fields and concept organization in model representations.
  • Data properties: Concepts may be nonlinearly separable, while different concepts may occupy subspaces with different intrinsic dimensions.These properties include magnitude-dependent or otherwise nonlinear boundaries and concepts ranging from unidimensional to multidimensional structure.
  • Architecture assumptions: ReLU and JumpReLU assume linear separability, while TopK assumes angular separability and the same dimensionality per concept.The compatibility of these assumptions with nonlinear separability and heterogeneity determines which concept structures each architecture can represent effectively.
  • SpaDE design: SpaDE combines competitive distance-based projections with adaptive sparsity to capture nonlinear separability and heterogeneous concept dimensionality.Its probability-simplex projection permits different concepts to activate different numbers of latents, while Euclidean distances to encoder prototypes support flexible receptive fields.
  • SpaDE design: SpaDE also assumes that Euclidean distances meaningfully separate concepts and can be used to disentangle them.Thus, the proposed architecture addresses the targeted properties without eliminating architecture-specific geometric assumptions.

5 Results: Empirical Validation of SAE behavior

Experiments on synthetic, formal-language, and vision activations test how SAE assumptions affect separability, dimensionality, and interpretability. Across these settings, SpaDE more consistently adapts to concept structure and isolates concepts than the other evaluated SAEs.

  • Separability experiment: SpaDE achieves an F1 score of 1.0 for both linearly and nonlinearly separable concepts, while ReLU and JumpReLU are bounded above by 0.5 on the nonlinear concept.ReLU and JumpReLU reach F1=1 for the separable concept, whereas their receptive fields overlap other concepts in the nonlinear case.
  • Heterogeneity experiment: TopK uses the same sparsity level across heterogeneous concepts and reconstructs higher-dimensional concepts worse, whereas SpaDE nearly matches each concept’s intrinsic dimension.ReLU, JumpReLU, and SpaDE show adaptive sparsity to varying degrees; SpaDE follows the intrinsic-dimension curve nearly perfectly for one hyperparameter choice.
  • Heterogeneity experiment: TopK falls below 20% normalized MSE only when its fixed sparsity exceeds a concept’s intrinsic dimension, while other SAEs remain below that threshold for nearly all concepts.For example, the six-dimensional concept crosses the threshold only after k=8.
  • Formal languages: In formal-language activations, SpaDE avoids latent co-occurrence across parts of speech and reaches perfect F1, while other SAEs show co-occurrence or varying separability.The different intrinsic dimensions and separability levels of parts of speech produce architecture-dependent sparsity and F1 behavior.
  • Vision: On DINOv2 vision activations, SpaDE produces the most monosemantic latents across classes and identifies interpretable object and scene concepts.Its latent co-occurrence is limited to certain pairs, compared with widespread co-occurrence in ReLU, JumpReLU, and TopK.

6 Discussion and Limitations

The experiments show that SAE architectures have distinct concept-discovery strengths and failure modes because their inductive biases may mismatch concept geometry. The paper therefore presents architecture design as geometry-dependent rather than proposing a universally optimal SAE.

  • Architecture-specific failure modes: ReLU and JumpReLU fail to capture nonlinear separability, while TopK struggles when concept dimensionality exceeds its K choice.These failures are associated with low F1 scores for ReLU and JumpReLU and high MSE for TopK under mismatched concept dimensions.
  • Architecture-specific failure modes: Latent co-occurrence across multiple concepts is common across architectures, indicating limited concept specialization.SpaDE instead achieves low latent co-occurrence for its most monosemantic latents.
  • Architecture-specific design: SpaDE combines Euclidean-distance computations, SparseMax, and adaptive sparsity to support structured concept representations.The paper describes SpaDE as an effective choice in the evaluated settings, not as an architecture that is optimal for all scenarios.
  • Broader implications: The paper argues that no single SAE is best for all contexts unless it integrates the data properties relevant to the specific problem.It recommends tailoring SAE design using prior knowledge about the underlying data geometry.
  • Broader applicability: The receptive-field analysis relating encoder structure to concept geometry can also study interpreter models such as transcoders.This extends the framework beyond sparse autoencoders while preserving its focus on representation geometry.
  • Limitations: The paper does not claim SpaDE is optimal for all scenarios and identifies additional data geometry, Euclidean separation, sparsity, and concept overlap as boundaries.These limitations include possible latent co-occurrence, overly specialized latents, and differing co-occurrence structure for overlapping concepts.

C.1 Separability experiment

The separability experiment evaluates four SAE architectures on synthetic Gaussian-cluster data using concept recovery, receptive-field, similarity, and spectral analyses. Its evaluation centers on F1-score from thresholded latent activations and examines how latent representations organize across concepts.

  • Dataset and models: The synthetic dataset contains six isotropic Gaussian concepts in 2D, arranged at angular intervals of 2π/6 with alternating cluster norms of 1 and 3.Each concept contributes 1 million samples, for 6 million total points.
  • Dataset and models: The comparison includes ReLU, JumpReLU, TopK, and SpaDE SAEs, with SpaDE using Euclidean distances and SparseMax for its encoder.The other three architectures follow their original formulations, with decoder activations normalized during the forward pass.
  • Evaluation: F1-score is the primary comparison metric, computed from precision and recall on 1,000 evaluated points per concept.Latent activations are thresholded at 10^-6 when computing precision and recall.
  • Evaluation: Receptive fields are analyzed on a 2D meshgrid, while pairwise cosine similarities organize latent relationships by class.The analysis also examines per-input and aggregated latent structure through similarity matrices.
  • Evaluation: Stable rank estimates the intrinsic dimension of the similarity matrix, and rounded-up stable rank determines the number of spectral-clustering groups.This provides a further analysis of correlations among SAE latent representations.

C.2 Heterogeneity experiment

The heterogeneity experiment constructs Gaussian concepts with different intrinsic dimensions and nested subspaces, then evaluates SAE reconstruction, sparsity, and latent specialization. The broader setup also applies the architectures to formal-language representations from a trained nanoGPT model.

  • Synthetic heterogeneity dataset: The synthetic dataset contains five Gaussian concepts with intrinsic dimensions 6, 14, 30, 62, and 126.These dimensions follow 2^q − 2 for q ∈ {3, 4, 5, 6, 7}.
  • Synthetic heterogeneity dataset: Lower-dimensional concept subspaces are nested within higher-dimensional subspaces, while each concept has constant total variance.The dataset contains 6.4 million points per concept, with approximately 22 million used for training.
  • Models and metrics: The four evaluated architectures are ReLU, JumpReLU, TopK, and SpaDE, with SpaDE using Euclidean distances and SparseMax.All models use the same single-hidden-layer autoencoder structure but differ in encoder and regularization choices.
  • Models and metrics: Evaluation reports normalized MSE as standard MSE divided by the corresponding concept variance and also measures average active latents per data point.The latter is reported as per-concept sparsity measured by L0.
  • Latent analysis: Latent specialization is assessed through cosine similarities between representations for different inputs and between latents aggregated over all data points.Each latent receives the label of the concept on which it is most frequently activated on average.
  • Formal-language setup: The formal-language setup trains a 2-layer nanoGPT on English-like PCFG strings and trains SAEs with 256 latents over a 128-dimensional residual stream.The model strings are padded to length 128, and SAE training lasts 10K iterations at a constant learning rate of 10^-3.

C.4 Vision experiment

The vision experiment analyzes representations from a pretrained DINOv2-base model on a 10-class ImageNet subset. SAEs are trained on spatial, CLS, and register tokens, with model selection controlled by sparsity and reconstruction.

  • Data and representations: The experiment uses DINOv2-base with registers and Imagenette, a 10-class ImageNet subset containing 1.5k images per class.Representations include 261 tokens per image.
  • SAE training: SAEs are trained on all spatial, CLS, and register tokens for 50 epochs with 200 latent dimensions.The setup processes approximately 200 million tokens over training.
  • SAE training: The best reconstruction is selected through a sparsity-controlled learning-rate sweep.TopK, ReLU, and SpaDE perform best at 5 × 10^-4, while JumpReLU performs best at 10^-4.

D Further Theory Results

SAE nonlinearities can be understood as projections onto constraint sets, whose geometry determines encoder receptive fields and therefore the concept structures an SAE assumes. SpaDE extends this framework with distance-based, adaptive representations designed for heterogeneous and nonlinearly separable concepts.

  • Projection nonlinearities: Projection nonlinearities map inputs onto constraint sets whose structure determines the encoder’s activation behavior, sparsity, and implicit data assumptions.ReLU, TopK, and JumpReLU correspond to projections onto different sets.
  • Projection nonlinearities: ReLU projects onto the positive orthant, TopK onto a union of k-sparse subspaces, and JumpReLU onto shifted hypercube corners.These projection sets induce different receptive-field structures.
  • Receptive fields and concept geometry: Under monosemanticity, an SAE’s receptive-field structure implies a corresponding assumption about the organization of concepts in data.Projection nonlinearities with flat boundaries yield piecewise-linear or constant behavior outside the constraint set.
  • SpaDE: SpaDE uses Euclidean distances to prototypes and a competitive projection, producing receptive fields that are unions of convex polytopes around prototypes.This locality and flexible shape allows latents to capture nonlinearly separable concepts.
  • SpaDE: SpaDE is exactly equivalent to a ReLU SAE with input-adaptive, piecewise-constant linear transformations and biases, while remaining piecewise linear and continuous.The equivalence is exact rather than a first-order Taylor approximation.
  • SpaDE: The probability simplex permits different sparsity levels across representations, enabling SpaDE to accommodate heterogeneous concept dimensions.For different choices of k, the simplex provides representations with varying numbers of active components.

E.1 Separability Experiment

The separability experiment compares SAE reconstruction, latent specialization, and concept separation across sparsity levels. SpaDE exhibits fewer dead latents, higher latent specialization, and lower cross-concept correlations than the other evaluated SAEs.

  • Reconstruction and sparsity: At dense representations, all SAEs have comparable MSE–sparsity curves, while TopK’s NMSE decreases substantially more than the others.The text attributes TopK’s lower NMSE to a redundant solution using two latents as an orthogonal basis for all data.
  • Dead latents: SpaDE has significantly fewer dead latents than ReLU, JumpReLU, and TopK across all sparsity levels.For the other architectures, dead-latent fractions increase at high sparsity and decrease as representations become denser.
  • Latent specialization: SpaDE has high stable ranks for data and latent correlations, whereas the other SAEs have much lower ranks than the desirable stable rank of 6.The stable rank of 6 corresponds to the six clusters in the data.
  • Activation profiles: Latent activation histograms show architecture-specific profiles across concepts, with SpaDE exhibiting a pointy profile that indicates a second mode beyond zero.The profiles share a common structure within each SAE despite variation across concepts.
  • Concept separation: SpaDE produces the lowest cross-concept correlations among the evaluated SAEs, while ReLU and JumpReLU correlations decrease little with sparsity.The corresponding latent-correlation analysis also shows increasing dead latents at higher sparsity.

E.2 Heterogeneity Experiment

The heterogeneity experiment shows that SAE training proceeds in two phases and that architecture behavior diverges as sparsity increases. Additional diagnostics examine reconstruction, latent structure, concept specialization, and low-dimensional projections across sparsity levels.

  • Training dynamics: At low sparsity, all SAEs reach similar normalized MSE, but their performance diverges at higher sparsity levels.Training first learns the data scale and then explains within-concept variation.
  • Training dynamics: The training dynamics exhibit two phases: matching each concept’s scale, followed by learning its within-concept variance.The transition is identified by the kink in comparisons with predictions based on concept centers.
  • Concept allocation: TopK with k = 32 cannot allocate enough latents to concepts with large intrinsic dimension, so it concentrates activations on fewer concepts.Latent activation profiles and cosine-similarity diagnostics compare concept and latent organization across sparsity levels.
  • Diagnostics: The figures visualize data correlations between datapoints, latent correlations, activation histograms, and PCA projections of sparse codes and latent specialization.These views are used to assess whether concepts and latents remain separated or specialized.
  • Diagnostics: The appendix evaluates normalized MSE, stable rank, monosemanticity, dead latents, concept-level errors, correlations, and PCA structure as sparsity changes.The formal-language diagnostics additionally examine parts-of-speech and positional dependence in sparse codes.
  • Formal-language comparison: Formal-language visualizations compare sparse-code geometry and latent activation specialization across parts-of-speech in two and three dimensions.The visualizations separately examine which datapoints latents activate for and which latents datapoints activate.

E.4 Vision Experiment

The vision experiment visualizes concepts learned by SpaDE through feature-attribution maps for monosemantic latents across five Imagenette classes. These maps provide class-specific views of the visual concepts identified by the method.

  • Feature attribution: Feature-attribution maps visualize monosemantic SpaDE latents for the Tench, Chainsaw, Church, Golf, and Springer classes.The visualizations are presented as class-specific examples of learned concepts.

NeurIPS Paper Checklist

The checklist reports that the paper provides theoretical, experimental, reproducibility, and ethics-related documentation. It also records stated limitations and the authors’ assessment of societal-impact and misuse concerns.

  • Claims and theory: The paper states that its theoretical claims are supported by Sections 3–4 and Appendix D, while experiments appear in Section 5 and Appendix E.The checklist records the authors’ assessment that the abstract and introduction accurately reflect the paper’s contributions and scope.
  • Scope and limitations: The paper states that its limitations are discussed in Section 6.The checklist identifies limitations as clearly discussed rather than reproducing their substantive contents.
  • Claims and theory: Theoretical results are reported as proved in Appendix D through derivations of receptive-field structure.The checklist marks the paper as providing the full assumptions and proofs for its theoretical results.
  • Reproducibility: Experimental setups are described briefly in Section 5 and extensively in Appendix C to support reproduction of the main results.The checklist also states that training and test details are specified sufficiently to understand the experiments.
  • Reproducibility: An anonymized code subset is provided for review, with all code planned for release after the review process.The checklist states that the formal-language experiment is currently covered by anonymized code.
  • Statistical reporting: The experiments report error bars when the experiments were not too expensive.The checklist marks the statistical-significance reporting as satisfactory.
  • Ethics and impact: The authors classify societal-impact discussion as not applicable because the work addresses limitations of interpretability tools and has no direct harmful impacts.They likewise state that safeguards for high-risk model or data release are not applicable.
Loading 2503.01822v2…