Do Sparse Autoencoders Capture Concept Manifolds?

Usha Bhalla; Thomas Fel; Can Rager; Sheridan Feucht; Tal Haklay; Daniel Wurgaft; Siddharth Boppana; Matthew Kowal; Vasudev Shyam; Jack Merullo; Atticus Geiger; Ekdeep Singh Lubana

Source-linked AI summary

Do Sparse Autoencoders Capture Concept Manifolds?

Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgaft, Siddharth Boppana, Matthew Kowal, Vasudev Shyam, Jack Merullo, Atticus Geiger, Ekdeep Singh Lubana

arXiv:2604.28119v1Apr 30, 2026cs.LGcs.AI

TL;DR

The paper studies whether SAEs can recover concepts organized as low-dimensional manifolds rather than independent directions. It develops a framework for manifold capture and finds that current SAEs typically represent these structures through fragmented, localized feature tilings, motivating geometry-centered interpretability. This approach remains limited by featurizers designed for one-dimensional atoms and by unreliable post-hoc grouping.

Problem
Existing SAE interpretations often assume concepts are independent linear directions, while many representations may instead contain structured, nonlinear geometries.
Method
The paper formalizes additive mixtures of manifolds and characterizes how sparse dictionaries can represent them through global subspaces or localized feature tilings.
Results
SAEs can represent manifolds in principle, but empirical analyses find fragmented representations in which geometry is tiled across many localized, partially redundant features.
Takeaways & Limitations
Interpretability should focus on recovering and manipulating geometric structures and on identifying coherent groups of atoms rather than isolated directions.
Takeaways & Limitations
Current featurizers use one-dimensional atoms and reconstruction losses that do not reward coherent treatment of curved geometric objects, while post-hoc recovery remains unreliable.

Abstract

from arXiv · show

Sparse autoencoders (SAEs) are widely used to extract interpretable features from neural network representations, often under the implicit assumption that concepts correspond to independent linear directions. However, a growing body of evidence suggests that many concepts are instead organized along low-dimensional manifolds encoding continuous geometric relationships. This raises three basic questions: what does it mean for an SAE to capture a manifold, when do existing SAE architectures do so, and how? We develop a theoretical framework that answers these questions and show that SAEs can capture manifolds in two fundamentally different ways: globally, by allocating a compact group of atoms whose linear span contains the entire manifold, or locally, by distributing it across features that each selectively tile a restricted region of the underlying geometry. Empirically, we find that SAEs suboptimally recover continuous structures, mixing the global subspace and local tiling solutions in a fragmented regime we call dilution. This explains why manifold structure is rarely visible at the level of individual concepts and motivates post-hoc unsupervised discovery methods that search for coherent groups of atoms rather than isolated directions. More broadly, our results suggest that future representation learning methods should treat geometric objects, not just individual directions, as the basic units of interpretability.

1 Introduction

The paper argues that neural representations often organize concepts along low-dimensional manifolds rather than isolated linear directions, creating challenges for SAE interpretation. It proposes studying manifold capture directly and using groups of features to recover geometric structure.

The paper asks whether SAEs recover additively superposed manifolds and how such recovery should be defined.
Continuous concepts can form curved, low-dimensional structures in model representations rather than isolated directions.
SAEs may preserve representation geometry through reconstruction while organizing it in ways that are not individually interpretable.
The paper therefore motivates manifold-oriented discovery methods that identify coherent groups of atoms rather than isolated directions.
Manifold structure can explain instability across learned dictionaries because multiple bases or tilings may represent the same geometry.
Single-feature interventions can be brittle when features encode local patches, while isolated-feature inspection can miss or misidentify the underlying object.

2 Notations: Sparse Coding and SAEs

The notation defines an SAE as a sparse coding system that maps an activation to latent coefficients through a learned dictionary. Its reconstruction lies in a sparse non-negative span of dictionary atoms.

An SAE extracts latent representation z from activation x using dictionary D through an optimization problem.
A sparsity-promoting regularizer can restrict the number of active coefficients, such as enforcing ∥z∥0 ≤ k.
The localized reconstruction x̂ = zD lies in a sparse non-negative span, geometrically forming a cone.

3 Manifolds are Ubiquitous in LLM Representations

The paper presents continuous concepts as low-dimensional geometric structures in LLM representations and tests whether those structures affect model behavior. Steering between concept centroids produces smooth behavioral changes for concept-dependent tasks.

Many real-world concepts vary continuously over a range and may therefore be represented by low-dimensional structured geometries.
Steering between prototypical concept centroids can move along these manifolds and smoothly change model outputs on dependent tasks.

4 Formalizing Manifold Capture in Sparse Representations

The paper extends sparse representation analysis from linear directions to additive mixtures of low-dimensional manifolds and defines capture through compact, consistent atom groups. It distinguishes global subspace capture from local tiling, including shattering and dilution, and validates post-hoc discovery through feature interactions in synthetic mixtures.

Representations as Additive Mixture of Manifolds: The Additive Mixture of Manifolds model represents neural activations as superpositions of smoothly embedded, low-dimensional manifolds, with linear representations as the special case of one-dimensional rays.Each manifold lies in a low-dimensional affine subspace and varies continuously along intrinsic coordinates.
Subspace Recovery via SAEs: An SAE captures a manifold when a small fixed group of decoder atoms spans a containing subspace and the encoder selects that group consistently across the manifold.This definition requires both parsimonious support and coherent activation, rather than merely low reconstruction error.
Subspace Recovery via SAEs: Subspace recovery is ensured when reconstruction error is sufficiently low, sparsity matches the manifold’s ambient dimension, and the dictionary is sufficiently incoherent.The result relies on classical sparse dictionary-learning conditions and distinguishes the ideal decoder from the trained encoder through an amortization gap.
From Capture to Tiling: When more atoms are available than the manifold’s ambient dimension, features can specialize to different regions and tile the manifold through localized receptive fields.Tiling uses overlapping local detectors whose joint activity encodes position along the manifold.
From Capture to Tiling: Shattering partitions a manifold into nearly disjoint active sets, whereas dilution uses overlapping redundant atoms without any compact group that spans the whole manifold.These are distinct forms of local tiling, separated by support overlap and the absence of a sufficiently small coherent atom group.
4.1 Ising Pairings and Regimes of Manifold Representation: Pairwise Ising models on binarized SAE codes separate structural co-activation from correlations induced by superposition and produce a more faithful feature-interaction matrix.The model uses fields for frequent activation and couplings for functional dependencies after indirect correlations are factored out.
4.1 Ising Pairings and Regimes of Manifold Representation: Positive within-group couplings indicate compact capture, negative couplings indicate shattering, and mixed signs indicate dilution with redundant overlapping atoms.Manifolds can therefore be recovered as communities with strong cooperative or inhibitory interactions rather than by decoder-direction similarity.
4.2 A Toy Model of Manifold Superposition: A controlled benchmark with eight manifold types tests the framework under known geometry, random embeddings, and sparse mixing, enabling direct evaluation of capture and discovery.The benchmark includes circles, spheres, tori, Möbius strips, Swiss rolls, helices, flat disks, and line segments.

5 Characterizing Manifold Capture in LLMs

Across Llama3.1-8B representations, SAEs preserve manifold geometry but distribute it across localized, overlapping features rather than compact atom groups. This dilution makes manifold structure meaningful primarily at the level of feature groups or subspaces.

Experimental setup: SAEs trained across five architectures, expansion factors, and sparsity settings were evaluated on Llama3.1-8B layer-19 representations from 500M Pile tokens.The study used Standard, JumpReLU, TopK, BatchTopK, and Matryoshka SAEs.
Compact capture and dilution: Variance explained plateaus only beyond each manifold’s ambient dimension, showing that SAEs fail to allocate compact atom groups spanning the manifold.The residual-variance protocol identifies this behavior as the dilution regime.
Local tiling: Individual SAE features reconstruct local manifold patches piecewise-linearly, while their union progressively covers the full geometry.This places current SAEs between compact subspace capture and complete shattering.
Architectural effects: Different SAE architectures produce distinct receptive-field biases, including angular separability for Top-K and linear separability for L1.These biases appear in feature selectivity across the days-of-week manifold.
Tuning and selectivity: On the years manifold, features respond smoothly and locally, with some encoding the ones digit periodically every 10 years and others encoding the decade.The overlapping responses resemble tuning curves in biological population codes.
Interpretation and limitation: Interpretability in the tiling regime requires analyzing feature groups or subspaces, because individual dictionary elements provide only narrow views of the captured geometry.The paper also identifies a design mismatch: one-dimensional atoms and reconstruction-focused losses do not directly target curved geometric objects.

6 Unsupervised Manifold Discovery

The paper evaluates post-hoc grouping methods for recovering manifolds from diluted SAE representations. Ising couplings and conditional co-activation best separate manifold communities, enabling recovery of known and novel structures.

Grouping strategy: Because manifold geometry is distributed across localized features, recovery requires grouping related atoms without prior knowledge of the underlying manifolds.The evaluated pipeline transfers an Ising-model approach from synthetic data to real language-model representations.
Similarity metrics: Ising couplings and conditional co-activation produce the clearest within-manifold and cross-manifold separation, whereas decoder cosine similarity and Pearson correlation fail to recover block structure.This supports treating manifold membership as a functional co-activation property rather than a decoder-direction property.
Unsupervised discovery: Applied to a BatchTopK SAE on Llama3.1-8B, the Ising pipeline recovers temperature, colors, and political bias as distinct communities and surfaces a novel epistemic-uncertainty manifold.The procedure therefore supports both recovery of known structures and hypothesis generation beyond them.

7 Conclusion

The paper argues that neural-network representations should be interpreted as geometric structures rather than isolated directions. SAEs preserve such geometry only implicitly, motivating manifold-aware featurizers and group-level feature analyses.

7 Conclusion: SAEs often represent manifolds through many localized, partially redundant features rather than coherent subspaces.This fragmented representation preserves geometry implicitly and obscures it at the level of individual features.
7 Conclusion: Interpretability should target recovery and manipulation of geometric structures rather than individual directions.The paper proposes manifold-targeted featurization and analysis tools that operate on coherent groups of features.
7 Conclusion: Continuous variables are typically encoded by populations of neurons with localized, overlapping receptive fields that collectively tile the underlying space.This neuroscience perspective provides a precedent for interpreting groups of SAE features as coherent geometric objects.
7 Conclusion: Existing manifold-learning and sparse-subspace-clustering methods generally associate each observation with a single subspace or manifold via a latent label.The paper’s additive mixture setting instead permits observations to participate in multiple manifolds simultaneously.
7 Conclusion: The evaluation uses last-token activations, PCA-based manifold interventions, and target-token probability tracking to test geometric relevance.The intervention sweeps continuous labels through PCA-space points and measures semantically diagnostic next-token probabilities.

B.2 SAE Training Details

The training pipeline uses layer-19 Llama-3.1-8B activations from The Pile and compares several SAE architectures under shared optimization settings. Main experiments retain models exceeding a variance-explained threshold, with lower-quality models included only for architectural comparison.

B.2 SAE Training Details: All SAEs use layer-19 residual-stream activations from Llama-3.1-8B, harvested from 500M tokens of The Pile.The activation dimension is d = 4096 and sequences have length 4096.
B.2 SAE Training Details: Optimization uses Adam with learning rate 10^-4, no weight decay, gradient clipping, and batches of 16,384 tokens.Activations are normalized by their mean ℓ2 norm before training, with linear warmup during the first training epoch.
B.2 SAE Training Details: The evaluated architectures include TopK, BatchTopK, JumpReLU, Matryoshka, and standard ℓ1 SAEs.Architecture-specific settings include auxiliary-loss weighting, straight-through-estimator bandwidth, nested feature groups, and sparsity weights.
B.2 SAE Training Details: Main experiments retain SAEs with variance explained above 0.85 on held-out activations.SpaDE and MFA remain included for architectural comparison despite VE values of approximately 0.62 and 0.55.

B.3 Platonic Representations?

The paper compares SAE representations across decoder, code, and manifold-restricted spaces. Individual features align weakly across models, whereas the induced coordinate systems on specific manifolds align strongly.

B.3 Platonic Representations?: Feature comparisons use optimal transport in decoder space, code space on random inputs, and code space restricted to specific manifolds.This tests whether different SAEs recover a shared representation of the same underlying geometry.
B.3 Platonic Representations?: Decoder directions and random-input code activations show weak or unclear alignment across SAEs.Point-based methods such as SpaDE differ substantially from direction-based SAEs in decoder space.
B.3 Platonic Representations?: Specific-manifold comparisons show strong alignment across SAEs despite differences in individual features.The consistent object is the manifold-level coordinate system rather than the feature identities themselves.
B.3 Platonic Representations?: PCA projections reveal global components, while SAE reconstructions approximate manifolds piecewise linearly through locally active features.Tuning curves illustrate mixed selectivity across hue and sentence-length manifolds.
B.3 Platonic Representations?: The paper distinguishes directional capture, where atom spans contain a manifold, from point-based capture, where landmarks form a topology-preserving sample.Point-based reconstruction lies within a sparse convex hull, unlike directional reconstruction in a sparse nonnegative span.

C.1 Simplicial recovery

Simplicial recovery treats dictionary atoms as landmarks whose convex combinations sample a manifold. Under additive superposition, however, point-based landmarks tile the joint manifold and cannot generally recover its individual factor manifolds.

C.1 Simplicial recovery: Point-based recovery requires active landmarks to form a sufficiently dense and faithful discrete sample of the manifold.The resulting Vietoris–Rips complex can recover connected components, loops, and higher-order topology under standard density and reach conditions.
C.1 Simplicial recovery: Under superposition, point-based SAEs face a fundamental obstruction to factorwise recovery because landmarks encode joint configurations.The simplex constraint couples all factors, so selecting landmark subsets cannot isolate one factor’s contribution.
C.1 Simplicial recovery: Convex combinations of landmarks on a joint manifold can produce points absent from the underlying factor manifolds.For additive factors, midpoint reconstructions generally combine chords rather than points lying on either individual manifold.
C.1 Simplicial recovery: Each landmark remains at least δ − ε away from the other factor manifold under the stated nondegeneracy condition.This formalizes why joint-manifold landmarks cannot simultaneously approximate the individual factors.
C.1 Simplicial recovery: Point-based dictionary learning tiles the joint manifold, whereas direction-based SAEs retain additive reconstructions that permit factorwise analysis.Recovering individual factors with point-based methods would require compositional constructions such as Minkowski sums or blockwise simplices.

D Conditions of Subspace capture

Subspace capture is guaranteed when a manifold lies in the span of a sufficiently incoherent subset of decoder atoms and reconstruction error is bounded. Under these conditions, sparse recovery keeps the learned representation close to the manifold at precision O(λ).

Recovery condition: The idealized analysis uses Orthogonal Matching Pursuit over the learned dictionary to separate dictionary quality from the behavior of a feedforward encoder.This enables classical sparse-recovery guarantees to be applied directly to the learned atoms.
Recovery condition: If a k-dimensional manifold subspace equals the span of k decoder atoms and dictionary coherence satisfies µ < 1/(2k −1), reconstruction error at most λ implies capture precision O(λ).The theorem assumes an orthonormal subspace basis, a µ-incoherent dictionary, and a support S⋆ of size k spanning the manifold.
Proof setup: Centering absorbs the affine offset into the SAE bias, so manifold points are analyzed as centered vectors in the relevant subspace.The proof represents each centered point using the atoms spanning the manifold subspace.
Proof mechanism: Under the exact recovery condition, the coefficient discrepancy outside the target support is O(λ), limiting the contribution of atoms outside the manifold subspace.The proof applies a noise-robust null-space property and bounds the off-support coefficient mass before bounding reconstruction outside the target span.
Proof mechanism: Unit-norm decoder atoms and the triangle inequality convert the coefficient bound into an O(λ) bound on the manifold-capture error.This final inequality establishes the claimed precision guarantee.

E Synthetic Experiment Details

The synthetic benchmark mixes normalized manifold instances in sparse superpositions, trains TopK SAEs across sparsity budgets, and evaluates whether learned decoder atoms recover each manifold’s ambient subspace. The experiment uses controlled geometry, known active masks, and restricted R2 around the embedding dimension.

Benchmark construction: Synthetic observations are generated by sampling manifold coordinates, embedding each manifold into ambient space, and forming sparse superpositions before SAE training.The pipeline evaluates both subspace capture for direction-based SAEs and simplicial capture for point-based SAEs.
Benchmark construction: The benchmark contains eight manifold types with intrinsic dimension d_i and embedding dimension k_i, distinguishing degrees of freedom from the atoms needed for subspace capture.A circle has d_i = 1 but k_i = 2, while a torus has intrinsic dimension 2 and requires a 4-dimensional embedding.
Benchmark construction: Normalization centers and isotropically rescales every manifold instance to equalize reconstruction-loss contributions while preserving angles, relative distances, curvature ratios, and topology.After normalization, each instance has RMS norm exactly 1 in local coordinates.
Benchmark construction: The benchmark uses 48 instances, 2,000,000 training samples, and a separate 1,000,000-sample evaluation set with known per-manifold contributions and active masks.The evaluation split measures capture on in-distribution superpositions with ground truth.
SAE training: TopK SAEs use 512 dictionary atoms in dimension 128, an expansion factor of 4, and sparsity budgets k ∈ {3, 4, 6, 8, 10, 14, 16, 20, 25}.Training uses a linear encoder with TopK selection and a linear decoder with unit-norm columns.
Evaluation: Restricted R2 greedily selects decoder directions that explain manifold-specific residual variance, then decodes masked codes to test whether approximately k_i atoms span the manifold.An R2 near 1 at n = k_i indicates compact subspace capture; selection operates on decoder directions rather than codes.
Statistical analysis: The Ising analysis replaces pairwise covariance with conditional-independence structure to identify relationships among binarized SAE activations.For Gaussian variables, the analogous object is the precision matrix; the binary case uses Ising couplings.

G A Geometric and Statistical View of Tiling

SAEs can organize a manifold through localized tiling rather than a compact global atom group, producing either near-exclusive shattering or overlapping redundant dilution. The paper identifies dilution as the dominant interpretability problem because geometry remains distributed across many features without coherent organization.

Tiling regimes: Trained SAEs rarely achieve compact capture; instead, they fragment manifolds into many partially overlapping receptive fields, a pattern called tiling.Tiling preserves manifold geometry through joint activity while failing to reuse a compact group across the entire manifold.
Shattering: Shattering assigns individual atoms to localized patches that partition the manifold, so movement along the manifold transfers activity from one atom to the next.The number of atoms scales with manifold volume in atom-units rather than ambient dimension.
Dilution: Dilution assigns overlapping groups of atoms to local regions, with smoothly varying subsets that redundantly encode the same geometry.Its total support can greatly exceed the manifold’s ambient dimension because multiple atoms cover each region.
Geometric distinction: Both shattering and dilution can achieve low reconstruction error, but they differ in what atoms represent: landmarks in shattering versus redundant local basis elements in dilution.This distinction separates geometric preservation from interpretable organization.
Statistical signatures: Shattering produces mutual exclusion, dilution combines positive within-region coupling with inhibition across disjoint regions, and capture produces broad co-firing.These distinct activation statistics motivate using Ising couplings to distinguish the regimes.
Statistical signatures: The support size |G| separates compact capture from both tiling regimes, while coupling signs distinguish shattering from dilution.Capture requires |G| no larger than ambient dimension; tiling requires a substantially larger support.
Interpretability implications: Empirical evidence places trained SAEs in dilution: restricted-R2 curves plateau beyond k_M, tuning fragments manifolds, and Ising couplings have mixed signs.Diluted representations encode manifold geometry across a redundant cover whose organization is not visible in individual features.