Verifying the Linear Representation Hypothesis: How Interpretable Are Vision SAEs?

• Polysemantic neurons activate in semantically distinct contexts, while superposition encodes more features than neurons using non-orthogonal directions, making the native basis difficult to interpret.Source: §1 · p. 1
• SAEs use sparse latent codes and reconstruction to separate superimposed features into distinct directions.Source: §1 · p. 1
• Vision SAE interpretability remains uncertain: common evaluations use reconstruction or dictionary properties, and whether concepts align with human perception remains debatable.Source: §1 · p. 2
• The paper adapts NLP's Autointerpretability Score to vision SAEs and validates it in a user survey, finding standard SAE interpretability metrics do not correlate.The authors argue that concept alignment is multifaceted and cannot be fully described by a single metric.Source: §1 · p. 2

• Vision SAEs range from standard ReLU-based designs to specialized sparse architectures, and target either global image features or localized patch-level concepts.Source: §2 · p. 2
• Reconstruction, sparsity, and dictionary orthogonality are indirect interpretability indicators, and sparsity alone can trade away completeness.Source: §2 · p. 2
• Other SAE proxies assess downstream task effects, similarity to input-space or ontology concepts, or dictionary similarity measures such as OOD score, Coherence, and Connectivity.Source: §2 · p. 2
• Ground-truth-oriented vision evaluations use VLMs and segmentation masks, annotated concepts, or user-defined concepts in training.Source: §2 · p. 3
• AIS was developed for sequence models and later made more compute-efficient; protein-language-model SAEs have also been assessed with AIS and Gene Ontology analysis.Source: §2 · p. 3

3.1 NOTATION

• SAEs are trained on patch embeddings mapped from images, producing higher-dimensional sparse codes; image-level codes can be obtained by max-pooling patch features.Source: §3.1 · pp. 3–4
• The encoder maps each patch embedding to sparse codes that the decoder uses with its dictionary matrix to reconstruct the embedding.Source: §3.1 · pp. 3–4

3.2 LINEAR REPRESENTATION HYPOTHESIS

• The LRH posits that high-dimensional representations decompose into superpositions over approximately orthogonal directions aligned with human-interpretable concepts.Source: §3.2 · p. 4
• The LRH premise implies that an SAE dictionary should converge toward a near-orthogonal basis whose sparse codes reconstruct the embedding.Source: §3.2 · p. 4
• Overcompleteness, low coherence, and sparse codes are structural proxies, but they are not sufficient to guarantee alignment with human-interpretable concepts.Source: §3.2 · p. 4
• Other discussed challenges include non-linearly separable, heterogeneous, or interacting concepts; Bhalla et al. (2026) argue for analyzing concept manifolds instead of individual directions.Source: §3.2 · pp. 4–5

4 METHODS

• The method uses patch captions to localize vision SAE activations, then scores whether an LLM can predict patch activations from its inferred concept.The section also situates AIS alongside standard SAE metrics, which are detailed in the appendices.Source: §4.1 · p. 5, §4 · p. 5

4.1 AIS PROCEDURE

• Captioning each image on a 4 × 4 grid makes patch-level embeddings comparable to AIS’s token-level localization in text.Grid resolution is a hyperparameter that can strongly influence downstream explanation quality; the experiments use LLaVa 1.5 for captions.Source: §4.1 · p. 5, §4.2 · p. 6
• For each selected neuron, the LLM explainer infers a common concept from caption–activation pairs drawn from five of its top-activating images.Activations are normalized to integers from 0 to 10, with negative values mapped to 0 and the maximum mapped to 10.Source: §4.1 · pp. 5–6
• The simulator predicts patch activations from the inferred concept using five other high-activating images and five random images; AIS is the Pearson correlation with true activations, averaged across neurons.This average is the paper’s global Mean AIS measure for an SAE.Source: Figure 1 · p. 5, §4.1 · p. 6

4.2 DATA AND EXPERIMENTS

• Experiments use CUB-200-2011, ImageNet100, and Caltech 101 with DINOv2, ViT, or SigLIP embeddings; the tested SAEs include Vanilla, TopK, and MP-SAE.ImageNet100 contains 100 classes with around 200 samples each; the algorithm lists LLaVa captions and Gemma as the LLM, while the study tested Gemma 3 and Gemma 4 and used the same model as explainer and simulator.Source: §4.2 · p. 6

1) LLM Explanations Align with Human Judgment

• Image-level AIS correlates positively with survey participants’ AIS (r = 0.6202), though the estimate is imprecise given the small sample.The LLM identified 4 degenerate all-zero cases; 7 of 16 users identified all or almost all of them.Source: §5 · p. 7, Figure 2 · p. 7
• On two MP SAE neurons, the Gemma pipeline outperforms users on AIS, while users show substantial disagreement and find activation simulation more challenging than identifying concepts.Most users identified “branches” and “water”; the authors suggest different model scales for explanation and simulation as a possible efficiency improvement.Source: Table 1 · p. 7, §5 · p. 7

2) High Reconstruction - Low Interpretability

• All SAE configurations achieve high embedding reconstruction R², yet their interpretability metrics indicate opaque features, with AIS and MS both below 0.5.Connectivity is maximal for almost all configurations; the patch-level SAEs also have lower MS than the cited image-level range.Source: §5 · p. 7, Table 2 · p. 8

3) Interpretability Metrics Do Not Correlate as Expected

• Only Sparsity–OOD Score correlates as expected; Sparsity–Coherence has the opposite relationship to the LRH expectation, and MS and AIS do not correlate with each other or highly with other metrics.The correlation matrix covers the SAE metrics; the authors treat ±0.6 as a suitable threshold for high correlation.Source: Figure 3 · p. 8, §5 · p. 8

Vision AIS Limitations

• Captioning may discard visual information, and errors from captioning, explanation, and simulation make it unclear how much a low AIS reflects poor SAE-concept interpretability.The authors also note that generic generated terms affect simulated activation quality.Source: §5 · p. 9

4) Captioning and Smaller LLMs Integrate Well in AIS

• With Vanilla SAE + DINO + CUB, some 4B Gemma versions achieve AIS similar to models three or six times larger, while shuffled captions yield much lower, even negative, AIS.The reported AIS uses the top 100 SAE features; the external NLP and PLM reference results use 150 and 200 features, respectively.Source: §5 · pp. 8–9, Table 3 · p. 9

Vision AIS Limitations

• A static captioning grid limits detail and provides no scene understanding, while dynamic-region methods or direct image analysis by VLMs are proposed alternatives.Source: §5 · p. 9

6 CONCLUSION

• Standard SAE metrics from within and outside the concept dictionary correlate neither with one another nor with AIS, while the caption-based vision AIS captures human-perceived explanation quality in a user study.The AIS adaptation had previously been shown to align with human explanations in NLP.Source: §6 · p. 9
• The paper supports calls for more formal explanation methods, deeper understanding of XAI metrics, and greater task- and audience-specific customization.It also points to supervision and ground-truth data for explainers, and to noise and uncertainty in explanations.Source: §6 · pp. 9–10
• The authors call for more in-depth study of SAE efficiency and interpretability beyond standard LRH assumptions.They argue that no all-purpose SAE design is guaranteed to deliver useful explanations across data and models.Source: §6 · p. 10
• They argue that SAE training and evaluation should interweave task-domain knowledge with reality-grounded metrics.Source: §6 · p. 10

A EXTENDED RELATED WORK

• Sparse coding represents signals with a small subset of an overcomplete basis, while distributed coding represents information across populations of neurons.Sparse coding and ICA both seek interpretable components, but ICA enforces statistical independence while sparse coding optimizes structural sparsity.Source: A EXTENDED RELATED WORK · p. 18
• SAEs adapt sparse dictionary learning to deep networks by using a latent sparsity penalty to reconstruct dense activations from a small set of latent directions.Earlier overcomplete sparse coding produced localized, Gabor-like filters analogous to simple cells in visual cortex.Source: A EXTENDED RELATED WORK · p. 18
• Image-level VLM descriptions can favor holistic objects over fine visual textures, while CLIP-based methods generally compare global visual and textual embeddings.Instruction-tuned VLMs such as InstructBLIP and LLaVA add spatial or context awareness through prompting.Source: A EXTENDED RELATED WORK · p. 18
• Localized and dense captioning methods use segmentation or patch-centric approaches to describe arbitrary regions, masks, or non-contiguous areas.Examples include Segment and Caption Anything, Describe Anything Model, and Patch-ioner.Source: A EXTENDED RELATED WORK · p. 18
• Vision-neuron description methods range from fixed-label alignment in Network Dissection to open-ended human-validated descriptions in MILAN and VLM scoring in CLIP-Dissect.Network Dissection is limited by its predetermined concept vocabulary; CLIP-Dissect does not require labeled probing data.Source: A EXTENDED RELATED WORK · pp. 18–19

B SAE TRAINING

• SAEs jointly minimize reconstruction error between original patch embeddings and their reconstructions while maintaining sparse latent codes.The general objective can include an MSE reconstruction term, a sparsity penalty, and an auxiliary term for inactive codes.Source: B SAE TRAINING · p. 19
• Vanilla SAEs use a ReLU projection and L1 sparsity penalty, while TopK SAEs retain only the K largest activations and may use an auxiliary loss to revive dead codes.The TopK formulation replaces the explicit sparsity penalty with direct control over the L0 norm.Source: B SAE TRAINING · p. 19
• MP-SAEs iteratively select the dictionary atom most aligned with the residual, yielding at most T active features and training with MSE alone.The encoder and decoder share a matrix, and each step updates the reconstruction and residual.Source: B SAE TRAINING · p. 19
• The largest SAE dictionary size was 12,288, and TopK configurations retained 20% of latent features while MP-SAE used T = 100 iterations.The dictionary size was 16 times the largest embedding dimension considered; this yielded average MP-SAE sparsity of approximately 0.0081.Source: B SAE TRAINING · p. 20
• SAEs were applied to final and intermediate backbone embeddings without training a task-solver head, and preliminary trials found that omitting encoder batch normalization degraded four reported dictionary metrics.Vanilla and TopK models trained for 30 epochs and MP-SAE for 10; the encoder used fixed running statistics at inference.Source: B SAE TRAINING · pp. 19–20

C OTHER SAE EVALUATION METRICS

• Monosemanticity Score weights pairwise image similarity by shared neuron activation to assess the global purity of each SAE neuron.The similarity matrix uses an embedder different from the one being explained.Source: C OTHER SAE EVALUATION METRICS · p. 20
• OOD Score measures how far dictionary atoms deviate from their closest patch-level embeddings; zero indicates exact matches, while higher values indicate drift from the data manifold.The metric uses patch-level embeddings because that is the feature space the SAE reconstructs.Source: C OTHER SAE EVALUATION METRICS · p. 21
• Coherence is the maximum pairwise cosine similarity between distinct dictionary rows, with lower values indicating more diverse directions and higher values indicating near-duplicate features.Source: C OTHER SAE EVALUATION METRICS · p. 21
• Connectivity measures the fraction of distinct image-level concept pairs that co-activate at least once; higher values indicate broader combinations, while lower values indicate more modular codes.The authors omit the subtraction from 1 used in the original implementation.Source: C OTHER SAE EVALUATION METRICS · p. 21

D MAX FILTER

• The MAX filter adaptively pools each feature map to a 4 × 4 grid, removes features below a pooled-patch firing-frequency threshold, then ranks the remainder by mean pooled activation and keeps 200.The stated threshold requires more than 320 positive patch-level activations.Source: D MAX FILTER · p. 21

E EXTRA AIS VALUES

• Table 4 reports the percentage of neurons with p < 0.05 for a test of uncorrelated predicted and ground-truth activations; errors and zero-pairs are excluded from AIS.Errors mean different predicted and ground-truth activation counts; NaNs denote zero-pairs.Source: Table 4 · p. 22
• Top-K SAEs score lower in Mean and Median AIS than Vanilla and MP SAEs across embedders and datasets, alongside lower significance rates.Significance is tested per neuron over 10 held-out images, so weaker correlations are more likely to fall below the threshold.Source: E EXTRA AIS VALUES · p. 22

F EXAMPLES LLM EXPLANATIONS AND SIMULATIONS

• Figures 4 and 5 show example prompts and outputs for a Vanilla SAE on DINO embeddings and CUB data, with Gemma4-12b as explainer and simulator.Figure 6 summarizes the frequency of common concepts in the generated explanations.Source: F EXAMPLES LLM EXPLANATIONS AND… · p. 22, Figure 4 · p. 23, Figure 5 · p. 24

G USER STUDY

• The study began with 21 participants, was anonymous, and was approved by the authors’ ethics board.Participants were told their annotations would be used solely for research; Figure 7 shows interface excerpts.Source: G USER STUDY · p. 26, Figure 7 · p. 26
• Participants identified concepts for two DINO-trained MP SAE neurons—tree branches and water—using five top-activating images and patch activation scores.The scores were displayed on images, mapped from 0–10 to low, medium, or high; the study averaged 25 minutes.Source: G USER STUDY · p. 26
• In the simulation task, participants annotated 10 top-random examples per neuron (5 top-activating and 5 random), with 16 scores per image; images in both tasks and neurons were randomly shuffled for each user.Figure 8 presents Gemma and example user explanations for the two surveyed neurons.Source: G USER STUDY · p. 26, Figure 8 · p. 27
• After removing very unspecific or missing explanations, 16 users’ responses were retained for statistics; three zero-variance non-zero annotations were excluded from average AIS values.The three annotations were among 224 simulation-task user annotations; such values were also absent from the SAE ground truth.Source: G USER STUDY · p. 26

H ABLATION STUDY: LAYER-WISE MS AND AIS

• The highest MS and AIS values occur in earlier DINO layers rather than the final embeddings on CUB.Table 5 compares Vanilla SAE features and raw DINO embeddings across layers; its right-half NaNs arise from zero-pairs and are excluded from AIS.Source: Table 5 · p. 28, H ABLATION STUDY: LAYER-WISE MS AND AIS · p. 28
• Raw input embeddings have higher MS but lower AIS than their corresponding SAE features across layers, showing the two metrics can disagree.The authors relate this inconsistency to the lack of correlation documented in Figure 3.Source: H ABLATION STUDY: LAYER-WISE MS AND AIS · p. 28
• No-SAE cases have more NaNs toward earlier layers; these arise from zero-pairs, and non-degenerate predictions show meaningful variance rather than defaulting to zero.Zero-pair consistency alone would not establish simulator calibration, since an always-zero simulator would also pass that check.Source: H ABLATION STUDY: LAYER-WISE MS AND AIS · p. 28