PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpreting CLIP with Hierarchical Sparse Autoencoders

Vladimir Zaigrajew, Hubert Baniecki, Przemyslaw Biecek

arXiv:2502.20578v2cs.CVcs.AIcs.LG

TL;DR

CLIP is difficult to interpret because its multimodal representations are distributed, while existing SAE approaches impose a trade-off between reconstruction quality and sparsity. The paper introduces MSAE, which learns nested representations at multiple granularities, and reports a leading frontier for CLIP together with concept-level applications and stated implementation limitations.

  • Problem

    CLIP’s distributed representations are difficult to interpret, while existing SAE methods face a reconstruction–sparsity trade-off and challenges in validating multimodal semantic features.

  • Method

    MSAE applies progressively increasing TopK operations and combines reconstruction losses across granularities to learn hierarchical CLIP representations.

  • Results

    MSAE achieves 0.99 cosine similarity, < 0.1 FVU, and ∼80% sparsity, while identifying over 120 validated concepts for similarity search and bias analysis.

  • Takeaways & Limitations

    MSAE provides a CLIP analysis tool for concept-based similarity search, controllable concept manipulation, and downstream gender-bias analysis.

  • Takeaways & Limitations

    Multiple decoder passes with different TopK activations introduce computational overhead, motivating optimized parallel CUDA kernels.

Abstract

from arXiv · show

Sparse autoencoders (SAEs) are useful for detecting and steering interpretable features in neural networks, with particular potential for understanding complex multimodal representations. Given their ability to uncover interpretable features, SAEs are particularly valuable for analyzing large-scale vision-language models (e.g., CLIP and SigLIP), which are fundamental building blocks in modern systems yet remain challenging to interpret and control. However, current SAE methods are limited by optimizing both reconstruction quality and sparsity simultaneously, as they rely on either activation suppression or rigid sparsity constraints. To this end, we introduce Matryoshka SAE (MSAE), a new architecture that learns hierarchical representations at multiple granularities simultaneously, enabling a direct optimization of both metrics without compromise. MSAE establishes a new state-of-the-art Pareto frontier between reconstruction quality and sparsity for CLIP, achieving 0.99 cosine similarity and less than 0.1 fraction of variance unexplained while maintaining ~80% sparsity. Finally, we demonstrate the utility of MSAE as a tool for interpreting and controlling CLIP by extracting over 120 semantic concepts from its representation to perform concept-based similarity search and bias analysis in downstream tasks like CelebA. We make the codebase available at https://github.com/WolodjaZ/MSAE.

1. Introduction

CLIP is widely used for multimodal representation and downstream vision-language systems, but its distributed representations remain difficult to interpret. The paper introduces MSAE, a hierarchical SAE designed to improve the reconstruction–sparsity trade-off while supporting concept-level analysis of CLIP.

  • Motivation: CLIP provides robust visual-textual representations used in multimodal understanding and downstream systems.Its representations support cross-modal similarity search, text-to-image generation, and other vision-language tasks.
  • Motivation: CLIP’s distributed representations and complex training objective make its information difficult to interpret.Existing gradient-, perturbation-, and concept-based approaches have limited success or depend on manually curated concepts.
  • Motivation: SAEs must balance reconstruction quality, sparsity, and semantic consistency across CLIP’s multimodal embedding space.Existing sparsity mechanisms include L1 and TopK proxy functions, each with distinct limitations.
  • Contribution: MSAE applies progressively larger TopK operations and combines losses across granularities to learn coarse-to-fine CLIP representations.This design avoids rigid TopK constraints and L1-associated activation shrinkage while targeting a stronger reconstruction–sparsity frontier.
  • Contribution: 0.99 cosine similarity, < 0.1 FVU, and ∼80% sparsity define the reported leading reconstruction–sparsity frontier.The paper also reports over 120 validated concepts and applications in concept-based similarity search and CelebA bias analysis.

2. Related Work

Prior work studies CLIP through components such as features, residual transformations, attention heads, and neurons, while mechanistic interpretability increasingly uses SAEs to recover interpretable features. Concept-based explainability has likewise progressed from curated concepts toward automated extraction and explicit concept learning.

  • Interpreting CLIP models: Direct CLIP interpretability examines feature attributions, residual transformations, attention heads, and individual neurons.These approaches target different components of CLIP’s representation and computation.
  • Mechanistic interpretability: Mechanistic interpretability frames neural-network analysis as reverse engineering, with SAEs helping recover monosemantic features from polysemantic representations.SAE-based methods have been applied beyond language models, including diffusion models and medical imaging.
  • Concept-based explainability: Concept-based explanations identify human-coherent concepts in latent spaces, evolving from manually curated datasets toward automated concept extraction and explicit concept learning.This line of work provides a complementary route to interpreting neural representations.

Matryoshka Sparse Autoencoder

MSAE extends sparse autoencoders with nested representations learned at multiple sparsity levels. It combines progressively increasing TopK operations, flexible inference-time sparsity, and modality-aware preprocessing to address limitations of L1 and fixed-TopK approaches.

  • Preliminaries: SAEs reconstruct activations as sparse combinations of learned directions, using encoder and decoder parameters to support interpretable latent features.The standard reconstruction objective minimizes squared reconstruction error.
  • Preliminaries: L1 regularization can shrink activations, whereas TopK enforces rigid sparsity; BatchTopK relaxes per-sample rigidity but still requires a predetermined k and may produce dead features.These limitations motivate a hierarchy of sparsity levels rather than one fixed threshold.
  • Matryoshka architecture: MSAE applies multiple TopK operations with increasing k values and optimizes reconstruction across all granularities simultaneously.The proposed k values are powers of 2 up to latent dimension d, providing coverage of the representation space at manageable computational cost.
  • Matryoshka architecture: At inference, MSAE can use any desired TopK granularity or remove TopK entirely and retain ReLU activations.This allows adaptive use of all neurons deemed necessary for reconstruction.
  • Hierarchical learning: Nested selections satisfy TopK_1 ⊆ TopK_2 ⊆ . . . ⊆ TopK_h, producing a hierarchy from sparse coarse features to denser representations.The hierarchy is motivated by samples requiring different numbers of concepts for optimal representation.
  • Sparsity weighting: Reverse weighting assigns larger loss weights to lower k values, improving sparsity without significant performance degradation relative to uniform weighting.Uniform weighting emphasizes reconstruction quality, whereas reverse weighting prioritizes sparse reconstructions.
  • Preprocessing: Embedding normalization centers and scales each modality so the sparsity coefficient has consistent effects across CLIP architectures and modalities.Inference can also remove TopK constraints from TopK-trained models to allow adaptive ReLU-based activation selection.

4. Evaluating MSAE

MSAE is evaluated against ReLU, TopK, and BatchTopK SAEs across sparsity–fidelity, semantic preservation, activation distributions, progressive recovery, and training-modality settings. Across these analyses, MSAE generally improves the sparsity–fidelity frontier, preserves semantic structure, and supports hierarchical recovery, while modality choice affects cross-domain generalization.

  • Sparsity–Fidelity Trade-off: MSAE achieves better sparsity than ReLU and better fidelity than TopK or BatchTopK, establishing a superior Pareto frontier with less than 1% dead neurons.The RW variant further improves sparsity with only minor fidelity degradation.
  • Semantic Preservation Analysis: MSAE preserves semantic structure beyond reconstruction distance, with alignment metrics showing its strength relative to alternative evaluation measures.The analysis reports that cosine similarity and FVU correlate with linear probing, while alignment assesses semantic structure preservation.
  • Activation Magnitudes Analysis: MSAE’s activation distribution resembles ReLU’s double curvature without activation shrinkage, while highly constrained TopK models show fewer valid named high-magnitude features.The paper associates high activations in constrained TopK models with more composite features; MSAE can also encode concepts using activations above 15.
  • Progressive Recovery: MSAE combines TopK’s high-sparsity performance with ReLU’s scaling capabilities during progressive recovery.It surpasses TopK’s plateau at lower sparsity and outperforms TopK (k = 32) and ReLU models on CKNNA beyond k = 10.
  • Training Modality: Text-trained MSAE variants generalize better across modalities, achieve higher sparsity on both modalities, and have no dead neurons.Both image- and text-trained models perform best in their respective training domains.

5. Interpreting CLIP with MSAE

MSAE supports interpretable and controllable analyses of CLIP embeddings through concept naming, similarity search, and downstream bias analysis. These applications include concept-based retrieval, controlled nearest-neighbor shifts, and gender-association analysis.

  • Concept naming and applications: MSAE enables neuron-concept mappings and two interpretability use cases using the RW variant with expansion rate 8.The paper applies these mappings to similarity search and downstream bias analysis.
  • Similarity search: MSAE extends image retrieval by exposing shared semantic patterns among nearest neighbors and enabling concept-based interpretation in CLIP and SAE spaces.The first nearest neighbor remains consistent across spaces, while SAE activations provide additional concept-level interpretability.
  • Similarity search: Increasing the germany concept preserves the original image as the top match but shifts the second nearest neighbor from an Irish to a German police vehicle.Larger concept magnitudes increase distance from the original embedding while preserving overall image structure.
  • Bias analysis: Approximately 0.99 F1 is achieved by a single-layer CelebA classifier, while MSAE reveals associations linking bearded with male and blonde with female classification.Concept manipulation also shows glasses and bearded concepts biasing toward male classification, while blonde biases toward female classification.
  • Concept naming and applications: Automated interpretability identifies the best-matching face neuron and compares its highest-activation text and image examples across TopK, ReLU, and MSAE.The comparison includes variants optimized for sparsity or reconstruction.

6. Conclusion

The paper concludes that MSAE improves CLIP interpretability through a flexible sparsity–fidelity trade-off and validated concept extraction. It also identifies computational and generalization boundaries for future work.

  • Conclusion: MSAE improves upon TopK and ReLU by providing flexible sparsity control and a superior sparsity–fidelity trade-off.The conclusion also reports near-optimal metrics and progressive feature recovery.
  • Conclusion: Over 120 validated concepts support concept-based similarity search and bias detection in downstream tasks.These applications demonstrate the use of extracted CLIP concepts for interpretation and control.
  • Limitations and future work: Multiple decoder passes with different TopK activations introduce computational overhead in the current implementation.The paper suggests optimized CUDA kernels for parallel processing across granularities as a future improvement.
  • Limitations and future work: The demonstrated evaluation uses CLIP embeddings, while application to SigLIP and modality-specific representations remains future work.The paper also notes that complex semantic features may require LLM-based interpretability methods.

A. Concept Discovery and Validation

The concept-discovery pipeline matches vocabulary concepts to CLIP SAE decoder features and validates assignments using similarity, uniqueness, and manual consistency checks. Results show that sparsity affects the number of interpretable concept neurons, while pre-activation bias substantially changes similarity magnitudes without usually changing rankings.

  • Concept matching: CLIP-based concept matching assigns each neuron the vocabulary concept whose embedding has maximum cosine similarity with its decoder column.Decoder columns are unit-magnitude, and a similarity of 1 indicates perfect alignment.
  • Similarity calibration: Retaining pre-activation bias yields similarities above 0.42 that correspond to correct concepts, whereas removing it clusters scores near 0.1–0.2.Both approaches preserve neuron rankings, with over 95% of concepts sharing identical highest-matching neurons.
  • Limitations: Concept assignments can remain poor when no strong match exists, while hierarchical concepts such as mammal may ambiguously match cat and dog neurons.The paper attributes this to non-orthogonal semantic vectors or incomplete vocabulary coverage.
  • Validation: Validation removes weak assignments using cosine similarity > 0.42, a best-to-second-best similarity ratio > 2.0, and one concept per neuron.The criteria target strong alignment, concept uniqueness, and monosemantic assignments.
  • Vocabulary and semantic consistency: The vocabulary consists of frequent LAION-400M caption unigrams, with manual evaluation used to verify semantic consistency and hierarchical relationships.Examples include recognizing dog breeds as specific concepts within the broader category dog.
  • Results across SAE architectures: Approximately 10% of TopK neurons and 1–3% of ReLU and Matryoshka neurons pass similarity validation at expansion rate 8.Across architectures, TopK yields 3–8 times more interpretable concept neurons than denser ReLU models, with Matryoshka intermediate.

B. Implementation Details

The experiments compare ReLU, TopK, and Matryoshka SAE configurations across expansion rates and analyze high-magnitude activations. They also introduce soft-capping to limit activation magnitudes and address associated concept-composition concerns.

  • MSAE uses expansion rates of 8, 16, and 32, scaling ViT-L/14 latent layers from 768 to 6144, 12288, and 24576 neurons.
  • TopK models use k values from 32 to 256, while MSAE uses K-lists spanning 32 or 64 through the latent dimension.
  • MSAE evaluates uniform and reverse weighting, with reverse weighting coefficients decreasing from 7 to 1 across granularities.
  • More constrained TopK models produce more samples with activations above 15, but their high-magnitude concepts have lower validity rates than MSAE and TopK k = 256.
  • MSAE can encode concepts with activation values above 15, which may yield more composite rather than atomic features.
  • Soft-capping bounds ReLU-based SAE activations to (0, softcap) and is intended to prevent excessive magnitudes from circumventing sparsity constraints.

D.2. Results.

Soft-capping improves MSAE sparsity and reduces dead neurons, especially in wider sparse layers, while causing only minimal reconstruction-fidelity impact.

  • MSAE RW (SC) achieves L0 values of 0.830 and 0.889 at latent sizes 6144 and 12288, respectively.
  • At latent size 12288, MSAE RW has 66 dead neurons compared with 491 for the 6144-size model.
  • Soft-capping improves L0 sparsity and reduces dead neurons, particularly for larger MSAE layers, with minimal impact on reconstruction fidelity.

E. CKNNA Alignment Metric

CKNNA measures similarity between CLIP embeddings and SAE activations by emphasizing shared local neighborhood structure rather than only global representation similarity.

  • CKNNA compares CLIP and SAE representations using only point pairs that are k-nearest neighbors in both spaces.
  • Unlike CKA, CKNNA focuses on local geometric relationships through a neighborhood constraint.
  • The evaluation sets k = 10 and computes CKNNA over randomly sampled batches of 10,000 representations.

F. Evaluating MSAE: Additional Results

Additional experiments test MSAE across modalities, architectures, expansion rates, and metrics. They report stable performance across settings, competitive metric trade-offs, and consistent activation and progressive-recovery patterns.

  • Sparsity–fidelity trade-offs: MSAE models show stable and elevated text-modality results, while lower-k and more strongly regularized models exhibit greater variance, especially on text.
  • Cross-architecture evaluation: MSAE’s superior performance and stability generalizes across CLIP architectures.
  • Metric comparison: MSAE RW achieves similar or better EVR, CS, and CKNNA results than TopK, while MSAE UW performs better on NDN.
  • Activation distributions: TopK and MSAE retain activation distributions closer to normal than ReLU’s shrinkage-affected maximum activations, with patterns persisting at higher expansion rates.
  • Progressive recovery: Progressive-recovery analyses evaluate EVR, CS, CKNNA, and NDN across image and text modalities at expansion rates 8, 16, and 32.
  • Progressive recovery: The progressive-recovery results remain stable at higher expansion rates.
  • Evaluation scope: The extended evaluation covers ReLU, TopK, BatchTopK, and MSAE across ViT-L/14 and ViT-B/16, multiple expansion rates, and both modalities.

G.1. Concept Visualization Analysis

The analysis validates MSAE concepts through multimodal activation examples, similarity search, gender-association analysis, and stability measurements. Valid concepts align across images and texts, while invalid concepts motivate explicit validation.

  • Concept validation: Six valid MSAE concepts are illustrated with their highest-activating ImageNet-1k images and CC3M texts.The examples include smile, alcoholic, trio, heart, running, and questions.
  • Concept validation: Two invalid concepts demonstrate that low similarity or broad activation patterns can reveal unreliable or overly general concepts.The invalid examples include the number “6” and a broader “h” concept rather than specific vocabulary entries.
  • Similarity search: SAE activation similarity produces nearest-neighbor results comparable to CLIP search while exposing shared semantic concepts between query and retrieved images.The SAE representation makes the semantic basis of similarity matches more interpretable.
  • Gender associations: Significant gender associations appear for the concepts bearded, blondes, and glasses in CelebA-based analysis.Highest-activation images and concept-manipulation examples are used to examine alignment and gender bias.
  • Stability: MSAE exhibits a stability–sparsity trade-off comparable to alternative SAE architectures.The analysis also reports that stability is highly correlated with sparsity.
Loading 2502.20578v2…