PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpreting CLIP's Image Representation via Text-Based Decomposition

Yossi Gandelsman, Alexei A. Efros, Jacob Steinhardt

arXiv:2310.05916v4cs.CVcs.AI

TL;DR

CLIP representations are useful across tasks, but the information they encode remains poorly understood. The paper decomposes CLIP-ViT representations into text-interpretable contributions from layers, heads, and image locations, finding property-specific heads and emergent localization. These findings support spurious-cue reduction and improved zero-shot segmentation.

  • Problem

    CLIP representations support many downstream tasks, but there is limited understanding of what information they encode.

  • Method

    The paper decomposes CLIP-ViT representations across layers, attention heads, and image locations, using the shared language space to label the resulting directions with text.

  • Results

    48% to 73%: removing heads associated with spurious cues improves Waterbirds worst-group accuracy, while the spatial decomposition yields a zero-shot segmenter that outperforms existing CLIP-based methods.

  • Takeaways & Limitations

    The analysis shows that interpreting CLIP’s internal structure can support reducing spurious cues and improving zero-shot segmentation.

  • Takeaways & Limitations

    The investigation analyzes only direct effects of model components and does not study indirect information flow from early to deeper layers.

Abstract

from arXiv · show

We investigate the CLIP image encoder by analyzing how individual model components affect the final representation. We decompose the image representation as a sum across individual image patches, model layers, and attention heads, and use CLIP's text representation to interpret the summands. Interpreting the attention heads, we characterize each head's role by automatically finding text representations that span its output space, which reveals property-specific roles for many heads (e.g. location or shape). Next, interpreting the image patches, we uncover an emergent spatial localization within CLIP. Finally, we use this understanding to remove spurious features from CLIP and to create a strong zero-shot image segmenter. Our results indicate that a scalable understanding of transformer models is attainable and can be used to repair and improve models.

1 INTRODUCTION

The paper develops a text-based decomposition of CLIP-ViT representations across layers, heads, and image locations to expose interpretable structure. It uses these findings to reduce spurious cues and improve zero-shot segmentation.

  • 1 INTRODUCTION: CLIP’s image representation is decomposed into text-explainable directions attributed to individual attention heads and image locations.Residual connections expose layer contributions, attention exposes location contributions, and the shared vision-language space supplies text labels.
  • 1 INTRODUCTION: Ablating all but the last 4 attention layers causes only a small zero-shot classification accuracy drop, indicating that late attention layers primarily construct the representation.
  • 1 INTRODUCTION: TEXTSPAN finds text-labeled bases for attention-head output spaces, revealing specialized roles such as shape processing.One head’s top directions include “A semicircular arch,” “A isosceles triangle,” and “oval.”
  • 1 INTRODUCTION: 48% to 73%: removing heads associated with spurious cues improves Waterbirds worst-group accuracy.The identified head roles also support retrieval based on discovered similarities such as color, location, and texture.
  • 1 INTRODUCTION: Attention-based spatial decomposition highlights image regions contributing to text directions and yields a zero-shot image segmenter that outperforms existing CLIP-based methods.Regions matching triangles primarily contribute to the direction labeled “isosceles triangle,” validating the text labels.

2 RELATED WORK

Prior explainability methods use heatmaps, feature inversion, neurons, or text concepts to interpret vision models. This paper positions its approach within text-based interpretation of CLIP representations and related vision-language applications.

  • 2 RELATED WORK: Heatmap methods identify image regions relevant to model outputs but do not explain non-spatial attributes such as object size or shape.
  • 2 RELATED WORK: Feature inversion and neuron- or connection-level analyses interpret vision models primarily through visual outputs.
  • 2 RELATED WORK: A small number of methods use text to interpret intermediate vision representations, including region descriptions, text-concept projections, multimodal neurons, and CLIP entanglement analyses.
  • 2 RELATED WORK: Contrastive vision-language models such as CLIP and ALIGN support zero-shot transfer and applications including OCR, geo-localization, classification, segmentation, 3D-scene querying, and text-based image generation.

3 DECOMPOSING CLIP IMAGE REPRESENTATION INTO LAYERS

The paper decomposes CLIP-ViT’s image representation into direct contributions from layers, attention heads, and image tokens, then interprets these contributions in the shared text-image space. Mean-ablation shows that the final four attention layers dominate the direct effect on zero-shot classification, while MLPs and earlier attention layers contribute little directly.

  • CLIP-ViT preliminaries: CLIP uses image and text encoders that map inputs into a shared latent space, enabling cosine-similarity-based zero-shot classification.Class names are encoded with a fixed template, and the class whose text representation has highest similarity to the image representation is predicted.
  • Decomposition into layers: ViT’s residual structure expresses the image representation as a sum of direct contributions from individual layers, including MLPs, MSAs, and the input class token.The analysis deliberately ignores indirect effects of one layer’s output on downstream layers.
  • Decomposition into layers: 1%-3% accuracy drop follows simultaneous mean-ablation of all MLPs, indicating that MLPs have a negligible direct effect on the image representation.Mean-ablation replaces each component with its mean over the ImageNet validation set and measures the resulting drop in zero-shot classification accuracy.
  • Decomposition into layers: Only the last four MSA layers have a large direct effect: ablating earlier MSAs changes accuracy little, whereas ablating the final MSAs sharply reduces performance.The accumulated-ablation experiment replaces MSA effects up to a given layer with their ImageNet-validation-set mean.
  • Fine-grained decomposition into heads and positions: Each MSA output can be decomposed into contributions from individual attention heads and image tokens, whose vectors remain interpretable through text representations.Contracting the contribution tensor across selected dimensions yields head-level or token-level contributions, and inner products with text representations measure similarity to descriptions.

4 DECOMPOSITION INTO ATTENTION HEADS

The paper decomposes CLIP’s image representation into attention-head contributions and labels each head’s directions with text, revealing property-specific roles that support interpretability and model repair.

  • TEXT-INTERPRETABLE DECOMPOSITION INTO HEADS: The decomposition expresses MSA outputs as sums of individual attention-head contributions in the joint text-image representation space.Text directions provide labels for these head-level contributions.
  • TEXT-INTERPRETABLE DECOMPOSITION INTO HEADS: TEXTSPAN greedily selects text descriptions whose projected representations explain orthogonal variance in each head’s output space.It starts from candidate descriptions and repeatedly projects selected components away before choosing the next one.
  • EXPERIMENTS: Applying TEXTSPAN to the last four layers yields text-interpretable bases that approximate each head’s 768-dimensional output in a 60-dimensional subspace.With 60 descriptions per head, ImageNet accuracy reaches 72.77%, compared with 75.25% base accuracy.
  • EXPERIMENTS: Many heads exhibit property-specific roles, including shape, texture, object count, color, counting, and location.Retrieved images and top descriptions qualitatively match the properties assigned to these heads.
  • EXPERIMENTS: Ablating heads associated with spurious location cues raises Waterbirds worst-group accuracy from 48% to 73% for ViT-L without additional training.The intervention removes geolocation and image-location heads identified through TEXTSPAN.
  • EXPERIMENTS: Head-specific contributions also enable image retrieval according to discovered properties such as color, location, and texture.Nearest neighbors differ by head and match the property captured by that head.

5 DECOMPOSITION INTO IMAGE TOKENS

The image-token decomposition attributes CLIP’s text-aligned representation to spatial locations, revealing regions matching described content and producing a state-of-the-art zero-shot segmenter.

  • DECOMPOSITION INTO IMAGE TOKENS: The resulting regions match the image parts described by the text, yielding a zero-shot semantic image segmenter.Joint head-and-location decomposition further links TEXTSPAN directions to contributing image regions.
  • DECOMPOSITION INTO IMAGE TOKENS: Grouping head contributions by image position decomposes CLIP’s representation into per-token contributions.The class token has negligible direct effect on zero-shot accuracy, so the analysis focuses on image tokens.
  • DECOMPOSITION INTO IMAGE TOKENS: For a text description t, each image position receives a heatmap score from the inner product between its token contribution and Mtext(t).This measures how strongly each location writes along the text direction.
  • DECOMPOSITION INTO IMAGE TOKENS: On ImageNet-segmentation, the image-token decomposition is more accurate than existing methods across all evaluated metrics.Heatmaps are thresholded into foreground/background masks under a standard explainability evaluation protocol.

6 LIMITATIONS AND DISCUSSION

The investigation finds that its analysis can improve downstream classification and zero-shot segmentation, while identifying limits in direct-effect analysis and head-level interpretability. Future work extends the analysis to other CLIP architectures and more complex structures.

  • The analysis reduced spurious cues in downstream classification and improved zero-shot segmentation.
  • The investigation analyzes only the direct effects of model components, leaving information flow from early layers to deeper ones for future study.
  • Not every attention head captures a single image property, possibly because properties are absent from the description pool or emerge only through head collaboration.
  • Future work will analyze other CLIP architectures and explore whether these insights can guide better image encoder architectures and feature extractors.

A.1 LAYER NORMALIZATION

The appendix adapts the decomposition to layer normalizations applied before the ViT layers and before the projection layer. These modifications preserve the corollaries about direct MLP and MSA contributions and are applied to OpenCLIP-based models.

  • Layer-normalization modifications address normalization before the projection layer and before each ViT layer.
  • Pre-projection layer normalization: The pre-projection normalization uses the forward-pass mean and standard deviation, absorbs the multiplicative term into projection matrix P, and incorporates the remaining contribution into the decomposition.
  • The described normalization modifications are applied when decomposing OpenCLIP-based models.
  • MLPs and MSAs input layer normalizations: Input layer normalizations modify the residual updates for MLP and MSA layers, including the corresponding token-level contributions.
  • These normalization changes do not affect the corollaries about direct MLP and MSA contributions because the analysis addresses their outputs.

A.2 MEAN-ABLATION OF THE CLASS-TOKEN ATTENDED FROM ITSELF

The appendix evaluates whether the class token’s direct contribution to the MSA term can be ignored when decomposing attention across tokens. Mean-ablating this contribution, together with all MLP contributions, causes only a negligible ImageNet zero-shot accuracy drop across three ViT models.

  • Mean-ablating the direct class-token contribution in the decomposed MSA term and all MLP contributions causes a negligible ImageNet zero-shot classification drop across three ViT models.

A.3 TEXT DESCRIPTIONS

The appendix describes the text-description sets used by the algorithm and its baselines. These include ChatGPT-generated image descriptions, common English words, class-specific descriptions, and random vectors, alongside prompts documented in Table 6.

  • General text descriptions: ChatGPT-generated image descriptions produced 3498 sentences, with manual prompting adding examples for patterns such as colors and letters.
  • Most common words: The most-common-word set contains 3498 English words selected using n-gram frequency analysis of Google’s Trillion Word Corpus.
  • Class-specific text descriptions: Class-specific prompting for each ImageNet class produced 28767 unique sentences.
  • Random vectors: The random-vector baseline consists of 3498 vectors sampled from a unit Gaussian.
  • Table 6 documents the prompts used to generate image descriptions.

A.4 ADDITIONAL INITIAL DESCRIPTION POOL ABLATION

Additional experiments show that class-specific description pools improve TEXTSPAN accuracy with fewer descriptions per head, while m = 60 yields similar accuracy across pools. The decomposition also produces text-aligned heatmaps and head-specific image retrieval results that generally reflect described attributes.

  • Initial description pool ablation: Class-specific description pools, containing around ×8 more examples, achieve higher accuracy with fewer descriptions per head than general pools.With m = 60, both pools produce relatively similar accuracy.
  • Initial description pool ablation: m = 60 produces relatively similar accuracy for the class-specific and general description pools.The comparison is reported across the evaluated initial description pools.
  • Attention-head interpretation: TEXTSPAN is applied to attention heads in the last four layers of CLIP ViT-L, with top-5 descriptions reported for layers 20 through 23.The reported results appear in Tables 7–10.
  • Image token decomposition: Text-description heatmaps highlight image objects mentioned in the text, while relative heatmaps isolate areas associated with one description rather than another.Relative heatmaps are formed by subtracting the two description heatmaps.
  • Attention-head interpretation: For randomly selected attention heads, the top retrieved text representations generally correspond to the attributes of the retrieved ImageNet images.The retrieval compares head contributions with the top 10 text representations found by TEXTSPAN and returns the three highest-scoring images per head.
  • Additional analyses: The appendix also reports additional joint decomposition examples, comparisons with other explainability methods, and Waterbirds evaluations after reducing spurious cues.The Waterbirds tables describe zeroing direct effects of property-specific heads and reducing background–class correlations.
Loading 2310.05916v4…