PaperlayerView exact revision on arXiv

Source-linked AI summary

ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference

Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, Wayne Zhang

arXiv:2407.12442v1cs.CV

TL;DR

CLIP’s strong image-level vision-language representations remain difficult to use for open-vocabulary semantic segmentation because local patch-text alignment is limited and segmentation maps are noisy. ClearCLIP decomposes the final representation and modifies its last layer by removing the residual connection, using self-self attention, and discarding the FFN. The resulting approach produces clearer segmentation and performs effectively across benchmark datasets and architectures.

  • Problem

    CLIP-based dense segmentation is limited by noisy patch predictions and poor alignment between local image regions and text representations.

  • Method

    ClearCLIP removes the final residual connection and FFN while adopting self-self attention to strengthen attention-based representations.

  • Results

    ClearCLIP consistently improves segmentation across architectures, while removing the residual connection and FFN together reaches an mIoU of 37.5 versus 27.3 for the baseline ablation.

  • Takeaways & Limitations

    Residual features are identified as the primary source of segmentation noise, while reducing their influence and reorganizing spatial information improves dense vision-language inference.

Abstract

from arXiv · show

Despite the success of large-scale pretrained Vision-Language Models (VLMs) especially CLIP in various open-vocabulary tasks, their application to semantic segmentation remains challenging, producing noisy segmentation maps with mis-segmented regions. In this paper, we carefully re-investigate the architecture of CLIP, and identify residual connections as the primary source of noise that degrades segmentation quality. With a comparative analysis of statistical properties in the residual connection and the attention output across different pretrained models, we discover that CLIP's image-text contrastive training paradigm emphasizes global features at the expense of local discriminability, leading to noisy segmentation results. In response, we propose ClearCLIP, a novel approach that decomposes CLIP's representations to enhance open-vocabulary semantic segmentation. We introduce three simple modifications to the final layer: removing the residual connection, implementing the self-self attention, and discarding the feed-forward network. ClearCLIP consistently generates clearer and more accurate segmentation maps and outperforms existing approaches across multiple benchmarks, affirming the significance of our discoveries.

1 Introduction

CLIP performs well across many vision-language tasks but struggles with dense semantic segmentation because its representations produce noisy, poorly localized patch predictions. ClearCLIP attributes much of this noise to the residual connection and improves segmentation by emphasizing clearer attention-based representations.

  • 1 Introduction: CLIP’s image-level training supports broad downstream generality but provides limited sensitivity to dense visual localization.Patch-level similarity can produce misclassified regions and substantial segmentation noise.
  • 1 Introduction: Residual features produce noticeably noisier segmentation maps than attention outputs, which provide clearer localization.The paper therefore identifies the residual connection as the main source of segmentation noise.
  • 1 Introduction: Residual connections have near-zero normalized entropy and larger peak values, while attention outputs maintain normalized entropy near 1.Peak values concentrated in a few channels make spatial feature vectors harder to distinguish with cosine similarity.
  • 1 Introduction: ClearCLIP removes the residual connection, applies self-self attention, and discards the final feed-forward network.These modifications strengthen attention output and produce clearer representations for open-vocabulary semantic segmentation.
  • 1 Introduction: Experiments across 8 benchmark datasets demonstrate the effectiveness of ClearCLIP.The approach is presented as a simple adaptation of CLIP for dense vision-language inference.

2 Related Work

Related work applies large-scale vision-language models to open-vocabulary semantic segmentation, with training-free methods modifying CLIP’s final attention operations. ClearCLIP follows this training-free direction while studying localization through feature decomposition.

  • 2 Related Work: Open-vocabulary semantic segmentation assigns arbitrary text-described categories to image regions.The task is also referred to as zero-shot semantic segmentation.
  • 2 Related Work: Training-free methods seek CLIP’s intrinsic localization ability with minimal modifications and commonly revise final-layer attention.Examples include identical attention and self-self variants such as value-value, query-query, and key-key attention.
  • 2 Related Work: These attention modifications reorganize spatial information and substantially improve segmentation performance.They are used to make CLIP representations more suitable for dense prediction without retraining.
  • 2 Related Work: ClearCLIP belongs to training-free open-vocabulary semantic segmentation and investigates CLIP localization through feature decomposition.Its perspective separates components of CLIP representations rather than introducing retraining-based supervision.

3 Methodology

ClearCLIP adapts CLIP for open-vocabulary semantic segmentation by decomposing final-layer representations and reducing residual-driven noise. Its analysis links poor dense localization to dominant, low-entropy residual features and shows that removing the residual connection, revising attention, and discarding the FFN improve segmentation.

  • Representation decomposition: The final transformer block is decomposed into residual and attention outputs to diagnose their separate effects on dense prediction.The residual connection and attention output are explicitly separated in the last vision-encoder layer.
  • Baseline: CLIP’s dense baseline classifies local patches by cosine similarity between visual and textual features, but often produces unsatisfactory segmentation.The limitation is associated with image-level contrastive training and poor alignment between local regions and text representations.
  • ClearCLIP design: ClearCLIP removes the residual connection and FFN in the final block while revising attention, with the solution consistently improving segmentation across architectures and datasets.The method is designed for open-vocabulary semantic segmentation and its cross-architecture results support the combined removal of residual and FFN components.
  • ClearCLIP design: Larger residual norms impede attention-based improvements, while increasing Xattn relative to Xres improves performance and motivates discarding the residual connection.The scaling experiment uses Xsum = Xres + αXattn, and the authors identify residual reduction and spatial reorganization through self-self attention as critical factors.
  • Representation decomposition: Xattn has clearer localization than Xres, while Xres alone achieves near-zero mIoU on COCOStuff and Xattn alone exceeds Xsum.The visual and quantitative comparison decomposes CLIP’s noisy map into a muddled residual map and a clearer attention map.
  • Statistical diagnosis: CLIP’s Xres and Xsum entropies decline toward zero, whereas Xattn entropy remains stable; Xres and Xsum maximum values can peak nearly 90 times higher in the final layer.Low entropy indicates concentration in a few peak values, which can compromise local information.

4 Experiments

Experiments show that removing CLIP’s residual connection, using self-self attention, and discarding the FFN improves open-vocabulary semantic segmentation across architectures and benchmarks. ClearCLIP produces stronger quantitative and qualitative results, while amplifying attention output also mitigates residual-feature effects.

  • Experimental setup: The evaluation covers eight open-vocabulary semantic-segmentation datasets, grouped by whether they include a background category.Five datasets lack a background class, while three include one.
  • Ablation study: Removing the residual connection raises CLIP-B/16 average mIoU from 27.3 to 33.7, while removing both the residual connection and FFN reaches 37.5.Removing the FFN alone does not improve performance.
  • Architectures and attention: Across architectures, removing both the residual connection and FFN consistently improves segmentation, and self-self attention outperforms vanilla q-k attention in the CLIP-B/16 setting.For CLIP-B/16 without the residual connection and FFN, q-q attention achieves 37.5 mIoU versus 27.6 for q-k attention.
  • Attention-output scaling: Increasing the attention-output scaling factor improves performance across attention types, whereas decreasing it from 1 to 0.5 causes a sharp decline.These results support reducing the residual connection’s relative influence by enlarging the attention output.
  • Quantitative results: ClearCLIP achieves the best results on four of five datasets without background classes and improves average mIoU by 4.4 over TCL and 3.3 over SCLIP.On datasets with background classes, ClearCLIP improves average mIoU over TCL by 3.8.
  • Qualitative results: Qualitatively, ClearCLIP produces clearer and more accurate fine-grained segmentation maps than CLIP, MaskCLIP, and SCLIP.MaskCLIP retains noise and incoherent segments, while SCLIP detects detailed features with less noise.

5 Conclusion

The paper attributes CLIP’s noisy segmentation primarily to residual features and links their lack of local information to high-level supervision emphasizing global direction. ClearCLIP addresses this with three final-layer changes and shows strong performance across the CLIP family.

  • Findings: Comparisons of feature statistics identify the residual connection as the primary source of segmentation noise.The study examines norm differences across CLIP backbones and contrasts residual and attention outputs.
  • Method: ClearCLIP removes the residual connection and FFN while adopting self-self attention for dense vision-language inference.The method is presented as a simple solution for decomposing CLIP representations.
  • Conclusion: ClearCLIP demonstrates superior performance and generalizability within the CLIP family.

A Ablation study with different backbones and datasets

Ablations across CLIP backbones and datasets show that removing the residual connection and FFN consistently improves segmentation, especially for ViT-L/14 models with larger residual norms.

  • Ablation across datasets: Removing the residual connection and FFN markedly enhances open-vocabulary semantic segmentation across all evaluated datasets.
  • Backbone effects: The improvement is especially pronounced for ViT-L/14, whose residual connection has a larger norm.

B Impact of channel-wise residual features

Channel-wise experiments support the view that high-value residual features introduce noise by emphasizing global feature direction. Removing at least 70% of those channels or normalizing the residual features yields the strongest behavior described.

  • Channel ablation: The best performance is achieved when at least 70% of high-value residual channels are eliminated.The experiments selectively reintroduce lower-average-value residual channels after removing high-value channels.
  • Feature normalization: Normalizing the residual features substantially reduces their scale and produces performance comparable to eliminating at least 70% of high-value channels.
  • Interpretation: These results support the hypothesis that high-level supervision emphasizes global feature direction in residual latent space, introducing segmentation noise.

C Integration across models

ClearCLIP integrates across architectures and segmentation models through minimal code changes while preserving the original attention mechanisms.

  • ClearCLIP applies across architectures and segmentation models with only 2–3 lines of code modification.The passage describes this as a free-lunch integration strategy.
  • For MaskCLIP and SCLIP, the method removes the residual connection and Feed-Forward Network from the last self-attention layer.
  • For GEM, ClearCLIP uses the final layer’s attention output as the final representation while preserving the original attention mechanisms.

D Visualization of feature maps

Feature-map visualizations show that residual-connection features dominate the final representation through unusually large single-channel responses.

  • Residual-connection feature maps show peak values in one channel that substantially exceed the other channels.The visualization covers two randomly selected samples and highlights the dominant channel in a red box.
  • The summed feature maps resemble the residual feature maps, indicating the residual component strongly influences the final feature.

E Additional qualitative examples

Additional visual comparisons across COCOStuff, ADE20K, and Pascal Context59 show that ClearCLIP produces less noisy results than other methods. The accompanying materials also describe feature-map visualizations and evaluation over five datasets without the background class.

  • ClearCLIP exhibits much less noise than other open-vocabulary segmentation methods on COCOStuff, ADE20K, and Pascal Context59.Figures 10 and 11 provide qualitative comparisons across these datasets.
  • Table 5 reports average mIoU over five datasets without the background class for ViT-Base and ViT-Large architectures.
  • The feature-map visualization uses two randomly selected COCOStuff examples, showing the first 64 maps and all 768 maps for each feature type.
Loading 2407.12442v1…