SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference

Feng Wang; Jieru Mei; Alan Yuille

Source-linked AI summary

SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference

Feng Wang, Jieru Mei, Alan Yuille

arXiv:2312.01597v4Dec 4, 2023cs.CV

TL;DR

CLIP performs strongly in zero-shot classification but struggles to localize visual features for semantic segmentation. SCLIP replaces the final vision-encoder self-attention block with Correlative Self-Attention, achieving 38.2% average zero-shot mIoU across eight benchmarks while using minimal modification to pretrained CLIP.

Problem
CLIP's visual representations support zero-shot classification but often produce inaccurate, noisy semantic-segmentation masks because visual features are wrongly localized.
Method
SCLIP replaces CLIP's final vision-encoder self-attention block with Correlative Self-Attention, reusing pretrained projection parameters for training-free segmentation adaptation.
Results
38.2% average zero-shot mIoU across eight semantic-segmentation benchmarks, outperforming existing state-of-the-art methods.
Takeaways & Limitations
Minimal modifications to pretrained CLIP substantially improve its functionality for dense vision-language inference.
Takeaways & Limitations
The evaluation excludes computationally intensive post-processing such as Dense CRF and generally discards PAMR from default settings.

Abstract

from arXiv · show

Recent advances in contrastive language-image pretraining (CLIP) have demonstrated strong capabilities in zero-shot classification by aligning visual representations with target text embeddings in an image level. However, in dense prediction tasks, CLIP often struggles to localize visual features within an image and fails to give accurate pixel-level predictions, which prevents it from functioning as a generalized visual foundation model. In this work, we aim to enhance CLIP's potential for semantic segmentation with minimal modifications to its pretrained models. By rethinking self-attention, we surprisingly find that CLIP can adapt to dense prediction tasks by simply introducing a novel Correlative Self-Attention (CSA) mechanism. Specifically, we replace the traditional self-attention block of CLIP vision encoder's last layer by our CSA module and reuse its pretrained projection matrices of query, key, and value, leading to a training-free adaptation approach for CLIP's zero-shot semantic segmentation. Extensive experiments show the advantage of CSA: we obtain a 38.2% average zero-shot mIoU across eight semantic segmentation benchmarks highlighted in this paper, significantly outperforming the existing SoTA's 33.9% and the vanilla CLIP's 14.1%.

1 Introduction

CLIP transfers well to image-level zero-shot classification but struggles with semantic segmentation because its patch representations are spatially misaligned. SCLIP addresses this by replacing the final self-attention block with Correlative Self-Attention, enabling training-free dense prediction and stronger segmentation results.

Motivation: CLIP achieves strong image-level zero-shot classification but only 3.1% mIoU on ADE20k and 5.7% mIoU on COCO-Stuff when patch features are directly matched with text.These results remain far below supervised counterparts producing around 40% mIoU on both benchmarks.
Motivation: Qualitative examples show that CLIP can roughly recognize image objects but often assigns categories to the wrong locations, producing noisy masks.For example, flamingo and water are predicted at opposite locations in one example.
Analysis: CLIP’s segmentation weakness is attributed to spatially misaligned patch representations and spatial-invariant features rather than failure to extract dense visual features.Final-layer attention maps show local tokens attending broadly, with similar patterns across source points and a focus on holistic information.
Method: CSA promotes spatial-covariant features by projecting the input once to compute pairwise token correlations, encouraging each token to attend to itself and similarly informative positions.Dense prediction requires local representations to change with their spatial positions.
Method: SCLIP replaces the final vision-encoder self-attention block with CSA and reuses CLIP’s pretrained projection parameters, requiring neither fine-tuning nor additional parameters.The resulting stand-alone model provides a tuning-free adaptation for semantic segmentation.
Results: 38.2% average mIoU across eight semantic segmentation benchmarks substantially exceeds MaskCLIP’s 30.3%, GroupViT’s 30.7%, and TCL’s 33.9%.The reported results include PASCAL Context and COCO-Stuff and cover zero-shot, open-vocabulary segmentation.

2 Related Work

Prior work develops transferable visual representations through self-supervised, contrastive, generative, and language-guided pretraining. CLIP is a representative language-guided foundation model whose extensions target scale, applications, and open-vocabulary visual inference.

Transferable Visual Foundation Models: Self-supervised models using masked image modeling or contrastive learning can adapt to visual tasks when sufficient downstream training data is available.Denoising diffusion models and Segment Anything models are also described as potential foundation models with transferable visual features.
Transferable Visual Foundation Models: Vision transformers are more suitable than ResNet encoders for zero-shot transfer to semantic segmentation because they provide global receptive fields and lower down-sampling ratios.The cited example contrasts ViT-Base/16’s 16× down-sampling with ResNet-50’s 32× ratio.
Open-Vocabulary Segmentation: Language guidance can make foundation models useful for open-vocabulary and zero-shot downstream visual tasks.CLIP aligns visual and textual features through contrastive pretraining and has inspired extensions in scale, applications, and inference protocols.

3 Method

SCLIP adapts CLIP for dense prediction by replacing the final vision-encoder self-attention with Correlative Self-Attention, which converts spatial-invariant features into spatial-covariant representations. CSA uses pairwise correlations among local tokens, preserves semantic relationships, and supports training-free inference with reused CLIP projections.

Motivation: CLIP’s spatial-invariant features hinder semantic segmentation because dense prediction requires local representations that vary with spatial position.The method targets spatial misalignment rather than a failure to extract dense visual features.
Correlative Self-Attention: CSA computes attention scores from pairwise correlations between projected local tokens, making attention depend on feature similarity across positions.Its attention uses a single projection matrix Wr and temperature τ in Softmax(XWrWr^TX^T/τ).
Correlative Self-Attention: CSA improves localization by increasing attention toward each token’s own position while also connecting tokens with similar semantic content.This combination is intended to produce localized yet smooth dense predictions, unlike strictly identity attention.
SCLIP Adaptation: SCLIP replaces the last CLIP vision-encoder self-attention block with CSA and reuses pretrained Wq and Wk projections, requiring no training.The projection matrices are combined as the CSA projection, yielding a training-free adaptation.
Post-processing: SCLIP produces good segmentation results without refinement or smoothing strategies, treating spatial continuity as part of inference rather than post-processing.The method is reported to be robust without relying on output-refinement procedures.

4 Experiments

Experiments evaluate SCLIP across eight semantic segmentation benchmarks, compare it with CLIP-based baselines, and ablate projection, localization, and preprocessing choices. Results support CSA’s robustness, while denser sliding inference improves accuracy at comparable computation.

Main Results: SCLIP consistently achieves the best performance across eight evaluated semantic segmentation benchmarks, averaging 38.0% mIoU versus TCL’s 33.9%.The comparison includes zero-shot and open-vocabulary segmentation baselines, while vanilla CLIP performs poorly with its original self-attention.
Main Results: PAMR raises SCLIP’s average mIoU by 1.9% across eight datasets, but the authors de-emphasize it because the refinement strategy is computationally intensive.GroupViT and TCL gain 1.4% and 3.3%, respectively, with PAMR.
Projection Ablation: Projection variants produce minimal performance differences across three datasets, and the default training-free CSA setting generally outperforms alternatives excluding learned projection.A single randomly initialized matrix still reaches 57.1% mIoU on PASCAL VOC, while learned projection offers only a modest improvement.
Feature Localization Ablation: Attention sharpening and local attention can improve vanilla CLIP under selected settings, but they remain below CSA across the evaluated datasets.Attention sharpening gains only 2.9% mIoU on PASCAL VOC with τ = 2, while local attention with window size three nearly matches MaskCLIP.
Image Pre-processing Ablation: Larger images with smaller windows and strides generally improve performance but increase computation, while the proposed preprocessing matches existing computation and achieves 58.9% mIoU versus TCL’s 51.2% on PASCAL VOC.The default protocol uses a 336-pixel shorter side, a 224×224 window, and stride 112; mode #1 reaches only 56.5% mIoU with too small an image size.

5 Conclusion

SCLIP enhances CLIP for dense prediction by introducing correlative self-attention as a task-specific decoder head. Across eight benchmarks, it substantially improves zero-shot semantic segmentation without fine-tuning or additional parameters.

38.2% average zero-shot mIoU across eight benchmarks demonstrates SCLIP’s improvement in dense vision-language inference.The reported average outperforms existing state-of-the-art models by a large margin.
Correlative self-attention functions as a task-specific decoder head for semantic segmentation.The approach enhances CLIP’s dense prediction capabilities through this novel mechanism.
SCLIP requires neither fine-tuning nor additional parameters while outperforming existing baseline methods.The result supports adapting pretrained CLIP models with minimal modifications.

Appendix: Additional Visualization Results

Additional visualizations compare SCLIP with vanilla CLIP, MaskCLIP, and ground truth on PASCAL VOC and COCO-Object. SCLIP generally produces clearer and more detailed masks, while complex categories and annotations still expose errors.

SCLIP yields clear segmentation masks in most cases, unlike the localization failures of vanilla CLIP and noisy segments from MaskCLIP.The comparison covers additional visualizations on PASCAL VOC and COCO-Object.
On PASCAL VOC, SCLIP detects detailed features such as small sheep legs and plant branches.Its plant-branch mask is slightly coarser than ground truth but substantially better than MaskCLIP’s merged background prediction.
On COCO-Object, semantic segmentation with 81 categories remains challenging for zero-shot models.MaskCLIP produces noisy predictions when semantic correlations between patch-level visual tokens are not considered.
Refinement or thresholding cannot simply resolve MaskCLIP’s noise because they may collapse predictions into one or very few categories.Such collapse degrades detailed visual-feature inference.
SCLIP skips a fence in one bird example and labels an SUV as car when ground truth marks it as bus.These examples show residual discrepancies between predictions and annotations.