PaperlayerView exact revision on arXiv

Source-linked AI summary

ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentation

Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, Wayne Zhang

arXiv:2408.04883v1cs.CV

TL;DR

Open-vocabulary segmentation must combine semantic labels with coherent image regions, but CLIP’s localization and VFMs’ semantic understanding are limited. ProxyCLIP uses VFM feature correspondence as proxy attention for CLIP, with adaptive normalization and masking. Across eight datasets it consistently achieves the best performance among the compared methods, and reports a 4.1 average mIoU improvement with CLIP-ViT-B/16.

  • Problem

    Open-vocabulary segmentation requires coherent regions labeled from arbitrary concepts, while CLIP’s localization and VFMs’ semantic understanding are limited.

    Source: §1 · pp. 1–2

  • Method

    ProxyCLIP uses VFM spatial feature correspondence as proxy attention for CLIP, with adaptive normalization and masking.

    Source: §1 · p. 2

  • Results

    Across eight datasets ProxyCLIP consistently achieves the best performance, with a 4.1 average mIoU improvement using CLIP-ViT-B/16 architectures.

    Source: Quantitative results · pp. 9, 11

  • Takeaways & Limitations

    ProxyCLIP is reported to retain CLIP’s zero-shot transfer ability while incorporating VFM spatial correspondence, and to benefit from larger CLIP models.

    Source: §1 · pp. 2–3

Abstract

from arXiv · show

Open-vocabulary semantic segmentation requires models to effectively integrate visual representations with open-vocabulary semantic labels. While Contrastive Language-Image Pre-training (CLIP) models shine in recognizing visual concepts from text, they often struggle with segment coherence due to their limited localization ability. In contrast, Vision Foundation Models (VFMs) excel at acquiring spatially consistent local visual representations, yet they fall short in semantic understanding. This paper introduces ProxyCLIP, an innovative framework designed to harmonize the strengths of both CLIP and VFMs, facilitating enhanced open-vocabulary semantic segmentation. ProxyCLIP leverages the spatial feature correspondence from VFMs as a form of proxy attention to augment CLIP, thereby inheriting the VFMs' robust local consistency and maintaining CLIP's exceptional zero-shot transfer capacity. We propose an adaptive normalization and masking strategy to get the proxy attention from VFMs, allowing for adaptation across different VFMs. Remarkably, as a training-free approach, ProxyCLIP significantly improves the average mean Intersection over Union (mIoU) across eight benchmarks from 40.3 to 44.4, showcasing its exceptional efficacy in bridging the gap between spatial precision and semantic richness for the open-vocabulary segmentation task.

1 Introduction

Open-vocabulary segmentation needs coherent image segments with labels from arbitrary concepts, but CLIP’s semantic strength and VFMs’ spatial consistency come with complementary limitations. ProxyCLIP uses VFM feature correspondence as proxy attention for CLIP, with adaptive normalization and masking, and reports benefits from smaller-patch VFMs and larger CLIP models.

1 Introduction

  • Open-vocabulary segmentation aims to partition images into coherent segments and label them with arbitrary visual concepts, but CLIP’s limited localization and VFMs’ limited semantic understanding leave complementary gaps.CLIP can recognize concepts but struggles to produce coherent segments; VFMs provide spatially consistent representations but may require downstream fine-tuning for semantic understanding.Source: §1 · pp. 1–2
  • CLIP-localization methods and CLIP fine-tuning can improve segmentation, but the former remain constrained by CLIP’s spatial-feature limitations and the latter may sacrifice original-model strengths.The paper also notes that contrastive learning does not explicitly encourage spatially consistent local features.Source: §1 · p. 2
  • ProxyCLIP integrates VFM feature correspondence as proxy attention for CLIP, aiming to retain CLIP’s zero-shot transfer while borrowing VFM spatial consistency.The paper reports that VFM feature correspondence exceeds several types of CLIP attention scores in semantic coherence.Source: §1 · p. 2
  • Adaptive normalization and masking are designed to keep proxy attention effective and consistent across VFMs with diverse visual-representation inductive biases.The paper identifies consistency and separateness of proxy attention as challenges across different VFMs.Source: §1 · p. 2
  • VFM architectures with smaller patch sizes produce clearer boundaries, which the authors associate with superior segmentation results.This is presented as a design finding alongside the framework’s cross-benchmark results.Source: §1 · p. 3
  • ProxyCLIP is reported as a state-of-the-art solution across eight segmentation benchmarks and as the first training-free method to benefit from larger CLIP models, unlike MaskCLIP.The authors also report that larger CLIP backbones improve ProxyCLIP’s performance.Source: §1 · p. 3

2 Related Work

Related work contrasts CLIP’s semantic recognition with the spatial coherence of vision foundation models, then surveys approaches that improve localization or combine their strengths. The paper positions ProxyCLIP as a training-free integration of CLIP and VFMs.

2 Related Work

  • CLIP’s image-level supervision supports zero-shot recognition and retrieval, but can leave local image regions poorly aligned with text for dense prediction.Weakly supervised methods use image-text pairs [53], while other approaches leverage visual grounding datasets.Source: Vision-language pre-training models · p. 3
  • Self-supervised VFM features from DINO and DINOv2 [34] provide spatially coherent representations, while SAM [20] produces precise but class-agnostic masks.Prior work applies VFM features to unsupervised segmentation and detection, among other tasks.Source: Vision foundation models · p. 3
  • CLIP localization approaches revise vanilla query-key attention [2] [25] [45] or exploit last-layer value embeddings; weakly supervised retraining adds group or prototype tokens [68], potentially sacrificing open-vocabulary properties.CLIP-DINOiser and SAM-CLIP integrate DINO/SAM spatial consistency with CLIP through knowledge distillation, aiming to enhance local patch representations for dense prediction.Source: Open-vocabulary semantic segmentation · p. 4

3 Method

ProxyCLIP uses VFM patch correspondences as proxy attention to reorganize CLIP value features, combining local coherence with CLIP’s semantic representations. Adaptive normalization and masking make the proxy attention consistent across different VFMs.

3.1 Preliminaries and Motivation

  • CLIP’s global attention can weaken local patch specificity, motivating the use of VFM features with strong localization to improve its dense representations.The paper aims to preserve CLIP’s zero-shot transfer ability while borrowing VFM feature correspondence.Source: CLIP’s representations · p. 5
  • Semantic coherence is evaluated by classifying whether patch pairs share a label, with patch labels assigned by majority voting over segmentation maps.The analysis collects CLIP attention scores and VFM patch representations on COCO Stuff.Source: Discussion and motivation of ProxyCLIP · p. 5
  • VFM feature correspondences, especially those from DINO and SAM, surpass CLIP attention in semantic coherence, while self-self attention scores exceed vanilla attention scores.The comparison uses average precision; the attention-map examples show DINO and SAM highlighting semantically relevant patches.Source: Figure 1 · p. 5, Discussion and motivation of ProxyCLIP · p. 6, Figure 2 · p. 6

3.2 ProxyCLIP

  • ProxyCLIP pairs frozen CLIP and VFM encoders: CLIP supplies value embeddings, while VFM features form proxy attention that reorganizes them into dense representations.The architecture overview depicts the two encoders and proxy attention module.Source: Architecture · p. 6, Proxy attention module · p. 7, Figure 3 · p. 7
  • Adaptive normalization aligns similarity statistics across VFMs, then masking suppresses negative similarities so patches positively integrate contextually relevant information.The method uses β = 1.2 and γ = 3 by default; the authors report these settings are robust.Source: Figure 4 · p. 7, Normalization and masking · p. 8
  • Interpolating VFM features to CLIP’s patch resolution lets smaller-patch VFMs such as DINO ViT-B/8 produce more detailed segmentation maps.Dense labels are assigned by comparing patch and text representations using cosine similarity.Source: Different resolution · p. 8, Open-vocabulary segmentation · p. 8

4 Experiments

ProxyCLIP is evaluated against open-vocabulary segmentation methods on eight datasets and consistently achieves the best performance. The experiments examine backbone choices, patch size, and adaptive normalization and masking.

4.1 Experiment Setups

4.2 Main results

  • 4.1, 14.0, and 15.3 average mIoU improvement over training-free competitors with CLIP-ViT-B/16, CLIP-ViT-L/14, and OpenCLIP-ViT-H/14, respectively.ProxyCLIP consistently achieves the best performance across eight datasets.Source: Quantitative results · pp. 9, 11, Table 1 · p. 10
  • ProxyCLIP improves average mIoU over CLIP-DINOiser by 2.0, 2.7, and 4.1 with CLIP-ViT-B/16, CLIP-ViT-L/14, and OpenCLIP-ViT-H/14, respectively.The authors also report that all training-based methods fall short of ProxyCLIP.Source: Quantitative results · p. 10

4.3 Ablation study

  • With CLIP-ViT-B/16, DINO, DINOv2, and SAM yield about 41.0 mIoU on average; using OpenCLIP-ViT-H/14 improves each variant by around 2.0 mIoU.The authors attribute SAM's comparable performance to the 336×336 input resolution, which does not benefit its reliance on high-resolution inputs.Source: Impact of the backbones · p. 12, Table 2 · p. 12
  • Adding DINO-ViT-B/8 proxy attention improves local consistency, while a larger OpenCLIP backbone improves classification accuracy.The qualitative comparison reports these differences between configurations (2)→(3) and (3)→(4).Source: Figure 6 · p. 12, Impact of the backbones · p. 13
  • With the same model size, DINO patch size 8 exceeds patch size 16 by about 1.0 mIoU on average and produces sharper boundaries.Figure 7 illustrates sharper boundaries with DINO-ViT-B/8.Source: Impact of the patch size · pp. 13–14, Table 3 · p. 13, Figure 7 · p. 13
  • Combining adaptive normalization and masking raises average mIoU from 25.0 to 42.3; the proxy-attention formula also outperforms SAM attention scores, 40.8 to 40.0 mIoU.DINO and DINOv2 attention scores yield significantly poorer outcomes than the proxy-attention approach.Source: Impact of the patch size · p. 14, Table 4 · p. 14

5 Conclusion

ProxyCLIP uses VFM spatial feature correspondence as proxy attention to enhance CLIP for open-vocabulary segmentation. Experiments indicate that it outperforms other training-free methods and weakly supervised approaches.

5 Conclusion

  • ProxyCLIP is a training-free method that uses VFM spatial feature correspondence as proxy attention to enhance CLIP for open-vocabulary segmentation.The paper also proposes adaptive normalization and masking and the use of small patch sizes.Source: §5 · p. 14

A Implementation for Stable Diffusion.

The stable diffusion implementation extracts dense features from the ninth UNet block at time step 0.

A Implementation for Stable Diffusion.

  • Stable diffusion features come from UNet block 9 at time step 0, producing a 22×22×1280 feature map after resizing the input from 336×336 to 672×672.The resulting feature map has a downsampling factor of 15.3.Source: A Implementation for Stable Diffusion · p. 21

B Hyperparameters.

ProxyCLIP performs consistently across a range of normalization parameters and uses fixed defaults across datasets. Its VFM adds inference computation relative to CLIP-only models.

B Hyperparameters.

  • Across four datasets, ProxyCLIP performs well for shifting factor β from 1.0 to 1.6 and scaling factor γ from 2.0 to 5.0; defaults are β = 1.2 and γ = 3.0.Figure 8 plots mIoU against each factor.Source: Figure 8 · p. 21, B Hyperparameters · p. 21

C Efficiency comparison.

  • Using an additional VFM such as DINO-B/8 consumes more inference resources than CLIP-only models, while a smaller ViT or larger patch size may balance accuracy and efficiency.The comparison uses an RTX 3090, batch size 1, 336×336 input, and fp16.Source: C Efficiency comparison · p. 22, Table 5 · p. 22

D Hard masking.

The hard-masking experiment varies the threshold on COCOStuff; different VFMs may need different thresholds, while adaptive normalization and masking performs consistently across VFMs.

D Hard masking.

  • The COCOStuff experiment evaluates the masking strategy alone while varying threshold α from 0.0 to 0.8.Selecting a suitable threshold is challenging because median attention scores vary across VFMs.Source: D Hard masking · p. 22
  • Adaptive normalization and masking consistently delivers promising results across different VFMs, unlike fixed thresholds that may need to vary by VFM.The experiment isolates masking and varies threshold α from 0.0 to 0.8 on COCOStuff.Source: D Hard masking · p. 22
  • Different VFMs may require different thresholds to achieve good results.This threshold variability motivates testing an adaptive strategy.Source: D Hard masking · p. 22

E Adopting attention embeddings of CLIP in PAM.

The study compares CLIP attention embeddings with VFM features in PAM. CLIP embeddings improve vanilla CLIP, but remain inferior to the VFM-based proxy.

E Adopting attention embeddings of CLIP in PAM.

F Additional quantitative results.

On ADE847 and Pascal Context459, ProxyCLIP performs on par with fully supervised methods and outperforms weakly supervised and training-free methods overall.

F Additional quantitative results.

  • The evaluation covers ADE847 and Pascal Context459, with 847 and 459 classes, respectively.Fully supervised results are taken from the respective papers, weakly supervised results from author-provided checkpoints, and training-free results from the authors’ implementation.Source: F Additional quantitative results · p. 23
  • In the reported comparison, training-free ProxyCLIP significantly outperforms all weakly-supervised and training-free methods, achieving an average of 10.5 mIoU versus 7.8 mIoU for CLIP-DINOiser.Source: F Additional quantitative results · p. 24
  • 11.1 mIoU on ADE847: ProxyCLIP surpasses most fully supervised methods and performs on par with fully supervised methods overall.Fully supervised methods typically train in-domain on COCOStuff with fully annotated labels.Source: F Additional quantitative results · pp. 23–24, Table 8 · p. 24

G Additional qualitative results.

Additional qualitative results show ProxyCLIP segmenting small objects on Cityscapes and producing accurate maps with clearer object boundaries on Pascal VOC and Pascal Context59.

G Additional qualitative results.

Related papers

Loading 2408.04883v1…