PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction

Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, Chen Change Loy

arXiv:2310.01403v2cs.CV

TL;DR

Open-vocabulary dense prediction requires transferring CLIP’s strong image-level alignment to local regions, a setting where CLIP ViTs struggle. CLIPSelf self-distills dense region representations toward corresponding image-crop representations without region-text pairs, achieving state-of-the-art detection and improved semantic and panoptic segmentation. Its validation is limited to the evaluated CLIP-based settings and does not include Swin-based models.

  • Problem

    CLIP ViTs generalize strongly for zero-shot image classification but have inferior dense region-language alignment when transferred from full images to local regions for open-vocabulary dense prediction.

  • Method

    CLIPSelf fine-tunes CLIP ViTs by aligning dense-map region representations with image representations of corresponding crops, using self-distillation without region-text pairs.

  • Results

    CLIPSelf reaches state-of-the-art performance on open-vocabulary detection benchmarks and improves current methods for semantic and panoptic segmentation.

  • Takeaways & Limitations

    Image-level CLIP representations can supervise improved dense region representations without additional labeled region-text pairs.

  • Takeaways & Limitations

    CLIPSelf is not directly validated on Swin-based models because suitable public CLIP-like models and further representation studies were unavailable.

Abstract

from arXiv · show

Open-vocabulary dense prediction tasks including object detection and image segmentation have been advanced by the success of Contrastive Language-Image Pre-training (CLIP). CLIP models, particularly those incorporating vision transformers (ViTs), have exhibited remarkable generalization ability in zero-shot image classification. However, when transferring the vision-language alignment of CLIP from global image representation to local region representation for the open-vocabulary dense prediction tasks, CLIP ViTs suffer from the domain shift from full images to local image regions. In this paper, we embark on an in-depth analysis of the region-language alignment in CLIP models, which is essential for downstream open-vocabulary dense prediction tasks. Subsequently, we propose an approach named CLIPSelf, which adapts the image-level recognition ability of CLIP ViT to local image regions without needing any region-text pairs. CLIPSelf empowers ViTs to distill itself by aligning a region representation extracted from its dense feature map with the image-level representation of the corresponding image crop. With the enhanced CLIP ViTs, we achieve new state-of-the-art performance on open-vocabulary object detection, semantic segmentation, and panoptic segmentation across various benchmarks. Models and code are released at https://github.com/wusize/CLIPSelf.

1 INTRODUCTION

Open-vocabulary dense prediction must transfer CLIP’s image-text alignment to local regions, but CLIP ViTs struggle to produce effective region representations from dense features. CLIPSelf addresses this mismatch through self-distillation without region-text pairs and improves downstream detection and segmentation.

  • Motivation: Open-vocabulary detection and segmentation extend recognition to visual concepts unseen in training datasets, beyond fixed-category dense prediction models.This motivates transferring CLIP’s vision-language alignment from full images to local regions.
  • Motivation: CLIP ViTs perform well on image-level zero-shot classification but struggle when dense feature maps are used for local region recognition.Image-crop classification remains satisfactory, whereas dense ViT features are inferior and noisier than CNN-based features.
  • Motivation: Region-text supervision is costly to annotate, while pseudo-label matching can introduce noisy associations between regions and object nouns.Image-crop representations are presented as a more reliable teacher for CLIP ViT region representations.
  • Method: CLIPSelf aligns dense-map region representations with image representations of corresponding crops, using image patches instead of region-text pairs.Regions can be obtained by partitioning an image into an m × n patch grid.
  • Results: CLIPSelf achieves state-of-the-art open-vocabulary detection on OV-COCO, OV-LVIS, and transfer detection benchmarks, while improving semantic and panoptic segmentation over current methods.The reported improvements include comparisons with Cat-Seg and ODISE.

2 RELATED WORK

Related work applies CLIP and other vision-language models to open-vocabulary dense prediction, but highlights a persistent weakness in CLIP ViT dense region-language alignment. Existing remedies modify ViT features or learn grounding separately, with sub-optimal or indirect use of aligned representations.

  • Open-Vocabulary Dense Prediction: Open-vocabulary dense prediction methods use CLIP features to recognize local visual concepts of arbitrary categories described by text.The related directions primarily include object detection and image segmentation.
  • Vision Transformers in Open-Vocabulary Learning: CLIP ViTs show superior open-vocabulary image recognition but inferior region-language alignment for open-vocabulary dense prediction.This mismatch motivates methods specifically targeting dense features rather than only image-level representations.
  • Vision Transformers in Open-Vocabulary Learning: Prior attempts improve CLIP ViT dense alignment by modifying the output layer or using masked attention, but their results remain sub-optimal.Detection-oriented Swin models instead learn region-language grounding through separate cross-modality heads.

3 METHODOLOGY

CLIPSelf analyzes how CLIP ViTs form image and dense representations, then improves local region representations by self-distilling dense features toward corresponding image-crop representations. The fine-tuned backbone is applied to open-vocabulary detection and segmentation.

  • CLIP Representations: CLIP represents whole images with the updated class embedding from its final residual attention block, whereas dense representations retain spatial image embeddings in a feature map.The dense feature map discards the class embedding and reshapes the remaining embeddings for region extraction.
  • Motivation: Image-crop representations substantially outperform CLIP ViT dense representations for region classification across input sizes, motivating alignment between the two representations.The comparison uses Top1 and Top5 region classification accuracy, and dense-feature performance does not increase with larger input images.
  • CLIPSelf: CLIPSelf fixes the original CLIP ViT as Teacher and fine-tunes a Student to align pooled dense-feature region representations with the Teacher’s corresponding image-crop representations.The Student is initialized from the Teacher, while region representations are pooled from the dense feature map.
  • CLIPSelf: During self-distillation, images are partitioned into randomly sampled m × n patch grids, enabling training across different patch sizes without region-text pairs.The implementation samples m and n from {1, ..., M}, with M set to 6.
  • Application to Open-Vocabulary Dense Prediction: The refined CLIP ViTs replace frozen CLIP backbones in a two-stage detector and provide improved backbones or inference features for semantic and panoptic segmentation.The detector trains detection heads, while Cat-Seg uses the fine-tuned backbone for initialization and ODISE uses the fine-tuned ViT at inference.

4 EXPERIMENTS

Experiments show that CLIPSelf improves CLIP ViT dense representations and transfers those gains to open-vocabulary detection and segmentation. Ablations, qualitative visualizations, and comparisons across training data and architectures support its effectiveness.

  • 4.1 ABLATION STUDY OF CLIPSELF: 72.1% Top1 mAcc from self-distillation substantially exceeds 52.1% from merely increasing the Student input size.The sanity check isolates the contribution of region-wise self-distillation from higher-resolution training alone.
  • 4.2 ENHANCEMENT OF DENSE REPRESENTATION BY CLIPSELF: CLIPSelf improves recognition of both region boxes and panoptic masks, establishing a general dense-representation enhancement for detection and segmentation.Table 2 evaluates Top1 and Top5 mean accuracy for boxes and thing and stuff masks.
  • 4.2 ENHANCEMENT OF DENSE REPRESENTATION BY CLIPSELF: Using region proposals improves foreground classification but reduces performance in recognizing background stuff regions.The proposal-based variant is trained with base-category annotations to preserve the open-vocabulary setting.
  • 4.2 ENHANCEMENT OF DENSE REPRESENTATION BY CLIPSELF: K-Means visualizations show that fine-tuned features group pixels from the same object more accurately and reduce false-positive clusters.Examples include separating the human face and hat into coherent clusters.
  • 4.4 DISCUSSION: CLIPSelf outperforms noisy region-text-pair training, also improves models trained on CC3M, and extends to local-window-attention architectures.These experiments test alternatives to downstream-data self-distillation and applicability beyond plain global-attention ViTs.

5 CONCLUSION

The paper analyzes CLIP ViT dense representations and introduces CLIPSelf to enhance them through self-distillation without region-text pairs. The refined models substantially improve open-vocabulary detection and segmentation, with further validation on local attention and web data.

  • 5 CONCLUSION: CLIPSelf fine-tunes CLIP ViT dense representations through self-distillation without region-text pairs, substantially improving open-vocabulary detection and image segmentation.The conclusion describes the approach as a straightforward solution for enhancing dense representations used in open-vocabulary dense prediction.

A.1 CLIP MODELS’ DENSE REPRESENTATION

CLIP ViT dense features tend to encode global image information rather than local regions, limiting their region recognition despite strong image-level performance. The appendix evaluates this behavior and contrasts extraction methods and attention masking.

  • A.1 CLIP MODELS’ DENSE REPRESENTATION: ViT image representations outperform dense representations on Top1 and Top5 region classification across input sizes, and dense accuracy does not improve with larger inputs.This behavior hinders dense prediction tasks that require high-resolution inputs.
  • A.1 CLIP MODELS’ DENSE REPRESENTATION: CNN-based CLIP dense features are highly effective for region recognition, whereas ViT dense features struggle with local regions.The comparison motivates adapting ViT representations before applying them to open-vocabulary dense prediction.
  • A.1 CLIP MODELS’ DENSE REPRESENTATION: Dense feature locations retrieve their corresponding images well, indicating that CLIP ViT locations tend to encode global image representations.The retrieval experiment uses one positive and 49 negative samples and reports recall at 1, 5, and 10.
  • A.1 CLIP MODELS’ DENSE REPRESENTATION: Masked attention improves stuff-mask recognition but worsens object-box and thing-mask classification, remaining well below CLIPSelf for stuff masks.The appendix compares alternative dense-feature extraction methods in Table A2.

A.2 CLIPSELF

The appendix shows that CLIPSelf transfers dense-feature matching from global images toward local regions. Across ViT variants, this produces stronger recognition of boxes and masks.

  • A.2 CLIPSELF: The Student uses 1024 × 1024 inputs for ViT-B/16 and 896 × 896 for ViT-L/14, while the Teacher uses 224 × 224 and 336 × 336 crops.Training updates all 12 or 24 attention layers of the respective Student ViT.
  • A.2 CLIPSELF: CLIPSelf effectively enhances dense representations across all evaluated ViT variants for recognizing boxes, thing masks, and stuff masks.Table A3 summarizes the improvements across these three region types.
  • A.2 CLIPSELF: CLIPSelf-refined dense features better match corresponding regions than images, revealing transfer from global image representation to local region representation.The result comes from the retrieval experiment summarized in Table A1.

A.3 OPEN-VOCABULARY OBJECT DETECTION

The section details the open-vocabulary detection benchmarks, detector design, and evaluation settings, then reports transfer and ablation findings. CLIPSelf-based detectors achieve strong performance while keeping the ViT backbone fixed.

  • OV-COCO evaluates 48 base and 17 novel categories, using novel-category AP50 as its main metric.
  • OV-LVIS uses 337 rare categories and reports mean mask AP on rare categories as its main metric.
  • Increasing the backbone learning rate improves base-category AP50 but decreases novel-category performance, motivating a fixed ViT backbone.
  • The experiments include comparisons on OV-COCO, OV-LVIS, OpenAI-model variants, and detailed system-level tables.
  • The LVIS-trained detector consistently outperforms previous methods when transferred to PASCAL VOC, COCO, and Objects365.

A.4 USING REGION-TEXT PAIRS

This section contrasts CLIPSelf with region-text-pair training approaches and positions the method as a way to adapt CLIP ViTs without requiring paired region descriptions. It also emphasizes applicability beyond plain ViTs.

  • RegionCLIP-style training uses pseudo-labelled region-text pairs formed from COCO captions, object nouns, and region proposals.
  • CLIPSelf fine-tunes CLIP ViTs through self-distillation rather than relying on paired region-text data.
  • The approach is presented as effective for open-vocabulary dense prediction and applicable beyond plain ViT architectures.

C LIMITATION

The paper identifies dependence on pretrained CLIP alignment and limited validation on Swin-based or other detection-oriented foundation models as important scope boundaries.

  • CLIPSelf performance is strongly influenced by the visual-language alignment of the original pretrained CLIP models.
  • The method is not directly validated on Swin-based models because public CLIP-like vision-language models with Swin backbones are unavailable.
  • Potential use with GLIP and Grounding DINO remains unvalidated because of resource limitations and insufficient study of their vision-language representations.

D VISUALIZATION

The visualization section presents qualitative detection and segmentation examples. Detection visualizations distinguish novel from base categories, while segmentation examples come from ADE20K evaluations.

  • Detection visualizations mark novel categories with red boxes and base categories with blue boxes.
  • The segmentation visualizations show open-vocabulary semantic segmentation results evaluated on ADE20K.
Loading 2310.01403v2…