Source-linked AI summary
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, Jiwen Lu
TL;DR
Dense prediction has seen limited transfer of knowledge from image-text pre-training because CLIP’s instance-level image-text objective differs from pixel-level prediction. DenseCLIP bridges this gap with pixel-text matching and context-aware language prompting, and reports superior performance across semantic segmentation, object detection, and instance segmentation. The framework is model-agnostic and also improves dense prediction with non-CLIP visual backbones.
Problem
Transferring CLIP knowledge from image-text pairs to complex dense prediction tasks remains barely explored, despite dense prediction requiring pixel-level outputs unlike CLIP’s instance-level representations.
Method
DenseCLIP converts image-text matching into pixel-text matching, uses score maps to guide dense prediction, and prompts the language model with visual context.
Results
DenseCLIP improves semantic segmentation, object detection, and instance segmentation, including +4.9%, +4.7% and +2.3% mIoU over ImageNet-pre-trained models for ResNet-50, ResNet101 and ViT-B.
Takeaways & Limitations
The model-agnostic framework transfers vision-language knowledge to arbitrary dense prediction systems and visual backbones, including ImageNet-pre-trained models.
Takeaways & Limitations
Improvements on detection are not as significant as those on segmentation, possibly because CLIP pre-training lacks locality constraints and provides less dense supervision for object-centered tasks.
Abstract
from arXiv · showhide
Recent progress has shown that large-scale pre-training using contrastive image-text pairs can be a promising alternative for high-quality visual representation learning from natural language supervision. Benefiting from a broader source of supervision, this new paradigm exhibits impressive transferability to downstream classification tasks and datasets. However, the problem of transferring the knowledge learned from image-text pairs to more complex dense prediction tasks has barely been visited. In this work, we present a new framework for dense prediction by implicitly and explicitly leveraging the pre-trained knowledge from CLIP. Specifically, we convert the original image-text matching problem in CLIP to a pixel-text matching problem and use the pixel-text score maps to guide the learning of dense prediction models. By further using the contextual information from the image to prompt the language model, we are able to facilitate our model to better exploit the pre-trained knowledge. Our method is model-agnostic, which can be applied to arbitrary dense prediction systems and various pre-trained visual backbones including both CLIP models and ImageNet pre-trained models. Extensive experiments demonstrate the superior performance of our methods on semantic segmentation, object detection, and instance segmentation tasks. Code is available at https://github.com/raoyongming/DenseCLIP
1. Introduction
DenseCLIP addresses the largely unexplored transfer of image-text contrastive knowledge to dense prediction by introducing pixel-text matching and context-aware language prompting. It improves dense prediction across backbones and tasks, including semantic segmentation, object detection, and instance segmentation.
- Motivation: DenseCLIP targets the largely unexplored transfer of image-text contrastive knowledge from CLIP to complex dense prediction tasks.Classification-oriented CLIP transfer methods remain close to image-text matching, whereas dense prediction requires pixel-level outputs.
- Method: The framework converts CLIP’s image-text matching into pixel-text matching and uses pixel-text score maps to guide dense prediction models.This provides an explicit mechanism for transferring pre-trained vision-language knowledge to dense prediction.
- Method: Contextual information from images is used to prompt the pre-trained language model, helping the framework exploit CLIP knowledge more effectively.The approach uses image context to modify the language-model input during dense prediction.
- Results: +4.9%, +4.7% and +2.3% mIoU improvements are reported over ImageNet-pre-trained models for ResNet-50, ResNet101 and ViT-B, respectively.These improvements are obtained with semantic FPN on ADE20K.
- Results: 46.5% mIoU is achieved on ADE20K by a ResNet-101 model with DenseCLIP and a lightweight semantic FPN decoder.The reported result uses only 1/3 computation relative to the compared state-of-the-art solutions.
- Scope: DenseCLIP is presented as a plug-and-play, model-agnostic framework applicable to arbitrary dense prediction systems and visual backbones with slight computation overhead.The paper reports improvements with ImageNet-pre-trained ResNets and Swin Transformers, as well as CLIP-pre-trained models.
2. Related Work
Prior work established pre-training and fine-tuning for downstream vision tasks, while vision-language pre-training expanded supervision through image-text interactions. DenseCLIP extends this line to dense prediction by using language information to guide fine-tuning.
- Pre-training and fine-tuning: The pre-training and fine-tuning paradigm trains models on large-scale datasets before adapting them to diverse downstream tasks.Prior examples include supervised or self-supervised pre-training followed by downstream fine-tuning.
- Pre-training and fine-tuning: Prompt-based methods reformulate downstream tasks to simulate tasks solved during original pre-training, following developments in NLP.This work is presented as inspired by that prompt paradigm.
- Vision-language models: Vision-language research spans tasks such as text-to-image retrieval, image captioning, visual question answering, and referring segmentation.Vision-language pre-training has received growing attention across these areas.
- Dense prediction: Dense prediction requires finer-grained pixel- or region-level representations than conventional instance-level classification.Semantic segmentation, instance segmentation, and object detection are cited as examples.
- Dense prediction: DenseCLIP introduces a fine-tuning strategy that leverages large-scale vision-language pre-training and uses language information to guide dense prediction.It is positioned as complementary to prior supervised and self-supervised dense-prediction pre-training approaches.
3. Approach
DenseCLIP adapts CLIP to dense prediction by extracting language-compatible spatial features, matching pixels with class text embeddings, and injecting contextual visual prompts. The framework uses these score maps for auxiliary supervision and supports different prompting strategies, dense prediction systems, and visual backbones.
- CLIP preliminaries: CLIP aligns visual and language embedding spaces using separate image and text encoders trained with contrastive learning on large-scale image-text pairs.The pre-trained model commonly transfers to classification through text prompts and image-text similarity.
- Language-guided dense prediction: DenseCLIP repurposes CLIP for dense prediction by using a spatial feature map from the image encoder as a language-compatible representation.The feature map retains spatial information and is aligned with language features after the encoder's attention-pooling pathway.
- Language-guided dense prediction: Pixel-text score maps are computed from normalized spatial image features and class text features, then used both as auxiliary segmentation outputs and as language-prior inputs to the decoder.The modified feature map concatenates the original feature map with the score maps, enabling use in segmentation or detection pipelines.
- Context-aware prompting: Context-aware prompting incorporates visual context into language representations through either pre-model prompting or post-model refinement of class embeddings.Pre-model prompting changes the text encoder inputs with Transformer-derived visual contexts, whereas post-model prompting refines text features after encoding.
- Context-aware prompting: The authors prefer post-model prompting because it avoids image-dependent text-encoder passes at inference and performs better empirically.Extracted text features can be stored after training to reduce inference overhead.
- Instantiations: The auxiliary score-map objective helps feature maps recover locality faster, benefiting both segmentation and detection, while DenseCLIP can also guide non-CLIP backbones.The framework is described as applicable to ImageNet-pretrained and self-supervised visual backbones as well as CLIP models.
4. Experiments
DenseCLIP improves dense prediction across semantic segmentation, object detection, and instance segmentation, while extending to non-CLIP backbones with modest computational overhead. Experiments also show that context-aware prompting and pixel-text guidance improve transfer from vision-language pre-training.
- Semantic segmentation: DenseCLIP improves ADE20K semantic segmentation over ImageNet-pretrained baselines and vanilla CLIP fine-tuning across ResNet-50, ResNet-101, and ViT-B backbones.The reported single-scale mIoU gains are +4.9%, +4.7%, and +2.3% over ImageNet-pretrained models, and +3.9%, +2.4%, and +1.2% over vanilla CLIP fine-tuning.
- Semantic segmentation: Post-model vision-to-language prompting performs better and uses fewer computational resources than pre-model prompting in the ResNet-50 ablation.Language-domain prompting alone provides a +2.5% mIoU boost, while both visual-context prompting strategies improve performance.
- Object detection and instance segmentation: DenseCLIP improves RetinaNet object detection over ImageNet1K pre-training by +1.5% and +2.6% AP on ResNet-50 and ResNet-101.It also improves vanilla CLIP fine-tuning by +0.9% and +0.6% AP on the same backbones.
- Object detection and instance segmentation: DenseCLIP consistently improves Mask R-CNN object detection and instance segmentation within an affordable computational budget.For instance segmentation, mask AP gains over ImageNet1K pre-training are +2.9% and +2.5%, while gains over vanilla CLIP fine-tuning are +0.8% and +0.7%.
- Any visual backbone: DenseCLIP also improves non-CLIP backbones, yielding ∼2.5% single-scale mIoU gains for ResNet-50/101 and ∼0.8% for Swin-T/S.The text encoder can be removed after training, providing a low-cost way to guide arbitrary pre-trained 2D backbones, although these results lag behind CLIP image encoders.
5. Conclusion and Discussion
DenseCLIP transfers vision-language knowledge to dense prediction through a model-agnostic, context-aware framework. It supports multiple dense tasks and backbones, while improving segmentation substantially but showing less significant gains for detection.
- Conclusion: DenseCLIP is a model-agnostic framework that transfers pre-trained vision-language knowledge to dense prediction tasks.It uses context-aware prompting and applies across different dense prediction systems and backbones.
- Conclusion: DenseCLIP applies to semantic segmentation, object detection, and instance segmentation.
- Limitations & societal impact: Detection improvements are not as significant as segmentation improvements, which the authors associate with CLIP's lack of locality during pre-training.The authors suggest dense supervision during pre-training or better recovery of locality as possible directions.
Appendix: More Analysis
The appendix analyzes optimization choices for DenseCLIP, finding that encoder learning-rate settings and learnable textual contexts materially affect training and performance.
- Effects of learning rate multipliers: Freezing the text encoder and using a 0.1 learning-rate multiplier for the image encoder yields the best performance.The final models use this configuration.
- Effects of optimization of the textual contexts: Learnable textual contexts bring notable improvements over prompts constructed directly from human-defined templates.Context optimization adds computation during training because gradients must be computed for the text encoder.
- Effects of γ: A learnable γ initialized with small values can improve performance.