Source-linked AI summary
How Much Can CLIP Benefit Vision-and-Language Tasks?
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, Kurt Keutzer
TL;DR
Existing V&L models rely on visual encoders trained with relatively limited manual annotations, motivating investigation of CLIP’s scalable language-supervised representations. The paper integrates CLIP as the visual encoder through direct task-specific fine-tuning and V&L pre-training, achieving competitive or better results across diverse tasks and state-of-the-art results on several benchmarks.
Problem
V&L visual encoders often rely on costly manually annotated data, creating a need for scalable representations that generalize beyond fixed visual labels.
Method
The paper evaluates CLIP as the visual encoder in direct task-specific fine-tuning and in V&L pre-training followed by downstream transfer.
Results
Competitive or better performance across diverse V&L tasks includes new state-of-the-art results of 76.70% on VQA test-std and 80.20% on SNLI-VE test.
Takeaways & Limitations
CLIP-ViL supports exploring stronger, web-trained visual encoders as alternatives to conventional annotated-data visual encoders in V&L models.
Takeaways & Limitations
On GQA, the model under-performs VinVL, potentially because GQA’s object-bounding-box construction favors region-based models trained on object data.
Abstract
from arXiv · showhide
Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pretraining usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks. We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown. We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V&L Navigation tasks. We release our code at https://github.com/clip-vil/CLIP-ViL.
1 Introduction
The paper studies whether CLIP can address visual-representation bottlenecks in V&L models by serving as their visual encoder across two integration scenarios. CLIP-ViL improves results across direct fine-tuning and V&L pre-training, including new state-of-the-art results on several tasks.
- Motivation: V&L models depend on visual encoders, but manually annotated visual data is difficult to scale and can constrain representations.The paper identifies visual representation as a performance bottleneck and motivates larger, more diverse supervision.
- Motivation: CLIP learns visual concepts from 400M noisy Internet image-text pairs with little human annotation, but direct zero-shot use is difficult for complex V&L reasoning.The paper therefore integrates CLIP’s visual encoder with existing V&L models rather than using CLIP directly as a zero-shot V&L system.
- Approach: The study evaluates CLIP in direct task-specific fine-tuning and in V&L pre-training followed by downstream transfer.These are denoted CLIP-ViL and CLIP-ViLp, respectively.
- Results: 1.4% accuracy on VQA v2.0, 6.5 CIDEr on COCO Captioning, and 4.0% success rate on Room-to-Room navigation show sizable gains over strong baselines.These results come from direct task-specific fine-tuning across three V&L tasks.
- Results: 76.70% on VQA test-std and 80.20% on SNLI-VE test establish new state-of-the-art results for CLIP-ViLp with V&L pre-training.CLIP-ViLp also outperforms models using BUTD ResNet101 and, with CLIP-Res50x4, surpasses VinVL-ResNeXt152.
2 Background and Motivation
V&L models combine visual and language processing, typically using visual encoders pre-trained on annotated classification or detection data. The paper motivates CLIP as a scalable alternative whose language supervision is less bounded by predefined visual labels.
- V&L Models: V&L models commonly contain visual and text encoders plus a cross-modal interaction module to ground language in visual observations.Prominent tasks include visual question answering, image captioning, navigation, and image-text retrieval.
- Training Pipeline: Training commonly proceeds through visual encoder pre-training, optional V&L pre-training, and task-specific fine-tuning.V&L pre-training uses paired image-caption data with reconstructive and image-text matching objectives.
- Visual Encoders: Region-based encoders use detection data, while grid-based encoders use image classification or detection data.These encoder families and their pre-training sources are contrasted in Figure 2.
- CLIP: CLIP requires aligned text rather than object detection or image classification annotations for visual representation learning.Its visual encoder is integrated into V&L models because CLIP’s strong vision zero-shot capability does not directly transfer to some complex V&L tasks.
- Motivation: Manually labeled datasets are expensive and hard to scale, and their fixed concepts provide supervision for only a limited set of predetermined visual categories.This motivates visual encoders trained on broader data sources with generalization to unseen objects and concepts.
3 CLIP-ViL
CLIP-ViL replaces conventional visual encoders with CLIP in task-specific fine-tuning across VQA, image captioning, and vision-and-language navigation. CLIP-based models improve over ImageNet-trained alternatives across these tasks, with larger CLIP models generally performing better, while CLIP-ViT-B can underperform in settings requiring localized feature maps.
- 3 CLIP-ViL: CLIP-ViL directly plugs CLIP visual encoders into task-specific models for VQA, image captioning, and vision-and-language navigation.The VQA experiments use Pythia and MCAN; captioning uses a Transformer; navigation uses an attentive neural agent.
- 3.1 Visual Question Answering: 4.01% and 4.26% gains over ImageNet-Res50 bring CLIP-Res50 to 65.55% with Pythia and 71.49% with MCAN on VQA Test-dev.CLIP-Res50x4 reaches 74.01% on Test-dev and 74.17% on Test-std with MCAN.
- 3.1 Visual Question Answering: Visual Genome detection pretraining boosts ImageNet encoders but reduces CLIP-Res50 performance by 5.54% with PythiaVG and 4.08% with MCANVG.The authors suggest that applying ImageNet-oriented Visual Genome fine-tuning to CLIP may hurt because CLIP uses different data and training.
- 3.2 Image Captioning: +9.1 / +1.5 in CIDEr / SPICE and +9.2 / +1.5 in CIDEr / SPICE are reported for CLIP-Res50 and CLIP-Res101 over ImageNet alternatives.CLIP-Res50x4 performs best among CLIP-ViL models, though it remains below models with interactive image-text pretraining.
- 3.3 Vision-and-Language Navigation: 8% higher SR and 6% higher SPL than EnvDrop are achieved on R2R, while RxR results include the best mono-lingual SR and nDTW.CLIP-ViL is competitive with vision-and-language navigation-specific pretrained models and improves prior nDTW results by 4.3% on RxR.
- 3.3 Vision-and-Language Navigation: CLIP-Res50 improves over ImageNet-Res152 on navigation, and performance scales with CLIP-Res101 and CLIP-Res50x4 on both R2R and RxR.CLIP-ViT-B has similar navigation results to CLIP-Res50, possibly because navigation uses a [CLS] feature rather than a feature map.
4 Vision-and-Language Pre-training
CLIP-ViLp replaces conventional region-based visual representations with CLIP and combines CLIP visual features with vision-and-language pre-training. Across three benchmarks, it outperforms most BUTD-based models and achieves state-of-the-art results on VQA and SNLI-VE.
- 4.1 Model and Pre-training: CLIP-ViLp combines a CLIP visual backbone with vision-and-language pre-training on paired image-text data.The model is subsequently fine-tuned on downstream V&L tasks.
- 4.1 Model and Pre-training: 9.18M image-text pairs are used for pre-training, drawn from COCO Captions, Visual Genome Captions, VQA, GQA, and VG-QA.The setup uses CLIP-Res50 and CLIP-Res50x4 variants and excludes test data from pre-training.
- 4.2 Experiments: CLIP-Res50-based models significantly outperform most BUTD-Res101-based models, including LXMERT trained with the same data and epoch budget.CLIP-ViLp with CLIP-Res50 exceeds LXMERT on VQA by 2.59.
- 4.2 Experiments: CLIP-Res50x4 surpasses VinVL on VQA while using significantly fewer V&L pre-training steps, but underperforms VinVL on GQA.The passage attributes GQA’s advantage to its construction from object bounding-box data.
- 4.2 Experiments: CLIP-Res50 significantly outperforms Pixel-BERT with ImageNet-Res50, indicating an advantage for CLIP initialization over ImageNet initialization.
5 Analysis
The analysis finds that CLIP performs poorly as a zero-shot VQA model but benefits substantially from interactive V&L pre-training and backbone unfreezing. It also identifies localization as a weakness of CLIP-ViT-B relative to CLIP-Res50.
- Zero-Shot Performance of CLIP in VQA: CLIP variants perform near chance on zero-shot VQA, and prompt engineering provides only a small improvement.Performance is worse for harder “other” questions than for “yes/no” questions.
- Zero-Shot Performance of CLIP in VQA: Zero-shot results suggest that VQA requires a deep interactive model together with additional pre-training or fine-tuning.
- Unfreezing the Visual Backbone: +9.25 improvement for CLIP-Res50 versus +5.72 for BUTD-Res101 after V&L pre-training, with CLIP-Res50 then significantly outperforming BUTD-Res101.Without pre-training, BUTD-Res101 performs better than CLIP-Res50.
- Unfreezing the Visual Backbone: Unfreezing the visual backbone during V&L pre-training allows CLIP-Res50 to adapt to the pre-training task.The comparison concerns CLIP models with an unfrozen backbone and BUTD with a frozen backbone.
- Low Detection Performance of CLIP-ViT-B: 0.03 Average Precision for CLIP-ViT-B detection versus 3.14 for its reproduced ImageNet-Res50 alternative.The analysis hypothesizes that CLIP-ViT-B’s degradation reflects limited visual localization inside its ViT feature map.
- Qualitative Comparison of CLIP Variants: Grad-CAM shows CLIP-Res50 localizes the woman’s shirt better than CLIP-ViT-B for the question about shirt color.
6 Conclusions
The paper evaluates CLIP as a visual encoder in task-specific fine-tuning and in V&L pre-training followed by downstream fine-tuning. Across diverse V&L tasks, both CLIP-ViL and CLIP-ViLp achieve competitive or better performance than strong baselines.
- 6 Conclusions: The paper studies CLIP as a visual encoder across different V&L models and tasks.
- 6 Conclusions: The two approaches are direct CLIP plugging during task-specific fine-tuning and CLIP integration with V&L pre-training before downstream fine-tuning.
- 6 Conclusions: CLIP-ViL and CLIP-ViLp achieve competitive or better performance than strong baselines across a variety of V&L tasks.The paper also reports analyses intended to explain observed phenomena and guide future V&L research.
A.1 Visual Question Answering
The VQA models encode questions with recurrent networks and fuse language and visual representations with multimodal attention or bilinear pooling. Their implementation follows established preprocessing and model configurations while replacing the visual backbone with CLIP.
- Model Architecture: Pythia uses an attention-based GRU and multimodal factorized bilinear pooling, while MCAN uses an LSTM and modular co-attention network.Both models apply an output classifier to the fused representation for answer prediction.
- Implementation Details: Input images are resized with a maximum shorter side of 600 pixels and longest side of 1000 while preserving aspect ratio.
- Implementation Details: The detector backbone is replaced with the CLIP visual module using Faster R-CNN in Detectron for Visual Genome training.
- Implementation Details: VQA training uses open-source hyperparameters for the large MCAN and base Pythia configurations.
A.2 Image Captioning
The captioning setup uses long-epoch training and self-critical sequence training, sampling multiple captions per image while optimizing CIDEr with REINFORCE.
- Training follows the publicly available implementation's “long epoch” hyperparameter.
- During self-critical training, the model samples 5 captions for each image.
- The SCST objective optimizes CIDEr using the REINFORCE algorithm.
A.3 Vision-and-Language Navigation
The navigation agent attends jointly to panoramic visual views and language instructions to predict actions, using pre-trained visual encoders and evaluation on R2R and RxR.
- Model: The navigation model uses an attentive neural agent based on prior work.
- Model: At each time step, an attentive agent combines panoramic views and the instruction to choose an action.
- Model: The visual encoder processes panoramic views, while a language LSTM represents the instruction before action prediction.
- Datasets: Evaluation covers Room-to-Room and Room-across-Room, with R2R training using 61 environments and 14,025 navigation instructions.
- Results: Table 9 compares grid, CLIP, and ImageNet-trained features on R2R using success rate and length-weighted success rate.
A.4 Details of CLIP-ViLp
CLIP-ViLp is fine-tuned on three V&L tasks after pre-training with CLIP-initialized components, while qualitative visualizations compare localization across CLIP backbones.
- Pre-training: The model is pre-trained with a batch size of 512 from BERTBASE and CLIP-initialized visual components.
- Optimization: The Transformer and ResNet components are optimized separately with AdamW and SGD, respectively.
- Fine-tuning: CLIP-ViLp is fine-tuned on VQA v2.0, SNLI-VE, and GQA.
- Fine-tuning: VQA uses binary cross-entropy for 5 epochs, while SNLI-VE uses negative log-likelihood for 2 epochs.
- Qualitative analysis: Grad-CAM visualizations suggest that CLIP-Res50 localizes question-relevant sentences better than CLIP-ViT-B.