PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment

Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, Furu Wei

arXiv:2203.07190v1cs.CVcs.CL

TL;DR

The paper asks whether CLIP’s language-supervised representations can transfer beyond standard vision tasks to vision-language understanding and few-shot learning. It evaluates zero-shot VQA and cross-modality visual entailment, then adds prompt generation and parameter-efficient fine-tuning. Experiments support CLIP as a vision-language few-shot learner without additional pre-training.

  • Problem

    The paper investigates whether CLIP’s zero-shot capabilities can transfer to vision-language understanding tasks and provide few-shot abilities beyond its role as a visual encoder.

  • Method

    The paper evaluates prompt-based zero-shot VQA, caption-trained/image-tested visual entailment, and BiNor fine-tuning that updates bias and normalization parameters.

  • Results

    CLIP demonstrates zero-shot cross-modality transfer and competitive zero/few-shot performance across VQA and visual entailment, supporting its use as a vision-language few-shot learner.

  • Takeaways & Limitations

    Language supervision and parameter-efficient adaptation allow CLIP models to function as vision-language few-shot learners without additional pre-training.

  • Takeaways & Limitations

    CLIP struggles with fine-grained object counting and subtle semantic distinctions, limitations that language knowledge may not resolve.

Abstract

from arXiv · show

CLIP has shown a remarkable zero-shot capability on a wide range of vision tasks. Previously, CLIP is only regarded as a powerful visual encoder. However, after being pre-trained by language supervision from a large amount of image-caption pairs, CLIP itself should also have acquired some few-shot abilities for vision-language tasks. In this work, we empirically show that CLIP can be a strong vision-language few-shot learner by leveraging the power of language. We first evaluate CLIP's zero-shot performance on a typical visual question answering task and demonstrate a zero-shot cross-modality transfer capability of CLIP on the visual entailment task. Then we propose a parameter-efficient fine-tuning strategy to boost the few-shot performance on the vqa task. We achieve competitive zero/few-shot results on the visual question answering and visual entailment tasks without introducing any additional pre-training procedure.

1 Introduction

This work studies whether CLIP’s language-supervised zero-shot abilities transfer to vision-language understanding and support few-shot learning. It evaluates VQA and visual entailment, introduces prompt generation and parameter-efficient fine-tuning, and reports that CLIP can act as a vision-language few-shot learner.

  • Motivation: Vision-language understanding requires systems to jointly capture visual and natural-language semantics, but existing approaches rely extensively on expensive human-annotated data.Such data include object-detection datasets and aligned image-text pairs.
  • Research Focus: The paper evaluates CLIP on visual question answering and visual entailment to study transfer from image-text pretraining to vision-language tasks.VQA requires answering image-based questions, while visual entailment predicts the relation between an image and a sentence.
  • Method: A two-step prompt generation strategy converts questions into masked templates and filters candidate answers with T5 to narrow the gap between CLIP pretraining and VQA.The resulting prompts pair candidate answers with images for CLIP image-text alignment scoring.
  • Method: Zero-shot cross-modality transfer is tested by training a classification layer on captions and evaluating with image-hypothesis pairs in visual entailment.This setup probes alignment between CLIP’s language and vision representations.
  • Results: BiNor fine-tunes only bias and normalization parameters, yielding better few-shot results than Frozen and supporting CLIP as a vision-language few-shot learner.The method is designed to make better use of limited examples.
  • Contributions: The paper’s contributions include studying CLIP transfer into vision-language tasks, demonstrating cross-modality transfer, and proposing parameter-efficient fine-tuning for few-shot VQA.These contributions are presented without adding a new pre-training procedure.

2 Preliminaries

CLIP independently encodes images and texts, aligns their representations with a dot-product score, and learns this alignment through contrastive pretraining. The paper applies this foundation to VQA classification and visual entailment prediction.

  • CLIP: CLIP uses a visual encoder and a text encoder to encode images and texts independently, then computes T(text) · V(image) as their alignment score.The encoders may be implemented with architectures such as ResNet or ViT for vision and a Transformer for text.
  • CLIP: Contrastive loss trains CLIP to distinguish aligned image-text pairs from randomly combined pairs using 400 million web-crawled image-text pairs.This language supervision enables image classification without task-specific optimization.
  • Visual Question Answering: VQA is formulated on VQAv2 as classification over 3,129 frequent answers covering questions such as counting and color identification.The answer types are yes/no, number, and other.
  • Visual Entailment: Visual entailment predicts entailment, neutral, or contradiction between an image-based premise and a hypothesis using the SNLI-VE dataset.SNLI-VE replaces textual premises from SNLI with images from Flickr30k.

3 Zero-shot VQA

The zero-shot VQA method narrows the mismatch between CLIP’s language-pretraining format and question answering by generating masked templates and filtering candidate answers before CLIP scores image-prompt pairs.

  • Directly applying CLIP to VQAv2 with a question-answer prompt produces nearly random-chance zero-shot performance.
  • Step I: Automatic Template Generation: TAP-C converts questions into masked statement templates using T5 demonstrations or dependency parsing.The parsing route compensates for out-of-coverage cases from T5 conversion.
  • Step II: Answer Filtering: A language model filters unlikely answers from the answer vocabulary, and T5 infills answer spans without assuming their length or position.
  • 3.2 TAP-C Method for VQA: TAP-C ensembles demonstration and parsing templates by preferring T5-generated templates unless generation confidence is low.
  • 3.2 TAP-C Method for VQA: TAP-C replaces each template mask with filtered answers and uses CLIP’s visual and text encoders to obtain a zero-shot VQA prediction.

4 Zero-shot Cross-modality Transfer

The paper tests zero-shot cross-modality transfer by training a classifier on text-only premise-hypothesis pairs and evaluating it on image-text pairs, keeping the CLIP encoders fixed.

  • The visual entailment experiment examines whether CLIP representations transfer from language-only training to image-text evaluation.
  • An MLP classifier is trained on fused CLIP text representations of premise-hypothesis pairs.Fusion concatenates the two vectors, their sum, difference, and elementwise product.
  • The CLIP text encoder remains frozen during training, and the trained MLP is reused when the premise is encoded by the visual encoder at inference.

5 Few-shot Learning for VQA

The few-shot VQA setup defines task structure around question and answer types, then adapts only a small subset of CLIP parameters while using demonstrations to improve answer filtering.

  • 5.1 Setup of Few-shot VQA: The 195 ways are defined as 65 question types multiplied by 3 answer types rather than the 3,129-answer vocabulary.
  • 5.1 Setup of Few-shot VQA: Few-shot VQA samples a 195-way K-shot subset, with support examples for training and query examples for performance evaluation.Each shot is an image, question, and answer; each epoch selects C ways and splits their examples into support and query sets.
  • 5.2 Parameter-efficient Fine-tuning: The fine-tuning strategy optimizes about 0.3 million bias and normalization parameters out of more than 100 million CLIP parameters.It applies to BatchNorm in ResNet and LayerNorm in Transformer components.
  • 5.2 Parameter-efficient Fine-tuning: The selected Bias and Normalization parameters are optimized with standard cross-entropy over image-prompt dot products.
  • 5.2 Parameter-efficient Fine-tuning: In-context demonstrations sampled from the same question type can improve TAP-C’s answer-filtering process by producing a demonstration-conditioned filtered vocabulary.

6 Experiments

Experiments evaluate CLIP on zero-shot VQA, zero-shot cross-modality transfer, few-shot VQA, and ablations of prompt generation and fine-tuning. Results show that question-aware prompting, limited-parameter adaptation, and cross-modal representations support competitive vision-language performance, while fine-grained counting and subtle semantic distinctions remain difficult.

  • Zero-shot VQA: TAP-C achieves strong zero-shot VQA performance by converting questions into templates and filtering candidate answers with a pretrained language model.Question-irrelevant prompting performs barely better than random guessing, whereas TAP-C is competitive with Frozen.
  • Zero-shot cross-modality transfer: CLIP transfers representations across modalities in both directions on SNLI-VE, with image masking reducing results to approximately random guessing.Language-to-vision transfer trains on caption-text premises and tests on image-text pairs; vision-to-language transfer reverses the setup.
  • Few-shot VQA: Few-shot VQA improves over zero-shot results across different shot counts, confirming that CLIP models can learn from limited examples.Performance gains are especially observed for the other category, while number-category gains slow as more examples are added.
  • Template ablations: The TAP-C template ensemble outperforms either dependency-parsing or T5-demonstration template alone because the two templates affect question types differently.The ablation indicates complementary strengths across question categories.
  • TAP-C ablations: Removing answer filtering lowers zero- and few-shot performance by about 20%, while removing template generation nearly halves results.Question-aware templates remain important in both zero-shot and few-shot settings.
  • Fine-tuning comparisons: BiNor and BitFit significantly outperform full fine-tuning in few-shot VQA because updating millions of parameters can overfit limited examples.BiNor additionally updates normalization parameters and increasingly outperforms BitFit as the number of shots grows.
  • Limitations: CLIP models struggle with fine-grained object counting and subtle semantic distinctions that language-based prompt improvements cannot reliably resolve.The paper gives background-versus-foreground identification as an example of the latter limitation and suggests stronger text encoders as future work.

7 Related Work

Prior work established vision-language models and prompting as foundations for zero- and few-shot vision-language learning. CLIP-based approaches extend this direction by leveraging aligned image-text representations and language-model prompting.

  • Vision-language models use aligned image-caption data to support strong zero-shot generalization in image classification and cross-modality retrieval.
  • A generative vision-language few-shot model can prompt a frozen language model with a vision prefix.
  • Language-model prompting mines knowledge from pretrained language models in zero-shot or few-shot settings.

8 Conclusions

The paper studies transferring CLIP models into vision-language understanding tasks through language prompts and parameter-efficient fine-tuning. Experiments on VQAv2 and SNLI-VE support CLIP as a vision-language few-shot learner, including zero-shot cross-modality transfer on visual entailment.

  • The study transfers CLIP models into vision-language understanding tasks using language prompts and parameter-efficient fine-tuning.
  • Experiments on VQAv2 and SNLI-VE confirm that CLIP models can be good vision-language few-shot learners.
  • CLIP models demonstrate zero-shot cross-modality transfer on the visual entailment task.

A Datasets Statistics

The dataset-statistics table summarizes the two datasets by example count, distinct-image count, and vocabulary size.

  • The table reports the number of examples, distinct images, and vocabulary size for two datasets.

B Details of Implementation

The implementation uses several CLIP variants and a pretrained T5 model, with reported hyperparameters for CLIP fine-tuning, entailment classification, and T5 generation. It also reports model architectures and parameter-count statistics, including the small share of BiNor parameters.

  • The experiments use CLIP variants and a publicly available T5large checkpoint pretrained on C4.
  • CLIP text encoders use transformers, while visual encoders use either ViT or ResNet architectures.
  • CLIP ViT-B/16 uses 12-layer, 512-hidden transformer encoders for both text and vision.
  • CLIP RN101 pairs a 12-layer, 512-hidden text transformer with a ResNet101 visual encoder of hidden size 512.
  • CLIP RN50x16 pairs a 12-layer text transformer with a ResNet50x16 visual encoder, both with hidden size 768.
  • The implementation reports shared hyperparameters for fine-tuning ResNet and ViT CLIP models.
  • The visual entailment MLP classifier uses grid-searched learning rate, batch size, and dropout configurations.
  • T5 hyperparameters support conditional generation for template generation and answer filtering.

C Few-shot Training Procedure

The few-shot procedure freezes most CLIP encoder parameters, samples C-way K-shot examples, and optimizes normalized image-text similarities with cross-entropy. It also uses demonstrations to generate masked question templates and answer prompts for different question and answer types.

  • Few-shot Training Procedure: The procedure freezes the visual and text encoders except bias and normalization parameters during few-shot training.The encoders may be a ResNet or ViT visual encoder and a Transformer text encoder.
  • Few-shot Training Procedure: Each iteration samples C-way K-shot examples, splits them into support and query sets, trains on support minibatches, and evaluates on the query set.Training repeats until the maximum iteration count N is reached.
  • Few-shot Training Procedure: The training loss uses τ-scaled dot products between normalized image and text embeddings with labels mapped from answers and prompts.The procedure updates parameters with Adam and cross-entropy loss.
  • Template Generation: Demonstrations cover five question types and three answer types while showing how questions are converted into masked templates.A demonstration contains a question and an answer that guide template generation.
  • Template Generation: T5 uses [mask] to mark where answer words should be placed, then the generated template is filled and further processed.For binary yes/no questions, the method directly generates positive and negative prompts.
Loading 2203.07190v1…