Conditional Prompt Learning for Vision-Language Models

Kaiyang Zhou; Jingkang Yang; Chen Change Loy; Ziwei Liu

Source-linked AI summary

Conditional Prompt Learning for Vision-Language Models

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei Liu

arXiv:2203.05557v2Mar 10, 2022cs.CVcs.AIcs.CLcs.LG

TL;DR

CoOp efficiently adapts vision-language models but its static learned context generalizes poorly beyond base classes. CoCoOp adds image-conditioned tokens and achieves stronger generalization, cross-dataset transfer, and domain generalization, while retaining training-efficiency limitations.

Problem
CoOp’s learned context is not generalizable to wider unseen classes within the same task, indicating overfitting to base classes.
Method
CoCoOp extends CoOp with conditional prompt learning by using a lightweight neural network to generate an input-conditional token for each image.
Results
CoCoOp achieves the best overall performance across 11 datasets and outperforms CoOp in unseen-class generalization, cross-dataset transfer, and domain generalization.
Takeaways & Limitations
Conditional prompts provide a simple parameter-efficient approach that performs strongly across base-to-new generalization, cross-dataset transfer, and domain generalization scenarios.
Takeaways & Limitations
CoCoOp trains slowly and uses substantial GPU memory for batch sizes larger than one because each image requires an independent text-encoder pass.

Abstract

from arXiv · show

With the rise of powerful pre-trained vision-language models like CLIP, it becomes essential to investigate ways to adapt these models to downstream datasets. A recently proposed method named Context Optimization (CoOp) introduces the concept of prompt learning -- a recent trend in NLP -- to the vision domain for adapting pre-trained vision-language models. Specifically, CoOp turns context words in a prompt into a set of learnable vectors and, with only a few labeled images for learning, can achieve huge improvements over intensively-tuned manual prompts. In our study we identify a critical problem of CoOp: the learned context is not generalizable to wider unseen classes within the same dataset, suggesting that CoOp overfits base classes observed during training. To address the problem, we propose Conditional Context Optimization (CoCoOp), which extends CoOp by further learning a lightweight neural network to generate for each image an input-conditional token (vector). Compared to CoOp's static prompts, our dynamic prompts adapt to each instance and are thus less sensitive to class shift. Extensive experiments show that CoCoOp generalizes much better than CoOp to unseen classes, even showing promising transferability beyond a single dataset; and yields stronger domain generalization performance as well. Code is available at https://github.com/KaiyangZhou/CoOp.

1. Introduction

Pre-trained vision-language models offer an efficient basis for downstream adaptation, but CoOp’s learned context overfits base classes and generalizes poorly to unseen classes. CoCoOp addresses this with image-conditioned prompts and shows stronger generalization, transfer, and domain performance.

Vision-language models generate class-specific classification weights through a prompted text encoder rather than fixed label weights.
Fine-tuning large vision-language models is impractical, while manual prompt engineering is time-consuming and does not guarantee an optimal prompt.
CoOp replaces prompt context words with learnable vectors and achieves large improvements over intensively tuned manual prompts using only a few labeled images.
CoOp’s learned context performs well on base classes but drops substantially on unseen classes within the same scene-recognition task, indicating overfitting to base classes.
CoCoOp extends CoOp with conditional prompt learning: a lightweight neural network generates an image-specific token combined with learnable context vectors.
Across 11 datasets, CoCoOp achieves the best overall performance against zero-shot and CoOp methods, with gains on unseen classes, cross-dataset transfer, and domain generalization.

2. Related Work

Related work spans vision-language models, NLP prompt learning, continuous prompt optimization, and zero-shot learning. CoCoOp builds on CoOp to address its weak generalizability through conditional prompts.

Vision-language models align images and text using jointly learned encoders, with CLIP using contrastive learning on 400 million image-text pairs.
Prompt learning originated in NLP, where prompts formulate downstream tasks for pretrained language models through cloze-style prediction.
Continuous prompt learning turns prompts into optimizable vectors, avoiding purely manual prompt design.
CoOp was an early vision-domain application of continuous prompt learning for adapting pretrained vision-language models.
Zero-shot learning similarly targets novel-class recognition from base-class training and relates to the seen-class bias problem.

3. Methodology

The method reviews CLIP and CoOp before introducing CoCoOp, which conditions learnable prompt context on each input image through a lightweight Meta-Net. This design targets CoOp’s weak generalization to unseen classes while retaining parameter efficiency.

CLIP: CLIP encodes images and text into a joint embedding space and performs zero-shot recognition by comparing image features with prompt-derived class representations.Its image encoder produces feature vectors, while the Transformer text encoder produces vectorized representations from token sequences.
CoOp: CoOp replaces fixed context words with M learnable vectors shared across classes and optimizes them for downstream recognition while freezing CLIP’s base model.The class prompt is formed by combining the shared context vectors with each class embedding, and training uses a differentiable cross-entropy objective.
Motivation: CoOp’s context is not generalizable to wider unseen classes, motivating instance-conditional context that shifts attention from specific classes toward each input instance.The proposed rationale is to reduce overfitting to the classes observed during training while addressing the broader task.
CoCoOp: CoCoOp adds a lightweight Meta-Net that generates an input-conditional token for each image and combines it with the learnable context vectors.A straightforward design using one network per context token would be larger; the Meta-Net provides a parameter-efficient alternative.
CoCoOp: For an input image, Meta-Net output π is added to each base context vector, producing prompts whose context varies with the image.The resulting class prompt is t_i(x) = {v_1(x), v_2(x), . . . , v_M(x), c_i}.
Optimization: CoCoOp jointly updates the context vectors and Meta-Net parameters using a two-layer Linear-ReLU-Linear bottleneck whose hidden layer reduces input dimension by 16×.The Meta-Net receives features produced by the image encoder, while more advanced designs are left for future work.

4. Experiments

Experiments evaluate CoCoOp on base-to-new generalization, cross-dataset transfer, domain generalization, and class-incremental recognition. Across these settings, instance-conditional prompts improve generalization and transferability over CoOp, while often preserving or improving overall performance.

Experimental Setup: The benchmark uses 11 diverse recognition datasets and evaluates 16-shot models across base-to-new generalization, cross-dataset transfer, and domain generalization.Learning-based models are trained from few-shot data, with results averaged over three runs.
Base-to-New Generalization: CoOp’s new-class accuracy trails its base-class accuracy by almost 20% on average, at 63.22% versus 82.69%.This gap exposes weak generalizability despite CoOp’s advantage in average performance over CLIP.
Base-to-New Generalization: CoCoOp raises unseen-class accuracy from 63.22% to 71.69%, substantially narrowing the gap with manual prompts.It improves accuracy on 5 of 11 datasets and reaches 70.43% on ImageNet, exceeding CLIP’s 68.14%.
Base-to-New Generalization: CoCoOp’s base-accuracy losses are mostly under 3%, while unseen-class gains outweigh them; on StanfordCars, -7.63% base accuracy accompanies +13.19% new-class accuracy.The StanfordCars changes combine to produce a 5.56% positive improvement for CoCoOp.
Cross-Dataset Transfer: When prompts learned from ImageNet transfer to 10 target datasets, CoCoOp mostly outperforms CoOp, including on fine-grained and specialized datasets.The advantage is especially evident on FGVCAircraft and DTD, despite lower absolute accuracy on distant categories.
Domain Generalization: CoCoOp is more domain-generalizable than CoOp, performing better on three of four target benchmarks while both prompt-learning methods beat CLIP on all targets.CoCoOp performs slightly worse than CoOp on ImageNetV2 but better on the other three target datasets.
Class-Incremental Test: In class-incremental recognition, the model must recognize new classes without training data from those classes, using only base-class training data.The experiment compares CLIP, CoOp, and CoCoOp across the 11 datasets.
Ablation Studies: Matching CoOp’s parameter size without its Meta-Net does not reproduce CoCoOp’s improvements, indicating that increased parameter count is not the key explanation.The comparison increases CoOp’s context-token count until the models have similar sizes.

5. Limitations

CoCoOp has two reported limitations: inefficient training and incomplete performance relative to CLIP on unseen classes.

CoCoOp is slow to train and consumes substantial GPU memory when batch size exceeds one.Its instance-conditional design requires an independent text-encoder forward pass for each image.
CoCoOp’s unseen-class performance still lags behind CLIP on 7 of 11 datasets.The authors identify closing or overturning these gaps as an open need.

6. Discussion and Conclusion

The paper studies parameter-efficient adaptation of costly foundation models and shows that conditional prompt learning improves generalization across several transfer settings. It also identifies efficiency and scalability as directions for future work.

Foundation models offer broad capabilities but require costly pre-training and contain very large numbers of parameters.The CLIP ViT-B/16 model used in the experiments has 150M parameters, motivating efficient adaptation methods.
CoCoOp provides insights into static-prompt generalizability and performs strongly in base-to-new, cross-dataset, and domain-generalization scenarios.The study follows the parameter-efficient prompt-learning line and evaluates conditional prompt learning across varied problem settings.
Future work includes more efficient conditional prompt learning, larger Meta-Net models, larger training sets, and heterogeneous mixed-dataset training.These directions are motivated by training efficiency and the observed transferability of instance-conditional prompts.

A. Results on DOSCO-2k

On DOSCO-2k, CoOp and CoCoOp outperform zero-shot CLIP, while CoCoOp surpasses CoOp on 4 of 7 datasets but has lower average performance.

CoOp and CoCoOp outperform CLIP by a large margin on DOSCO-2k despite tuning only a small number of parameters.The benchmark evaluates domain generalization across unseen domains using seven datasets and four architectures.
CoCoOp beats CoOp on 4 of 7 DOSCO-2k datasets, but CoOp achieves higher average performance.The results are presented as evidence that efficient adaptation methods have potential for transfer-learning problems.