PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Prompt for Vision-Language Models

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei Liu

arXiv:2109.01134v6cs.CVcs.AIcs.LG

TL;DR

Prompt engineering for vision-language models is time-consuming and sensitive to wording. CoOp learns prompt context vectors while freezing pretrained parameters, and across 11 datasets it beats hand-crafted prompts with as few as one or two shots, averaging a 15% margin with 16 shots.

  • Problem

    Prompt engineering for vision-language models requires substantial time and expertise, while small wording changes can significantly affect downstream performance.

    Source: §1 · p. 2

  • Method

    CoOp replaces manually tuned context words with learnable vectors while freezing pretrained parameters, using unified or class-specific contexts.

    Source: §1 · p. 2

  • Results

    Across 11 datasets, CoOp beats hand-crafted prompts with as few as one or two shots and averages a 15% margin with 16 shots.

    Source: §1 · pp. 2–3, §4.1 · p. 7

  • Takeaways & Limitations

    CoOp turns pretrained vision-language models into data-efficient visual learners with stronger domain-shift robustness than manual prompts.

    Source: §1 · p. 3, §5 · p. 10

    CoOp’s continuous prompts are relatively difficult to interpret and are sensitive to noisy labels, as shown by weak Food101 performance.

    Source: §5 · p. 10

Abstract

from arXiv · show

Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Different from the traditional representation learning that is based mostly on discretized labels, vision-language pre-training aligns images and texts in a common feature space, which allows zero-shot transfer to a downstream task via prompting, i.e., classification weights are synthesized from natural language describing classes of interest. In this work, we show that a major challenge for deploying such models in practice is prompt engineering, which requires domain expertise and is extremely time-consuming -- one needs to spend a significant amount of time on words tuning since a slight change in wording could have a huge impact on performance. Inspired by recent advances in prompt learning research in natural language processing (NLP), we propose Context Optimization (CoOp), a simple approach specifically for adapting CLIP-like vision-language models for downstream image recognition. Concretely, CoOp models a prompt's context words with learnable vectors while the entire pre-trained parameters are kept fixed. To handle different image recognition tasks, we provide two implementations of CoOp: unified context and class-specific context. Through extensive experiments on 11 datasets, we demonstrate that CoOp requires as few as one or two shots to beat hand-crafted prompts with a decent margin and is able to gain significant improvements over prompt engineering with more shots, e.g., with 16 shots the average gain is around 15% (with the highest reaching over 45%). Despite being a learning-based approach, CoOp achieves superb domain generalization performance compared with the zero-shot model using hand-crafted prompts.

1 Introduction

Vision-language pre-training enables prompt-based transfer, but prompt engineering is time-consuming and sensitive to wording. CoOp automates this adaptation with learnable prompt-context vectors, achieving data-efficient recognition and stronger domain-shift robustness across 11 datasets.

Motivation

  • Vision-language pre-training aligns images and texts, enabling transfer to downstream tasks through prompting.CLIP and ALIGN use separate image and text encoders with a contrastive objective that pulls matched pairs together and pushes unmatched pairs apart.Source: §1 · pp. 1–2
  • Prompt engineering is time-consuming, wording-sensitive, and not guaranteed to produce optimal downstream prompts.For Caltech101, adding “a” before the class token yields more than 5% improvement, yet extensive tuning still does not ensure optimality.Source: §1 · p. 2

Method

  • CoOp automates prompt engineering by representing context words with learnable vectors while keeping the pre-trained model parameters fixed.The context vectors can be initialized randomly or with pre-trained word embeddings, and two implementations address different recognition tasks.Source: §1 · p. 2

Results

  • One or two shots suffice for CoOp to beat hand-crafted prompts with a decent margin across 11 diverse visual-recognition datasets.The benchmark covers generic objects, scenes, actions, fine-grained categories, textures, and satellite imagery.Source: §1 · p. 2
  • 16 shots produce an average margin of around 15% over hand-crafted prompts, with the highest exceeding 45%.CoOp also outperforms the linear probe baseline and is more robust than the manual-prompt zero-shot model under domain shifts.Source: §1 · p. 3

2 Related Work

Vision-language models enable generic visual representations and zero-shot classification through prompting, building on Transformer text encoders, large-minibatch contrastive learning, and web-scale image-text data. This work connects that line to continuous prompt learning in NLP and applies prompt learning to vision-language model adaptation, introducing unified and class-specific contexts.

Vision-language learning

  • Vision-language models support generic visual representations and zero-shot transfer to downstream classification tasks through prompting.This line includes models such as CLIP and ALIGN.Source: §2.1 · p. 3
  • CLIP and ALIGN were driven by Transformer text representation learning, large-minibatch contrastive learning, and web-scale image-text datasets.CLIP uses 400 million curated image-text pairs, whereas ALIGN uses 1.8 billion noisy image-text pairs.Source: §2.1 · p. 3
  • Mapping images and text into a common embedding space has been studied for nearly a decade using earlier word-vector, TF-IDF, and metric-learning technologies.The passage contrasts these earlier approaches with newer vision-language technologies.Source: §2.1 · p. 3

Prompt learning in NLP

  • Continuous prompt learning methods optimize vectors in word-embedding space, but make the learned “words” difficult to visualize compared with discrete-token search.These methods are closely related to the approach developed here.Source: §2.2 · p. 3

Prompt learning for vision

  • The work is presented as the first application of prompt learning to adapting large vision-language models in computer vision, with benefits for transfer performance and robustness to domain shifts.The authors frame this direction as important for democratizing foundation models.Source: §2.2 · pp. 3–4

3 Methodology

CoOp adapts CLIP-like vision-language models by learning continuous prompt-context vectors while freezing pre-trained parameters. It supports unified and class-specific contexts, optimized through classification loss and text-encoder gradients for downstream recognition.

CLIP Background

  • CLIP uses separate image and Transformer-based text encoders to map visual and linguistic inputs into embedding spaces.The image encoder may be a CNN such as ResNet-50 or a ViT, while the text encoder generates representations from natural language.Source: §3.1 · p. 4

Zero-Shot Inference

  • Zero-shot recognition compares image features with class-specific classification weights synthesized by the text encoder from prompts such as “a photo of a [CLASS].”Each class weight is generated by replacing the class token with a specific class name.Source: §3.1 · p. 4

Context Optimization

  • CoOp replaces manually tuned context words with continuous vectors learned end-to-end from data while keeping massive pre-trained parameters frozen.The learned vectors have the same dimensionality as CLIP word embeddings, 512 for CLIP.Source: §3.2 · p. 5

Context Implementations

  • Unified context shares one learned context across all classes, whereas class-specific context assigns independent context vectors to each class.Class-specific context is particularly useful for some fine-grained classification tasks.Source: §3.2 · p. 5

Training

  • Training minimizes standard cross-entropy classification loss, back-propagating gradients through the text encoder to optimize task-relevant context in the word-embedding space.Continuous representations enable exploration of the full word-embedding space while leveraging knowledge encoded in the frozen parameters.Source: §3.2 · p. 5

4 Experiments

Experiments on 11 image-classification datasets show that CoOp is an effective few-shot learner, outperforming hand-crafted prompts and linear probing while retaining strong robustness to distribution shifts. Performance depends on context design, with trade-offs between accuracy and robustness across context length, context sharing, and vision backbones.

Comparison with Hand-Crafted Prompts

  • At 16 shots, CoOp improves over hand-crafted prompts by over 45% on EuroSAT and over 20% on DTD.Improvements exceeding 10% also occur on Flowers102, StanfordCars, FGVCAircraft, SUN397, and UCF101.Source: §4.1 · p. 7

Comparison with Linear Probe

  • CoOp outperforms linear-probe CLIP in the low-data regime, while linear probing needs more than 4 shots on average merely to match zero-shot CLIP.CoOp’s advantage is especially large with one or two shots and grows as more shots become available.Source: §4.1 · pp. 7–8

Context and Vision Backbones

  • Unified context performs better on average, while class-specific context is preferred for StanfordCars, Flowers102, and FGVCAircraft; stronger vision backbones improve performance across all architectures.The backbone comparison covers CNNs and ViTs, and CoOp’s advantage over hand-crafted prompts remains significant across architectures.Source: §4.1 · p. 8, §4.3 · p. 9

Domain Generalization

  • CoOp improves robustness to distribution shifts across ImageNet-derived target datasets, whereas linear probing performs much worse on those domains.The evaluation transfers from ImageNet to ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.Source: §4.2 · pp. 8–9

Comparison with Other Fine-tuning Methods

CoOp outperforms alternative fine-tuning strategies, while its performance is largely insensitive to whether context vectors use manual or random initialization. Interpreting the learned continuous prompts remains difficult and requires vocabulary-based approximation.

Comparison with Other Fine-tuning Methods

  • CoOp largely outperforms image-encoder fine-tuning, text-output transformation layers, and text-output bias terms.Image-encoder fine-tuning performs poorly; transformation layers slightly improve the zero-shot model, while bias terms show promise but remain substantially below CoOp.Source: Comparison with Other Fine-tuning… · p. 10

Interpreting the Learned Prompts

  • Interpreting CoOp’s learned prompts is difficult because their context vectors are optimized in a continuous space.The authors approximate interpretation by finding vocabulary items closest to learned vectors using Euclidean distance, although CLIP’s BPE vocabulary includes subwords.Source: Comparison with Other Fine-tuning… · p. 10

5 Conclusion, Limitations and Future Work

The study shows that prompt learning can adapt CLIP-like vision-language models into data-efficient learners, with CoOp outperforming manual prompts in domain generalization. It also identifies interpretability and noisy-label sensitivity limitations while motivating extensions to broader adaptation settings.

Conclusion

  • CoOp demonstrates that prompt learning can make CLIP-like models data-efficient and substantially improve domain generalization over manual prompts.The paper presents the first comprehensive study of adapting large vision models with prompt learning.Source: §5 · p. 10

Limitations

  • CoOp’s strong performance is relatively difficult to interpret, consistent with other continuous prompt learning methods in NLP.This limitation concerns the interpretability of the learned continuous prompts.Source: §5 · p. 10
  • CoOp is sensitive to noisy labels, as indicated by its weak performance on Food101.The Food101 result is cited as experimental evidence of this sensitivity.Source: §5 · p. 10

Future Work

  • CoOp’s simplicity enables future work on cross-dataset transfer, test-time adaptation, and generic adaptation methods for mega-size vision models.The paper identifies these directions as open questions for extending prompt learning and adapting larger vision models.Source: §5 · p. 10

Appendix · A Datasets Details

The appendix documents dataset statistics, zero-shot prompt specifications, and preprocessing choices for the 11 datasets and four ImageNet variants. It also references analyses of learned context vectors and ImageNet fine-tuning comparisons.

A Datasets Details

  • Table 6 reports detailed statistics for 11 datasets and four ImageNet variants.The table also specifies the hand-crafted prompts used for zero-shot CLIP.Source: A Datasets Details · p. 10
  • The appendix provides the hand-crafted prompts used for zero-shot CLIP.These prompts are detailed alongside the dataset statistics in Table 6.Source: A Datasets Details · p. 10
  • For Caltech101, the “BACKGROUND Google” and “Faces easy” classes are discarded.This is an explicit dataset construction choice described in the appendix.Source: A Datasets Details · p. 10
  • For UCF101, the middle frame of each video is used as input to the image encoder.The appendix treats this video-to-image conversion as the dataset input procedure.Source: A Datasets Details · p. 10
  • Table 4 lists the nearest words for each of the 16 context vectors learned by CoOp.Distances are shown in parentheses, and N/A denotes non-Latin characters.Source: Table 4 · p. 11
  • Table 5 compares CoOp with other fine-tuning methods on ImageNet using 16 shots.The table defines ∆ as the difference with the zero-shot model.Source: Table 5 · p. 11

B Results on DOSCO-2k

On DOSCO-2k, CoOp and CoCoOp achieve substantially better domain generalization than zero-shot CLIP while tuning only a small number of parameters. The benchmark evaluates transfer across seven datasets with broader contextual domain shifts.

DOSCO-2k

  • DOSCO-2k contains 7 image recognition datasets spanning generic objects, fine-grained aircraft recognition, and action recognition.The benchmark focuses on broader contextual domain shift rather than manually defined image-style variations.Source: B Results on DOSCO-2k · p. 11

Results

  • CoOp and CoCoOp outperform zero-shot CLIP by a large margin on unseen-domain tests.Models are trained on the training set, with checkpoints selected by validation performance for final testing in unseen domains.Source: B Results on DOSCO-2k · p. 11
  • The results indicate that CoOp and CoCoOp have strong potential for transfer learning despite requiring only a small number of parameters to tune.Four different architectures are evaluated in the benchmark results.Source: B Results on DOSCO-2k · p. 11

Related papers

Loading 2109.01134v6…