PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Prompt for Vision-Language Models

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei Liu

arXiv:2109.01134v6cs.CVcs.AIcs.LG

TL;DR

Prompt engineering for vision-language models is time-consuming and sensitive to wording. CoOp learns prompt context vectors while freezing pretrained parameters, and across 11 datasets it beats hand-crafted prompts with as few as one or two shots, averaging a 15% margin with 16 shots.

  • Problem

    Prompt engineering for vision-language models requires substantial time and expertise, while small wording changes can significantly affect downstream performance.

  • Method

    CoOp replaces manually tuned context words with learnable vectors while freezing pretrained parameters, using unified or class-specific contexts.

  • Results

    Across 11 datasets, CoOp beats hand-crafted prompts with as few as one or two shots and averages a 15% margin with 16 shots.

  • Takeaways & Limitations

    CoOp turns pretrained vision-language models into data-efficient visual learners with stronger domain-shift robustness than manual prompts.

  • Takeaways & Limitations

    CoOp’s continuous prompts are relatively difficult to interpret and are sensitive to noisy labels, as shown by weak Food101 performance.

Abstract

from arXiv · show

Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Different from the traditional representation learning that is based mostly on discretized labels, vision-language pre-training aligns images and texts in a common feature space, which allows zero-shot transfer to a downstream task via prompting, i.e., classification weights are synthesized from natural language describing classes of interest. In this work, we show that a major challenge for deploying such models in practice is prompt engineering, which requires domain expertise and is extremely time-consuming -- one needs to spend a significant amount of time on words tuning since a slight change in wording could have a huge impact on performance. Inspired by recent advances in prompt learning research in natural language processing (NLP), we propose Context Optimization (CoOp), a simple approach specifically for adapting CLIP-like vision-language models for downstream image recognition. Concretely, CoOp models a prompt's context words with learnable vectors while the entire pre-trained parameters are kept fixed. To handle different image recognition tasks, we provide two implementations of CoOp: unified context and class-specific context. Through extensive experiments on 11 datasets, we demonstrate that CoOp requires as few as one or two shots to beat hand-crafted prompts with a decent margin and is able to gain significant improvements over prompt engineering with more shots, e.g., with 16 shots the average gain is around 15% (with the highest reaching over 45%). Despite being a learning-based approach, CoOp achieves superb domain generalization performance compared with the zero-shot model using hand-crafted prompts.

1 Introduction

Vision-language pre-training enables prompt-based transfer, but prompt engineering is time-consuming and sensitive to wording. CoOp automates this adaptation with learnable prompt-context vectors, achieving data-efficient recognition and stronger domain-shift robustness across 11 datasets.

  • Motivation: Vision-language pre-training aligns images and texts, enabling transfer to downstream tasks through prompting.CLIP and ALIGN use separate image and text encoders with a contrastive objective that pulls matched pairs together and pushes unmatched pairs apart.
  • Motivation: Prompt engineering is time-consuming, wording-sensitive, and not guaranteed to produce optimal downstream prompts.For Caltech101, adding “a” before the class token yields more than 5% improvement, yet extensive tuning still does not ensure optimality.
  • Method: CoOp automates prompt engineering by representing context words with learnable vectors while keeping the pre-trained model parameters fixed.The context vectors can be initialized randomly or with pre-trained word embeddings, and two implementations address different recognition tasks.
  • Results: One or two shots suffice for CoOp to beat hand-crafted prompts with a decent margin across 11 diverse visual-recognition datasets.The benchmark covers generic objects, scenes, actions, fine-grained categories, textures, and satellite imagery.
  • Results: 16 shots produce an average margin of around 15% over hand-crafted prompts, with the highest exceeding 45%.CoOp also outperforms the linear probe baseline and is more robust than the manual-prompt zero-shot model under domain shifts.

2 Related Work

Vision-language models enable generic visual representations and zero-shot classification through prompting, building on Transformer text encoders, large-minibatch contrastive learning, and web-scale image-text data. This work connects that line to continuous prompt learning in NLP and applies prompt learning to vision-language model adaptation, introducing unified and class-specific contexts.

  • Vision-language learning: Vision-language models support generic visual representations and zero-shot transfer to downstream classification tasks through prompting.This line includes models such as CLIP and ALIGN.
  • Vision-language learning: CLIP and ALIGN were driven by Transformer text representation learning, large-minibatch contrastive learning, and web-scale image-text datasets.CLIP uses 400 million curated image-text pairs, whereas ALIGN uses 1.8 billion noisy image-text pairs.
  • Vision-language learning: Mapping images and text into a common embedding space has been studied for nearly a decade using earlier word-vector, TF-IDF, and metric-learning technologies.The passage contrasts these earlier approaches with newer vision-language technologies.
  • Prompt learning in NLP: Continuous prompt learning methods optimize vectors in word-embedding space, but make the learned “words” difficult to visualize compared with discrete-token search.These methods are closely related to the approach developed here.
  • Prompt learning for vision: The work is presented as the first application of prompt learning to adapting large vision-language models in computer vision, with benefits for transfer performance and robustness to domain shifts.The authors frame this direction as important for democratizing foundation models.

3 Methodology

CoOp adapts CLIP-like vision-language models by learning continuous prompt-context vectors while freezing pre-trained parameters. It supports unified and class-specific contexts, optimized through classification loss and text-encoder gradients for downstream recognition.

  • CLIP Background: CLIP uses separate image and Transformer-based text encoders to map visual and linguistic inputs into embedding spaces.The image encoder may be a CNN such as ResNet-50 or a ViT, while the text encoder generates representations from natural language.
  • Zero-Shot Inference: Zero-shot recognition compares image features with class-specific classification weights synthesized by the text encoder from prompts such as “a photo of a [CLASS].”Each class weight is generated by replacing the class token with a specific class name.
  • Context Optimization: CoOp replaces manually tuned context words with continuous vectors learned end-to-end from data while keeping massive pre-trained parameters frozen.The learned vectors have the same dimensionality as CLIP word embeddings, 512 for CLIP.
  • Context Implementations: Unified context shares one learned context across all classes, whereas class-specific context assigns independent context vectors to each class.Class-specific context is particularly useful for some fine-grained classification tasks.
  • Training: Training minimizes standard cross-entropy classification loss, back-propagating gradients through the text encoder to optimize task-relevant context in the word-embedding space.Continuous representations enable exploration of the full word-embedding space while leveraging knowledge encoded in the frozen parameters.

4 Experiments

Experiments on 11 image-classification datasets show that CoOp is an effective few-shot learner, outperforming hand-crafted prompts and linear probing while retaining strong robustness to distribution shifts. Performance depends on context design, with trade-offs between accuracy and robustness across context length, context sharing, and vision backbones.

  • Comparison with Hand-Crafted Prompts: At 16 shots, CoOp improves over hand-crafted prompts by over 45% on EuroSAT and over 20% on DTD.Improvements exceeding 10% also occur on Flowers102, StanfordCars, FGVCAircraft, SUN397, and UCF101.
  • Comparison with Linear Probe: CoOp outperforms linear-probe CLIP in the low-data regime, while linear probing needs more than 4 shots on average merely to match zero-shot CLIP.CoOp’s advantage is especially large with one or two shots and grows as more shots become available.
  • Context and Vision Backbones: Unified context performs better on average, while class-specific context is preferred for StanfordCars, Flowers102, and FGVCAircraft; stronger vision backbones improve performance across all architectures.The backbone comparison covers CNNs and ViTs, and CoOp’s advantage over hand-crafted prompts remains significant across architectures.
  • Domain Generalization: CoOp improves robustness to distribution shifts across ImageNet-derived target datasets, whereas linear probing performs much worse on those domains.The evaluation transfers from ImageNet to ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.

Comparison with Other Fine-tuning Methods

CoOp outperforms alternative fine-tuning strategies, while its performance is largely insensitive to whether context vectors use manual or random initialization. Interpreting the learned continuous prompts remains difficult and requires vocabulary-based approximation.

  • Comparison with Other Fine-tuning Methods: CoOp largely outperforms image-encoder fine-tuning, text-output transformation layers, and text-output bias terms.Image-encoder fine-tuning performs poorly; transformation layers slightly improve the zero-shot model, while bias terms show promise but remain substantially below CoOp.
  • Interpreting the Learned Prompts: Interpreting CoOp’s learned prompts is difficult because their context vectors are optimized in a continuous space.The authors approximate interpretation by finding vocabulary items closest to learned vectors using Euclidean distance, although CLIP’s BPE vocabulary includes subwords.

5 Conclusion, Limitations and Future Work

The study shows that prompt learning can adapt CLIP-like vision-language models into data-efficient learners, with CoOp outperforming manual prompts in domain generalization. It also identifies interpretability and noisy-label sensitivity limitations while motivating extensions to broader adaptation settings.

  • Conclusion: CoOp demonstrates that prompt learning can make CLIP-like models data-efficient and substantially improve domain generalization over manual prompts.The paper presents the first comprehensive study of adapting large vision models with prompt learning.
  • Limitations: CoOp’s strong performance is relatively difficult to interpret, consistent with other continuous prompt learning methods in NLP.This limitation concerns the interpretability of the learned continuous prompts.
  • Limitations: CoOp is sensitive to noisy labels, as indicated by its weak performance on Food101.The Food101 result is cited as experimental evidence of this sensitivity.
  • Future Work: CoOp’s simplicity enables future work on cross-dataset transfer, test-time adaptation, and generic adaptation methods for mega-size vision models.The paper identifies these directions as open questions for extending prompt learning and adapting larger vision models.

Appendix · A Datasets Details

The appendix documents dataset statistics, zero-shot prompt specifications, and preprocessing choices for the 11 datasets and four ImageNet variants. It also references analyses of learned context vectors and ImageNet fine-tuning comparisons.

  • A Datasets Details: Table 6 reports detailed statistics for 11 datasets and four ImageNet variants.The table also specifies the hand-crafted prompts used for zero-shot CLIP.
  • A Datasets Details: The appendix provides the hand-crafted prompts used for zero-shot CLIP.These prompts are detailed alongside the dataset statistics in Table 6.
  • A Datasets Details: For Caltech101, the “BACKGROUND Google” and “Faces easy” classes are discarded.This is an explicit dataset construction choice described in the appendix.
  • A Datasets Details: For UCF101, the middle frame of each video is used as input to the image encoder.The appendix treats this video-to-image conversion as the dataset input procedure.
  • A Datasets Details: Table 4 lists the nearest words for each of the 16 context vectors learned by CoOp.Distances are shown in parentheses, and N/A denotes non-Latin characters.
  • A Datasets Details: Table 5 compares CoOp with other fine-tuning methods on ImageNet using 16 shots.The table defines ∆ as the difference with the zero-shot model.

B Results on DOSCO-2k

On DOSCO-2k, CoOp and CoCoOp achieve substantially better domain generalization than zero-shot CLIP while tuning only a small number of parameters. The benchmark evaluates transfer across seven datasets with broader contextual domain shifts.

  • DOSCO-2k: DOSCO-2k contains 7 image recognition datasets spanning generic objects, fine-grained aircraft recognition, and action recognition.The benchmark focuses on broader contextual domain shift rather than manually defined image-style variations.
  • Results: CoOp and CoCoOp outperform zero-shot CLIP by a large margin on unseen-domain tests.Models are trained on the training set, with checkpoints selected by validation performance for final testing in unseen domains.
  • Results: The results indicate that CoOp and CoCoOp have strong potential for transfer learning despite requiring only a small number of parameters to tune.Four different architectures are evaluated in the benchmark results.
Loading 2109.01134v6…