PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Prompt Learning for Vision-Language Models

Tony Huang, Jack Chu, Fangyun Wei

arXiv:2204.03649v2cs.CV

TL;DR

Prompt engineering is difficult, and supervised prompt-learning methods require labeled target data. The paper introduces UPL, which learns prompts from unlabeled target images using pseudo-labeling and self-training, and reports stronger transfer than original CLIP across 11 datasets, with its enhanced version competitive with 8-shot supervised methods.

  • Problem

    Designing effective prompts is laborious, while existing adaptation methods require labeled samples from target datasets, restricting scalability.

  • Method

    UPL generates pseudo labels for unlabeled target images, selects top-K confident samples per class, and optimizes a learnable prompt representation through self-training.

  • Results

    UPL improves over original CLIP with prompt engineering across ImageNet and 10 other datasets, while UPL* is competitive with 8-shot CoOp and 8-shot Tip-Adapter on most datasets.

  • Takeaways & Limitations

    Unsupervised prompt learning provides an alternative to labeled-data adaptation while avoiding prompt engineering for downstream image recognition.

Abstract

from arXiv · show

Contrastive vision-language models like CLIP have shown great progress in transfer learning. In the inference stage, the proper text description, also known as prompt, needs to be carefully designed to correctly classify the given images. In order to avoid laborious prompt engineering, recent works such as CoOp, CLIP-Adapter and Tip-Adapter propose to adapt vision-language models for downstream image recognition tasks on a small set of labeled data. Though promising improvements are achieved, requiring labeled data from the target datasets may restrict the scalability. In this paper, we explore a different scenario, in which the labels of the target datasets are unprovided, and we present an unsupervised prompt learning (UPL) approach to avoid prompt engineering while simultaneously improving transfer performance of CLIP-like vision-language models. As far as we know, UPL is the first work to introduce unsupervised learning into prompt learning. Experimentally, our UPL outperforms original CLIP with prompt engineering on ImageNet as well as other 10 datasets. An enhanced version of UPL is even competitive with the 8-shot CoOp and the 8-shot TIP-Adapter on most datasets. Code and models are available at https://github.com/tonyhuang2022/UPL.

Introduction

Vision-language models require carefully designed prompts for downstream image recognition, while supervised adaptation methods depend on labeled target data. UPL instead learns prompts from unlabeled target images through pseudo-labeling and self-training, improving transfer performance.

  • Motivation: CLIP-like vision-language models classify target images using text prompts, but selecting effective prompts often requires domain knowledge and laborious engineering.A representative template is “a photo of a [CLS]”.
  • Motivation: Supervised methods such as CoOp, CLIP-Adapter, and Tip-Adapter adapt vision-language models with small sets of labeled target images, limiting scalability.They respectively optimize continuous prompts, learn refined features with additional networks, or construct a query-key cache from few-shot supervision.
  • UPL: UPL generates pseudo labels for unlabeled target images and optimizes a learnable prompt representation through self-training on selected samples.The approach replaces hand-crafted prompts with the optimized representation during transfer.
  • UPL: UPL selects the top-K confident samples per class because class-biased predictions can create imbalanced pseudo-data distributions under threshold filtering.The method also notes that confidence scores do not clearly correlate with pseudo-label accuracy.
  • Results: UPL outperforms original CLIP with prompt engineering on ImageNet and 10 other datasets, while its enhanced version is competitive with 8-shot CoOp and 8-shot TIP-Adapter on most datasets.The paper presents UPL as an unsupervised alternative to labeled-data prompt learning.

Related Work

Related work establishes vision-language models as transferable two-tower systems and situates prompt learning as a way to avoid hand-crafted prompt design. Prior adaptation methods differ in whether they optimize prompts or add feature-adaptation modules.

  • Vision-language Models: Vision-language models align images and raw texts in a shared embedding space using image and text encoders trained on large-scale image-text pairs.Examples discussed include CLIP, ALIGN, FLIP, Wukong, and Florence.
  • Vision-language Models: These models have inherent transfer capacity for image recognition and have been adapted to tasks including detection, segmentation, action recognition, retrieval, captioning, and 3D recognition.The related-work discussion uses CLIP as the representative framework.
  • Prompt Learning: Prompt learning addresses the difficulty of hand-crafted prompt design by optimizing continuous prompt representations or training additional adapter networks.CoOp optimizes prompts, whereas CLIP-Adapter and related methods adapt text or visual features with extra networks.

Method

UPL generates pseudo labels without target annotations, selects balanced top-K samples per class, and optimizes a shared learnable prompt representation for CLIP transfer.

  • Overview of UPL: UPL uses a pretrained vision-language model to generate pseudo labels for unlabeled target images, then optimizes a learnable prompt representation through self-training.
  • Pseudo Label Generation: Top-K confident samples are selected per class because class-biased accuracy and weak confidence–accuracy correlation make fixed thresholds unreliable.This strategy avoids imbalanced pseudo-labeled data and limits some categories from overwhelming training.
  • Pseudo Label Generation: CLIP predicts each image’s class using cosine similarity between image and class embeddings, scaled by the learned temperature parameter.The predicted label is the class with maximum probability.
  • Prompt Representation Optimization: The continuous prompt is shared across categories and concatenated with each fixed class-word embedding before encoding class prompts.The learnable representation has default length L = 16, while the CLIP encoder weights remain fixed during optimization.
  • Prompt Representation Optimization: Prompt optimization minimizes cross-entropy on pseudo-labeled images, back-propagating through the text encoder to update only the prompt representation.After optimization, UPL replaces hand-crafted prompts with the learned representations for inference.

Experiment

Across 11 datasets, UPL improves CLIP transfer without target labels by combining pseudo-labeling, prompt ensembling, and related design choices. Experiments examine dataset setup, pseudo-labeling strategies, prompt representations, and robustness to noisy pseudo-labels.

  • Method: UPL uses CLIP models to generate pseudo-labels, then applies self-training to optimize a learnable prompt representation; additional models can improve pseudo-label quality.UPL* uses varied vision architectures only for pseudo-labeling while retaining CLIP with ResNet-50 for training and inference.
  • Pseudo-labeling: Increasing the number of pseudo-labeled samples improves performance in the top-K ablation.Figure 6 varies K to study this relationship.
  • Main Results: 68.37 averaged accuracy is achieved by UPL*, while UPL improves over prompt-engineered CLIP by +4.2 points across 11 datasets.UPL* is competitive with 8-shot CoOp and 8-shot Tip-Adapter on most listed datasets.
  • Pseudo-labeling: Top-K pseudo-labeling is compared with confidence thresholds because high thresholds imbalance pseudo-labeled data and low thresholds introduce excessive noise.The strategy is evaluated on UCF101 against commonly used threshold-based self-training approaches.
  • Prompt Representation: Prompt-representation ensembling nearly saturates at N = 16, while different learned representations show class-specific preferences that motivate ensembling.The prompt ensemble study evaluates N = 2, 4, 8, and 16 on UCF101.
  • Robustness: UPL remains robust to noisy pseudo-labels: low per-class pseudo-label accuracy can coexist with significant transfer improvement.The study finds no obvious correspondence between pseudo-label accuracy and transfer improvement on UCF101.

Conclusion

The paper proposes UPL as an unsupervised alternative to prompt engineering and few-shot adaptation for CLIP transfer. Across ImageNet and 10 other datasets, UPL improves over prompt-engineered CLIP, while UPL* approaches supervised few-shot methods on most datasets.

  • Conclusion: UPL avoids prompt engineering and target-dataset annotations by adapting pretrained vision-language models for downstream image recognition.The paper contrasts UPL with supervised CoOp, CLIP-Adapter, and Tip-Adapter.
  • Conclusion: UPL outperforms original CLIP with prompt engineering on ImageNet and 10 additional image-classification datasets.The evaluation covers 11 datasets in total.
  • Conclusion: UPL surpasses 2-shot CoOp on averaged accuracy, while UPL* is competitive with 8-shot CoOp and 8-shot Tip-Adapter on most datasets.These comparisons are reported across the evaluated datasets.

More Experiments

Additional experiments show that UPL generalizes across prompt-learning frameworks and remains robust to prompt-count and token-position choices. UPL also outperforms CLIP in both single-prompt and multiple-prompt settings.

  • Framework Generalization: UPL can be applied to CoCoOp, CLIP-Adapter, and Tip-Adapter-F, and all tested UPL-equipped methods outperform original CLIP.The results support generalization beyond the CoOp-style structure used in the main paper.
  • Prompt Design: UPL is robust to the position of the [CLS] token because three tested insertion positions achieve similar performance.The comparison is reported in Table 6.
  • Prompt Count: UPL demonstrates superiority over original CLIP with both one prompt and multiple prompts.The comparison addresses whether the main result depends on using multiple prompts.

Visualization Results

The visualizations examine prompt representations and pseudo-label quality, emphasizing that confidence and pseudo-label accuracy do not fully determine transfer gains. They also motivate top-K selection and prompt ensembling.

  • Prompt Visualization: Nearest-vocabulary analysis is used to interpret continuous prompt representations, whose vectors are optimized rather than directly expressed as words.The analysis searches for vocabulary or subword tokens closest to the learned vectors.
  • Pseudo-label Visualization: Top-K pseudo-labeling is motivated by cases where low-confidence samples are correctly predicted, especially on fine-grained datasets such as FGVCAircraft.The visualization compares this phenomenon on DTD and FGVCAircraft.

Dataset Details and Hand-Craft Prompts

The paper documents dataset-specific hand-crafted prompts used for pseudo-labeling and analyzes optimized prompt representations through nearest-vocabulary words. It also examines why confidence alone may not fully indicate pseudo-label quality.

  • Optimized Prompt Representations: Tables 8 and 9 identify vocabulary words nearest to optimized UPL prompt representations using Euclidean distance.Table 8 covers five datasets, while Table 9 examines four prompt representations on ImageNet; N/A denotes non-Latin characters.
  • Dataset Details and Hand-Craft Prompts: Table 10 lists each dataset’s class count, train/test sizes, and hand-crafted prompt used for pseudo-labeling.The prompts use the simplest form without complex prompt engineering.
  • Pseudo-Label Confidence: Figure 9 studies DTD and FGVCAircraft to show that prediction confidence does not completely reflect pseudo-label quality.The figure notes that some low-confidence predictions may still be correct.
Loading 2204.03649v2…