PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Prompt Learning for Vision-Language Models

Tony Huang, Jack Chu, Fangyun Wei

arXiv:2204.03649v2cs.CV

TL;DR

Vision-language model transfer requires carefully designed prompts, while supervised adaptation uses labeled target images. UPL optimizes prompt representations from pseudo-labeled unlabeled images, outperforming original CLIP with prompt engineering by +4.2 point of averaged accuracy across 11 datasets.

Abstract

from arXiv · show

Contrastive vision-language models like CLIP have shown great progress in transfer learning. In the inference stage, the proper text description, also known as prompt, needs to be carefully designed to correctly classify the given images. In order to avoid laborious prompt engineering, recent works such as CoOp, CLIP-Adapter and Tip-Adapter propose to adapt vision-language models for downstream image recognition tasks on a small set of labeled data. Though promising improvements are achieved, requiring labeled data from the target datasets may restrict the scalability. In this paper, we explore a different scenario, in which the labels of the target datasets are unprovided, and we present an unsupervised prompt learning (UPL) approach to avoid prompt engineering while simultaneously improving transfer performance of CLIP-like vision-language models. As far as we know, UPL is the first work to introduce unsupervised learning into prompt learning. Experimentally, our UPL outperforms original CLIP with prompt engineering on ImageNet as well as other 10 datasets. An enhanced version of UPL is even competitive with the 8-shot CoOp and the 8-shot TIP-Adapter on most datasets. Code and models are available at https://github.com/tonyhuang2022/UPL.

Introduction

Vision-language models need carefully designed prompts for downstream classification, while existing adaptation methods rely on labeled target data; UPL instead learns prompts from unlabeled target images. The introduction presents UPL as improving transfer performance through pseudo-labeling and prompt ensembles.

  • Unlike supervised CoOp (Zhou et al. 2021), CLIP-Adapter (Gao et al. 2021), and Tip-Adapter (Zhang et al. 2021a), UPL adapts vision-language models without target labels.Those methods require annotated target samples, which may limit scalability.Source: Introduction · p. 1
  • UPL uses a pretrained vision-language model to pseudo-label target images, then self-trains a learnable prompt representation on selected pseudo-labeled samples.Replacing the hand-crafted prompt with the optimized representation improves CLIP transfer performance.Source: Introduction · pp. 1–2
  • Top-K confident samples per class avoid threshold-based filtering that can imbalance pseudo-data, since confidence scores do not reliably indicate pseudo-label accuracy.Despite noisy labels, UPL is robust because all classes use the same prompt representation.Source: Introduction · p. 2
  • UPL adds pseudo-label and prompt-representation ensembles to further improve transfer performance.Source: Introduction · p. 2
  • UPL significantly outperforms prompt-engineered CLIP on ImageNet and 10 other datasets, while an enhanced version is competitive with 8-shot CoOp and TIP-Adapter on most datasets.Source: Introduction · p. 2

Related Work

Prior work establishes vision-language models as transferable image-text representations and develops prompt adaptation methods that typically depend on labeled target data. UPL instead applies self-training to optimize prompts without target annotations while keeping both encoders fixed.

Vision-language Models

  • Vision-language models use two-tower image and text encoders with contrastive learning, supporting transfer to image recognition and other vision tasks.Examples include CLIP (Radford et al. 2021), ALIGN (Jia et al. 2021), and Florence (Yuan et al. 2021); CLIP has also been adapted to tasks including detection and segmentation.Source: Related Work · p. 2

Prompt Learning

Self-training

  • Traditional self-training generates pseudo labels and fine-tunes with labeled and pseudo-labeled data; UPL uses self-training to optimize prompts while keeping both encoders fixed.The paper identifies UPL as the first use of self-training in prompt learning for vision-language models.Source: Related Work · p. 2

Method

UPL avoids target-dataset annotations by generating pseudo-labels for unlabeled images, learning a shared prompt representation, and replacing hand-crafted prompts with it at inference. Its pseudo-labeling selects top-K samples per class, while prompt optimization updates the prompt representation with the CLIP encoders frozen.

Overview of UPL

  • UPL generates pseudo-labels with a pre-trained vision-language model, then optimizes a shared learnable prompt representation on selected unlabeled samples.At transfer, it replaces hand-crafted prompts with the optimized representation and uses the CLIP inference pipeline.Source: Overview of UPL · p. 3

Pseudo Label Generation

  • Top-K selection per class avoids imbalanced pseudo-labeled data and reliance on confidence as a proxy for label accuracy; UPL sets K = 16.The method averages predictions from multiple CLIP vision architectures to generate enhanced pseudo-labels.Source: Pseudo Label Generation · pp. 3–4

Prompt Representation Optimization

  • UPL learns a shared prompt representation V with default length L = 16, minimizing cross-entropy against pseudo-labels while keeping both CLIP encoders fixed.Gradients pass through the text encoder to update V; multiple independently initialized prompt representations can also be averaged at inference.Source: Prompt Representation Optimization · pp. 4–5
  • UPL outperforms 2-shot CoOp, and UPL* is competitive with 8-shot CoOp and 8-shot Tip-Adapter on most datasets.Both UPL and UPL* improve on original CLIP with prompt engineering across 11 datasets.Source: Table 1 · p. 5

Experiment

UPL avoids prompt engineering and improves CLIP’s averaged accuracy across 11 datasets; UPL* further improves performance using additional CLIP models for pseudo-labeling. The experiments also examine pseudo-label selection, prompt representation choices, and robustness to noisy pseudo labels.

Implementation Details

  • UPL uses CLIP with ResNet-50, while UPL* uses additional CLIP architectures only to generate pseudo labels and retains ResNet-50 for training and inference.The default uses a simple prompt and the top 16 confident samples per class; prompt representations are initialized from a zero-mean Gaussian with standard deviation 0.02, and length L = 16. Training uses SGD for 50 epochs, batch size 32, and a cosine-decay learning rate starting at 0.002, with a first-epoch warmup at 1e-5.Source: Implementation Details · pp. 5–6
  • Experiments cover 11 public image-classification datasets spanning general objects, fine-grained categories, and textures.Source: Implementation Details · p. 6

Main Results

  • UPL improves averaged accuracy by +4.2 points over CLIP, while UPL* reaches 68.37 averaged accuracy and is competitive with 8-shot CoOp and Tip-Adapter on most datasets.UPL also surpasses 2-shot CoOp; UPL* is competitive on ImageNet, Caltech101, EuroSAT, Food101, OxfordPets, SUN397, and StandfordCars.Source: Main Results · p. 6

Ablation Study

  • UPL is less sensitive to prompt representation length L on Caltech101, DTD, and StandfordCars.Source: Table 4 · p. 6
  • On UCF101, top-K pseudo-labeling ensures a balanced pseudo-labeled distribution and prevents categories with many samples from overwhelming training; as K increases from 2, 4, 8 to 16, performance increases, so K = 16.A threshold of 0.9 causes an imbalanced pseudo-labeled distribution, while 0.3 introduces too much noise and disturbs training.Source: Ablation Study · p. 6, Figure 6 · p. 6, Table 2 · p. 6
  • Prompt ensembling nearly saturates at N = 16, while individually learned prompts have similar overall accuracy but significantly different per-class accuracy.On UCF101, PR-1, PR-2, and PR-3 score 64.58, 64.66, and 64.39, respectively; their differing class preferences motivate ensembling.Source: Ablation Study · p. 7, Figure 7 · p. 7
  • On UCF101, transfer improvement has no obvious correspondence with per-class pseudo-label accuracy, and UPL improves some classes despite low pseudo-label accuracy.Because all classes share one prompt representation, samples across classes jointly optimize it, allowing qualified pseudo-labeled samples to offset noise.Source: Ablation Study · p. 7, Figure 8 · p. 7

Conclusion

UPL avoids prompt engineering through unsupervised adaptation of pre-trained vision-language models and improves transfer performance across image-classification datasets. It outperforms prompt-engineered CLIP and compares favorably with few-shot approaches.

  • UPL outperforms prompt-engineered original CLIP on ImageNet and 10 other datasets and beats 2-shot CoOp in average accuracy across 11 datasets.Source: Conclusion · p. 7
  • An improved UPL version is competitive with 8-shot CoOp and 8-shot TIP-Adapter on most datasets.Source: Conclusion · p. 7
  • UPL is presented as the first unsupervised framework for adapting pre-trained vision-language models to downstream image recognition, avoiding time-consuming prompt engineering.Source: Conclusion · p. 7
  • Experiments evaluate UPL on ImageNet and 10 widely used image-classification datasets.Source: Conclusion · p. 7

More Experiments

Additional experiments show that UPL transfers across prompt-learning frameworks, outperforms original CLIP (Radford et al. 2021) with one or multiple prompts, and is robust to [CLS] token position.

Generalization of UPL

Position of the [CLS] Token

Comparison with CLIP of Single Prompt

Visualization Results

UPL’s optimized prompt representations have no obvious correspondence to vocabulary subwords, though some nearest subwords correlate with their datasets. Prediction visualizations show that low-confidence samples can still be correctly predicted, especially for fine-grained FGVCAircraft.

Nearest Words of Prompt Representations

  • UPL’s nearest subwords show no obvious overall correspondence to optimized prompts, but some correlate strongly with datasets, including “pie” for Food101 and “cat” for OxfordPets.The authors search vocabulary words by Euclidean distance to prompts composed of 16 learnable vectors; CLIP uses BPE tokenization.Source: Nearest Words of Prompt Representations · p. 10

Visualization of Predictions

  • On DTD and FGVCAircraft, visualizations show that low-confidence samples may still be correctly predicted, a phenomenon more evident on fine-grained FGVCAircraft.This motivates the authors’ advocacy for top-K pseudo-labeling because confidence alone does not fully reflect pseudo-label quality.Source: Visualization of Predictions · p. 10

Dataset Details and Hand-Craft Prompts

This section documents dataset-specific settings for pseudo-label generation and emphasizes using the simplest hand-crafted prompt rather than complex prompt engineering.

Dataset Details and Hand-Craft Prompts

Related papers

Loading 2204.03649v2…