PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Vision and Language Prompt Learning

Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, Chen Change Loy

arXiv:2210.07225v1cs.CVcs.AI

TL;DR

The paper studies why text-only and visual-only prompt tuning do not improve consistently across downstream vision datasets. It introduces Unified Prompt Tuning, which jointly tunes both modalities through a lightweight network, and reports a better trade-off than unimodal methods across few-shot learning and domain generalization. UPT is promising but not uniformly optimal, since alternative designs can sometimes perform better.

  • Problem

    Unimodal prompt tuning does not consistently improve over zero-shot CLIP because downstream tasks vary in visual-feature and text-embedding variances.

  • Method

    UPT learns unified modality-agnostic prompts with a lightweight self-attention network for jointly tuning the text and visual encoders.

  • Results

    UPT achieves a better trade-off than unimodal prompt tuning across few-shot learning and domain generalization benchmarks.

  • Takeaways & Limitations

    Prompt learning should exploit correspondences between visual and text modalities rather than relying only on isolated single-modal prompts.

  • Takeaways & Limitations

    UPT is not uniformly optimal: ablations show that replacing its Transformer with an MLP can sometimes perform better.

Abstract

from arXiv · show

Prompt tuning, a parameter- and data-efficient transfer learning paradigm that tunes only a small number of parameters in a model's input space, has become a trend in the vision community since the emergence of large vision-language models like CLIP. We present a systematic study on two representative prompt tuning methods, namely text prompt tuning and visual prompt tuning. A major finding is that none of the unimodal prompt tuning methods performs consistently well: text prompt tuning fails on data with high intra-class visual variances while visual prompt tuning cannot handle low inter-class variances. To combine the best from both worlds, we propose a simple approach called Unified Prompt Tuning (UPT), which essentially learns a tiny neural network to jointly optimize prompts across different modalities. Extensive experiments on over 11 vision datasets show that UPT achieves a better trade-off than the unimodal counterparts on few-shot learning benchmarks, as well as on domain generalization benchmarks. Code and models will be released to facilitate future research.

1 INTRODUCTION

Large vision-language models are efficient to adapt with prompt tuning, but text-only and visual-only prompts respond differently to downstream feature variances. UPT jointly tunes both modalities to achieve a better trade-off across tasks.

  • Motivation: Prompt tuning freezes the vision-language model and learns only extra input prompts for efficient downstream adaptation.This avoids directly fine-tuning the full model, whose huge parameter size makes that approach often prohibitive.
  • Unimodal prompt tuning: Text prompt tuning modifies the text encoder, whereas visual prompt tuning modifies the visual encoder to address limitations of fixed prompts.These approaches correspond to the two encoder branches in a typical vision-language model.
  • Unimodal prompt tuning: Single-modal prompt tuning lacks consistent gains because downstream tasks differ in visual intra-class variance and text inter-class variance.The study measures these distribution statistics and relates them to prompt-tuning improvements.
  • Unimodal prompt tuning: CoOp beats VPT by 8.1% on Flowers102, while VPT beats CoOp by 8.4% on EuroSAT.The opposing dataset-level outcomes make the choice of prompt modality task-dependent.
  • Unified Prompt Tuning: UPT uses a shared initial prompt and a lightweight self-attention network to generate prompts for both CLIP encoders.The design jointly optimizes text and visual modalities rather than using isolated modality-specific prompts.

2 METHODOLOGY

The paper reviews text and visual prompt tuning for CLIP, analyzes when each modality-specific approach fails, and introduces Unified Prompt Tuning to optimize both modalities jointly. UPT uses shared prompts and a lightweight Transformer to tune text classifiers and visual features while keeping the encoders frozen.

  • 2.1 PRELIMINARIES: CLIP uses frozen image and text encoders to map inputs into a shared space, with text embeddings producing classifiers for image prediction.The image feature is z=fφ(x), while class text representations form W=fψ(A).
  • 2.1 PRELIMINARIES: Text prompt tuning learns prompt parameters for the text encoder while keeping both CLIP encoders frozen.CoOp replaces the handcrafted template with learnable prompts and optimizes them using a task-specific objective.
  • 2.1 PRELIMINARIES: Visual prompt tuning inserts learnable prompts into the visual encoder to modulate image features without changing the encoder parameters.VPT-shallow inserts prompts in the first Transformer layer, whereas VPT-deep inserts them at every layer; the prompts are data-independent after learning.
  • 2.2 ANALYSIS: CoOp performs poorly with high intra-class visual variance, while VPT is ineffective when inter-class text embeddings have low variance.EuroSAT illustrates the former failure and Flowers102 illustrates the latter: CoOp barely changes text classifiers on EuroSAT, while VPT does not substantially modulate visual features on Flowers102.
  • 2.3 UNIFIED PROMPT TUNING: UPT is presented as a unified strategy that outperforms single-modal prompt tuning particularly in few-shot learning and domain generalization.The method is motivated by the difficulty of obtaining consistent improvements from CoOp and VPT across downstream tasks.
  • 2.3 UNIFIED PROMPT TUNING: UPT learns shared modality-agnostic prompts and transforms them with a lightweight self-attention network before sending separate prompt parts to the text and visual encoders.The encoders remain frozen; only the unified prompts and lightweight Transformer parameters are optimized, jointly tuning classifiers W and visual features z.

3 EXPERIMENTS

Experiments evaluate prompt-tuning strategies on few-shot classification, domain generalization, and multimodal design ablations. Across these settings, UPT generally improves over single-modal approaches, while results expose dataset- and design-dependent trade-offs.

  • Few-shot learning: Few-shot experiments use 11 datasets with 1/2/4/8/16 training shots, evaluating average accuracy over three runs against Zero-shot CLIP, CoOp, and VPT baselines.The protocol uses the whole test set for evaluation.
  • Few-shot learning: CoOp and VPT vary substantially across datasets: CoOp is stronger on StanfordCars and SUN397, whereas VPT-deep beats CoOp by over 12% on EuroSAT.The results also verify that VPT-deep outperforms VPT-shallow in this setting.
  • Few-shot learning: UPT improves over CoOp by 0.48/1.36/1.29/2.46/3.19% and over VPT-deep by 0.89/2.70/2.03/2.40/2.01% at 1/2/4/8/16 shots, respectively.UPT also surpasses both baselines on large and fine-grained datasets, with smaller gains on OxfordPets and Food101.
  • Domain generalization: Domain-generalization evaluation trains on ImageNet and tests on four shifted ImageNet variants, reporting source-target and target-only average accuracy.The protocol uses 16 shots and evaluates ImageNet V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.
  • Domain generalization: UPT is generally better than single-modal baselines, comparable with CoCoOp, and best on four of five target datasets in domain generalization.VPT-shallow exceeds VPT-deep on OOD accuracy, while text prompt tuning methods outperform visual prompt tuning approaches.
  • Ablation studies: Ablations show that joint CoOp-VPT training reaches 78.70% versus 79.39% for VPT-deep, direct prompt sharing performs worst, and a two-layer MLP remains below self-attention UPT on average.The MLP baseline nevertheless achieves the best performance on two datasets.
  • Qualitative results: Visualizations show stronger self-attention responses for UPT than VPT, a possible explanation for UPT's performance on few-shot and OOD generalization settings.The comparison uses self-attention response maps between visual prompts and image patch tokens.

4 RELATED WORK

Vision-language models use separate image and text encoders, motivating distinct text and visual prompt-tuning approaches for efficient downstream adaptation. Prior work includes learnable text prompts for CLIP, visual input tokens for Vision Transformers, and methods such as neural architecture search for prompt configuration.

  • Vision-Language Models: Vision-language models align image and text encoders in a common feature space using web-scale image-text pretraining.They transfer to tasks including point-cloud and video classification, object detection, semantic segmentation, few-shot learning, and open-vocabulary perception.
  • Text Prompt Tuning: Text prompt tuning adapts CLIP by replacing input context tokens in its text branch with learnable vectors.CoOp is a representative method, with follow-ups including CoCoOp, DualCoOp, ProGrad, and ProDA.
  • Visual Prompt Tuning: Visual prompt tuning adapts pretrained Vision Transformers by adding trainable tokens to the visual input space.VPT and Visual Prompting add tokens to Transformer inputs, while NOAH uses neural architecture search to identify prompt configurations.

5 CONCLUSION

The paper argues that efficient adaptation of large vision-language models should examine and exploit correspondences between modalities. UPT offers a better trade-off than unimodal prompt tuning, although ablations show that alternative designs can sometimes perform better.

  • 5 CONCLUSION: Efficient downstream adaptation is increasingly important as vision models scale and are deployed more broadly.The paper studies this challenge through prompt learning for large vision-language models such as CLIP.
  • 5 CONCLUSION: UPT achieves a better trade-off than unimodal prompt-tuning methods across a wide range of vision datasets.The reported settings include few-shot learning and domain generalization.
  • 5 CONCLUSION: Ablation studies find that alternative designs, including an MLP instead of a Transformer, can sometimes outperform UPT.The authors present multimodal prompt learning as a framework that may improve with more advanced and efficient designs.

A INTRA-/INTER- CLASS VARIANCE

The appendix defines intra-class visual variance from CLIP image features and inter-class text variance from CLIP class embeddings. These quantities characterize visual dispersion within classes and text-feature dispersion across classes.

  • Intra-class Visual Variance: Intra-class visual variance is computed from CLIP image features for images grouped by ground-truth class.For each class, the method uses the class image set and its mean feature before aggregating across all classes.
  • Inter-class Text Variance: Inter-class text variance is computed from CLIP text features for the dataset’s class labels.The calculation first obtains each class feature and the mean feature across the k classes, then computes the variance.

B IMPLEMENTATION DETAILS

The experiments use a ViT-B/16 CLIP backbone and standardized prompt, optimization, and batching settings across datasets.

  • B IMPLEMENTATION DETAILS: The implementation uses ViT-B/16 as the CLIP backbone and context length m = 4 for CoOp and VPT.Zero-shot CLIP and VPT use the default template “a photo of a [CLS].”
  • B IMPLEMENTATION DETAILS: SGD starts at learning rate 0.002 with cosine annealing, and all datasets use batch size 32.
Loading 2210.07225v1…