Source-linked AI summary
Prompt Distribution Learning
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, Xinmei Tian
TL;DR
Downstream recognition with pre-trained VLMs needs task-related prompts, but manual design can be biased and a single prompt cannot capture diverse visual representations. ProDA learns a distribution of diverse, low-bias prompts from few samples by modeling their output embeddings with a Gaussian and training with a surrogate objective. Across 12 datasets, it consistently outperforms existing methods, including a 9.1% relative average improvement over human-crafted prompts in the 1-shot setting.
Problem
Prompt design for downstream VLM recognition is difficult because manual prompts can be biased, while single prompts neglect diverse visual representations.
Method
ProDA learns a distribution of diverse soft prompts from downstream data by modeling prompt output embeddings with a multivariate Gaussian and optimizing a surrogate objective.
Results
ProDA consistently and significantly outperforms existing baselines across 12 datasets, improving the average result by 9.1% relative to human-crafted prompts with 1 sample per category.
Takeaways & Limitations
Learning low-bias prompt distributions provides task-related content that captures visual variation for adapting pre-trained VLMs to downstream recognition.
Takeaways & Limitations
The proposed prompt distribution learning method focuses on object and image recognition and cannot currently be applied to tasks such as detection or semantic segmentation.
Abstract
from arXiv · showhide
We present prompt distribution learning for effectively adapting a pre-trained vision-language model to address downstream recognition tasks. Our method not only learns low-bias prompts from a few samples but also captures the distribution of diverse prompts to handle the varying visual representations. In this way, we provide high-quality task-related content for facilitating recognition. This prompt distribution learning is realized by an efficient approach that learns the output embeddings of prompts instead of the input embeddings. Thus, we can employ a Gaussian distribution to model them effectively and derive a surrogate loss for efficient training. Extensive experiments on 12 datasets demonstrate that our method consistently and significantly outperforms existing methods. For example, with 1 sample per category, it relatively improves the average result by 9.1% compared to human-crafted prompts.
1. Introduction
Pre-trained vision-language models support downstream recognition, but performance depends on task-appropriate prompts that are difficult to design manually and must also represent diverse visual content. ProDA learns low-bias, diverse prompt distributions from few samples and trains efficiently through output embeddings, improving average performance across 12 datasets.
- Motivation: VLM recognition performance is highly sensitive to the form of the provided task-related content.Aligned image-text embeddings let VLMs use category descriptions for downstream recognition, including zero-shot inference.
- Motivation: Manual prompt design can introduce artificial bias, require repetitive expert attempts, and need a large validation set for selection.Task-specific wording can improve fine-grained recognition, but suitable prompts are difficult to determine automatically or manually.
- Motivation: Intra-class variation from pose, deformation, and lighting makes diverse prompts desirable for representing visual content.Existing practice ensembles 80 hand-crafted prompts on ImageNet, including prompts describing different object sizes.
- ProDA: ProDA automatically learns low-bias soft prompts from a few downstream samples and estimates a distribution over diverse prompts rather than one prompt.It also differentiates prompts in construction and semantics to improve diversity and generalization to various and unknown samples.
- ProDA: ProDA learns distributions of prompt output embeddings, whose within-category representations are adjacent and can be modeled with a multivariate Gaussian.This replaces direct modeling of scattered raw or input embeddings with a simpler distributional representation.
- Results: 9.1% relative average improvement is reported over human-crafted prompts with 1 sample per category across 12 downstream datasets.The experiments report consistent and significant improvement over existing baselines.
2. Related Work
Related work establishes vision-language pre-training as a transferable recognition framework and traces prompt learning from manually designed templates to automatically learned continuous prompts. In computer vision, CoOp learns a single soft prompt, whereas ProDA addresses the neglected diversity of visual representations by learning prompt distributions.
- Vision-Language Pre-Trained Models: Contrastive VLMs such as CLIP and ALIGN learn aligned image-text representations from web-scale noisy pairs and transfer them to downstream tasks.Their contrastive loss brings matching image-text pairs closer and separates mismatching pairs.
- Prompt Learning: Prompt learning formalizes tasks through prompt templates, evolving from human-crafted prompts to automatically generated discrete prompts and learned soft prompts.Discovering an appropriate prompt is central because manual design is sensitive and difficult.
- Prompt Learning: In computer vision, hand-crafted prompts support zero-shot VLM inference, while CoOp learns a soft prompt by minimizing target-task classification loss.Both approaches use prompts for downstream recognition, but they differ in whether prompts are manually designed or data-driven.
- Prompt Learning: CoOp’s single-prompt formulation neglects the diversity of visual representations that ProDA explicitly models with a prompt distribution.This distinction motivates distribution learning for downstream visual recognition.
3. Method
ProDA adapts a pre-trained VLM by learning a distribution of classifier weights induced by diverse prompts, rather than a single prompt. It models output embeddings with a Gaussian distribution and optimizes a surrogate loss for efficient training.
- Prompt Tuning: Prompt tuning learns continuous prompts from a few target-task samples to generate classifier weights while keeping the pre-trained VLM parameters frozen.The prompt is concatenated with each class name, and the resulting text embeddings form the target classifier weights.
- Optimization: The desired prompt distribution minimizes marginal classification loss, but exact multi-class marginal-likelihood computation remains intractable even under the Gaussian assumption.ProDA therefore derives an upper bound that can be optimized efficiently.
- Learning the Prompt Distribution: ProDA estimates a distribution over classifier weights generated by a collection of learnable prompts, indirectly modeling the prompt distribution.The weights are the output embeddings of category descriptions, and their mean and covariance are estimated from multiple prompt-induced classifiers.
- Learning the Prompt Distribution: A multivariate Gaussian models the classifier-weight distribution because high-level embeddings of diverse descriptions within a category are usually adjacent.The original prompt distribution may be complex, but the corresponding output embeddings are treated as more tractable to model.
- Optimization: Minimizing the surrogate upper-bound loss trains the prompt collection used to estimate the weight distribution for test-sample classification.At inference, ProDA predicts with the expected class probability under the learned weight distribution, using numerical approximations when needed.
- Improving Prompt Diversity: ProDA further improves prompt diversity by explicitly encouraging the learnable prompts to have dissimilar semantics.The method uses semantic embeddings and a semantic orthogonality loss for this purpose.
4. Experiments
ProDA is evaluated against prompt-based and linear-probe baselines across 12 downstream classification datasets under few-shot transfer, with ablations examining its distribution-learning and diversity components. It consistently improves recognition, while larger prompt collections help estimate prompt distributions more precisely and mini-batch sampling preserves performance with lower memory use.
- Evaluation Setup: ProDA is evaluated on 12 downstream classification datasets spanning general, fine-grained, remote-sensing, and texture recognition.The evaluation uses 1, 2, 4, 8, and 16 labeled samples per class, with results averaged over 3 runs.
- Main Results: 9.1% relative average improvement over hand-crafted prompts is achieved with 1 training sample per class, increasing to 25% with 16 samples.In the 1-shot setting, improvements reach 40% on EuroSAT and 25% on DTD.
- Main Results: 8.5% relative average improvement over CoOp is achieved in 1-shot evaluation, with a 4.3% improvement in 16-shot evaluation.The comparison reports consistent and significant outperformance of prompt tuning.
- Main Results: In 1-shot evaluation, ProDA reaches a 77% higher average score than linear-probe CLIP, with scores of 67.0% versus 37.8%.The comparison uses the same pre-trained CLIP model and indicates the benefit of category names in few-shot recognition.
- Ablation Study: Ablations show that learning the weight distribution outperforms aggregating classifiers, while positional and semantic diversity constraints improve average scores.The ablation study reports average scores across 12 downstream tasks at different training-sample counts.
- Ablation Study: More prompts improve CIFAR-100 recognition by providing diverse descriptions, while mini-batch prompt sampling achieves similar results to using all prompts with lower memory requirements.Using all prompts requires eight times the input-text size, whereas sampling reduces memory overhead.
5. Discussion and Conclusion
The paper presents prompt learning that models a distribution of diverse prompts for downstream visual recognition with a pre-trained VLM. It reports substantial improvement over the linear-probe baseline while remaining limited to object and image recognition.
- The approach demonstrates substantial improvement over the linear-probe baseline in few-shot learning.
- Current prompt distribution learning focuses on object and image recognition rather than detection, semantic segmentation, or image style transfer.The authors state that dedicated modifications would be needed for some other vision tasks.
A. Proof of Proposition 1
The proposition assumes concatenated prompt output embeddings follow a Gaussian distribution and defines covariance-derived quantities for prompt pairs. Under this assumption, the expected recognition loss is bounded by a surrogate upper loss involving prompt means and covariance terms.
- The proposition models concatenated prompt embeddings as a Gaussian distribution with mean µ_1:C(P^K) and covariance Σ_1:C(P^K).
- The proof derives an upper-bound loss L_upper(P^K) using class-prompt means and covariance terms in the softmax denominator.
- Jensen’s inequality supplies the stated inequalities, while the moment-generating function evaluates the Gaussian expectation.
B. Datasets
The evaluation uses 12 downstream datasets, with each dataset’s accuracy metric following the CLIP evaluation protocol.
- The study evaluates downstream performance on 12 datasets.
- Each dataset uses the accuracy metric defined by CLIP.
- Dataset details are provided in Table 3.
C. Baselines
The baseline setup selects Linear Probe CLIP regularization using validation-set hyperparameter sweeps, while CoOp uses a specified SGD training configuration.
- Linear Probe CLIP regularization is selected separately for each dataset using its validation set.
- The Linear Probe CLIP hyperparameter sweep follows the strategy used in CLIP.
- CoOp is trained with SGD at learning rate 0.001, batch size 20, cosine decay, 100 epochs, and prompt length 16.
D. Results
The experiments evaluate various methods using the same pre-trained CLIP RN50 model across 12 downstream datasets. Table 4 reports detailed performance, with results organized by training samples per class.
- Table 4 compares various methods using the same pre-trained CLIP model, RN50, on 12 downstream datasets.
- Table 4 reports detailed performance in percentages across the 12 downstream datasets.
- “# Shot” denotes the number of training samples per class.