Source-linked AI summary
Self-regulating Prompts: Foundational Model Adaptation without Forgetting
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, Fahad Shahbaz Khan
TL;DR
Prompt learning efficiently adapts foundational models but can overfit downstream objectives and reduce CLIP’s generalization. PromptSRC addresses this with mutual agreement to frozen features, Gaussian-weighted prompt self-ensembling, and textual diversity. Across four benchmarks, it reports favorable performance while retaining pre-trained CLIP generalization.
Problem
Task-specific prompt optimization can overfit downstream distributions and compromise CLIP’s original generalization to new tasks.
Method
PromptSRC jointly regularizes task-specific and task-agnostic representations through mutual agreement, Gaussian-weighted prompt self-ensembling, and diverse text templates.
Results
+1.42% harmonic-mean gain over MaPLe and +8.26% over CLIP is reported across 11 base-to-novel generalization datasets, with competitive results on other benchmarks.
Takeaways & Limitations
PromptSRC steers prompts toward downstream performance without compromising pre-trained CLIP generalization.
Abstract
from arXiv · showhide
Prompt learning has emerged as an efficient alternative for fine-tuning foundational models, such as CLIP, for various downstream tasks. Conventionally trained using the task-specific objective, i.e., cross-entropy loss, prompts tend to overfit downstream data distributions and find it challenging to capture task-agnostic general features from the frozen CLIP. This leads to the loss of the model's original generalization capability. To address this issue, our work introduces a self-regularization framework for prompting called PromptSRC (Prompting with Self-regulating Constraints). PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations using a three-pronged approach by: (a) regulating prompted representations via mutual agreement maximization with the frozen model, (b) regulating with self-ensemble of prompts over the training trajectory to encode their complementary strengths, and (c) regulating with textual diversity to mitigate sample diversity imbalance with the visual branch. To the best of our knowledge, this is the first regularization framework for prompt learning that avoids overfitting by jointly attending to pre-trained model features, the training trajectory during prompting, and the textual diversity. PromptSRC explicitly steers the prompts to learn a representation space that maximizes performance on downstream tasks without compromising CLIP generalization. We perform extensive experiments on 4 benchmarks where PromptSRC overall performs favorably well compared to the existing methods. Our code and pre-trained models are publicly available at: https://github.com/muzairkhattak/PromptSRC.
1. Introduction
PromptSRC addresses prompt overfitting by jointly preserving task-specific adaptation and frozen CLIP’s task-agnostic knowledge. It combines mutual agreement, prompt self-ensembling, and textual diversity, improving generalization across evaluated benchmarks.
- Prompt learning can overfit task-specific data, causing prompted models to lose frozen CLIP’s original generalization capability.
- PromptSRC regulates prompts through mutual agreement with frozen vision-language features while adapting them to downstream tasks.
- Prompt self-ensembling aggregates complementary prompt knowledge learned at different training epochs using weighted aggregation.
- +1.42% harmonic-mean gain over MaPLe and +8.26% over CLIP is achieved across 11 base-to-novel generalization datasets.
- PromptSRC is evaluated on four diverse image-recognition benchmarks and reports competitive results across base-to-novel, cross-dataset, domain-generalization, and few-shot settings.
- Textual diversity uses multiple text augmentations and templates to address the diversity mismatch between textual labels and visual samples.
2. Related Work
Related work establishes vision-language models as broadly transferable foundations and positions prompt learning as an efficient adaptation strategy. Existing methods address contextual diversity, visual conditioning, and multimodal prompting, while regularization methods generally constrain learning or modify inputs, parameters, or annotations.
- Vision-Language models: Vision-language models align image-text pairs in a shared embedding space using contrastive pre-training on large web-scale corpora.
- Vision-Language models: These models encode open-vocabulary concepts and support downstream applications including recognition, detection, segmentation, and other vision-language tasks.
- Prompt learning: Prompt learning adapts foundational models without full fine-tuning by learning prompt vectors while retaining pretrained representations for downstream tasks.
- Prompt learning: Prior approaches learn multiple prompt sets, condition prompts on visual features, or jointly learn hierarchical prompts across vision and language branches.
- Network regularization: Network regularization includes constraint-based techniques and strategies that modify inputs, model parameters, or annotations.
3. Proposed Method
PromptSRC addresses prompt overfitting by combining task-specific optimization with regularization that preserves general information from frozen CLIP. Its framework uses mutual agreement, prompt self-ensembling, and textual diversity to improve downstream adaptation without sacrificing generalization.
- Motivation: Prompt learning adapts frozen CLIP with learnable prompts, but supervised training can overfit downstream data and reduce generalization to new classes and datasets.Shortening training can balance base and novel performance, but lowers supervised-task performance.
- Results: PromptSRC’s consistency constraints improve base-class performance while also improving novel-class generalization as training proceeds.Figure 2 contrasts this behavior with naive supervised prompt training, whose generalization deteriorates as the schedule increases.
- Self-regularization framework: The framework self-ensembles prompts learned across epochs using Gaussian-weighted aggregation to combine complementary training-stage information.Early prompts receive less weight because they are immature, while late prompts receive less weight because they favor the supervised distribution.
- Prompt learning setup: The method uses jointly learned vision and language prompts with frozen CLIP encoders, producing prompted image and text features for downstream classification.Deep prompting adds separate prompt sets at transformer blocks, and prompts are optimized with cross-entropy on downstream data.
- Self-regularization framework: PromptSRC combines task-specific and task-agnostic learning by explicitly maximizing agreement between prompted and frozen CLIP features.The self-consistency loss operates on visual and textual features within the frozen CLIP latent space.
- Self-regularization framework: PromptSRC introduces textual diversity because each class has one textual label, unlike the multiple image samples available to the visual encoder.Textual augmentations produce diverse frozen vision-language text features that are averaged for regularization.
4. Experiments
PromptSRC is evaluated across base-to-novel generalization, few-shot learning, cross-dataset transfer, and domain generalization, with ablations examining its regularizers and efficiency. Across these settings, it improves generalization and task performance while using relatively few tunable parameters and modest training overhead.
- Self-regulating prompts: PromptSRC’s regularization components increase average novel-class performance by 4.31% and harmonic mean by 2.46% over the IVLP baseline.Mutual agreement, Gaussian prompt aggregation, and textual diversity are added progressively across the ablation.
- Base-to-novel generalization: PromptSRC improves zero-shot CLIP novel-class accuracy by 1.88% while also significantly improving base-class performance.The result supports retaining task-agnostic features while adapting to base classes.
- Base-to-novel generalization: PromptSRC achieves averaged base, novel, and harmonic-mean accuracies of 84.26%, 76.10%, and 79.97%, respectively, across 11 datasets.It improves performance over MaPLe on 8 of 11 datasets while using 46K rather than 3.55M tunable parameters.
- Few-shot experiments: Against MaPLe, PromptSRC gains 3.05%, 2.72%, 2.59%, 1.80%, and 1.07% at 1, 2, 4, 8, and 16 shots, respectively.The largest gains occur in the minimal-data settings, while improvements remain consistent across all shot counts.
- Cross-dataset evaluation: PromptSRC achieves better cross-dataset generalization than CoOp and CoCoOp on 8/10 and 7/10 datasets, and improves over MaPLe on 5/10 datasets.It uses 46K tunable parameters versus MaPLe’s 3.55M.
- Domain generalization: PromptSRC reaches the highest average accuracy of 60.65% on out-of-distribution target datasets under domain shifts.The model is trained on ImageNet and evaluated directly on shifted target datasets.
5. Conclusion
PromptSRC addresses prompt overfitting by self-regulating prompts to preserve CLIP’s generalization while adapting to downstream tasks. Its framework combines consistency constraints, Gaussian-weighted prompt aggregation, and textual diversity, with benefits shown across multiple benchmarks.
- PromptSRC explicitly encourages agreement between prompted representations and frozen-model representations through self-consistency constraints.
- Gaussian-weighted self-ensembling aggregates prompts learned across training epochs to capture complementary features and improve generalization.
- Textual diversity supplements mutual-agreement regulation to address prompt overfitting and improve generalization.
- Extensive evaluations on multiple benchmarks show benefits from the self-regulating prompt-learning approach.
Adaptation without Forgetting
The supplemental section provides implementation details, additional comparisons, and ablative analyses of PromptSRC. It organizes these materials into three appendices.
- Appendix A contains additional implementation details for PromptSRC.
- Appendix B contains additional results comparisons.
- Appendix C contains additional ablative analyses.
A. Additional Implementation details
PromptSRC uses fixed training settings across benchmarks, Gaussian Weighted Prompt Aggregation, diverse text templates, and several evaluation protocols. The implementation reports harmonic mean as the main base-to-novel generalization metric.
- PromptSRC uses ViT-B/16 CLIP with d = 512, learning rate 0.0025, SGD, and a single NVIDIA A100 GPU.Training lasts 50 epochs for few-shot settings and 20 epochs for the other three benchmark settings.
- Gaussian Weighted Prompt Aggregation samples Gaussian weights to aggregate prompt knowledge learned at different training epochs.The supplied implementation passage introduces GPA but does not provide its remaining procedural details.
- The textual-diversity component randomly selects 60 prompt templates from the complete template list.Examples include templates describing photos, sculptures, graffiti, renderings, cartoons, sketches, and pixelated photos of a category.
- Base-to-novel evaluation reports top-1 base-class and novel-class accuracy together with their harmonic mean, the main generalization metric.
- Few-shot evaluation reports top-1 test-set accuracies for K = 1, 2, 4, 8, and 16 shots using CoOp-provided splits.
- Cross-dataset and domain-generalization experiments likewise report top-1 test-set accuracies, with an algorithm describing PromptSRC’s implementation.
B. Additional results comparison
Additional experiments examine PromptSRC across video understanding, alternate textual-diversity designs, recent vision-language models, and component ablations. These comparisons report consistent generalization improvements in the supplied results.
- Generalization across tasks: PromptSRC is evaluated on a video action-recognition generalization benchmark using Kinetics-400-pre-trained ViFi-CLIP.
- PromptSRC procedure: Algorithm 1 summarizes learning self-regulating prompts from prompted and pre-trained visual-textual features.
- Textual diversity: Ensembling multiple text templates achieves better generalization than alternate textual-diversity designs.
- Additional results: PromptSRC consistently improves generalization on 10/11 datasets and gains +2.09% absolute average HM over IVLP on EVA CLIP.
- Component analysis: The complete regularization framework improves performance over naive IVLP prompt learning in the base-to-novel setting.
- GPA analysis: Figure 6 analyzes GPA hyper-parameters on ImageNet.
C. Additional ablation study
Additional ablations examine alternate textual-diversity construction and GPA hyperparameters, alongside per-dataset comparisons of PromptSRC components and few-shot methods.
- Textual diversity is alternatively tested by randomly selecting one of N prompt templates instead of ensembling frozen CLIP textual features, with results averaged over 11 datasets.
- Varying GPA’s σ2 has a minute effect on ImageNet performance, whereas increasing µ gives greater weight to prompts learned in later epochs.The reported increase in µ improves base-class performance slightly.
- Table 12 compares individual PromptSRC components and reports absolute gains over IVLP on individual datasets.
- Table 13 compares PromptSRC with various methods on individual datasets in the few-shot setting.