PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-regulating Prompts: Foundational Model Adaptation without Forgetting

Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, Fahad Shahbaz Khan

arXiv:2307.06948v2cs.CV

TL;DR

Prompt learning efficiently adapts foundational models but can overfit downstream objectives and reduce CLIP’s generalization. PromptSRC addresses this with mutual agreement to frozen features, Gaussian-weighted prompt self-ensembling, and textual diversity. Across four benchmarks, it reports favorable performance while retaining pre-trained CLIP generalization.

  • Problem

    Task-specific prompt optimization can overfit downstream distributions and compromise CLIP’s original generalization to new tasks.

    Source: §1 · p. 1, §3 · p. 3, §3.2 · p. 4

  • Method

    PromptSRC jointly regularizes task-specific and task-agnostic representations through mutual agreement, Gaussian-weighted prompt self-ensembling, and diverse text templates.

    Source: §1 · p. 1

  • Results

    +1.42% harmonic-mean gain over MaPLe and +8.26% over CLIP is reported across 11 base-to-novel generalization datasets, with competitive results on other benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PromptSRC steers prompts toward downstream performance without compromising pre-trained CLIP generalization.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Prompt learning has emerged as an efficient alternative for fine-tuning foundational models, such as CLIP, for various downstream tasks. Conventionally trained using the task-specific objective, i.e., cross-entropy loss, prompts tend to overfit downstream data distributions and find it challenging to capture task-agnostic general features from the frozen CLIP. This leads to the loss of the model's original generalization capability. To address this issue, our work introduces a self-regularization framework for prompting called PromptSRC (Prompting with Self-regulating Constraints). PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations using a three-pronged approach by: (a) regulating prompted representations via mutual agreement maximization with the frozen model, (b) regulating with self-ensemble of prompts over the training trajectory to encode their complementary strengths, and (c) regulating with textual diversity to mitigate sample diversity imbalance with the visual branch. To the best of our knowledge, this is the first regularization framework for prompt learning that avoids overfitting by jointly attending to pre-trained model features, the training trajectory during prompting, and the textual diversity. PromptSRC explicitly steers the prompts to learn a representation space that maximizes performance on downstream tasks without compromising CLIP generalization. We perform extensive experiments on 4 benchmarks where PromptSRC overall performs favorably well compared to the existing methods. Our code and pre-trained models are publicly available at: https://github.com/muzairkhattak/PromptSRC.

1. Introduction

PromptSRC addresses prompt overfitting by jointly preserving task-specific adaptation and frozen CLIP’s task-agnostic knowledge. It combines mutual agreement, prompt self-ensembling, and textual diversity, improving generalization across evaluated benchmarks.

  • Prompt learning can overfit task-specific data, causing prompted models to lose frozen CLIP’s original generalization capability.Source: §1 · p. 1
  • PromptSRC regulates prompts through mutual agreement with frozen vision-language features while adapting them to downstream tasks.Source: §1 · pp. 1–2
  • Prompt self-ensembling aggregates complementary prompt knowledge learned at different training epochs using weighted aggregation.Source: §1 · pp. 1–2
  • +1.42% harmonic-mean gain over MaPLe and +8.26% over CLIP is achieved across 11 base-to-novel generalization datasets.Source: §1 · p. 2
  • PromptSRC is evaluated on four diverse image-recognition benchmarks and reports competitive results across base-to-novel, cross-dataset, domain-generalization, and few-shot settings.Source: §1 · p. 2, Figure 1 · p. 2
  • Textual diversity uses multiple text augmentations and templates to address the diversity mismatch between textual labels and visual samples.Source: §1 · p. 2

2. Related Work

Related work establishes vision-language models as broadly transferable foundations and positions prompt learning as an efficient adaptation strategy. Existing methods address contextual diversity, visual conditioning, and multimodal prompting, while regularization methods generally constrain learning or modify inputs, parameters, or annotations.

Vision-Language models

  • Vision-language models align image-text pairs in a shared embedding space using contrastive pre-training on large web-scale corpora.Source: §2 · p. 2
  • These models encode open-vocabulary concepts and support downstream applications including recognition, detection, segmentation, and other vision-language tasks.Source: §2 · p. 2

Prompt learning

  • Prompt learning adapts foundational models without full fine-tuning by learning prompt vectors while retaining pretrained representations for downstream tasks.Source: §2 · pp. 2–3
  • Prior approaches learn multiple prompt sets, condition prompts on visual features, or jointly learn hierarchical prompts across vision and language branches.Source: §2 · p. 3

Network regularization

  • Network regularization includes constraint-based techniques and strategies that modify inputs, model parameters, or annotations.Source: §2 · p. 3

3. Proposed Method

PromptSRC addresses prompt overfitting by combining task-specific optimization with regularization that preserves general information from frozen CLIP. Its framework uses mutual agreement, prompt self-ensembling, and textual diversity to improve downstream adaptation without sacrificing generalization.

Motivation

  • Prompt learning adapts frozen CLIP with learnable prompts, but supervised training can overfit downstream data and reduce generalization to new classes and datasets.Shortening training can balance base and novel performance, but lowers supervised-task performance.Source: §3 · p. 3, §3.2 · pp. 4–5

Results

  • PromptSRC’s consistency constraints improve base-class performance while also improving novel-class generalization as training proceeds.Figure 2 contrasts this behavior with naive supervised prompt training, whose generalization deteriorates as the schedule increases.Source: Figure 2 · p. 3, §3.2.1 · p. 5

Self-regularization framework

  • The framework self-ensembles prompts learned across epochs using Gaussian-weighted aggregation to combine complementary training-stage information.Early prompts receive less weight because they are immature, while late prompts receive less weight because they favor the supervised distribution.Source: §3 · p. 3

Prompt learning setup

  • The method uses jointly learned vision and language prompts with frozen CLIP encoders, producing prompted image and text features for downstream classification.Deep prompting adds separate prompt sets at transformer blocks, and prompts are optimized with cross-entropy on downstream data.Source: §3.1 · pp. 3–4

Self-regularization framework

  • PromptSRC combines task-specific and task-agnostic learning by explicitly maximizing agreement between prompted and frozen CLIP features.The self-consistency loss operates on visual and textual features within the frozen CLIP latent space.Source: §3 · p. 3, §3.2.1 · p. 5
  • PromptSRC introduces textual diversity because each class has one textual label, unlike the multiple image samples available to the visual encoder.Textual augmentations produce diverse frozen vision-language text features that are averaged for regularization.Source: Figure 3 · p. 4, §3.2.3 · p. 6

4. Experiments

PromptSRC is evaluated across base-to-novel generalization, few-shot learning, cross-dataset transfer, and domain generalization, with ablations examining its regularizers and efficiency. Across these settings, it improves generalization and task performance while using relatively few tunable parameters and modest training overhead.

Self-regulating prompts

  • PromptSRC’s regularization components increase average novel-class performance by 4.31% and harmonic mean by 2.46% over the IVLP baseline.Mutual agreement, Gaussian prompt aggregation, and textual diversity are added progressively across the ablation.Source: §4.2 · p. 7

Base-to-novel generalization

  • PromptSRC improves zero-shot CLIP novel-class accuracy by 1.88% while also significantly improving base-class performance.The result supports retaining task-agnostic features while adapting to base classes.Source: §4.3 · p. 7
  • PromptSRC achieves averaged base, novel, and harmonic-mean accuracies of 84.26%, 76.10%, and 79.97%, respectively, across 11 datasets.It improves performance over MaPLe on 8 of 11 datasets while using 46K rather than 3.55M tunable parameters.Source: §4.3 · p. 7

Few-shot experiments

  • Against MaPLe, PromptSRC gains 3.05%, 2.72%, 2.59%, 1.80%, and 1.07% at 1, 2, 4, 8, and 16 shots, respectively.The largest gains occur in the minimal-data settings, while improvements remain consistent across all shot counts.Source: §4.4 · p. 7

Cross-dataset evaluation

  • PromptSRC achieves better cross-dataset generalization than CoOp and CoCoOp on 8/10 and 7/10 datasets, and improves over MaPLe on 5/10 datasets.It uses 46K tunable parameters versus MaPLe’s 3.55M.Source: §4.5 · p. 8

Domain generalization

  • PromptSRC reaches the highest average accuracy of 60.65% on out-of-distribution target datasets under domain shifts.The model is trained on ImageNet and evaluated directly on shifted target datasets.Source: §4.6 · p. 8, Table 4 · p. 8

5. Conclusion

PromptSRC addresses prompt overfitting by self-regulating prompts to preserve CLIP’s generalization while adapting to downstream tasks. Its framework combines consistency constraints, Gaussian-weighted prompt aggregation, and textual diversity, with benefits shown across multiple benchmarks.

  • PromptSRC explicitly encourages agreement between prompted representations and frozen-model representations through self-consistency constraints.Source: §5 · p. 9
  • Gaussian-weighted self-ensembling aggregates prompts learned across training epochs to capture complementary features and improve generalization.Source: §5 · p. 9
  • Textual diversity supplements mutual-agreement regulation to address prompt overfitting and improve generalization.Source: §5 · p. 9
  • Extensive evaluations on multiple benchmarks show benefits from the self-regulating prompt-learning approach.Source: §5 · p. 9

Adaptation without Forgetting

The supplemental section provides implementation details, additional comparisons, and ablative analyses of PromptSRC. It organizes these materials into three appendices.

A. Additional Implementation details

PromptSRC uses fixed training settings across benchmarks, Gaussian Weighted Prompt Aggregation, diverse text templates, and several evaluation protocols. The implementation reports harmonic mean as the main base-to-novel generalization metric.

B. Additional results comparison

Additional experiments examine PromptSRC across video understanding, alternate textual-diversity designs, recent vision-language models, and component ablations. These comparisons report consistent generalization improvements in the supplied results.

Generalization across tasks

PromptSRC procedure

Textual diversity

  • Ensembling multiple text templates achieves better generalization than alternate textual-diversity designs.Source: Table 10 · p. 13

Additional results

Component analysis

GPA analysis

C. Additional ablation study

Additional ablations examine alternate textual-diversity construction and GPA hyperparameters, alongside per-dataset comparisons of PromptSRC components and few-shot methods.

  • Textual diversity is alternatively tested by randomly selecting one of N prompt templates instead of ensembling frozen CLIP textual features, with results averaged over 11 datasets.Source: C. Additional ablation study · p. 14
  • Varying GPA’s σ2 has a minute effect on ImageNet performance, whereas increasing µ gives greater weight to prompts learned in later epochs.The reported increase in µ improves base-class performance slightly.Source: C. Additional ablation study · p. 14
  • Table 12 compares individual PromptSRC components and reports absolute gains over IVLP on individual datasets.Source: Table 12 · p. 15
  • Table 13 compares PromptSRC with various methods on individual datasets in the few-shot setting.Source: Table 13 · p. 16

Related papers

Loading 2307.06948v2…