PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt-aligned Gradient for Prompt Tuning

Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, Hanwang Zhang

arXiv:2205.14865v4cs.CV

TL;DR

Prompt tuning can overfit few-shot data, reducing generalization and potentially underperforming zero-shot predictions. ProGrad aligns prompt updates with general knowledge from the original prompt, and experiments across multiple settings report consistent improvements over existing methods.

  • Problem

    Prompt tuning can reduce generalization and even underperform zero-shot transfer, while existing remedies rely heavily on early stopping and data augmentation.

  • Method

    ProGrad regularizes prompt updates by retaining gradient components aligned with the original prompt’s general knowledge and discarding conflicting updates.

  • Results

    ProGrad improves over CoOp across all 11 evaluated datasets, improves base-to-new harmonic mean over CoOp and CoCoOp, and improves both source and target domain-generalization accuracy.

  • Takeaways & Limitations

    Experiments across few-shot classification, base-to-new generalization, domain generalization, and cross-dataset transfer demonstrate ProGrad’s effectiveness and efficiency.

  • Takeaways & Limitations

    Existing prompt-tuning transfer methods are not directly applicable because prompt tuning transfers only from general knowledge to downstream knowledge.

Abstract

from arXiv · show

Thanks to the large pre-trained vision-language models (VLMs) like CLIP, we can craft a zero-shot classifier by "prompt", e.g., the confidence score of an image being "[CLASS]" can be obtained by using the VLM provided similarity measure between the image and the prompt sentence "a photo of a [CLASS]". Therefore, prompt shows a great potential for fast adaptation of VLMs to downstream tasks if we fine-tune the prompt-based similarity measure. However, we find a common failure that improper fine-tuning may not only undermine the prompt's inherent prediction for the task-related classes, but also for other classes in the VLM vocabulary. Existing methods still address this problem by using traditional anti-overfitting techniques such as early stopping and data augmentation, which lack a principled solution specific to prompt. We present Prompt-aligned Gradient, dubbed ProGrad, to prevent prompt tuning from forgetting the the general knowledge learned from VLMs. In particular, ProGrad only updates the prompt whose gradient is aligned (or non-conflicting) to the "general direction", which is represented as the gradient of the KL loss of the pre-defined prompt prediction. Extensive experiments demonstrate the stronger few-shot generalization ability of ProGrad over state-of-the-art prompt tuning methods. Codes are available at https://github.com/BeierZhu/Prompt-align.

1. Introduction

Prompt tuning adapts CLIP to downstream classification but can forget general knowledge and underperform zero-shot CLIP with scarce data. ProGrad addresses this by retaining only prompt updates aligned with the general-knowledge direction, and experiments report improvements across few-shot and generalization settings.

  • Prompt-based adaptation: CLIP uses prompt-image similarity as a zero-shot confidence score for classifying images with natural-language class prompts.A prompt such as “a photo of a [CLASS]” connects image and language representations for classification.
  • Prompt-tuning challenge: Prompt tuning replaces handcrafted prompts with tunable word embeddings, but continued training can reduce generalization and underperform zero-shot CLIP.CoOp’s accuracy can drop by at most 4% as training continues, and it may underperform without augmentation or enough samples.
  • Prompt-aligned Gradient: ProGrad measures general knowledge with the KL-loss gradient between zero-shot prompted CLIP and the few-shot model, alongside a cross-entropy gradient for domain knowledge.The two gradients are called the general direction and domain-specific direction, respectively.
  • Prompt-aligned Gradient: ProGrad decomposes the domain-specific gradient and discards the component opposing the general direction, updating the prompt only along non-conflicting directions.Orthogonal components are retained, while parallel components are accepted only when aligned with general knowledge.
  • Experiments: Across 15 image-classification benchmarks, ProGrad improves over CoOp on all 11 reported datasets and improves base-to-new harmonic means over CoOp and CoCoOp.The evaluation covers generic objects, fine-grained recognition, and action classification, as well as domain generalization and cross-dataset transfer.

2. Related Work

The related-work discussion places prompt tuning among methods for adapting VLMs to downstream tasks. It notes that existing adaptation methods are not directly applicable when transfer proceeds only from general knowledge to downstream knowledge.

  • VLMs Adaptation: VLM adaptation methods support downstream tasks including visual question answering, visual grounding, image retrieval, and semantic segmentation, while this paper focuses on image classification.Prior approaches include classifier-based fine-tuning such as Linear Probe and feature adapters such as CLIP-Adapter.
  • VLMs Adaptation: Methods designed for other adaptation settings are not directly applicable to prompt tuning because its transfer direction runs only from general knowledge to downstream knowledge.The paper states that their failure in several ablation studies is shown in the appendix.

3. Methodology

ProGrad adapts CLIP prompts for few-shot classification by aligning domain-specific updates with the general knowledge encoded by zero-shot predictions. It preserves this knowledge by modifying conflicting gradients and theoretically bounds generalization error using both pretrained and downstream data.

  • Prompt-based learning: CoOp replaces hand-crafted prompts with learnable continuous context vectors optimized from few-shot downstream examples.The prompt combines the learnable context vectors with each class token and is optimized using classification loss.
  • Gradient alignment: ProGrad estimates general knowledge from the KL divergence between zero-shot and fine-tuned predictions, while domain knowledge comes from cross-entropy with ground-truth labels.The corresponding gradients are Gg = ∇vLkl(v) and Gd = ∇vLce(v).
  • Gradient alignment: When domain and general gradients align, ProGrad uses the domain gradient; when they conflict, it projects the domain gradient onto the orthogonal direction of the general gradient.Conflict is defined by an angle larger than 90°, and the projection avoids increasing the KL loss.
  • Generalization analysis: ProGrad's learned model can be viewed as optimizing empirical risk on both pretrained and downstream domains.Its generalization analysis uses Rademacher complexity, domain discrepancy, and sample-size-dependent estimation terms.
  • Training pipeline: Only context vectors are learnable, and λ controls general-knowledge guidance, with λ = 1 enforcing orthogonal projection and λ = 0 reducing ProGrad to CoOp.The resulting update is intended to prevent gradients from overfitting few-shot downstream samples.
  • Generalization analysis: Under a small pretrained-to-downstream domain gap, CoOp's estimation-error bound is at least twice ProGrad's, yielding a lower theoretical error bound for ProGrad in few-shot settings.The bound depends inversely on training-sample counts, with downstream sample size typically small in few-shot learning.

4. Experiments

ProGrad is evaluated across few-shot classification, generalization, transfer, robustness, and further analyses. It consistently improves over prompt-tuning baselines, while analyses examine distillation, gradient conflicts, oracle directions, conventional fine-tuning, and regularization strength.

  • Experimental settings: ProGrad is evaluated in few-shot classification, domain generalization, base-to-new generalization, and cross-dataset transfer across 15 image-classification benchmarks.The benchmark collection covers generic objects, fine-grained recognition, actions, satellite imagery, textures, and scenes.
  • Few-shot classification: Prompt ensembling improves both zero-shot CLIP and ProGrad: CLIP++ rises from 58.77% to 59.38%, while ProGrad++ reaches 75.03% from ProGrad's 74.28% at 16 shots.ProGrad++ uses prompt ensembling only to generate the general direction while optimizing a single prompt.
  • Base-to-new generalization: ProGrad achieves the highest average performance across base-to-new metrics, while CoOp and CoCoOp consistently underperform zero-shot CLIP on new classes.The harmonic mean evaluates the trade-off between base- and new-class accuracy.
  • Cross-dataset transfer: ProGrad outperforms other baselines on 9 of 10 target datasets and achieves the highest source-dataset performance in cross-dataset transfer.The transfer setting trains on ImageNet and evaluates on ten target datasets.
  • Further analysis: ProGrad outperforms knowledge distillation across few-shot settings, whereas LwF improves low-shot CoOp but drops at 8 and 16 shots.The comparison uses Ltotal = Lce+α·Lkd for LwF.
  • Further analysis: A more accurate oracle general direction produces a stronger ProGrad model, while smaller λ generally weakens regularization and lowers low-shot performance.For DTD at 16 shots, λ = 0.9 improves results, possibly because stronger pre-trained regularization is detrimental when downstream data is sufficient.

5. Conclusion

ProGrad addresses overfitting in few-shot prompt tuning by preventing updates from conflicting with the hand-crafted prompt’s general knowledge. Experiments across several generalization settings and 11 datasets demonstrate its effectiveness and efficiency.

  • ProGrad regularizes each tuning step so it does not conflict with the general knowledge of the hand-crafted prompt.
  • Experiments cover few-shot classification, base-to-new generalization, domain generalization, and cross-dataset transfer over 11 datasets.
  • The paper identifies overfitting issues in existing prompt tuning methods that rely heavily on early stopping and data augmentation.
  • Future work will investigate applying ProGrad to object detection and segmentation.

A. Justification from Generalization Error

The analysis bounds ProGrad’s downstream generalization error using training risk, the gap between pre-trained and downstream domains, and sample-dependent estimation errors. When the domain gap is small, the bound is lower than for conventional fine-tuning.

  • The expected and empirical risks are defined for classifiers on the downstream and pre-trained domains.
  • Conventional fine-tuning trains on downstream empirical risk, whereas zero-shot CLIP is treated as trained on pre-trained-domain empirical risk.
  • ProGrad is initialized from the pre-trained model and regularizes each step to avoid increasing KL divergence between ProGrad and pre-trained predictions.
  • The bound assumes bounded inputs, bounded layer Frobenius norms, 1-Lipschitz positive-homogeneous activations, and a softmax output over c classes.
  • ProGrad’s generalization error is bounded by empirical training risk, the pre-trained–downstream domain gap, and estimation errors that decrease with sample counts.The bound includes terms involving Nd and Np, the downstream and pre-trained sample numbers.
  • When the pre-trained and downstream domain gap is very small, ProGrad’s estimation-error bound is at least 2 times lower than conventional fine-tuning’s.

B. Additional Implementation Details

The implementation initializes ProGrad’s learnable context from a hand-crafted prompt and follows CoOp’s training settings. Prompt templates for the evaluated datasets and ImageNet variants are documented separately.

  • ProGrad initializes the learnable context vector with the word embeddings of the zero-shot hand-crafted prompt.With context length M=16 and “a photo of a”, the first 12 vectors are zero-initialized and the final 4 use the prompt’s embeddings.
  • The implementation follows CoOp’s training settings, including SGD for all prompt-based models.
  • Hand-crafted prompts for 11 datasets and ImageNet variants are listed in Table 8, with selected ImageNet ensemble prompts shown in Table 9.

D.1. Additional Few-shot Classification Results

Additional results compare ProGrad with conventional fine-tuning, prompt tuning, prompt regularization, knowledge distillation, and gradient-modification alternatives. CoOp + ProGrad achieves the best average performance and the comparisons support ProGrad’s broader benefit.

  • CoOp + ProGrad achieves the best average performance among the compared methods in Table 10.
  • Both conventional pre-train-then-fine-tune and prompt tuning benefit from adding ProGrad.
  • Directly constraining the learned prompt to remain close to the hand-crafted prompt provides limited improvement.
  • Introducing general knowledge improves performance through CoOp + KD and CoOp + GM, but both remain below CoOp + ProGrad.
  • CoOp + KD can still learn downstream knowledge conflicting with general knowledge, while CoOp + MD relies on estimating domain-specific gradients from limited data.

D.2. Additional Results for Base-to-New Generalization

This section presents base-to-new generalization results separately for each of the 11 datasets.

  • Table 12 reports base-to-new generalization results for each of the 11 datasets.
  • The evaluation is organized at the individual-dataset level rather than only as an aggregate result.
  • The reported setting concerns generalization from base classes to new classes.

D.3. Additional Results for Domain Generalization

This section collects additional evaluation material, including prompt ensembling examples, few-shot results across 11 datasets, base-to-new evaluation, and domain generalization results.

  • Domain Generalization: Table 13 provides averaged accuracies with standard deviations for domain generalization.
  • Prompt Ensembling: The section also includes prompt ensembling examples for ImageNet.
  • Few-Shot Learning: Few-shot learning results cover 11 datasets, with prompt-based methods using context length M=16.
  • Base-to-New Generalization: Base-to-new evaluation uses prompt methods learned from base classes with 4 shots and reports accuracy and harmonic mean H.
  • Domain Generalization: Domain generalization results are reported with standard deviation.
Loading 2205.14865v4…