PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual-Language Prompt Tuning with Knowledge-guided Context Optimization

Hantao Yao, Rui Zhang, Changsheng Xu

arXiv:2303.13283v1cs.CVcs.CL

TL;DR

CoOp-based prompt tuning can overfit seen classes and lose the general textual knowledge needed for unseen-class generalization. KgCoOp minimizes the discrepancy between learned and hand-crafted textual embeddings alongside contrastive loss. Across multiple benchmarks, it reports higher performance with less training time, although seen-class discrimination can degrade.

  • Problem

    CoOp-based prompt tuning can forget general textual knowledge and perform worse than CLIP on unseen classes.

  • Method

    KgCoOp adds a regularization term that minimizes the discrepancy between learnable and hand-crafted textual embeddings alongside contrastive loss.

  • Results

    KgCoOp achieves higher performance with less training time across base-to-new generalization, few-shot classification, and domain generalization evaluations.

  • Takeaways & Limitations

    Constraining learned prompts toward hand-crafted prompts improves generalization to unseen classes without requiring additional computation beyond the described prompt-tuning procedure.

  • Takeaways & Limitations

    KgCoOp may degrade discriminative ability on seen classes, with the paper reporting badly degraded Base performance in some cases.

Abstract

from arXiv · show

Prompt tuning is an effective way to adapt the pre-trained visual-language model (VLM) to the downstream task using task-related textual tokens. Representative CoOp-based work combines the learnable textual tokens with the class tokens to obtain specific textual knowledge. However, the specific textual knowledge is the worse generalization to the unseen classes because it forgets the essential general textual knowledge having a strong generalization ability. To tackle this issue, we introduce a novel Knowledge-guided Context Optimization (KgCoOp) to enhance the generalization ability of the learnable prompt for unseen classes. The key insight of KgCoOp is that forgetting about essential knowledge can be alleviated by reducing the discrepancy between the learnable prompt and the hand-crafted prompt. Especially, KgCoOp minimizes the discrepancy between the textual embeddings generated by learned prompts and the hand-crafted prompts. Finally, adding the KgCoOp upon the contrastive loss can make a discriminative prompt for both seen and unseen tasks. Extensive evaluation of several benchmarks demonstrates that the proposed Knowledge-guided Context Optimization is an efficient method for prompt tuning, \emph{i.e.,} achieves better performance with less training time.

1. Introduction

Prompt tuning adapts pretrained visual-language models with task-related textual tokens, but CoOp-based methods can replace general knowledge with task-specific knowledge and lose generalization on unseen classes. KgCoOp addresses this by constraining learned textual embeddings toward hand-crafted prompts while retaining contrastive optimization.

  • Visual-language models learn general knowledge from large-scale image-text pairs, supporting generalization across tasks.
  • Prompt tuning adapts pretrained VLMs by applying task-related textual tokens, while hand-crafted prompts provide general textual knowledge and learnable prompts provide task-specific knowledge.
  • The distance between learnable and hand-crafted textual embeddings is associated with the severity of performance degradation on New classes.A larger distance corresponds to more severe degradation, with σclip and σcoop denoting CLIP and CoOp New-class accuracies.
  • CoOp-based methods can overfit few-shot seen classes, producing worse performance on unseen classes than CLIP.Reported New accuracies are 74.22% for CLIP, 63.22% for CoOp, and 71.69% for CoCoOp.
  • KgCoOp reduces forgetting of general textual knowledge by minimizing the discrepancy between learnable and hand-crafted textual embeddings.It adds this knowledge-guided constraint to the contrastive loss used to optimize learnable prompts.
  • Experiments across 11 image-classification datasets and four ImageNet variants report higher performance with less training time for KgCoOp.The reported training time is the same as CoOp and faster than CoCoOp and ProGrad.

2. Related Work

Prior work uses pretrained visual-language models and prompt tuning to adapt to downstream tasks. CoOp, CoCoOp, and ProGrad progressively modify prompts or updates to improve task adaptation, while KgCoOp aligns learned knowledge with general knowledge without discarding conflicting knowledge.

  • Visual-Language Models: Visual-language models use image-text association pairs to learn powerful visual-language representations.
  • Prompt Tuning: Prompt tuning adapts pretrained VLMs by inferring task-specific textual knowledge from task-related textual tokens.CLIP uses the hand-crafted template “a photo of a [CLASS]” for zero-shot class embeddings.
  • Prompt Tuning: CoOp replaces hand-crafted prompts with learnable soft prompts inferred from labeled few-shot samples, whereas CoCoOp generates image-conditional context.
  • Comparison with Existing Methods: KgCoOp adds a low-discrepancy constraint relative to CoOp and keeps conflicting knowledge rather than discarding it as ProGrad does.The paper reports that KgCoOp is more efficient than ProGrad because it requires no additional computation.

3. Methodolgy

The method reviews CLIP and CoOp prompt tuning, then adds KgCoOp to preserve general textual knowledge while adapting prompts to downstream tasks. KgCoOp minimizes the discrepancy between learned and hand-crafted textual embeddings alongside the standard contrastive loss.

  • 3.1. Preliminaries: CLIP uses frozen visual and textual encoders with a hand-crafted prompt to generate class embeddings and compute visual-textual similarity for prediction.The template is “a photo of a [class-name]”, and cosine similarity with a learnable temperature produces prediction probabilities.
  • 3.1. Preliminaries: CoOp replaces fixed context with M learnable vectors concatenated with each class token to generate task-related textual embeddings.The learnable context tokens are optimized from few-shot samples while the pretrained encoders remain frozen.
  • 3.2. Knowledge-guided Context Optimization: 69.34% versus 82.89% Base accuracy shows CoOp improves seen-class performance over CLIP, but 74.22% versus 63.22% New accuracy shows poorer unseen-class performance.Across 11 datasets, larger distance between learned and fixed prompts is associated with more severe unseen-class performance degradation; DTD has a 20.63% drop ratio.
  • 3.2. Knowledge-guided Context Optimization: KgCoOp measures the discrepancy between special and general textual knowledge using the Euclidean distance between learned and hand-crafted textual embeddings.The method minimizes this distance because the paper reports that lower distance corresponds to lower unseen-class performance degradation.
  • 3.2. Knowledge-guided Context Optimization: The final objective combines the standard cross-entropy loss Lce with the knowledge-guided constraint Lkg, weighted by λ.Lkg minimizes the squared discrepancy between learned embeddings wi and hand-crafted embeddings wclip_i across Nc seen classes.

4. Experiments

Experiments across base-to-new generalization, few-shot classification, and domain generalization show that KgCoOp improves unseen-class and overall performance while requiring less training time. Its gains come from constraining learned textual embeddings toward fixed general textual embeddings, with λ balancing this regularization.

  • Generalization From Base-to-New Classes: KgCoOp improves New-class performance over CoCoOp by 1.89% and 1.91% in the 8-shot and 16-shot ViT-B/16 settings, respectively, while maintaining similar Base accuracy.Base accuracy is 78.36% and 80.73% for KgCoOp versus 78.56% and 80.47% for CoCoOp.
  • Generalization From Base-to-New Classes: KgCoOp raises Harmonic mean over ProGrad from 75.2% to 76.06% in 8-shot and from 76.16% to 77.0% in 16-shot settings.The comparison reflects a trade-off: ProGrad performs better on Base classes, while KgCoOp performs better on New classes.
  • Generalization From Base-to-New Classes: KgCoOp achieves higher New performance on eight of 11 datasets, whereas existing CoOp-based methods are often worse than CLIP on unseen classes.CoCoOp and ProGrad each retain advantages on particular datasets, but KgCoOp has the strongest New-class coverage reported here.
  • Domain Generalization: KgCoOp improves average target-domain performance over CoCoOp from 59.90% to 60.11% in domain generalization.The comparison evaluates ImageNet-trained prompts on ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.
  • Few-shot Classification: KgCoOp obtains higher average performance than CoOp, CoCoOp, and ProGrad in 4-shot few-shot classification.The few-shot setting trains and evaluates on datasets with the same categories.
  • Analysis: Higher λ generally increases H by emphasizing general knowledge, but λ=10.0 yields 76.79%, below the 77.0% obtained with λ=8.0.Thus, the regularization weight has an effective range rather than improving performance without bound.
  • Analysis: Adding the Lkg constraint improves New and Harmonic mean performance over CoCoOp and ProGrad, with more than 3% improvement on New performance.Lkg minimizes the discrepancy between general and specific textual embeddings; lower Lkg is associated with higher H.

5. Conclusion

KgCoOp is presented as a prompt-tuning method that improves unseen-class generalizability by minimizing the discrepancy between general and learnable specific textual embeddings, but it can reduce seen-class discrimination.

  • KgCoOp minimizes discrepancy between general textual embeddings and learnable specific textual embeddings to improve unseen-class generalizability.
  • KgCoOp may improve unseen-class generalizability while degrading discriminative ability on seen classes.The authors identify poor Base performance on seen classes as a limitation and leave balancing both settings for future work.

Appendix A. Comparison for Cross-Dataset Transfer

Cross-dataset transfer evaluates prompts learned on ImageNet when applied to other downstream datasets. KgCoOp generally outperforms CoOp and ProGrad, although CoCoOp achieves the best average performance and EuroSAT is an exception.

  • CoCoOp achieves the best average performance across existing methods because its prompts combine textual prompts with visual descriptions.The passage also identifies CoCoOp as time-consuming.
  • KgCoOp outperforms CoOp and ProGrad on almost all downstream datasets except EuroSAT.
  • The cross-dataset results support KgCoOp's high generalizability for transfer from ImageNet to other downstream datasets.

Appendix B. Effect of Context Length

The appendix studies context length, initialization, hand-crafted templates, and discrepancy measurements for prompt tuning. Eight context tokens perform best across all three reported metrics, while the final model uses four tokens for fair comparison.

  • Effect of Context Length: 8 context tokens achieve higher average performance than 4 or 16 tokens on all three reported metrics across 11 datasets.Lengths 4 and 16 obtain similar performance.
  • Effect of Context Length: The final model uses 4 context tokens to enable fair comparison with CoOp and CoCoOp.
  • Effect of Initialization: The appendix compares word-embedding initialization using “a photo of a” with random Gaussian initialization for prompt tokens.Random initialization uses a zero-mean Gaussian distribution with standard deviation 0.02.
  • Effect of Hand-Crafted Prompts: Six hand-crafted prompt templates produce different performance, with T6, “a photo of a {}, a type of {}”, obtaining the highest performance.
  • Additional Comparisons: The appendix includes comparisons across different K-shot sample counts and discrepancy measurement methods in the base-to-new setting.

Appendix E. How to reduce the discrepancy between special knowledge and general knowledge?

KgCoOp reduces the discrepancy between special and general knowledge through an embedding-level regularization term. Among the tested discrepancy measures, the proposed Lkg achieves the strongest New and H performance.

  • Discrepancy Measurement: KgCoOp uses Lkg to minimize the distance between general and specific textual embeddings.
  • Discrepancy Measurement: Lpt performs worse than CoOp on H, indicating that directly constraining prompt-token similarity is not effective in this comparison.
  • Discrepancy Measurement: Lkg and Lkl outperform CoOp, while Lkg achieves the best New and H performance among the compared methods.

Appendix F. Failure cases

The failure-case analysis examines predictions that KgCoOp gets wrong while CoOp gets right, comparing them with zero-shot CLIP errors. A high proportion of these cases are also misclassified by CoOp.

  • Failure-case comparison: The analysis counts how often zero-shot CLIP models also fail on cases where KgCoOp is wrong but CoOp is correct.The comparison is presented in Figure A4's confusion matrix.
  • Failure-case comparison: A high proportion of KgCoOp's failure cases are also misclassified by CoOp.

Appendix G. Disscussion about the generalization on new class

The appendix attributes lower new-class performance to domain discrepancy between seen and new classes, which makes generalization harder. Large domain gaps can bias learned prompts toward seen classes.

  • Generalization on new classes: Domain discrepancy between seen and new classes makes generalization to new classes harder.
  • Generalization on new classes: CoOp loses more than 10% New performance on DTD, EuroSAT, and UCF101.
  • Generalization on new classes: Serious domain gaps between new and seen classes bias the learned prompt toward seen classes.The appendix links this bias to CoOp's behavior in Figure A4.

Appendix H. Detailed Results

Detailed experiments compare KgCoOp with CoOp-based methods across visual backbones and few-shot settings for base-to-new generalization. The appendix reports comparisons for ViT-B/16 and ResNet-50 at multiple shot counts, with context length M=4 for prompt-based methods.

  • Experimental comparisons: The experiments compare KgCoOp with CoOp, CoCoOp, and ProGrad across different backbones and K-shot samples.
  • Experimental comparisons: The evaluated visual encoders are ResNet-50 and ViT-B/16.
  • Experimental comparisons: The few-shot settings include 4-shot, 8-shot, and 16-shot comparisons.
  • ViT-B/16 results: Tables A5 and A6 report ViT-B/16 base-to-new results for 4-shot and 8-shot settings.The context length M is 4 for prompt-based methods, and H denotes harmonic mean.
  • ResNet-50 results: Tables A7–A9 report ResNet-50 base-to-new results for 4-shot, 8-shot, and 16-shot settings.The context length M is 4 for prompt-based methods, and H denotes harmonic mean.
Loading 2303.13283v1…