PaperlayerView exact revision on arXiv

Source-linked AI summary

TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model

Hantao Yao, Rui Zhang, Changsheng Xu

arXiv:2311.18231v2cs.CV

TL;DR

Textual prompt tokens can generalize poorly to unseen classes when they do not model class distributions. TCP uses TKE to turn class-level textual knowledge into class-aware prompts for the Text Encoder, and evaluations report higher performance with less training time across several tasks. The method’s class-aware prompts rely on the discriminative ability of general textual embeddings, and TCP performs poorly on FGVCAircraft.

  • Problem

    Domain-shared and image-conditional textual prompts have limited performance on unseen classes because they do not explicitly model class distributions.

    Source: §1 · p. 1

  • Method

    TCP uses TKE to map class-level textual knowledge into class-aware prompt tokens inserted into the Text Encoder to generate classifiers, including for unseen classes.

    Source: §1 · p. 2

  • Results

    Across 11 image-classification datasets, TCP achieves higher performance with less training time on base-to-new, cross-dataset, and few-shot evaluations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Evaluations on several benchmarks and tasks indicate that class-aware prompts are effective for prompt tuning.

    Source: §5 · p. 8

    TCP relies on the discriminative ability of general textual embeddings and performs poorly on FGVCAircraft.

    Source: §5 · p. 8

Abstract

from arXiv · show

Prompt tuning represents a valuable technique for adapting pre-trained visual-language models (VLM) to various downstream tasks. Recent advancements in CoOp-based methods propose a set of learnable domain-shared or image-conditional textual tokens to facilitate the generation of task-specific textual classifiers. However, those textual tokens have a limited generalization ability regarding unseen domains, as they cannot dynamically adjust to the distribution of testing classes. To tackle this issue, we present a novel Textual-based Class-aware Prompt tuning(TCP) that explicitly incorporates prior knowledge about classes to enhance their discriminability. The critical concept of TCP involves leveraging Textual Knowledge Embedding (TKE) to map the high generalizability of class-level textual knowledge into class-aware textual tokens. By seamlessly integrating these class-aware prompts into the Text Encoder, a dynamic class-aware classifier is generated to enhance discriminability for unseen domains. During inference, TKE dynamically generates class-aware prompts related to the unseen classes. Comprehensive evaluations demonstrate that TKE serves as a plug-and-play module effortlessly combinable with existing methods. Furthermore, TCP consistently achieves superior performance while demanding less training time. Code:https://github.com/htyao89/Textual-based_Class-aware_prompt_tuning/

1. Introduction

Existing textual prompt methods struggle to model class distributions for unseen classes. TCP uses Textual Knowledge Embedding (TKE) to project class-level textual knowledge into class-aware prompts, and evaluations report higher performance with less training time.

1. Introduction

  • Domain-shared prompts can underperform on unseen classes, while image-conditional prompts add image-specific knowledge but do not explicitly model class distributions.The paper identifies a need to relate learnable prompts dynamically to each class’s textual knowledge.Source: §1 · pp. 1–2
  • The paper focuses on textual prompt tuning and does not consider visual prompt tuning.Source: §1 · p. 1
  • TCP adds TKE to map class-level textual knowledge into class-aware prompts inserted into the Text Encoder to generate a class-aware classifier.At inference, TKE generates prompts for unseen classes from their textual knowledge.Source: §1 · p. 2
  • Across 11 image-classification datasets, TCP achieves higher performance with less training time on base-to-new, cross-dataset, and few-shot evaluations.The paper describes TCP as steering prompts to learn class-aware knowledge for downstream generalization and discrimination.Source: §1 · pp. 1–2
  • The paper proposes incorporating each class’s prior knowledge into learnable prompt tokens to enhance class-distribution discrimination.It also presents TKE as a plug-and-play module for existing prompt-tuning methods.Source: §1 · p. 2

2. Related Works

The related work situates prompt tuning within VLM adaptation, where textual tokens are used to produce task-specific representations. Prior prompt methods vary in how they condition or regularize tokens; TCP adds class-level textual knowledge to address unseen-class performance.

2.1. Vision-Language Models

  • VLMs use visual and textual modalities trained on large-scale image-text pairs, and CLIP is a representative framework used for downstream prompt tuning.The paper describes CLIP as using independent visual and text encoders trained with contrastive loss on 400 million image-text association pairs.Source: §2.1 · p. 2

2.2. Prompt Tuning

  • Textual prompt tuning combines learnable textual tokens with class tokens to produce task-specific textual knowledge, whereas CLIP’s hand-crafted template supports zero-shot prediction.Source: §2.2 · p. 2
  • Prompt-tuning methods include domain-shared, image-conditional, knowledge-constrained, distribution-aware, and multimodal approaches, while TCP projects class-level textual knowledge into class-aware prompts.The related work also covers ensembling, optimal transport, external knowledge, and adapter-based approaches.Source: §2.2 · p. 3

3. Methodolgy

The method builds on CLIP and CoOp textual prompting, then adds TKE-generated class-aware tokens to the Text Encoder. The resulting class-aware embeddings are optimized with contrastive and knowledge-guided consistency losses.

3.1. Preliminaries

  • CoOp replaces CLIP’s hand-crafted textual tokens with learnable tokens concatenated with each class token, then optimizes them using image and class embeddings.CLIP’s hand-crafted template supplies general class-level textual embeddings.Source: §3.1 · p. 3
  • KgCoOp adds consistency between generated and general embeddings, and the robust prompt-tuning objective weights this consistency term by 8.0.The objective also includes contrastive loss.Source: §3.1 · p. 3

3.2. Textual-based Class-aware Prompt tuning

  • TCP’s TKE projects general class-level textual embeddings through down- and up-projection layers into class-aware prompt tokens reshaped to M × D for insertion into the Text Encoder.The prompt length M and dimension D determine the projected token shape.Source: §3.2 · p. 4
  • TCP inserts the class-aware prompts at layer l alongside shared learnable and class tokens, then uses the remaining Text Encoder layers to produce embeddings optimized with contrastive and consistency losses.The first l layers produce the intermediate representation into which the class-aware prompts are inserted.Source: §3.2 · p. 4

4. Experiments

The experiments evaluate TCP on base-to-new, cross-dataset, and few-shot tasks, then examine TKE, prompt design, and efficiency. TCP reports strong results across these settings and ablations.

4. Experiments

  • The evaluation covers base-to-new generalization, few-shot learning, and cross-dataset generalization on 11 benchmarks.Experiments use CLIP with a ViT-B/16 backbone, random prompt initialization, and 50 training epochs.Source: §4 · p. 4, §4.1 · pp. 4–5

4.2. Base-to-New Class Generalization

  • TCP scores 84.35%/75.28%/79.56% on Base/New/H, versus KgCoOp’s 80.73%/73.6%/77.0%, and exceeds PromptSRC’s H (79.51% vs 79.31%).In comparison with DePT, TCP improves Base/New/H by 0.51%/0.32%/0.41%.Source: Table 1 · p. 5, §4.2 · pp. 5–6

4.3. Cross-Dataset Generalization

  • TCP reaches 66.29% average performance on cross-dataset evaluation, above DePT’s 65.55% and comparable to DAPT’s 66.31%.TCP is trained on ImageNet and evaluated on ten unrelated datasets.Source: Table 2 · p. 6, §4.3 · p. 6

4.5. Ablation Study

  • TKE improves five existing methods, and class-aware prompts outperform domain-shared prompts on Base/New performance.TKE gains are 1.49%, 2.51%, 1.30%, 1.57%, and 3.65% for CoOp, KgCoOp, ProGrad, PromptSRC, and DAPT, respectively; class-aware prompts improve Base/New from 80.73%/73.6% to 84.05%/75.18%.Source: §4.5 · pp. 6–7, Table 3 · p. 6, Table 5 · p. 7

4.4. Few-shot Classification

  • TCP achieves 76.32% average performance in the 4-shot classification setting across 11 datasets.Evaluation uses the same class space as training.Source: §4.4 · p. 6, Table 4 · p. 7

4.5. Ablation Study

  • Prompt length M=8, TKE dimension Dmid=128, and insertion at layer 8 perform best; TCP is relatively insensitive to prompt length.Across prompt lengths 1, 2, 4, 8, and 16, H ranges from 79.26% to 79.63%; the random template outperforms the handcrafted template.Source: §4.5 · p. 7, Figure 3 · p. 7, Table 6 · p. 7
  • Single-layer TCP-Shallow outperforms deeper variants, while TCP with TKE outperforms CLIP-Adapter and CoOp-Adapter on all 11 datasets.TCP-Shallow with 8-layer reaches H of 79.51% and New-class performance of 75.36%.Source: §4.5 · p. 8, Table 7 · p. 8, Figure 6 · p. 8
  • TCP is reported to perform better with less running time, although TKE adds slight time over CoOp and KgCoOp; Figure 7 compares prediction probabilities.Training time is measured as average processing time per ImageNet image in the 16-shot setting.Source: §4.5 · p. 8, Table 8 · p. 8, Figure 7 · p. 8

5. Conclusion

TCP uses class-level textual knowledge to create class-aware prompts, and the paper reports effectiveness across several benchmarks. Its reliance on discriminative general textual embeddings constrains its performance.

5. Conclusion

  • TCP uses Textual Knowledge Embedding to transfer class-level textual embeddings into class-aware prompts, combined with pre-trained class tokens for task-specific textual knowledge.Source: §5 · p. 8
  • Several benchmarks and tasks show the class-aware prompt is effective for prompt tuning.Source: §5 · p. 8
  • TCP relies on the discriminative ability of general textual embeddings; weaker embeddings can produce weaker textual classifiers.The paper notes that TCP does not perform well on FGVCAircraft.Source: §5 · p. 8

A. Effect of Prompt Fusion

TCP fuses class-aware prompts with mid-level textual tokens. The paper reports that higher fusion weights are associated with better performance, especially on unseen classes.

A. Effect of Prompt Fusion

  • The method replaces the M-th textual tokens with the obtained class-aware prompt.Source: A. Effect of Prompt Fusion · p. 12
  • The fusion formulation combines class-aware prompt T′_m with mid-level textual token F_l,m to form fused token m.Source: A. Effect of Prompt Fusion · p. 12
  • Higher ω is associated with higher performance, with an obvious improvement in New performance on unseen classes.The paper attributes this pattern to higher ω considering fewer mid-level textual tokens biased toward the training domain for testing.Source: A. Effect of Prompt Fusion · p. 12

B. Class-aware Prompt vs regularize Lkg

The paper reports that combining the class-aware prompt with the regularizer yields higher performance than using either alone.

B. Class-aware Prompt vs regularize Lkg

C. Comparison of training time and model complexity

With the same ViT-B/16 backbone, methods have similar inference time because the backbone mainly controls inference time.

C. Comparison of training time and model complexity

  • Methods using the same ViT-B/16 backbone have similar inference time, as the backbone mainly controls it.The paper states that prompt-tuning parameters are far fewer than fixed parameters.Source: C. Comparison of training time and… · p. 12

D. Domain Generalization

Domain generalization is evaluated by testing on target datasets with the same classes but different distributions from the source. TCP is trained on few-shot ImageNet and evaluated on four ImageNet variants.

  • Domain generalization tests models on target datasets that share classes with the source domain but have different data distributions.Source: D. Domain Generalization · p. 12
  • TCP is conducted on few-shot ImageNet and evaluated on ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R; results are summarized in Table 11.Source: D. Domain Generalization · p. 12

E. Datasets

The evaluation uses 11 datasets spanning several recognition categories. Table 12 provides detailed dataset statistics.

  • Evaluation covers 11 datasets, including ImageNet, Caltech, OxfordPets, StanfordCars, Flowers, Food101, FGVCAircraft, EuroSAT, UCF101, DTD, and SUN397.Source: E. Datasets · p. 12
  • The datasets span general object, satellite image, scene, texture, action, and fine-grained object recognition; Table 12 gives their detailed statistics.Source: E. Datasets · p. 12, Table 12 · p. 13

Related papers

Loading 2311.18231v2…