PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptKD: Unsupervised Prompt Distillation for Vision-Language Models

Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, Jian Yang

arXiv:2403.02781v5cs.CV

TL;DR

Prior prompt research focused on prompt formats learned with scarce labeled data, leaving prompt-driven distillation from a larger teacher comparatively unexplored. PromptKD uses a pretrained CLIP teacher, shared text-feature class vectors, and learnable student prompts for distillation on unlabeled images. Across 11 recognition datasets, it achieves state-of-the-art performance, with average gains on both base and new classes.

  • Problem

    Prior prompt research primarily focused on effective prompt formats learned with scarce labeled data, rather than prompt-driven distillation from a larger teacher.

    Source: §1 · p. 2

  • Method

    PromptKD pretrains a large CLIP teacher on few-shot domain labels, then uses shared teacher text-feature class vectors and learnable student prompts to align teacher and student logits on unlabeled images.

    Source: §1 · p. 2

  • Results

    Across 11 recognition datasets, PromptKD achieves state-of-the-art performance, with average improvements of 2.70% on base classes and 4.63% on new classes using ViT-B/16 CLIP.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PromptKD enables student training on extensive unlabeled domain images without requiring labeled images for student training.

    Source: §1 · p. 2

    The full-fine-tuning comparison used a dataset much smaller than common CLIP training sets and a 20-epoch schedule; the authors note larger datasets and longer schedules would improve it.

    Source: §8 · p. 13

Abstract

from arXiv · show

Prompt learning has emerged as a valuable technique in enhancing vision-language models (VLMs) such as CLIP for downstream tasks in specific domains. Existing work mainly focuses on designing various learning forms of prompts, neglecting the potential of prompts as effective distillers for learning from larger teacher models. In this paper, we introduce an unsupervised domain prompt distillation framework, which aims to transfer the knowledge of a larger teacher model to a lightweight target model through prompt-driven imitation using unlabeled domain images. Specifically, our framework consists of two distinct stages. In the initial stage, we pre-train a large CLIP teacher model using domain (few-shot) labels. After pre-training, we leverage the unique decoupled-modality characteristics of CLIP by pre-computing and storing the text features as class vectors only once through the teacher text encoder. In the subsequent stage, the stored class vectors are shared across teacher and student image encoders for calculating the predicted logits. Further, we align the logits of both the teacher and student models via KL divergence, encouraging the student image encoder to generate similar probability distributions to the teacher through the learnable prompts. The proposed prompt distillation process eliminates the reliance on labeled data, enabling the algorithm to leverage a vast amount of unlabeled images within the domain. Finally, the well-trained student image encoders and pre-stored text features (class vectors) are utilized for inference. To our best knowledge, we are the first to (1) perform unsupervised domain-specific prompt-driven knowledge distillation for CLIP, and (2) establish a practical pre-storing mechanism of text features as shared class vectors between teacher and student. Extensive experiments on 11 datasets demonstrate the effectiveness of our method.

1. Introduction

PromptKD uses a large CLIP teacher and learnable student prompts to distill domain knowledge on unlabeled images, reusing teacher text features as shared class vectors. On 11 recognition datasets, it outperforms previous methods and achieves state-of-the-art performance.

1. Introduction

  • CLIP uses separate image and text encoders trained with contrastive loss to align multimodal representations in a shared embedding space.Prompt-learning methods may adapt text prompts, visual prompts, or both modalities.Source: §1 · pp. 1–2
  • PromptKD transfers knowledge from a large CLIP teacher to a lightweight student through learnable visual prompts, aligning their logits with KL divergence.An extra projector adjusts for the teacher–student feature-dimension disparity.Source: §1 · p. 2
  • Prior prompt research focused on learning prompt formats with scarce labeled data, while PromptKD treats prompts as domain-knowledge distillers from a larger teacher.Source: §1 · p. 2
  • The teacher is pretrained on few-shot labeled domain data, then its text encoder computes and stores class vectors for reuse.The vectors are computed once and shared by teacher and student image encoders to calculate logits without additional text-branch computation.Source: §1 · p. 2
  • 2.70% and 4.63% average improvements on base and new classes, respectively, accompany PromptKD's state-of-the-art performance across 11 recognition datasets.The evaluation uses the ViT-B/16 image encoder CLIP model.Source: §1 · p. 2
  • The teacher supplies soft labels for extensive unlabeled domain images, so student training does not require labeled images; the student encoder and stored vectors are used for inference.Source: §1 · p. 2

2. Related Work

Related work covers prompt learning for adapting vision-language models, knowledge-distillation strategies, and methods that apply distillation to CLIP.

Prompt Learning in Vision-Language Models

  • Learnable text or visual soft prompts adapt pretrained VLM representations to downstream tasks without completely retraining the original model.MaPLe learns prompts in both branches, while PromptSRC regularizes prompt learning with original features.Source: §2 · p. 3

Knowledge Distillation

  • Knowledge distillation transfers knowledge from a large pretrained teacher to a lightweight student through logits alignment, feature imitation, or sample-relationship matching.Source: §2 · p. 3
  • CLIP distillation methods include CLIP-KD's feature mimicry with MSE loss and TinyCLIP's cross-modal feature alignment in affinity space.PromptKD instead learns student prompts while keeping the student's original CLIP weights frozen.Source: §2 · p. 3

3. Method

PromptKD transfers a domain-adapted CLIP teacher’s predictions to a student through learnable visual prompts trained on unlabeled domain images. It reuses precomputed teacher text features as class vectors across teacher and student image encoders.

3.2. PromptKD: Prompt Distillation for VLMs

  • PromptKD trains student visual prompts to imitate teacher predictions on unlabeled domain images, using KL divergence and a projector to align feature dimensions.The framework has teacher pretraining followed by student prompt distillation.Source: §3 · p. 3

3. Method

  • Figure 3 depicts the sequence from labeled teacher pretraining and saved text features, through student prompt and projector training, to inference with the student and stored vectors.The pre-saved class vectors avoid text-encoding computation during distillation.Source: §3 · p. 3, Figure 3 · p. 4

3.1. Background

  • CLIP uses image and text encoders to map both modalities into a joint embedding space, with text features serving as classification weights for image features.The output logit is the image–class feature product, and temperature is used in calculating probabilities.Source: §3.1 · p. 4
  • Soft prompts are learnable text or visual inputs that adapt representations, while knowledge distillation uses KL divergence to match teacher and student output distributions.Visual prompts are added to image patches; the distillation loss uses temperature-softened logits.Source: §3.1 · p. 4

3.2. PromptKD: Prompt Distillation for VLMs

  • The framework has teacher pretraining and student prompt distillation stages; the CLIP teacher is pretrained on labeled domain data using MaPLe [21], PromptSRC [22], or a publicly available pretrained CLIP model.Source: §3.2 · p. 5
  • Teacher text features are stored as shared class vectors and reused to compute teacher and student logits without text-branch computation during distillation.The stored teacher text features are also used with the trained student image encoder for inference.Source: §3.2 · p. 5

4. Experiments

Experiments evaluate a ViT-L/14 CLIP teacher and ViT-B/16 student across base-to-novel, cross-dataset, and unlabeled-data settings, then test design choices and inference cost.

4.2. Base-to-novel Generalization

  • PromptKD outperforms prior methods on all 11 base-to-novel datasets with a ViT-B/16 student and ViT-L/14 teacher.The comparison uses the same ViT-B/16 image encoder for the compared methods.Source: Table 1 · p. 6, §4.2 · p. 7

4.3. Cross-dataset Evaluation

  • PromptKD achieves a 5.52% average improvement over the previous method on cross-dataset evaluation, with better performance on 9 of 10 datasets.The cross-dataset evaluation uses unlabeled target-domain data in a transductive setting.Source: Table 2 · p. 6, §4.3 · p. 7

4.4. Comparison with Other Methods

  • On Flowers102, PromptKD performs better than previous methods that also use unlabeled data for training.The comparison uses ViT-B/16 CLIP and 16-shot labels for the methods using few-shot labels.Source: Table 3 · p. 7, §4.4 · p. 7

4.5. Ablation Study

  • As the number of unlabeled ImageNet images per class increases, student accuracy improves, with gains eventually beginning to plateau.The default ablation setting uses 64 images per class, or 64,000 images across 1,000 classes.Source: §4.5 · p. 7, Figure 4
  • Logit-based distillation performs significantly better than feature-based distillation in the reported comparison.The authors suggest that aligning image feature spaces may be harder because teacher and student model structures differ.Source: Table 4 · p. 7, §4.5 · p. 7
  • Prompt-based distillation and shared teacher class vectors are identified as crucial to PromptKD’s final performance.The alternatives include projector-only training, full fine-tuning, and using the student’s own text encoder.Source: §4.5 · p. 8, Table 5 · p. 8
  • Higher-accuracy pretrained teachers and stronger-capacity teachers lead to better student distillation performance.The reported comparisons include different teacher pretraining methods and ViT-B/16 versus ViT-B/32 teachers.Source: §4.5 · p. 8, Table 6 · p. 8, Figure 5 · p. 8
  • PromptKD has lower inference cost than CoOp, CoCoOp, and PromptSRC in the reported SUN397 comparison.Inference costs were measured on a single A100 GPU.Source: Table 7 · p. 8, §4.5 · p. 8

5. Conclusion

The conclusion presents PromptKD as a two-stage framework for transferring a large CLIP teacher’s knowledge to a lightweight student using prompts and unlabeled domain data. Its effectiveness depends on whether those unlabeled samples represent the target domain.

5. Conclusion

  • PromptKD pretrains a large teacher on few-shot labeled domain data, then distills to a student using extensive unlabeled domain data.The method reuses pre-stored teacher text features as class vectors for distillation and inference.Source: §5 · p. 8
  • Experiments across 11 recognition datasets demonstrate the effectiveness of the framework.Source: §5 · p. 8
  • When distillation samples do not represent the target domain, the student’s generalization to that domain may be biased or weakened.The authors identify regularization methods to mitigate this issue as future work.Source: §5 · p. 8

6. Experimental Settings

The experimental settings cover recognition, base-to-novel, cross-dataset, and domain-generalization evaluations, using CLIP teachers and students with different image-encoder capacities.

Datasets

  • The evaluation uses 11 datasets for base-to-novel and cross-dataset tests, and ImageNet-1K plus four variants for domain generalization.The broader evaluation also includes 15 recognition datasets.Source: §6 · p. 12

Implementation Details

  • The default teacher is ViT-L/14 CLIP and the target student is ViT-B/16 CLIP, with base and novel accuracy and their harmonic mean averaged over three runs.Source: §6 · p. 12

Training Data Usage

  • Students train for 20 epochs with SGD, batch size 8, and learning rate 0.005; distillation uses the full training dataset under a transductive zero-shot setting.The default temperature is τ=1, and the teacher is pretrained using PromptSRC.Source: §6 · p. 12

7. Additional Experiments

Additional experiments examine domain generalization, projector and student choices, temperature, and training duration. The reported results favor target-domain distillation, a two-layer projector, τ=1, and longer schedules.

Domain Generalization

  • PromptKD shows a clear advantage on four domain-generalization target datasets and consistent improvement across all of them.The teacher is pretrained on ImageNet, while students train on unlabeled target-dataset images.Source: Table 9 · p. 12, §7 · p. 12

Layer of Projector

  • Two MLP layers are effective for the projector; using more or fewer layers can cause overfitting or underfitting.Teacher accuracy is also reported for the base-to-novel and cross-dataset experiments.Source: §7 · p. 12, Table 12 · p. 13, Table 10 · p. 13

Distillation with Different Students

  • Student models with different capacities achieve consistent improvements through PromptKD.The comparison includes CLIP students with different image-encoder capacities.Source: §7 · pp. 12–13, Table 13 · p. 13

Temperature Hyperparameter

  • The best performance occurs at τ=1; increasing temperature rapidly reduces performance, and τ=4 is unsuitable for this task.Source: Figure 6 · p. 13, §7 · p. 13

Distillation with Longer Schedules

  • Student performance increases with longer training schedules across the tested 20-, 40-, and 60-epoch settings.Source: Table 14 · p. 13, §7 · p. 13

8. Discussion

The discussion links full fine-tuning’s lower results to its smaller training dataset and short schedule. It also reports that PromptKD can outperform a baseline with a weaker teacher, while attributing this behavior to keeping CLIP weights frozen and training only prompts.

8. Discussion

  • Full fine-tuning results are lower than those of other distillation methods by a large margin (>2%).The authors attribute this to a training dataset smaller than common CLIP training datasets and a short training schedule of 20 epochs; they state that larger datasets and longer schedules would improve full fine-tuning.Source: §8 · pp. 13–14
  • 71.87%>70.22%: with a weaker ViT-B/32 teacher than the ViT-B/16 student, PromptKD outperforms the baseline method.The passage contrasts this result with traditional distillation, where poor teachers often lead to a significant decline in student performance.Source: §8 · p. 14
  • PromptKD trains only learnable prompts while keeping the original CLIP model weights frozen, which remain influential in prediction.The authors say the trained prompts do not substantially bias model inference.Source: §8 · p. 14
Loading 2403.02781v5…