Source-linked AI summary
Domain Adaptation via Prompt Learning
Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, Gao Huang
TL;DR
UDA must transfer models from labeled source domains to unlabeled target domains without sacrificing semantic structure through domain alignment. DAPL uses domain-specific and domain-agnostic prompt contexts with pre-trained vision-language models, achieving state-of-the-art benchmark performance while optimizing few parameters.
Problem
UDA faces distribution shift, while aligning source and target features can distort semantic structures and reduce class discriminability.
Method
DAPL embeds domain-specific and domain-agnostic continuous contexts with class labels in prompts and trains them using contrastive learning.
Results
74.5%/86.9% accuracy is achieved on Office-Home and VisDA-2017, respectively, and accuracy improves by 2.5%/2.5% over CLIP.
Takeaways & Limitations
DAPL provides a prompt-learning paradigm for UDA that avoids feature alignment and learns distinct domain representations for source and target domains.
Takeaways & Limitations
The method’s disentanglement analysis is conditioned on an assumption about the visual representation f(xd).
Abstract
from arXiv · showhide
Unsupervised domain adaption (UDA) aims to adapt models learned from a well-annotated source domain to a target domain, where only unlabeled samples are given. Current UDA approaches learn domain-invariant features by aligning source and target feature spaces. Such alignments are imposed by constraints such as statistical discrepancy minimization or adversarial training. However, these constraints could lead to the distortion of semantic feature structures and loss of class discriminability. In this paper, we introduce a novel prompt learning paradigm for UDA, named Domain Adaptation via Prompt Learning (DAPL). In contrast to prior works, our approach makes use of pre-trained vision-language models and optimizes only very few parameters. The main idea is to embed domain information into prompts, a form of representations generated from natural language, which is then used to perform classification. This domain information is shared only by images from the same domain, thereby dynamically adapting the classifier according to each domain. By adopting this paradigm, we show that our model not only outperforms previous methods on several cross-domain benchmarks but also is very efficient to train and easy to implement.
1. Introduction
UDA transfers knowledge from labeled source domains to unlabeled target domains under distribution shift. DAPL uses domain-aware prompt learning instead of conventional domain alignment to preserve semantic structure while adapting classification.
- UDA addresses knowledge transfer from a labeled source domain to an unlabeled target domain when distribution shift limits direct generalization.
- Conventional alignment and adversarial methods learn domain-invariant features, but aligning domains can lose semantic information and class discriminability.
- DAPL introduces prompt learning for UDA using domain-agnostic context, domain-specific context, and class labels in continuous prompt representations.
- DAPL trains prompts with a contrastive objective, making image-text pairs positive only when their domains and categories match.
- 74.5%/86.9% accuracy is achieved on Office-Home and VisDA-2017, respectively, while DAPL improves accuracy by 2.5%/2.5% over CLIP.
- The paper presents DAPL as the first application of prompt learning to unsupervised domain adaptation, to the authors’ knowledge.
2. Related Work
Related work includes discrepancy-based and adversarial UDA methods that seek domain-invariant representations, alongside approaches designed to preserve class discrimination. DAPL instead learns domain-specific visual concepts through prompt learning.
- Discrepancy-based UDA methods align source and target distributions using measures such as MMD and related statistical objectives.
- Adversarial UDA methods use domain discriminators and minimax training to obtain representations intended to be domain-invariant.
- Recent UDA methods address the loss of class discrimination associated with distorted semantic feature structures through discriminant features or prediction regularization.
- DAPL learns domain-specific visual concepts, such as transparent backgrounds for the product domain, rather than relying only on shared prompts.
3. Method
DAPL uses learnable prompts that combine domain-agnostic, domain-specific, and class information with CLIP image-text contrastive learning for domain adaptation.
- Preliminaries: DAPL uses CLIP image and text encoders to map images and prompts into a shared feature space for classification.Classification is based on cosine similarity between encoded image and text features.
- Optimization: DAPL trains with contrastive alignment of image-text pairs whose domain and class both match, using cross-entropy and pseudo-labels for confident target samples.Target pseudo-labels are generated from the highest predicted probability when it exceeds threshold τ, and source and target losses are combined.
- Prompt learning: Continuous prompt embeddings replace fixed manually designed tokens to describe semantic features more precisely.The prompt-learning formulation optimizes continuous context variables rather than relying only on discrete token embeddings.
- Domain-specific prompts: DAPL combines domain-agnostic context, domain-specific context, and a class token, with domain-specific context shared across categories within each domain.Source and target domains use separate prompts, while domain-specific tokens are shared across all categories in a domain.
- Class-specific prompts: Class-specific context can provide finer-grained category representations than using only a class token.The main results use class-specific context together with domain-specific context.
4. Experimental Results
Experiments on Office-Home and VisDA-2017 evaluate DAPL across benchmark comparisons, efficiency measurements, ablations, and confidence visualizations. DAPL achieves strong accuracy while requiring little training time, with domain-specific context central to its gains.
- Quantitative Evaluation: DAPL achieves state-of-the-art performance on Office-Home and VisDA-2017, reaching 74.5% and 86.9% accuracy, respectively.The evaluation covers 12 Office-Home tasks and 12 VisDA-2017 classes.
- Quantitative Evaluation: DAPL surpasses SRDC by 3.2% on Office-Home average accuracy and STAR by 4.2% on VisDA-2017 average accuracy.The reported comparisons use average accuracy across 12 Office-Home tasks and 12 VisDA-2017 classes.
- Training Time Analysis: DAPL trains in 5.3 hours on VisDA-2017, compared with 13.4 hours for MCD and 38.3 hours for DANN.The efficiency is attributed to fine-tuning only the prompt with very few parameters.
- Visualization: Across VisDA-2017 and Office-Home, DAPL produces the highest ground-truth-class prediction confidence among the compared prompting methods.The comparison includes manually designed, domain-agnostic, and proposed prompts.
- Ablation Study: Domain-specific context is crucial: DAPL improves accuracy by 2.5% over the hand-crafted prompt baseline on VisDA-2017.Adding domain-specific context to unified or class-specific context also produces further gains, including a 1.4% improvement over the domain-agnostic unified-context setting.
- Ablation Study: Token length has little effect overall, although performance is slightly lower when M1 < M2.This suggests continuous representations can be learned with a small number of tokens.
- Ablation Study: Performance is not sensitive to pseudo-label threshold τ from 0.4 to 0.7 because fewer, more confident labels can offset reduced label quantity.At τ = 0.7, pseudo-label quality may compensate for the performance drop caused by fewer pseudo labels.
- Visualization: Visualization examples show that combining learnable domain-agnostic and domain-specific contexts improves confidence when prompts better match target-domain visual concepts.Domain-specific context helps recognize product-domain imagery such as a backpack with a transparent background.
5. Conclusion
The paper introduces a prompt-learning method for UDA that avoids aligning features between domains by using domain-specific context to learn distinct domain representations. Results demonstrate its advantage, while prompt learning may extend to other visual UDA tasks such as semantic segmentation.
- DAPL avoids conventional feature alignment by designing domain-specific context for each domain.
- The method builds a bridge between multimodality methods and domain adaptation methods.
- Prompt learning may extend to other visual tasks in UDA, including semantic segmentation.