PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Domain Generalization via Domain Prompt Learning

Shiming Chen, Shuhuang Chen, Salman Khan, Fahad Shahbaz Khan

arXiv:2610.30680v1cs.CV

TL;DR

Domain generalization methods often rely on source-domain data, which can be costly to collect and may not cover new or scarce classes. DoP uses text prompts and test-image-based visual refinement for zero-shot domain generalization, reporting competitive results across DG benchmarks and ImageNet. Its results on challenging datasets remain an acknowledged limitation.

  • Problem

    Existing domain-generalization methods rely on source-domain data, which can be costly or infeasible to collect and may not include new or scarce classes.

    Source: §1 · p. 2

  • Method

    DoP uses text domain learning to generate diverse style tokens and visual domain learning to refine a test-relevant token with test-image data.

    Source: §1 · p. 2

  • Results

    DoP reports competitive performance over current state-of-the-art methods on PACS, VLCS, OfficeHome, DomainNet, and ImageNet.

    Source: §1 · p. 3

  • Takeaways & Limitations

    DoP is presented as a way to learn target-domain knowledge in text and visual modalities for zero-shot domain generalization.

    Source: §5 · p. 11

    Results on challenging datasets such as DomainNet and ImageNet show room for improvement, which the authors suggest may reflect limited domain knowledge in text space or insufficient information from a single test image.

    Source: §5 · p. 12

Abstract

from arXiv · show

Domain generalization (DG) aims to improve the generalization of the model to any unseen domains. Targeting this goal, existing DG methods mainly focus on learning domain-invariant features or prompts from the given source domains. However, these methods rely on source-domain data, and collecting and annotating large-scale single- or multi-source datasets is often infeasible, limiting the practical applicability of DG. To tackle this challenge, we propose {\underline{Do}}main \underline{P}rompt Learning (DoP), a method built on a large vision-language model (e.g., CLIP) for zero-shot domain generalization, which recognizes unseen classes in unseen target domains without access to source-domain data. DoP essentially learns the target domain knowledge via prompt both in text and visual modalities to overcome the domain shift in text-visual modalities under the test-time adaptation formulation. Specifically, DoP first takes text domain learning (TDL) to simulate various distribution shifts by learning diverse style tokens implicitly in the text space with inter-style diversity loss and intra-style consistency loss. Subsequently, DoP employs visual domain learning (VDL) to further refine the target-domain-relevant style token generated by TDL by making use of the test visual data with self-supervised learning. Extensive experiments on popular DG benchmark datasets (e.g., PACS, VLCS, OfficeHome, and DomainNet) and ImageNet show that our DoP consistently outperforms the state-of-the-art DG approaches. Codes are available at: https://github.com/shiming-chen/DoP.

1 Introduction and Motivation

DoP addresses zero-shot domain generalization, where models must recognize unseen classes in unseen domains without source-domain data. It learns target-domain knowledge through text prompts and test-image-driven visual refinement, and reports competitive results across DG benchmarks and ImageNet.

1 Introduction and Motivation

  • Standard domain-generalization methods learn domain-invariant features or prompts from source data, using strategies such as augmentation, cross-domain alignment, knowledge decomposition, meta-learning, and ensembles.The introduction frames this source-data dependence as a practical constraint on deployment.Source: §1 · pp. 1–2
  • Zero-shot domain generalization targets recognition of unseen classes in unseen domains without source-domain data, addressing the cost of collecting and annotating such data and the difficulty of anticipating new or scarce classes.The paper also notes uncertainty about which source domains, or how many, are sufficient when unseen domains are arbitrary.Source: §1 · p. 2
  • The central challenge is domain shift across text and visual modalities in unseen domains, which the paper describes as more challenging than visual-only shift in standard domain generalization.The proposed direction uses pretrained VLM zero-shot capacity and discovers target-domain knowledge in both modalities.Source: §1 · p. 2
  • DoP uses text domain learning to synthesize diverse style tokens with inter-style diversity and intra-style consistency losses, then selects a token relevant to the test image and refines it through visual domain learning.The refinement uses test-image information and self-supervised learning during testing.Source: §1 · pp. 2–3
  • DoP reports competitive performance over current state-of-the-art methods on PACS, VLCS, OfficeHome, DomainNet, and ImageNet.The paper presents these results as evidence that DoP learns domain knowledge for zero-shot domain generalization.Source: §1 · p. 3

2 Related Work

Related work spans source-data-based classical domain generalization and VLM-based approaches. DoP differs from prior source-free style prompting by selecting a test-image-relevant style token and refining it with visual data.

2 Related Work

  • Classical domain-generalization methods use source-domain strategies including augmentation, feature alignment, domain-specific/invariant knowledge decomposition, meta-learning, and ensembles.These approaches aim to improve target-domain generalization by learning from source domains.Source: §2 · p. 3
  • VLM-based domain-generalization methods include prompt learning and fine-tuning, but the surveyed approaches generally optimize using source-domain data and face distribution-shift or cross-dataset-bias challenges.The paper distinguishes DoP from multimodal DG methods that learn robust representations from multiple observed modalities.Source: §2 · p. 3
  • PromptStyler learns diverse pseudo-style words without source data to train a fixed classifier, whereas DoP selects a test-image-conditioned style token and further refines it visually.This contrast motivates DoP’s dynamic domain prompt learning.Source: §2 · p. 3
  • DoP’s pipeline uses text domain learning to generate and select a target-relevant style token, then visual domain learning to refine it before CLIP class prediction.The method is framed as implicitly learning domain knowledge in text and explicitly refining it in visual space.Source: Figure 2 · p. 4, §2 · p. 4

3 The Proposed Method

DoP performs zero-shot domain generalization by learning domain prompts in text and visual spaces under test-time adaptation, without downstream source images or target labels. It generates style tokens, selects a target-relevant token, refines it with test data, and uses the resulting prompt for CLIP prediction.

3 The Proposed Method

  • DoP follows a test-time prompt adaptation protocol without downstream source-domain images or target labels, while keeping pretrained CLIP encoders frozen during visual refinement.TDL generates candidate style tokens; VDL refines the selected token using unlabeled test data.Source: §3 · p. 4

3.1 Text Domain Learning

  • TDL creates style prompts and jointly learns diverse style tokens with diversity and consistency objectives, then selects a token for each test image by cosine similarity.The consistency objective makes samples close to their corresponding style features and distant from other styles, highlighting shared style characteristics across classes.Source: §3.1 · pp. 4–5

3.2 Visual Domain Learning

  • VDL uses augmented test-image views and entropy-based self-supervision to refine the selected style token, filters low-confidence patches, and discards the adapted token after each prediction.The averaged prediction is formed from patch predictions; refinement is independent for each test image, so adaptation is not accumulated across samples.Source: §3.2 · p. 6

3.3 Class Prediction

  • After VDL refinement, DoP uses the domain prompt as CLIP text input and predicts the test image's class from the resulting scores.The softmax temperature is set to 100 following CLIP.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate DoP on four domain-generalization benchmarks and ImageNet, with comparisons, ablations, qualitative analyses, hyperparameter studies, and computational timing. DoP reports strong benchmark performance, benefits from combining TDL and VDL, and reduces test time relative to TPT.

4 Experiments

  • DoP achieves the best average accuracies of 74.9% and 82.0% with ResNet-50 and ViT-B/16, respectively, on four DG datasets.It improves over PromptStyler by 0.2 and 1.2 percentage points with those backbones; on ImageNet variants it achieves a competitive average accuracy of 61.0%.Source: §4 · p. 7, Table 2 · p. 8, Table 3 · p. 9

4.2 Qualitative Results

  • TDL clusters text samples by style, while learned styles remain diverse; synthesized prompts with learned tokens capture PACS domain styles unlike randomly initialized tokens.The illustrated PACS styles are painting, cartoon, photo, and sketch.Source: §4.2 · pp. 8–9, Figure 5 · p. 10

4 Experiments

  • On PACS, DoP improves Art accuracy over PromptStyler by 1.0 and 1.2 percentage points with ResNet-50 and ViT-B/16, respectively.With ResNet-50 it also improves Cartoon and Sketch by 0.2 and 0.9 points while maintaining Photo accuracy; with ViT-B/16, Sketch decreases by 0.8 points.Source: §4 · p. 7, Table 5 · p. 10

4.1 Ablation Study

  • Joint TDL and VDL improve average accuracy by 5.6% over DoP without VDL and 4.3% over DoP without TDL.TDL-only and VDL-only variants decrease average performance by 6.2% and 4.9%, respectively; style token selection adds 0.6% average performance gain.Source: §4.1 · p. 8, Table 6 · p. 10

4.3 Hyperparameter Analysis

  • Accuracy is insensitive to confidence-selection percentile ρ, while too few or too many initialized tokens K slightly decrease performance.The authors attribute the K effect to incomplete domain coverage when K is too small and insufficient text samples to learn many tokens when K is too large.Source: §4.3 · pp. 10–11, Figure 7 · p. 11

4.4 Computational Efficiency

  • DoP takes 170.5 ms per PACS image and reduces testing time by 63.2% compared with TPT by refining only the selected style token.The timing comparison uses all 9,991 PACS images with CLIP ViT-B/16.Source: Table 7 · p. 11, §4.4 · p. 11

5 Conclusion

DoP addresses zero-shot domain generalization by learning domain knowledge in text and visual modalities through prompts. The authors report results on four DG benchmarks, while noting remaining challenges on datasets with more domains and classes.

5 Conclusion

  • DoP uses text domain learning and visual domain learning to learn domain knowledge in both modalities for recognizing unseen classes in unseen domains without source-domain data.The authors report quantitative and qualitative results on four widely used DG benchmarks.Source: §5 · p. 11
  • The authors present domain prompt learning as a promising direction for improving machine-learning models’ generalization.Source: §5 · p. 12
  • The authors identify DomainNet and ImageNet as challenging settings where performance still has room for improvement.They suggest limited domain knowledge in the constructed text space and insufficient information from a single test image as possible reasons.Source: §5 · p. 12

Declarations

The experiments use eight publicly available benchmark datasets; the authors declare no competing interests and list their contributions.

Declarations

  • The experiments use eight publicly available benchmarks, including PACS, VLCS, OfficeHome, DomainNet, and four ImageNet variants.Source: Declarations · p. 12
  • The authors declare no competing interests and report no funding.Source: Declarations · p. 12
  • Shiming Chen wrote the manuscript and conducted experiments, Shuhuang Chen supported coding, and Salman Khan and Fahad Shahbaz Khan supervised and polished the manuscript.Source: Declarations · p. 12

Related papers

Loading 2610.30680v1…