Source-linked AI summary
DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning
Sikai Bai, Jie Zhang, Shuaicheng Li, Song Guo, Jingcai Guo, Jun Hou, Tao Han, Xiaocheng Lu
TL;DR
Federated domain generalization must handle unseen target domains, while existing approaches commonly rely on client–domain alignment or sample domain labels. DiPrompT uses global and domain prompts with adaptive domain querying, and the paper reports stronger performance than state-of-the-art methods, including some that use domain labels.
Problem
Federated learning methods may perform poorly on unseen target domains, while domain-generalization approaches often assume client–domain alignment or labeled samples, which can be costly and risk privacy leakage.
Source: §1 · pp. 1–2Method
DiPrompT learns global and domain-specific prompts, uses adaptive image-text querying to select domains for unlabeled samples, and ensembles prompt knowledge for target prediction.
Source: §1 · p. 2Results
DiPrompT outperforms state-of-the-art methods on multiple domain-generalization tasks, including some methods that use domain labels.
Source: §1 · p. 2Takeaways & Limitations
The approach provides complementary general and domain-specific knowledge for predicting on unseen target domains without requiring sample domain labels.
Source: §1 · p. 2The considered setting allows the number of clients to exceed the number of source domains and assumes domain labels are not given.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Federated learning (FL) has emerged as a powerful paradigm for learning from decentralized data, and federated domain generalization further considers the test dataset (target domain) is absent from the decentralized training data (source domains). However, most existing FL methods assume that domain labels are provided during training, and their evaluation imposes explicit constraints on the number of domains, which must strictly match the number of clients. Because of the underutilization of numerous edge devices and additional cross-client domain annotations in the real world, such restrictions may be impractical and involve potential privacy leaks. In this paper, we propose an efficient and novel approach, called Disentangled Prompt Tuning (DiPrompT), a method that tackles the above restrictions by learning adaptive prompts for domain generalization in a distributed manner. Specifically, we first design two types of prompts, i.e., global prompt to capture general knowledge across all clients and domain prompts to capture domain-specific knowledge. They eliminate the restriction on the one-to-one mapping between source domains and local clients. Furthermore, a dynamic query metric is introduced to automatically search the suitable domain label for each sample, which includes two-substep text-image alignments based on prompt tuning without labor-intensive annotation. Extensive experiments on multiple datasets demonstrate that our DiPrompT achieves superior domain generalization performance over state-of-the-art FL methods when domain labels are not provided, and even outperforms many centralized learning methods using domain labels.
1. Introduction
Federated domain generalization must address shifts between decentralized training data and unseen target domains, while existing approaches often assume client–domain alignment and labeled domains. DiPrompT separates general and domain-specific knowledge and adaptively combines it for target prediction without requiring domain labels.
1. Introduction
- Existing federated domain-generalization methods mainly learn domain-invariant features and assume a one-to-one mapping between clients and domains.When clients outnumber source domains, knowledge from a shared domain can be dispersed across clients, making sufficient invariant knowledge difficult to learn.Source: §1 · pp. 1–2
- Domain labels for each local sample can be costly to annotate in decentralized settings and may risk privacy leakage.Source: §1 · p. 2
- DiPrompT separates global and domain-specific knowledge with prompts, queries suitable domains for samples without labels, and ensembles the prompts for target prediction.The G-Prompt maintains domain-invariant representation; D-Prompts capture domain-specific knowledge, and the Q-Prompt searches for a domain through image-text alignment.Source: §1 · p. 2
- Experiments on multiple domain-generalization tasks report DiPrompT outperforming state-of-the-art methods, including some methods using domain labels.The paper also describes the approach as lightweight and based on small-scale operations.Source: §1 · p. 2
2. Related Work
Prior work covers federated learning, centralized domain generalization, and prompt tuning, but the paper targets federated generalization with flexible client counts and unknown domain labels. Its method uses prompt-tuning modules and a staged query-and-optimization procedure.
2. Related Work
- Centralized domain-generalization methods require access to all source-domain data and sample domain labels, while federated methods considered here address flexible client counts and unknown labels.Some label-free methods use image style or clustering to assign pseudo-labels but do not use text information about underlying domains.Source: §2 · p. 2
- During training, DiPrompT updates the Q-Prompt, selects a D-Prompt, jointly optimizes G- and D-Prompts, then applies a moving-average update to D-Prompts.Source: Figure 2 · p. 3
- Prompt tuning adds learnable embeddings to input tokens for adaptation, and DiPrompT extends it to federated domain generalization.The paper contrasts this with continuous-vector prompts in CLIP language branches used for few-shot visual recognition.Source: §2 · p. 3
3. Methodology
DiPrompT separates shared and domain-specific knowledge into lightweight prompts, then uses dynamic querying and target-sample-dependent ensembling to apply them. Its setting permits more clients than source domains and does not provide domain labels during training.
3.1. Preliminary and Notations
- The setting relaxes the conventional one-to-one client–domain mapping, allowing K ≫ M and leaving domain labels unavailable during training.Multiple clients may hold data from a common source domain.Source: §3.1 · p. 3
3.2. Disentangled Prompt Learning
- Disentangled prompt learning pairs a globally optimized prompt for shared knowledge with domain prompts for valuable domain-specific knowledge.The design addresses the risk that a single shared prompt filters out information present only on subsets of clients.Source: §3.2 · p. 3
3.2.1 Global Prompt Tuning (G-Prompt)
- G-Prompt uses learnable context vectors with frozen CLIP encoders and optimizes image–text classification against class labels.Image and text embeddings are matched by cosine similarity, and the global prompt is trained with cross-entropy loss.Source: §3.2.1 · p. 4
3.2.2 Domain Prompts Tuning (D-Prompts)
- D-Prompts extract domain-specific knowledge with domain-wise prompts trained using classification and contrastive losses.The contrastive objective uses handcrafted prompts for the same domain as positive pairs and prompts from other domains as negatives.Source: §3.2.2 · p. 4
- For shared domains across clients, domain-wise aggregation combines prompts from clients associated with the same domain; beta momentum averaging is used to reduce client drift and preserve initial domain information.Only updated prompts are aggregated, and the paper sets β = 0.2.Source: §3.2.2 · p. 4
3.3. Dynamic Query Scheme (Q-Prompt)
- Q-Prompt automatically selects domain prompts through two alignments: classify each input into a category, then group it into an underlying domain.Training selects the domain with the highest probability under the ground-truth class; target testing selects across all categories.Source: §3.3 · p. 5
- Q-Prompt tuning uses feature-level MSE alignment and logits-level KL-divergence constraints against its beta-momentum counterpart.The overall objective combines G-Prompt loss with D-Prompt loss for the domain predicted by Q-Prompt.Source: §3.3 · p. 5
3.4. Collaborative Ensemble Process
- At inference, DiPrompT dynamically combines prompt knowledge into target-sample text features, accounting for differences in feature distance to source domains.The ensemble draws on both the global prompt and domain prompts.Source: §3.4 · p. 5
4. Experiments
The experiments assess DiPrompT on three benchmark datasets with domain distribution shifts.
4. Experiments
- Experiments on three benchmark datasets with domain distribution shifts assess the method’s effectiveness and robustness.Additional experiments are reported in the supplementary material.Source: §4 · p. 5
4.1. Experimental Setup
Experiments use leave-one-domain-out evaluation on three domain-generalization datasets, comparing DiPrompT with centralized, federated, and federated domain-generalization methods under shared implementation settings.
4.1.1 Datasets.
- PACS, Officehome, and VLCS use leave-one-domain-out evaluation, with one domain held out as target and the rest used as sources.The datasets contain four domains each.Source: §4.1.1 · p. 5
4.1.2 Baselines.
- Baselines include centralized domain-generalization methods, standard FL methods, FL+DG methods, and prompt- or adapter-based FL methods.The listed methods include SWAD, I2ADR, FedAvg, FedProx, FedADG, FedSR, FedGMA, PromptFL, and FedCLIP.Source: §4.1.2 · pp. 5–6
4.1.3 Implementation Details.
- Most methods use Adam at 5e-4 and shared hyperparameters, including λ = 1.0, batch size 16, K = 20 clients, and 100 global rounds.FedGMA uses its best-reported optimizer parameters; ResNet50 and a masked self-attention Transformer serve as image and text encoders.Source: §4.1.3 · p. 6
4.2. Performance Comparison
DiPrompT is evaluated against centralized and federated methods in a flexible setting without one-to-one client-domain mapping. It achieves the best average accuracy across four benchmark datasets and leads in 11 of 12 target-domain settings.
4.2. Performance Comparison
- The reported results cover 12 domain-generalization tasks on PACS, Officehome, and VLCS, averaged over three runs.Bold text denotes the best results.Source: §4.2 · p. 6
- With K > M and no one-to-one client-domain mapping, prior FL+DG methods perform poorly, often below FedAvg and FedProx.The authors suggest FL+DG methods may struggle to retain valuable knowledge when client volume greatly exceeds source-domain count.Source: §4.2 · p. 7
- DiPrompT achieves the best average accuracy across four benchmark datasets and outperforms prior methods in 11 of 12 target-domain settings.The comparison includes centralized, FL, and FL+DG methods; DiPrompT uses ResNet50 without domain labels.Source: §4.2 · p. 7
4.3. Ablation Study
Ablations and stress tests examine DiPrompT's components, data scarcity, backbone choices, hyperparameters, and efficiency. The reported findings support complementary prompt components, robustness across settings, and lower communication costs than selected FL+DG baselines.
4.3.1 Effectiveness of Components.
- Removing either prompt type, learnable query prompts, collaborative ensembling, or Q-Prompt losses reduces performance; the contrastive-loss removal causes a slight drop.Using domain labels during local training brings DiPrompT close to its upper bound.Source: §4.3.1 · p. 7
4.3.2 Results on Few-shot Recognition.
- DiPrompT improves performance across 1, 2, 4, 8, and 16-shot settings on PACS and VLCS, with larger gains at 1–2 shots.All methods improve as training samples per class increase.Source: §4.3.2 · pp. 7–8
4.3.3 Impacts of Backbone Architecture.
- DiPrompT outperforms other state-of-the-art methods across the tested backbones on PACS and VLCS.The tested image encoders are ResNet50, ViT-B/32, and ViT-B/16; stronger backbones improve performance for all methods.Source: §4.3.3 · p. 8
4.3.4 Effect of Hyperparameters.
- As client count rises from 3 to 50, DiPrompT's performance drops about 1% and remains ahead of other methods; λ = 1 gives optimal performance.Figure 4 analyzes client count K and the weight coefficient λ on PACS.Source: §4.3.4 · p. 8, Figure 4 · p. 8
4.3.5 Cost Analysis.
- DiPrompT saves at most 830 times communication cost per round versus FedADG and 625 times versus FedGM; on the same mini-batch, it outperforms FedADG in GPU time around 2 times and is comparable to FedGMA.Communication cost is measured by uploaded data size per round; computation cost is compared by GPU time on the same mini-batch.Source: §4.3.5 · p. 8
5. Conclusion
DiPrompT outperforms state-of-the-art methods and many centralized strategies using domain labels. It learns general and domain-specific knowledge across clients and adaptively searches for a suitable domain when labels are unavailable.
- DiPrompT outperforms state-of-the-art methods and is even better than many centralized learning strategies using domain labels.Source: §5 · p. 8
- The framework learns general and valuable specific information across clients, including when client and source-domain counts differ or clients share a domain.Source: §5 · p. 8
- Its prompt-tuning-based adaptive query mechanism searches for a suitable domain for each sample when local data lack domain labels.Source: §5 · p. 8