Source-linked AI summary
Federated Adversarial Domain Adaptation
Xingchao Peng, Zijun Huang, Yizhe Zhu, Kate Saenko
TL;DR
Federated learning preserves privacy but can suffer domain shift when source-node data differ from a new target node. This paper formulates UFDA and proposes FADA, combining adversarial alignment, feature disentanglement, and dynamic attention; experiments report improved adaptation across image and text tasks.
Problem
Federated learning methods can fail to generalize across non-i.i.d. nodes because source data differ from a new node’s unlabeled target data, without extra user supervision.
Method
FADA trains separate source-node models, aggregates source gradients for the target, aligns domains adversarially, disentangles representations, and applies dynamic attention.
Results
FADA improves adaptation performance over baseline methods on all evaluated image and text classification tasks.
Takeaways & Limitations
UFDA provides a federated framework for transferring knowledge from distributed source domains to an unlabeled target domain while retaining the federated privacy setting.
Takeaways & Limitations
The theoretical error bound holds only when model weights on all nodes are fully synchronized.
Abstract
from arXiv · showhide
Federated learning improves data privacy and efficiency in machine learning performed over networks of distributed devices, such as mobile phones, IoT and wearable devices, etc. Yet models trained with federated learning can still fail to generalize to new devices due to the problem of domain shift. Domain shift occurs when the labeled data collected by source nodes statistically differs from the target node's unlabeled data. In this work, we present a principled approach to the problem of federated domain adaptation, which aims to align the representations learned among the different nodes with the data distribution of the target node. Our approach extends adversarial adaptation techniques to the constraints of the federated setting. In addition, we devise a dynamic attention mechanism and leverage feature disentanglement to enhance knowledge transfer. Empirically, we perform extensive experiments on several image and text classification tasks and show promising results under unsupervised federated domain adaptation setting.
1 INTRODUCTION
Federated learning preserves local data and parameters but faces domain shift across non-i.i.d. nodes. The paper formulates UFDA and proposes FADA, combining privacy-preserving federated training with adversarial alignment, feature disentanglement, and dynamic attention.
- Federated learning keeps data and model parameters on local devices, supporting decentralized training without sharing them.
- Non-i.i.d. data across nodes creates domain shift, motivating knowledge transfer to a new unlabeled target node without additional user supervision.
- The federated setting prevents standard domain adaptation methods from accessing both labeled source and unlabeled target data, while node models converge differently and contribute unequally.
- FADA extracts domain-invariant features through adversarial domain alignment and feature disentanglement, with dynamic attention handling varying node convergence rates.
- FADA trains one model per source node and updates the target model using aggregated source gradients while preserving data privacy.
- Extensive image and language experiments improve adaptation performance over baselines on all evaluated tasks.
2 RELATED WORK
Prior work covers unsupervised domain adaptation and privacy-preserving federated learning, but the paper identifies a gap at their intersection. It positions the proposed framework as the first federated learning framework to consider unsupervised domain adaptation.
- Unsupervised Domain Adaptation: Unsupervised domain adaptation transfers knowledge from labeled source data to an unlabeled target domain using discrepancy, reconstruction, or adversarial approaches.
- Federated Learning: Federated learning enables collaborative model training while keeping training data and model parameters on local devices.
- Federated Learning: Existing federated methods mainly learn a single global model and lack convergence guarantees for non-i.i.d. data, while federated multi-task learning learns separate node models.
- The proposed work is described as the first federated learning framework to consider unsupervised domain adaptation.
- Feature Disentanglement: Feature disentanglement seeks to remove irrelevant and domain-specific features so representations model relevant factors of data variation.
3 GENERALIZATION BOUND FOR FEDERATED DOMAIN ADAPTATION
The paper derives a weighted generalization bound for UFDA with separate source hypotheses and an aggregated target hypothesis. The bound highlights source-target discrepancy and source weights as central factors in federated adaptation.
- The analysis reviews single-source domain adaptation bounds before deriving an error bound for unsupervised federated domain adaptation.
- The notation defines source and target risks, empirical risks, the H-divergence, and the hypothesis-class-based distribution discrepancy used in the analysis.
- UFDA distributes source data across nodes, prohibits sharing during training, and therefore learns separate models for each distributed source domain.
- The target hypothesis is formed by aggregating the parameters of the source hypotheses.
- The bound includes source weights, H∆H discrepancy, and VC-dimension-related constraints for a novel unlabeled target domain rather than a source mixture.
- The weighted bound extends the single-source result and becomes equivalent to it when only one source domain exists (N = 1).
4 FEDERATED ADVERSARIAL DOMAIN ADAPTATION
FADA combines federated adversarial alignment, representation disentanglement, and dynamic attention to reduce domain discrepancy while preserving decentralized data access.
- Dynamic Attention: Dynamic attention masks source gradients according to each domain’s contribution to improving target-feature clustering.The contribution is measured by the gap-statistics gain between consecutive iterations, and the resulting mask adjusts source-gradient weights.
- Federated Adversarial Alignment: FADA aligns source and target representations through adversarial training while keeping source data local and sharing only feature outputs with domain identifiers.Each source-target pair trains local feature extractors and an adversarial domain identifier; the extractors are trained to confuse it.
- Representation Disentanglement: Representation disentanglement separates extracted features into domain-invariant and domain-specific branches to support domain-invariant transfer.The method uses disentanglers, class identifiers, mutual-information estimation, and reconstruction to separate and preserve feature information.
- Experiments: Experiments evaluate FADA on digit classification, object recognition, and sentiment analysis across four datasets spanning image and text domains.The evaluated datasets are Digit-Five, Office-Caltech10, DomainNet, and Amazon Review.
- Optimization: The model jointly minimizes federated alignment, representation disentanglement, and task losses in an end-to-end stochastic-gradient training procedure.Algorithm 1 aggregates source feature-extractor and classifier parameters to update the target model after local training steps.
5 EXPERIMENTS
Experiments evaluate FADA across digit, object-recognition, and sentiment-classification tasks under UFDA protocols. FADA outperforms compared baselines, while analyses support contributions from dynamic attention, adversarial alignment, and representation disentanglement.
- Digit classification: FADA achieves 73.6% average accuracy on Digit-Five, outperforming the baselines.Model III combines dynamic attention, adversarial alignment, and representation disentanglement.
- Representation analysis: FADA learns more domain-invariant features, with smaller intra-class and larger inter-class variance than f-DANN and f-DAN on the visualized digit task.Its features also have smaller A-distance than raw ResNet and f-DANN features, making source and target representations harder to distinguish.
- Object recognition: 86.5% accuracy with AlexNet and 87.1% with ResNet outperform compared baselines on Office-Caltech10.Performance is similar across models for several target domains but worse when Amazon is the target, likely because of its large domain gap and white backgrounds.
- Object recognition: 28.9% accuracy with AlexNet and 30.3% with ResNet outperform baselines on the large-scale DomainNet dataset.All models perform less desirably when Infograph or Quickdraw is the target because of large domain shifts.
- Sentiment analysis: FADA achieves 78.9% accuracy on Amazon Review, showing effectiveness on linguistic as well as vision tasks under UFDA.The improvement from Model II to Model III is limited, indicating that linguistic features are harder to disentangle than visual features.
- Ablation analysis: Dynamic attention is essential: removing it reduces performance in most Digit-Five, Office-Caltech10, and Amazon Review experiments.It addresses differing source-domain convergence rates and adjusts contributions from source nodes.
6 CONCLUSION
The paper formulates UFDA, derives a generalization bound, and proposes FADA with dynamic attention for transferring knowledge to unlabeled target domains. Experiments across vision and linguistic benchmarks support FADA's efficacy, while feature disentanglement improves performance.
- The paper introduces unsupervised federated domain adaptation and derives a theoretical generalization bound for it.
- FADA transfers knowledge from distributed source domains to an unlabeled target domain using a novel dynamic attention schema.
- Feature disentanglement boosts FADA performance in unsupervised federated domain adaptation tasks.
- Extensive vision and linguistic benchmark evaluations demonstrate FADA's efficacy against several domain adaptation baselines.
7 NOTATIONS
This section provides explanations of the paper's notation and organizes those definitions in a dedicated notation table.
- The paper provides explanations for the notation used throughout the study.
- The notation material serves as a reference for interpreting the paper's formal descriptions.
- Table 6 is dedicated to the notations occurring in the paper.
8 MODEL ARCHITECTURE
The model-architecture appendix specifies the architectures used for digit recognition and image recognition, including layer dimensions and regularization settings.
- Table 7 lists the model architecture for digit recognition on the Digit-Five dataset.
- For convolutional layers, the architecture table reports input and output dimensions, kernel size, stride, and padding.
- The fully connected layers specify input and output dimensions, while dropout layers specify the probability of zeroing an element.
9 DETAILS OF DATASETS
The dataset-details section documents experimental data organization across Digit-Five, DomainNet, Office-Caltech10, and Amazon Review, including splits, image counts, and sentiment-label counts.
- For Digit-Five and DomainNet, the appendix provides the train/test split for each domain.
- For Office-Caltech10, the appendix reports the number of images in each domain.
- For Amazon Review, the appendix gives detailed counts of positive and negative reviews for each merchandise category.
- Table 8 describes the model architecture for cross-domain sentiment analysis on the Amazon Review dataset.
- Tables 9 describes the image-recognition model architecture for Office-Caltech10 and DomainNet.
- Table 10 summarizes the detailed number of samples used in the experiments.
10 PROOF OF THEOREM 2
Theorem 2 derives a weighted error bound for unsupervised federated domain adaptation by treating the source domains as a weighted mixture and relating source error, source–target divergence, and optimal joint risk. The bound assumes fully synchronized model weights in the federated setting.
- Theorem 3 states a weighted error bound for federated domain adaptation over a hypothesis class with VC-dimension d.
- The bound decomposes target performance into weighted source error, source–target divergence, and the optimal hypothesis risk on the source–target mixture.
- The proof constructs a combined source domain as a mixture of the N source domains with mixture weights α.
- The derivation bounds the mixture divergence using the corresponding divergences for the individual source domains and uses triangle-inequality arguments.
- The theoretical error bound holds only when model weights on all nodes are fully synchronized, because the source data are treated as a mixture despite remaining unshared.