Source-linked AI summary
KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation
Hao-Zhe Feng, Zhaoyang You, Minghao Chen, Tianye Zhang, Minfeng Zhu, Fei Wu, Chao Wu, Wei Chen
TL;DR
Privacy-constrained UMDA cannot rely on direct source-data access, and conventional approaches face domain-distance, communication, privacy, and negative-transfer challenges. KD3A adapts through knowledge distillation across source-domain models, combining consensus learning, dynamic weighting, and decentralized distance optimization. On DomainNet, it outperforms state-of-the-art UMDA approaches, while reducing communication costs and remaining robust to negative transfer.
Problem
Conventional UMDA assumes direct access to source domains, but privacy-preserving settings keep source data and computations decentralized, limiting data-based distance calculations and other adaptation methods.
Source: §1 · p. 1Method
KD3A performs decentralized domain adaptation through knowledge distillation on source-domain models, using Knowledge Vote, Consensus Focus, and BatchNorm MMD.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
KD3A reduces communication cost 100× compared with other decentralized UMDA methods and is robust to negative transfer and privacy leakage attacks.
Source: §1 · p. 2, §6 · p. 9The formulation assumes a C-way classification task shared by the source and target domains.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Conventional unsupervised multi-source domain adaptation (UMDA) methods assume all source domains can be accessed directly. This neglects the privacy-preserving policy, that is, all the data and computations must be kept decentralized. There exists three problems in this scenario: (1) Minimizing the domain distance requires the pairwise calculation of the data from source and target domains, which is not accessible. (2) The communication cost and privacy security limit the application of UMDA methods (e.g., the domain adversarial training). (3) Since users have no authority to check the data quality, the irrelevant or malicious source domains are more likely to appear, which causes negative transfer. In this study, we propose a privacy-preserving UMDA paradigm named Knowledge Distillation based Decentralized Domain Adaptation (KD3A), which performs domain adaptation through the knowledge distillation on models from different source domains. KD3A solves the above problems with three components: (1) A multi-source knowledge distillation method named Knowledge Vote to learn high-quality domain consensus knowledge. (2) A dynamic weighting strategy named Consensus Focus to identify both the malicious and irrelevant domains. (3) A decentralized optimization strategy for domain distance named BatchNorm MMD. The extensive experiments on DomainNet demonstrate that KD3A is robust to the negative transfer and brings a 100x reduction of communication cost compared with other decentralized UMDA methods. Moreover, our KD3A significantly outperforms state-of-the-art UMDA approaches.
1. Introduction
KD3A addresses privacy-constrained UMDA by adapting through knowledge distillation on source-domain models. It combines consensus learning, domain weighting, and decentralized distance optimization, and reports strong DomainNet accuracy alongside lower communication costs and robustness to negative transfer.
1. Introduction
- Decentralized UMDA prevents access to source data for pairwise domain-distance calculations, while communication costs, privacy risks, and bad sources constrain conventional methods.Irrelevant or malicious sources can cause negative transfer.Source: §1 · p. 1
- KD3A performs decentralized domain adaptation by distilling knowledge from models trained on different source domains.Its components are Knowledge Vote for domain consensus, Consensus Focus for identifying malicious or irrelevant domains, and BatchNorm MMD for decentralized H-divergence optimization.Source: §1 · pp. 1–2
- KD3A reduces communication cost 100× compared with other decentralized UMDA methods and is robust to privacy leakage attacks.It also assigns low weights to malicious or irrelevant domains, supporting robustness to negative transfer.Source: §1 · p. 2
- On the large-scale DomainNet dataset, KD3A achieves 51.1% accuracy and significantly outperforms the state-of-the-art UMDA approaches, according to the reported result.Source: §1 · p. 2
2. Related work
Prior UMDA work uses distance minimization, adversarial training, or model-level distillation, but decentralization constrains data access and communication. Federated learning and federated domain adaptation offer distributed approaches, with trade-offs in communication, privacy, and resilience to negative transfer.
2.1. Unsupervised Multi-Source Domain Adaptation
- MMD variants require pairwise source-target data calculations, while adversarial training exchanges model parameters after every batch and consumes substantial communication.The cited MMD variations include multikernel MMD, class-weighted MMD, and domain-crossing MMD.Source: §2.1 · p. 2
- Teacher-student knowledge distillation ensembles source-trained teachers on the target, but conventional strategies may fail with irrelevant or malicious source domains.Prior work reports this strategy outperforming other UMDA methods in practice.Source: §2.1 · p. 2
2.2. Federated Learning
- Federated learning aggregates local-model updates into a global model, with more communication rounds associated with better performance but higher cost and privacy leakage.Source: §2.2 · p. 2
- FADA applies adversarial training without accessing data but has high communication costs and vulnerability to privacy leakage; Model Adaptation and SHOT address single-source, not multi-source, negative transfer.Source: §2.2 · p. 2
3. KD3A: Decentralized Domain Adaptation via Knowledge Distillation
KD3A adapts models across decentralized source domains through knowledge distillation, without accessing source data. Its pipeline creates consensus knowledge, weights source domains by their consensus contribution, and uses BatchNorm statistics to optimize domain distance.
3. KD3A: Decentralized Domain Adaptation via Knowledge Distillation
- KD3A applies knowledge distillation without accessing source-domain data, for a target task shared with the source domains.The decentralized setting keeps source data local and makes trained source models and unlabeled target data accessible during communication rounds.Source: §3 · p. 3
3.1. Extending Source Domains With Consensus Knowledge
- A distilled model trained on target examples with source-model consensus extends the source domains, and KD loss optimizes its task risk.The target model is then formed by aggregating the source models, including the distilled model.Source: §3.1 · p. 3
3.2. Knowledge Vote: Producing Good Consensus
- Knowledge Vote filters uncertain predictions, votes among the remaining models for a consensus class, then averages supporting models’ predictions.It records the number of supporting domains and gives examples with no models surviving the confidence gate a low weight.Source: §3.2 · p. 4
3.3. Consensus Focus: Against Negative Transfer
- Consensus Focus weights source domains by their marginal contribution to consensus quality, assigning low focus to domains whose removal does not reduce quality.Its quality measure favors consensus supported by more domains with higher confidence; unlike H-divergence reweighting, it does not require source data and uses label information to identify malicious domains.Source: §3.3 · pp. 4–5
3.4. BatchNorm MMD: Decentralized Optimization Strategy of H−divergence
- BatchNorm MMD uses mean and variance parameters from source-model BatchNorm layers to optimize H-divergence without accessing source data.The paper notes that fully connected-layer features may lose detailed 2-D information.Source: §3.4 · p. 5
3.5. The Algorithm of KD3A
- KD3A trains source models, applies Knowledge Vote and Consensus Focus, then optimizes the target model with BatchNorm MMD.The confidence gate is KD3A’s only hyperparameter: too high a threshold eliminates most target data, while too low a threshold reduces consensus quality; the paper gradually increases it from about 0.8 to 0.95.Source: §3.5 · pp. 5–6
4. Generalization Bound For KD3A
KD3A’s generalization analysis relates its target model to an extended source domain formed through Knowledge Vote and weighted through Consensus Focus. The bound’s quality depends on consensus quality, with tighter-bound conditions distinguishing useful from harmful source domains.
4. Generalization Bound For KD3A
- The theorem defines its bound using extended source domains produced through Knowledge Vote and domain weights assigned by Consensus Focus.These are the quantities specified alongside the target model in the theorem statement.Source: §4 · p. 6
- KD3A’s bound depends on consensus quality: Knowledge Vote improves consensus for good sources, while Consensus Focus filters knowledge from bad sources.The analysis describes these mechanisms as heuristically achieving the bound’s two conditions.Source: §4 · p. 6
- The KD3A bound is tighter than the original bound when the task-risk gap between the distillation domain and target is below the stated source-wise upper bound.The condition must hold for all source domains.Source: §4 · p. 6
- Tighter-bound conditions favor good sources with low H-divergence and low optimal task risk, while high-divergence, high-risk domains should be filtered.The analysis says KD3A keeps the target-risk bound close to good domains and away from bad domains.Source: §4 · p. 6
5. Experiments
On DomainNet, KD3A leads the reported UMDA comparisons, while experiments also examine negative transfer, communication rounds, and privacy leakage. Consensus Focus gives low weights to bad domains, and KD3A maintains strong performance with less communication than FADA.
5.1. Domain Adaptation Performance
- 51.1% accuracy: KD3A significantly outperforms all DomainNet baselines and matches oracle performance on Clipart and Sketch.Table 1 reports UMDA accuracy on DomainNet.Source: Table 1 · p. 7, §5.1 · p. 7
- All UMDA methods fail on Quickdraw, where the oracle reaches 66.8% and the source-only baseline reaches 13.3%.KD3A slightly outperforms source-only there by using pseudo-labels from good samples and assigning low weights to bad samples.Source: §5.1 · p. 7
- Knowledge Vote, Consensus Focus, and BatchNorm MMD each improve accuracy; Knowledge Vote contributes most, and KD3A can work without BatchNorm MMD.The ablation study reports these component contributions across target domains.Source: Figure 3 · p. 7, §5.1 · p. 8
5.2. Robustness To Negative Transfer
- Consensus Focus assigns low weights to irrelevant domains and 5% to MA-30; it outperforms alternative weighting strategies on malicious domains.Domain-drop outperforms the other methods for IR-qdr and MA-(30,50), while Consensus Focus can use information from less malicious domains.Source: §5.2 · p. 8, Table 2 · p. 8
5.3. Communication Efficiency And Privacy Security
- At r = 1, KD3A uses 100× less communication than FADA; reducing rounds from 100 to 1 drops its accuracy by only 0.9%.At r = 0.5 and r = 0.2, KD3A still achieves state-of-the-art results while FADA falls below source-only.Source: Table 3 · p. 8, §5.3 · p. 9
- The gradient leakage attack recovers source images used in FADA, while KD3A is reported robust to the attack.The authors attribute KD3A’s robustness to its low communication cost.Source: Figure 5 · p. 8, §5.3 · p. 9
6. Conclusions
KD3A performs domain adaptation through knowledge distillation without accessing source-domain data. DomainNet experiments report improved UMDA performance, robustness to negative transfer, and advantages in communication efficiency and privacy-attack robustness.
6. Conclusions
- KD3A performs domain adaptation through knowledge distillation without accessing source-domain data.The conclusion presents this as its main approach to decentralized UMDA.Source: §6 · p. 9
- Experiments on large-scale DomainNet show KD3A outperforms state-of-the-art UMDA approaches and is robust to negative transfer.Source: §6 · p. 9
- KD3A has a communication-efficiency advantage and is robust to the privacy-leakage attack.Source: §6 · p. 9
7. Appendix
The appendix supports the KD3A analysis with a knowledge-distillation argument and generalization bounds. It also derives an empirical-risk learning bound for finite training samples.
7.1. Appendix A
- Appendix A applies Pinsker’s inequality to class probabilities for target inputs when relating knowledge-distillation loss to model outputs.Source: §7.1 · p. 9
7.2. Appendix B
- Proposition 1 bounds the task-risk gap between the extended distillation source domain and the target domain.The appendix then derives an empirical-risk learning bound using NT sampled training data.Source: §7.2 · pp. 9–10
- The empirical bound’s probability statement uses δ, and its model-complexity term refers to VC-dimension d.The appendix defines d as the VC-dimension of model space H.Source: §7.2 · p. 10
7.3. Appendix C
Appendix C states a condition under which the KD3A bound is tighter than the original bound: the knowledge-distillation bound must be tighter than each single-source bound.
7.3. Appendix C
- The KD3A bound is tighter than the original bound when the knowledge-distillation bound is tighter than the single-source bound for every source domain.The condition is stated for all source domains and all target hypotheses.Source: §7.3 · p. 10
- The appendix frames the original bound as a weighted combination of source-domain bounds and the KD3A bound as combining that original bound with the distillation bound.The comparison therefore depends on the distillation bound being tighter than each source-specific bound.Source: §7.3 · p. 10
- The tighter-bound condition is expressed through a bound on the task-risk gap between the knowledge-distillation domain and the target domain.The supplied passages identify this gap and state that it must satisfy a condition, but do not provide the full condition.Source: §7.3 · pp. 10–11
7.4. Appendix D: Representation Invariant Bounds For KD3A.
Appendix D discusses the bound’s dependence on the joint risk term and revises the analysis to account for representation changes during H-divergence optimization. It also states that KD3A applies to the revised bounds through BatchNorm-MMD distance minimization.
7.4. Appendix D: Representation Invariant Bounds For KD3A.
- The authors replace the original bound with a bound using ˜H-divergence and a constant term C because H-divergence optimization may change the representation space and λ.They treat λ as constant under fixed model backbones, while acknowledging that representation changes may alter it.Source: §7.4 · p. 11
- 92.0% accuracy is reported for KD3A on Digit-5, where the table caption states that it outperforms all other baselines.Source: Table 6 · p. 12
- KD3A also works with the revised bounds because ˜H-divergence can be optimized by minimizing the BatchNorm-MMD distance.The authors state that the proof for the new bounds applies directly.Source: §7.4 · p. 12
7.5. Appendix E: The Implementation of BatchNorm MMD
Appendix E presents a computation-efficient implementation of BatchNorm MMD. It replaces direct optimization across BatchNorm layers with a global optimum that is substituted into the target model.
7.5. Appendix E: The Implementation of BatchNorm MMD
- Directly optimizing the BatchNorm-MMD loss requires traversing all BatchNorm layers and is described as time-consuming.Source: §7.5 · p. 12
- The alternative derives the global optimal solution for each layer’s target-domain parameter and substitutes the solution into every BatchNorm layer of the target model.The authors describe this as a two-step computation-efficient method.Source: §7.5 · p. 12
- The authors report that the computation-efficient implementation achieves the same performance as the original maximization step.They characterize the implementation as potentially heuristic.Source: §7.5 · p. 12
7.6. Appendix F
Across Amazon Review, Digit-5, and Office-Caltech10, KD3A is reported to outperform other UMDA and advanced decentralized UMDA methods. The appendix also describes its implementation settings and mixup augmentation experiments.
7.6.1. IMPLEMENTATION DETAILS.
- Experiments use a 3-layer MLP for Amazon Review, a 3-layer CNN for Digit-5, and ImageNet-pretrained ResNet101 for Office-Caltech10 and DomainNet.The method comparisons use the same backbones and training epochs; SGD and cosine learning-rate decay are used.Source: §7.6.1 · p. 13
- Mixup with α = 0.2 is used as a unified augmentation strategy, and KD3A is reported to achieve good results even without data augmentation.Results are compared both with and without augmentation.Source: §7.6.1 · p. 13
7.6.2. RESULTS ON AMAZON REVIEW, DIGIT-5 AND OFFICE-CALTECH10.
- KD3A outperforms other UMDA methods and advanced decentralized UMDA methods on Amazon Review, Digit-5, and Office-Caltech10.The authors report better consensus knowledge on hard domains such as MNISTM in Digit-5, with a large margin over other methods.Source: §7.6.2 · p. 13