Source-linked AI summary
One-shot federated unsupervised domain adaptation with Smoothed Knowledge Distillation and teacher refinement
Ali Abedi, Q. M. Jonathan Wu, Ning Zhang, Farhad Pourpanah
TL;DR
FUDA seeks to adapt models to unlabeled target domains while preserving privacy, yet existing approaches face unstable pseudo-labels, complex aggregation, and communication demands. SKD-ETR uses smoothed distillation from source-model teachers and EMA-based teacher refinement; evaluations report competitive results across three benchmarks with fewer communication rounds and trainable parameters.
Problem
Existing FUDA methods can struggle with noisy or inconsistent pseudo-label supervision, heuristic aggregation, and communication inefficiency.
Source: §1 · p. 2Method
SKD-ETR trains a randomly initialized student on unlabeled target data using smoothed ensemble pseudo-labels, then updates source-model teachers toward the student with EMA.
Source: §1 · pp. 3–4, §3.5 · p. 10Results
SKD-ETR achieves superior or comparable performance to state-of-the-art methods on OfficeHome, Office-Caltech, and DomainNet while requiring fewer trainable parameters and communication rounds.
Source: §6 · pp. 26–27Takeaways & Limitations
The reported results support SKD-ETR as a communication- and computation-efficient approach for practical federated settings.
Source: §6 · p. 27Random initialization can hinder adaptation when the target distribution differs drastically from all sources, as in the challenging Quickdraw domain.
Source: §5.5 · p. 25
Abstract
from arXiv · showhide
Federated unsupervised domain adaptation (FUDA) addresses the challenge of adapting models to an unlabeled target domain using decentralized source domains while preserving data privacy. However, existing FUDA methods typically require multiple communication rounds, rely on complex aggregation strategies, and often struggle with noisy pseudo labels and inconsistent source knowledge. To address these challenges, we propose SKD-ETR, a novel one-shot FUDA framework that combines Smoothed Knowledge Distillation (SKD) and Exponential moving average-based Teacher Refinement (ETR). SKD trains a student model on the target domain using smoothed soft pseudo labels generated by the ensemble of source models. This reduces overconfidence, mitigates noise, and improves robustness. ETR further refines each source model by interpolating its parameters toward the student via exponential moving average (EMA), thereby transferring target-domain knowledge back to the teachers. This bidirectional refinement enhances pseudo-label quality and student generalization without additional communication overhead. SKD-ETR eliminates complex aggregation by initializing the student model randomly and performing a single-round distillation process. Extensive experiments on OfficeHome, Office-Caltech, and DomainNet demonstrate that SKD-ETR achieves competitive performance while being communication- and computation-efficient, and robust under noisy supervision. The code for this paper is available at https://github.com/A-Abedi/SKD_ETR.
1. Introduction
FUDA must adapt to unlabeled target domains while preserving source-data privacy, but existing approaches face communication costs, unstable pseudo-labels, and complex aggregation. SKD-ETR combines smoothed student distillation with EMA teacher refinement, and evaluations report competitive performance across standard benchmarks.
1. Introduction
- One-shot FL [7] uses a single communication round, reducing bandwidth requirements and repeated exposure of intermediate model states.The paper links this design to reduced privacy-leakage risk and suitability for resource-constrained, privacy-sensitive settings.Source: §1 · p. 2
- Existing FUDA methods [21] can propagate noisy supervision or discard useful signals, use heuristic aggregation, and incur communication costs.These limitations motivate a framework that stabilizes pseudo-labels while reducing reliance on costly aggregation.Source: §1 · p. 3
- SKD trains a student on unlabeled target data using ensemble-generated soft pseudo-labels and label smoothing in the KL loss to mitigate noise.The paper describes smoothing as an implicit regularizer that improves student robustness and generalization.Source: §1 · pp. 3–4
- The framework is presented as one-shot FUDA that uses knowledge distillation with unlabeled target data and pretrained source models.The paper summarizes its contributions in the introduction before outlining the remaining sections.Source: §1 · pp. 3–4
- ETR moves each source model's parameters toward the student with EMA, transferring target-domain knowledge to teachers without additional communication.The refined teachers can generate more reliable pseudo-labels.Source: §1 · pp. 3–4
- SKD-ETR achieves competitive performance on standard benchmarks, with the paper reporting consistent outperformance of existing approaches.Evaluations cover OfficeHome, Office-Caltech, and DomainNet.Source: §1 · pp. 3–4
2. Related Work
The related-work review places SKD-ETR alongside multi-source and federated domain adaptation, as well as knowledge-distillation approaches. It distinguishes methods by their data access, adaptation strategy, and use of teacher knowledge.
2.1. Multi-source Unsupervised Domain Adaptation
- MSUDA methods such as SSD, SImpAl, and MDAMA adapt using multiple source domains, but assume source-data access during adaptation.SSD selects informative source samples and ranks domains through two-step distillation; SImpAl enforces classifier agreement, while MDAMA uses a classifier-aware margin discrepancy.Source: §2.1 · p. 5
- MSFDA methods adapt from pretrained source models without source-data access, using approaches such as frozen classifiers, source weighting, or transformer-based self-distillation.Examples include SHOT, DECISION, and TransDA [37].Source: §2.1 · p. 5
2.2. Federated Unsupervised Domain Adaptation
- FUDA distributes source models across clients for privacy-preserving adaptation; methods include adversarial debiasing, confidence-weighted distillation, and one-shot aggregation.Examples include FADE, FDKD, KD3A, FDAC, and SEA+MSPL [21].Source: §2.2 · p. 5
2.3. Knowledge Distillation
- Knowledge-distillation approaches for adaptation vary in how they form or refine pseudo-labels, including consensus voting, teacher weighting, dual distillation, and self-distillation.Examples include KD3A, FDKD, SSD, DECISION, and TransDA [37].Source: §2.3 · p. 6
3. Methodology
The method adapts a student to an unlabeled target domain using privately trained models from heterogeneous source domains. SKD transfers ensemble supervision to the student, while ETR moves teachers toward the student through EMA updates.
3.1. Problem Formulation
- Each client trains on a distinct, private labeled source distribution, while the objective is to learn a student that performs well on an unlabeled target domain.Raw data, features, and statistics are not shared between clients or with the server.Source: §3.1 · p. 6
3.2. Model Overview
- SKD-ETR proceeds in two stages: source teachers guide student training on unlabeled target data, then EMA updates refine the teachers without additional communication rounds.The refined teachers are used during training; only the student is retained afterward.Source: Figure 1 · p. 7, §3.2 · p. 8
3.3. Model Architecture
- The model combines a frozen pre-trained DINOv2 backbone, a trainable bottleneck, and a trainable classification head.The bottleneck transforms backbone features, and the head maps them to class logits.Source: §3.3 · p. 8
3.4. Source Model Training and Server-Side Aggregation
- The framework uses a randomly initialized student for server-side adaptation instead of aggregating source models, avoiding complex aggregation and source-domain bias in initialization.Clients send trained model parameters to the server, where source models guide target-domain adaptation as teachers.Source: §3.4 · p. 9
3.5. Smoothed Knowledge Distillation
- SKD trains the student on unlabeled target data by matching its predictions to smoothed ensemble pseudo-labels with a KL-divergence loss.Smoothing softens overconfident teacher predictions, and the loss matches the softened distribution rather than hard labels.Source: §3.5 · pp. 9–10
3.6. Teacher Refinement with EMA
- ETR moves source-model parameters toward the target-adapted student, reducing teacher–student prediction disagreement on target data and, under smoothness assumptions, tightening a target-risk discrepancy term.The refined teachers provide supervision in subsequent iterations without additional client communication; only the student is retained for inference.Source: §3.6 · pp. 11–12
3.6.1. Effect on Decision Boundaries
- Under the stated smoothness assumptions, moving a teacher’s parameters toward the student contracts hypothesis discrepancy; the authors describe the resulting decision boundary as softened toward the student’s boundary.The proposed interpretation links the student’s target-domain supervision to smoother teacher boundaries and better generalization to unseen target samples.Source: §3.6 · p. 11, §3.6.1 · p. 14
4. Experimental settings
The experiments evaluate SKD-ETR on three multi-domain benchmarks using a leave-one-domain-out protocol, with each source domain assigned to a client and top-1 target-domain accuracy reported.
4.1. Datasets
- OfficeHome, Office-Caltech, and DomainNet span 65, 10, and 345 categories, respectively, across four, four, and six domains.OfficeHome has approximately 15,500 images; Office-Caltech has 2,533; DomainNet has 48,129.Source: §4.1 · p. 14
4.2. Implementation Details
- Local clients train for 20 epochs, and the server trains the student for 10 epochs using SKD.Training uses SGD with momentum 0.9, learning rate 3e-2, and batch size 32; the smoothing factor is 0.5 and EMA momentum is 0.15.Source: §4.2 · pp. 14–15
4.3. Baselines and Evaluation Metric
- Each experiment treats one domain as the unlabeled target and the remaining domains as labeled sources, with one client per source domain.Methods are evaluated using top-1 target-domain accuracy, reported per domain and averaged across domains.Source: §4.2 · p. 15, §4.3 · p. 16
4.2. Implementation Details
- The leave-one-domain-out experiments use random cropping and horizontal flipping, ImageNet normalization, and the same backbone and training setup across datasets.The bottleneck and head follow the design in [43].Source: §4.2 · p. 15
4.3. Baselines and Evaluation Metric
- The baselines cover MSUDA, MSFDA, and FUDA; FedSD2C and FedPFT are one-shot FL approaches extended to FUDA, while SEA+MSPL is one-shot FUDA.The listed FUDA baselines also include FADA, FADE, KD3A, FDKD, and FDAC.Source: §4.3 · p. 15
5.1. Results
SKD-ETR achieves the highest average accuracy on OfficeHome and DomainNet and performance comparable to the best baseline on Office-Caltech. The adaptation also changes student representations and decision boundaries on OfficeHome.
5.1. Results
- SKD-ETR achieves the highest average accuracy on OfficeHome and DomainNet, and is comparable to the best baseline on Office-Caltech while reducing trainable parameters, communication overhead, and sharable model size.Source: §5.1 · p. 16
- On OfficeHome, t-SNE visualizations and high Flip Rate and KL Divergence values indicate changed student representations and predictions after adaptation.The authors report improved feature alignment and class separability in the target domain.Source: §5.1 · p. 16
5.2. Ablation Study
The ablation study examines SKD and ETR, student initialization, pseudo-labeling strategies, and backbone architectures. Its findings support both components and the chosen random initialization and smoothed supervision.
5.2.1. Effect of Different Components of the Model
- Removing either SKD or ETR reduces performance, while removing both yields the lowest accuracy, supporting the contribution of both components.Source: §5.2.1 · p. 18, Table 5 · p. 18
5.2.2. Effect of Different Student Model Initializations
- Random initialization achieves the highest average performance across the four OfficeHome domains, whereas source-model parameter transfer can introduce domain-specific bias or noisy supervision.The authors use random initialization in subsequent experiments.Source: §5.2.2 · p. 19, Table 6 · p. 19
5.2.3. Effect of Pseudo Labeling Strategies
- SKD achieves the highest average performance across domains by retaining all target samples and smoothing supervision, unlike alternatives that can discard data or bias predictions toward overconfidence.The compared alternatives are confidence thresholding, confidence weighting, and entropy weighting.Source: §5.2.3 · p. 19, Table 7 · p. 20
5.2.4. Effect of Different Backbones
- SKD-ETR improves performance across ResNet-50, ResNet-101, ViT, and DINOv2 backbones, with more pronounced gains on transformers; DINOv2 achieves the highest average performance.All backbones are frozen, and the comparison baseline uses KL divergence without SKD and ETR.Source: §5.2.4 · p. 20, Table 8 · p. 20
5.3. Sensitivity Analysis
The sensitivity analysis studies smoothing, EMA decay, parameter noise, and teacher refinement. It reports preferred parameter settings, robustness under injected noise, and improved teacher accuracy and alignment after refinement.
5.3.1. Effect of
- ϵ = 0.5 achieves the best target performance, balancing preservation of source signals with reduction of pseudo-label noise.Lower smoothing can retain incorrect high-confidence predictions, while very high smoothing can obscure class distinctions.Source: §5.3.1 · p. 21, Figure 5 · p. 21
- Entropy-adaptive and time-decayed smoothing generally underperform fixed ϵ, which the authors find more stable under long-tailed and highly divergent target domains.Entropy-adaptive smoothing yields slight improvements in some extreme-shift scenarios.Source: §5.3.1 · pp. 21–22, Table 9 · p. 22
5.3.2. Effect of α
- α = 0.15 achieves the best results, balancing retained source-model knowledge with incorporation of generalized student knowledge.Source: §5.3.2 · p. 22, Figure 6 · p. 22
5.3.3. Robustness to Parameter Noise
- SKD-ETR remains stable with minor performance drops at Gaussian noise levels σ = 0.01 and σ = 0.02, and retains reasonable accuracy at σ = 0.05.The authors report much larger degradation for FedAvg under the same setting.Source: §5.3.3 · p. 23, Table 10 · p. 23
5.3.4. Effect of ETR on Teacher models
- ETR improves every teacher's target-domain accuracy and reduces pairwise KL divergence, although refined teachers remain less accurate than the adapted student.For Art as target, the Clipart teacher improves from 72.7% to 81.2%; the student is retained for inference for its accuracy–efficiency trade-off.Source: §5.3.4 · pp. 23–24, Figure 7 · p. 24, Table 11 · p. 24
5.4. Computational Overhead and Resource Efficiency
SKD-ETR uses modest computational and energy resources for adaptation, while teacher-side pseudo-labeling and refinement scale linearly with the number of clients and add no inference cost after training.
5.4. Computational Overhead and Resource Efficiency
- Each teacher requires 70.7 GFLOPs per forward–backward pass, 364 MB of parameters, 0.81 GB peak GPU memory, and 1,560 seconds of training time.The resource measurements are for an NVIDIA GeForce GTX 1080 Ti GPU; Table 12 summarizes teacher and student resource usage on OfficeHome.Source: §5.4 · p. 24, Table 12 · p. 25
- Student training consumes approximately 0.05 kg CO2, and inference takes 0.01 seconds per batch of 32 samples.The adaptation increases memory usage and training time moderately while maintaining the teachers' FLOP count and parameter size.Source: §5.4 · p. 25
- Ensemble pseudo-label generation and EMA refinement scale linearly with client count M, run only once during adaptation, and leave inference cost at O(1).Teachers remain in non-training mode and contribute only lightweight forward passes during adaptation.Source: §5.4 · p. 25
5.5. Limitations and Failure Modes
The reported weakness is concentrated in extreme domain shifts: random initialization can hinder adaptation, with Quickdraw standing out as DomainNet's largest-distance outlier.
5.5. Limitations and Failure Modes
- Quickdraw has the largest pairwise Fréchet distances among DomainNet domains, marking it as an outlier in the measured domain structure.The distances are computed using embeddings from a frozen DINOv2 backbone.Source: §5.5 · p. 25, Figure 8 · p. 26
- Random initialization avoids source-specific bias but can hinder adaptation when the target distribution differs drastically from all sources.For Quickdraw, SKD-ETR performance declines relative to aggregation-based SEA+MSPL [21], which benefits from source-informed initialization.Source: §5.5 · pp. 25–26
- SKD-ETR nevertheless attains the best average performance overall, despite the Quickdraw challenge..Source: §5.5 · pp. 25–26
- Adaptive initialization and progressive distillation are proposed as future strategies for handling extreme domain shifts.Source: §5.5 · p. 26
6. Conclusion and Future Work
SKD-ETR combines smoothed student distillation with EMA-based teacher updates for one-shot FUDA without source-data access or complex aggregation. Experiments report competitive benchmark performance with fewer trainable parameters and communication rounds.
6. Conclusion and Future Work
- SKD regularizes student training against noisy pseudo labels with label smoothing, while ETR updates source models using student parameters to improve subsequent pseudo-labels without additional communication.The paper also provides a theoretical justification for the EMA update and reports that the framework avoids complex aggregation mechanisms.Source: §6 · p. 26
- Experiments on OfficeHome, Office-Caltech, and DomainNet show superior or comparable performance to state-of-the-art methods while requiring significantly fewer trainable parameters and communication rounds.The authors characterize the approach as communication- and computation-efficient in practical federated settings.Source: §6 · pp. 26–27
- Future work will extend SKD-ETR to heterogeneous model architectures and semi-supervised target domains, and explore adaptive smoothing and model compression.Source: §6 · p. 27