Source-linked AI summary
Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning
Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, Ling-Yu Duan
TL;DR
Non-IID data causes local and global optimization to diverge, while direct aggregation can degrade the global model. FedFTG fine-tunes that model on the server using data-free distillation with generated hard samples and label-aware techniques, and experiments report superiority over SOTA methods and compatibility as a plugin for several optimizers.
Problem
Non-IID data causes drifted local models, knowledge forgetting, slow convergence, and degraded performance, while existing approaches overlook performance loss from direct global-model aggregation.
Method
FedFTG uses a server-side generator, hard-sample mining, customized label sampling, and class-level ensemble for data-free knowledge distillation from local models to the aggregated global model.
Results
FedFTG achieves superior performance against SOTA FL methods on five benchmarks and can enhance FedAvg, FedProx, FedDyn, SCAFFOLD, and MOON as a plugin.
Takeaways & Limitations
Because FedFTG modifies only server-side aggregation, it can be embedded into several local FL optimizers while using server computation to refine the global model.
Takeaways & Limitations
FedFTG roughly doubles per-round training time relative to FedAvg and is more applicable to cross-silo settings with sufficient server computation.
Abstract
from arXiv · showhide
Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. Data heterogeneity is one of the main challenges in FL, which results in slow convergence and degraded performance. Most existing approaches only tackle the heterogeneity challenge by restricting the local model update in client, ignoring the performance drop caused by direct global model aggregation. Instead, we propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG), which relieves the issue of direct model aggregation. Concretely, FedFTG explores the input space of local models through a generator, and uses it to transfer the knowledge from local models to the global model. Besides, we propose a hard sample mining scheme to achieve effective knowledge distillation throughout the training. In addition, we develop customized label sampling and class-level ensemble to derive maximum utilization of knowledge, which implicitly mitigates the distribution discrepancy across clients. Extensive experiments show that our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
1. Introduction
Non-IID data makes local optimization inconsistent with global optimization, causing drift, knowledge forgetting, slower convergence, and degraded performance. FedFTG addresses the overlooked aggregation problem by fine-tuning the aggregated global model on the server through data-free distillation from local models.
- Non-IID client data makes local empirical-loss minimization inconsistent with global empirical-loss minimization, causing drifted local models, knowledge forgetting, slower convergence, and degraded performance.
- Existing methods constrain local updates, but direct server aggregation can still ignore local knowledge incompatibility and induce knowledge forgetting in the global model.
- FedFTG fine-tunes the aggregated global model on the server through data-free knowledge distillation from local models, using an auxiliary generator to produce pseudo data.
- Hard sample mining explores samples that induce prediction disagreement between local and global models, helping correct model shift after aggregation.
- Customized label sampling and class-level ensemble address label-distribution shift by exploiting distribution correlations and maximizing knowledge utilization.
- FedFTG modifies only server-side aggregation, making it orthogonal to FedAvg, FedProx, FedDyn, SCAFFOLD, and MOON and usable as a plugin.
2. Related Work
Related work covers federated optimizers, federated knowledge distillation, and data-free knowledge distillation. FedFTG differs by transferring local-model knowledge to the global model through server-side hard-sample distillation and label-aware techniques.
- Federated Optimizer: Federated optimizers such as FedAvg, FedProx, SCAFFOLD, and FedDyn primarily align local and global models by modifying local training or objectives.
- Knowledge Distillation in Federated Learning: Federated distillation methods including FedDF, FedAUX, and FedBE use unlabeled data or generated global models to combine knowledge from local models.
- Data-Free Knowledge Distillation: Data-free knowledge distillation generates pseudo data from teacher-model responses and uses it to transfer teacher knowledge to a student model.
- FedFTG distinction: FedGen also uses a data-free generator, but applies it to regularize local training rather than fine-tuning the aggregated global model.
- FedFTG distinction: FedFTG adds hard-sample mining, customized label sampling, and class-level ensemble to transfer local knowledge under data heterogeneity.
3. Methodology
FedFTG fine-tunes the preliminary aggregated global model on the server through data-free distillation from local models, using generated pseudo-data and class-aware knowledge integration.
- Server-side global-model fine-tuning: FedFTG aggregates local models into a preliminary global model, then fine-tunes it on the server instead of broadcasting it directly.The server extracts knowledge from local models for this additional global-model update.
- Integration and optimization: FedFTG modifies only server-side aggregation, making it compatible with FedAvg, FedProx, FedDyn, and SCAFFOLD.The server update alternates generator training to explore hard samples with global-model updates to transfer local knowledge.
- Pseudo-data generation: A conditional generator produces pseudo-data from Gaussian noise and sampled class labels to capture clients’ input distributions without accessing their data.The generator is trained with semantic and diversity objectives so outputs fit class distributions while avoiding collapse.
- Data-free distillation: The global model minimizes prediction discrepancy with local models on generated pseudo-data, transferring local knowledge through weighted model-discrepancy losses.The discrepancy uses KL divergence between global and local softmax predictions, with weights α_t^{k,y} controlling each local model’s class-specific contribution.
- Hard sample mining: Hard sample mining makes the generator seek pseudo-data that exposes disagreement between the global and local models, because easy samples produce little distillation signal.The overall server objective is adversarial: the generator explores hard samples while the global model reduces model discrepancy, subject to fidelity and diversity terms.
- Label-distribution adaptation: Customized label sampling and class-level ensemble account for heterogeneous client label distributions and class-dependent knowledge importance.Majority classes receive higher pseudo-data generation probability, supporting distillation under class imbalance and distribution shift.
4. Experiments
Experiments evaluate FedFTG on heterogeneous CIFAR10 and CIFAR100 settings, comparing accuracy, convergence, robustness, ablations, and compatibility with existing optimizers. FedFTG consistently improves performance, with input-level generation and its proposed components contributing to the gains.
- Performance Comparison: FedFTG achieves the best test accuracy across CIFAR10 and CIFAR100 scenarios, surpassing the second-best method by at least 1.5%.Experiments are repeated over 3 random seeds.
- Performance Comparison: FedFTG reaches target accuracies efficiently, achieving the best results on CIFAR100 and reducing SCAFFOLD's required communication rounds in every scenario.FedDyn can initially rise faster, but its final accuracy is lower than FedFTG's on CIFAR10 and CIFAR100.
- Robustness to Heterogeneity and Participation: FedFTG remains strongest as data heterogeneity varies and as the fraction of active clients changes, with larger gains under extreme heterogeneity at β = 0.2.Accuracy increases for all methods as heterogeneity decreases and more clients participate.
- Orthogonality to FL Optimizers: Combining FedFTG with existing optimizers boosts each optimizer's performance; SCAFFOLD+FedFTG has the best accuracy, while FedDyn+FedFTG reaches targets in fewer rounds.FedAvg+FedFTG also exceeds the methods in Table 1 except SCAFFOLD.
- Ablation Study: Removing any FedFTG module or loss reduces and destabilizes accuracy, while replacing KL divergence with mean average square causes model collapse.The evaluated modules are hard sample mining, customized label sampling, and class-level ensemble.
- Ablation Study: Input-level generation significantly outperforms feature-level generation because feature-level fine-tuning updates only the last few global-model layers.FedFTG with feature-level generation still exceeds the other methods in Table 1.
5. Discussion
FedFTG introduces privacy and efficiency trade-offs: it may risk revealing client information and adds server-side computation, while requiring little additional communication.
- Privacy issue: FedFTG may violate FL privacy regulations because it recovers client training data on the server.The authors state that pseudo data captures high-level shared patterns and is not human-understandable, but label statistics may also leak privacy.
- Privacy issue: Adding noise below 10% to label statistics changes CIFAR10 performance by less than 0.1% under β = 0.3.
- Communication cost: FedFTG adds only negligible transmission cost by sending clients’ label statistics, and no extra cost when those statistics are reported before training.
- Computation efficiency: FedFTG requires about double FedAvg’s training time per communication round because it additionally trains the global model.It is therefore more applicable to cross-silo FL, where servers can have sufficient computation resources.
6. Conclusion
The paper proposes FedFTG, a data-free distillation method that fine-tunes aggregated global models using local-model knowledge and addresses label distribution shifts.
- FedFTG fine-tunes the global model through data-free knowledge distillation to boost federated learning performance.
- Hard sample mining transfers local-model knowledge effectively, while customized label sampling and class-level ensemble address label distribution shifts.
- Extensive experiments on five benchmarks validate FedFTG’s effectiveness.
7. Supplementary
The supplementary material examines long-tailed data, client distributions, generator architectures, convergence, and classifier architectures across additional experiments and configurations.
- Long-Tail Problem: Increasing class imbalance widens the gap between total-test and partitioned-test accuracy because models favor majority classes.Figure 7 uses CIFAR10 subsets generated by Dir(β), where smaller β indicates greater imbalance.
- Long-Tail Problem: Biased models generate pseudo data with less than 10% accuracy for minority classes, sometimes performing worse than random noise.The finding motivates customized label sampling during data generation.
- Client Data Distributions: Smaller Dirichlet β values produce more divergent client distributions, including clients with no data for some classes.Figure 9 visualizes per-class allocations across ten clients for different β values.
- Architectures and Hyperparameters: The supplementary material specifies baseline hyperparameters and generator dimensions for CIFAR10, CIFAR100, and larger-image datasets.Generator output dimensions are adjusted to match the classifier input, including 512 dimensions for ResNet18.
- Supplementary Results: At β = 0.6, FedFTG achieves the second-best CIFAR10 convergence and best CIFAR100 convergence by communication rounds.It also reduces the rounds required by SCAFFOLD.
- Supplementary Results: FedFTG yields the best test accuracy across all supplementary VGG11 and ResNet34 scenarios.Table 10 evaluates CIFAR10 with β = 0.3.