PaperlayerView exact revision on arXiv

Source-linked AI summary

DENSE: Data-Free One-Shot Federated Learning

Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, Chao Wu

arXiv:2112.12371v2cs.LGcs.CV

TL;DR

One-shot FL lowers communication but existing methods can require auxiliary or public data, additional information transfers, or homogeneous models. DENSE uses data generation and model distillation to train a global model without auxiliary data while supporting model heterogeneity, and experiments report superior performance, including a 5.08% improvement over Fed-ADI on CIFAR10. The paper notes potential privacy attacks as a future direction.

  • Problem

    Existing one-shot FL methods face practical limitations including public or auxiliary dataset requirements, additional data transfers, and homogeneous client models.

  • Method

    DENSE is a two-stage data-free one-shot FL framework that trains a generator from ensemble client models and then distills their knowledge into a global model.

  • Results

    DENSE achieves the highest accuracy across the evaluated datasets and outperforms Fed-ADI by 5.08% on CIFAR10 when α = 0.3.

  • Takeaways & Limitations

    DENSE provides a practical data-free one-shot FL framework that requires no auxiliary dataset or additional information beyond model parameters and supports heterogeneous models.

  • Takeaways & Limitations

    The authors identify potential privacy attacks in one-shot FL as an unresolved direction for future work.

Abstract

from arXiv · show

One-shot Federated Learning (FL) has recently emerged as a promising approach, which allows the central server to learn a model in a single communication round. Despite the low communication cost, existing one-shot FL methods are mostly impractical or face inherent limitations, \eg a public dataset is required, clients' models are homogeneous, and additional data/model information need to be uploaded. To overcome these issues, we propose a novel two-stage \textbf{D}ata-fre\textbf{E} o\textbf{N}e-\textbf{S}hot federated l\textbf{E}arning (DENSE) framework, which trains the global model by a data generation stage and a model distillation stage. DENSE is a practical one-shot FL method that can be applied in reality due to the following advantages: (1) DENSE requires no additional information compared with other methods (except the model parameters) to be transferred between clients and the server; (2) DENSE does not require any auxiliary dataset for training; (3) DENSE considers model heterogeneity in FL, \ie different clients can have different model architectures. Experiments on a variety of real-world datasets demonstrate the superiority of our method.For example, DENSE outperforms the best baseline method Fed-ADI by 5.08\% on CIFAR10 dataset.

1 Introduction

One-shot FL reduces federated learning to a single communication round, but existing approaches can require public or distilled data and remain difficult to apply in practical settings. DENSE addresses these constraints with a two-stage framework that generates data and distills model knowledge while supporting heterogeneous client models.

  • Motivation: One-shot FL permits only one communication round, reducing communication compared with the original frequently communicating FL framework.It is motivated by settings such as model markets and concerns about interception or reconstruction attacks from frequent communication.
  • Limitations of Existing Methods: Existing one-shot FL methods may require public datasets, transmitted distilled data, or other impractical information transfers.These requirements can add communication cost and potential privacy leakage, and may be unsuitable for sensitive domains.
  • DENSE Framework: DENSE trains a global model through data generation followed by model distillation using an ensemble of client-uploaded local models.The ensemble models train a generator for synthetic data, which is then used with the ensemble models to train the global model.
  • Practical Setting: DENSE requires no auxiliary dataset or additional transferred information beyond model parameters and supports clients with different model architectures.Unlike FedAvg-based aggregation, DENSE does not require averaging model parameters, enabling heterogeneous models.

2 Data-Free One-Shot Federated Learning

DENSE performs one-shot federated learning through data generation followed by model distillation, without auxiliary data and while supporting heterogeneous client models.

  • Framework Overview: DENSE trains the global model in two stages: an auxiliary generator produces synthetic data, then ensemble models and synthetic data train the global model.The server receives client models and uses their ensemble in both stages.
  • Data Generation: The generator is trained from client-model outputs rather than direct access to client training data, aiming for similar data distribution without directly computing data similarity.The generator uses random noise and labels, while client-model logits provide the similarity signal.
  • Data Generation: DENSE combines cross-entropy, batch-normalization, and boundary-support objectives to improve synthetic-data similarity, stability, and transferability.Boundary support encourages samples between ensemble and global-model decision boundaries, where they better support knowledge transfer.
  • Data Generation: Using only cross-entropy can yield low performance because non-IID client data may make the generator unstable, trapped in sub-optimal minima, or overfit synthetic data.The paper identifies this as a limitation of the CE-only generator objective.
  • Data Generation: The generated data are intended to have similar features but differ from client training data, reducing the probability of leaking sensitive client information.The paper distinguishes visual privacy from distributional similarity for utility.
  • Model Distillation: Unlike parameter averaging, averaging logits can support both heterogeneous and homogeneous federated learning systems.DENSE places no restriction on client local models and can combine with arbitrary local training techniques.

3.1 Experimental Setup

Experiments evaluate one-shot FL on six real-world datasets under Dirichlet-generated non-IID partitions, comparing DENSE with FedAvg, FedDF, Fed-DAFL, and Fed-ADI.

  • Datasets: Experiments use MNIST, FMNIST, SVHN, CIFAR10, CIFAR100, and Tiny-ImageNet.
  • Data partitioning: Dirichlet sampling creates non-IID client partitions, with smaller α producing more highly skewed data.
  • Baselines: DENSE is compared with FedAvg, FedDF, Fed-DAFL, and Fed-ADI, while methods requiring auxiliary models or datasets are omitted.
  • Implementation: Client training uses SGD with momentum 0.9, learning rate 0.01, batch size 128, 200 local epochs, and five clients.

3.2 Results

Across non-IID datasets, DENSE achieves the strongest reported accuracy and remains effective with heterogeneous client architectures. Distillation also outperforms direct FedAvg aggregation, whose performance can degrade as local training increases.

  • Evaluation on real-world datasets: DENSE achieves the highest accuracy across all evaluated datasets and non-IID settings.At α = 0.3 on CIFAR10, DENSE outperforms Fed-ADI by 5.08%.
  • Evaluation on real-world datasets: 5.08%: DENSE outperforms Fed-ADI on CIFAR10 when α = 0.3.
  • Evaluation on real-world datasets: As data become more imbalanced, all methods lose performance, although DENSE still significantly outperforms other methods under highly skewed data.
  • Impact of model distillation: With E = 40, FedAvg reaches 34% test accuracy, while larger local-training values can cause degradation or collapse.The paper attributes this behavior to inconsistent optimization objectives and weight divergence under non-IID data.
  • Impact of model distillation: DENSE outperforms each client’s local model, whereas FedAvg underperforms each local model when E = 400.The comparison supports model distillation over direct parameter aggregation in one-shot FL under non-IID data.
  • Results in heterogeneous FL: DENSE significantly outperforms other baselines when five heterogeneous CNN models train on CIFAR10 under α = {0.1, 0.3, 0.5}.FedAvg is omitted because it does not support heterogeneous models.

3.3 Analysis of Our Method

The analysis examines how DENSE behaves as client count and data imbalance vary, and visualizes its synthetic data. Performance generally declines with more clients, while LDAM improves results under highly skewed non-IID data.

  • Impact of the number of clients: Accuracy decreases as the number of clients increases on CIFAR10 and SVHN, but DENSE still outperforms other baselines.The evaluated client counts are m = {5, 10, 20, 50, 100}.
  • Combination with imbalanced learning: LDAM can mitigate the impact of local data imbalance when incorporated into clients’ local training.The paper presents this combination as a way to build a more powerful ensemble for federated learning.
  • Combination with imbalanced learning: DENSE+LDAM significantly improves performance, especially under highly skewed non-IID data with α = 0.1.Figure 5 visualizes the accuracy curve and client data distribution for CIFAR10 at α = 0.1.
  • Visualization of synthetic data: Synthetic data on CIFAR10 and SVHN look much different from the original data, reducing the probability of leaking sensitive client information.Figure 6 places original data in the first and third rows and synthetic data in the second and last rows.
  • Visualization of synthetic data: Despite their visual dissimilarity from original data, synthetic data enable DENSE to achieve higher performance than other baseline methods.The paper states that ideally synthetic data should be visually distinct from real data.

4 Conclusion

The conclusion presents DENSE as a data-free one-shot FL framework that combines data generation with model distillation. Experiments across varied settings support its efficacy, while privacy attacks remain a future direction.

  • 4 Conclusion: DENSE trains the global model through a data generation stage followed by a model distillation stage.The conclusion characterizes DENSE as an effective one-shot federated learning method.
  • 4 Conclusion: Experiments across various settings validate the efficacy of DENSE.
  • 4 Conclusion: DENSE is presented as a practical data-free one-shot FL framework that supports model heterogeneity.The conclusion identifies potential privacy attacks in one-shot FL as a promising future direction.

Checklist

The checklist reports that the paper describes its contributions, limitations, training details, compute resources, and existing assets, while not reporting error bars or potential negative societal impacts.

  • Checklist: The authors report that they described the contributions, scope, and limitations of the work.
  • Checklist: The authors report no discussion of potential negative societal impacts.
  • Checklist: The paper reports complete theoretical assumptions and proofs where theoretical results are included.
  • Checklist: The authors report providing reproducibility materials and training details, including code, data, instructions, splits, and hyperparameters.
  • Checklist: Error bars were not reported, although the total compute amount and resource type were reported.
  • Checklist: The paper reports citing existing assets and not including new assets in supplementary material or through a URL.

6 Appendix

The appendix reviews standard FL aggregation, one-shot FL, prior practical limitations, model heterogeneity, and data-free distillation. It frames DENSE as addressing one-round heterogeneous FL without auxiliary data or additional information.

  • 6.1.1 Federated Learning: In standard FL, clients upload local model parameters and the server computes an aggregated global model parameter before another training round.The procedure requires frequent client-server communication and can incur high communication cost.
  • 6.1.2 One-shot Federated Learning: One-shot FL limits each client to one local-model upload and uses one unidirectional communication between clients and the server.The appendix states that this reduces communication cost and attack risk, but convergence is difficult, especially with non-IID client data.
  • 6.1.2 One-shot Federated Learning: Prior one-shot methods use public or distilled data, while existing studies also do not address model heterogeneity.These settings motivate the question of one-shot FL without additional information or auxiliary data under heterogeneous models.
  • 6.1.3 Model Heterogeneity: Knowledge distillation can support independently designed client models, but proxy-data requirements make that approach impractical for many applications.The appendix identifies heterogeneous-model one-shot FL as more challenging and practical.
  • 6.1.3 Model Heterogeneity: Data-free ensemble distillation transfers teacher knowledge through synthetic data, but earlier approaches can require repeated communication or expose privacy concerns.One cited approach broadcasts generators to clients and later requires clients to return them to the server.
Loading 2112.12371v2…