PaperlayerView exact revision on arXiv

Source-linked AI summary

FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning

Yuanhao Xiong, Ruochen Wang, Minhao Cheng, Felix Yu, Cho-Jui Hsieh

arXiv:2207.09653v1cs.LGcs.CR

TL;DR

Federated learning faces unbalanced, non-i.i.d. client data that make iterative model averaging communication-intensive. FedDM builds local surrogate objectives from synthetic data and sends them to the server; across three image-classification tasks, it outperforms FL counterparts in communication efficiency and final model performance.

  • Problem

    Unbalanced, non-i.i.d. client data make local gradients inconsistent, requiring many communication rounds for shared-model training.

    Source: §1 · pp. 1–2

  • Method

    FedDM learns synthetic data that approximate local training objectives through distribution matching, enabling the server to build a global surrogate.

    Source: §1 · p. 2

  • Results

    Across three tasks, FedDM outperforms FL counterparts in communication efficiency and final model performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The server obtains a better global view of the loss landscape than by aggregating local model updates.

    Source: §5 · p. 9

    Potential user data leakage still warrants careful consideration when deploying differentially private algorithms in real-world sensitive applications.

    Source: Reproducibility & Ethics Statements · p. 9

Abstract

from arXiv · show

Federated learning~(FL) has recently attracted increasing attention from academia and industry, with the ultimate goal of achieving collaborative training under privacy and communication constraints. Existing iterative model averaging based FL algorithms require a large number of communication rounds to obtain a well-performed model due to extremely unbalanced and non-i.i.d data partitioning among different clients. Thus, we propose FedDM to build the global training objective from multiple local surrogate functions, which enables the server to gain a more global view of the loss landscape. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data through distribution matching. FedDM reduces communication rounds and improves model quality by transmitting more informative and smaller synthesized data compared with unwieldy model weights. We conduct extensive experiments on three image classification datasets, and results show that our method can outperform other FL counterparts in terms of efficiency and model performance. Moreover, we demonstrate that FedDM can be adapted to preserve differential privacy with Gaussian mechanism and train a better model under the same privacy budget.

1 Introduction

Federated learning must contend with highly unbalanced, non-i.i.d. client data and costly communication, while iterative model-averaging methods can require many rounds to converge. FedDM instead sends synthetic data that form local surrogate objectives, and the paper reports improved efficiency and model performance across three tasks, with a differential-privacy adaptation.

1 Introduction

  • Unbalanced, non-i.i.d. client data make local gradients inconsistent, creating a substantial communication-round burden for shared-model training.Existing iterative methods still require considerable communication rounds, and some improvements increase the information required by the server.Source: §1 · pp. 1–2
  • FedDM learns synthetic data to approximate each client's local objective, then sends that data so the server can build a global surrogate instead of aggregating local model updates.The approach uses distribution matching in embedding space with maximum mean discrepancy (MMD).Source: §1 · p. 2
  • FedDM is adapted to guarantee (ϵ, δ)-differential privacy with the Gaussian mechanism and train a better model under the same privacy budget.The paper identifies privacy as an important factor for deploying federated-learning systems.Source: §1 · p. 2
  • Across three tasks, FedDM outperforms FL counterparts in communication efficiency and final model performance.The paper also reports that FedDM can guarantee (ϵ, δ)-differential privacy with the Gaussian mechanism and train a better model under the same privacy budget.Source: §1 · p. 2

2 Related work

Related work covers federated-learning methods for non-i.i.d. data and communication efficiency, differential-privacy approaches, and techniques for constructing smaller informative datasets.

Federated Learning

  • FedAvg and its variants address federated learning under client heterogeneity, while other work targets communication costs through compression or one-shot training.FedProx modifies the loss; FedNova and SCAFFOLD use auxiliary information to balance distribution shift.Source: §2 · p. 2

Differential Privacy

  • Differential privacy quantifies information disclosure about individuals, with methods developed for deep-network training and federated user-level or local privacy.The cited work also considers privacy for hyperparameter tuning.Source: §2 · pp. 2–3

Dataset Distillation

  • Dataset-distillation methods learn smaller synthetic datasets using approaches including meta-learning, gradient matching, distribution matching, neural kernels, and generative models.Coreset selection instead uses heuristic criteria and faces a performance-versus-data-size trade-off.Source: §2 · p. 3

3 Methodology

FedDM builds the global objective from local surrogate functions learned through distribution matching, giving the server a broader view of the loss landscape. Its methodology describes iterative surrogate minimization, client-side synthetic data, and a Gaussian-mechanism privacy adaptation.

3.1 Iterative surrogate minimization framework for federated learning

  • Each round, clients approximate their local objectives near the current weights; the server aggregates those surrogates and minimizes the result within a radius-limited neighborhood.The neighborhood is local rather than the full parameter space; a first-order Taylor surrogate recovers large-batch gradient descent.Source: §3.1 · p. 4

3.2 Local distribution matching

  • Clients learn class-wise synthetic samples by matching real and synthetic feature and logit distributions around the current weights, then synchronize weights for the next round.The matching uses a latent-space MMD estimate because the underlying data distribution is inaccessible.Source: §3.2 · pp. 4–5
  • ≈2.8 × 10^6 versus 3.2 × 10^6 float parameters for FedDM and ConvNet [39], respectively, on CIFAR10 under Dir10(0.5) (cpc=9, ipc=10).The paper also reports significantly fewer communication rounds and smaller explicit messages with FedDM.Source: §3.2 · p. 5

3.3 Differential privacy of FedDM

  • FedDM applies the Gaussian mechanism during client-side synthetic-set optimization and uses parallel composition to retain (ϵ, δ)-differential privacy across clients with disjoint datasets.The synthetic set is treated as equivalent to network parameters in DP-SGD; the system guarantee holds when each client satisfies the same privacy guarantee.Source: §3.3 · p. 6

4 Experiments

Across three image-classification datasets, FedDM is evaluated for communication efficiency, non-uniform data partitioning, differential privacy, and design trade-offs. The experiments report stronger performance than iterative model-averaging baselines, including under challenging partitions and added noise.

4.1 Experimental setup

  • The experiments use MNIST, CIFAR10, and CIFAR100 with Dirichlet-simulated non-i.i.d. partitions; the default is Dir10(0.5) with 10 clients.FedDM is compared with FedAvg, FedProx, FedNova, and SCAFFOLD; experiments use 20 communication rounds and ConvNet by default, with ResNet-18 also evaluated.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 Communication efficiency and convergence rate

  • 69.66 ± 0.13% test accuracy on CIFAR10 versus SCAFFOLD’s 66.12 ± 0.17% after 20 communication rounds; FedDM also converges fastest by message size.The evaluation uses Dir10(0.5); Figure 2 compares test accuracy against communication rounds and message size.Source: §4.2 · p. 7, Figure 2 · p. 8

4.3 Evaluation on different data partitioning

  • With Dir10(0.1) and Dir10(0.01), FedDM performs more consistently than iterative model-averaging methods, whose performance degrades sharply relative to Dir10(0.5).The authors attribute FedDM’s robustness to distribution matching that more accurately approximates the global training objective.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Performance with DP guarantee

  • Under the same differential-privacy guarantee, FedDM outperforms other FL counterparts in convergence rate and final performance across the tested noise levels.The comparison uses Gaussian noise levels σ = 1, 3, and 5 with gradient norm bound C = 5; FedDM is also reported as most resistant to perturbed optimization versus its no-noise accuracy.Source: §4.4 · p. 8, Figure 3 · p. 9

4.5 Analysis of FedDM

  • On CIFAR10 with Dir10(0.5), accuracy rises from 53.64 ± 0.35% to 69.62 ± 0.14% as ipc increases from 3 to 10, while communication burden also grows.On ResNet-18, FedDM works well and retains its original message size while baseline messages grow; with ipc = 10, REAL reaches 68.66 ± 0.08%, below FedDM’s 69.62 ± 0.14%.Source: §4.5 · p. 9

5 Conclusions and limitations

FedDM uses iterative distribution matching to make federated learning more communication-efficient by learning client synthetic datasets that approximate local objectives. The conclusion reports Gaussian-mechanism differential privacy and identifies synthetic-set size versus final performance as a trade-off, especially with hundreds of clients or classes.

5 Conclusions and limitations

  • FedDM learns a synthetic dataset for each client to approximate its local objective, helping the server obtain a more global view of the loss landscape than aggregating local updates.The paper presents this iterative distribution-matching approach as a route to more communication-efficient federated learning.Source: §5 · p. 9
  • FedDM can preserve differential privacy with the Gaussian mechanism.Source: §5 · p. 9
  • Synthetic-set size trades off against final performance, particularly for classification tasks with hundreds of clients or classes; reducing the set to save communication is a future direction.Source: §5 · p. 9

Reproducibility & Ethics Statements

The authors state that experimental setups and the algorithm are specified and source code is provided in supplemental material. They also report Gaussian-mechanism differential privacy while cautioning that potential data leakage must be considered in sensitive real-world deployments.

Reproducibility & Ethics Statements

A Synthetic Binary Classification

A synthetic 1-D binary classification task illustrates matching a surrogate training objective to the original objective near w = 0.

A Synthetic Binary Classification

  • The experiment constructs a dataset of n = 100 synthetic pairs for a one-dimensional binary classification problem to illustrate the advantage of learning a surrogate function for the training objective.Source: A Synthetic Binary Classification · p. 13
  • Predictions use Sigmoid(wxi), with w trainable, and binary cross entropy as the training objective.Source: A Synthetic Binary Classification · p. 13
  • Twenty randomly initialized examples are used to match the objective around w = 0 and compare it with the original objective and gradient tangent.The comparison is plotted in Figure 1.Source: A Synthetic Binary Classification · p. 13

B Proof of Eq. (15)

The proof divides the client loss into two parts and derives gradients with respect to the synthetic set.

B Proof of Eq. (15)

C Message size of different FL methods

FedDM uploads smaller messages than model-size-based baselines in some settings, with savings especially notable for unbalanced partitions.

C Message size of different FL methods

D.1 Learning curves of FL methods

The appendix provides learning curves across data partitions and additional settings; FedDM is reported to outperform other baselines in near-i.i.d. and higher-client settings.

D.1 Learning curves of FL methods

D.2 Visualization of the synthetic dataset

Synthesized images remain illegible even without added gradient noise, and become harder to recognize as noise increases. FedDM samples and clips weights within a ρ-radius ball, with similar performance across tested radii and ρ = 5 selected for the experiments.

D.2 Visualization of the synthetic dataset

  • As σ increases, synthesized images become harder to recognize; even without added gradient noise, they remain illegible from their original classes.The visualizations compare no noise with σ = 2, 3, and 5.Source: §D.2 · p. 17, Figure 14 · p. 17, Figure 15 · p. 18

D.3 ρ-radius ball

  • FedDM samples weights from a truncated Gaussian and clips them to a ρ-radius ball around wr; the server also clips its training weights to that ball.The radius is a local parameter-space constraint, not a requirement to construct a surrogate across the entire parameter space.Source: §D.3 · p. 19
  • Test accuracy was similar for ρ values of 3, 5, and 10, with ρ = 5 relatively best and selected for all experiments.The comparison is on CIFAR10 after 20 communication rounds under the default Dir10(0.5) setting.Source: §D.3 · p. 19, Table 4 · p. 19
Loading 2207.09653v1…