FedGAN: Federated Generative Adversarial Networks for Distributed Data

Mohammad Rasouli; Tao Sun; Ram Rajagopal

Source-linked AI summary

FedGAN: Federated Generative Adversarial Networks for Distributed Data

Mohammad Rasouli, Tao Sun, Ram Rajagopal

arXiv:2006.07228v2Jun 12, 2020cs.LGcs.CVcs.MAstat.ML

TL;DR

FedGAN targets GAN training across privacy-constrained, non-iid distributed data sources where local data may be insufficient and communication bandwidth limited. It trains local generators and discriminators with periodic intermediary synchronization, proves convergence under equal and two time-scale updates, and reports similar performance to general distributed GANs with reduced communication and robustness to less frequent communication.

Problem
FedGAN addresses GAN training across distributed non-iid data sources when privacy prevents pooling and limited local data or bandwidth constrain centralized or local training.
Method
FedGAN trains local generators and discriminators, periodically averages and broadcasts their parameters through an intermediary, and analyzes updates using stochastic approximations and communication-efficient SGD.
Results
FedGAN converges, has similar performance to general distributed GAN, reduces communication complexity, and remains robust to reduced communications across toy, image, and time-series experiments.
Takeaways & Limitations
Periodic synchronization enables a communication-efficient distributed GAN while preserving the paper's reported performance and convergence across non-iid sources.
Takeaways & Limitations
The analysis assumes all agents participate in communication, while added privacy noise and non-responsive agents remain future research directions.

Abstract

from arXiv · show

We propose Federated Generative Adversarial Network (FedGAN) for training a GAN across distributed sources of non-independent-and-identically-distributed data sources subject to communication and privacy constraints. Our algorithm uses local generators and discriminators which are periodically synced via an intermediary that averages and broadcasts the generator and discriminator parameters. We theoretically prove the convergence of FedGAN with both equal and two time-scale updates of generator and discriminator, under standard assumptions, using stochastic approximations and communication efficient stochastic gradient descents. We experiment FedGAN on toy examples (2D system, mixed Gaussian, and Swiss role), image datasets (MNIST, CIFAR-10, and CelebA), and time series datasets (household electricity consumption and electric vehicle charging sessions). We show FedGAN converges and has similar performance to general distributed GAN, while reduces communication complexity. We also show its robustness to reduced communications.

1 Introduction

FedGAN addresses privacy- and bandwidth-constrained GAN training across distributed, non-iid data sources. It combines local models, periodic synchronization, convergence analysis, and evaluations across toy, image, and time-series datasets.

Distributed sources may lack enough local data diversity to train an accurate population-wide GAN, while privacy constraints prevent central pooling.
FedGAN places local generators and discriminators at each source and periodically synchronizes them through an intermediary.
The paper proves FedGAN convergence for non-iid sources under equal and two time-scale generator and discriminator updates.
The analysis connects stochastic-approximation results for GANs with communication-efficient stochastic-gradient methods for federated learning.
Experiments cover toy systems, MNIST, CIFAR-10, CelebA, household electricity consumption, and electric-vehicle charging sessions.
The paper examines convergence, efficiency, and robustness to reduced communication across these datasets.

2 Literature Review

Prior distributed GAN and federated-learning approaches address parts of the distributed training problem but leave communication or convergence gaps. FedGAN places both model components at agents and provides a fuller theoretical and experimental study.

Distributed GAN research includes centralized generators with distributed discriminators and gossip-based approaches without an intermediary.
Federated learning averages locally trained model parameters through an intermediary to support communication-efficient distributed learning.
FedAvg-GAN still communicates discriminator and generator information at every generator update because its generator remains centralized.
FedGAN places both generators and discriminators at agents, synchronizing parameters only every K local updates.
Earlier architectures similar to FedGAN lacked theoretical convergence studies and had limited experimental results, whereas this paper provides a complete study.

3 FedGAN Algorithm

FedGAN trains local generators and discriminators on non-iid data, periodically synchronizing their parameters through an intermediary. The paper analyzes its communication properties and convergence using stochastic-approximation arguments under equal and two time-scale updates.

FedGAN Algorithm: The agents use local datasets from individual, potentially non-iid distributions, with stochastic mini-batch gradients and shared learning-rate schedules.Agent data are weighted by local dataset size in the federated model.
FedGAN Algorithm: FedGAN places both generators and discriminators at the agents, which train locally and synchronize averaged parameters through an intermediary every K steps.The intermediary does not train either model; it averages and broadcasts parameters.
Communication and Computation Complexity: Increasing the synchronization interval K reduces average communication, although it may reduce trained FedGAN performance and requires experimental evaluation.FedGAN communicates generator and discriminator parameters only at synchronization times rather than every update step.
Communication and Computation Complexity: FedGAN shifts computation toward agents because each trains a local generator, while the intermediary only averages parameters.Agent computation is roughly doubled relative to distributed GAN, at the same order.
Convergence Analysis: Under standard assumptions, FedGAN is shown to converge for both equal and two time-scale generator and discriminator updates with non-iid data sources.The assumptions include Lipschitz gradients, bounded iterates, martingale-difference noise, bounded stochastic-gradient variance, and gradient closeness conditions.
Convergence Analysis: The convergence proof shows FedGAN’s update trajectory asymptotically tracks the centralized GAN’s limiting ODE, without implying convergence to the same point.The argument bounds deviations between averaged federated parameters and centralized-GAN updates, then invokes centralized GAN ODE convergence results.

4 Experiments

FedGAN is evaluated across toy, image, and energy time-series datasets, with experiments examining convergence, similarity to real data, and robustness to reduced communication. The image and time-series studies use non-iid agent splits and periodic synchronization.

Experiments: Experiments cover toy examples, MNIST, CIFAR-10, CelebA, household electricity consumption, and EV charging sessions.The study uses these datasets to assess convergence, generated-data quality, and communication robustness.
Image datasets: MNIST is split across B = 5 agents with two image classes per agent, using synchronization interval K = 20.FedGAN generated images close to real MNIST images.
Image datasets: CIFAR-10 FedGAN FID scores remain close to distributed GAN scores even for large synchronization intervals K, except in the tail part.The experiment tests K = 10, 20, 1000, 500, 3000, and 6000.
Image datasets: CelebA experiments report performance close to distributed GAN and robust to reduced communications as synchronization intervals increase.CelebA uses B = 5 agents and tests K = 10, 20, 50, 100, and 200.
Time-series datasets: Energy time-series data are divided across B = 5 non-iid agents with K = 20, and k-means compares real and generated profiles.The datasets include 500k PG&E households and 12.4 million EV charging sessions; Figures 3 and 4 show normalized top-9 clusters.

5 Conclusions and Future Directions

The paper concludes that FedGAN provides communication-efficient distributed GAN training under privacy constraints, with convergence for non-iid data and performance comparable to general distributed GANs. It identifies broader scalability, theory, privacy, and reliability questions for future work.

Conclusions: FedGAN converges for non-iid data sources with both equal and two time-scale generator and discriminator updates.The conclusion presents this as a theoretical result of the proposed algorithm.
Conclusions: Across toy, image, and time-series experiments, FedGAN has similar performance to general distributed GAN while reducing communication complexity.The experiments also examine robustness to reduced communications.
Future directions: Future work includes testing more federated datasets and applications, and studying robustness as the number of agents increases.The authors note that larger agent counts require GPU resources beyond the engineering capacity of this research.
Future directions: Open theoretical questions include explaining robustness to reduced communication and identifying the convergence rate.The authors also propose studying differential-privacy noise and non-responsive agents.

6 Broader Impact

The paper describes FedGAN as a potential way to support shared learning across agents while preserving local-data privacy and keeping communication costs low. It highlights energy applications related to demand prediction and EV charging-station planning.

Potential positive impact: FedGAN provides shared learning across agents with local data while preserving privacy and low communication cost.The paper frames this as helping democratize data power.
Energy applications: The work emphasizes energy applications including household demand prediction and electric vehicle charging station planning.These applications are presented in the context of sustainability and reversing global warming.

A Convergence Analysis of FedGAN with Two Time-Scale Updates

FedGAN’s two time-scale updates are analyzed by separating the faster discriminator dynamics from the slower generator dynamics. Under local stability and learning-rate assumptions, the iterates converge almost surely to an equilibrium of the corresponding centralized GAN dynamics.

Assumptions: The analysis assumes b(n) = o(a(n)), making discriminator updates faster than generator updates.This relationship determines the relative generator and discriminator learning rates.
Assumptions: The limiting discriminator ODE has a locally asymptotically stable attractor λ(θ), while the generator ODE has a locally asymptotically stable equilibrium θ∗.Both stability conditions are imposed within specified domains of attraction.
Scope: The convergence guarantee is conditioned on the iterates eventually entering a compact region within the local attractor’s domain of attraction.The paper identifies this as an unverifiable event under the relaxed local stability assumption.
ODE analysis: With θ treated as quasi-static, FedGAN discriminator updates asymptotically track the faster ODE ˙w(t) = g(w(t)).The time-scale separation allows analysis of w(t) while θ changes slowly.
Convergence results: For fixed θ, the discriminator sequence converges almost surely to a compact connected internally chain transitive invariant set of the limiting ODE.This result follows from the stochastic-approximation tracking argument.
Convergence results: The coupled iterates (w_n, θ_n) converge almost surely to (λ(θ∗), θ∗), an equilibrium representing centralized GAN dynamics with two time-scale updates.The result extends the equal-time-scale analysis to jointly varying generator and discriminator parameters.

B Proof of Results in Section 3.3

The proof establishes that FedGAN’s interpolated parameter trajectories track the limiting ODE despite local updates and periodic synchronization. Bounding the iterates and synchronization errors yields almost-sure asymptotic tracking.

Trajectory bounds: The proof bounds generator and discriminator interpolated trajectories over finite time windows using Lipschitz conditions and Gronwall’s inequality.The resulting bound is finite almost surely under bounded parameter assumptions.
Trajectory bounds: The corresponding drift terms remain finite almost surely, providing boundedness needed for the ODE approximation.The bounds apply separately to discriminator and generator dynamics.
Error control: The proof combines bounds on drift, stochastic variation, and parameter discrepancies to control the interpolated trajectory’s deviation from the limiting ODE.The analysis uses inequalities involving variance bounds, gradient divergence, and synchronization errors.
Conclusion: Because K_T,n → 0 almost surely, linear interpolation completes the proof of asymptotic ODE tracking.The synchronization error vanishes as the iteration index grows.

C Toy Examples in Section 4.1: 2D System, Mixed Gaussian and Swiss Roll

Toy experiments evaluate FedGAN on a partitioned 2D system, mixed Gaussians, and Swiss roll data. The experiments show convergence to the expected 2D point and generated samples that nearly coincide with pooled real data in the other two tasks.

2D system: In the 2D experiment, five agents partition the data domain into equal segments, creating distributed local data for FedGAN.The generator and discriminator are evaluated through their parameter trajectory.
2D system: FedGAN converges to (1, 0) for synchronization intervals K = 1, 5, 20, 50 in the 2D experiment.The trajectory is shown for generator θ and discriminator φ, with the red dot marking initial values.
2D system: Increasing K and reducing communications has low impact on the 2D experiment’s result.The paper describes this as robustness to reduced communications.
Mixed Gaussian and Swiss roll: The mixed Gaussian and Swiss roll experiments divide data among four agents and use synchronization interval K = 5.Each agent owns two Gaussians or a different equal-sized part of the roll.
Mixed Gaussian and Swiss roll: Generated and real points nearly coincide after N = 15000 iterations for mixed Gaussians and N = 27000 iterations for Swiss roll.The orange points denote real data and green points denote generated data.

D Hyperparameters and Generated Images for Experiments in Section 4.2 and 4.3

The paper specifies hyperparameters for image and time-series experiments and describes distributed EV charging data with non-iid station-level partitions. Figures cover generated images, charging profiles, and the effects of synchronization settings.

Image experiments: CIFAR-10 and CelebA experiments use separate hyperparameter tables, while MNIST reuses CIFAR-10 settings with one-channel image dimensions.Batch normalization is abbreviated BN in the image-experiment tables.
EV time-series data: EV charging-profile generation uses POI category, POI subcategory, maximum power, battery capacity, month, and day of week as CGAN labels.The time-series experiments use hyperparameters listed in Table 3.
EV time-series data: The EV charging dataset is split across five agents by charging-station category, producing non-iid charging-profile distributions.Figure 10 compares Tuesday charging sessions from a high-tech workplace and a shopping center.
Image experiments: CIFAR-10 generated images are evaluated with B = 5 and K = 10, 20, 100, 500, 3000, alongside distributed GAN.The comparison is performed at N = 30000 iterations.
Image experiments: CelebA generated images are evaluated with B = 5 and K = 10, 20, 50, 100, 200, alongside distributed GAN.The comparison is performed at N = 47500 iterations.