MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets

Corentin Hardy; Erwan Le Merrer; Bruno Sericola

Source-linked AI summary

MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets

Corentin Hardy, Erwan Le Merrer, Bruno Sericola

arXiv:1811.03850v2Nov 9, 2018cs.LGstat.ML

TL;DR

GANs are computationally demanding because they combine two deep neural networks and train on large datasets, yet are typically trained centrally. This paper introduces MD-GAN, which distributes training across workers with local data, and compares it with federated learning adapted to GANs. MD-GAN halves worker learning complexity and reports better performance than federated learning on MNIST and CIFAR10.

Problem
GANs are usually trained on a single server despite computational demands and datasets distributed across multiple workers.
Method
MD-GAN uses one server-hosted generator and worker-hosted discriminators that exchange peer-to-peer while sending feedback for generator updates.
Results
MD-GAN reduces worker learning complexity by a factor of two and provides better performance than federated learning on MNIST and CIFAR10.
Takeaways & Limitations
MD-GAN offers a distributed GAN design for geo-distributed or edge-device learning setups while retaining competitive results on the reviewed datasets.

Abstract

from arXiv · show

A recent technical breakthrough in the domain of machine learning is the discovery and the multiple applications of Generative Adversarial Networks (GANs). Those generative models are computationally demanding, as a GAN is composed of two deep neural networks, and because it trains on large datasets. A GAN is generally trained on a single server. In this paper, we address the problem of distributing GANs so that they are able to train over datasets that are spread on multiple workers. MD-GAN is exposed as the first solution for this problem: we propose a novel learning procedure for GANs so that they fit this distributed setup. We then compare the performance of MD-GAN to an adapted version of Federated Learning to GANs, using the MNIST and CIFAR10 datasets. MD-GAN exhibits a reduction by a factor of two of the learning complexity on each worker node, while providing better performances than federated learning on both datasets. We finally discuss the practical implications of distributing GANs.

I. INTRODUCTION

GANs generate realistic data but require large datasets and substantial computation, motivating distributed training over data held across workers. The paper proposes MD-GAN as a distributed GAN approach and compares it with federated learning adapted to GANs.

GANs are deep neural-network generative models that learn to generate new realistic data from a dataset’s probability distribution.
Large training datasets make centralized GAN training computationally demanding, while data volumes can prevent timely centralization.The paper cites an image super-resolution application using 350,000 ImageNet images as an example.
Distributed GAN training must preserve the tightly coupled generator–discriminator process while keeping communication and worker computation reasonable.
MD-GAN is presented as the first approach to distribute GANs across worker machines using a single server-hosted generator and peer-to-peer discriminator communication.
The paper compares MD-GAN with standalone training and federated learning adapted to GANs, experimenting on MNIST and CIFAR10 with GPUs.

1) Discriminator learning:

The paper establishes GAN and distributed-training foundations before describing federated learning as a comparison method. It also notes weak convergence guarantees and scope boundaries for more advanced GAN techniques.

Discriminator learning: Discriminator learning minimizes Jdisc while updating discriminator parameters θ against real and generated batches.Xr contains real data from the local training dataset, and Xg contains generated data from G.
Generator learning: Generator learning minimizes Jgen to adapt generator parameters w to the discriminator parameters θ.This update uses a batch of random vectors and occurs once per iteration.
Generator learning: GAN training alternates discriminator and generator updates over different batches, but convergence guarantees remain weak.
Distributed setup: The considered distributed setup keeps workers’ local datasets in place and assumes those datasets are independently and identically distributed.
Scope: More advanced GAN techniques are outside the paper’s scope because distributing them would require redesigning their protocols.
FL-GAN: FL-GAN trains a generator–discriminator pair locally and averages both parameter sets at the server every E epochs.

IV. THE MD-GAN ALGORITHM

MD-GAN places one generator on the server and distributes discriminators across workers, which exchange discriminators peer-to-peer. Each global iteration combines local discriminator training with server-side generator updating.

Architecture: MD-GAN uses a single server-hosted generator while workers host discriminators that can exchange peer-to-peer.Worker discriminator architectures and initial parameters may differ, although the paper assumes they are the same for simplicity.
Design rationale: The design forms a 1-versus-N game in which the generator faces all worker discriminators and workers use local data to distinguish real from generated samples.
Global iteration: Each global iteration generates k batches on the server, sends selected batches to workers, and trains each worker discriminator on local real data.The server chooses two distinct generated batches for each worker, and each discriminator performs L learning iterations.
Global iteration: Workers compute error feedback with their discriminators and send it to the server, which uses all feedbacks to update generator parameters w.The server computes the generator gradient and applies an optimizer such as Adam.
Design rationale: MD-GAN does not fully follow the parameter-server model because workers do not synchronize the same model architecture hosted at the server.
Design comparison: Figure 1 contrasts MD-GAN’s single server generator and peer-to-peer discriminator swapping with FL-GAN’s worker generators and server averaging.

B. The generator learning procedure (server-side)

MD-GAN places the generator on the server, which generates batches and distributes them to workers for discriminator and generator-error computations. The server aggregates worker feedbacks to update the generator parameters.

The server hosts generator G and updates its parameters using error feedbacks from workers.
At each global iteration, G generates k batches of size b, with k ≤ N, and each worker receives two batches.The two batches support separate discriminator and generator computations.
Workers send generator error terms F_n to the server after computing them on generated data.Each feedback contains b vectors associated with the generated batch.
The server computes gradients from all feedbacks, merges parallel updates by averaging, and applies Adam to update generator parameters.The parameter update is represented as w_j(t) = w_j(t − 1) + Adam(∆w_j).
Algorithm 1 separates worker discriminator-learning steps from server-side batch generation, feedback collection, gradient computation, and parameter updates.

3) Workload at the server:

Placing the generator on the server shifts generator computation and feedback processing away from workers, increasing server workload with batch count, data dimension, and model size.

The server generates k batches of b data and receives N error feedbacks of size bd during each global iteration.
O(kb|w|) is the server’s batch-generation complexity under G_op = O(|w|) and G_a = O(|w|).Feedback merging and gradient computation require O(b(dN + k|w|)) memory and computation.

4) The complexity vs. data diversity trade-off:

MD-GAN controls a trade-off between generator workload and feedback diversity through the number k of generated batches distributed across workers.

k = 1 reduces server workload but makes all workers compute feedback on the same training batch, reducing feedback diversity.
k = N gives each worker a different batch, avoiding feedback conflicts on concurrently processed data but increasing generator workload.
The experiments use k = 1 or k = ⌊log(N)⌋ to evaluate this workload–performance trade-off.
Each worker trains its discriminator on its local dataset B_n and uses generated batches to compute generator error feedback.
Discriminator parameters are swapped between workers after E epochs in a random gossip pattern to avoid over-specialization on one local dataset.The paper links excessive local training to reduced discriminator generalization.

1) The swapping of discriminators:

MD-GAN distributes discriminator state and generated-data feedback differently from FL-GAN, with communication costs depending strongly on batch size and model or data dimensions. Its traffic is competitive at smaller batch sizes but can become more expensive as batches grow.

1) The swapping of discriminators:: MD-GAN workers handle discriminator parameters and error feedbacks, while generator computation is performed on the server.
1) The swapping of discriminators:: MD-GAN communicates generated batches from server to workers, error terms from workers to server, and discriminator parameters between workers.
1) The swapping of discriminators:: Table II highlights a factor-of-two reduction in computation complexity and memory on workers for MD-GAN relative to adapted federated learning.
1) The swapping of discriminators:: When data-object size approaches GAN parameter count, MD-GAN communication may be expensive; the paper illustrates this with GoogLeNet image dimensions and parameter count.
1) The swapping of discriminators:: Figure 2 compares maximal ingress traffic per communication for MD-GAN and FL-GAN as batch size varies, distinguishing worker and server ingress.
1) The swapping of discriminators:: MD-GAN is competitive below approximately b = 550 for MNIST and b = 400 for CIFAR10, while larger batches can cross FL-GAN’s traffic level.

2) Computation complexity:

MD-GAN removes generator computation from workers by hosting a single generator at the server, reducing worker-side complexity while adding communication. The experiments evaluate this setup across worker counts, datasets, architectures, and GAN metrics.

Worker computation: The single-generator design removes generator tasks from workers, leaving them to process discriminator parameters and error feedbacks.Workers communicate with the server after local iterations, while the generator remains centralized.
Complexity comparison: The workload on MD-GAN workers is half that of FL-GAN, although the architecture determines the exact memory and computation gain.The reduction is generally about half when generator and discriminator complexities are similar.
Evaluation: Convergence is assessed using MNIST score or Inception Score, together with Fréchet Inception Distance across iterations.Higher MNIST and Inception scores are better, whereas lower FID is better.

B. Experiment results

The experiments report competitor scores over iterations and examine how MD-GAN behaves as the number of workers changes. The worker-scaling study also compares normal swapping with a configuration where swapping is disabled.

Iteration results: Figure 3 reports competitor scores as a function of training iterations, with the resulting curves smoothed for readability.The reported metrics are evaluated through the iteration axis.
Worker scaling: Figure 4 tracks MNIST score and Fréchet Inception Distance for MD-GAN with the MLP model as the number of workers varies.The figure focuses on worker-count scaling rather than competitor comparison.
Worker scaling: The worker-scaling experiments include a comparison with swapping processing disabled.This provides a variant of MD-GAN for comparison within the worker-count analysis.

1) Competitor scores:

Across MNIST and CIFAR10 experiments, MD-GAN is compared with standalone GANs and FL-GAN under varying scalability, communication, and crash conditions. Results show benefits from its single-generator, multi-discriminator design, while performance depends on workload allocation, discriminator swapping, dataset complexity, and failure timing.

FL-GAN does not converge on MLP experiments, while MD-GAN achieves better FID and MS scores than the standalone competitor.
On CNN experiments, MD-GAN and FL-GAN have close MNIST FID and MS scores, while MD-GAN obtains better CIFAR10 IS and MS scores.
MD-GAN exploits a single generator trained against multiple discriminators distributed across workers.
Constant worker workload leads to better results but increases server cost.
Swapping discriminators improves results, although its FID gain is marginal under constant server workload.
Crash effects depend on dataset complexity: MNIST performance is largely unaffected, whereas CIFAR10 diverges after early failures but remains comparable to standalone performance through 8 crashed workers.

4) Validation on a larger dataset:

The CelebA validation compares MD-GAN with FL-GAN and a standalone GAN using Inception Score and Fréchet Inception Distance. MD-GAN’s Inception Scores are comparable and slightly higher, while its FID is worse than the standalone approach.

Validation setup: The experiment trains GAN competitors on CelebA, using 200K celebrity images at 128 × 128 pixels and evaluating 10K test images across N = {1, 5} workers.The training images are distributed equally and i.i.d. over the workers.
Evaluation metrics: The reported metrics are Inception Score and Fréchet Inception Distance for the three competitors on CelebA.
Results: All Inception Scores are comparable, with MD-GAN slightly above the other competitors.
Results: MD-GAN and FL-GAN are distanced by the standalone approach on FID, as in the CNN experiment on MNIST.
Related work: Prior multi-discriminator and multi-generator GAN studies targeted convergence improvements rather than learning over distributed datasets.

1) Asynchronous setting:

The paper considers asynchronous updates, communication constraints, worker failures, adversarial workers, and scalability as open issues for distributed GAN learning. Asynchrony can reduce waiting, but inconsistent parameters and broader system constraints remain important boundaries.

Asynchronous setting: Asynchrony means a worker’s parameters may differ between generating data and sending its feedback to the server.
Communication constraints: The parameter-server model creates a communication bottleneck toward the central server, motivating gradient and data compression methods.
Faults and future scope: Distributed GAN learning remains exposed to worker crashes and potentially manipulated discriminator feedback, while generator–discriminator coupling may complicate future scalability.
Scaling the number of workers: The experiments used up to 50 parallel workers, while the bottleneck limiting larger-scale gains remains unclear.The paper identifies dataset size and conflicting asynchronous updates as possible factors.