PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Diverse Generative Adversarial Networks

Arnab Ghosh, Viveka Kulharia, Vinay Namboodiri, Philip H. S. Torr, Puneet K. Dokania

arXiv:1704.02906v3cs.CVcs.AIcs.GRcs.LGstat.ML

TL;DR

GANs often struggle with mode collapse, limiting their ability to represent diverse data modes. MAD-GAN uses multiple generators and a source-identifying discriminator, and reports diverse high-quality generations, modality disentanglement, and theoretical mixture-model behavior. The paper also identifies uncertainty about how many generators a dataset requires.

  • Problem

    GAN training can suffer mode collapse because reaching the true equilibrium is difficult in practice, motivating methods that explicitly capture diverse modes.

  • Method

    MAD-GAN combines multiple generators with one discriminator whose outputs identify both real samples and the generator responsible for each fake sample.

  • Results

    MAD-GAN reports high-quality diverse generations across synthetic and real tasks, modality disentanglement, unsupervised feature-representation efficacy, and mixture-component behavior with a stated convergence optimum.

  • Takeaways & Limitations

    The framework captures diverse modes while producing high-quality samples and can disentangle inter-class and intra-class variation in challenging diverse-class data.

  • Takeaways & Limitations

    The paper leaves open how to estimate the number of generators required for a dataset because the true data distribution is unavailable.

Abstract

from arXiv · show

We propose MAD-GAN, an intuitive generalization to the Generative Adversarial Networks (GANs) and its conditional variants to address the well known problem of mode collapse. First, MAD-GAN is a multi-agent GAN architecture incorporating multiple generators and one discriminator. Second, to enforce that different generators capture diverse high probability modes, the discriminator of MAD-GAN is designed such that along with finding the real and fake samples, it is also required to identify the generator that generated the given fake sample. Intuitively, to succeed in this task, the discriminator must learn to push different generators towards different identifiable modes. We perform extensive experiments on synthetic and real datasets and compare MAD-GAN with different variants of GAN. We show high quality diverse sample generations for challenging tasks such as image-to-image translation and face generation. In addition, we also show that MAD-GAN is able to disentangle different modalities when trained using highly challenging diverse-class dataset (e.g. dataset with images of forests, icebergs, and bedrooms). In the end, we show its efficacy on the unsupervised feature representation task. In Appendix, we introduce a similarity based competing objective (MAD-GAN-Sim) which encourages different generators to generate diverse samples based on a user defined similarity metric. We show its performance on the image-to-image translation, and also show its effectiveness on the unsupervised feature representation task.

1. Introduction

MAD-GAN addresses GAN mode collapse with multiple generators and one discriminator designed to identify both real samples and their generating source. Experiments and analysis report diverse, high-quality generations, including modality disentanglement and mixture-model behavior.

  • GANs can suffer mode collapse because practical training may fail to reach the true equilibrium, motivating methods that explicitly capture diverse modes.
  • MAD-GAN uses multiple generators with one discriminator, while allowing parameter sharing or separation according to dataset modality.Initial layers may be shared for homogeneous data but need not be shared across completely different modalities.
  • MAD-GAN disentangles inter-class variations while preserving intra-class diversity on datasets containing forests, icebergs, and bedrooms.Different generators produce different classes, and different noise inputs yield diverse samples within a class.
  • Experiments report that MAD-GAN outperforms compared GAN variants on synthetic and compositional MNIST settings while generating high-quality samples and capturing many modes.The paper also reports diverse generations for image-to-image translation and face generation, plus efficacy for unsupervised feature representation.
  • At convergence, the generators can learn as mixture components, achieving global optimum value −(k + 1) log(k + 1) + k log k.Here, k denotes the number of generators.
  • The discriminator identifies whether a sample is real or which generator produced it, pushing generators toward different identifiable high-probability modes.

2. Related Work

Related work addresses mode collapse through latent-information objectives, encoder-decoder regularization, optimization changes, and multiple-generator or multiple-discriminator designs. MAD-GAN is distinguished by explicitly enforcing diversity among multiple generators.

  • InfoGAN maximizes mutual information between factored latent variables and the generator distribution to encourage factor-specific generations.
  • ModeGAN uses an encoder-decoder paradigm so reconstructed samples are likely to remain in the same mode as the corresponding true samples.Its approach assumes sufficient true samples exist from a mode for the generator to capture it.
  • Coupled GAN trains two generators with shared parameters on separate domains, while multiple discriminators have been used to stabilize generator training.
  • W-GAN replaces the original GAN’s Jensen-Shannon divergence with an integral probability metric based on earth mover distance, while related methods modify equilibrium or discriminator training.The passage also describes BEGAN, DCGAN, GoGAN, and a formulation emphasizing multiple generators and discriminators.
  • Among multiple-generator methods, MAD-GAN is closest to several prior approaches but explicitly enforces generators to capture diverse modes.

3. Preliminaries

GAN training learns a generator distribution through a minimax game between a generator and discriminator. The discriminator separates real data from generated samples, while the generator learns to make generated samples appear real.

  • GANs seek generator parameters that produce an approximate data distribution from prior input noise.The generator is written as G(z; θg), with z sampled from a prior such as a normal distribution.
  • The discriminator D(x; θd) learns to distinguish real samples from the true distribution pd and fake samples from the generator distribution pg.
  • The GAN objective is optimized block-wise by alternately optimizing discriminator parameters θd and generator parameters θg.
  • The discriminator increases scores for true samples and decreases them for fake samples, whereas the generator maximizes discriminator mistakes on generated samples.
  • At theoretical equilibrium, the generator produces real samples, corresponding to pg = pd.

4. Multi-Agent Diverse GAN

MAD-GAN addresses mode collapse by combining multiple generators with one discriminator whose generator-identification task encourages distinct modes. Its objective supports a mixture-model interpretation, with a stated global optimum when the averaged generator distribution matches the data distribution.

  • Multi-Agent GAN Architecture: MAD-GAN uses multiple generators and one discriminator to increase the capacity for capturing diverse modes.The architecture is proposed specifically to address mode collapse, while allowing generators to share selected initial parameters when appropriate.
  • Multi-Agent GAN Architecture: Generators receiving the same latent input can otherwise converge to similar samples, motivating an explicit mechanism for diversity.Parameter sharing is task-dependent: it can support homogeneous data but should be avoided when generators must capture class-specific structures.
  • Enforcing Diverse Modes: The discriminator outputs k + 1 scores to distinguish real samples from fake samples and identify which generator produced each fake sample.This generator-identification requirement is intended to push different generators toward different identifiable modes, while each generator retains the standard GAN objective.
  • Optimization: The proposed updates require only minor modifications to standard GAN optimization and allow all generators to be updated in parallel.Generator-specific and discriminator gradients are computed from the corresponding discriminator outputs.
  • Theoretical Analysis: At the stated global optimum, the generators form a mixture model whose averaged distribution matches the true data distribution.The objective value is −(k + 1) log(k + 1) + k log k, and the single-generator case recovers the standard GAN objective with value −log 4.

5. Experiments

MAD-GAN is evaluated against GAN variants across synthetic and real-world settings, with experiments focused on distribution matching, mode recovery, diverse generation, and representation learning. Across these tasks, the reported results show strong mode coverage, diverse outputs, and improved unsupervised features.

  • Experimental design: MAD-GAN is compared with multiple GAN variants on synthetic, MNIST, image-to-image translation, diverse-class, face-generation, and representation-learning tasks.The evaluation combines quantitative criteria such as KL divergence and recovered modes with qualitative comparisons of generated samples.
  • Non-Parametric Density Estimation: In the 1D Gaussian-mixture experiment, MAD-GAN captures all clustered modes, including significantly overlapping modes, and obtains the minimum Chi-square distance and KL-divergence.The experiment uses five Gaussian components and compares estimated generated distributions with the ground truth.
  • Non-Parametric Density Estimation: Increasing MAD-GAN’s generator count to four makes sampling progressively more realistic in the synthetic experiment.The comparison varies the number of generators while keeping the underlying five-component Gaussian-mixture setup fixed.
  • Stacked and Compositional MNIST: On Stacked-MNIST, MAD-GAN outperforms the compared GAN variants on KL divergence and recovered modes; on Compositional-MNIST, it recovers all 1000 modes and is closest to the true distribution by KL divergence.The MNIST comparisons use three MAD-GAN generators.
  • Diverse Samples for Image-to-Image Translation and Comparison to InfoGAN: For image-to-image translation, MAD-GAN generators produce distinct handbag outputs and varied night-to-day images across colors, textures, patterns, lighting, skies, and weather.The handbag comparison reports that InfoGAN outputs are nearly identical across categorical codes, whereas MAD-GAN outputs differ meaningfully.
  • Diverse Generation and Unsupervised Representation Learning: MAD-GAN also separates classes while retaining within-class variation in diverse-class data, generates high-quality diverse faces, and achieves a 17.5% SVHN misclassification error versus DCGAN’s 22.48%.The diverse-class result concerns inter-class disentanglement and intra-class variation; the face result uses three DCGAN generators within MAD-GAN.

6. Conclusion

MAD-GAN is presented as a simple framework for generating diverse, meaningful samples, supported by empirical comparisons and theoretical analysis. The paper identifies estimating the required number of generators as an unresolved direction.

  • MAD-GAN generates diverse and meaningful samples while capturing diverse modes and producing high-quality outputs.The framework is compared with various GAN variants and analyzed theoretically under conditions for global optimality.
  • The number of generators needed for a particular dataset remains difficult to estimate because the true data distribution is unavailable.The paper also leaves open how generator count should relate to data-distribution complexity.
  • MAD-GAN-Sim extends the framework with a similarity-based competing objective for encouraging diverse samples.The objective uses a user-defined similarity function to determine diversity.

A. Insights for Diversity in MAD-GAN

MAD-GAN’s diversity mechanism relies on multiple generators and a shared discriminator, while MAD-GAN-Sim uses task-specific similarity constraints. The analysis explains when generator collapse can occur and how training avoids it in practice.

  • Insights for Diversity in MAD-GAN: MAD-GAN can theoretically reach its minimum objective even when all generators learn the data distribution identically.Thus, the architecture does not theoretically rule out identical generator behavior.
  • Insights for Diversity in MAD-GAN: Random initialization and sufficient discriminator and generator capacity help avoid the trivial solution in which all generators focus on the same data region.Experiments support diverse generations under these conditions.
  • B. Similarity based competing objective: MAD-GAN-Sim adds a diversity term that makes generator dissimilarity depend on a user-defined task-specific function.The approach is motivated by the expectation that samples from different modes should differ in visual cues such as texture, color, or shading.
  • B. Similarity based competing objective: The MAD-GAN-Sim architecture otherwise matches MAD-GAN, with the competing objective placed at the discriminator’s end.MAD-GAN uses generator identification, whereas MAD-GAN-Sim uses competing objectives.
  • B.1. Approach: The similarity objective maps generated samples into a feature space and increases the discriminator-score margin when two samples are more similar.The similarity function lies in [0, 1], and the average constraint is used because it trains more stably than the maximally violated constraint.
  • B.1. Approach: MAD-GAN-Sim computes constraints by passing the same noise sample through all generators, then resamples noise when updating another generator.This procedure is described in Algorithm 1.

B.2. Experiments

The paper reports experiments demonstrating the efficacy of MAD-GAN-Sim on real-world datasets.

  • B.2. Experiments: MAD-GAN-Sim is evaluated on real-world datasets.

B.3. Diverse Samples for Image-to-Image Translation

MAD-GAN-Sim produces diverse, meaningful image-to-image translations and improves unsupervised feature representation on SVHN relative to DCGAN.

  • MAD-GAN-Sim generates meaningful and diverse handbags from edges and day images from night images using cosine-based similarity.Each generator produces diverse images in both image-to-image translation settings.
  • 18.3% misclassification error is achieved by MAD-GAN-Sim on SVHN, compared with 22.48% for DCGAN.The paper reports this as evidence of better unsupervised feature representation.

C. Network Architectures and Parameters

The experiments use architectures and training settings tailored to each task, including shared multi-generator layers and generator-identification outputs. MAD-GAN-Sim produces diverse image-to-image outputs across visual attributes and conditions.

  • The non-parametric density-estimation experiment uses Xavier initialization with normally distributed random sampling, while other experiments match their adapted base architectures.
  • MAD-GAN-Sim generates diverse handbag and night-to-day outputs varying in color, texture, lighting, weather, sky patterns, and landscapes.The figure layouts show one input column followed by three generator-output columns.
  • MAD-GAN and MA-GAN use four generators with shared parameters in their first two hidden layers.Each generator has two 128-neuron fully connected hidden layers with exponential linear units.
  • Each generator receives 64-dimensional uniform noise U(−1, 1), while InfoGAN additionally concatenates a five-dimensional categorical code.
  • MAD-GAN uses multi-label cross-entropy and samples 128 examples from every generator and the real distribution per mini-batch.Training uses Adam with batch size 128 and learning rate 1e−4; MA-GAN instead uses binary cross-entropy and combines generator samples.
  • The synthetic density-estimation dataset is a five-component Gaussian mixture with means 10, 20, 60, 80, and 110, including overlapping and isolated modes.The first two modes overlap significantly, while the fifth is peaky and isolated.

C.2. Stacked and compositional MNIST Experiments

The experiments use MNIST-derived datasets and U-Net or DCGAN-style architectures, with MAD-GAN variants modifying discriminator outputs and generator inputs to represent multiple modes. Image-to-image settings add categorical codes or noise to control diversity.

  • Stacked and compositional MNIST: Stacked- and compositional-MNIST experiments use MNIST handwritten-digit data with architectures based on prior stacked-MNIST and DCGAN designs.
  • Image-to-image translation: The image-to-image architecture uses a U-Net generator and a patch-based discriminator with 4 × 4 stride-2 convolutions and 50% dropout in CDk layers.
  • Stacked and compositional MNIST: MAD-GAN discriminator architectures adapt the final output to k + 1 dimensions for k generators, whereas InfoGAN uses a Q network for salient variables.The stacked-MNIST and compositional-MNIST discriminator tables specify these task-specific output adaptations.
  • Image-to-image translation: The U-Net decoder uses skip-connections, and MAD-GAN keeps the decoder’s last layer separate across generators.The decoder specification is CD512-CD1024-CD1024-C1024-C1024-C512-C256-C128.
  • Image-to-image translation: MAD-GAN-Sim maps discriminator features to a one-dimensional sigmoid output and uses penultimate-layer activations for cosine-similarity features.
  • Image-to-image translation: The diversity term represents three modes with a categorical code appended channel-wise to the input image; one experiment additionally appends a spatial matrix of normal noise.Training uses Adam at learning rate 2e−4, L1 weight 10, and batch size 1.

C.4. Diverse-Class Data Generation

Diverse-class and face-generation experiments adapt DCGAN with residual layers and multiple generators, using parameter sharing differently across settings. The evaluation also includes SVHN feature extraction and generator-count ablations.

  • Diverse-class data: The diverse-class dataset contains 24,000 randomly selected images from islets, icebergs, broadleaf forests, bamboo forests, and bedrooms.
  • Diverse-class data: The diverse-class architecture uses DCGAN with residual layers and shares all generator parameters except the last layer.The residual layers address the complexity of the data and generator manifolds by providing the discriminator more capacity.
  • Diverse-class data: The discriminator uses a k + 1-dimensional softmax output for MAD-GAN with k generators.The architecture tables identify this output design for diverse-class generation and face generation.
  • Face and image generation: The face-generation setup uses CelebA, while ImageNet-1k training uses 14,197,122 images with three generators and 64 × 64 resizing.
  • Unsupervised feature representation: SVHN features extracted from the discriminator are used to train a regularized linear L2-SVM, with an ablation varying the number of generators.The evaluation uses the SVHN train, validation, and test splits described in the dataset preparation.
Loading 1704.02906v3…