PaperlayerView exact revision on arXiv

Source-linked AI summary

DeLiGAN : Generative Adversarial Networks for Diverse and Limited Data

Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, Venkatesh Babu Radhakrishnan

arXiv:1706.02071v1cs.CV

TL;DR

Typical GANs require large datasets to capture diversity in complex image modalities. DeLiGAN learns a Gaussian-mixture latent space jointly with GAN parameters, and experiments show diverse generation in limited-data settings across several modalities, while remaining subject to capacity and mixture assumptions.

  • Problem

    Typical GAN-based approaches require large amounts of training data to capture diversity across complex image modalities.

  • Method

    DeLiGAN reparameterizes the GAN latent space as a mixture of Gaussians and learns the mixture parameters along with the GAN parameters.

  • Results

    DeLiGAN generates diverse images for handwritten digits, photo objects, and hand-drawn sketches in limited-data regimes.

  • Takeaways & Limitations

    Across digits, sketches, and color photos, the modification helps stabilize training and produce diverse samples even with low data.

  • Takeaways & Limitations

    The model is constrained by the underlying GAN’s modeling capacity and uses simplifying assumptions of diagonal component covariances and equally weighted components.

Abstract

from arXiv · show

A class of recent approaches for generating images, called Generative Adversarial Networks (GAN), have been used to generate impressively realistic images of objects, bedrooms, handwritten digits and a variety of other image modalities. However, typical GAN-based approaches require large amounts of training data to capture the diversity across the image modality. In this paper, we propose DeLiGAN -- a novel GAN-based architecture for diverse and limited training data scenarios. In our approach, we reparameterize the latent generative space as a mixture model and learn the mixture model's parameters along with those of GAN. This seemingly simple modification to the GAN framework is surprisingly effective and results in models which enable diversity in generated samples although trained with limited data. In our work, we show that DeLiGAN can generate images of handwritten digits, objects and hand-drawn sketches, all using limited amounts of data. To quantitatively characterize intra-class diversity of generated samples, we also introduce a modified version of "inception-score", a measure which has been found to correlate well with human assessment of generated samples.

1. Introduction

GANs generate realistic images but typically require substantial training data to capture complex, diverse image distributions. DeLiGAN targets small-yet-diverse datasets by enabling diverse generation across multiple image modalities and introducing a modified inception score.

  • GAN background: GANs use generator and discriminator networks to produce images intended to be indistinguishable from real data.The generator transforms random vectors into images, while the discriminator estimates whether images come from the true distribution.
  • Motivation: Complex image categories combine high visual detail with substantial inter- and intra-category diversity.Deep, high-capacity generators are typically used to model these factors, but they require large amounts of training data.
  • Contributions: DeLiGAN is proposed for small-yet-diverse data scenarios.The framework is designed to generate diverse images despite limited training data.
  • Contributions: The paper demonstrates modality-specific DeLiGAN models for handwritten digits, photo objects, and hand-drawn sketches.These experiments evaluate generation across several distinct image modalities under limited-data regimes.
  • Contributions: The authors introduce a modified inception score to quantitatively characterize intra-class diversity in generated samples.The measure is described as correlating well with human assessment of generated samples.

2. Related Work

Prior work improved GAN image quality and training stability, while related latent-space methods mainly arose in variational inference or alternative latent-space learning procedures. DeLiGAN places a mixture model directly in the GAN latent space.

  • GAN-related work: Deep convolutional GANs and algorithmic techniques such as mini-batch discrimination improved image quality and training stability.These developments extended the original GAN framework with deeper networks and training-oriented modifications.
  • Latent distributions: Mixture models have commonly been used to represent complex approximate posteriors in variational inference.Related work includes mixture-based latent distributions and normalizing flows for constructing more complex densities.
  • Latent-space modification: Other approaches modify latent spaces by alternating latent-factor and generator training or by using MCMC sampling.These methods explore learned latent distributions in variational or adversarial autoencoders.

3. Generative Adversarial Networks (GANs)

A GAN learns a generator that maps samples from a simple latent prior to images and a discriminator that distinguishes generated from real data. Adversarial optimization trains both networks toward indistinguishable generated samples.

  • GAN components: A typical GAN contains a generator G and discriminator D, usually implemented as two neural networks.The generator maps a random vector z to an image, while the discriminator evaluates whether an image belongs to the true data distribution.
  • GAN components: The generator transforms a latent vector z from an easy-to-sample distribution into an image xG = G(z).The standard prior can be uniform, such as z ∼ U(−1, 1).
  • Training objective: GAN training assumes a fixed, easy-to-sample prior pz(z) and trains the generator to produce samples from the data distribution.The generator seeks to maximize the probability that generated samples belong to the true data distribution.
  • Adversarial training: The discriminator outputs the probability pD(I) that an input image comes from the true data distribution.It is trained to assign low probability to generated images and high probability to real training data.
  • Adversarial training: Generator and discriminator are trained adversarially through alternating optimization phases.The generator maximizes D(G(z)), while the discriminator separates generated samples from real data.
  • Training objective: At proper convergence, generated images are essentially indistinguishable from real images, with pD(G(z)) = 0.5.The corresponding GAN objective reaches its lowest Jensen–Shannon divergence value.

4. Our model - DeLiGAN

DeLiGAN increases the modeling power of the latent prior rather than network depth by representing latent space as a trainable Gaussian mixture. Samples use reparameterization, while mixture parameters are learned jointly with GAN parameters and regularized against variance collapse.

  • Model motivation: DeLiGAN addresses limited, diverse data by reparameterizing the latent space as a Mixture-of-Gaussians model.The approach increases prior modeling power instead of increasing generator depth.
  • Latent sampling: Latent samples are generated by selecting one Gaussian component and applying the reparameterization trick.Each component uses a diagonal covariance matrix, and the selected Gaussian sample is expressed using its parameters and auxiliary noise.
  • Joint learning: The model learns the mixture means µ and scales σ jointly with the GAN parameters to maximize generated-data probability.The learned latent mixture is optimized through gradients arising from the generator loss.
  • Initialization: Mixture components are initialized with means sampled from U(−1, 1) and scales set to 0.2.The mixture weights are uniform and not trainable in the described model.
  • Training stabilization: An L2 regularizer prevents Gaussian scales σ from collapsing to zero during training.Without regularization, generator optimization can reduce scales to concentrate samples near high-probability regions.
  • Training procedure: Increasing samples per Gaussian improves gradient accuracy and can speed training through averaging over auxiliary noise.The procedure extends naturally to batches of images for mini-batch training.

5. Experiments

The experiments evaluate DeLiGAN and GAN variants across toy, MNIST, CIFAR-10, and sketch data, using modified inception score to quantify sample diversity. Across limited-data settings, DeLiGAN generally produces more diverse and stable generations, while its mixture size introduces a memory–capacity trade-off.

  • Model and Comparisons: DeLiGAN uses a learnable mixture of Gaussian components in latent space, with N chosen empirically according to data complexity.The experiments use 50–100 components; increasing N raises memory requirements and eventually yields redundant components.
  • Modified Inception Score: The modified inception score (m-IS) measures intra-class diversity together with generated-sample quality.It compares classifier label distributions for samples predicted to belong to the same class and reports per-class and combined scores.
  • Toy Data: On bimodal toy data, DeLiGAN models both distributions and their local Gaussian structure more convincingly than baseline variants.The other higher-parameter variants model the two modes but struggle with local structure, while baseline training is more unstable in practice.
  • MNIST: With 500 MNIST training images, DeLiGAN produces crisper, more diverse digits and remains more stable than baseline GAN.Baseline GAN frequently diverges and can produce homogeneous non-digit samples; the deformation artifact is much less common with DeLiGAN.
  • CIFAR-10: On CIFAR-10 reduced to 2000 images, DeLiGAN produces visibly diverse samples and captures under-represented categories better than baseline GAN.GAN performs better for visually similar cat and dog categories, while DeLiGAN achieves stronger coverage of other under-represented categories and outperforms traditional GANs overall.
  • Hand-drawn Sketches: For hand-drawn sketches, DeLiGAN samples are visibly better and more diverse than GAN samples for dissimilar categories, as reflected by m-IS scores.The corresponding comparison uses sketches from five dissimilar categories.

6. Discussion

DeLiGAN’s learnable Gaussian-mixture latent space improves stability and diversity in low-data, multimodal settings by shifting components toward high-probability regions. The approach also avoids sampling low-probability voids, but remains constrained by the underlying GAN framework.

  • Latent-space mixture model: Learnable Gaussian mixtures can approximate arbitrarily complex latent distributions when enough components are used.This replaces the conventional unit Gaussian or uniform latent distribution.
  • Latent-space mixture model: Gradient updates move mixture means toward regions that increase discriminator scores and generated-sample probability.Components initialized in lower-probability regions gradually drift toward regions producing higher-probability samples.
  • Empirical behavior: DeLiGAN produces very few non-digit-like samples by emulating low-probability voids between data modes in its latent distribution.The paper illustrates this behavior with toy experiments and MNIST results compared with baseline GAN.
  • Empirical behavior: In multimodal low-data settings, baseline GAN can fit the mode containing the most data, overlooking other modes and reducing generated diversity.Limited data can make image-space modes appear more numerous because connecting examples are unavailable.
  • Scope and implications: DeLiGAN’s latent-mixture layer can be added to almost any GAN framework, but generated quality remains limited by the underlying framework’s modelling capacity.The authors expect stronger GAN backbones to yield more realistic, higher-quality samples with the mixture layer.

7. Conclusions and Future Work

The paper concludes that mixture-model reparameterization makes GANs more stable and diverse across several modalities in low-data scenarios. It also identifies simplifying mixture assumptions as a limitation and direction for future work.

  • Conclusions: Across digits, hand-drawn object sketches, and color photos, mixture-model latent spaces produce diverse samples in low-data scenarios.The authors report that the modification also helps stabilize the model.
  • Limitations and future work: The current setup assumes diagonal covariance matrices and equally weighted mixture components.These simplifying assumptions limit approximation of more complex latent distributions.
  • Limitations and future work: Future learning schemes can incorporate these mixture parameters to better approximate the underlying latent distribution.The paper also provides source code for training DeLiGAN models and computing the modified inception score.
Loading 1706.02071v1…