Mode Regularized Generative Adversarial Networks

Tong Che; Yanran Li; Athul Paul Jacob; Yoshua Bengio; Wenjie Li

Source-linked AI summary

Mode Regularized Generative Adversarial Networks

Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, Wenjie Li

arXiv:1612.02136v5Dec 7, 2016cs.LGcs.AIcs.CVcs.NE

TL;DR

GANs are unstable and prone to missing modes because discriminator behavior can provide unreliable training guidance. The paper introduces regularizers based on more predictable geometric signals and evaluates mode coverage and probability-mass fairness. The authors report reduced variance, more stable training, and a unified remedy for missing modes, with a variant improving sample quality over DCGAN.

Problem
GANs are hard to train and often collapse probability mass onto a few modes, producing lower-entropy samples with reduced variety.
Method
The paper adds geometric and autoencoder-based regularizers that provide stable similarity signals, penalize missing modes, and support mode-coverage evaluation.
Results
The regularizers reduce model variance, stabilize training, address missing modes, and can improve sample quality relative to the DCGAN baseline.
Takeaways & Limitations
The approach provides a unified way to stabilize GAN training while encouraging fairer probability-mass distribution across generated modes.
Takeaways & Limitations
On large datasets such as CelebA, diversity improvements may come with lower sample quality unless hyperparameters are carefully tuned.

Abstract

from arXiv · show

Although Generative Adversarial Networks achieve state-of-the-art results on a variety of generative tasks, they are regarded as highly unstable and prone to miss modes. We argue that these bad behaviors of GANs are due to the very particular functional shape of the trained discriminators in high dimensional spaces, which can easily make training stuck or push probability mass in the wrong direction, towards that of higher concentration than that of the data generating distribution. We introduce several ways of regularizing the objective, which can dramatically stabilize the training of GAN models. We also show that our regularizers can help the fair distribution of probability mass across the modes of the data generating distribution, during the early phases of training and thus providing a unified solution to the missing modes problem.

1 INTRODUCTION

GANs achieve strong results but remain difficult to train, with instability and missing modes reducing generated diversity. The paper attributes these problems to uncontrolled discriminator behavior and proposes regularizers using more predictable similarity signals.

Motivation: GAN training is unstable and sensitive to hyper-parameters, often collapsing probability mass onto only a few data modes.Missing modes produce lower-entropy samples with less variety than the data distribution.
Motivation: The authors argue that discriminator functions can become highly nonlinear between real and generated manifolds, creating bad plateaus and wrong maxima.These shapes can provide harmful gradients to the generator.
Approach: The proposed regularizers supplement discriminator gradients with predictable similarity metrics such as the L2 norm.The paper combines this idea with penalties intended to address missing modes.
Evaluation: The paper introduces metrics for evaluating both mode diversity and fairness of probability mass, including well-trained and collapsed generators.These metrics are presented as more robust for judging complex generative models.
Results: When correctly applied, the regularizers reduce model variance, stabilize training, and address missing modes without negative effects on generated samples.A variant can also improve sample quality relative to the DCGAN baseline.

2 RELATED WORK

Related work develops GAN architectures, conditioning methods, feature-based stabilization, and joint latent-image approaches. The paper positions its autoencoder-based losses as regularizers rather than sample-generating components.

GAN foundations: The original GAN framework uses deep neural networks for both the generator and discriminator.
Conditional models: Conditional GAN variants add information vectors or conditioning inputs for tasks including image prediction, text synthesis, and edge-map synthesis.
Architectural extensions: LAPGAN and DCGAN extend GANs toward higher-resolution and photo-realistic image generation through convolutional architectures.
Training stabilization: Feature matching and related intermediate-feature statistics were proposed to stabilize GAN training.
Joint representations: Other approaches combine feature-space and image-space information, sometimes training an encoder jointly with the generator and discriminator.
Positioning: Unlike VAEGAN, this paper uses its jointly trained autoencoder-based losses to penalize missing modes and improve training stability and sample quality.

3 MODE REGULARIZERS FOR GANS

The paper attributes GAN instability and mode missing to poorly controlled discriminator behavior, then introduces metric and mode regularizers to provide more stable gradients and distribute probability mass across modes.

Motivation: GAN training uses the discriminator as a learned evaluation metric, whose local gradient ∇log D(G) directs generated samples toward the data manifold.Unlike fixed distance functions, this learned target can provide unstable gradients during early training.
Motivation: When data and generation manifolds are disjoint, the discriminator can become nearly zero around generated samples, causing vanishing gradients.The discriminator objective does not directly constrain its behavior away from the two manifolds.
Geometric metrics regularizer: The metric regularizer adds an encoder-based reconstruction loss Ex∼pd[d(x, G∘E(x))] so geometric similarity metrics supplement discriminator gradients.The distance d may be pixel-wise L2 or a learned feature distance.
Mode regularizer: The mode regularizer is added to the GAN objective to further penalize missing modes and encourage fairer probability-mass distribution.The proposed regularizers target both training stability and the missing-mode problem.
Mode regularizer: Missing modes arise because rarely visited regions receive few corrective updates while the discriminator may assign high values near both missing and represented modes.Most generator gradients can therefore push samples toward a major mode, while a minor mode is updated only when generated samples are already nearby.
Manifold-diffusion training and evaluation: The manifold-diffusion algorithm separates training into manifold matching and probability-mass diffusion, using an encoder, geometric loss, and two discriminators.Experiments estimate mode missing and sample quality with multiple metrics, including MODE score and a third-party discriminator.

4 EXPERIMENTS

Experiments on MNIST and CelebA evaluate regularized GANs for stability, sample quality, and mode coverage. Across quantitative and qualitative tests, regularization improves mode preservation and often maintains or improves visual quality.

4.1 MNIST: MNIST experiments use ten dominant modes, defined as connected components of the data manifold, and a large-scale hyperparameter grid search.The grid search uses λ1 = 0.2 and λ2 = 0.4, with identical settings for GAN and Regularized GAN.
4.1.1 Grid Search for MNIST GAN Models: The regularizer significantly improves MODE scores, demonstrating benefits for GAN stability and generated sample quality.MODE scores are computed using a pretrained four-layer CNN classifier trained on MNIST digits.
4.1.2 Compositional MNIST Data with 1000 Modes: On compositional MNIST with 1000 modes, regularization substantially reduces both missed modes and KL divergence.#Miss counts classifier-reported modes never generated, while KL measures divergence between generated and training distributions.
4.2.1 Missing Modes Estimation on CelebA: On CelebA, Regularized-GAN and MDGAN outperform baseline DCGAN across all tested settings, with MDGAN showing the strongest mode-preservation performance.Regularized GAN is also more consistent than DCGAN when the prior noise dimension changes from 100 to 200.
4.2.1 Missing Modes Estimation on CelebA: MDGAN captures some difficult small modes missed by all models and performs well on sideface, paleface, black, and beret attributes missed only by DCGAN.The hardest shared-missed examples include rare caps and background types.
4.2.2 Qualitative Evaluation of Generated Samples: Qualitatively, MDGAN and Regularized-GAN generate clear, natural-looking faces, while MDGAN exhibits fewer distortions and sharper, more coherent samples than several compared models.The authors attribute reduced distortions to autoencoder regularization altering generation manifolds and helping learn fine-grained details.

5 CONCLUSIONS

The paper concludes that GAN instability and missing modes are linked to limited control over discriminator behavior. Autoencoder-based regularizers provide more stable geometric gradients while penalizing missing modes and encouraging fairer probability allocation.

5 Conclusions: GAN training is highly unstable, hyperparameter-sensitive, and prone to missing modes or concentrating probability mass on a few modes.Successful training often requires substantial human and computational effort for hyperparameter tuning.
5 Conclusions: The proposed autoencoder-based regularizers systematically measure and reduce missing modes while stabilizing training.Their key mechanism combines geometric metrics that provide more stable gradients with an encoder-based regularization strategy.

A APPENDIX: PSEUDO CODE FOR MDGAN

The appendix outlines MDGAN training as a manifold step followed by a diffusion step. It presents the detailed training procedure for an MDGAN example.

A Appendix: Pseudo Code for MDGAN: The appendix provides the detailed training procedure for an MDGAN example discussed earlier in the paper.The procedure is presented as pseudocode and includes a manifold-training stage.
A Appendix: Pseudo Code for MDGAN: The manifold step samples data points and updates discriminator D1 using stochastic-gradient ascent.The pseudocode begins by sampling {x1, x2, · · · xm} from pdata(x).

B APPENDIX: ARCHITECTURE FOR EXPERIMENTS

The experimental architecture retains DCGAN’s generator and discriminator while adding an encoder that reverses the generator’s layer structure. Batch-normalization statistics are separated for noise-derived and encoder-derived generator inputs.

B Appendix: Architecture for Experiments: The compositional MNIST and CelebA architectures retain DCGAN’s generator and discriminator and add an encoder as the generator’s inverse.The encoder reverses the layer order and replaces deconvolutional layers with convolutional layers.
B Appendix: Architecture for Experiments: The generator uses separate batch statistics for sampled-noise and encoder-derived inputs while sharing batch-normalization parameters.This addresses the two data classes passing through generator batch-normalization layers.

C APPENDIX: ADDITIONAL SYNTHESIZED EXPERIMENTS

The appendix evaluates mode-regularized GANs on a synthesized six-mode 2D Gaussian mixture using a simple GAN architecture. Compared with standard GAN training, the regularized model better preserves the target modes during training.

Experimental setup: The experiment uses a six-Gaussian mixture arranged around a circle and a simple generator producing 2D samples from 3D uniform noise.Both generator and discriminator are trained with Adam at learning rate 1e-4.
Experimental setup: The regularized model uses λ1 = λ2 = 0.005 for the comparison with standard GAN training.
Results: Figure 9 compares generator-distribution heatmaps across training epochs with the target data distribution.The columns show training progress, while the rightmost column presents the target distribution.
Results: Standard GAN training oscillates among modes and recovers only one data mode at a time.
Results: The regularized GAN is contrasted with standard GAN because it is intended to distribute probability mass across the target modes more effectively.

D APPENDIX: COMPARISON WITH VAEGAN

The appendix compares Reg-GAN and MDGAN with VAEGAN theoretically and empirically. The paper attributes their differences to using plain auto-encoder regularization without VAE-specific assumptions, reporting better sample quality and mode preservation, while noting tuning limits on large datasets.

Comparison scope: The comparison evaluates theoretical differences, sample quality, and missing modes between the proposed mode-regularized GANs and VAEGAN.
Theoretical comparison: VAEGAN relies on a variational bound and assumes the true posterior can be approximated by a factorized Gaussian.
Theoretical comparison: The paper argues that GAN posteriors need not satisfy VAEGAN’s factorized-Gaussian assumption, motivating a plain auto-encoder objective instead of a VAE objective.The proposed regularization is intended to avoid conflicts between a good GAN generator and the regularization objective.
Evaluation protocol: The VAEGAN comparison uses official code, with VAEGAN trained for 30 epochs and the proposed models for 20 epochs; the best of three VAEGAN runs was selected visually.
Empirical comparison: The proposed regularizers produce a huge drop in the number of missing modes, while five-person blind evaluation favors their sample diversity four to one, with one tie.
Empirical comparison: The paper concludes that Reg-GAN and MDGAN achieve better sample quality and mode-preserving ability than VAEGAN, while auto-encoder regularization is associated with fewer fine-grained distortions.