PaperlayerView exact revision on arXiv

Source-linked AI summary

Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis

Qi Mao, Hsin-Ying Lee, Hung-Yu Tseng, Siwei Ma, Ming-Hsuan Yang

arXiv:1903.05628v6cs.CV

TL;DR

cGANs often neglect noise vectors in multimodal conditional generation, causing mode collapse, while existing remedies can be task-specific and computationally expensive. MSGAN maximizes generated-image distance relative to latent-code distance and integrates this term into existing cGANs. Across categorical, image-to-image, and text-to-image synthesis, it improves diversity while maintaining visual quality without changing network architectures or adding training overhead.

  • Problem

    cGANs can ignore noise vectors that should provide output variation, causing mode collapse, while some existing remedies require task-specific components and objectives.

  • Method

    MSGAN adds a generator regularizer that maximizes the ratio of generated-image distance to corresponding latent-code distance.

  • Results

    Across categorical generation, image-to-image translation, and text-to-image synthesis, MSGAN improves diversity without sacrificing visual quality.

  • Takeaways & Limitations

    The regularization term is readily integrated into existing cGAN frameworks without training overhead or network-structure modifications.

Abstract

from arXiv · show

Most conditional generation tasks expect diverse outputs given a single conditional context. However, conditional generative adversarial networks (cGANs) often focus on the prior conditional information and ignore the input noise vectors, which contribute to the output variations. Recent attempts to resolve the mode collapse issue for cGANs are usually task-specific and computationally expensive. In this work, we propose a simple yet effective regularization term to address the mode collapse issue for cGANs. The proposed method explicitly maximizes the ratio of the distance between generated images with respect to the corresponding latent codes, thus encouraging the generators to explore more minor modes during training. This mode seeking regularization term is readily applicable to various conditional generation tasks without imposing training overhead or modifying the original network structures. We validate the proposed algorithm on three conditional image synthesis tasks including categorical generation, image-to-image translation, and text-to-image synthesis with different baseline models. Both qualitative and quantitative results demonstrate the effectiveness of the proposed regularization method for improving diversity without loss of quality.

1. Introduction

Conditional generation often requires multiple plausible outputs, but cGANs can ignore noise vectors and collapse onto only a few modes. MSGAN adds a simple regularization term and improves diversity across three conditional image-synthesis tasks without sacrificing image quality.

  • Motivation: Conditional mappings are multimodal, with contexts determining content and noise vectors providing output variations.Examples include class-conditioned generation, image-to-image translation, and text-to-image synthesis.
  • Motivation: cGANs often focus on high-dimensional conditional contexts, causing noise vectors to have minor effects and generators to miss distribution modes.The resulting mode collapse produces samples from only a single or few modes.
  • Method: MSGAN maximizes the ratio between distances in generated images and their corresponding latent codes, encouraging generators to explore minor modes.The regularizer can also provide discriminators with gradients from generated samples representing otherwise ignored modes.
  • Experiments: The method is evaluated with different baseline models on categorical generation, image-to-image translation, and text-to-image synthesis.The evaluation uses DCGAN, Pix2Pix, DRIT, and StackGAN++ across several datasets.
  • Evaluation: NDB, JSD, and FID evaluate distribution similarity, mode coverage, and image quality alongside diversity.The evaluation explicitly checks whether improved diversity comes without reduced realism.
  • Results: Across the evaluated tasks, the regularization improves diversity without loss of image quality and requires marginal overhead with minimal framework changes.The reported contributions emphasize generalizability across conditional generation settings.

2. Related Work

Prior mode-collapse remedies use discriminator changes or auxiliary networks, while cGAN-specific approaches often rely on task-specific encoders and objectives. MSGAN instead uses a simple regularization term intended for broad conditional-generation use with no added training overhead.

  • Conditional generative adversarial networks: Conditional GANs synthesize images from external contexts such as low-resolution images, visual domains, and other conditioning information.The related-work discussion places MSGAN within the broader GAN and cGAN image-synthesis framework.
  • Motivation: Figure 2 motivates detecting collapse through the ratio of image-distance change to latent-distance change.An anomalous ratio appears when nearby latent vectors map disproportionately close together.
  • Reducing mode collapse: GAN mode-collapse methods modify discriminator optimization or divergence metrics, while others add auxiliary networks such as multiple generators or encoders.These approaches target missing modes through discriminator-side or generator-supporting mechanisms.
  • cGAN mode collapse: Existing cGAN diversity methods introduce additional encoders and task-specific objectives, which reduce generalizability and increase training loads.The cited approaches focus primarily on image-to-image translation.
  • MSGAN: MSGAN contrasts with those approaches by using a simple regularization term that requires no training overhead or network-structure changes and applies across tasks.The paper notes a concurrent work using a similar loss term for cGAN mode-collapse reduction.

3. Diverse Conditional Image Synthesis

MSGAN addresses cGAN mode collapse from the generator perspective by separating outputs associated with different latent codes. It appends this regularizer to existing objectives so the generator explores more modes while adversarial training preserves distributional realism.

  • GAN training: Adversarial training guides the generator toward realistic samples by providing gradients from a discriminator that distinguishes real and generated data.The GAN training process is formulated as a discriminator-generator mini-max problem.
  • Mode collapse: Mode collapse occurs when generators favor larger modes and fail to generate samples from minor modes.The literature attributes missing modes partly to the lack of a penalty when collapse occurs.
  • Conditional mode collapse: In cGANs, high-dimensional structured contexts can dominate noise vectors, worsening mode missing despite noise being responsible for diversity.Images and sentences are examples of such conditional contexts.
  • Mode Seeking GANs: MSGAN maximizes the distance between G(c, z1) and G(c, z2) relative to the distance between z1 and z2.The distance metric d*(·) is used to measure the relevant image and latent-code distances.
  • Mode Seeking GANs: The regularizer encourages exploration of the image space and can make the discriminator attend to samples from minor modes.This interaction is described as increasing the chance that unexplored modes receive subsequent adversarial training signals.
  • Integration: The new objective appends the mode-seeking term to the original cGAN objective, with λms controlling its importance.The original objective may be a simple or task-specific complex loss.

4. Experiments

Experiments apply MSGAN to categorical generation, image-to-image translation, and text-to-image synthesis while preserving existing architectures and evaluating diversity, distribution similarity, and quality. Across these tasks, MSGAN improves mode coverage and diversity while maintaining visual quality.

  • Evaluation metrics: NDB and JSD measure similarity between generated and real-data bin distributions, while FID evaluates generated-image quality.Lower NDB, JSD, and FID indicate closer distributional similarity or better quality; LPIPS is used for diversity where applicable.
  • Conditioned on Class Label: On CIFAR-10, MSGAN mitigates mode collapse in most classes while maintaining image quality.The categorical-generation evaluation reports NDB, JSD, and FID results; LPIPS is omitted because 32×32 images make upsampling degrade quality.
  • Conditioned on Paired Images: On facades and maps, MSGAN improves all metrics over Pix2Pix and achieves diversity comparable to BicycleGAN.BicycleGAN uses an additional encoder network, whereas MSGAN retains the baseline framework and adds the proposed regularization.
  • Conditioned on Unpaired Images: On Yosemite and cat⇌dog translation, MSGAN performs favorably against DRIT on all metrics and gains substantial diversity on the challenging cat⇌dog task.The bin visualization shows severe mode collapse for DRIT and improvement with MSGAN; qualitative results report more modes without loss of visual quality.
  • Conditioned on Text: On CUB-200-2011, MSGAN improves StackGAN++ diversity while maintaining visual quality, with fixed text codes producing varied bird appearances, poses, and backgrounds.Interpolation results also show smooth changes in coat colors and patterns or bird orientations and foothold appearances as latent codes vary.

5. Conclusions

The paper presents a mode seeking regularization term for cGANs that improves diversity across three conditional generation tasks without sacrificing visual quality.

  • The regularization term addresses mode collapse in cGANs by maximizing generated-image distance relative to latent-code distance.
  • The method integrates with existing cGAN frameworks without training overhead or network-structure modifications.
  • The method generalizes across categorical generation, image-to-image translation, and text-to-image synthesis.
  • Qualitative and quantitative results show improved diversity without sacrificing the visual quality of generated images.

Appendix A. Implementation Details

The implementation adds the proposed regularization term to existing baselines while largely retaining their original architectures and training settings, with task-specific details for DCGAN, Pix2Pix, DRIT, and StackGAN++.

  • All baseline methods retain their original objective functions while incorporating the proposed regularization term.The original network architecture design and default hyper-parameter settings are retained.
  • DCGAN: For CIFAR-10, DCGAN uses modified generator and discriminator structures for 32×32 images, with batch size 32 and learning rate 0.0002.Training uses Adam with β1 = 0.5 and β2 = 0.999 for both baseline and MSGAN.
  • Pix2Pix: Pix2Pix uses a BicycleGAN-style U-Net generator with latent codes injected into every layer and a two-scale PatchGAN discriminator.Pix2Pix and MSGAN use the officially released version 1 hyper-parameters.
  • DRIT: DRIT applies the mode seeking regularization only to generators in the first translation stage.The first stage is modified using the officially released code.
  • StackGAN++: StackGAN++ computes the regularization term from the last generator’s output images and input latent codes.The implementation is based on the officially released code.

Appendix B. Evaluation Details

Evaluation samples are generated repeatedly for each conditional context, and diversity, distribution similarity, and image quality are measured using standardized metrics across conditioning types.

  • FID, NDB, JSD, and LPIPS are evaluated using official implementations, with K-means clusters used for NDB and JSD.The number of bins is set to K ≈ Ntrain/20, and performance is reported as consistent across a large range of K.
  • Results average five independent trials and report the mean and standard derivation from each trial.
  • Conditioned on Class Label: For class-conditioned generation, 5000 images are generated per class, with FID computed against all training samples and 250 clusters used for NDB and JSD.
  • Conditioned on Image: For image-conditioned generation, 50 images are generated per test input, while LPIPS uses 50 image pairs per context.A total of 5000 generated samples is formed for FID, and 100 input images are randomly selected.
  • Conditioned on Text: For text-conditioned generation, 10 images are generated for each of 200 sentences, producing 2000 samples for evaluation.FID uses 2000 training samples, while NDB and JSD use 100 bins; LPIPS averages 10 pairs per sentence.

C.1. The Weighting Parameter λms

The ablation study varies λms on facades and finds that larger weights improve diversity until excessive weighting destabilizes training and reduces quality.

  • Increasing λms improves the diversity of generated images on the facades dataset.
  • When λms exceeds 1.0, training becomes unstable and produces low quality and potentially low diversity.
  • Table 6 reports quantitative results for different λms values on facades.

C.2. The Design Choice of the Distance Metric

The paper compares image-distance choices for its mode-seeking regularizer and finds that L1 distance provides higher diversity than discriminator feature distance, while retaining marginal computational overhead.

  • Distance metric comparison: Discriminator feature distance improves over Pix2Pix on facades, but L1 distance achieves higher diversity.The feature-distance variant follows feature matching loss and uses discriminator-layer representations.
  • Distance metric comparison: Table 7 reports quantitative results for Pix2Pix and MSGAN variants on the facades dataset.The supplied table caption identifies the evaluation as quantitative results on facades.
  • Computational overhead: MSGAN incurs marginal computational overheads, whereas BicycleGAN requires longer per-iteration time and more memory.BicycleGAN also uses an additional encoder and another discriminator network.

Appendix E. Additional Results

The appendix provides additional qualitative results across categorical generation, image-to-image translation, and text-to-image synthesis, alongside architecture and dataset-statistics documentation.

  • Task and architecture documentation: Table 9 summarizes training and testing image counts and the baseline model for each generation task.The appendix therefore documents dataset statistics alongside its additional visual results.
  • Categorical generation: Additional categorical-generation results are shown for DCGAN with mode-seeking regularization on CIFAR-10.The appendix also provides the generator and discriminator architecture for DCGAN.
  • Image-to-image translation: Additional image-to-image translation results cover facades, maps, Yosemite Summer→Winter, Yosemite Winter→Summer, Cat→Dog, and Dog→Cat.The figures organize facades and maps by rows and provide separate results for the Yosemite and animal translation directions.
  • Text-to-image synthesis: Additional text-to-image synthesis results are shown for CUB-200-2011.The appendix also includes example text describing a bird with a small fat bill and red belly.
Loading 1903.05628v6…