PaperlayerView exact revision on arXiv

Source-linked AI summary

Diversity-Sensitive Conditional Generative Adversarial Networks

Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, Honglak Lee

arXiv:1901.09024v1cs.LGstat.ML

TL;DR

Conditional GANs often collapse diverse conditional distributions into nearly deterministic outputs, despite practical ambiguity in tasks such as video prediction. The paper adds a generator regularizer that encourages latent-code-dependent outputs and applies it across three tasks. The method produces diverse generations, with experiments showing a controllable realism–diversity trade-off and strong comparisons against task-specific approaches.

  • Problem

    Conditional GANs often represent only a small subset of multimodal output distributions and may ignore latent-code variation, especially for high-dimensional image and video generation.

  • Method

    The method directly regularizes the generator to produce different outputs for different latent codes, integrating the regularizer into existing cGAN objectives with λ controlling diversity.

  • Results

    Across image translation, image inpainting, and video prediction, adding the regularization induces diverse outputs; it can improve diversity and realism relative to baselines and task-specific methods.

  • Takeaways & Limitations

    A simple generator-side addition provides a broadly applicable way to obtain stochastic conditional generations while explicitly tuning the balance between realism and diversity.

  • Takeaways & Limitations

    Automatically learning λ and τ to achieve an appropriate realism–diversity balance and match the actual data distribution remains future work.

Abstract

from arXiv · show

We propose a simple yet highly effective method that addresses the mode-collapse problem in the Conditional Generative Adversarial Network (cGAN). Although conditional distributions are multi-modal (i.e., having many modes) in practice, most cGAN approaches tend to learn an overly simplified distribution where an input is always mapped to a single output regardless of variations in latent code. To address such issue, we propose to explicitly regularize the generator to produce diverse outputs depending on latent codes. The proposed regularization is simple, general, and can be easily integrated into most conditional GAN objectives. Additionally, explicit regularization on generator allows our method to control a balance between visual quality and diversity. We demonstrate the effectiveness of our method on three conditional generation tasks: image-to-image translation, image inpainting, and future video prediction. We show that simple addition of our regularization to existing models leads to surprisingly diverse generations, substantially outperforming the previous approaches for multi-modal conditional generation specifically designed in each individual task.

3 METHOD

The method regularizes a cGAN generator to produce different outputs for different latent codes, directly addressing mode collapse without requiring an encoder network. Its formulation is broadly integrable and exposes a control over output diversity.

  • Mode-collapse problem: Standard cGAN optimization can ignore latent-code stochasticity and produce a deterministic mapping from conditioning input to output.Prior encoder-based approaches require architectural modifications and introduce additional training challenges.
  • Generator regularization: The proposed regularization penalizes mode-collapsing behavior by encouraging the generator to map latent codes to different outputs.This replaces a many-to-one latent-code mapping with a more diverse mapping from latent codes to outputs.
  • Objective and control: The full objective adds the regularization to the adversarial objective, with λ controlling its importance and the generator’s degree of stochasticity.For bounded generator outputs, the margin can be removed, leaving λ as the sole additional hyper-parameter.
  • Generality: The regularization is designed to integrate with existing cGAN objectives across different loss functions, architectures, tasks, and data modalities.It can also use alternative distance metrics, including feature-space and sequence-data distances.

4 ANALYSIS OF THE PROPOSED REGULARIZATION

The analysis explains mode collapse through the generator’s latent-code sensitivity: maximizing output differences increases generator gradients and shrinks neighborhoods that can collapse into one mode. This also connects the method to encoder-based approaches while preserving direct diversity control.

  • Generator-gradient connection: The regularization corresponds to a lower bound on the generator’s averaged gradient norm along the line between two latent codes.Optimizing it therefore increases the generator gradient norm with respect to the latent code.
  • Gradient perspective: Increasing generator sensitivity to latent codes is presented as an alternative to discriminator regularization for addressing vanishing-gradient effects.The goal is to spread outputs over the output space from fixed conditioning inputs.
  • Optimization perspective: A single gradient update can move a large neighborhood of latent codes toward the same output mode, causing collapse across many generator outputs.The neighborhood is bounded by a radius determined by the update and local generator behavior.
  • Optimization perspective: Constraining the ratio of output distance to latent-code distance shrinks the neighborhood of latent codes that can collapse into a mode.This directly targets the mechanism by which one update can collapse many latent codes.
  • Connection to BicycleGAN: The method is related to BicycleGAN because both discourage vanishing generator sensitivity to latent codes, but DSGAN directly optimizes generator-gradient magnitude.Unlike BicycleGAN, it also controls diversity through λ.

5 EXPERIMENTS

The experiments apply the same diversity regularization to cGAN baselines across image translation, inpainting, and future-frame prediction, producing diverse outputs while exposing a realism–diversity trade-off.

  • Experimental setup: DSGAN applies the proposed regularization to existing cGAN baselines across image translation, image inpainting, and future frame prediction using the same networks.The method is added to each baseline objective rather than replacing its architecture.
  • Image-to-image translation: LPIPS diversity increases while FID initially decreases as λ grows, but λ = 20 yields FID = 191 and LPIPS = 0.20 with less realistic outputs.The ablation demonstrates a controllable trade-off between visual quality and diversity.
  • Image-to-image translation: Compared with the cGAN baseline, DSGAN and BicycleGAN learn multimodal output distributions, while DSGAN generates more diverse outputs than BicycleGAN.The comparison fixes DSGAN’s λ at 8 across datasets and uses BicycleGAN’s optimal settings.
  • Image-to-image translation: DSGAN maintains consistent performance across latent-code sizes because it samples codes from the prior distribution, avoiding BicycleGAN’s encoder–prior matching difficulty.BicycleGAN’s generation quality degrades with high-dimensional latent codes.
  • High-resolution synthesis: On Cityscape, adding the regularization increases output diversity while evaluating generated images with FID, LPIPS, and segmentation accuracy.The segmentation metric compares input labels with predictions obtained from generated images.
  • Image inpainting and video prediction: In inpainting and video prediction, regularization induces multimodal outputs; DSGAN’s latent-space samples are realistic, diverse, and partly encode meaningful attributes.For video prediction, DSGAN is reported to outperform SAVP in diversity and realism with fewer parameters and simpler training, including on limited-diversity videos.

6 CONCLUSION

The paper concludes that generator regularization is a simple, general way to induce diversity in conditional GANs across three tasks. It identifies learning λ and τ to balance realism and diversity as future work.

  • The proposed regularization integrates broadly into conditional GANs across loss functions, network architectures, and data modalities.
  • Adding the regularization to existing cGAN objectives effectively induces diversity in image translation, image inpainting, and future video prediction.
  • Learning λ and τ so the learned distribution matches the actual data distribution is identified as an interesting direction for balancing realism and diversity.

A DERIVATION OF LOWER-BOUND OF GRADIENT NORM

The derivation formalizes how a generator update affecting one latent code can also affect nearby latent codes. This provides a bound on the neighborhood attracted toward the same mode.

  • The derivation uses a straight-line path γ connecting latent samples z1 and z2, then applies expectation over standard Gaussian latent samples.
  • Modes are defined as sets of images close to a real image, and attraction measures whether a gradient step moves an output toward that mode by ϵ.
  • If z1 is attracted to mode M by ϵ, then every z2 in a neighborhood Nr(z1) is attracted to M by ϵ/2.
  • The neighborhood Nr(z1) can be arbitrarily large, but it is bounded by an open ball of radius r.

C.1 APPLICATION TO UNCONDITIONAL GAN

The unconditional GAN experiment tests whether the regularization alleviates mode collapse on a synthetic mixture of eight Gaussian modes. Compared with vanilla GAN, DSGAN captures substantially more high-quality samples.

  • The experiment trains vanilla GAN and its regularized counterpart on eight 2D Gaussian distributions arranged in a ring, using λ = 0.1.
  • Vanilla GAN places substantial probability mass around a single output mode and produces 0 high-quality samples in the reported evaluation.
  • DSGAN obtains 8% Samples and 81.4% Samples (%) in the quantitative evaluation, compared with vanilla GAN’s 0 and 0.0.
  • The regularization is also evaluated after discriminator-gradient issues are simulated by retraining from a converged baseline discriminator.
  • FID 52.32 and LPIPS 0.16 are achieved with regularization, close to the balanced-discriminator values of FID 57.20 and LPIPS 0.18.
  • The regularization loss decreases quickly early in training, helping exploration when discriminator gradients are vanishing.

D ADDITIONAL EXPERIMENT RESULTS

The supplementary section provides additional experiment details, including baseline architectures for image-to-image translation. These baselines preserve established network structures while adapting stochastic inputs where needed.

  • The supplementary material contains additional experiment details and results omitted from the main paper because of space restrictions.
  • The BicycleGAN baseline uses a U-Net generator and a two-scale PatchGAN-style discriminator with BicycleGAN’s hyperparameters.
  • The pix2pixHD baseline uses one generator and injects latent code after downsampling through spatial tiling and depth-wise concatenation.

D.1.2 EVALUATION METRICS

The evaluation uses perceptual diversity, distributional similarity, and human judgments to assess generated-image diversity and quality. These protocols compare samples generated under the same input conditions.

  • Perceptual metrics: LPIPS measures generated-sample diversity using average pairwise L1 distances between pretrained AlexNet features.Twenty samples are generated for each validation image, and scores are averaged across validation images.
  • Distributional metric: FID compares a generated dataset with the training dataset using samples produced for each validation input.The protocol samples 20 outputs per input and computes FID after resizing training images when needed.
  • Human evaluation: AMT compares visual plausibility and condition matching between our method and baseline cGAN or BicycleGAN.Turkers perform side-by-side comparisons using two image sets generated from the same input condition.

D.1.3 QUALITATIVE RESULTS

Qualitative results show that DSGAN preserves plausible details and variation in image translation, while its latent interpolations remain smooth and meaningful. Compared with BicycleGAN, DSGAN produces more coherent changes across ambiguous regions.

  • Qualitative comparisons: DSGAN and baseline cGAN capture missing shoelace details in edge-to-photo translation, whereas some BicycleGAN outputs omit them.On maps-to-images, DSGAN and baseline cGAN also generate natural, variable vegetation while BicycleGAN tends toward plain texture with global color variation.
  • Latent-space analysis: DSGAN’s Cityscapes latent interpolations produce reasonable intermediate generations with smooth transitions.The result implies a smooth manifold in the learned latent space.
  • Latent-space analysis: On maps-to-images interpolation, DSGAN generates meaningful diversity in ambiguous regions and changes smoothly between latent codes.BicycleGAN instead shows limited meaningful changes and occasional sudden output changes.

D.2 IMAGE INPAINTING

The image-inpainting setup modifies a baseline cGAN for image quality while adding regularization to control diversity. Increasing the regularization weight produces more meaningful, realistic variations, and latent interpolation remains smooth.

  • Network architecture: The inpainting generator predicts a missing region from a masked 256 × 256 image, then combines the prediction with the unmasked input using the binary mask.Feature matching and a patchGAN-style discriminator are added for image quality, independently of the proposed regularization.
  • Regularization analysis: Increasing λ from 2 to 5 changes outputs from subtle facial variations to meaningful differences in hairstyle, age, and identity.The larger weight maintains visual quality and alignment with the input condition while increasing diversity.
  • Latent-space analysis: Interpolated inpainting outputs transition smoothly between two latent-code endpoints, and most intermediate samples appear realistic.The experiment tests whether the learned conditional distribution varies continuously across latent codes.

D.3.1 DATASET

The video-prediction evaluation uses BAIR and KTH videos to predict future frames from two conditioning frames, with testing extended beyond the training horizon. It compares against SAVP and evaluates diversity and closest-sample quality.

  • Dataset protocol: Both datasets use two conditioning frames to train prediction of 10 future frames and test prediction of 28 future frames at 64×64 resolution.The setup uses k = 2 and T = 10 during training, then T = 28 during testing.
  • Datasets: BAIR contains diverse robot-arm movements and objects against a static background, while KTH contains six human activities in static-background videos.KTH videos are augmented by randomly skipping 1–3 frames to add motion diversity.
  • Comparison model: The method is compared with SAVP, a stochastic video-prediction model based on a conditional GAN–VAE hybrid.The baseline cGAN uses SAVP’s GAN generator and discriminator components for comparison.
  • Evaluation metrics: Video diversity is measured by pairwise frame-wise MSE distance, while Distmin and Simmax measure closest-sample quality against ground truth.Distmin uses MSE distance; Simmax uses VGGNet-feature cosine similarity.

D.3.4 MORE EXAMPLES

Across BAIR and KTH video prediction examples, the method produces diverse future predictions while often improving clarity and capturing meaningful dynamics that baselines miss.

  • When input motion is ambiguous, the model explores meaningfully different futures, such as hand-waving instead of hand-clapping.
  • In BAIR robot-arm examples, baseline cGAN and SAVP often produce noisy backgrounds, whereas our method generates clearer outputs.
  • Our method sometimes predicts interactions between foreground and background objects by modeling more meaningful video dynamics from latent code z.
  • Our method generates diverse future predictions on both BAIR and KTH, while SAVP is less diverse, especially on KTH.
Loading 1901.09024v1…