PaperlayerView exact revision on arXiv

Source-linked AI summary

Consistency Regularization for Generative Adversarial Networks

Han Zhang, Zizhao Zhang, Augustus Odena, Honglak Lee

arXiv:1910.12027v2cs.LGcs.CVstat.ML

TL;DR

GAN training is unstable and sensitive to hyperparameters, while existing discriminator regularizers can be computationally expensive and interact poorly with spectral normalization. The paper proposes consistency regularization for GAN discriminators and finds it effective across settings, including state-of-the-art FID improvements for conditional generation.

  • Problem

    GAN training is unstable, highly sensitive to hyperparameters, and difficult to characterize theoretically, motivating discriminator-focused stabilization methods.

  • Method

    Consistency regularization augments discriminator inputs and penalizes sensitivity, encouraging consistent discriminator outputs for semantically preserved data augmentations.

  • Results

    Consistency regularization achieves the best FID across evaluated regularization settings and improves BigGAN⋆ FID from 20.42 to 11.48 on CIFAR-10 and from 7.75 to 6.66 on ImageNet.

  • Takeaways & Limitations

    The method is simple, computationally cheap, compatible with spectral normalization, and boosts existing GAN models across losses, architectures, optimizers, and datasets.

Abstract

from arXiv · show

Generative Adversarial Networks (GANs) are known to be difficult to train, despite considerable research effort. Several regularization techniques for stabilizing training have been proposed, but they introduce non-trivial computational overheads and interact poorly with existing techniques like spectral normalization. In this work, we propose a simple, effective training stabilizer based on the notion of consistency regularization---a popular technique in the semi-supervised learning literature. In particular, we augment data passing into the GAN discriminator and penalize the sensitivity of the discriminator to these augmentations. We conduct a series of experiments to demonstrate that consistency regularization works effectively with spectral normalization and various GAN architectures, loss functions and optimizer settings. Our method achieves the best FID scores for unconditional image generation compared to other regularization methods on CIFAR-10 and CelebA. Moreover, Our consistency regularized GAN (CR-GAN) improves state-of-the-art FID scores for conditional generation from 14.73 to 11.48 on CIFAR-10 and from 8.73 to 6.66 on ImageNet-2012.

1 INTRODUCTION

GAN training is unstable and sensitive to hyperparameters, motivating a discriminator regularizer based on consistency under semantic-preserving augmentations. The proposed method is computationally inexpensive and performs strongly across architectures and datasets, including state-of-the-art FID results.

  • 1 INTRODUCTION: GAN training is unstable and highly sensitive to hyperparameters because it involves finding a Nash equilibrium in a non-convex, high-dimensional game.The convergence properties of GAN training remain mostly an open problem.
  • 1 INTRODUCTION: Consistency regularization augments images before discriminator processing and penalizes discriminator sensitivity to those augmentations.The approach adapts a technique widely used in semi-supervised learning.
  • 1 INTRODUCTION: The method is simple, effective, and less computationally expensive than prior techniques, while improving performance when combined with spectral normalization.Its gains persist across a large range of GAN variants and datasets.
  • 1 INTRODUCTION: CR-GAN achieves the best FID scores for unconditional image generation on CIFAR-10 and CelebA.The experiments evaluate different GAN variants and demonstrate effective interaction with spectral normalization.
  • 1 INTRODUCTION: 14.73 to 11.48 on CIFAR-10 and 8.73 to 6.66 on ImageNet-2012 are the reported conditional-generation FID improvements from applying the proposed technique.The paper reports these improvements as boosting state-of-the-art GAN models.

2 METHOD

The method regularizes GAN discriminators by enforcing consistency under semantics-preserving data augmentations. The consistency cost is added to discriminator updates, while the generator update remains unchanged, and the design works with different adversarial losses.

  • GAN objectives: GANs use generator and discriminator networks trained alternately with adversarial objectives, including non-saturating, hinge-loss, and Wasserstein formulations.The generator maps latent variables into observation space, while the discriminator distinguishes generated from empirical data.
  • Consistency regularization: Consistency regularization penalizes discriminator sensitivity when input images undergo semantics-preserving augmentations such as flipping or cropping.The discriminator’s real-versus-fake decision should remain invariant under valid domain-specific transformations.
  • Consistency regularization: The regularizer compares discriminator outputs before activation across augmented and original inputs, with layer-specific weights and an L2 norm.Consistency can be enforced from layer m through layer n, where λ_j weights the contribution from layer j.
  • Optimization: The consistency cost is added to the discriminator loss with hyper-parameter λ, while the generator objective and update remain unchanged.The paper reports that enforcing consistency only on the discriminator’s last pre-activation layer is sufficient in experiments.
  • Optimization: CR-GAN is designed to work with any valid adversarial losses and adds limited overhead by processing one additional image through the discriminator during discriminator updates.The algorithm uses λ = 10 by default.

3 EXPERIMENTS

The experiments evaluate consistency regularization across datasets, GAN architectures, losses, optimizers, spectral normalization, and regularization strengths. Across these settings, it achieves strong FID results, improves state-of-the-art conditional GANs, and is faster than gradient-based regularizers.

  • Experimental setup: Experiments compare consistency regularization with Gradient Penalty, DRAGAN, and JS-Regularizer across CIFAR-10 and CelebA.The study varies loss functions, optimizer parameters, regularization coefficients, and neural architectures.
  • Impact of loss function: Consistency regularization improves the baseline across all evaluated loss functions and both datasets, while achieving the lowest FID in all reported Figure 2 cases.Gradient Penalty and DRAGAN show mixed results across settings.
  • Computational cost: 1.7 times faster than gradient-based regularization techniques, including DR, GP, and JSR.Those methods require computing the gradient of the gradient norm ∥∇x(D)∥.
  • Impact of the regularization coefficient: Consistency regularization has the best FID on both datasets and is more robust to changes in λ than other GAN regularization techniques.The coefficient is evaluated at λ values of 0.1, 1, 10, and 100.
  • Unconditional generation: 14.56 FID on CIFAR-10 and 16.97 FID on CelebA are achieved for unconditional image generation across the evaluated architectures and datasets.The CIFAR-10 result is lower than the 14.73 FID reported for class-conditional image synthesis by Brock et al. (2018).
  • Conditional generation: 11.48 FID on CIFAR-10 and 6.66 FID on ImageNet are obtained by adding consistency regularization to BigGAN⋆ under otherwise identical architecture and training settings.The corresponding BigGAN⋆ baselines are 20.42 and 7.75, respectively.

4 ABLATION STUDIES AND DISCUSSION

The ablations separate the effects of augmentation from consistency regularization. Augmentation reduces discriminator over-fitting, while consistency regularization provides additional FID gains and depends on semantically meaningful representations.

  • Ablation design: Consistency regularization combines input augmentation with an explicit consistency constraint between original and augmented data.The constraint penalizes sensitivity to the augmentation rather than merely assigning transformed real images the same label.
  • Discussion: Consistency regularization pulls semantically similar original and transformed images closer in the discriminator representation space.The proposed interpretation is that this encourages an implicit manifold structure rather than only reducing classification over-fitting.
  • How much augmentation matters: Augmentation alone reduces discriminator over-fitting, but consistency regularization alone does little to address over-fitting.Training and held-out test accuracies are low for the baseline and essentially indistinguishable for augmentation-only and consistency-regularized models.
  • How much augmentation matters: Consistency regularization achieves a better FID score than both the baseline and augmentation-only settings, despite similar discriminator accuracy to augmentation alone.The comparison suggests its FID benefit operates through a mechanism other than reducing discriminator over-fitting.
  • Augmentation choices: FID 16.04 is the best result among four tested augmentation types, obtained with random shifting and horizontal flipping without cutout.The alternatives included pixel-space Gaussian noise, cutout, and a combination of cutout with shifting and flipping.

5 CONCLUSION

The paper concludes that consistency regularization is a simple, computationally cheap GAN regularizer that remains effective across architectures, losses, optimizers, datasets, and spectral-normalization settings. Its broad empirical gains include robust performance across optimizer settings and improved state-of-the-art conditional-generation FIDs.

  • Conclusion: Consistency regularization improves GAN performance across the many evaluated contexts and is compatible with spectral normalization.The conclusion also reports effectiveness under different loss functions, neural architectures, and optimizer hyper-parameter settings.
  • Conclusion: The method is computationally cheap and more effective than other regularization methods across the evaluated settings.The paper reports a thorough study of its design choices and hyper-parameters.
  • Optimizer settings: Figure A1 compares FID scores for all methods across optimizer settings A–G.The comparison focuses on whether consistency regularization produces better samples under different optimizer configurations.
  • Optimizer settings: Consistency regularization remains fairly robust under seven optimizer settings, including aggressive learning-rate and momentum configurations.Settings C and D generally work best for SNDCGAN and ResNet, respectively.

B COMPARISON OF DIFFERENT REGULARIZATION METHODS WHEN SPECTRAL NORMALIZATION IS NOT USED

Without spectral normalization, consistency regularization still improves the baseline and usually achieves the best FID scores. Adding spectral normalization further boosts results, while consistency regularization does not depend on it.

  • Results without spectral normalization: Consistency regularization always improves the baseline model when spectral normalization is not used.The comparison is reported in Figure B1 using the W/O baseline condition.
  • Results without spectral normalization: Consistency regularization achieves the best FID scores in most cases without spectral normalization.The result demonstrates that the method does not depend on spectral normalization.
  • Interaction with spectral normalization: Adding spectral normalization further boosts the results compared with the corresponding settings without it.The paper makes this comparison with the results shown in Figures 2 and 4.

C TRAINING SPEED

Consistency regularization reduces the computational cost of discriminator updates relative to gradient-based regularization techniques. On SNDCGAN trained on CIFAR-10 with an NVIDIA Tesla V100, it is reported as approximately 1.7 times faster.

  • Training speed: 1.7 times faster discriminator updates are achieved by consistency regularization than by gradient-based regularization techniques.The measurement uses SNDCGAN on CIFAR-10 with an NVIDIA Tesla V100.

D GENERATED SAMPLES FOR UNCONDITIONAL IMAGE GENERATION

The figures compare unconditional samples on CelebA and CIFAR-10, including a difficult ResNet setting where consistency regularization is the only method producing satisfactory samples.

  • The section presents visual comparisons of generated samples for unconditional image generation on CelebA and CIFAR-10.
  • Consistency regularization is the only tested regularization method producing satisfactory samples in a difficult CIFAR-10 ResNet setting.The setting uses Wasserstein loss and spectral normalization; the baseline and previous regularization methods fail to generate reasonable samples.

E GENERATED SAMPLES FOR CONDITIONAL IMAGE GENERATION

The figures compare conditionally generated images across CIFAR-10 classes and between BigGAN* and CR-BigGAN* on ImageNet.

  • On CIFAR-10, each row in the conditional-generation comparison represents one class.
  • The ImageNet comparisons place CR-BigGAN* samples on the left and BigGAN* samples on the right.The same left-right arrangement is used in the additional ImageNet results.

F COMPARISON WITH INCEPTION SCORE

This section evaluates unconditional CIFAR-10 generation with Inception Score across architectures, loss settings, and the number of discriminator layers receiving consistency regularization.

  • Consistency regularization achieves the best Inception Score with both SNDCGAN and ResNet architectures on CIFAR-10.
  • Table F1 reports the best Inception Score for unconditional image generation on CIFAR-10.
  • The loss-setting comparisons report Inception Score for SNDCGAN and ResNet models trained on CIFAR-10.
  • Regularizing only the discriminator’s final layer achieves reasonably good results, while adding consistency to early intermediate layers harms performance.The authors therefore use final-layer consistency regularization in the remaining experiments.
  • The layer-comparison experiment evaluates two weighting schemes: inverse feature dimension and equal weight across layers.

H CONSISTENCY REGULARIZATION ON THE GENERATED SAMPLES

The section compares consistency regularization applied to real samples, generated samples, both, or neither in unconditional and conditional generation settings.

  • The experiment compares four settings: no consistency regularization, CR-Real, CR-Fake, and CR-All.CR-Real regularizes real samples, CR-Fake regularizes generated samples, and CR-All regularizes both.
  • CR-Fake has similar computational cost to CR-Real, whereas CR-All doubles the computational cost.
  • Figure H1 evaluates these settings for CIFAR-10 unconditional generation with SNDCGAN and ResNet, and CIFAR-10 conditional generation with CR-BigGAN*.
Loading 1910.12027v2…