PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved Consistency Regularization for GANs

Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, Han Zhang

arXiv:2002.04724v2stat.MLcs.LG

TL;DR

GAN consistency regularization can improve synthesis but may introduce augmentation artifacts when applied only to real images. The paper proposes ICR, combining balanced regularization of real and generated images with latent-space regularization of the discriminator and generator. ICR achieves best known or state-of-the-art FID results across CIFAR-10, CelebA, and ImageNet-2012 settings.

  • Problem

    Consistency regularization can introduce augmentation artifacts into generated samples when augmentations are applied only to real images, while existing methods do not directly regularize the generator.

  • Method

    ICR combines balanced consistency regularization on real and generated images with latent consistency regularization that makes the discriminator insensitive and the generator sensitive to small latent perturbations.

  • Results

    ICR yields best known FID scores for unconditional CIFAR-10 and CelebA, improves conditional CIFAR-10 from 11.48 to 9.21, and improves ImageNet-2012 BigGAN from 6.66 to 5.38.

  • Takeaways & Limitations

    The techniques fix augmentation artifacts and achieve the best known FID numbers for unconditional and conditional CIFAR-10 and fixed-size conditional ImageNet synthesis.

  • Takeaways & Limitations

    FID values using the PyTorch implementation are not comparable to values from the official TensorFlow implementation because the implementations output substantially different numbers.

Abstract

from arXiv · show

Recent work has increased the performance of Generative Adversarial Networks (GANs) by enforcing a consistency cost on the discriminator. We improve on this technique in several ways. We first show that consistency regularization can introduce artifacts into the GAN samples and explain how to fix this issue. We then propose several modifications to the consistency regularization procedure designed to improve its performance. We carry out extensive experiments quantifying the benefit of our improvements. For unconditional image synthesis on CIFAR-10 and CelebA, our modifications yield the best known FID scores on various GAN architectures. For conditional image synthesis on CIFAR-10, we improve the state-of-the-art FID score from 11.48 to 9.21. Finally, on ImageNet-2012, we apply our technique to the original BigGAN model and improve the FID from 6.66 to 5.38, which is the best score at that model size.

1 Introduction

Consistency regularization can improve GAN performance but may create augmentation artifacts when applied only to real images. The paper introduces ICR, combining balanced image-space regularization with latent-space and generator regularization, and reports strong FID results across datasets and settings.

  • Motivation: Consistency regularization improves GAN performance but applying augmentations only to real images can cause generated samples to inherit undesirable augmentation artifacts.The imbalance also leaves the generator indirectly constrained rather than directly regularized.
  • Contribution: ICR applies consistency regularization to generated images, latent vectors, and the generator by combining balanced consistency regularization with latent consistency regularization.The combined method is designed to address both augmentation imbalance and the lack of direct generator regularization.
  • Results: ICR achieves the best known FID scores for unconditional image synthesis on CIFAR-10 and CelebA across various GAN architectures.
  • Results: 11.48 to 9.21: ICR improves the state-of-the-art FID score for conditional image synthesis on CIFAR-10.
  • Results: 6.66 to 5.38: ICR improves the FID of the original BigGAN model on ImageNet-2012, achieving the best score at that model size.

2 Improved Consistency Regularization

The paper develops two extensions of GAN consistency regularization: bCR balances image-space regularization across real and generated images, while zCR regularizes responses to latent perturbations and encourages generator diversity. Their combination, ICR, regularizes both discriminator and generator behavior across image and latent spaces.

  • Background: Consistency regularization trains models to produce consistent predictions for instances and their semantics-preserving augmentations.
  • CR-GAN: CR-GAN regularizes discriminator sensitivity between an image x and its augmented version T(x), but applying this only to real images can make augmentations appear to be real-data features.
  • Balanced Consistency Regularization (bCR): Cutout experiments show that vanilla CR-GAN can generate visible black-rectangle artifacts, while bCR fixes this issue by augmenting generated fake images as well.
  • Balanced Consistency Regularization (bCR): bCR augments both real images and generated samples, adding discriminator consistency terms for both corresponding image pairs.The real and fake regularization strengths can be adjusted with λreal and λfake.
  • Latent Consistency Regularization (zCR): zCR perturbs latent vectors with small noise, encourages discriminator insensitivity to the resulting outputs, and encourages the generator to produce diverse corresponding images.The generator diversity term helps avoid collapse to specific samples for every latent vector.
  • Putting It All Together (ICR): ICR combines bCR and zCR, regularizing discriminator consistency across real-image, generated-image, and latent-perturbation pairs while encouraging generator diversity.The combined method applies augmentations in both image and latent spaces and adds terms to both discriminator and generator objectives.

3 Experiments

The experiments evaluate bCR, zCR, and their combination ICR across datasets, architectures, loss functions, and conditional settings. ICR achieves the best reported results across the studied unconditional settings and improves conditional FID scores on CIFAR-10 and ImageNet.

  • Experimental setup: The study compares bCR and zCR with multiple GAN regularization baselines across datasets, architectures, and loss functions.The baselines include Gradient Penalty, DRAGAN, Jensen-Shannon Regularizer, and vanilla Consistency Regularization.
  • Experimental setup: FID is the primary evaluation metric, computed against real test images using 10K CIFAR-10, 3K CelebA, and 50K ImageNet images.Inception Scores are also reported in the appendix.
  • Unconditional GAN models: On DCGAN CIFAR-10 with hinge loss, bCR improves vanilla CR by more than 3.0 FID, while zCR improves scores less but remains above measurement variance.The reported bCR settings use λreal = λfake = 10; zCR uses σnoise = 0.03, λgen = 0.5, and λdis = 5.
  • Unconditional GAN models: On ResNet-style CIFAR-10, both proposed methods outperform all baselines, but zCR performs better than bCR.This reverses the ordering observed for DCGAN CIFAR-10.
  • ICR and conditional synthesis: Combining bCR and zCR into ICR produces the best results in all three studied unconditional synthesis settings and the best reported unconditional CIFAR-10 ResNet results.For conditional synthesis, ICR reaches FID 9.21 on CIFAR-10 and 5.38 on ImageNet at the original BigGAN model size.
  • Unconditional GAN models: On CelebA DCGAN, bCR improves vanilla CR by more than 2.0 FID and zCR improves it by around 1.0 FID.The improvements are smaller than on CIFAR-10 but remain substantial.

4 Ablation Studies

The ablations examine artifact removal through balanced consistency regularization and performance sensitivity to bCR and zCR hyper-parameters. They find that stronger fake-image regularization reduces artifacts, while moderate latent generator regularization performs best in the tested setup.

  • Artifact analysis: bCR fixes augmentation artifacts by applying consistency regularization to both real and generated images.Vanilla CR-GAN produces cutout artifacts, whereas bCR largely removes them when λfake ≥ λreal.
  • Artifact analysis: Higher λfake increasingly alleviates generation artifacts, although a few remain when 0 < λfake ≪ λreal.The artifact analysis examines 600 generated images from three random runs with λreal = 10.
  • Hyper-parameter analysis: For bCR, the λreal and λfake coefficients weight consistency costs on real images and generated samples, respectively.Their interplay is evaluated with a spectrally normalized DCGAN-type model on CIFAR-10 using hinge loss.
  • Hyper-parameter analysis: λgen = 0.5 works best in the tested zCR setup, as measured by FID.zCR also studies σnoise and λdis, which control latent perturbation magnitude and discriminator sensitivity.

5 Related Work

The related work spans GAN scaling, consistency regularization, robustness, and augmentation-based training. ICR differs by regularizing the compound discriminator-generator mapping and by acting in both image and latent spaces.

  • GAN scaling: Prior work scales conditional GANs to ImageNet-2012, with successive models defining the task’s recent state of the art.The cited models differ in reported performance and, in some cases, model size.
  • Robustness and latent regularization: zCR regularizes D(G(z)) against D(G(z + Δz)), whereas adversarial-robustness work perturbs generated images and regularizes D(G(z)) against D(G(z) + v).zCR also regularizes the generator rather than only the discriminator.
  • Consistency regularization: Consistency regularization originated largely in semi-supervised learning and was later applied to GANs with initial performance gains.These methods regularize predictions under perturbations, motivating consistency-based GAN training.
  • Augmentation-based training: Concurrent work studies GAN training with limited data and systematically investigates augmentation effectiveness.These efforts indicate an emerging direction centered on augmentations in GAN training.

6 Conclusion

The paper introduces bCR and zCR as complementary improvements to GAN consistency regularization. Together, they remove observed augmentation artifacts and achieve leading FID results across CIFAR-10 and ImageNet settings.

  • Contributions: bCR augments generator samples alongside training data, while zCR perturbs prior draws and discourages discriminator but encourages generator sensitivity.Their combination is called Improved Consistency Regularization, or ICR.
  • Results: ICR fixes augmentation artifacts and achieves the best known FID numbers for unconditional and conditional CIFAR-10 synthesis.It also achieves the best FID numbers for conditional ImageNet synthesis at the original BigGAN parameter count.
  • Practical implications: The techniques are simple to implement, not particularly computationally burdensome, and relatively insensitive to hyper-parameters.The authors propose them as additions to the GAN training toolkit.

A GAN Losses

GAN training uses a generator that maps latent samples to target-like images and a discriminator that distinguishes real from generated samples. The appendix summarizes non-saturating, hinge, and Wasserstein formulations alongside spectral normalization as a stabilization technique.

  • GAN formulation: A GAN generator maps z ∼ p(z) to samples G(z), while the discriminator distinguishes p_real(x) from generated samples.The discriminator’s distinction encourages the generator to reduce the discrepancy between target and generated distributions.
  • Non-saturating GAN: The non-saturating GAN uses the generator loss LG = −E_z∼p_z[log D(G(z))].The paper identifies this original formulation as the non-saturating GAN.
  • Hinge loss: Hinge loss is presented as an alternative discriminator objective used to improve training stability and generation quality.The appendix places hinge loss within broader redesigns of the discriminator loss.
  • Wasserstein GAN: WGAN measures a 1-Lipschitz-constrained Wasserstein distance between target and generated distributions in discriminator output space.Its objectives are introduced as another successful GAN reformulation.
  • Normalization: Spectral normalization is described as a widely used discriminator weight-normalization method for stabilizing GAN training.The paper notes its adoption across recent GAN models.

B Evaluation in Inception Score

The section evaluates unconditional generated samples with Inception Score on CIFAR-10 and CelebA, reporting the best results for both SNDCGAN and ResNet architectures.

  • Inception Score evaluates the unconditional generated samples on CIFAR-10 and CelebA.
  • Improved Consistency Regularization achieves the best Inception Score results with both SNDCGAN and ResNet architectures.
  • The evaluation includes SNDCGAN on CIFAR-10 with hinge loss and ResNet on CIFAR-10 with non-saturating loss.

C Hyper-Parameters’ Effect on Performance

The section analyzes how hyper-parameters affect Balanced and Latent Consistency Regularization, with visualized results reported in the appendix. For Balanced Consistency Regularization, similar λfake and λreal magnitudes generally perform best and outperform vanilla consistency regularization.

  • The experiments analyze hyper-parameter effects for Balanced Consistency Regularization and Latent Consistency Regularization.The visualizations are presented in the appendix because of space limitations.
  • Balanced Consistency Regularization essentially always outperforms vanilla consistency regularization across the tested λfake and λreal values.The analysis trains DCGAN on CIFAR-10 with hinge loss.
  • Balanced Consistency Regularization generally performs best when λfake and λreal have similar magnitudes.

D Additional Quantitative Results

Additional experiments quantify the effect of Balanced Consistency Regularization under cutout and color-jittering augmentations. The results support its improvement over baseline consistency regularization across both augmentation settings.

  • FID evaluations quantify the cutout experiments corresponding to Figure 5.
  • Color-jittering experiments produce similar results when comparing consistency regularization with Balanced Consistency Regularization.The jittering augmentation adds bounded random noise to normalized image color channels.
  • The color-jittering results support the reported findings and improvements over baselines.

E Qualitative Examples

The section presents qualitative ImageNet samples from ICR-BigGAN alongside samples from baseline CR-BigGAN. The comparison uses models with FID scores of 5.38 and 6.66, respectively.

  • ICR-BigGAN samples on ImageNet are shown as qualitative examples for different class labels.
  • 5.38 is the reported FID for ICR-BigGAN, compared with 6.66 for baseline CR-BigGAN.The figures show ICR-BigGAN samples on the left and CR-BigGAN samples on the right.
  • The qualitative figures provide additional random ImageNet samples from ICR-BigGAN trained on ImageNet.
Loading 2002.04724v2…