PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect

Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, Liqiang Wang

arXiv:1803.01541v1cs.CVcs.LGstat.ML

TL;DR

GANs are difficult to train, motivating WGAN’s use of a 1-Lipschitz discriminator, but gradient penalties may not enforce continuity near the real-data manifold. The paper adds a consistency term based on perturbed real points and integrates it with semi-supervised learning, reporting stronger generative and semi-supervised results on CIFAR-10. The method is evaluated within the scope of sampled regions around the real manifold and the paper’s stated datasets and settings.

  • Problem

    GAN training can be unstable and suffer mode collapse, while gradient penalties examine only sampled points and may leave regions near the real-data manifold unchecked.

  • Method

    The paper adds a Lipschitz-based consistency term that compares discriminator responses to two dropout-induced virtual points near each real point and integrates it with GAN semi-supervised learning.

  • Results

    The approach reports state-of-the-art CIFAR-10 generative and semi-supervised results, including an inception score of 8.81 ± 0.13 and a semi-supervised result of 9.98 ± 0.21 using 4,000 labeled images.

  • Takeaways & Limitations

    The proposed regularization is reported to improve sample quality, reduce overfitting with limited data, and support competitive semi-supervised training.

  • Takeaways & Limitations

    The method limits its continuity checks to the real-data manifold and surrounding regions because checking all input pairs and regions is impractical.

Abstract

from arXiv · show

Despite being impactful on a variety of problems and applications, the generative adversarial nets (GANs) are remarkably difficult to train. This issue is formally analyzed by \cite{arjovsky2017towards}, who also propose an alternative direction to avoid the caveats in the minmax two-player training of GANs. The corresponding algorithm, called Wasserstein GAN (WGAN), hinges on the 1-Lipschitz continuity of the discriminator. In this paper, we propose a novel approach to enforcing the Lipschitz continuity in the training procedure of WGANs. Our approach seamlessly connects WGAN with one of the recent semi-supervised learning methods. As a result, it gives rise to not only better photo-realistic samples than the previous methods but also state-of-the-art semi-supervised learning results. In particular, our approach gives rise to the inception score of more than 5.0 with only 1,000 CIFAR-10 images and is the first that exceeds the accuracy of 90% on the CIFAR-10 dataset using only 4,000 labeled images, to the best of our knowledge.

1 INTRODUCTION

GANs are powerful but difficult to train because their dynamics can be unstable and may collapse modes. WGAN addresses these issues through a 1-Lipschitz discriminator, while this paper adds consistency constraints around real data to improve WGAN training and data efficiency.

  • GAN training is difficult without heuristics because dynamics can be unstable and generated samples may collapse to limited modes.
  • WGAN replaces the original training direction with Wasserstein-distance minimization and uses a discriminator constrained to be 1-Lipschitz.Weight clipping can limit network capacity and cause gradient vanishing problems.
  • Gradient penalty regularizes sampled points between real and generated data, but finite training leaves other support regions unexamined.Early in training, these sampled points may remain distant from the real-data manifold.
  • The proposed consistency term additionally enforces Lipschitz continuity around the real-data manifold by bounding discriminator-response differences for two perturbed points.The method perturbs each real point twice to obtain x′ and x′′ and applies the Lipschitz bound to their responses.
  • The approach improves gradient-penalty training, reduces overfitting on small datasets, and integrates with GANs for semi-supervised learning.The paper reports no obvious overfitting when training on only 1,000 CIFAR-10 images.
  • 8.81 ± 0.13 is the reported CIFAR-10 inception score, while semi-supervised learning reaches 9.98 ± 0.21 using 4,000 labeled images.

2 APPROACH

The approach improves WGAN training by adding a consistency regularization based on perturbed real data and discriminator dropout. It also integrates this regularization into a semi-supervised learning framework.

  • Improving the improved training of WGAN: Gaussian input noise produced blurry generated samples, while input dropout caused cut-off MNIST samples, motivating hidden-layer dropout instead.With a small dropout rate, the perturbed discriminator output is treated as the clean discriminator’s response to a nearby virtual data point.
  • Improving the improved training of WGAN: The method adds a consistency term that bounds discriminator-response differences between two dropout-perturbed versions of each real data point.The perturbations are applied to hidden discriminator layers, producing virtual points near the observed real data.
  • Improving the improved training of WGAN: The final consistency term includes discriminator outputs and a second-to-last-layer distance, which slightly improves performance by controlling that intermediate representation.The regularization is written with margin parameter M′, whose reported best values range from 0 to 0.2; results use M′ = 0.
  • Improving the improved training of WGAN: The consistency regularization checks Lipschitz continuity over the real-data manifold and surrounding regions, complementing the gradient penalty used in improved WGAN training.The method focuses on pairs of nearby perturbed points rather than checking one data point at a time.
  • A seamless connection with a semi-supervised learning method: For semi-supervised learning, the discriminator uses K + 1 output neurons and adds the consistency regularization to the objective.The additional neuron contrasts generated samples with real data using the Wasserstein distance, while dropout-based consistency supports temporal self-ensembling.

3 EXPERIMENTAL RESULTS

Experiments on MNIST and CIFAR-10 compare CT-GAN with GP-WGAN and other methods across sample quality, convergence, overfitting, inception score, and semi-supervised learning. CT-GAN produces more realistic samples, is less prone to overfitting, and achieves strong reported CIFAR-10 results.

  • MNIST: MNIST experiments use 1,000 images for WGAN training and 100 revealed labels among 60,000 training examples for semi-supervised learning.No data augmentation is used in the semi-supervised MNIST setup.
  • MNIST: CT-GAN generates more realistic MNIST samples than GP-WGAN after 50,000 generator iterations, with generally sharper foreground-background contrast.The comparison is shown in Figure 3.
  • MNIST: CT-GAN discriminator costs consistently decrease on MNIST training and test sets, whereas GP-WGAN test costs saturate relatively early.The authors report the same overfitting observation for CIFAR-10 in Appendix E.
  • MNIST: CT-GAN is among the best semi-supervised learning methods on MNIST.The comparison is reported in Table 1.
  • CIFAR-10: CIFAR-10 tests a small CNN with 1,000 images and a ResNet with the whole training set; CT-GAN produces more photo-realistic samples for both comparisons.Figures 5 and 7 provide the qualitative comparisons.
  • CIFAR-10: The CT-GAN discriminator weights occupy the smaller, more symmetric range [−0.67, 0.96], compared with GP-WGAN’s [−2.00, 10.12].The authors say this partially explains CT-GAN’s lower susceptibility to overfitting.
  • CIFAR-10: 5.13 ± 0.12 is CT-GAN’s inception score with a small CNN, compared with 2.98 ± 0.11 for GP-WGAN.The models are trained using the small CNN-based generator.
  • CIFAR-10: CT-GAN achieves state-of-the-art CIFAR-10 semi-supervised learning results using only 4,000 labels and outperforms GAN-based methods by a large margin.Results are averaged over five experimental rounds with standard errors.

4 CONCLUSION

The paper introduces a consistency term derived from Lipschitz inequality to improve GAN performance and ease over-fitting when data are limited. Experiments report state-of-the-art CIFAR-10 results for semi-supervised learning and generative modeling.

  • The proposed consistency term is derived from Lipschitz inequality and is intended to boost GAN performance.
  • The consistency term is demonstrated to ease over-fitting when the amount of data is limited.
  • Experiments report state-of-the-art CIFAR-10 accuracy and Inception score for both semi-supervised learning and generative-model tasks.

APPENDIX A NETWORK ARCHITECTURES

The appendix identifies the network architectures used for semi-supervised learning on MNIST and CIFAR-10. It points to dedicated architecture tables for both datasets.

  • The classification-purpose CT-GAN architecture is detailed for MNIST and CIFAR-10 in Tables 4 and 5.The classifiers follow widely used semi-supervised network designs, with weight normalization rather than batch normalization.
  • Table 4 presents the network architectures used for semi-supervised learning on MNIST.
  • Table 5 presents the network architectures used for semi-supervised learning on CIFAR-10.

APPENDIX B HYPER-PARAMETERS AND OTHER TRAINING DETAILS

The appendix reports training settings for semi-supervised and generative-model experiments, including dataset-specific epochs, learning rates, and architecture controls. Generative-model comparisons largely retain the improved WGAN setup while adding dropout layers.

  • For semi-supervised learning, λ = 1.0 is used in Eq.(7), with 1,000 CIFAR-10 epochs at learning rate 0.0003.MNIST uses 300 epochs at learning rate 0.003; other hyper-parameters match the improved GAN.
  • The MNIST and CIFAR-10 generative-model architectures are documented in Tables 6 and 7.
  • Generative-model experiments keep the improved WGAN network structure and hyper-parameters, adding three dropout layers to some hidden layers.

APPENDIX C ABLATION STUDY OF OUR APPROACH TO SSL

The SSL ablation study examines the proposed consistency term, GAN component, and second-to-last-layer regularization. Removing these components worsens test error or slightly reduces performance, while the consistency term connects GANs with temporal ensembling.

  • The consistency term connects GANs with temporal ensembling and contributes to the method’s superior SSL results.The paper reports that this dual effect is verified by the ablation study.
  • 14.98 test error results when the consistency term is removed, indicating the effectiveness of CT regularization.
  • Removing GAN reduces the approach nearly to temporal ensembling, but the resulting error remains significantly larger than for the overall method.
  • Removing regularization on the second-to-last layer produces a small performance drop.

APPENDIX D EXAMINING THE 1-LIPSCHITZ CONTINUITY

The experiments compare CT-GAN and GP-WGAN on gradient norms and Lipschitz-continuity measures during training with 1,000 CIFAR-10 images. CT-GAN maintains the 1-Lipschitz condition more effectively across training.

  • Gradient norm: CT-GAN achieves lower discriminator-input gradient norms on CIFAR-10 than GP-WGAN when trained with 1,000 images.The comparison is reported using the maximum ℓ2 norm of discriminator gradients on the CIFAR-10 testing set.
  • Definition of the Lipschitz continuity: Figures 9 and 10 track CT over training iterations, with Figure 10 showing the CT actually used to train the generative model.Both experiments use 1,000 CIFAR-10 images.
  • Definition of the Lipschitz continuity: CT-GAN’s Lipschitz-continuity curve converges below a certain value much faster than GP-WGAN.The curves are computed by pairing randomly selected real data points during training.

APPENDIX E GP-WGAN WITH DROPOUT

Adding dropout to GP-WGAN reduces overfitting, but the CT-GAN regularization is more effective and produces a higher inception score on 1,000-image CIFAR-10 training.

  • Inception score: 5.13 ± 0.12 is CT-GAN’s inception score, compared with 4.29 ± 0.12 for GP-WGAN+dropout and 2.98 ± 0.11 for GP-WGAN.The experiment uses 1,000 CIFAR-10 images; GP-WGAN+dropout removes the CT term while retaining the dropout layers.
  • Generated samples: GP-WGAN+dropout generates the samples shown in Figure 11(a).The figure accompanies the reported inception-score comparison.
  • Convergence: Dropout reduces GP-WGAN overfitting but is less effective than CT-GAN according to the discriminator-cost convergence curves.The comparison includes GP-WGAN, GP-WGAN+Dropout, and CT-GAN.

APPENDIX F EXPERIMENTS ON LARGE DATASET

The paper evaluates CT-GAN on ImageNet and LSUN bedroom data using the GP-WGAN experimental setup. On 64×64 ImageNet, CT-GAN attains a higher inception score than GP-WGAN after 200,000 generator iterations.

  • ImageNet: 10.27 ± 0.15 is CT-GAN’s ImageNet inception score after 200,000 generator iterations, versus 9.85 ± 0.17 for GP-WGAN.The ImageNet experiment uses 64×64 images and the same setup as the GP-WGAN work.
  • ImageNet: CT-GAN’s ImageNet inception score becomes higher than GP-WGAN’s after early generator iterations.Figure 13 plots the inception-score comparison across generator iterations.
  • LSUN bedroom: CT-GAN generates LSUN bedroom image samples after 20k training iterations.These samples are presented in Figure 15.
Loading 1803.01541v1…