PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved Training of Wasserstein GANs

Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, Aaron Courville

arXiv:1704.00028v3cs.LGstat.ML

TL;DR

GAN training remains difficult to stabilize, and WGAN weight clipping can cause optimization problems and pathological critic behavior. This paper introduces WGAN-GP, which replaces clipping with a gradient penalty and demonstrates more stable training, improved performance, and high-quality generation across varied architectures and datasets.

  • Problem

    Consistently stable training of GANs remains an open problem despite extensive efforts to stabilize optimization.

  • Method

    The paper introduces WGAN-GP, which replaces WGAN’s weight clipping with a penalty on the critic’s input-gradient norm.

  • Results

    The method enables stable training across varied GAN architectures, improves performance over weight clipping, and produces high-quality generations.

  • Takeaways & Limitations

    WGAN-GP provides a more stable alternative to weight clipping for training GANs across diverse architectures and modeling tasks.

  • Takeaways & Limitations

    The paper does not claim that every weight-clipping pathology always occurs in practice or that the identified mechanisms are exhaustive.

Abstract

from arXiv · show

Generative Adversarial Networks (GANs) are powerful generative models, but suffer from training instability. The recently proposed Wasserstein GAN (WGAN) makes progress toward stable training of GANs, but sometimes can still generate only low-quality samples or fail to converge. We find that these problems are often due to the use of weight clipping in WGAN to enforce a Lipschitz constraint on the critic, which can lead to undesired behavior. We propose an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input. Our proposed method performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning, including 101-layer ResNets and language models over discrete data. We also achieve high quality generations on CIFAR-10 and LSUN bedrooms.

1 Introduction

GANs are powerful but difficult to train, while WGAN improves theoretical properties yet relies on potentially problematic critic weight clipping. The paper proposes gradient penalty and demonstrates more stable, improved training across architectures, high-quality image generation, and a character-level language model without discrete sampling.

  • GANs generate visually appealing samples but are often hard to train, motivating extensive work on stabilizing their training.
  • WGAN uses the Wasserstein distance for better theoretical properties but enforces its critic’s 1-Lipschitz constraint through weight clipping.
  • Toy-dataset experiments show that critic weight clipping can produce undesired behavior.
  • The paper proposes gradient penalty, called WGAN-GP, as an alternative that avoids the same problems as weight clipping.
  • Experiments demonstrate stable training across varied GAN architectures, improvements over weight clipping, high-quality image generation, and a character-level GAN language model without discrete sampling.

2 Background

The background contrasts standard GAN training, whose divergences and saturating discriminator can cause optimization difficulty, with WGAN’s Wasserstein objective and 1-Lipschitz critic. It also motivates replacing weight clipping by gradient-based control through properties of the optimal critic.

  • GAN training: GANs train a generator to fool a discriminator that distinguishes generated samples from true data samples.The generator maps noise into the input space, while the discriminator receives either generated or real data.
  • GAN training: When the discriminator reaches optimality, the standard GAN objective minimizes Jensen-Shannon divergence but can produce vanishing gradients as the discriminator saturates.The commonly used non-saturating generator loss partly circumvents this difficulty but can still misbehave.
  • Wasserstein GAN: WGAN replaces typical GAN divergences with the Earth-Mover distance, which is continuous everywhere under mild assumptions.The Earth-Mover distance is informally the minimum mass-times-distance cost required to transform one distribution into another.
  • Wasserstein GAN: WGAN minimizes the Wasserstein distance using a critic restricted to 1-Lipschitz functions, whose input gradients are better behaved and whose value empirically correlates with sample quality.The critic evaluates distributions rather than classifying samples.
  • Weight clipping: WGAN enforces the critic’s Lipschitz constraint by clipping weights to [−c, c], restricting the critic to a subset of k-Lipschitz functions determined by c and its architecture.The paper identifies issues with this approach and proposes an alternative in later sections.
  • Optimal critic: An optimal WGAN critic can be chosen 1-Lipschitz, and its gradient norm is 1 almost everywhere under the real and model distributions.These properties are used to motivate the paper’s analysis of why weight clipping is problematic.

3 Difficulties with weight constraints

Weight constraints in WGANs create optimization difficulties and can bias critics toward pathological, overly simple value functions. These issues persist across several constraint variants and can remain even with batch normalization, especially in very deep critics.

  • Optimization difficulties: Weight clipping causes optimization difficulties and can produce a pathological critic value surface even when optimization succeeds.The authors caution that these mechanisms do not necessarily occur in every practice setting or exhaust all possible mechanisms.
  • Optimization difficulties: L2 norm clipping, weight normalization, and L1 or L2 weight decay exhibit problems similar to hard weight clipping.These alternatives were tested as additional weight constraints or soft constraints.
  • Optimization difficulties: Even with batch normalization, very deep WGAN critics often fail to converge.Batch normalization mitigates these problems to some extent, but does not resolve convergence failures in very deep critics.
  • Pathological critic surfaces: Weight clipping biases k-Lipschitz critics toward extremely simple functions instead of functions attaining the desired gradient norm.The observed behavior concerns architectures attempting to attain their maximum gradient norm k.
  • Pathological critic surfaces: Clipped critics ignore higher moments of the data distribution and learn simple approximations to optimal functions, unlike the proposed approach.This finding is reported across several toy distributions with the generator distribution fixed to real data plus unit-variance Gaussian noise.
  • Optimization difficulties: Without careful tuning of clipping threshold c, interactions between the weight constraint and cost function produce vanishing or exploding gradients.On a Swiss Roll dataset with 12-layer ReLU MLPs and c in [10^-1, 10^-2, 10^-3], gradients grew or decayed exponentially across layers.

4 Gradient penalty

Section 4 replaces WGAN weight clipping with a soft penalty on the critic’s input-gradient norm, evaluated on randomly sampled inputs. The method samples uniformly along lines between real and generated points, uses λ = 10, avoids critic batch normalization, and favors a two-sided penalty targeting gradient norm 1.

  • Gradient penalty: The method enforces the Lipschitz constraint by penalizing deviations of the critic’s input-gradient norm from 1 on random samples.A differentiable function is 1-Lipschitz exactly when its gradients have norm at most 1; the paper uses a soft constraint for tractability.
  • Gradient penalty: The sampling distribution is uniform along straight lines joining pairs of real-data and generated samples.This is motivated by the optimal critic having gradient-norm-1 lines connecting coupled points from the real and generator distributions.
  • Gradient penalty: λ = 10 works well across architectures and datasets ranging from toy tasks to large ImageNet CNNs.The paper uses this penalty coefficient in all experiments.
  • Gradient penalty: The critic omits batch normalization because batch normalization makes its output depend on an entire batch, invalidating the per-input gradient penalty objective.The penalized objective assumes the critic maps a single input to a single output.
  • Gradient penalty: The method uses a two-sided penalty that drives gradient norms toward 1 rather than a one-sided penalty that only keeps them below 1.The authors report slightly better early empirical performance but do not investigate the difference fully.

5 Experiments

Experiments show that WGAN-GP trains a broad range of GAN architectures more reliably than standard GAN objectives and weight clipping, often with shared default hyperparameters. It also improves convergence and sample quality on image datasets and enables meaningful character-level language modeling where the standard GAN objective failed.

  • Architecture robustness: WGAN-GP successfully trains many sampled 32×32 ImageNet architectures that the standard GAN objective cannot, across most inception-score thresholds.The study evaluates 200 random architectures, defining success as inception score > min score.
  • Architecture robustness: WGAN-GP is the only method that trains every tested LSUN-bedroom architecture using a shared set of hyperparameters.Other methods—including weight-clipped WGAN, DCGAN, and LSGAN—show instability or mode collapse on some architectures.
  • CIFAR-10 training: With the same RMSProp optimizer and learning rate, gradient penalty converges faster and reaches a better CIFAR-10 score than weight clipping; Adam improves it further.Compared with DCGAN, WGAN-GP converges more slowly in wall-clock time but has a more stable score at convergence.
  • CIFAR-10 quality: WGAN-GP achieves state-of-the-art unsupervised CIFAR-10 Inception score, while its conditional model outperforms every published model except SGAN.The same architecture is used for both the unsupervised and label-conditioned results.
  • Discrete language modeling: A continuous-generator WGAN-GP character model learns substantial language statistics on Google Billion Word despite frequent spelling errors, whereas the standard GAN objective yields no comparable results.The generator deterministically outputs 32 one-hot character vectors, decoded by taking each output vector’s argmax.

6 Conclusion

The paper identifies weight clipping as problematic in WGAN and introduces a critic-loss penalty that avoids those issues, yielding stable, strong performance across diverse architectures.

  • 6 Conclusion: The proposed critic-loss penalty avoids the problems observed with weight clipping in WGAN.The authors present it as an alternative constraint mechanism.
  • 6 Conclusion: The method demonstrates strong modeling performance and stability across a variety of architectures.The authors suggest this stability could enable stronger modeling on large-scale image datasets and language.

A Proof of Proposition 1

The proof uses compactness to obtain an optimal f ∗ and analyzes pairs in an optimal coupling satisfying f ∗(y)−f ∗(x)=∥y−x∥. Along the segment from x to y, equality forces linear potential growth and, where differentiable, the gradient to point from xt toward y.

  • Existence and coupling: Compactness of X guarantees the existence of an optimal f ∗, and Theorem 5.10 characterizes optimal couplings π.The proof invokes parts (iii) and (ii) of Theorem 5.10 of.
  • Segment analysis: For π-almost-every pair (x,y) with f ∗(y)−f ∗(x)=∥y−x∥, define ψ(t)=f ∗(xt)−f ∗(x) along the segment.The proof safely assumes x≠y because this holds under π with probability 1.
  • Segment analysis: The Lipschitz inequalities must be equalities, yielding ψ(t)=t∥x−y∥ and f ∗(xt)=f ∗(x)+t∥x−y∥.Equality at the endpoints forces equality throughout the bounding inequalities.
  • Gradient characterization: At differentiable xt, the 1-Lipschitz bound and equality in the Pythagorean inequality imply ∇f ∗(xt)=v.Here v is a unit vector aligned with the segment direction.
  • Gradient characterization: Thus, with probability 1 under π, ∇f ∗(xt)=y−xt ∥y−xt∥ for pairs attaining equality in the optimal potential difference.This is the proof’s concluding gradient-direction statement.

B More details for training random architectures within a set

This section details training random architectures on 32 × 32 ImageNet for 100K generator iterations, using separate Adam settings for standard GAN and WGAN-GP objectives. It also reports outcomes across 200 random architectures and compares them with a standard DCGAN score of 7.24.

  • Training setup: Models were trained on 32 × 32 ImageNet for 100K generator iterations using Adam.The standard GAN used α = 0.0002, β1 = 0.5, β2 = 0.999; WGAN-GP used α = 0.0001, β1 = 0, β2 = 0.9.
  • Training setup: The standard GAN used α = 0.0002, β1 = 0.5, β2 = 0.999, while WGAN-GP used α = 0.0001, β1 = 0, β2 = 0.9.These were the recommended Adam settings for the respective objectives.
  • Normalization and regularization: When discriminator batch normalization or layer normalization was used, λ = 10−3 weight decay slightly helped both algorithms.The decay was applied as a small discriminator weight decay.
  • Architecture outcomes: Table 5 reports outcomes for 200 random architectures at different success thresholds, compared with a standard DCGAN score of 7.24.The supplied passage identifies the experiment scope and comparison baseline but does not provide the threshold-specific outcomes.

C Experiments with one-sided penalty … F CIFAR-10 ResNet architecture

Across the experiments, the two-sided gradient penalty generally outperformed the one-sided version, although the comparison was not statistically significant. The paper also notes a theoretical issue with nonsmooth activations and specifies robustness-experiment hyperparameters and a CIFAR-10 residual architecture.

  • C Experiments with one-sided penalty: The authors observed that the two-sided penalty seemed to perform slightly better than penalizing only gradients larger than 1.The one-sided penalty would penalize gradients larger than 1 but not gradients smaller than 1.
  • C Experiments with one-sided penalty: The two-sided penalty achieved a higher Inception score in 100 of 174 architecture trials, versus 77 trials for the one-sided penalty.The result was not statistically significant at p < 0.05 and used only one metric and architecture distribution.
  • D Nonsmooth activation functions: With ReLU and other common nonsmooth activations, the gradient penalty objective can be undefined at some points and discontinuous with respect to network parameters.The objective gradient contains terms involving second derivatives of activation functions, and gradient descent is not guaranteed to succeed in this setting.
  • E Hyperparameters used for LSUN robustness experiments: For each method, the experiments used hyperparameters recommended in that method’s paper, with an additional LSGAN learning-rate search.The search was added because the LSGAN paper did not specify a learning rate.
  • E Hyperparameters used for LSUN robustness experiments: WGAN with gradient penalty used Adam with α = .0001, β1 = .5, and β2 = .9.These were the listed hyperparameters for the LSUN robustness experiments.
  • F CIFAR-10 ResNet architecture: The CIFAR-10 generator and critic are residual networks using pre-activation blocks with two 3 × 3 convolutions and ReLU nonlinearities.The critic uses mean pooling for some downsampling blocks, the generator uses nearest-neighbor upsampling, and batch normalization appears only in the generator.
  • F CIFAR-10 ResNet architecture: The CIFAR-10 model was optimized with Adam at learning rate 2 × 10−4, decayed linearly to 0 over 100K generator iterations.The architecture includes a generator progression from 128 × 4 × 4 to 128 × 32 × 32 before a 3 × 3 tanh convolution, while the critic downsamples to a scalar output.

G CIFAR-10 ResNet samples

Figure 6 presents unconditional CIFAR-10 samples generated by the proposed unsupervised model alongside conditional samples produced by adding AC-GAN conditioning.

  • CIFAR-10 ResNet samples: Figure 6 shows unconditional and conditional CIFAR-10 samples, with conditional samples organized by class in columns.The conditional samples result from adding AC-GAN conditioning to the unconditional model.

H More LSUN samples

This section lists LSUN experiments comparing DCGAN, LSGAN, clipped WGAN, and WGAN-GP across multiple generator–discriminator architectures and nonlinearities. The configurations include DCGANs, MLPs, models without normalization, gated multiplicative nonlinearities, tanh nonlinearities, and 101-layer ResNets.

  • Architectures: The tested configurations include DCGAN generators and discriminators, 4-layer 512-dim ReLU MLP generators, and 101-layer ResNet generators and discriminators.The passages also list DCGAN setups with no batch normalization and constant filter counts.
  • Architectural variants: The experiments vary normalization and nonlinearities, including no normalization in either generator or discriminator, gated multiplicative nonlinearities, and tanh nonlinearities.These variants appear across the DCGAN, LSGAN, clipped WGAN, and WGAN-GP method listings.
  • Experimental configurations: The LSUN experiments compare DCGAN, LSGAN, WGAN with clipping, and WGAN-GP across several architecture configurations.The listed methods are paired with DCGANs, MLPs, normalization variants, nonlinearities, and 101-layer ResNets.
Loading 1704.00028v3…