PaperlayerView exact revision on arXiv

Source-linked AI summary

On the regularization of Wasserstein GANs

Henning Petzka, Asja Fischer, Denis Lukovnikov

arXiv:1709.08894v3stat.MLcs.LG

TL;DR

GAN training can be stabilized by replacing its original divergence with the Wasserstein distance, but this introduces a Lipschitz constraint requiring regularization. The paper analyzes weaknesses in gradient penalties that force gradient norms toward one and proposes a one-sided penalty that directly penalizes violations above one, with experiments supporting better stability and lower sensitivity to the penalty weight.

  • Problem

    WGANs improve convergence but require enforcing a Lipschitz constraint, and existing gradient penalties may impose overly strong assumptions about sampling and differentiability.

  • Method

    The paper proposes penalizing only gradient-norm violations above one, rather than penalizing every deviation from one, while directly targeting the Lipschitz constraint.

  • Results

    The proposed WGAN-LP penalty performs better for larger λ values and is less sensitive to λ than WGAN-GP in the reported experiments.

  • Takeaways & Limitations

    A weaker one-sided gradient penalty provides more stable learning and better-behaved critic scores across penalty weights in the reported toy-data experiments.

  • Takeaways & Limitations

    The theoretical proposition assumes a differentiable optimal critic and samples drawn from an optimal coupling, whereas practice samples independently from marginal distributions.

Abstract

from arXiv · show

Since their invention, generative adversarial networks (GANs) have become a popular approach for learning to model a distribution of real (unlabeled) data. Convergence problems during training are overcome by Wasserstein GANs which minimize the distance between the model and the empirical distribution in terms of a different metric, but thereby introduce a Lipschitz constraint into the optimization problem. A simple way to enforce the Lipschitz constraint on the class of functions, which can be modeled by the neural network, is weight clipping. It was proposed that training can be improved by instead augmenting the loss by a regularization term that penalizes the deviation of the gradient of the critic (as a function of the network's input) from one. We present theoretical arguments why using a weaker regularization term enforcing the Lipschitz constraint is preferable. These arguments are supported by experimental results on toy data sets.

1 INTRODUCTION

WGANs replace the original GAN metric with the Wasserstein-1 distance, improving convergence properties but imposing a 1-Lipschitz constraint. This paper reviews existing regularization arguments and proposes a less restrictive alternative supported by theory and experiments.

  • WGANs minimize the Wasserstein-1 distance instead of the Jensen-Shannon distance to address convergence problems.
  • The Wasserstein formulation requires the discriminator function to belong to the space of 1-Lipschitz functions.
  • Weight clipping was introduced as a simple way to enforce the Lipschitz constraint by restricting network parameters.
  • The paper proposes a less restrictive regularization term for WGANs based on theoretical considerations and empirical results.
  • The authors argue that gradient regularization can be harmful when samples are independently drawn from marginals or when the discriminator is nondifferentiable.

2 OPTIMAL TRANSPORT

The paper introduces couplings as joint distributions with specified marginals and reviews optimal transport's primal and dual formulations. Kantorovich duality connects optimal couplings with optimal 1-Lipschitz functions.

  • A coupling of probability distributions µ and ν on Rn is a distribution on Rn × Rn whose marginals are µ and ν.
  • The set of all couplings of µ and ν is denoted Π(µ, ν).
  • Kantorovich duality characterizes the transport problem through a maximum over 1-Lipschitz functions.
  • An optimal coupling and an optimal 1-Lipschitz function satisfy f*(x) − f*(y) = ||x − y||2 on the coupling's support.
  • Optimal transport computes the minimum expected distance between paired points under a coupling, with regularized variants adding terms such as entropy or Ω(π).

3 WASSERSTEIN GANS

WGAN training replaces the original GAN's Jensen-Shannon objective with Wasserstein-1 distance optimization over 1-Lipschitz critics. The critic guides generation through confidence values rather than binary real/fake classification.

  • Original GAN training alternates updates to a generator and discriminator that distinguish generated data from real data.
  • The original GAN objective minimizes the Jensen-Shannon divergence when an optimal discriminator is found at each iteration.
  • WGAN replaces Jensen-Shannon divergence minimization with Wasserstein-1 distance minimization.
  • The WGAN dual objective maximizes over the set of all 1-Lipschitz functions.
  • Alternating gradient updates train the generator and neural-network critic, whose values guide generation toward real-looking data.
  • Unlike the original discriminator, the WGAN critic's values are not interpreted directly as probabilities of real or fake data.

4 IMPROVED TRAINING OF WGANS

The paper argues that gradient-penalty regularization relies on restrictive assumptions about optimal couplings and critic differentiability. It motivates directly penalizing violations of the Lipschitz constraint instead.

  • Regularizing the critic: Weight clipping restricts neural-network critic parameters to enforce an α-Lipschitz class, but the optimal function may not belong to that class.The paper presents this as a limitation of the original WGAN enforcement strategy.
  • Gradient penalty: WGAN-GP penalizes deviations of the critic’s input-gradient norm from one along interpolations between independently sampled real and generated points.The interpolation distribution uses x ∼ µ, y ∼ ν, and t ∼ U[0,1].
  • Sampling from the marginals instead of the optimal coupling: The unit-gradient condition is guaranteed only when the critic is differentiable and points are sampled from the optimal coupling π∗.Independent marginal sampling can produce pairs outside the support of π∗.
  • Sampling from the marginals instead of the optimal coupling: For arbitrary real-generated pairs, equality along interpolation lines need not hold; higher-dimensional examples can contain points where the Lipschitz inequality is strict.The one-dimensional example still has gradient norm one almost everywhere, whereas the higher-dimensional example need not.
  • Differentiability of the critic: Optimal critics can be nondifferentiable, including at points arising from discrete distributions and at branching points of nondeterministic optimal couplings.A nondeterministic optimal coupling with non-collinear destinations forces nondifferentiability at the shared source point.
  • Differentiability of the critic: Because neural-network approximations are differentiable almost everywhere, enforcing unit gradient near critic nondifferentiabilities can impose a strong constraint.The authors therefore argue against assuming unit gradient on every line between arbitrary real and generated samples.

5 HOW TO REGULARIZE WGANS

The paper proposes WGAN-LP, a one-sided regularization that directly penalizes violations of the Lipschitz constraint rather than deviations of gradient norms from one. Theoretical analysis connects this choice to optimal transport and experiments indicate greater robustness to the penalty weight.

  • Penalizing the violation of the Lipschitz constraint: WGAN-LP directly penalizes violations of the Lipschitz inequality between independently sampled real and generated points.The penalty squares deviations so larger violations receive greater weight.
  • Penalizing the violation of the Lipschitz constraint: The one-sided penalty penalizes critic gradients only when their norm exceeds one, enforcing the relevant upper bound.This targets the steepest-descent direction without penalizing gradients below one.
  • Penalizing the violation of the Lipschitz constraint: WGAN-LP alternates discriminator updates using the penalty with generator updates minimizing the negative critic value on generated samples.The sampling strategy determines the interpolation point used in the regularization term.
  • The connection to regularized optimal transport: The proposed formulation has a connection to regularized optimal transport through a soft constraint that uses a single critic function.The corresponding dual perspective relates the regularization weight to the primal regularization parameter.
  • Comparison and generalization: For small λ, WGAN-LP and WGAN-GP have approximately agreeing optimal scores; for larger λ, WGAN-LP performs better and depends less on λ.The paper also notes that the framework extends to other metrics, including Wasserstein-2.
  • Comparison and generalization: The authors frame the proposal as a less restrictive alternative to gradient-penalty regularization, while noting contrary conclusions to earlier empirical claims.The cited discussion specifically contrasts the paper’s experiments with claims that two-sided penalties converge faster and reach better optima.

6 EXPERIMENTS

Experiments on toy distributions and CIFAR-10 compare WGAN-GP with WGAN-LP across critic behavior, estimated Wasserstein distance, and sample quality. WGAN-LP generally remains stable across penalty weights and outperforms WGAN-GP at larger weights.

  • Experimental setup: Experiments compare WGAN-GP and WGAN-LP on 8Gaussians, 25Gaussians, Swiss Roll, and CIFAR-10.Toy-data penalties were applied to randomly sampled points on lines between real and generated samples.
  • Level sets of the critic: With λ = 10, WGAN-GP learns neither a good critic nor a good data model, whereas WGAN-LP learns a regular critic in few iterations.WGAN-GP stabilizes with λ = 1, while WGAN-LP is less sensitive to larger penalty weights.
  • Evolution of the critic loss: For λ = 5, GP critic loss oscillates heavily around zero, while LP critic loss smoothly decreases to zero.The comparison uses critic losses without their regularization terms.
  • Estimating the Wasserstein distance: For λ = 5, WGAN-LP produces a smaller estimated Wasserstein distance with fewer training fluctuations than WGAN-GP.The estimate uses minimum-assignment costs between real and generated sample sets.
  • Sample quality on CIFAR-10: WGAN-LP matches or slightly exceeds WGAN-GP’s CIFAR-10 Inception score for λ ≤10 while remaining more stable across hyperparameter choices.The reported table values are maximal mean scores during training, computed over 10 image sets.
  • Related penalties: Related penalties perform well on toy data but yield considerably worse and highly unstable CIFAR-10 training.These alternatives are the regularization terms from Equations (7) and (9).
  • Gradient penalty comparison: For λ = 5, LP and GP behave similarly in the gradient penalty’s below-one contribution, while at λ = 100 GP penalties are larger and performance suffers more.Figure 6 separates below-one and above-one contributions and plots per-iteration values with sliding-window means.

7 CONCLUSION

The paper proposes a weaker gradient penalty for enforcing WGAN’s Lipschitz constraint and reports theoretical and empirical evidence supporting it over weight clipping and a stronger penalty.

  • The proposed penalty is Êx∼τ[(max {0, ||∇f(̂x)|| −1})2].
  • The weaker penalty performs better than weight clipping and the stronger penalty Êx∼τ[(||∇f(̂x)||2 −1)2].
  • The proposed regularization produces more stable learning behavior.
  • The proposed penalty is less sensitive to λ, with smooth convergence and well-behaved critic scores across penalty-weight values.

A PROPERTIES OF AN OPTIMAL CRITIC FUNCTION OF WGANS

WGAN critic values need not order real and generated points in a way that directly distinguishes them, even when the critic is optimal.

  • The original GAN discriminator can provide no local incentive to move generated values closer to real data when both are classified as generated.
  • Some generated points have higher critic values than some real points, so critic values alone cannot distinguish fake from real points.
  • In the illustrated setting, an optimal coupling connects the left-most real point to the left-most generated point and matches the remaining points arbitrarily.
  • The indicated critic, with slope 1 or −1 almost everywhere, is optimal by attaining equality in Kantorovich duality.
  • The one- and two-dimensional examples use optimal couplings and critic functions, with the displayed transport and critic objectives both equal to 1 in the two-dimensional case.

B THE ISSUE WITH THE WEIGHT CLIPPING APPROACH

Weight clipping enforces a common Lipschitz bound but restricts neural-network critics so severely that optimal critics are generally excluded.

  • An optimal WGAN critic exhausts the relevant Lipschitz constraint at some pair of points.
  • Weight clipping is therefore not a good strategy for enforcing the critic’s Lipschitz constraint.
  • Clipping weights imposes a common Lipschitz constant over all functions representable by the clipped network.
  • For ReLU networks, exhausting that bound requires constant ±cmax columns in the first layer and all-cmax entries in later layers.
  • Because optimal critics exhaust the Lipschitz constraint but clipped networks rarely do, the optimal critic is usually outside the clipped network class.
  • The layerwise Lipschitz bound is achieved only when suitable point pairs witness the maximal constants through every linear and ReLU layer.
  • For layers i ≥ 2, the required matrices contain cmax in every entry, leaving very few clipped-network functions that exhaust the common bound.

C.1 PROOF OF THEOREM 1

The appendix derives optimal-transport and gradient-based properties used to analyze WGAN critics, including coupling optimality and conflicts caused by differentiability assumptions.

  • The appendix derives its Kantorovich formulation from Villani’s theorem, identifying c-convexity with 1-Lipschitz continuity for Euclidean cost.
  • Optimal couplings are supported on pairs where the optimal critic exhausts the Lipschitz inequality.
  • Along a transport segment attaining equality, the critic must vary linearly with unit slope under the 1-Lipschitz constraint.
  • For symmetric, non-overlapping distributions with connected support, the proof transports left and right halves separately and uses ϕ(x) = −|x| as an optimal critic.
  • The constructed coupling and critic attain equality in Kantorovich duality, establishing their optimality.
  • The analysis also shows that two distinct unit directional derivatives at one differentiable point would require gradient norm greater than one.
  • The appendix includes critic-loss and approximated-EM-distance evolutions for WGAN-GP with λ = 1, plus critic level sets for 8Gaussian and 25Gaussian data.

D.4 DIFFERENT SAMPLING METHODS

Local perturbation reduces some instability from GP regularization, while LP remains less fluctuating; a Wasserstein-2 experiment also shows smooth training behavior.

  • Local perturbation: Adding noise only to training examples stabilizes WGAN-GP later than perturbing both training and generated samples.The critic loss remains more fluctuating than with WGAN-LP.
  • Local perturbation: The experiments compare GP- and LP-penalties while perturbing training samples alone or both training and generated samples.The comparison includes local perturbation variants of the sampling procedure.
  • Local perturbation: The approximated Wasserstein-1 distance fluctuates more with GP than LP under local perturbation.These GP fluctuations are less severe than with Gulrajani et al.’s sampling procedure.
  • Wasserstein-2 training: For Wasserstein-2 training with λ = 10, both critic loss and approximated EM distance decrease smoothly on Swiss Roll.The authors identify Wasserstein-2 as an interesting candidate for further investigation.

D.6 EXPERIMENTAL RESULTS ON CIFAR

On CIFAR, WGAN-LP is less sensitive to the regularization weight than WGAN-GP, while conditional training yields similar performance for both methods and the displayed diagnostics track these behaviors.

  • Inception score: The Inception score evaluates generated-image quality, with higher scores indicating better generative-model performance.It combines conditional label confidence with variability across generated samples.
  • Inception scores and validation loss: WGAN-GP and WGAN-LP perform similarly for small λ, but WGAN-GP performs much worse for larger λ.WGAN-LP is less dependent on the choice of λ.
  • Conditional CIFAR10: Conditional CIFAR10 training produced similar Inception scores: 8.537 ± 0.133 for WGAN-GP and 8.462 ± 0.115 for WGAN-LP.Both models used λ = 10 and label information.
  • Inception scores and validation loss: Figure 16 compares total validation loss, unregularized loss, and regularization loss for WGAN-GP and WGAN-LP at λ = 5 and λ = 100.Light curves show true values, while dark lines average five iterations.
  • Critic level sets: On CIFAR-10, Equation (7) initially improves Inception scores before diverging, whereas Equation (9) produces greater instability.The corresponding level-set training on the displayed toy setting converges properly.
Loading 1709.08894v3…