PaperlayerView exact revision on arXiv

Source-linked AI summary

Training GANs with Optimism

Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, Haoyang Zeng

arXiv:1711.00141v2cs.LGcs.GTstat.ML

TL;DR

GAN training can exhibit limit cycling because simultaneous gradient dynamics may fail to converge in the last iterate. The paper applies Optimistic Mirror Descent and Optimistic Adam to WGAN training, proving last-iterate convergence for bi-linear games and reporting improved empirical performance.

  • Problem

    GANs are difficult to train because zero-sum no-regret dynamics can cycle, while parameter averaging is impractical for deep non-convex-concave games.

  • Method

    The paper replaces gradient descent with Optimistic Mirror Descent for WGANs and introduces Optimistic Adam as an optimistic Adam variant.

  • Results

    OMD converges to equilibrium in the last iterate for bi-linear zero-sum games, while OMD-based training outperforms GD variants on DNA KL divergence and Adam on CIFAR10 inception score.

  • Takeaways & Limitations

    Optimistic dynamics provide a training approach that addresses limit cycling and supports stable last-iterate behavior in the settings studied.

  • Takeaways & Limitations

    Theoretical guarantees beyond convex-concave settings are unavailable for GD, while the paper’s OMD analysis assumes λ∞ ≤ 1.

Abstract

from arXiv · show

We address the issue of limit cycling behavior in training Generative Adversarial Networks and propose the use of Optimistic Mirror Decent (OMD) for training Wasserstein GANs. Recent theoretical results have shown that optimistic mirror decent (OMD) can enjoy faster regret rates in the context of zero-sum games. WGANs is exactly a context of solving a zero-sum game with simultaneous no-regret dynamics. Moreover, we show that optimistic mirror decent addresses the limit cycling problem in training WGANs. We formally show that in the case of bi-linear zero-sum games the last iterate of OMD dynamics converges to an equilibrium, in contrast to GD dynamics which are bound to cycle. We also portray the huge qualitative difference between GD and OMD dynamics with toy examples, even when GD is modified with many adaptations proposed in the recent literature, such as gradient penalty or momentum. We apply OMD WGAN training to a bioinformatics problem of generating DNA sequences. We observe that models trained with OMD achieve consistently smaller KL divergence with respect to the true underlying distribution, than models trained with GD variants. Finally, we introduce a new algorithm, Optimistic Adam, which is an optimistic variant of Adam. We apply it to WGAN training on CIFAR10 and observe improved performance in terms of inception score as compared to Adam.

1 INTRODUCTION

GAN training is framed as a zero-sum game, but simultaneous gradient-based updates can cycle instead of converging in the last iterate. The paper proposes optimistic methods and reports theoretical, toy-example, DNA-sequence, and CIFAR10 evidence supporting their use.

  • Motivation: GAN training commonly solves a Generator–discriminator zero-sum game with simultaneous stochastic gradient-based updates.The Generator maps random noise to data-space samples, while the discriminator distinguishes real from generated samples.
  • Motivation: No-regret dynamics can oscillate around equilibrium, and averaging parameters is impractical for non-convex-concave games involving deep networks.The paper therefore focuses on methods whose last iterate approaches equilibrium rather than relying only on averaged parameters.
  • Contributions: The paper proposes Optimistic Mirror Descent for WGANs, using predicted opponent gradients to obtain faster regret rates than gradient descent.OMD is presented as a small modification of gradient descent that exploits predictability between successive opponent gradients.
  • Contributions: For bi-linear zero-sum games, OMD converges to equilibrium in the last iterate, providing theoretical evidence for improved training stability.This contrasts with the limit-cycle behavior associated with gradient-descent dynamics.
  • Empirical evidence: Toy simulations show GD variants can cycle while OMD converges pointwise, and OMD-based WGANs achieve lower KL divergence on DNA-sequence generation.The paper also introduces Optimistic Adam and reports better CIFAR10 inception score than Adam.

2 PRELIMINARIES: WGANS AND OPTIMISTIC MIRROR DESCENT

WGANs formulate generation as a zero-sum game whose standard simultaneous gradient updates lack last-iterate guarantees. Optimistic Mirror Descent replaces these updates with prediction-based dynamics and extends naturally to stochastic training.

  • GAN and WGAN formulation: A GAN uses a generator that maps random noise to samples and a discriminator that classifies samples as real or fake in a zero-sum game.The generator seeks to fool the discriminator.
  • GAN and WGAN formulation: A powerful classifier discriminator can cause vanishing generator gradients, motivating WGANs’ use of a 1-Lipschitz function to approximate Wasserstein-1 distance.WGANs replace the classifier-style discriminator with an arbitrary 1-Lipschitz function.
  • Gradient dynamics: The classic WGAN approach updates both players simultaneously with stochastic gradient descent on the zero-sum loss inf_θ sup_w L(θ,w).The gradients are evaluated with respect to the current generator and discriminator parameters.
  • Gradient dynamics: In convex-concave games, GD guarantees an approximate equilibrium for averaged parameters, but not for the last-iterate pair; beyond that setting, guarantees are unavailable.The average-parameter guarantee assumes bounded convex sets and an appropriately chosen step size.
  • Optimistic Mirror Descent: OMD predicts the next iteration’s gradient, often using the previous gradient, and augments the Follow-the-Regularized-Leader formulation with that prediction.The paper analyzes last-gradient prediction theoretically and explores other predictors experimentally.
  • Optimistic Mirror Descent: OMD’s update is presented as inherently different from existing GAN adaptations such as Nesterov momentum and gradient penalty.The paper separately compares these approaches in illustrative dynamics.
  • Stochastic training: Replacing exact gradients with unbiased single-sample or minibatch estimates yields stochastic GD and stochastic OMD algorithms.This permits training with empirical distributions when expected gradients are impractical to compute directly.

3 AN ILLUSTRATIVE EXAMPLE: LEARNING THE MEAN OF A DISTRIBUTION

In a simple WGAN that learns the mean of a multivariate normal distribution, GD produces limit cycles while OMD converges to the target in the last iterate, including with stochastic gradients.

  • The example uses Q = N(v, I), with the generator learning the unknown mean parameter v.
  • The WGAN uses a linear discriminator and an additive generator displacement of Gaussian input noise.
  • The game’s unique equilibrium has generator parameter θ = v and discriminator parameter w = 0.
  • GD dynamics produce a limit cycle, whereas OMD dynamics converge to v in terms of the last iterate.
  • OMD’s stability carries over to stochastic gradients when the batch size is sufficiently large.
  • Gradient penalty, Nesterov momentum, and multiple discriminator updates narrow GD’s cycle but leave it non-vanishing, unlike OMD.

4 LAST-ITERATE CONVERGENCE OF OPTIMISTIC ADVERSARIAL TRAINING

The paper proves that OMD converges in the last iterate to equilibrium for bilinear zero-sum games, contrasting with GD’s divergence or cycling behavior.

  • The theoretical analysis studies min_x max_y x^T A y and establishes final-iterate rather than only average-iterate convergence for OMD.
  • The OMD iteration uses initialization in the relevant column spaces, with x−1 = 2x0 and y−1 = 2y0.
  • The analysis assumes λ∞ ≤ 1, enforceable by positively scaling A without changing the min-max solutions.
  • For the bilinear game, equilibrium pairs have x in the null space of A^T and y in the null space of A.
  • GD diverges for min_x max_y x^T y from any initialization with nonzero x0 and y0.
  • For sufficiently small η, OMD’s last iterate eventually lies within O(√η · γ√∆0) of the equilibrium space.

5 EXPERIMENTAL RESULTS FOR GENERATING DNA SEQUENCES

The DNA-sequence experiment trains WGANs on samples from a position-weight-matrix distribution and compares stochastic OMD and SGD variants using KL divergence.

  • DNA sequences sharing a cellular function are modeled as samples from a nucleotide distribution represented by a position-weight matrix.
  • The experiment generates 40,000 six-nucleotide sequences, holds out 10% for validation, and evaluates WGANs using KL divergence.
  • The study compares GD variants including momentum, Nesterov momentum, and Adagrad.
  • OMD variants use last-iteration, running-average, or hyperbolically discounted gradient predictors, with either 5:1 or 1:1 discriminator-generator updates.
  • When selecting by the lowest validation discriminator loss, Stochastic OMD achieves lower KL divergence than competing SGD variants.
  • When evaluated at the last epoch, optimistic Adam achieves the best performance across learning rates.

6 GENERATING IMAGES FROM CIFAR10 WITH OPTIMISTIC ADAM

The paper applies Optimistic Adam to WGAN image generation on CIFAR10 and reports higher inception scores than Adam, including stronger performance with 1:1 training.

  • Optimistic Adam is introduced as an optimistic variant of Adam for training WGANs on images.The algorithm maintains exponential first- and second-moment estimates and performs an optimistic gradient step.
  • Optimistic Adam outperforms Adam in terms of inception score on CIFAR10.Inception score is used as a standard metric for WGAN image quality.
  • With Optimistic Adam, training the discriminator once per generator iteration outperforms the 1:5 generator-discriminator training scheme.This comparison uses matched learning rates, betas, gradient-penalty coefficients, and batch sizes across methods.
  • Vanilla Adam performs worse than Optimistic Adam with 1:1 training, even when Adam uses five discriminator updates per generator update.

A VARIANTS OF GD TRAINING

This section gives the update rules for several gradient-descent modifications used to compare alternative GAN training dynamics.

  • The section describes exact update rules for multiple modifications of gradient-descent training.
  • Basic gradient descent updates the generator and discriminator parameters using their respective gradients and learning rates.
  • Momentum augments gradient updates with exponentially decayed velocity terms for both players.
  • Nesterov-style updates evaluate gradients at look-ahead parameters before applying momentum-based updates.

B PERSISTENCE OF LIMIT CYCLES IN GD TRAINING

The experiments examine whether common gradient-descent adaptations eliminate cycling, finding that oscillations persist, while asymmetric training only reduces their range.

  • Oscillations persist across the modified gradient-descent dynamics shown in Figure 5, although some adaptations alleviate them.
  • Gradient penalty: Gradient penalty modifies WGAN training by constraining the discriminator’s gradient to support the Lipschitz condition.
  • Momentum and Nesterov momentum: Momentum and Nesterov momentum provide alternative gradient-descent dynamics, with Nesterov matching momentum without gradient penalty in the bi-linear example.
  • Asymmetric training: Asymmetric training is the most effective tested modification for reducing the range of cycles and bringing the last iterate closer to equilibrium.
  • Asymmetric training: Asymmetric training does not eliminate cycles; it only makes their range smaller.

C ANOTHER EXAMPLE: LEARNING A CO-VARIANCE MATRIX

This section extends the OMD-versus-GD comparison to covariance learning, using Gaussian data and linear generators. The experiments examine deterministic and stochastic dynamics, with regularization and weight clipping used in the stated settings.

  • Comparison: The section compares OMD and GD dynamics beyond the bilinear case through covariance-learning simulations.The authors state that the example is not bilinear and tests whether the theoretical behavior carries over qualitatively.
  • Model: The example models a mean-zero multivariate Gaussian with unknown covariance using quadratic discriminators and linear generators.The generator maps z ∼ N(0, I) linearly, with both parameter matrices W and V of size d × d.
  • Equilibrium: Writing Σ = UU^T simplifies the WGAN loss, with equilibrium V_ik = U_ik and W_ij = 0.In one dimension, the equilibrium is V = σ and W = 0.
  • Stabilization: The covariance-learning setup adds ℓ2 regularization because generator non-convexity and non-unique optima can arise when Σ is not strictly positive definite.Regularization contributes 2λW_t and 2λV_t to the respective gradient terms.
  • Stochastic dynamics: Stochastic experiments replace gradients with unbiased estimates or small-minibatch averages to compare GD instability with OMD stability.Figures 8 and 9 use discriminator weight clipping in [−1, 1].
  • Experiments: Figures 6 and 7 examine stability for two- and three-dimensional Gaussians under weight clipping in [−1, 1].The captions identify OMD-versus-GD stability as the comparison in the covariance-learning problem.

D.1 PROOF OF THEOREM 1

This section proves last-iterate convergence of OMD for bilinear min-max problems under stated assumptions and characterizes the general affine case. It contrasts this behavior with GD, whose iterates diverge in the bilinear example.

  • Proof: Under the theorem’s spectral-norm and sufficiently small step-size conditions, the proof establishes bounds on the OMD iterates and their distance to equilibrium.The initialization and auxiliary lemmas support an induction over time.
  • Equilibrium: The equilibrium solutions of the homogeneous bilinear problem are pairs with x in the null space of A^T and y in the null space of A.This identifies the equilibrium space used in the convergence statement.
  • General bilinear case: For finite affine problems, OMD retains last-iterate convergence after decomposing the linear terms and applying the homogeneous result.The analysis shifts variables before invoking the bilinear convergence theorem.
  • General bilinear case: When the affine problem is infinite or undefined, OMD travels to infinity, with the analysis characterizing its motion.This is the stated boundary of the general-case result.
  • Gradient descent comparison: For the special GD dynamics on f(x,y) = x^T y, the running iterate diverges from equilibrium for every η > 0.The squared distance obeys d(t) = (1 + η^2)d(t − 1).

E DNA-GENERATION WGAN ARCHITECTURE

The DNA-generation architecture specifies separate generator and discriminator pipelines for short DNA sequences, including convolutional processing, upsampling, and a gradient penalty.

  • Configuration: The DNA setup uses sequence length L = 6, gradient penalty λ = 1e−4, and batch size 512.The generator input dimension is 50.
  • Generator: The generator projects noise through fully connected layers, reshapes it, upsamples, and applies four [1 × 3] convolutions.The final convolution outputs 4 × 1 × L before the discriminator receives the generated sequence representation.
  • Discriminator: The discriminator processes the 4 × 1 × L input with sixteen [1 × 3] convolutions followed by fully connected layers ending in one linear output.The listed discriminator output is scalar-valued.
  • CIFAR10 configuration: The CIFAR10 configuration uses gradient penalty λ = 10 and batch size 64.Its generator and discriminator architecture is listed separately from the DNA configuration.

G CIFAR10 GENERATOR IMAGE SAMPLES

The CIFAR10 image-sample section compares Optimistic Adam with Adam at 1:1 and 5:1 training ratios. The reported early-epoch comparison favors Optimistic Adam visually, while Figure 4 tracks inception scores across training.

  • Image samples: Figure 10 presents CIFAR10 samples from a generator trained with Optimistic Adam.The caption identifies the optimizer but not a separate quantitative outcome.
  • Image samples: At epoch 19, Optimistic Adam with a 1:1 training ratio is compared against Adam with 1:1 and 5:1 ratios on CIFAR10.The corresponding figures show generator samples for each configuration.
  • Image samples: The authors report that Optimistic Adam had already produced visually appealing results at epoch 19, unlike the two vanilla Adam configurations.This is a qualitative comparison of early generated images.
  • Inception scores: Figure 4 compares inception scores across 30 epochs for Optimistic Adam at ratio 1 and Adam at ratio 5.The curves summarize 35 runs with 10%-90% confidence intervals.
Loading 1711.00141v2…