PaperlayerView exact revision on arXiv

Source-linked AI summary

A Closer Look at the Optimization Landscapes of Generative Adversarial Networks

Hugo Berard, Gauthier Gidel, Amjad Almahairi, Pascal Vincent, Simon Lacoste-Julien

arXiv:1906.04848v3cs.LGstat.ML

TL;DR

GANs are successful generative models but remain challenging to train, and whether rotation predicted by game theory occurs in practical GAN settings was unresolved. The paper visualizes the game vector field and finds rotation around stable stationary points, including generator-loss saddles that can still have good performance.

  • Problem

    Whether adversarial rotation predicted by game theory occurs in practical GAN training remained open beyond toy examples far from realistic deep-network image generation.

    Source: §1 · p. 1

  • Method

    The paper studies the concatenated player-gradient game vector field with Path-angle visualizations and Jacobian eigenvalue analysis.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Across GAN formulations, optimization methods, and datasets, training shows rotational components around locally stable stationary points, and often ends at generator-loss saddles rather than minima despite good performance.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Good performance at locally stable stationary points raises whether GANs need a Nash equilibrium to achieve good generator performance.

    Source: §6 · p. 10

    The §3.3 theorem is stated under the assumption that equation (6) holds with equality.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Generative adversarial networks have been very successful in generative modeling, however they remain relatively challenging to train compared to standard deep neural networks. In this paper, we propose new visualization techniques for the optimization landscapes of GANs that enable us to study the game vector field resulting from the concatenation of the gradient of both players. Using these visualization techniques we try to bridge the gap between theory and practice by showing empirically that the training of GANs exhibits significant rotations around Local Stable Stationary Points (LSSP), similar to the one predicted by theory on toy examples. Moreover, we provide empirical evidence that GAN training converge to a stable stationary point which is a saddle point for the generator loss, not a minimum, while still achieving excellent performance.

1 INTRODUCTION

GANs pose a distinct landscape problem because adversarial training jointly optimizes multiple objectives, while practical rotation had remained an open question. The paper studies the game vector field with Path-angle and reports rotation around stable stationary points, alongside convergence to stable points that are not local Nash equilibria.

1 INTRODUCTION

  • Existing theory associated adversarial games with rotation, but whether it occurs in practical GAN training remained open because prior examples were far from realistic deep-network image generation.Source: §1 · p. 1
  • The paper studies the concatenated player-gradient game vector field rather than individual loss surfaces, using Path-angle to visualize its behavior near stationary points.Source: §1 · pp. 1–2
  • Experiments with NSGAN and WGAN-GP, Adam and ExtraAdam, and MoG, MNIST, and CIFAR10 provide evidence that existing GAN training methods do not converge to local Nash equilibria.Source: §1 · p. 2
  • Across mixture-of-Gaussians, MNIST, and CIFAR10 experiments, varied GAN formulations show significant rotation around locally stable stationary points and can settle at generator-loss saddles rather than minima.The paper also reports that these stable stationary points achieve good generator performance metrics.Source: §1 · p. 2

2 RELATED WORK

Prior work studies GAN stabilization, game-theoretic stability, and neural-network loss landscapes through different methods. This paper relates those approaches to empirical analysis of GAN landscapes and stable points.

2 RELATED WORK

  • Prior GAN stabilization efforts focused mainly on new objectives or added regularization, while this work characterizes landscapes across formulations and their relation to training improvement.Source: §2 · p. 2
  • Local Jacobian-eigenvalue analyses provide theoretical stability guarantees under assumptions such as a generator able to fully capture the real distribution; this work tests their practical applicability.Source: §2 · pp. 2–3
  • Earlier work treated game rotations theoretically, whereas this paper investigates whether they occur in practice for GANs with highly non-convex losses and real-world datasets.Source: §2 · p. 2
  • Prior studies found that locally stable stationary points need not be local Nash equilibria, and this work argues that good performance may be reached without attaining one.Source: §2 · p. 2
  • Linear-path visualizations and local Hessian analyses offer different views of neural-network landscapes; for games, the Jacobian can have complex eigenvalues, which this work uses to assess GAN stability.Source: §2 · pp. 2–3

3 FORMULATIONS FOR GAN OPTIMIZATION AND THEIR PRACTICAL

The section reframes GAN optimization as joint game dynamics, distinguishing local Nash stability from stability of the interacting players. It formalizes attraction and rotation around locally stable stationary points and motivates visualizing these dynamics.

3.2 AN ALTERNATIVE FORMULATION BASED ON THE GAME VECTOR FIELD

  • The game vector field treats the players' parameters as a joint state and concatenates their gradients, capturing interactions that a single loss surface omits.In practice, this joint vector field is not generally the gradient of a single function.Source: §3.2 · p. 3, §3.1 · p. 3
  • A differentiable Nash equilibrium is sufficient but not necessary for a locally stable stationary point, and the two notions are generally neither equivalent nor interchangeable.The section contrasts static stability of each player's loss with joint dynamic stability.Source: Table 1 · p. 4, §3.2 · p. 4
  • An LSSP is a stationary point whose game-vector-field Jacobian has eigenvalues with positive real parts, so nearby points are attracted to it.The section interprets the condition as local attraction and formalizes it through the Jacobian spectrum.Source: §3.2 · p. 4
  • In the example, discriminator updates rotate the generator's saddle so that its descent direction becomes an ascent direction, stabilizing the joint dynamics.The generator loss remains unstable for fixed discriminator parameters, but becomes stable through their dynamic interaction.Source: §3.2 · p. 4
  • A Nash equilibrium describes static stability with the other player's parameters fixed, whereas an LSSP describes joint dynamic stability.The distinction matters because the generator's loss can have a saddle at a jointly stable point.Source: §3.1 · p. 3, §3.2 · p. 4
  • The game vector-field visualization shows trajectories rotating around and converging to a stationary point that is a saddle in the generator's fixed-discriminator loss.The accompanying loss plot is only a snapshot at fixed discriminator parameters.Source: §3.2 · p. 4, Figure 1 · p. 5

3.3 ROTATION AND ATTRACTION AROUND LOCALLY STABLE STATIONARY POINTS IN GAMES

  • Under the stated equality and diagonalizability assumptions, the dynamics decompose in a suitable basis into attraction for real eigenvalues and rotation for complex eigenvalues.The proposition's components do not interact in that basis; rotation is absent around a local minimum in single-objective minimization because its Hessian eigenvalues are real.Source: §3.3 · p. 5

4 VISUALIZATION FOR THE VECTOR FIELD LANDSCAPE

The section adapts low-dimensional path visualizations to the joint game vector field, introducing path-norm and path-angle measures alongside Jacobian eigenvalues. These tools distinguish attraction from rotation near an LSSP.

4.1 STANDARD VISUALIZATIONS FOR THE LOSS SURFACE

  • Standard loss-landscape visualizations follow paths between parameter values, often using a linear path, while nonlinear paths can test whether minima are connected.Neural-network landscapes require low-dimensional visualizations because they have many parameters.Source: §4.1 · p. 6, §4 · p. 6

4.2 PROPOSED VISUALIZATION: PATH-ANGLE

  • Linear interpolation between early- and late-training parameters provides a path for inspecting the game vector field, with pure attraction and pure rotation as reference behaviors.The expected practical behavior lies between these extremes.Source: §4.2 · p. 6
  • Path-norm checks whether the joint vector field is small near a stationary point; one player's near-zero gradient alone can be misleading.Source: §4.2 · p. 6
  • Path-angle is the cosine between the game vector field and the linear path toward a candidate final point, evaluated along that path.The path is evaluated near an LSSP; the interval can be extrapolated beyond the endpoint when useful.Source: §4.2 · p. 6
  • Top-k eigenvalues of the game-vector-field Jacobian indicate local stability through their real parts and rotational behavior through large imaginary parts.Eigenvalues of the diagonal Jacobian blocks can also indicate whether an LSSP is not a local Nash equilibrium.Source: §4.2 · p. 6
  • Unlike Jacobian eigenvalue computation, Path-angle requires only gradients and describes behavior along a path rather than at one point.The paper notes that second-order derivatives may be prohibitively expensive for deep networks.Source: §4.2 · p. 6

4.3 ARCHETYPAL BEHAVIORS OF THE PATH-ANGLE AROUND A LSSP

  • For pure attraction, path-angle changes sign around α = 1; for pure rotation it has a bump there, and general LSSPs combine both patterns.The bump grows as behavior approaches pure rotation, while low-dimensional projection can keep the measured angle below 1.Source: §4.3 · pp. 6–7, Figure 2 · p. 7

5 NUMERICAL RESULTS ON GANS

The experiments examine GAN losses, datasets, and optimizers using path-angle and game-Jacobian visualizations. They find rotational dynamics around many locally stable stationary points, while generators often converge to saddles rather than local Nash equilibria despite good performance.

5 NUMERICAL RESULTS ON GANS

  • The experiments compare NSGAN and WGAN-GP on MoG, MNIST, and CIFAR10, using extragradient for MoG and Adam or ExtraAdam for the image datasets.Adam sometimes converged to unstable points, so the reported image-dataset observations use ExtraAdam; its Inception scores were slightly better.Source: §5 · pp. 7–8

5.1 EVIDENCE OF ROTATION AROUND LOCALLY STABLE STATIONARY POINTS IN GANS

  • Path-angle bumps and large imaginary game-Jacobian eigenvalues indicate rotational components; NSGAN eigenvalues cluster nearer the imaginary axis, while WGAN-GP spreads rightward.Most endpoints are LSSPs, although some eigenvalues indicate an unstable point; the authors include those results because performance is similar to an LSSP.Source: Figure 3 · p. 8, Figure 4 · p. 8, §5.1 · p. 9

5.2 THE LOCALLY STABLE STATIONARY POINTS OF GANS ARE NOT LOCAL NASH EQUILIBRIA

  • Generators reach saddle points rather than local minima, so the observed LSSPs are not local Nash equilibria despite good evaluation performance.The Hessian results cover NSGAN and WGAN-GP; for CIFAR10, both players are at saddle points under NSGAN.Source: Figure 5 · p. 9, §5.2 · p. 10, Figure 6 · p. 10

6 DISCUSSION

Across GAN formulations, optimization methods, and datasets, training consistently fails to converge to local Nash equilibria, while LSSPs with generator saddle points can still yield good performance. The observed rotational components may contribute to training difficulty and motivate methods designed to handle rotation.

6 DISCUSSION

  • Across GAN formulations, standard optimizers, and datasets, training consistently does not converge to local Nash equilibria, yet LSSPs with generator saddle points can deliver good performance.The authors also argue that game-specific rotations may contribute to training difficulty and that low-dimensional examples capture some challenges of large-scale GANs.Source: §6 · p. 10

A.1 PROOF OF THEOREM 1

The appendix derives how Jacobian eigenvalues shape local game dynamics, shows that differential Nash equilibria and locally stable stationary points need not coincide, and describes a Jacobian-vector-product approach to estimating top-k eigenvalues.

A.1 PROOF OF THEOREM 1

  • Real Jacobian eigenvalues yield pure attraction, zero-real-part eigenvalues yield pure rotation, and other eigenvalues combine attraction and rotation.The proof constructs a real basis from the complex-conjugate eigenvalue pairs to express these dynamics.Source: §A.1 · p. 13

A.2 BEING A DNE IS NEITHER NECESSARY OR SUFFICIENT FOR BEING A LSSP

  • The examples show that a differential Nash equilibrium need not be locally stable, while a locally stable stationary point need not be a differential Nash equilibrium.The stationary point (0, 1) is a DNE but not an LSSP; (0, −1) is an LSSP but not a DNE.Source: §A.2 · p. 14

B COMPUTATION OF THE TOP-K EIGENVALUES OF THE JACOBIAN

  • The top-k Jacobian eigenvalues are computed using automatic differentiation for Jacobian-vector products and Scipy’s sparse.linalg.eigs.The procedure first computes the game vector field’s gradient over a subset of the dataset.Source: B COMPUTATION OF THE TOP-K EIGENVALUES… · p. 15

C.1 MIXTURE OF GAUSSIAN EXPERIMENT

The Mixture of Gaussians experiments use a fixed full-batch sample, one-hidden-layer networks, and separate setup choices for WGAN-GP and NSGAN.

C.1 MIXTURE OF GAUSSIAN EXPERIMENT

  • The Mixture of Gaussians dataset uses 10,000 independently sampled points from a two-component one-dimensional Gaussian mixture, reused for full-batch training.The components are N(2, 0.5) and N(−2, 1), each with weight one-half; latent variables are standard normal.Source: §C.1 · p. 15
  • Both generator and discriminator are one-hidden-layer networks with 100 hidden units and ReLU activations.Source: §C.1 · p. 15
  • For WGAN clipping, the path-angle gradient is filtered to exclude discriminator-gradient components that clipping would leave unchanged after a gradient step.A coordinate is filtered when its clipped value would not change after the step.Source: §C.1 · p. 15
  • WGAN-GP and NSGAN each use 30,000 iterations with full-batch size 10,000, while their generator learning rates are 1 × 10−2 and 1 × 10−1, respectively.Source: §C.1 · p. 15

C.2 MNIST EXPERIMENT

The MNIST experiments specify the data, architecture, and training configurations, while the path-angle plots summarize behavior near selected low-gradient, good-performance points.

C.2 MNIST EXPERIMENT

  • The experiments train on 50,000 MNIST examples scaled to [−1, 1] using DCGAN generators and discriminators with NSGAN and WGAN-GP objectives.The discriminator’s Batch-norm layer is replaced by Spectral-norm, which the authors report finding stabilizes training.Source: §C.2 · p. 15
  • NSGAN runs use batch size 100 for 100,000 iterations, while WGAN-GP runs use batch size 100 for 200,000 iterations.Generator learning rates are 2 × 10−4 for NSGAN and 8.6 × 10−5 for WGAN-GP; Adam and ExtraAdam configurations are reported for both objectives.Source: §C.2 · p. 16
  • The MNIST inception scores are computed with a pretrained LeNet classifier, and real MNIST data has an average score of 9.9.Source: §C.2 · p. 16

C.3 PATH-ANGLE PLOT

  • Path-angle plots use initialization as the start and multiple nearby, good-performance training points as endpoints, reporting median angles and interquartile ranges.The multiple endpoints account for continued movement around a low-gradient LSSP neighborhood under stochastic noise.Source: §C.3 · p. 16

C.4 INSTABILITY OF GRADIENT DESCENT

On the Mixture of Gaussians task, gradient descent is reported as unstable, while extra-gradient reaches a low gradient norm that indicates convergence.

C.4 INSTABILITY OF GRADIENT DESCENT

  • On the Mixture of Gaussians dataset, gradient descent produces unstable results compared with extra-gradient.Source: §C.4 · p. 16
  • Gradient descent’s gradient norm has much greater variance than extra-gradient’s in the reported comparison.The variance comparison is shown in Figure 7.Source: §C.4 · p. 17, Figure 7 · p. 17
  • Extra-gradient reaches a low gradient norm, indicating convergence, while gradient descent does not appear to converge.Source: Figure 7 · p. 17

C.5 ADDITIONAL RESULTS WITH ADAM

The additional Adam results show path-angle and eigenvalue visualizations on MNIST and CIFAR10. For NSGAN on CIFAR10, the passage reports eigenvalues with negative real parts and interprets the point as unstable.

C.5 ADDITIONAL RESULTS WITH ADAM

  • Figure 8 presents path-angle and eigenvalue results computed with Adam on MNIST.Source: Figure 8 · p. 17
  • On CIFAR10, NSGAN has eigenvalues with negative real parts, which the passage interprets as indicating an unstable point.Figure 9 presents the path-angle and eigenvalue results computed with Adam.Source: Figure 9 · p. 18
Loading 1906.04848v3…