Gradient descent GAN optimization is locally stable

Vaishnavh Nagarajan; J. Zico Kolter

Source-linked AI summary

Gradient descent GAN optimization is locally stable

Vaishnavh Nagarajan, J. Zico Kolter

arXiv:1706.04156v3Jun 13, 2017cs.LGcs.AImath.OCstat.ML

TL;DR

GAN optimization is poorly understood because practical simultaneous parameter updates do not satisfy the convex-concave assumptions behind standard convergence analyses. The paper analyzes these dynamics through local stability theory, showing stability for traditional GANs under suitable conditions, instability for WGANs, and regularization that stabilizes both formulations.

Problem
GAN optimization lacks theoretical understanding in the practical setting where generator and discriminator parameters are updated simultaneously, rather than through idealized convex-concave or fully optimized procedures.
Method
The paper analyzes GAN updates as dynamical systems using stochastic-approximation ODE methods, Jacobian stability conditions, and gradient-based regularization.
Results
Traditional GAN equilibria are locally asymptotically stable under suitable conditions, WGANs can have non-convergent limit cycles, and the proposed regularizer stabilizes both formulations.
Takeaways & Limitations
Local stability can hold despite non-convex-concave GAN objectives, while gradient-based regularization shows practical promise for faster convergence and averting mode collapse.
Takeaways & Limitations
The analysis is local near equilibrium points and does not establish that equilibria necessarily exist.

Abstract

from arXiv · show

Despite the growing prominence of generative adversarial networks (GANs), optimization in GANs is still a poorly understood topic. In this paper, we analyze the "gradient descent" form of GAN optimization i.e., the natural setting where we simultaneously take small gradient steps in both generator and discriminator parameters. We show that even though GAN optimization does not correspond to a convex-concave game (even for simple parameterizations), under proper conditions, equilibrium points of this optimization procedure are still \emph{locally asymptotically stable} for the traditional GAN formulation. On the other hand, we show that the recently proposed Wasserstein GAN can have non-convergent limit cycles near equilibrium. Motivated by this stability analysis, we propose an additional regularization term for gradient descent GAN updates, which \emph{is} able to guarantee local stability for both the WGAN and the traditional GAN, and also shows practical promise in speeding up convergence and addressing mode collapse.

1 Introduction

The paper studies simultaneous gradient updates to GAN generators and discriminators, addressing optimization outside the convex-concave setting. It establishes local stability for traditional GANs and proposes regularization that stabilizes both traditional GANs and WGANs.

Motivation: GAN optimization remains theoretically and practically challenging because standard convergence analysis assumes convex-concavity, function-space updates, and fully optimized discriminators.These assumptions do not match simultaneous parameter updates used in practice.
Approach: The paper analyzes gradient descent GAN optimization, where generator and discriminator parameters receive simultaneous stochastic gradient updates without inner or outer optimization loops.Neither network is assumed to be optimized to convergence before the other is updated.
Stability results: Despite not forming a convex-concave game even with simple linear representations, GAN dynamics can be locally exponentially stable under suitable generator and discriminator representational conditions.Updates converge exponentially to an equilibrium within some region around that equilibrium.
Stability results: The traditional GAN can satisfy the stability conditions, whereas WGANs can exhibit non-convergent limit cycles under gradient descent.This distinguishes local behavior of the two formulations near equilibrium.
Regularization: A regularizer based on the norm of the discriminator gradient yields locally exponentially stable equilibria for all GAN classes, including WGANs.Preliminary experiments indicate faster convergence and reduced mode collapse.

2 Background and related work

The paper situates its analysis among theoretical, practical, and dynamical-systems work on GAN optimization. It studies local stability through deterministic differential equations and Jacobian conditions, extending beyond convex-concave game analysis.

GAN optimization and theory: Related GAN theory studies instability from disjoint data and generated supports, whereas this paper focuses on local stability of parameter-update dynamics.The paper presents these questions as complementary.
GAN optimization and theory: Practical GAN research has proposed new objectives, architectures, and optimization heuristics to improve stability and address mode collapse.The paper's regularizer is positioned within this broader line of work.
GAN optimization and theory: The proposed parameter-gradient regularizer differs from related gradient-norm regularization that computes gradients with respect to datapoints.The comparison distinguishes the variable with respect to which the gradient norm is calculated.
Stochastic approximation and nonlinear systems: The analysis uses the ODE method from stochastic approximation and nonlinear differential equations to relate stochastic updates to deterministic continuous-time dynamics.Under standard conditions, the stochastic process converges to an equilibrium of the associated ODE.
Stochastic approximation and nonlinear systems: Local stability is assessed around equilibria using Jacobian linearization, with Hurwitz Jacobians implying convergence in a non-empty neighborhood.The paper also considers zero eigenvalues associated with subspaces of equilibria.
Stochastic approximation and nonlinear systems: A Lyapunov analysis identifies a non-negative energy function that decreases over time, providing additional insight into convergence dynamics.The energy measures distance from an equilibrium or equilibrium subspace.

3 GAN optimization dynamics

Gradient descent GAN updates can be locally exponentially stable around suitable equilibria despite non-convex-concave dynamics. WGAN updates may cycle, while a discriminator-gradient regularizer restores local stability under stated conditions.

Local stability: GAN gradient descent updates are locally exponentially stable around good equilibria under suitable representational and smoothness conditions.Good equilibria match the data distribution with a discriminator that is zero on its support; nearby bad equilibria are implicitly excluded.
Local stability: The stability result uses the dynamical-systems meaning of equilibrium: any point where the gradient update is zero.This differs from the usual GAN usage referring to a Nash equilibrium of the min-max problem.
Why stability is non-trivial: GAN objectives can be concave in both discriminator and generator parameters, making the minimization side difficult even near equilibrium.This occurs for a large part of discriminator space and is not limited to the simple linear parameterization considered in the main example.
Assumptions: The realizable stability assumption requires the generator to match the true distribution and the representations to exclude nearby bad equilibria.A non-realizable alternative removes the distribution-matching requirement but restricts the discriminator to be linear in its parameters; both retain the all-zero discriminator condition.
WGAN behavior: Without regularization, WGAN gradient descent can cycle indefinitely around an equilibrium rather than converge.The paper attributes this contrast to the WGAN loss lacking the strict concavity required by the traditional stability argument.
Regularization: A penalty based on the discriminator-gradient norm preserves equilibrium parameters and makes the updates locally exponentially stable for WGANs and other concave GAN objectives.For the regularized WGAN, Theorem 3.2 states local exponential stability under appropriate conditions; for the general result, η must satisfy η < 1/(2λmax(−JDD)).

4 Experimental results

The experiments suggest gradient regularization can spread generated samples across modes more quickly than 10-unrolled GANs and avoid the gradual redistribution seen in conventional updates.

On a mixture of eight Gaussians, gradient regularization quickly spreads generated points across all modes instead of exploring only a few initially.Conventional GAN updates are reported to enter mode collapse in this setup.

5 Conclusion

The paper establishes local stability for traditional GAN optimization under suitable conditions, finds instability for WGANs, and proposes a gradient-based regularizer that stabilizes both in practice. The conclusions remain local, do not establish equilibrium existence, and include an implementation link.

The analysis shows traditional GAN optimization can be locally asymptotically stable under proper conditions, while WGANs can exhibit non-convergent limit cycles.
The analysis is local near equilibrium, does not establish that equilibria exist, and stronger global convergence is only described as plausible for simple settings.
A gradient-based regularizer stabilizes both traditional GANs and WGANs and can improve convergence speed in practice.An implementation is provided at the authors’ repository.
Figure 1 compares gradient-regularized GANs with 10-unrolled GANs, while Figures 2 and 4 compare gradient-regularized and traditional GAN variants.
Figure 3 presents streamline plots around equilibrium (0, 1) for conventional GAN and WGAN updates across η values.

A Preliminaries

The preliminaries define local stability and develop linearization and Lyapunov tools for systems with multiple nearby equilibria. An extension shows that stability can follow from contraction in non-equilibrium directions while equilibrium directions remain neutral.

Stability definitions: Asymptotic and exponential stability strengthen stability by requiring convergence, with exponential stability imposing an exponential convergence rate.The definitions distinguish stability, asymptotic stability, and exponential stability through increasingly strong convergence conditions.
Stability definitions: Local stability requires trajectories initialized sufficiently near an equilibrium to remain within any prescribed surrounding neighborhood.A stable system need not converge; it may orbit indefinitely within the neighborhood.
Linearization and Lyapunov analysis: Lyapunov analysis uses a positive function whose time derivative decreases near equilibrium; LaSalle’s extension permits instantaneous constancy when only equilibria remain invariant.This handles systems where a quadratic Lyapunov function is not strictly decreasing everywhere.
Linearization and Lyapunov analysis: Linearization reduces local nonlinear stability analysis to the Jacobian: a Hurwitz Jacobian, whose eigenvalues have negative real parts, implies local exponential stability.The nonlinear remainder is quadratic near equilibrium, so its effect is dominated by the linearized system’s quadratic Lyapunov decrease.
Multiple equilibria: For parameters θ and γ with nearby equilibria along γ, it suffices to show that the Jacobian in θ is Hurwitz.A smaller neighborhood is identified so γ remains controlled, keeping trajectories inside the region where θ contracts exponentially.
Multiple equilibria: Within that smaller neighborhood, θ converges exponentially to zero while γ approaches a bounded value, yielding convergence to an equilibrium.The trajectory remains in the stability region until convergence.

B GANs are not concave-convex near equilibrium

The paper demonstrates that GAN objectives are not generally concave-convex near equilibrium, even for simple parameterizations. This failure occurs arbitrarily close to equilibrium in both ordinary GAN settings and WGAN-related cases.

Counterexamples to concavity: A one-dimensional distribution-learning example is used to test local curvature in generator and discriminator parameter spaces.The analysis considers discriminator and generator dimensions dD,dG ≥ 1 with an arbitrary target distribution and standard-normal latent distribution.
Counterexamples to concavity: When f′′(x) < 0, the objective is concave in most generator parameters arbitrarily close to the all-zero discriminator parameter.This follows by choosing one nonzero discriminator coordinate while setting the others to zero.
Counterexamples to concavity: When f′′(x) = 0 and dD > 2, the objective is likewise concave in generator parameters in the region w2 > 0 if f′(x) > 0.The condition f′(x) > 0 holds for WGANs, so this establishes the relevant local curvature behavior there.

C Local exponential stability of GANs

The paper proves local exponential stability for gradient-descent GAN dynamics under representational and curvature conditions, despite nearby equilibrium subspaces. The proof projects away equilibrium directions and shows the remaining Jacobian is Hurwitz.

Jacobian structure: The analysis derives the GAN update Jacobian and identifies its discriminator, generator, and cross-parameter blocks.The generator-generator block is zero under the support-preservation assumption, while the lower cross-block is the negative transpose of the upper block.
Projected stability analysis: Under the stated assumptions, the projected Jacobian is Hurwitz, which yields local exponential stability around an equilibrium and nearby equilibria.The proof uses the multiple-equilibria linearization theorem and bounds convergence through the Jacobian eigenvalues.
Equilibrium subspaces: Null spaces of the discriminator and generator curvature matrices correspond to local subspaces of equilibrium discriminators and generators.Perturbations along these null spaces preserve zero updates, so nearby points remain equilibria.
Projected stability analysis: The system is rotated and projected onto the space orthogonal to the equilibrium subspace, where the Jacobian block has the required rank and curvature properties.The projected construction removes zero-eigenvalue equilibrium directions while retaining the dynamically contracting directions.
Convergence rate: The convergence rate is governed by Jacobian eigenvalues with strictly negative real parts, with separate bounds for real and non-real eigenvalues.The paper provides explicit upper bounds involving f′(0), f′′(0), and curvature-matrix eigenvalues.
Convergence rate: A squared-distance Lyapunov function to the equilibrium subspace decreases or is instantaneously constant away from equilibrium.This supplies a Lyapunov-based interpretation of convergence for the projected linearized dynamics.

C.1 Realizable case with a relaxed assumption

The relaxed realizable-case analysis allows generator supports to change under perturbation, replacing support preservation with discriminator conditions. Under these conditions, local exponential stability still holds, while violating the second condition can destroy stability.

Relaxed assumptions: The relaxed assumption permits perturbed generators to have different supports, provided discriminator behavior satisfies explicit local conditions.The equilibrium discriminator must vanish on the support of nearby generator perturbations, and the relevant flat directions must align.
Relaxed assumptions: The first relaxed condition requires the equilibrium discriminator to be zero beyond the equilibrium generator’s support, covering supports of nearby generators.This prevents discriminator values from inducing generator updates over the expanded local support.
Relaxed assumptions: The second condition requires shared flat directions so discriminator perturbations along those directions remain zero on nearby generator supports.This preserves the equilibrium-discriminator property throughout the local perturbation region.
Examples: A uniform target on [−1,1], interval-supported generator, and polynomial discriminator provide an example satisfying the relaxed conditions.In this example, the relevant Hessian is positive definite at equilibrium, satisfying the second condition.
Examples: A point-mass target and point-mass generator with a linear discriminator violate the second condition despite having a unique equilibrium.The first Hessian has a full null space, whereas the second Hessian is nonzero arbitrarily close to equilibrium.
Stability conclusion: When the relaxed conditions hold, the modified proof again establishes local exponential stability.The argument extends the equilibrium-subspace and Jacobian results while retaining the original proof’s core structure.

C.2 The non-realizable case

In the non-realizable setting, the paper establishes local stability under a linear-parameter discriminator, even when the generator space cannot represent the true distribution. Nearby perturbations along specified null spaces remain equilibria under the stated assumptions.

Setup: The analysis considers a generator space that may exclude the true distribution, with a linear discriminator and a generator equilibrium paired with a zero-on-support discriminator.The representation must match true and generated means, and a richer representation may eliminate such an equilibrium generator.
Jacobian analysis: The non-realizable proof derives the Jacobian at equilibrium and relies on the discriminator’s linearity to eliminate discriminator-parameter Hessian terms.The remaining Jacobian terms match the realizable case because the equilibrium discriminator is identically zero.
Conclusion: The resulting equilibrium therefore lies in a subspace of equilibria, and the main stability theorem follows under the non-realizable assumptions.The proof connects the null-space lemmas to the main theorem through the same argument used earlier.
Equilibrium structure: Perturbations along Null(KDD) and Null(KDG) preserve the discriminator and generator equilibrium requirements, so the system remains at equilibrium along those directions.The relevant objectives are locally constant along the corresponding null-space directions.

D Linear Quadratic GAN – Gaussian example

The linear-quadratic Gaussian example provides a concrete equilibrium where a linear generator matches the target Gaussian and a quadratic discriminator is zero. Under curvature and rank conditions, this equilibrium is locally exponentially stable.

Model: The LQ example uses a quadratic discriminator and linear generator to learn N(µ, Σ) from standard-normal latent variables.The target covariance is positive definite, and the generator is G(z)=Az+b.
Equilibrium: At A=Σ1/2, b=µ, W2=0, and w1=0, all updates vanish because the generated distribution equals N(µ, Σ).This identifies the parameter setting as an equilibrium corresponding to the true distribution.
Stability conditions: The Jacobian is Hurwitz when f′′(0)<0, f′(0)≠0, and JDG has full column rank.JDD is negative definite under the curvature condition, while the rank condition controls the coupled Jacobian.

E WGANs are not necessarily asymptotically stable

For the LQ WGAN Gaussian system, arbitrarily near-equilibrium initializations can follow periodic orbits instead of converging. The construction extends from one dimension to higher dimensions through invariant eigenspaces.

Counterexample: The paper studies an LQ WGAN learning a zero-mean Gaussian and identifies nearby initializations that periodically return to their starting point.The one-dimensional construction is later extended to the multidimensional Gaussian case.
One-dimensional system: The system has equilibria at w2=0, w1=0, a=±σ, and b=0; with w1=b=0, the dynamics reduce to w2 and a.The reduced subsystem is sufficient for analyzing stability near a=σ.
One-dimensional system: When w2>0, the squared distance from the equilibrium never decreases, so convergence would require the trajectory to leave this half-space.The distance derivative is 2w2(a−σ)^2.
Limit cycles: For arbitrarily close initializations, the trajectory mirrors its earlier path across w2=0 and returns to its initial state after 2T, forming a perpetual loop.The symmetry gives (w2(2T),a(2T))=(0,a(0)), ruling out asymptotic stability.
Higher dimensions: The higher-dimensional system preserves the covariance eigenspace, decomposing into independent one-dimensional systems whose orbiting components produce the same instability.The resulting equilibrium is not asymptotically stable.

F.1 Local stability of gradient-regularized GANs

The proposed gradient penalty modifies the equilibrium Jacobian so that its relevant diagonal blocks become negative definite. Under a step-size condition, it yields local exponential stability for conventional GANs and, under analogous conditions, for WGANs.

Main result: Theorem 3.2 guarantees local exponential stability for conventional GANs under that step-size bound and establishes analogous WGAN stability for any η under appropriate conditions.For WGANs, the convergence rate is governed by Jacobian eigenvalues with strictly negative real parts.
Regularized update: The regularizer adds −∇θG||∇θDV(DθD,GθG)||^2 to the generator update.At equilibrium, the added term itself is zero, but its Jacobian changes the local dynamics.
Jacobian analysis: The regularized Jacobian differs from the original one in the rows corresponding to generator parameters.The discriminator-related blocks remain those of the original Jacobian.
Stability proof: A quadratic Lyapunov function proves exponential stability after projecting away the subspace of equilibria when η < 1/(2λmax(−JDD)).The proof requires the Lyapunov matrix to be positive definite and its derivative to be strictly negative away from equilibrium.
Local stability: The modified Jacobian has negative-definite block-diagonal terms, with the relevant matrix shown to be negative definite.This is the local-stability mechanism supplied by the regularizer.

F.1.1 Exponential stability of gradient-regularized WGAN

Under WGAN-specific assumptions, unregularized updates need not be asymptotically stable, whereas gradient regularization yields locally exponentially stable equilibria. The analysis uses projected Jacobians and Lyapunov-style arguments to establish stability around equilibrium subspaces.

Even under the stated WGAN conditions, the unregularized update does not ensure asymptotic stability.
The WGAN analysis relaxes one discriminator assumption because linearity permits equilibrium discriminators that are constant, rather than identically zero, on the data support.
The regularized system has locally exponentially stable equilibria under the stated assumptions, including for WGANs.
The proof projects the linearized dynamics onto a subspace orthogonal to equilibrium directions and analyzes the resulting Jacobian.
A squared-distance Lyapunov function is non-increasing and decreases away from equilibrium, supporting convergence of the projected system.

G Eigenvalue bounds

The eigenvalue analysis establishes negativity of the real parts for Jacobian eigenvalues under positive-definiteness and rank conditions, while deriving bounds through quadratic inequalities and eigenvalue decompositions. The appendix also notes that some bounds apply only to complex eigenvalues.

Positive definiteness of Q and full column rank of P imply that the associated structured matrices are Hurwitz.
If Q is symmetric positive definite and P has full column rank, every eigenvalue λ of the relevant Jacobian satisfies Re(λ) < 0.
The proof analyzes real and imaginary parts of a normalized eigenvector equation to derive constraints on eigenvalue components.
Eigenvalue upper bounds are obtained by converting the analysis into a quadratic in the real eigenvalue component and examining its discriminant and monotonicity.
The appendix provides only partial upper bounds for the related structured matrices, specifically for complex eigenvalues.