PaperlayerView exact revision on arXiv

Source-linked AI summary

A Variational Inequality Perspective on Generative Adversarial Networks

Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, Simon Lacoste-Julien

arXiv:1802.10551v5cs.LGmath.OCstat.ML

TL;DR

GAN training is difficult because basic SGD can oscillate and fail on two-player games. The paper casts GAN optimization as variational inequalities and develops averaging, extrapolation, and extrapolation from the past for SGD and Adam. These techniques improve reported CIFAR-10 GAN metrics over a prior baseline, while the theory relies on monotonicity, boundedness, and related assumptions.

  • Problem

    Basic SGD can oscillate and fail to converge on GAN games, motivating optimization methods with stronger convergence guarantees.

  • Method

    The paper casts GAN games as variational inequalities and extends averaging, extrapolation, and extrapolation from the past to SGD and Adam.

  • Results

    The proposed techniques improved inception score and Fréchet inception distance by 4-6% over Miyato et al. (2018) on CIFAR-10 using WGAN-GP and a ResNet generator.

  • Takeaways & Limitations

    The techniques provide practical GAN optimizers that are orthogonal to GAN objective and architecture design and yielded better-trained models in experiments.

  • Takeaways & Limitations

    The stochastic analysis assumes bounded expected squared gradient norms and monotonicity or related conditions that may not hold for unconstrained or general neural-network GANs.

Abstract

from arXiv · show

Generative adversarial networks (GANs) form a generative modeling approach known for producing appealing samples, but they are notably difficult to train. One common way to tackle this issue has been to propose new formulations of the GAN objective. Yet, surprisingly few studies have looked at optimization methods designed for this adversarial training. In this work, we cast GAN optimization problems in the general variational inequality framework. Tapping into the mathematical programming literature, we counter some common misconceptions about the difficulties of saddle point optimization and propose to extend techniques designed for variational inequalities to the training of GANs. We apply averaging, extrapolation and a computationally cheaper variant that we call extrapolation from the past to the stochastic gradient method (SGD) and Adam.

1 INTRODUCTION

The paper frames GAN training instability as an optimization problem requiring methods designed for adversarial games rather than direct applications of basic SGD. It casts GAN games in the variational inequality framework and develops averaging, extrapolation, and extrapolation from the past for SGD and Adam.

  • GANs produce realistic samples but are difficult to train and often behave unstably.
  • Basic SGD can oscillate and fail to converge on some games, even without stochasticity.The paper argues that this reflects a mismatch between minimization methods and two-player game optimization.
  • The paper casts multi-player games, including minimax and non-zero-sum GANs, as variational inequality problems.This connects GAN optimization to techniques from the mathematical programming literature.
  • It extends SGD and Adam with averaging, extrapolation, and extrapolation from the past.Extrapolation from the past requires one gradient computation per update, unlike extrapolation, which computes the gradient twice.
  • The proposed techniques improve inception score and Fréchet inception distance by 4-6% over Miyato et al. (2018) on CIFAR-10 with WGAN-GP and a ResNet generator.

2 GAN OPTIMIZATION AS A VARIATIONAL INEQUALITY PROBLEM

GAN objectives define coupled two-player optimization problems, including zero-sum and non-zero-sum formulations. The paper represents these games as variational inequalities, while noting that non-convexity and equilibrium assumptions constrain the theory.

  • GAN training is a game in which a generator produces samples and a discriminator classifies them as real or fake.The generator ultimately seeks samples realistic enough to fool the discriminator.
  • GAN formulations include zero-sum minimax objectives and non-zero-sum objectives with different costs for the two players.The non-saturating formulation is an example of a non-zero-sum game.
  • The two players’ optimization problems are coupled and must be considered jointly.
  • Standard GAN objectives are non-convex, so pure equilibria may not exist and their existence is not well understood.
  • The paper studies GAN games through stationary conditions and variational inequalities, which generalize stationary conditions for two-player games.Both saddle-point and non-zero-sum GAN optimization can be cast as variational inequality problems.

3 OPTIMIZATION OF VARIATIONAL INEQUALITIES (BATCH SETTING)

In the batch setting, the paper analyzes averaging and extrapolation methods for variational inequalities using bilinear game dynamics. It introduces extrapolation from the past as a cheaper stabilizing variant that reuses a gradient.

  • The paper studies exact-gradient variational inequality methods in the batch setting before extending them to stochastic methods.The main techniques are averaging, extrapolation, and extrapolation from the past.
  • Basic gradient updates can oscillate on bilinear operators, while averaged iterates converge for bounded monotone operators and extragradient can converge at O(1/t).Extragradient uses a lookahead step to compute a more stable update direction.
  • Averaging: Averaging can be implemented online through recursive updates, including uniform and geometric averaging.Geometric averaging is also called exponential moving averaging.
  • Averaging: For alternating bilinear updates, averaging can yield convergence, whereas averaging fails when simultaneous iterates diverge.The analysis contrasts bounded but non-convergent alternating iterates with geometrically diverging simultaneous iterates.
  • Extrapolation: Extrapolation evaluates the gradient at a lookahead point, stabilizing dynamics and approximating a more stable implicit method.For the toy bilinear problem, extrapolation has convergence behavior similar to the implicit method while remaining more practical.
  • Extrapolation from the past: Extrapolation from the past reuses a stored extrapolated gradient, requiring one gradient computation per update instead of two.The paper proves its convergence for strongly monotone operators and in the stochastic variational inequality setting.

4 OPTIMIZATION OF VIP WITH STOCHASTIC GRADIENTS

The paper extends stochastic gradient methods for variational inequalities with averaging, extrapolation, and extrapolation from the past, analyzing their convergence under explicit assumptions.

  • Stochastic setting: Stochastic VIP optimization uses unbiased mini-batch gradient estimates rather than exact operators, motivating stochastic extensions of batch techniques.The stochastic operator is defined as an unbiased estimate of the expected gradient, as in GAN mini-batch training.
  • Algorithms: The proposed methods are AvgSGD, AvgExtraSGD, AvgPastExtraSGD, and a mini-batch-reuse variant, all returning averaged iterates.AvgExtraSGD adds extrapolation, while AvgPastExtraSGD uses extrapolation from the past.
  • Assumptions: The convergence analysis uses monotonicity, compact convex constraints, and either bounded variance or bounded expected squared stochastic-gradient norm.The stronger squared-norm assumption is mainly reasonable for bounded constraint sets and fails for unconstrained bilinear objectives.
  • Convergence: SGD with extrapolation improves convergence guarantees and avoids the strong bounded-norm assumption required for averaged SGD.Its variance term is smaller than the corresponding term for averaged SGD when σ ≪ M.
  • Convergence: Extrapolation from the past achieves a convergence bound similar to extrapolation while making each iteration computationally half as costly.It uses one gradient computation per update instead of extrapolation's two.
  • Scope: Convexity is necessary but not sufficient for operator monotonicity, and general neural-network GANs produce non-monotone VIPs.The convergence results therefore apply under a restricted monotone setting rather than to all neural-network GANs.

5 COMBINING THE TECHNIQUES WITH ESTABLISHED ALGORITHMS

The paper combines variational-inequality techniques with established SGD and Adam optimizers, including an Adam implementation with extrapolation and extrapolation from the past.

  • Combining techniques: The proposed averaging and extrapolation techniques are combined with Adam and SGD, two standard optimizers for training deep neural networks.The combinations are intended for practical use with existing optimization algorithms.
  • Adam variants: Extra-Adam is the proposed Adam variant that incorporates an extrapolation step.The algorithm supports standard extrapolation and extrapolation from the past as alternative update choices.
  • Adam variants: The Adam procedure updates first- and second-moment estimates from stochastic gradients before projecting the parameter update onto the constraint set.The implementation can load a previously saved stochastic gradient for extrapolation from the past.

6 RELATED WORK

Related work includes extragradient methods, GAN-specific optimization and stabilization approaches, and prior practical uses of averaging; this paper provides a variational-inequality motivation for these techniques.

  • Variational-inequality optimization: Extragradient is a standard variational-inequality algorithm with established deterministic and stochastic extensions.The cited literature analyzes stochastic extragradient methods, including settings with bounded constraints.
  • GAN optimization: Prior GAN stabilization methods include alternative game formulations, gradient penalties, momentum-based optimization, objective dualization, and gradient-norm regularization.These approaches address GAN training through objectives or optimizer modifications.
  • Averaging: Earlier work briefly used geometric averaging for GANs, whereas this paper formally motivates averaging through the variational-inequality perspective.The paper also discusses subsequent work that explored averaging empirically in more depth.

7 EXPERIMENTS

Experiments test the proposed optimization techniques on a stochastic bilinear problem and CIFAR10 GANs, finding strong performance for averaged extrapolation methods, especially ExtraAdam.

  • 7 EXPERIMENTS: The experiments aim to improve optimization of standard GANs without introducing new objectives, architectures, or state-of-the-art architectural changes.The techniques are presented as orthogonal to GAN objective design and architectural choices.
  • 7.1 BILINEAR SADDLE POINT (STOCHASTIC): On a constrained finite-sum bilinear problem with n = 10^3 and d = 10^3, AvgAltSGD1 and AvgPastExtraSGD perform best.The operator in this task is monotone and the domain is [−1, 1]^d.
  • 7.2 WGAN AND WGAN-GP ON CIFAR10: The GAN experiments train DCGAN and ResNet architectures with WGAN and WGAN-GP objectives on CIFAR10, evaluating inception score and FID.Inception score is computed on 50,000 samples, with five random seeds and 500,000 generator updates.
  • 7.2 WGAN AND WGAN-GP ON CIFAR10: ExtraAdam converges faster than the Adam baselines on WGAN-GP, while extrapolation-based methods are less sensitive to learning-rate tuning.The figure reports means and standard deviations over five runs for the inception score.
  • 7.2 WGAN AND WGAN-GP ON CIFAR10: Extrapolation and averaging consistently improve Adam baselines, and ExtraAdam with averaging outperforms the other methods on both GAN tasks.The comparison uses best-performing learning rates and reports improvements over prior CIFAR10 IS and FID values.

8 CONCLUSION

The paper reframes GAN objectives as variational inequality problems and develops principled optimization techniques for training them. These techniques produce practical algorithms and improve trained models while remaining orthogonal to GAN architectures and objectives.

  • GAN objectives are addressed within the variational inequality framework.
  • The paper develops practical optimization algorithms for a wide range of GAN objectives, including non-zero-sum games and constrained projections.
  • The techniques are orthogonal to designing new GAN architectures and objectives.
  • Experiments verify that the techniques can yield better-trained models and improve the previous state of the art.

B.1 PROOF OF PROPOSITION 1

The proof contrasts simultaneous and alternating updates for the analyzed game. Simultaneous iterates diverge geometrically, whereas alternating iterates remain bounded but do not converge directly to equilibrium.

  • The simultaneous iterates diverge geometrically.
  • The alternating iterates are bounded but do not converge to zero.
  • The proof analyzes the alternating update rule through its explicit two-step recurrence.

B.2 IMPLICIT AND EXTRAPOLATION METHOD

The paper analyzes implicit and extrapolation methods for bilinear variational inequalities, showing that extrapolation approximates implicit updates while remaining more practical. A reduction through singular-value coordinates extends the analysis to general unconstrained bilinear objectives.

  • B.3 GENERALIZATION TO GENERAL UNCONSTRAINED BILINEAR OBJECTIVE: A singular-value decomposition reduces a broad class of first-order methods on the general bilinear problem to independent one-dimensional problems.Each coordinate uses the same method with a step size rescaled by the corresponding singular value σ_i.
  • B.3 GENERALIZATION TO GENERAL UNCONSTRAINED BILINEAR OBJECTIVE: The simultaneous iterates diverge geometrically, while alternating iterates remain bounded but do not converge to the equilibrium.Their uniform averages have different behaviors: simultaneous-update averages diverge, whereas alternating-update averages converge to zero.
  • B.3 GENERALIZATION TO GENERAL UNCONSTRAINED BILINEAR OBJECTIVE: Extrapolation and implicit updates converge linearly on unconstrained bilinear problems for 0 < η < 1/σmax(A).For extrapolation, η = 1/(2σmax(A)) yields N^2_{t+1} ≤ (1 − 1/(8κ))^t N^2_0.
  • B.2 IMPLICIT AND EXTRAPOLATION METHOD: Extrapolation from the past is analyzed under strong monotonicity and Lipschitz continuity, with updates using η = 1/(4L) that converge linearly.The method is presented as a projected extrapolation technique for variational inequalities.
  • C.2 ON THE IMPORTANCE OF THE MERIT FUNCTION: The primal-dual gap is not generally a valid merit function without strong convex-concavity, because it can vanish away from an optimum.For L(θ, ϕ) = θ · ϕ, the gap is zero for every θ and ϕ despite only the origin being an equilibrium.

D ANOTHER WAY OF IMPLEMENTING EXTRAPOLATION TO SGD

The paper introduces a re-used-mini-batch implementation of stochastic extrapolation that correlates the extrapolation and update estimators. It provides convergence properties under stochastic assumptions, while requiring bounded iterates for its analysis.

  • D ANOTHER WAY OF IMPLEMENTING EXTRAPOLATION TO SGD: ReExtraSGD reuses the extrapolation mini-batch for the current update to improve the correction of stochastic oscillations.The intended mechanism is correlation between the two gradient estimators.
  • D ANOTHER WAY OF IMPLEMENTING EXTRAPOLATION TO SGD: Under Assumptions 1 and 4, ReExtraSGD with constant η ≤ 1/√(2L) has stated convergence properties for any T ≥ 1.The supplied passage introduces these properties but does not include the resulting bounds.
  • D ANOTHER WAY OF IMPLEMENTING EXTRAPOLATION TO SGD: The analysis assumes that the algorithm’s iterate sequence is bounded, an assumption described as strong.The paper notes that this assumption has also been made in prior work.
  • D ANOTHER WAY OF IMPLEMENTING EXTRAPOLATION TO SGD: Compared with the cited SGD-with-prediction analysis, the resulting variance term includes an additional σ^2/2 under the shared convergence certificate.Using the paper’s notation, the resulting variance is summarized as (M^2(1 + L) + σ^2)/2.

F.1 PROOF OF THM. 2

The paper establishes convergence results for stochastic variational-inequality methods and evaluates their GAN adaptations on toy and CIFAR-10 experiments.

  • Theorem 2 gives a convergence-rate result for Algorithm 1 under the stated assumptions with a constant step-size.
  • Under Lipschitzness and the stated assumptions, Algorithm 2 has a convergence guarantee for a sufficiently small constant step-size.
  • AvgPastExtraSGD also has a convergence guarantee under the stated assumptions and a sufficiently small constant step-size.
  • Algorithm 5 is analyzed under Lipschitzness and related assumptions, with convergence properties for constant step-size η ≤ 1/(√2L).
  • In the toy non-convex GAN, some methods fail depending on initialization, while extrapolation performs better than the other methods.
  • On CIFAR-10, extrapolation and averaging improve Adam baselines, with ExtraAdam outperforming other methods on both evaluated tasks.

G.4 COMPARISON OF THE METHODS WITH THE SAME LEARNING RATE

With the same learning rate, ExtraAdam is less sensitive to learning-rate choice, while AvgExtraAdam maintains more stable sample quality across step-sizes.

  • ExtraAdam is less sensitive to learning-rate choice and can use higher learning rates with less degradation.
  • For AvgExtraAdam, sample quality changes little across step-sizes, whereas AltAdam5 is sensitive to step-size tuning.
  • Robustness to step-size tuning is presented as useful because it can reduce time spent tuning other learning-procedure hyperparameters.
  • Uniform averaging improves inception score but produces somewhat blurrier samples.
  • FID is more sensitive to blurriness, explaining why averaging can perform worse on FID despite improving inception score.
Loading 1802.10551v5…