Source-linked AI summary
The Numerics of GANs
Lars Mescheder, Sebastian Nowozin, Andreas Geiger
TL;DR
GAN training is difficult because simultaneous gradient ascent encounters problematic Jacobian eigenvalues. The paper analyzes these dynamics through smooth two-player games and designs Consensus Optimization to improve convergence. The method provides stable training across many GAN architectures, including architectures known to be hard to train.
Problem
GANs are notoriously hard to train, and the paper addresses the numerical difficulties of finding local Nash-equilibria in their smooth two-player-game formulation.
Method
The paper analyzes the gradient vector field and Jacobian spectrum of simultaneous gradient ascent, then designs Consensus Optimization to address the identified problems.
Results
Consensus Optimization leads to stable training on many GAN architectures, including architectures known to be hard to train.
Takeaways & Limitations
The method separates computation of equilibria from strategies that establish their existence and can be combined with different GAN architectures and divergence measures.
Takeaways & Limitations
High regularization can stabilize formerly unstable stationary points but may lead to poor solutions, and deeper architectures become less stable without gradient-scale rescaling.
Abstract
from arXiv · showhide
In this paper, we analyze the numerics of common algorithms for training Generative Adversarial Networks (GANs). Using the formalism of smooth two-player games we analyze the associated gradient vector field of GAN training objectives. Our findings suggest that the convergence of current algorithms suffers due to two factors: i) presence of eigenvalues of the Jacobian of the gradient vector field with zero real-part, and ii) eigenvalues with big imaginary part. Using these findings, we design a new algorithm that overcomes some of these limitations and has better convergence properties. Experimentally, we demonstrate its superiority on training common GAN architectures and show convergence on GAN architectures that are known to be notoriously hard to train.
1 Introduction
GANs are powerful but notoriously difficult to train. This paper studies their training as smooth two-player games, identifies numerical obstacles in simultaneous gradient ascent, and proposes Consensus Optimization to improve convergence.
- GANs have been applied successfully to diverse tasks but remain notoriously hard to train.
- The paper analyzes local Nash-equilibrium computation in smooth two-player games using the gradient vector field associated with GAN objectives.
- Its convergence is hindered by Jacobian eigenvalues with zero real-part or large imaginary parts.
- Consensus Optimization is designed to address these numerical difficulties and empirically enables stable training across GAN architectures and divergence measures.
- Simultaneous gradient ascent is the standard method, updating both players’ parameters concurrently by gradient ascent.
3 Convergence Theory
The convergence analysis explains why simultaneous gradient ascent can be unstable or slow. Its step-size constraints depend on the Jacobian spectrum, especially eigenvalues near the imaginary axis or with large imaginary-to-real-part ratios.
- Simultaneous gradient ascent can fail because the gradient-vector-field Jacobian has eigenvalues with zero real-part or large imaginary part.
- When convergence holds, fixed-point iterations converge at least linearly, with error O(|λmax|^k).
- If all Jacobian eigenvalues have negative real-part, the algorithm is locally convergent for sufficiently small step size h.
- The maximum allowable step size depends on the largest real part and q = |ℑ(λ)/ℜ(λ)| across eigenvalues.
- As q grows, the required step size scales as O(q^-2), while eigenvalues near 1 cause very slow convergence.
4 Consensus Optimization
Consensus Optimization modifies the game’s gradient field with a regularizer that encourages agreement between players. The resulting method improves the Jacobian spectrum and is locally convergent under stated assumptions, though practical limitations remain.
- Derivation: The method seeks stationary points by modifying the original vector field while minimizing L(x) = 1/2∥v(x)∥2.
- Derivation: Consensus Optimization subtracts γL from both players’ utility functions, producing a regularizer that encourages agreement.
- Practical considerations: Naive mini-batch implementation produces biased estimates of L(x), although the bias decreases linearly with batch size and can be debiased.
- Convergence: The algorithm preserves stationary points when A(x) is invertible, and for zero-sum games it converges locally to a local Nash-equilibrium for sufficiently small h.
- Convergence: The method addresses approximately zero real-part eigenvalues and can make the imaginary-to-real-part quotient arbitrarily small with an appropriate γ.
5 Experiments
Experiments compare consensus optimization with simultaneous or alternating gradient ascent on synthetic and image datasets. Consensus optimization converges smoothly on the Gaussian mixture, stabilizes GAN losses, and trains architectures known to be difficult.
- Mixture of Gaussians: The Gaussian-mixture experiment evaluates densities after 0, 5000, 10000, and 20000 iterations against the target density.
- Mixture of Gaussians: On a circular mixture of eight Gaussians, SimGA jumps between modes and fails to converge, whereas consensus optimization smoothly reaches the target distribution.Both methods use RMSProp with learning rate 10^-4 for 20,000 steps; consensus optimization uses γ = 10.
- CIFAR-10 and CelebA: On CIFAR-10, consensus optimization makes generator and discriminator losses almost constant, while alternating gradient ascent produces highly fluctuating losses.The inception scores use 6400 samples; convergence rates are comparable and consensus optimization achieves slightly better end results.
- Mixture of Gaussians: Near the Nash-equilibrium, unmodified Jacobian eigenvalues lie close to the imaginary axis, while consensus regularization moves them to the left.The comparison uses the Jacobians of v(x) and w(x) = v(x) −γ∇L(x).
- CIFAR-10 and CelebA: Consensus optimization successfully trains DC-GAN-like models on CIFAR-10 and CelebA without batch normalization, despite these architectures being hard to optimize with gradient ascent.The CelebA model additionally uses constant filter counts and extra RESNET layers.
6 Discussion
The discussion places consensus optimization alongside implicit Euler and second-order interpretations while distinguishing computation of equilibria from ensuring their existence. It also records assumptions and practical stability boundaries.
- Global convergence: The theoretical analysis cannot explain why generator and discriminator losses remain almost constant during training.
- Limitations: The theoretical results assume that a Nash-equilibrium exists, which may fail for misspecified generator dimensionality when minimizing an f-divergence.The authors leave closer theoretical investigation of this case to future research.
- Limitations: A high regularization parameter can stabilize formerly unstable stationary points and thereby lead to poor solutions.The method also becomes less stable for deeper architectures, where gradient scales can differ substantially.
- Interpretations: Consensus optimization can be viewed as an approximation to implicit Euler integration, whose stability properties support local Nash-equilibrium convergence.Implicit Euler itself requires solving a nonlinear equation at every iteration.
- Interpretations: The method can also be interpreted as a second-order method, motivating further study of second-order optimization for saddle-point problems.
- Related work: Unlike architecture, objective, noise, or divergence strategies, this work focuses on stabilizing training across a wide range of architectures and divergence functions.The related-work discussion notes practical costs or restrictions associated with several alternative approaches.
- Conclusion: The paper reports local convergence to Nash-equilibria despite problematic Jacobian eigenvalues and experimental stabilization across GAN architectures, including mitigation of mode collapse.
The Numerics of GANs: Supplementary Material
The supplied supplementary-material passages identify the paper’s authors and their research groups.
- Lars Mescheder is affiliated with the Autonomous Vision Group.
- Sebastian Nowozin is affiliated with the Machine Intelligence and Perception Group.
- Andreas Geiger is affiliated with the Autonomous Vision Group.
Abstract
The document supplements the main paper with omitted proofs and additional experimental results supporting stable GAN training across architectures and divergence measures.
- The document contains proofs omitted from the main text.
- Additional experiments demonstrate stable GAN training across architectures and divergence measures.
Proofs
The proofs establish local convergence conditions for fixed-point iterations based on the gradient vector field and its Jacobian. They also characterize convergence rates and the role of eigenvalue conditions.
- The convergence error decreases at least linearly as O(|λmax|^k), where λmax has the largest absolute eigenvalue of F′(¯x).
- For eigenvalues λ = −a + bi, the condition |1 + hλ|^2 = 1 + h^2b^2 + h^2a^2 − 2ha determines whether the update lies within the unit ball.
- If the Jacobian v′(¯x) has only eigenvalues with negative real parts, Algorithm 1 is locally convergent for sufficiently small h.
- Fixed points of the analyzed iteration coincide with stationary points of v when A(x) is invertible, and A(¯x)v′(¯x) is negative definite under the stated assumptions.
- At a local Nash-equilibrium, the modified iteration converges from an open neighborhood when h is sufficiently small.
Additional Theoretical Results
The additional results extend convergence analysis to ordinary gradient ascent, momentum, rescaling, and alternating updates, while examining minibatch bias in consensus optimization.
- Gradient ascent: For ordinary gradient ascent, negative-definite ∇2f(x) yields local convergence toward a fixed point for sufficiently small h.
- Momentum: Momentum transforms each eigenvalue through λ(λ + γ), and with real non-positive eigenvalues of v′(¯x), the iteration is locally convergent for small h.
- Gradient rescaling: Gradient rescaling is locally convergent when α ∈ (0, 1) and the eigenvalues of I + h√ϵv′(¯x) lie in the unit ball.
- Alternating gradient ascent: For alternating gradient ascent, eigenvalues of the update approach those of the gradient-field Jacobian as h becomes small, giving local convergence when their real parts are negative.
- Consensus optimization: The minibatch estimator of consensus loss is biased because its expectation exceeds L(x) by 1/2 Var[vB(x)], while the bias decreases linearly with batch size.
Additional Experimental Results
The experiments compare simultaneous gradient ascent, alternating gradient ascent, consensus optimization, and a smoothing variant across challenging DC-GAN architectures and generator objectives. Consensus optimization generally trains more architectures successfully, though its performance depends on the setting.
- Architecture comparisons: Consensus optimization successfully trains DC-GAN variants without batch-normalization, with constant filters, and with a Jensen-Shannon generator objective, while struggling with the standard architecture.The comparison includes simultaneous gradient ascent and alternating gradient ascent as baselines.
- Smoothing comparison: The smoothing optimizer trains a batch-normalized DC-GAN where consensus optimization fails, but usually performs worse than consensus optimization.Smoothing removes the regularization term from the generator loss while retaining it for the discriminator loss.
- Smoothing comparison: With an additional discriminator fully connected layer, the smoothing optimizer fails on the constant-filter architecture where consensus optimization succeeds.The experiments are conducted on CIFAR-10 and otherwise show similar results to the preceding comparison.
- Generator objectives: Consensus optimization can train GANs using a large variety of divergence functions on a DC-GAN without batch-normalization and with constant filters.The samples are generated from the resulting models.
- Architecture comparisons: On a modified DC-GAN without batch-normalization, constant filters, and additional ResNet layers, only consensus optimization produces visually compelling results.Simultaneous gradient ascent completely fails, while alternating gradient ascent reaches a bad solution.
- Architecture comparisons: Consensus optimization achieves much better end results than alternating gradient ascent on a four-layer DC-GAN trained on CIFAR-10.The comparison uses generator and discriminator losses and inception score computed from 6400 samples.
Effect of Hyperparameters
The experiments examine how learning rate and regularization affect alternating gradient ascent and consensus optimization with RMSProp. Consensus optimization is tested with architecture-specific regularization settings and a fixed learning rate.
- Learning rate: Learning rate is varied for alternating gradient ascent on a DC-GAN architecture using RMSProp.The experiment isolates learning-rate effects while keeping the optimizer fixed.
- Learning rate: Consensus optimization is evaluated across learning rates using RMSProp, with γ = 0.1 for the three-layer architecture and γ = 10 for the four-layer architecture.The regularization parameter differs by architecture in this experiment.
- Regularization: The effect of γ on consensus optimization is evaluated with RMSProp at a fixed learning rate of 2 · 10^-4.The experiment varies the regularization parameter while holding the learning rate constant.