PaperlayerView exact revision on arXiv

Source-linked AI summary

The Unusual Effectiveness of Averaging in GAN Training

Yasin Yazıcı, Chuan-Sheng Foo, Stefan Winkler, Kim-Hui Yap, Georgios Piliouras, Vijay Chandrasekhar

arXiv:1806.04498v2stat.MLcs.CVcs.LG

TL;DR

GAN training can cycle or fail to converge, motivating stabilization methods that do not alter the adversarial game. This paper analyzes moving and exponential parameter averaging theoretically and experimentally, finding that both improve GAN results while EMA reduces cycle amplitude and is typically more robust. The approach remains limited when averaging brings optimization near a poor local optimum and can slow convergence under certain spectral conditions.

  • Problem

    GANs are difficult to train because adversarial dynamics can cycle around optima or spiral outward instead of converging.

  • Method

    The paper analyzes moving average and exponential moving average of GAN parameters outside the training loop, including bilinear dynamics and local GAN stability.

  • Results

    Both averaging techniques significantly improve generated-image quality across datasets, architectures, and GAN objectives, while EMA shrinks cyclic amplitudes and typically performs more consistently than MA.

  • Takeaways & Limitations

    Parameter averaging is simple to implement, has minimal computation overhead, and can stabilize GAN training without altering the game’s optimal points.

  • Takeaways & Limitations

    Averaging can bring generators near a local optimum without finding a good solution, and EMA may slow convergence when β exceeds the relevant Jacobian eigenvalue magnitudes.

Abstract

from arXiv · show

We examine two different techniques for parameter averaging in GAN training. Moving Average (MA) computes the time-average of parameters, whereas Exponential Moving Average (EMA) computes an exponentially discounted sum. Whilst MA is known to lead to convergence in bilinear settings, we provide the -- to our knowledge -- first theoretical arguments in support of EMA. We show that EMA converges to limit cycles around the equilibrium with vanishing amplitude as the discount parameter approaches one for simple bilinear games and also enhances the stability of general GAN training. We establish experimentally that both techniques are strikingly effective in the non-convex-concave GAN setting as well. Both improve inception and FID scores on different architectures and for different GAN objectives. We provide comprehensive experimental results across a range of datasets -- mixture of Gaussians, CIFAR-10, STL-10, CelebA and ImageNet -- to demonstrate its effectiveness. We achieve state-of-the-art results on CIFAR-10 and produce clean CelebA face images.\footnote{~The code is available at \url{https://github.com/yasinyazici/EMA_GAN}}

1 INTRODUCTION

GANs are difficult to train because adversarial dynamics can cycle or spiral rather than converge. This work studies parameter averaging outside the training loop to stabilize that behavior without changing the adversarial game.

  • Motivation: GAN training is unstable and often fails to converge because it is formulated as a two-player zero-sum game.Prior work addresses instability through function families, objectives, regularization, time-scale techniques, and extra-gradient methods.
  • Motivation: Cycling around an optimum or slow outward spiraling is identified as a key source of non-convergence.Possible causes include Jacobian eigenvalues with zero real parts, discriminator update limitations, sampling noise, and generator learning-rate effects.
  • Approach: The paper averages generator parameters over time outside the training loop to address cycling without influencing the adversarial game.Averaging both players is known to be optimal in convex-concave min-max games, but exponential discounting lacks guarantees even for bilinear games.
  • Contributions: EMA theoretically shrinks the amplitude of cyclic behavior, while both averaging techniques improve results across datasets, architectures, and GAN objectives.The paper also compares averaging with other methods targeting cycling and non-convergence.

2 RELATED WORK

Related work improves GAN stability through regularization, historical mixtures, time-scale updates, and extra-gradient methods. This paper distinguishes parameter averaging for saddle-point optimization from averaging in maximum-likelihood objectives and examines uniform versus exponential averaging.

  • GAN stability methods: Historical averaging, mixtures of historical generators, and two-time-scale updates have been used to improve stability or approach local equilibrium.These methods respectively penalize deviation from parameter averages, update discriminators using historical generators, or use different update time scales.
  • GAN stability methods: Jacobian-based regularization and extra-gradient methods target non-convergence, but some approaches alter the min-max objective by adding regularization terms.The cited Jacobian issue involves eigenvalues with zero real parts and large imaginary parts.
  • Parameter averaging: Averaging results from maximum-likelihood optimization cannot be formally extrapolated to GANs because GANs use saddle-point optimization.The paper emphasizes the distinction between convex or non-convex optimization and saddle-point objectives.
  • Parameter averaging: For minimax objectives, prior work used uniform averaging, while this paper analyzes its shortcomings and argues that exponential decay averaging performs better.The paper notes that earlier use of this strategy lacked detailed analysis or theoretical insight.

3 METHOD

The method maintains averaged generator parameters outside GAN training, using moving average for uniform weighting and exponential moving average to discount older iterates. EMA offers a finite effective time window and is typically more robust in practice.

  • Moving Average: Moving Average maintains an efficient uniform average of parameters without storing every past iterate.Its average approximates the mean when generator iterates reach a stationary distribution.
  • Moving Average: Averaging iterates from changing distributions can worsen results because the generator may not remain in one stationary distribution.The method also faces uncertainty about when stationarity begins.
  • Exponential Moving Average: Exponential Moving Average is introduced to address the practical issues of uniform averaging over potentially changing iterates.EMA discounts earlier iterates rather than weighting the full history equally.
  • Exponential Moving Average: EMA has an effective time window controlled by β, and as β approaches 1 it averages over longer windows.Both EMA and MA operate outside the training loop, with minimal computation overhead and no influence on the game’s optimal points.
  • Practical considerations: EMA typically provides more robust performance, and multiple β values can be evaluated in one training run by tracking several averages.This supports convenient hyperparameter search without rerunning training.

4 WHY DOES AVERAGING WORK?

The paper studies why parameter averaging can stabilize GAN training, first analyzing EMA and MA in simple bilinear games and then examining EMA near local Nash equilibria. The analysis shows that EMA reduces cyclic oscillations while preserving the local stability properties of the underlying training dynamics, though it may slow convergence.

  • 4 WHY DOES AVERAGING WORK?: Parameter averaging is analyzed as a way to stabilize GAN training without changing the adversarial game.The theoretical analysis uses bilinear saddle problems and a discrete-time non-convex model to study averaging behavior.
  • 4.1 GLOBAL ANALYSIS FOR SIMPLE BILINEAR GAMES: Time-averaging converges to equilibria in bilinear zero-sum games, whereas EMA had lacked comparable guarantees even in this setting.The paper contrasts known convergence of moving averages with the limited prior understanding of exponentially discounted averages.
  • 4.1 GLOBAL ANALYSIS FOR SIMPLE BILINEAR GAMES: EMA applied to the periodic trajectories of a bilinear saddle problem produces a lower-amplitude periodic sequence centered around the equilibrium.For β close to 1, the residual oscillation has approximately ln(β) amplitude and does not vanish completely.
  • 4.2 LOCAL STABILITY ANALYSIS FOR GANS: The general GAN analysis represents training as a single-iterate update xt = F(xt−1) and augments it with EMA iterates that require the two previous iterates.This formulation includes simultaneous and alternating gradient descent methods.
  • 4.2 LOCAL STABILITY ANALYSIS FOR GANS: EMA preserves local stability exactly when the eigenvalues of the underlying training Jacobian lie within the unit ball.The augmented EMA Jacobian adds eigenvalues β while retaining the eigenvalues of the original training operator.
  • 4.2 LOCAL STABILITY ANALYSIS FOR GANS: EMA can slow convergence when β exceeds the magnitude of all other eigenvalues of the underlying training operator.This is a rate limitation rather than a loss of local stability under the stated condition 0 < β < 1.

5 EXPERIMENTS

Experiments across illustrative and real-world datasets evaluate EMA and MA against non-averaged training and competing stabilization methods. EMA consistently improves GAN metrics and stability, while MA can degrade when averaging spans changing parameter regions.

  • Experimental setup: Experiments cover mixtures of Gaussians, CIFAR-10, STL-10, CelebA, and ImageNet using multiple architectures and GAN objectives.The evaluation includes illustrative and real-world datasets, with comparisons against Optimistic Adam, Consensus Optimization, and Zero-GP.
  • Mixture of Gaussians: In the mixture-of-Gaussians experiment, EMA outperforms non-averaging, MA, Optimistic Adam, Consensus Optimization, and Zero-GP on Wasserstein-1 distance.MA improves results in some cases but less than EMA; uniform averaging over long iterates can hurt performance, so MA starts later in training.
  • Mixture of Gaussians: EMA produces more balanced and stable mode support across iterations, whereas the non-averaged generator fluctuates by alternately omitting modes.OMD, CO, and Zero-GP outperform the baseline but appear less stable across iterations than EMA.
  • Quantitative scores: Across CIFAR-10, STL-10, and ImageNet, EMA significantly improves Inception and FID scores and performs more consistently than MA.The experiments report scores at selected generator iteration counts and repeat experiments three times.
  • CIFAR-10: On CIFAR-10, EMA consistently outperforms the non-averaged model in FID and Inception scores, while MA FID worsens as its averaging window lengthens.The authors attribute this to averaging parameters from different regions when generator iterates do not remain in one parameter region.
  • CelebA: On CelebA, image quality improves with longer EMA windows, but β values near one change attributes and β = 0.999 and β = 0.9999 look better than MA.The results indicate an optimal β or averaging-window size, while EMA also yields smoother attribute changes over training.
  • STL-10 and ImageNet: For STL-10 and ImageNet, EMA improves quantitative scores, but visual improvements are small and both models remain largely unrecognizable.The authors interpret this as averaging approaching a local optimum without guaranteeing that the local optimum is good enough.

6 CONCLUSION

The paper studies moving average and exponential moving average parameter averaging outside the GAN training loop. Both improve generated-image quality across datasets, architectures, and objectives, while EMA also admits a theoretical stability analysis.

  • 6 CONCLUSION: The study evaluates moving average (MA) and exponential moving average (EMA) parameter averaging outside the GAN training loop.Both methods are designed to improve GAN training without being part of the adversarial update loop.
  • 6 CONCLUSION: Both averaging techniques significantly improve generated-image quality across datasets, network architectures, and GAN objectives.
  • 6 CONCLUSION: EMA converges to stable limit cycles of small amplitude around the saddle-problem solution in simple bilinear settings.The theoretical result concerns EMA's behavior even though it does not converge directly to equilibrium.
  • 6 CONCLUSION: Averaging methods are easy to implement and impose minimal computational overhead.The paper presents them as applicable across a wide range of settings.

A.1 PARAMETERIZED ONLINE AVERAGING

Parameterized online averaging generalizes moving averaging through an exponent α that controls weighting. In CIFAR-10 experiments, delayed starts and several α values help, but none matches EMA.

  • A.1 PARAMETERIZED ONLINE AVERAGING: Parameterized online averaging uses α to regulate the importance of terms in the average.When α = 1, the method corresponds to moving average.
  • A.1 PARAMETERIZED ONLINE AVERAGING: The algorithm is restricted to t ≥α because t−α can otherwise be negative.
  • A.1 PARAMETERIZED ONLINE AVERAGING: On CIFAR-10, α = 1 fails because averaging starts from the first iteration.The experiment uses the same setting as the first row of Table 2.
  • A.1 PARAMETERIZED ONLINE AVERAGING: α = 10, 100, and 1000 improve scores by varying amounts, but none performs as well as EMA.

A.2 QUANTITATIVE EFFECTS OF DIFFERENT β FOR EMA

The paper evaluates EMA across β values from 0.9 to 0.9999 on CIFAR-10. Both Inception and FID improve progressively as β increases, motivating β = 0.9999 for most experiments.

  • A.2 QUANTITATIVE EFFECTS OF DIFFERENT β FOR EMA: EMA experiments on CIFAR-10 vary β from 0.9 to 0.9999 using four repeated 500k-iteration runs.Inception and FID are collected every 20k iterations, and the best scores are reported with means and standard deviations.
  • A.2 QUANTITATIVE EFFECTS OF DIFFERENT β FOR EMA: Both Inception and FID show clear, progressive improvement as β increases from 0.9 to 0.9999.

B FURTHER RESULTS

The further-results figures document generated samples and training-time FID and Inception scores across datasets, averaging settings, objectives, architectures, and discriminator-to-generator update ratios.

  • B FURTHER RESULTS: Figure 9 shows CIFAR-10 generations after 300k iterations.
  • B FURTHER RESULTS: Figures 10 and 11 show STL-10 and ImageNet generations after 500k iterations.
  • B FURTHER RESULTS: Figures 12 and 13 track CIFAR-10 FID and Inception scores during training under original GAN and WGAN-GP objectives with ResNet architectures.The original GAN setting uses ndis = 5, whereas the WGAN-GP setting uses ndis = 1.
  • B FURTHER RESULTS: Figures 14–18 compare CelebA generations for non-averaged, EMA, and MA generators across β values and training iterations.The rows include β = 0.9, 0.99, 0.999, 0.9999, and MA at 100k, 150k, 200k, 300k, and 350k iterations.

C NETWORK ARCHITECTURES

The experiments use conventional and ResNet generator architectures, with specified latent priors and normalization, while implementation settings are organized across architecture and optimization tables.

  • Architecture design: ResNet layers use doubled feature numbers under spectral normalization, while generator priors are 512-dimensional for conventional architectures and 128-dimensional for ResNet.Latent samples are normalized onto a unit hypersphere before entering the generator.
  • Architecture design: Architecture specifications for conventional 32x32-resolution generators and discriminators are provided in Tables 5 and 6.These tables define the conventional architecture settings used in the experiments.
  • Architecture design: The conventional generator and discriminator each use four 256-unit layers, with an additional generator projection layer and discriminator classifier layer.ReLU follows each affine transformation.
  • Experimental settings: Mixture-of-Gaussians settings are separately reported for the baseline, Optimistic Adam, Consensus Optimization, and Zero-GP experiments.The corresponding settings appear in Tables 7–10.
  • Experimental settings: Table 11 collects settings for CIFAR-10, STL-10, CelebA, and ImageNet experiments.The table covers the multi-dataset experiments beyond the mixture-of-Gaussians setup.
Loading 1806.04498v2…