Source-linked AI summary
Stabilizing Training of Generative Adversarial Networks through Regularization
Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, Thomas Hofmann
TL;DR
GAN training can fail when model and data distributions have dimensional mismatch or non-overlapping support, leaving the relevant f-divergence undefined. The paper derives a low-cost, noise-motivated weighted gradient-norm regularizer and reports stable training across models and benchmark tasks. Its conclusion is that this modification improves stability, reduces overfitting, and supports better generalization within the evaluated scope.
Problem
Dimensional misspecification or negligible support overlap can make the f-divergence used by f-GANs non-finite, undermining GAN training.
Method
The paper derives an efficiently computable regularizer from Gaussian-noise convolution, implemented as a discriminator weighted gradient-norm penalty.
Results
The regularizer yields stable GAN training and higher-quality samples across a Gaussian mixture and benchmark image datasets, with stable training reported across a broad set of models.
Takeaways & Limitations
The modification turns GANs into more reliable deep-learning building blocks that can be trained indefinitely without collapse, according to the paper’s conclusion.
Takeaways & Limitations
The analytic approximation requires assumptions near the optimum, including convergence of the regularized objective to ψ∗ + O(γ).
Abstract
from arXiv · showhide
Deep generative models based on Generative Adversarial Networks (GANs) have demonstrated impressive sample quality but in order to work they require a careful choice of architecture, parameter initialization, and selection of hyper-parameters. This fragility is in part due to a dimensional mismatch or non-overlapping support between the model distribution and the data distribution, causing their density ratio and the associated f-divergence to be undefined. We overcome this fundamental limitation and propose a new regularization approach with low computational cost that yields a stable GAN training procedure. We demonstrate the effectiveness of this regularizer across several architectures trained on common benchmark image generation tasks. Our regularization turns GAN models into reliable building blocks for deep learning.
1 Introduction
GANs train a generator and discriminator in an adversarial game, but dimensional misspecification can make f-divergences undefined. The paper derives a noise-motivated, efficiently computable gradient regularizer that stabilizes training and improves sample quality.
- GAN framework: GANs pair a generator producing Qθ with a discriminator that distinguishes the model distribution from the data distribution P.The discriminator represents a class of distribution-discrepancy objectives optimized through a supremum and saddle-point problem.
- Regularization approach: The proposed regularizer is motivated by convolving P and Q with white Gaussian noise and is implemented as a weighted squared-gradient penalty on the discriminator.Unlike naive norm penalization, the weighting depends on the discriminator output and the selected regularization scale γ.
- Motivation: Dimensional misspecification occurs when P and Q lack a common density or have negligible support intersection, making the f-divergence non-finite.This mismatch can arise when the data or model is confined to a low-dimensional manifold.
- Empirical evaluation: The approach is demonstrated on a Gaussian mixture and benchmark image datasets, where it yields stable GAN training and higher visual sample quality.The paper also compares regularized and unregularized GANs using a pairwise cross-testing protocol.
- Contributions: The paper’s stated contributions are a novel efficiently computable f-GAN regularizer, robustness to dimensional misspecification, and stable training across many models.These contributions position the method as a broadly applicable GAN training modification.
2 Background
The background contrasts f-divergences with integral probability metrics and reviews noise-based remedies for dimensional misspecification. It motivates analytic Gaussian convolution as a lower-variance alternative to injecting high-dimensional discriminator noise.
- f-divergence: f-divergences generalize GANs’ Jensen-Shannon objective and admit variational formulations that replace density-based evaluation with expectations over P and Q.These expectations can be estimated by sampling from the two distributions.
- Integral Probability Metrics: Integral probability metrics distinguish P and Q through a witness function class, including the function classes used by maximum mean discrepancy and Wasserstein GANs.Different choices of the function class produce different IPM methods.
- Integral Probability Metrics: Wasserstein GANs restrict the witness functions to bounded Lipschitz functions, a hard constraint implemented with weight clipping that is empirically difficult to satisfy.This restriction yields a weaker notion of convergence than the original GAN’s Jensen-Shannon divergence.
- Training with Noise: Adding continuous discriminator noise can smooth distributions and break dimensional misspecification, but high-dimensional noise increases parameter-estimation variance and sample requirements.The paper instead proposes analytic convolution of P and Q with Gaussian noise.
- Related regularization: Other regularizers target mode dropping and may add supervision, auto-encoders, or multi-step procedures that can be computationally expensive.These approaches are presented as complementary to the paper’s regularization method.
3 Noise-Induced Regularization
The paper interprets input-noise training as discriminator regularization and derives a tractable weighted gradient-norm approximation. This approach avoids sampling noise, explicit density convolution, and costly Hessian computations while relying on convergence near the optimum.
- Noise-induced regularization: Adding noise convolves the data and model distributions, giving them full support and making the generalized f-divergence well-defined under dimensional misspecification.The formulation applies noise to both P and Q, with λ(x) > 0 and (p ∗ λ)(x) > 0.
- Noise-induced regularization: Equivalent convolution reformulation restricts the discriminator to a smaller, smoother family, producing a low-pass or short-sighted adversary.The distributions remain unchanged in this view, while the discriminator’s accessible perspective is smoothed.
- Analytic approximations: For small γ, a Taylor approximation replaces analytic convolution with a Laplacian correction based on the trace of the Hessian.The Laplacian measures deviation of scalar fields from their local average and acts as an infinitesimal proxy for exact convolution.
- Analytic approximations: Computing Laplacians for composed discriminator networks can require a full intermediate-layer Hessian, which is computationally demanding.The Laplacian itself avoids the quadratic Hessian blow-up, but composition through deep networks creates the intermediate Hessian burden.
- Gradient-norm approximation: The proposed simplification uses the optimizer’s density-ratio characterization to cancel Laplacian terms near ψ∗ and retain a weighted squared-gradient penalty.The assumption is that optimizing the regularized objective brings ψ to ψ∗ + O(γ); cancellation is not assumed far from the optimum.
- Gradient-norm approximation: Convexity of f_c makes the gradient-norm weighting non-negative, so the regularized maximization remains well-defined.The regularizer is upper bounded by zero under f_c′′ ≥ 0.
4 Regularizing GANs
The paper turns the noise analysis into a class of f-GAN discriminator regularizers, including a Jensen–Shannon GAN form. It gives an implementable training procedure, recommends annealing γ, and cautions that regularizing the generator can destabilize convergence.
- Regularizer construction: The proposed f-GAN regularizers arise from the equivalence between training with noise and regularizing the discriminator.The construction is presented as a class of regularizers inspired by the preceding analytic analysis.
- Regularized Jensen-Shannon GAN: The commonly used Jensen–Shannon GAN parameterization receives an analogous regularizer expressed using the discriminator logit φ = σ−1(ϕ).The authors prefer gradients of φ because they are easier to implement and more robust than gradients after the sigmoid.
- Training procedure: Algorithm 1 specifies regularized JS-GAN training with initial noise variance γ0, annealing rate α, discriminator updates nϕ, minibatches, and T iterations.The listed defaults are γ0 = 2.0, α = 0.01 with annealing, γ = 0.1 without annealing, and nϕ = 1.
- Training procedure: Discriminator regularization efficiently convolves the distributions and addresses dimensional misspecification without explicitly performing that convolution.The paper leaves open using the regularizer in the generator objective, but warns that this can destabilize training near convergence.
- Training procedure: A practical annealing scheme starts with large γ early in training and exponentially decays it to a small non-zero value.The paper leaves the question of determining an optimal annealing schedule for future work.
5 Experiments
Experiments across synthetic and benchmark image-generation settings show that regularization stabilizes GAN training, including under dimensional misspecification, while preserving or improving sample quality.
- 5.1 2D submanifold mixture of Gaussians in 3D space: The regularized GAN trained essentially indefinitely without collapse on a dimensionally misspecified 2D submanifold mixture, unlike the unregularized model.The unregularized GAN collapsed in every run after around 50k iterations, whereas the regularized variant remained stable beyond 200k iterations across noise variances and annealing settings.
- 5.3 Training time: Regularization increased overall training time by a marginal factor of roughly 1.4 because discriminator-gradient backpropagation adds computation.The paper contrasts this cost with the reported faster convergence or sample generation of regularized f-GANs relative to WGANs.
- 5.3 Training time: Regularized GANs remained stable across ResNet, DCGAN, DCGAN without normalization, and DCGAN with tanh activations on LSUN bedrooms.Samples were produced after 200k generator iterations using exponentially annealed γ0 = 2.0 and an alternative generator loss.
- 5.3 Training time: Regularized ResNet GANs on CelebA could be trained essentially indefinitely without collapse, with superior sample quality evident after 100k generator iterations.The experiments varied the initial regularization level γ0 and exponentially annealed it during training.
- 5.5 Cross-testing protocol: The regularizer stabilized CIFAR-10 DCGANs across a broad range of noise levels and produced higher-quality images than unregularized variants after 50 training epochs.The comparison included explicitly added noise with noise-to-signal ratio 4 and regularized or unregularized DCGANs using an alternative generator loss.
- 5.5 Cross-testing protocol: In CIFAR-10 cross-testing, the regularized GAN classified unregularized samples as fake with FP 0.0, while the unregularized GAN classified regularized samples as real with FP 1.0.These results indicate asymmetric cross-testing behavior between the two models.
6 Conclusion
The paper introduces a weighted discriminator-gradient regularizer for GANs. It addresses dimensional misspecification, improves stability and generalization-related behavior, and enables training essentially indefinitely without collapse.
- 6 Conclusion: The proposed method penalizes the discriminator’s weighted gradient norm to address dimensional misspecification and non-overlapping supports.The regularization scheme modifies standard GAN training while retaining a simple implementation.
- 6 Conclusion: Experiments show that the regularizer improves stability, prevents GAN overfitting, and leads to better generalization properties under the cross-testing protocol.The conclusion presents these properties as the main empirical consequences of the regularized training procedure.
- 6 Conclusion: The resulting GAN models can essentially be trained indefinitely without collapse, providing reliable building blocks for deep learning.The conclusion frames this as the paper’s main result and practical consequence.
7 APPENDIX
The appendix derives the Jensen–Shannon regularizer, examines its well-definedness under dimensional misspecification, and reports experiments on synthetic and image-generation settings. It also specifies architectures and training configurations used for the empirical evaluations.
- Regularizer derivation: The Jensen–Shannon regularizer is derived by Taylor-expanding the noise-convolved objective and cancelling Laplacian terms at the optimal discriminator.The approximation assumes small noise variance and yields the corresponding JS regularizer.
- Regularizer derivation: The original GAN and Fenchel-dual f-GAN objectives are equivalent at the optimum, but their regularizers use different parametrizations.The original objective uses the Jensen–Shannon-specific form, whereas the Fenchel-dual objective uses the general f-GAN form.
- Well-definedness: The regularizer remains well-defined under dimensional misspecification because the noisy objective has a maximizer for every positive noise variance and its regularizers converge pointwise.This contrasts with the possible nonexistence of an unregularized optimum when the support or dimensionality mismatch makes the supremum infinite.
- Synthetic experiment: The synthetic target is a seven-component Gaussian mixture embedded as a tilted two-dimensional submanifold in three-dimensional space, making it degenerate under the ambient base measure.The displayed mixture uses enlarged component variance for visualization, while the experimental mixture uses the smaller variance described in the appendix.
- Experimental results: The regularized ResNet LSUN experiment generates full-resolution 64 × 64 images after 200k generator iterations while exponentially annealing γ from 2.0 to 0.01.The appendix also presents unregularized comparisons across ResNet, DCGAN, and modified DCGAN architectures.