Source-linked AI summary
A Large-Scale Study on Regularization and Normalization in GANs
Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, Sylvain Gelly
TL;DR
GAN training is difficult because performance depends on losses, architectures, regularization, normalization, and hyperparameter choices, while fair comparisons face reproducibility issues. This paper performs a large-scale empirical analysis of these choices and concludes that non-saturating loss and spectral normalization are practical defaults, with gradient penalty useful given additional computation.
Problem
GAN training is sensitive to design and hyperparameter choices, while inconsistent implementations and nondeterminism complicate accurate, fair comparison across approaches.
Method
The study evaluates losses, normalization, regularization, and architectures across modern large-scale datasets using literature settings and sequential Bayesian hyperparameter optimization.
Results
The study finds non-saturating loss and spectral normalization suitable as default choices under limited computation, while gradient penalty can be added with more budget and convergence training.
Takeaways & Limitations
The work provides practical guidance, reference implementations, trained models, and concrete actions for improving fair and reproducible GAN comparisons.
Takeaways & Limitations
Reproducing GAN results remains difficult because GPU-operation nondeterminism can prevent identical scores and disabling it can impose an order-of-magnitude runtime penalty.
Abstract
from arXiv · showhide
Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion. While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant number of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of "tricks". The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs resulted in a plethora of proposed losses, regularization and normalization schemes, as well as neural architectures. In this work we take a sober view of the current state of GANs from a practical perspective. We discuss and evaluate common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub.
1. Introduction
GANs learn complex distributions through a generator–discriminator game, but their deep-network minimax optimization is notoriously difficult. This motivates a large-scale empirical analysis of losses, normalization, regularization, architectures, and hyperparameter choices.
- GANs train a generator to transform a simple input distribution into a complex distribution that a discriminator cannot distinguish from real samples.
- Deep neural network parameterizations make the GAN minimax problem notoriously hard to solve.
- Training is sensitive to losses, architectures, regularization, normalization, and stochastic optimization choices, prompting extensive work on these challenges.
- The study empirically analyzes competing GAN approaches across large-scale datasets using hyperparameter optimization and literature-reported settings.
- The authors decompose normalization, regularization, and loss effects, finding gradient penalty and spectral normalization useful for high-capacity architectures and non-saturating loss stable across datasets and hyperparameters.
2. The GAN Landscape
The GAN landscape is defined by interacting choices of loss functions, regularization and normalization schemes, architectures, and evaluation metrics. The paper examines these choices and hyperparameter sensitivity, including their practical effects on training and measurement.
- Design choices: GAN design varies across loss functions, regularization or normalization schemes, and neural architectures, making models sensitive to these choices.The study focuses on DCGAN and ResNet architectures, along with several loss and discriminator-control methods.
- Loss functions: The non-saturating loss is used because prior empirical evidence indicates it outperforms the minimax variant.In this formulation, the generator maximizes the probability that generated samples are classified as real.
- Loss functions: WGAN replaces the Jensen-Shannon divergence with the Wasserstein distance and requires a 1-Lipschitz discriminator.The discriminator output is real-valued, and enforcing Lipschitzness is identified as a key challenge.
- Regularization: Gradient penalty regularizes discriminator gradient norms, but its computation can be intensive and it may depend on the changing model distribution.The study also considers L2 regularization as a general discriminator-complexity regularizer.
- Normalization: Normalization can target optimization or representation properties, with batch normalization, layer normalization, and spectral normalization applied to discriminators.Spectral normalization divides each weight matrix by its spectral norm to control operator norms and is argued to produce higher-rank discriminators than competing approaches.
- Evaluation metrics: FID compares embedded real and generated samples through their estimated Gaussian statistics, while KID is evaluated as an unbiased alternative.KID and FID are reported to be very strongly correlated, with Spearman coefficients of 0.994 for LSUN-BEDROOM and 0.995 for CELEBA-HQ-128.
3. Experimental Results and Discussion
The study evaluates GAN design choices through controlled slices of a large search space, measuring FID quality, stability, and computational trade-offs. Across architectures and settings, spectral normalization is consistently beneficial, while gradient penalty can help but requires more tuning and computation.
- Evaluation methodology: The study varies selected GAN components and evaluates top-5% FID distributions, where lower median FID indicates better quality and lower variance indicates greater optimization stability.It also measures the minimum attainable FID under fixed computational budgets and repeats training across five random initializations.
- Regularization and normalization: Spectral normalization improves model quality and is more computationally efficient than gradient penalty.The comparison fixes the loss and architecture while varying normalization and regularization schemes.
- Regularization and normalization: Gradient penalty can improve performance, but tuning its regularization strength is non-trivial and may require a large computational budget.Models using GP benefit from a 5:1 discriminator-to-generator update ratio and may improve with an additional 100K optimization steps.
- Neural architectures: Spectral normalization improves sample quality across both ResNet-based and SNDCGAN architectures, with both architectures achieving comparable results.The figure compares FID distributions for the top 5% of models using non-saturating loss.
- Simultaneous regularization and normalization: Combining gradient penalty with spectral or layer normalization strongly improves performance over the baseline.The authors partially explain this by noting that spectral normalization does not ensure a 1-Lipschitz discriminator because of convolutional-layer normalization.
4. Challenges of a Large-Scale Study
The study identifies reproducibility and comparison challenges arising from metric definitions, architecture details, dataset processing, implementation mismatches, and nondeterministic computation. These issues can compound and hinder both practical use and research progress.
- Metrics: FID scores can differ by more than 10% because studies use different datasets, sample counts, and implementations.The authors argue that FID should be computed against the test dataset.
- Neural architectures: ResNet comparisons can be unfair because papers often omit design choices such as layer order, activation placement, resampling, and filter counts.The authors recommend their reported architectures as solid baselines.
- Datasets: Unclear upscaling and cropping procedures can make nominally identical datasets inconsistent across studies.The issue is highlighted for LSUN-BEDROOM.
- Implementation and non-determinism: Paper-code mismatches and GPU-operation randomness make exact reproduction difficult, while disabling nondeterministic optimizations can impose an order-of-magnitude runtime penalty.These problems remain even after fixing data ordering and initial weights.
- Overall challenge: Individually minor reproducibility issues compound to create friction in practical applications and research.The paper frames this accumulation as a broader obstacle to progress.
5. Related Work
This work extends earlier GAN comparisons by studying more complex architectures and datasets alongside broader regularization and normalization choices. Its related evidence also highlights that combining gradient penalty with normalization can outperform a baseline.
- Prior large-scale studies: Earlier large-scale work found no statistically significant difference between recent GAN models and the original non-saturating GAN in a limited setting.That study used low-to-medium complexity datasets and a single neural architecture.
- Combined methods: Gradient penalty coupled with spectral or layer normalization strongly improves performance over the baseline.The cited comparison uses FID distributions for the top 5% of models.
- Combined methods: Spectral normalization’s limitation in guaranteeing a 1-Lipschitz discriminator partially explains why combining it with gradient penalty can help.The explanation concerns how convolutional layers are normalized.
- Scope expansion: The present study extends prior comparisons to more complex neural architectures, datasets, and regularization and normalization schemes.It builds on open-source code from the earlier study.
6. Conclusions and Future Work
The study evaluates regularization, normalization, losses, and architectures for GAN training, then identifies practical defaults and reproducibility pitfalls. It also outlines evaluation areas that remain for future work.
- The study varies regularization, normalization, loss functions, and neural architectures, assessing generated-sample quality with quantitative metrics.
- For limited computational budgets, the authors suggest non-saturating loss and spectral normalization as default choices; with more budget, they suggest adding gradient penalty and training to convergence.
- Both evaluated neural-architecture classes can perform well across the considered datasets, while most ResNet-style modifications yield only marginal sample-quality improvements.
- The study identifies metrics, dataset preprocessing, nondeterminism, and missing implementation details as obstacles to accurate and fair comparisons.
- Future work should evaluate large-scale models, custom architectures, newer stabilization methods, conditional GANs, and additional metrics beyond FID and IS.
A. FID and Inception Scores on CIFAR10
The CIFAR10 study evaluates SNDCGAN and ResNet CIFAR architectures and recovers results reported by Miyato et al. (2018). It includes Inception Score comparisons across the studied models.
- The empirical study uses SNDCGAN and ResNet CIFAR architectures on CIFAR10 and reproduces results reported by Miyato et al. (2018).
- Figure 6 reports the Inception Score for each model in the study, corresponding to results reported by Miyato et al. (2018).
B. Empirical Comparison of FID and KID
The study compares FID and KID across models and finds very strong rank correlation on LSUN-BEDROOM and CELEBA-HQ-128. It therefore treats either metric as practically usable for the studied applications.
- The Spearman correlation between KID and FID is approximately 0.994 for LSUN-BEDROOM and 0.995 for CELEBA-HQ-128.
- The study compares the overlap between models ranked in the top K by FID and KID for K ∈ 5, 10, 20, 50, 100.
- The experiment suggests that FID and KID are very strongly correlated, so practical applications can choose either metric and FID-based conclusions should transfer to KID-based studies.
C.2. ResNet Architecture
The ResNet CIFAR architecture is documented through its discriminator architecture, residual-block definition, and associated implementation components. The materials identify mean pooling, batch normalization, residual resampling, and path merging as key elements.
- The RS column records residual-block resampling as downscale, upscale, or none, while MP and BN denote mean pooling and batch normalization.
- Each residual block merges a shortcut path with one convolution and a second path with two convolutions.
- The architecture materials include the ResNet19, ResNet CIFAR, and residual-block specifications.
D. ResNet Architecture Ablation Study
The study tests minor ResNet implementation differences through ablations on CIFAR10. CIN produced the worst FID, but combining it with OPT restored performance to the level of the other variants.
- Experimental setup: The ablation study evaluates six minor ResNet implementation differences on CIFAR10.The experiment codes include DEFAULT, SKIP, CIN, OPT, CIN OPT, SUM, TAN, EPS, and ALL.
- Results: CIN obtained the worst FID score among the ablation variants.
- Results: Combining CIN with OPT improved results to the same level as the other variants.The paper attributes this improvement to avoiding a three-channel bottleneck in the first block.
- Results: Overall, the tested ResNet differences had minor impact on CIFAR10.
E. Recommended Hyperparameter Settings
The paper recommends hyperparameter settings for three regularization regimes and compares model stability using FID across architectures and datasets. Spectral normalization performs best among the normalizers on SNDCGAN and ResNet CIFAR, while layer and spectral normalization both work well on ResNet19.
- Recommended settings: The authors propose best parameters without regularizers, with one regularizer, and with at most two regularizers.Tables 8–10 summarize the top two parameters for SNDCGAN, ResNet19, and ResNet CIFAR, ranked by median FID across five seeds.
- Evaluation: Models are ranked by median FID across five random seeds, while best-seed FID is listed separately to show stability.Ranking by the best FID across seeds can produce better but unstable results.
- Normalization results: Spectral normalization outperforms other normalizers on SNDCGAN and ResNet CIFAR architectures.
- Normalization results: Layer normalization and spectral normalization both work well on ResNet19.
- Visual results: Generated examples are selected from the best-FID run and supplemented with two plots having higher FID scores.Figures 8–10 show examples on CELEBA-HQ-128, LSUN-BEDROOM, and CIFAR10.
- Hyperparameter analysis: Figure 11 presents heatmaps of learning rate, β1, β2, ndisc, and λ for each architecture–dataset combination.