Source-linked AI summary
Are GANs Created Equal? A Large-Scale Study
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, Olivier Bousquet
TL;DR
The paper asks how GAN algorithms can be compared objectively when likelihood metrics are unavailable and existing comparisons differ in metrics and computational budget. It conducts a neutral, large-scale evaluation using FID and precision–recall-based tasks, finding that algorithmic differences become less relevant as computational budget increases. The authors consequently advocate more systematic, neutral evaluation while noting important scope and metric limitations.
Problem
There is no clear consensus on which GAN algorithms perform objectively better because metrics and computational budgets are not consistently controlled.
Method
The study conducts a fair, comprehensive comparison using FID, precision, recall, and F1 across controlled architectures, hyperparameter searches, random seeds, data sets, and computational budgets.
Results
Nearly all tested GANs can reach similar FID values with sufficient computational budget, and no algorithm clearly dominates others as budget increases.
Takeaways & Limitations
GAN comparisons should report distributions of results at fixed computational budgets rather than only the minimum FID achieved.
Takeaways & Limitations
The study explores only a subset of comparison conditions and cannot exclude significant differences under unexplored settings or more complex, higher-resolution data sets.
Abstract
from arXiv · showhide
Generative adversarial networks (GAN) are a powerful subclass of generative models. Despite a very rich research activity leading to numerous interesting GAN algorithms, it is still very hard to assess which algorithm(s) perform better than others. We conduct a neutral, multi-faceted large-scale empirical study on state-of-the art models and evaluation measures. We find that most models can reach similar scores with enough hyperparameter optimization and random restarts. This suggests that improvements can arise from a higher computational budget and tuning more than fundamental algorithmic changes. To overcome some limitations of the current metrics, we also propose several data sets on which precision and recall can be computed. Our experimental results suggest that future GAN research should be based on more systematic and objective evaluation procedures. Finally, we did not find evidence that any of the tested algorithms consistently outperforms the non-saturating GAN introduced in \cite{goodfellow2014generative}.
1 Introduction
GANs learn a generator–discriminator game to match a data distribution, but comparing the many proposed algorithms remains difficult because metrics and computational budgets vary. The study addresses this with a fair, comprehensive evaluation and introduces additional evaluation tasks.
- GANs learn a deterministic generator transformation that aims to match the data distribution against a discriminator distinguishing real from generated samples.
- No clear consensus identifies which GAN algorithms perform objectively better, partly because evaluation metrics and computational budgets are inconsistent.
- Qualitative sample comparisons are subjective and potentially misleading because the model likelihood is generally unavailable for GANs.
- The study provides a fair, comprehensive comparison of state-of-the-art GANs and finds that nearly all can reach similar FID values with sufficient computational budget.
2 Background and Related Work
GAN evaluation relies on classifier-based metrics because likelihood-based evaluation is difficult in high-dimensional settings. IS and FID offer useful comparisons but retain important limitations, motivating broader evaluation procedures.
- Likelihood-based evaluation is often intractable for complex, high-dimensional data, making quantitative GAN evaluation a central challenge.
- Inception Score rewards confident predictions and diverse marginal labels, but it is insensitive to the prior label distribution and is not a proper distance.
- FID compares Gaussian approximations of real and generated Inception embeddings using their Fréchet distance.
- FID can detect intra-class mode dropping and is more robust to noise than IS, but both metrics fail to detect overfitting.
- This study differs from a smaller IS-focused comparison by evaluating state-of-the-art GANs with FID, precision, and recall in a large-scale experiment.
3 Flavors of Generative Adversarial Networks
The study focuses on unconditional GANs and compares objectives that differ in discriminator output constraints, losses, and gradient penalties. These design choices define several major GAN variants.
- Unconditional GANs learn from unlabeled data, while their optimization problems differ in discriminator output constraints, losses, and gradient norm penalties.
- The original formulation includes minimax and non-saturating losses, with the latter maximizing the probability that generated samples are classified as real.
- WGAN uses an unbounded discriminator and Wasserstein objective, while WGAN-GP replaces weight clipping with a gradient-norm constraint encouraging 1-Lipschitz behavior.
4 Challenges of a Fair Comparison
Fair GAN comparison requires choices about metrics, reporting targets, architectures, hyperparameters, random seeds, data sets, and computational budgets. The study therefore evaluates FID alongside precision and recall while systematically varying comparison conditions.
- Which metric to use?: The study evaluates FID across all data sets and precision, recall, and F1 on proposed tasks because likelihood metrics cannot be applied.
- Which metric to use?: FID shows slight bias but low variance at sample size 10000, extreme sensitivity to mode dropping, and Spearman’s ρ = 0.9 across InceptionNet and VGG on CELEBA for FID < 200.
- How to compare models?: Model scores can vary with architecture, hyperparameters, random initialization, and data set, so best, average, median, and worst scores represent different comparison targets.
- Pragmatic comparison choices: The study uses a shared architecture, explores data-set-specific and transferred hyperparameter searches, reports seed confidence intervals, evaluates four data sets, and varies the hyperparameter budget.
- Hyperparameter optimization: Random-search hyperparameter optimization is necessary in practice, while sequential Bayesian optimization produced comparable results.
5 Metrics
The paper evaluates FID alongside precision, recall, and F1, while proposing manifold-based data sets that make precision and recall computable. These complementary measures expose quality, diversity, and overfitting limitations that IS and FID may miss.
- Metrics: FID and Inception Score cannot detect overfitting, so the proposed precision-and-recall method complements rather than replaces them.A memory GAN storing all training samples could score perfectly under both existing measures.
- Metrics: Figure 2 illustrates that high precision can coexist with low recall, whereas high recall can coexist with low precision.The examples associate low recall with limited diversity and low precision with failure to capture convexity.
- Metrics: FID robustness is assessed by partitioning data into empirical data and model distributions drawn from separate random subsets.The study measures bias, variance, encoding-network sensitivity, and sensitivity to mode dropping.
- Metrics: Precision measures generated-sample proximity to the data manifold, while recall measures recovery of samples across that manifold.The proposed tasks define these quantities through efficiently computable distances to constructed manifolds.
- Metrics: The proposed convex-polygon data sets provide controlled-complexity tasks for evaluating generative models with precision and recall.Their manifold distances are efficiently computable, turning evaluation into distance estimation.
- Metrics: Distance to the manifold is approximated by constrained gradient descent over polygon vertices, repeated five times from random initializations.For generated samples, latent representations are obtained by minimizing reconstruction error over z.
6 Large-scale Experimental Evaluation
The study compares GANs under controlled architectures, broad hyperparameter searches, computational budgets, and repeated random initializations. Results show no consistently dominant model and indicate that score distributions and tuning budgets matter more than isolated best scores.
- Hyperparameter sensitivity: 100-sample wide hyperparameter searches reveal extreme training sensitivity and no significantly more stable model.The experiments use common architectural choices and comparable expressive power to support fair comparisons.
- Experimental design: The experiments select models through large-scale hyperparameter searches, evaluate best FID during training, and estimate stability with 50 different initialization seeds.FID is computed every five epochs from 10,000 generated and test samples during the search.
- Budget and model comparison: No algorithm clearly dominates, and limited budgets prevent statistically significant model comparisons.The study also finds that a model considered “bad” can outperform a “good” model when trained with a larger budget.
- Results: Table 2 shows that performance depends heavily on the data set and no model strictly dominates the others.Asterisks identify significant outlier runs, including severe mode collapse or training failures.
- Precision, recall, and F1: When optimizing F1, NS GAN and WGAN achieve high precision and recall, whereas BEGAN and VAE perform underwhelmingly on this data set.Figure 5 reports maximum F1 distributions with 95% confidence intervals across computational budgets.
- Results: Better scores can result from wider hyperparameter searches, making distributions of obtained scores more informative than a model’s single best score.Hyperparameter ranges transfer across data sets for some models, such as NS GAN, but are more sensitive for others, such as WGAN.
- Random initialization: Most models are relatively robust to random initialization, but variance remains significant and should be considered when comparing models.LSGAN is the exception, showing lower robustness to initialization changes.
7 Limitations of the Study
The study’s conclusions are bounded by the datasets, architectures, metrics, and hyperparameter-search conditions it explores. The authors identify unresolved risks in FID and practical limitations in transferring or selecting hyperparameters.
- Data sets, neural architectures, and optimization issues: Higher-resolution or more complex datasets may require larger networks, potentially producing larger quantitative differences between methods.The study uses one architecture that achieves good FID on all considered datasets.
- Metrics: FID may be unstable under major changes to the encoding network and may be fooled by artifacts tailored to that network.The authors also note that FID cannot detect overfitting to the training dataset.
- Exploring the space of hyperparameters: Hyperparameter optimization remains necessary, while transfer-based tuning can be unfair or impractical when recommended settings are unavailable or dataset-specific.Sequential Bayesian optimization is identified as a possible future alternative but is beyond the study’s computational scope.
8 Conclusion
The paper proposes neutral, fair GAN comparisons using FID, precision, recall, and F1, emphasizing result distributions rather than minimum scores. Its evidence suggests algorithmic differences become less relevant with greater computational budgets, while unexplored conditions remain possible.
- Conclusion: The study recommends comparing distributions of the minimum achievable FID under a fixed computational budget instead of reporting only the minimum FID.This addresses randomness in optimization and model instability.
- Conclusion: As computational budgets increase, empirical differences among state-of-the-art GAN algorithms become less relevant.With limited compute, a supposedly weaker algorithm might still outperform a supposedly stronger one.
- Conclusion: The study explores only a subset of model-comparison conditions, so it cannot exclude significant performance differences under unexplored settings.The authors therefore support more systematic experiments and neutral comparison grounds.
A Wide and narrow hyperparameter ranges
The study uses wide and narrow hyperparameter searches to examine parameter sensitivity, transferability, FID distributions, and the effect of random initialization across four datasets. Narrow ranges are selected from wide-search results, while repeated retraining exposes variation across seeds.
- Search ranges: Wide and narrow searches sample hyperparameters uniformly or on a log-scale, with narrow ranges optimized from the FASHION-MNIST wide search.Table 3 defines the wide search, while Table 4 uses 50 samples per model in the narrow search.
- Sensitivity analysis: Scatter plots for FASHION-MNIST, MNIST, CIFAR10, and CELEBA estimate how each hyperparameter affects final FID and guide narrow-range selection.The plots organize algorithms by column and parameters by row.
- FID distributions: FID distributions and corresponding samples are reported for MNIST, FASHION-MNIST, CIFAR10, and CELEBA across models.These figures connect metric distributions with generated-image quality.
- Narrow-range search: Narrow-range tuning can produce practical differences between models because some parameter ranges transfer across datasets while others are more sensitive to the dataset.The comparison allows 50 hyperparameter samples per model.
- Random initialization: Repeated retraining with fixed best hyperparameters produces slight final-FID variance, making the full result distribution more informative than the best seed.Each model is retrained 50 times with random weight initialization.
E Precision, Recall and F1 as a Function of the Budget
The study evaluates precision, recall, and F1 under different optimization thresholds. The supplied figures compare F1 at thresholds δ = 1.0 and δ = 0.5, and precision at δ = 1.0.
- F1: F1 is evaluated with threshold δ = 1.0.
- F1: F1 is also evaluated with threshold δ = 0.5.
- Precision: Precision is evaluated with threshold δ = 1.0.
F Impact of the optimization algorithm.
Across 100 WGAN hyperparameter settings, ADAM and RMSPROP produced similar score distributions, while ADAM performed better on the recommended parameters. The experiments also examine how precision, recall, and F1 relate to FID under different optimization targets and thresholds.
- 100 hyperparameter settings were tested for WGAN training using ADAM and RMSPROP, with similar score distributions and no clear overall optimizer winner.
- ADAM outperformed RMSPROP on recommended parameters: CIFAR10 154.5 vs 161.2 and CELEBA 97.9 vs 216.3.
- The results highlight the need for hyperparameter search when comparing optimization algorithms.
- Precision and recall were evaluated under thresholds δ = 0.5 and δ = 1.0 across the reported optimization settings.
- The proposed measure is particularly suitable for detecting a loss in recall when examining its correlation with FID, precision, recall, and F1.