PaperlayerView exact revision on arXiv

Source-linked AI summary

The GAN is dead; long live the GAN! A Modern GAN Baseline

Yiwen Huang, Aaron Gokaslan, Volodymyr Kuleshov, James Tompkin

arXiv:2501.05441v1cs.LGcs.CV

TL;DR

GANs are widely viewed as difficult to train because instability, mode dropping, and ad-hoc architectural tricks remain common. The paper introduces a regularized relativistic loss with local convergence guarantees, then uses it to build the minimalist R3GAN baseline, which compares favorably across multiple datasets and model families.

  • Problem

    GAN training is considered difficult because minimax optimization can diverge, reduce sample diversity, and require poorly understood empirical tricks.

  • Method

    The paper combines RpGAN with zero-centered gradient penalties and replaces StyleGAN2’s non-essential features with modern backbone designs to construct R3GAN.

  • Results

    R3GAN compares favorably by FID with StyleGAN, other SOTA GANs, and diffusion models across FFHQ, ImageNet, CIFAR, and Stacked MNIST.

  • Takeaways & Limitations

    The results support a simpler GAN baseline with increased stability, modern architectures, and fewer ad-hoc training tricks.

  • Takeaways & Limitations

    R3GAN is not suitable for direct downstream use in image editing or controllable generation because it lacks image-inversion and disentangled-synthesis features.

Abstract

from arXiv · show

There is a widely-spread claim that GANs are difficult to train, and GAN architectures in the literature are littered with empirical tricks. We provide evidence against this claim and build a modern GAN baseline in a more principled manner. First, we derive a well-behaved regularized relativistic GAN loss that addresses issues of mode dropping and non-convergence that were previously tackled via a bag of ad-hoc tricks. We analyze our loss mathematically and prove that it admits local convergence guarantees, unlike most existing relativistic losses. Second, our new loss allows us to discard all ad-hoc tricks and replace outdated backbones used in common GANs with modern architectures. Using StyleGAN2 as an example, we present a roadmap of simplification and modernization that results in a new minimalist baseline -- R3GAN. Despite being simple, our approach surpasses StyleGAN2 on FFHQ, ImageNet, CIFAR, and Stacked MNIST datasets, and compares favorably against state-of-the-art GANs and diffusion models.

1 Introduction

The paper challenges the view that GANs require brittle tricks by combining a regularized loss with modernized architectures. It introduces R3GAN, a streamlined baseline that compares favorably across several generative-model benchmarks.

  • GAN optimization is difficult because minimax training can diverge and generated samples can lose diversity through mode collapse.
  • Popular GAN backbones rely on poorly understood empirical tricks and lag behind modern architectures common in diffusion models.
  • The proposed objective augments RpGAN with zero-centered gradient penalties and has the same local-convergence guarantee as regularized classic GANs.
  • The authors develop a simpler backbone by removing non-essential StyleGAN features and borrowing designs from modern ConvNets and transformers.
  • R3GAN compares favorably by FID with StyleGAN, other SOTA GANs, and diffusion models across FFHQ, ImageNet, CIFAR, and Stacked MNIST.

2 Serving Two Masters: Stability and Diversity with RpGAN +R1 + R2

The paper combines relativistic pairing with R1 and R2 gradient penalties to address GAN stability and diversity together. The resulting regularized objective has local convergence guarantees and empirically improves stability, mode coverage, and KL divergence on StackedMNIST.

  • GAN objectives must address both training stability and sample diversity, motivating a theoretically grounded regularizer for RpGAN.
  • The classic GAN objective can encourage mode collapse or mode dropping because empirical updates move all fake samples just beyond one discriminator boundary.
  • RpGAN compares real and fake critic outputs pairwise, changing the loss landscape and addressing mode dropping.
  • Unregularized RpGAN does not always converge, including for data distributions approaching a delta distribution, so additional regularization is necessary.
  • R1 penalizes discriminator gradients on real data, while R2 penalizes them on fake data; together they regularize both distributions.
  • RpGAN with either R1 or R2 is locally convergent under assumptions similar to prior analyses, because the regularized Jacobian has eigenvalues with negative real parts.
  • With both R1 and R2, classic GAN and RpGAN training became stable; RpGAN achieved full mode coverage and reduced DKL from 0.9270 to 0.0781.
  • R1 alone can fail early because regularizing real data does not sufficiently control discriminator behavior on fake data, allowing gradient explosion.

3 A Roadmap to a New Baseline — R3GAN

The roadmap strips StyleGAN2 to a minimal baseline, then restores stability with a well-behaved loss and modernizes the backbone through ResNet and ConvNeXt-inspired design choices. This progression yields stable training and improved performance while retaining comparable model capacity.

  • Minimum baseline: Config B removes StyleGAN2’s style-based features, image-manipulation enhancements, and training tricks, retaining only the raw backbone and basic image generation.The removed components include mapping and style injection, weight modulation/demodulation, noise injection, mixing and path-length regularization, and several optimization tricks.
  • Minimum baseline: 12.46 FID is obtained by the simplified Config B on FFHQ-256, while training remains stable and sample quality remains reasonable.Config B reduces z to dimension 64 and lowers the learning rate from 2.5×10^-3 to 5×10^-5 after removing equalized learning rate.
  • Bottleneck modernization: The modernized architecture uses increased width with grouped convolution and an inverted bottleneck, while retaining fewer activations and separate resampling layers.Large-kernel depthwise convolution, GELU, normalization changes, and patchify stems are not adopted as generally beneficial choices in this setting.
  • Network architecture details: The final minimalist design uses symmetric 25 M-parameter generators and discriminators, with one transition layer and two residual blocks per resolution stage.Bilinear resampling and optional 1×1 convolutions handle spatial-size and channel changes, while fix-up initialization addresses variance explosion without normalization.

4 Experiments Details

Experiments show that stabilizing the loss enables architectural modernization, yielding R3GAN results that outperform StyleGAN2 and compare favorably with GAN and diffusion baselines across several datasets.

  • Roadmap on FFHQ-256: Config A reaches FID 7.52 on FFHQ-256, while removing StyleGAN2 tricks in Config B raises FID to 12.46.
  • Roadmap on FFHQ-256: Config D’s modernized generator and discriminator achieve FID 9.95, while retaining StyleGAN2 output skips worsens FID to 10.17.
  • Roadmap on FFHQ-256: Config E reaches FID 7.51 after widening grouped-convolution bottlenecks and 7.05 after inverting stem and bottleneck dimensions, surpassing StyleGAN2.
  • FFHQ-256: On FFHQ-256, R3GAN outperforms existing StyleGAN methods and four diffusion-based methods without using the bCR trick.
  • FFHQ-64: On FFHQ-64, R3GAN outperforms EDM with fewer than half its parameters and one function evaluation.
  • CIFAR-10: On CIFAR-10, R3GAN outperforms many GANs despite relatively small capacity, using 40 M combined generator and discriminator parameters.
  • ImageNet: On ImageNet-32 and ImageNet-64, R3GAN remains competitive with smaller discriminators or models and one-step sampling, including outperforming larger diffusion models on ImageNet-64.

5 Discussion and Limitations

The discussion presents R3GAN as a simple, easily convergent baseline rather than a universal solution, with explicit limits for downstream functionality, scalability, and societal risk.

  • The paper argues that simplifying GANs around a stable RpGAN + R1 + R2 objective can produce diverse outputs and competitive FID across four datasets.
  • R3GAN prioritizes simplicity over functionality and is not claimed to outperform every existing model on every dataset or task.
  • The model is unsuitable for direct image editing or controllable generation because it lacks dedicated inversion and disentangled-synthesis features.
  • The scalability evaluation has not yet verified performance on higher-resolution ImageNet or large-scale text-to-image generation.
  • The paper notes that generative models, especially models of people, can enable personalized deepfakes and spread disinformation.

6 Conclusion

R3GAN combines a provably locally convergent regularized relativistic loss with modern architectures, eliminating commonplace GAN tricks while achieving competitive performance within its parameter size class.

  • R3GAN is a new baseline GAN with increased stability, modern architectures, and no commonplace ad-hoc tricks.Its regularized relativistic loss is presented as central to the approach and is argued to improve training stability.

Appendices

GAN training can be analyzed as a dynamical system whose local convergence depends on Jacobian eigenvalues, or, for small learning rates, on the gradient-field Jacobian spectrum.

  • GAN training is formulated as a dynamical system with update operator Fh(θ, ψ) = (θ, ψ) + hv(θ, ψ).Here, h is the learning rate and v is the gradient vector field.
  • Jacobian eigenvalues with absolute value above 1 imply non-convergence, while values below 1 imply linear convergence.If all eigenvalues have absolute value 1, convergence behavior is undetermined.
  • For sufficiently small h, local training behavior can instead be assessed using eigenvalues of the gradient vector field Jacobian.Negative real parts imply local linear convergence; positive real parts imply non-convergence, while zero real parts are inconclusive.

B DiracRpGAN: A demonstration of non-convergence

The DiracRpGAN example analytically demonstrates non-convergence without regularization: its equilibrium has imaginary-axis eigenvalues and trajectories circle indefinitely. Adding gradient regularization gives all eigenvalues negative real parts and yields local convergence for small learning rates.

  • DiracRpGAN uses a univariate Dirac generator, a linear discriminator, and true data concentrated at zero.Its training objective is then specified for this simplified problem setting.
  • Without regularization, DiracRpGAN has a unique equilibrium at θ = ψ = 0 with Jacobian eigenvalues ±f′(0)i on the imaginary axis.The convergence behavior is therefore inconclusive from the eigenvalue criterion, and the paper proceeds to show analytically that training does not converge.
  • Every unregularized gradient-field trajectory preserves θ(t)^2 + ψ(t)^2, so its distance from the equilibrium remains constant.Consequently, training runs in circles rather than converging.
  • With gradient regularization, all eigenvalues have negative real parts when γ > 0, making gradient descent locally convergent for small enough learning rates.The analysis investigates both R1 and R2 regularization forms.

C General Convergence Results

The general analysis extends prior convergence proofs to gradient-penalized RpGAN under stated regularity, equilibrium, and discriminator-expressiveness assumptions. It proves local convergence to the generator and discriminator reparameterization manifolds at least linearly for sufficiently small learning rates.

  • Assumptions: The proofs adapt Mescheder et al.’s techniques to RpGAN without requiring supp pD = supp pθ.The analysis also considers the realizable case, where some generator reproduces the true data distribution.
  • Assumptions: At equilibrium, RpGAN permits the discriminator to equal any constant C on the data support rather than requiring zero logits.This reflects that RpGAN is defined on critic differences rather than raw logits.
  • Assumptions: The convergence theorem assumes f′(0) ≠ 0, f′′(0) < 0, local manifold regularity, and a discriminator strong enough to detect generator deviations.The discriminator condition is identified as the only assumption about discriminator expressiveness.
  • Analysis: The regularized gradient field is analyzed with either R1 or R2 regularization, and the result extends trivially to applying both.The proof uses the Jacobian restricted orthogonally to the reparameterization manifolds.
  • Result: For small enough learning rates, gradient descent for both regularized fields converges locally to MG × MD at least linearly.This is the main general convergence result for gradient-penalized RpGAN.

D Hyperparameters, training configurations, and compute

The experiments specify implementation, augmentation, conditioning, capacity, precision, and training-schedule choices, with compute requirements varying substantially across datasets.

  • Training uses StyleGAN3 support code for EMA, non-leaky augmentation, and metric evaluation, while the loss and models are implemented from scratch.
  • A cosine burn-in schedule adjusts learning rate, regularization strengths, Adam β2, EMA half-life, and augmentation probability before target values.Lower initial Adam β2 helps adapt to the initial large learning rate, while large initial γ smooths separated real and fake distributions.
  • The models use horizontal flips and non-leaky augmentation where enabled, including pixel blitting, geometric and color transforms, and cutout.
  • Capacity follows resolution-specific allocations, with two residual blocks per resolution and wider or deeper models than StyleGAN2 at comparable model sizes.The ImageNet configuration doubles channels, producing roughly 4× as many parameters as default StyleGAN2.
  • Training uses mixed precision only at the four highest resolutions, with parameters retained in FP32; BFloat16 works whereas FP16 cripples training.
  • Class-conditional generation concatenates latent and class embeddings in G and uses a projection discriminator in D without normalization-based conditioning.
  • Training required 7 hours for Stacked MNIST, 4 days for CIFAR-10, roughly 3 weeks for FFHQ, and about 5000 H100 hours for ImageNet.

E Negative Results and Future Work

The study reports several design choices that underperformed or complicated the minimalist model, while qualitative examples document generation across datasets.

  • Negative Results: GELU, Swish, and SMU considerably deteriorated FID when applied to both G and D.
  • Negative Results: Leaky ReLU is retained because activations with little negative-region gradient can sparsify D’s gradient and reduce useful information reaching G.The underperformance of SMU remains unexplained despite its smooth approximation to leaky ReLU.
  • Negative Results: Group normalization did not improve FID or training stability, although the authors limit their claim to normalization layers standardizing activation-map statistics.
  • Negative Results: Applying R1 and R2 lazily once every 8 minibatches slightly worsened FFHQ and CIFAR-10 FID and caused complete convergence failure on Stacked MNIST and toy datasets.
  • Qualitative Examples: The appendix includes qualitative sample-generation examples for Stacked-MNIST, FFHQ, CIFAR-10, ImageNet-32, and ImageNet-64.

G Training Curves

The paper presents training curves for CIFAR-10, FFHQ, and ImageNet configurations, while checklist material bounds claims to specific datasets and reports reproducibility details without variance bars.

  • Training Curves: Figures 11–15 provide training curves for CIFAR-10, FFHQ-64, FFHQ-256, ImageNet-32, and ImageNet-64.
  • Claims and Scope: The paper’s stability, convergence, and state-of-the-art GAN claims are supported by figures, appendices, and Section 4, with claims bound to specific datasets.
  • Reproducibility: Supplemental tables and sections disclose the hyperparameters and training configurations needed to reproduce and understand the experiments.
  • Reproducibility: The submission had no code available at submission time, although the authors aimed to release it by publication with reproduction instructions.
  • Statistical Significance: The experiments do not report error bars because individual runs require many days or weeks of computation.
Loading 2501.05441v1…