PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Andrew Brock, Jeff Donahue, Karen Simonyan

arXiv:1809.11096v2cs.LGstat.ML

TL;DR

Generating diverse, high-fidelity images from complex datasets such as ImageNet remained difficult. This paper scales GAN training, introduces architectural and regularization changes, and uses truncation to trade variety for fidelity, achieving state-of-the-art ImageNet synthesis with IS 166.5 and FID 7.4 at 128×128.

  • Problem

    GANs had limited fidelity and variety on complex datasets such as ImageNet, leaving a gap between generated and real-world images.

  • Method

    The paper scales GANs with architectural and regularization changes, analyzes large-scale instabilities, and applies truncation to trade sample variety for fidelity.

  • Results

    BigGANs substantially improve ImageNet class-conditional synthesis, reaching IS 166.5 and FID 7.4 at 128×128 resolution.

  • Takeaways & Limitations

    Scaling improves the fidelity and variety of generated natural images, establishing a new performance level among ImageNet GAN models.

  • Takeaways & Limitations

    Training remains unstable: enforcing stability can substantially reduce performance, while achieving better final results may require allowing late-stage collapse.

Abstract

from arXiv · show

Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal. To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities specific to such scale. We find that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick," allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator's input. Our modifications lead to models which set the new state of the art in class-conditional image synthesis. When trained on ImageNet at 128x128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.5 and Frechet Inception Distance (FID) of 7.4, improving over the previous best IS of 52.52 and FID of 18.6.

1 INTRODUCTION

The paper targets the gap between GAN-generated and real ImageNet images by scaling GANs, introducing architectural and regularization changes, and analyzing large-scale instabilities. These modifications enable truncation-based control of fidelity versus variety and substantially improve class-conditional synthesis results.

  • The authors aim to narrow the fidelity and variety gap between GAN-generated images and real-world ImageNet images.
  • Scaling models to two to four times more parameters and eight times the batch size, alongside architectural and regularization changes, improves scalability and performance.
  • The modifications make GANs compatible with the truncation trick, enabling fine-grained control over the trade-off between sample variety and fidelity.
  • Large-scale GANs exhibit specific instabilities; combining novel and existing techniques reduces them, but complete stability substantially harms performance.
  • 166.5 IS and 7.4 FID improve the ImageNet 128×128 state of the art from 52.52 IS and 18.65 FID.

2 BACKGROUND

The section introduces GANs as adversarial generator–discriminator models trained through a min-max objective, while emphasizing their brittleness and the range of stabilization, architectural, conditioning, and evaluation approaches developed around them.

  • GAN foundations: GANs use a Generator to map random noise to samples and a Discriminator to distinguish real from generated samples within a two-player min-max objective.The original formulation seeks a Nash equilibrium.
  • GAN foundations: For images, GANs typically use convolutional networks, with latent noise drawn from distributions such as N(0, I) or U[−1, 1].Without auxiliary stabilization, training is notoriously brittle and requires finely tuned hyperparameters and architectural choices.
  • Stabilization approaches: Stabilization research modifies GAN objectives or constrains the Discriminator to encourage convergence and improve training stability.The section situates these approaches within growing empirical and theoretical insights.
  • Stabilization approaches: Spectral Normalization enforces Lipschitz continuity on the Discriminator, while generator performance has been linked to the conditioning of its Jacobian.Spectral Normalization normalizes parameters using running estimates of their first singular values.
  • Architectural approaches: Architectural approaches include self-attention for modeling global structure and progressively training high-resolution GANs across increasing resolutions.These examples are represented by SA-GAN and ProGAN, respectively.
  • Conditional GANs: Conditional GANs incorporate class information through noise concatenation, conditional BatchNorm gains and biases, or class-embedding-based Discriminator evidence.These mechanisms support generation or discrimination conditioned on class information.

3 SCALING UP GANS

Scaling larger batches and wider models substantially improves GAN performance, while shared embeddings reduce computational costs and accelerate training. Orthogonal regularization makes truncation-based quality–variety control practical, though large-scale training can still collapse.

  • Scaling model and batch size: An 8× larger batch improves state-of-the-art IS by 46%, while covering more modes and providing better gradients for both networks.The scaled models also reach better final performance in fewer iterations but become unstable and undergo collapse.
  • Scaling model and batch size: Increasing layer width by 50%, approximately doubling parameters in both networks, yields a further 21% IS improvement.The authors attribute this gain to increased model capacity relative to dataset complexity.
  • Architectural changes: A shared class embedding projected to each layer’s gains and biases reduces computation and memory costs and improves training speed by 37%.This replaces separate embedding layers for conditional BatchNorm.
  • The truncation trick: Sampling from a truncated normal instead of the training distribution immediately boosts IS and FID while improving individual sample quality.The truncation trick resamples latent values whose magnitude exceeds a chosen threshold.
  • Orthogonal Regularization: Orthogonal Regularization increases the share of models amenable to truncation from 16% to 60%.The selected penalty uses β = 10^-4 and removes diagonal terms while minimizing pairwise filter cosine similarity without constraining norms.
  • Scaling limitations: Large-scale models can reach 512×512 resolution without explicit multiscale methods, but training collapse necessitates early stopping.The authors report that current GAN techniques support distributed, large-batch training and substantially improve the state of the art.

4 ANALYSIS

The analysis attributes GAN instability to poorly conditioned generator and discriminator spectra whose interaction drives collapse. Spectral constraints improve conditioning but cannot reliably prevent collapse without substantial performance loss, making delayed collapse a practical compromise.

  • Generator conditioning: The top singular values of generator weight matrices, especially in an over-complete first layer, grow throughout training and explode at collapse.The analysis monitors σ0, σ1, and σ2 as potential precursors of collapse.
  • Generator conditioning: Generator spectral regularization or clamping prevents singular-value growth and explosion but no tested combination prevents training collapse.These techniques can mildly improve performance, suggesting improved conditioning is helpful but insufficient.
  • Discriminator conditioning: Discriminator spectra are noisy, with singular values growing during training and jumping at collapse rather than exploding.Smooth Frobenius norms indicate that the noise is concentrated in the top few singular directions, possibly from strongly perturbing generator batches.
  • Discriminator conditioning: 45% reduction in IS results from the default γ=10 gradient penalty despite stabilized, smoother, and bounded spectra; γ=1 still reduces IS by 20%.Lowering the penalty partially alleviates performance degradation but produces increasingly ill-behaved spectra.
  • Overall conclusion: Stability arises from generator–discriminator interaction: conditioning is necessary but insufficient, while strong discriminator constraints stabilize training at a dramatic performance cost.Relaxing conditioning and allowing collapse later can yield better final performance after the model is sufficiently trained.

5 EXPERIMENTS

The experiments evaluate BigGAN on ImageNet across multiple resolutions and quality–variety settings, showing that BigGAN-deep improves substantially over BigGAN. Additional analyses test memorization, characterize failure modes, and demonstrate stable, transferable performance on the much larger JFT-300M dataset.

  • ImageNet evaluation: BigGAN is evaluated on ImageNet at 128×128, 256×256, and 512×512 resolutions using IS and FID across three quality–variety settings.The models’ samples are shown in Figure 4, with additional samples in the appendix and online.
  • Architecture comparison: BigGAN-deep substantially outperforms BigGAN across all tested resolutions and metrics, indicating that increased depth improves sample quality across architectures.BigGAN-deep is four times deeper and uses a different residual-block configuration.
  • Memorization analysis: The experiments test memorization through class-wise nearest-neighbor analyses in pixel and pretrained-classifier feature spaces, alongside sample and class-wise interpolations.These analyses hold z constant for class-wise interpolations.
  • Failure modes: Partially trained models exhibit class leakage and class-dependent difficulty, in addition to familiar failure modes such as local artifacts, texture blobs, and mode collapse.Class leakage occurs when images from one class contain properties of another.
  • JFT-300M evaluation: 292M images from JFT-300M, spanning 8.5K labels, provide a dataset two orders of magnitude larger than ImageNet for testing scalability.The full dataset contains 300M images labeled with 18K categories, while the experiments retain the 8.5K most common labels.
  • JFT-300M evaluation: JFT-300M models remain stable for many hundreds of thousands of iterations and improve over a baseline without changing the underlying models, training, or regularization beyond expanded capacity.Unlike ImageNet models, their training does not collapse without heavy regularization, suggesting larger datasets may partially alleviate GAN stability issues.

6 CONCLUSION

Scaling up GANs for multi-category natural-image modeling improves both sample fidelity and variety, establishing a new performance level among ImageNet GANs. The paper also analyzes large-scale GAN training behavior and stability through the singular values of their weights.

  • 6 CONCLUSION: Scaling up GANs improves the fidelity and variety of generated natural-image samples across multiple categories.The conclusion states that large-scale training benefits both dimensions of sample quality.
  • 6 CONCLUSION: The resulting models improve ImageNet GAN performance by a large margin and establish a new state-of-the-art level.The passage characterizes this as a new level of performance among ImageNet GAN models.
  • 6 CONCLUSION: The paper analyzes large-scale GAN training behavior and characterizes training stability using the singular values of the models’ weights.This analysis concerns the training behavior specific to large-scale GANs.

APPENDIX A ADDITIONAL SAMPLES, INTERPOLATIONS, AND NEAREST NEIGHBORS FROM IMAGENET MODELS · APPENDIX B ARCHITECTURAL DETAILS

Appendix A presents BigGAN samples, interpolations, and nearest-neighbor comparisons across ImageNet models and resolutions, while Appendix B specifies BigGAN and BigGAN-deep architectural variants. The appendices also describe modeling difficulties for globally structured, high-resolution classes and architectural changes across resolutions.

  • APPENDIX A ADDITIONAL SAMPLES, INTERPOLATIONS, AND NEAREST NEIGHBORS FROM IMAGENET MODELS: BigGAN generates samples at 256×256 and 512×512 resolutions.Figures 5 and 6 show samples at these two resolutions.
  • APPENDIX A ADDITIONAL SAMPLES, INTERPOLATIONS, AND NEAREST NEIGHBORS FROM IMAGENET MODELS: 512×512 modeling is easier for textural, common classes than for unaligned human faces or crowds.The latter involve more dynamic global structure and details to which human observers are more sensitive.
  • APPENDIX A ADDITIONAL SAMPLES, INTERPOLATIONS, AND NEAREST NEIGHBORS FROM IMAGENET MODELS: Interpolations vary both z and c, and pose semantics are frequently maintained when interpolating c with z held constant.The figure also indicates that grayscale is encoded in the joint z, c space rather than in z alone.
  • APPENDIX A ADDITIONAL SAMPLES, INTERPOLATIONS, AND NEAREST NEIGHBORS FROM IMAGENET MODELS: Nearest neighbors are compared in VGG-16-fc7, ResNet-50-avgpool, and pixel spaces.In each figure, the generated image appears in the top left.
  • APPENDIX B ARCHITECTURAL DETAILS: BigGAN uses the ResNet GAN architecture of Zhang et al. (2018), with a modified discriminator channel pattern and a shared class embedding in G.The discriminator’s first convolution in each block matches the output-filter count rather than the input-filter count.
  • APPENDIX B ARCHITECTURAL DETAILS: BigGAN-deep concatenates the entire z with the class embedding and passes the resulting vector to each residual block through skip connections.It uses residual blocks with bottlenecks incorporating additional 1 × 1 convolutions.
  • APPENDIX B ARCHITECTURAL DETAILS: The 256 × 256 BigGAN architecture adds a ResBlock at 16×16 and moves G’s non-local block to 128 × 128, while memory constraints retain D’s placement.These changes are relative to the 128 × 128 architecture.
  • APPENDIX B ARCHITECTURAL DETAILS: The 512 × 512 BigGAN architecture adds a ResBlock at 512 × 512 and moves both networks’ non-local blocks back to 64 × 64.These changes are relative to the 256 × 256 architecture and are forced by memory constraints.

APPENDIX C EXPERIMENTAL DETAILS … LATENTS

The appendix specifies BigGAN’s optimization and normalization setup, documents batch-size-invariant sampling through standing statistics, and reports benchmark scores. It also evaluates latent-space choices, finding that several sparse or discrete designs outperform the standard Gaussian prior while low-dimensional latents remain viable.

  • APPENDIX C EXPERIMENTAL DETAILS: BigGAN uses SA-GAN architectures with non-local blocks, orthogonal initialization, Adam, and constant learning rates of 2 · 10−4 for D and 5 · 10−5 for G.The models use TensorFlow and the architectures detailed in Appendix B.
  • C.1 BATCHNORM STATISTICS AND SAMPLING: Standing statistics address BatchNorm’s batch-size dependence by aggregating activation moments over typically 100 noise batches, making outputs invariant to batch size and device count.Test batch size can drastically change performance, especially with exponential moving averages of G’s weights.
  • C.2 CIFAR-10: CIFAR-10 achieves an IS of 9.22 and an FID of 14.73 without truncation.The experiment uses the settings from Table 1, row 8.
  • C.3 INCEPTION SCORES OF IMAGENET IMAGES: ImageNet’s training and validation IS values are 233 and 166 at 128×128, 377 and 234 at 256×256, and 348 and 241 at 512×512, respectively.The discrepancy is attributed to the Inception classifier being trained on the training data.
  • APPENDIX D ADDITIONAL PLOTS: Appendix plots compare IS and FID at 128×128, 256 pixels, 512 pixels, and on JFT-300M at 256×256 across truncation values and model capacities.Scores are averaged across three random seeds where specified, and JFT-300M curves correspond to rows in Table 3.
  • APPENDIX E CHOOSING LATENT SPACES: Latent-space experiments find successful training with z ∈R8 and only a minimal performance drop at z ∈R32, compared with the default z ∈R128.The authors caution that comparison with single-class networks using z ∈R512 is improper because the datasets and network settings differ.
  • LATENTS: Bernoulli {0, 1} improves IS by 8% over N(0, I) and requires 60% fewer iterations, while Censored Normal improves IS by 15-20% and tends to require fewer iterations.Censored Normal is max (N(0, I), 0), combining sparsity with continuous nonnegative variation.

APPENDIX F MONITORED TRAINING STATISTICS · APPENDIX G ADDITIONAL DISCUSSION: STABILITY AND COLLAPSE

The appendices investigate training stability and collapse through monitored statistics and additional stability experiments. Unmodified models collapse after 200000 iterations, while tested regularization methods avoid collapse but achieve limited maximum Inception Scores.

  • APPENDIX F MONITORED TRAINING STATISTICS: 200000 iterations precede collapse in a typical model without special modifications.
  • APPENDIX F MONITORED TRAINING STATISTICS: The monitored statistics cover both generator and discriminator behavior across unmodified, regularized, gradient-penalized, and dropout models.
  • APPENDIX F MONITORED TRAINING STATISTICS: 125000 iterations precede collapse when σ0 in G is regularized towards 1, in both G and D statistics.
  • APPENDIX F MONITORED TRAINING STATISTICS: An R1 Gradient Penalty of strength 10 on D prevents collapse but reaches a maximum IS of 55.
  • APPENDIX F MONITORED TRAINING STATISTICS: Dropout with keep probability 0.8 on D’s last feature layer prevents collapse but reaches a maximum IS of 70.
  • APPENDIX G ADDITIONAL DISCUSSION: STABILITY AND COLLAPSE: Appendix G presents additional investigations into model stability, expanding upon the discussion in Section 4.

G.1 INTERVENING BEFORE COLLAPSE · G.2 SPIKES IN THE DISCRIMINATOR’S SPECTRA

Collapse occurs abruptly, but interventions show that keeping the discriminator optimal and favoring it over the generator can delay or prevent breakdown. The discriminator’s spectra exhibit noisy, slowly decaying behavior that may reflect memorization and attenuated real-data gradients, motivating alternative loss designs.

  • G.1 INTERVENING BEFORE COLLAPSE: Sample quality can fall from peak to minimum within a few hundred iterations, while exploding generator singular values detect collapse without a consistent threshold.This motivated testing checkpoints taken several thousand iterations before collapse.
  • G.1 INTERVENING BEFORE COLLAPSE: Intervention experiments used collapsed-model checkpoints ten or twenty thousand iterations before collapse to test altered training setups, timing, and final performance.The experiments measured whether collapse occurred, when it occurred relative to the original collapse, and performance attained at collapse.
  • G.1 INTERVENING BEFORE COLLAPSE: Doubling learning rates from 2·10−4 in D and 5·10−5 in G to 4·10−4 in D and 1·10−4 in G caused immediate collapse.Changing Adam’s β1 and β2 or resetting momentum vectors generally made no difference or produced limited effects.
  • G.1 INTERVENING BEFORE COLLAPSE: Reducing G’s learning rate while keeping D’s unchanged delayed collapse by over one hundred thousand iterations in some cases but crippled training.Reducing D’s learning rate while keeping G’s unchanged instead led to immediate collapse.
  • G.1 INTERVENING BEFORE COLLAPSE: Freezing G left D stable as both loss components slowly approached zero, whereas freezing D caused G to immediately and dramatically collapse.The experiments support the conclusion that D must remain optimal relative to G for stability and useful gradient information.
  • G.2 SPIKES IN THE DISCRIMINATOR’S SPECTRA: D’s top three singular values contain substantial noise, grow throughout training, respond weakly to collapse, and have a slowly decaying spectrum with the first-two-value ratio centered around one.Figure 29 provides a closeup of D’s spectra at a noise spike.
  • G.2 SPIKES IN THE DISCRIMINATOR’S SPECTRA: D may memorize training data until confident predictions make both the original GAN and hinge losses provide zero gradients on correctly classified examples.Attenuating real-data gradients can eventually bias D toward signals from generated data.
  • G.2 SPIKES IN THE DISCRIMINATOR’S SPECTRA: An unbounded Wasserstein loss was unstable beyond a few thousand iterations despite gradient penalties and optimizer retuning, so the authors explored changing the hinge-loss margin.Spectral Normalization constrains output scale, making the margin selection meaningful.

APPENDIX H NEGATIVE RESULTS

The appendix reports techniques that degraded performance, failed to improve it, or introduced scaling and optimization issues in this specific setup. It cautions that these negative results were not evaluated as thoroughly as the main architectural choices and should not be generalized beyond the tested applications.

  • The authors emphasize that these findings are specific to their particular setup and that the appendix evaluations were less thorough than those for the main architectural choices.The stated purpose was to save future researchers time and provide a fuller account of attempted improvements to performance or stability.
  • Doubling network depth by adding a Residual block after every up- or down-sampling block hampered performance.
  • Sharing class embeddings between G and D scaled poorly and was sensitive to optimization hyperparameters, especially the number of D steps per G step.Initial experiments suggested faster training, but this apparent benefit did not persist at scale.
  • Replacing or removing BatchNorm in G, or adding BatchNorm to D alongside Spectral Normalization, crippled training.The tested replacements included WeightNorm in G and Spectral Normalization without BatchNorm in G.
  • At 128×128, changing attention-block placement provided no noticeable benefit while substantially increasing compute and memory costs.Moving the attention block up one stage benefited the 256×256 setting.
  • Alternative convolutional designs generally failed: dilation degraded performance, bilinear upsampling degraded performance, and filter sizes other than 3 usually degraded performance.A filter size of 5 in G alone provided only a small improvement over baseline at unjustifiable compute cost.

APPENDIX I HYPERPARAMETERS

Hyperparameter sweeps identified learning-rate settings, regularization strengths, and optimizer parameters that shaped performance and training stability. The experiments selected 10^-4 for modified Orthogonal Regularization in G and found trade-offs between stabilization and performance for R1 and DropOut.

  • Learning rates: Halving the learning rates used at higher batch sizes produced the settings adopted for the experiments.At lower batch sizes, the optimal SA-GAN settings were G at 10^-4 and D at 4 · 10^-4; higher-batch-size sweeps tested halving and doubling.
  • Regularization: R1 penalty strength correlated negatively with performance, while settings above 0.5 improved training stability.The sweep covered [10^-3, 10^-2, 10^-1, 0.5, 1, 2, 3, 5, 10].
  • Regularization: DropOut had a stabilizing effect similar to R1 but degraded performance.The final layer of D was swept over keep probabilities [0.5, 0.6, 0.7, 0.8, 0.9, 0.95].
  • Optimizer parameters: D’s Adam β1 provided light regularization without significantly improving results, while higher β1 values crippled training in either network.The sweep for D’s Adam β1 used [0.1, 0.2, 0.3, 0.4, 0.5].
  • Regularization: 10^-4 was selected for the modified Orthogonal Regularization penalty strength in G.The sweep covered [10^-5, 5 · 10^-5, 10^-4, 5 · 10^-4, 10^-3, 10^-2].
Loading 1809.11096v2…