PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Generative Adversarial Networks with Limited Data

Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, Timo Aila

arXiv:2006.06676v2cs.CVcs.LGcs.NEstat.ML

TL;DR

Limited training data makes GAN discriminators overfit, causing training to diverge and making ordinary augmentation risky because transformations can leak into generated images. The paper introduces adaptive discriminator augmentation, which adjusts augmentation strength to control overfitting while avoiding leakage. Across datasets, it enables strong results with a few thousand images and improves CIFAR-10 FID from 5.59 to 2.42.

  • Problem

    Small datasets cause discriminator overfitting and divergent GAN training, while dataset augmentation can leak unwanted transformations into generated images.

  • Method

    Adaptive discriminator augmentation adjusts the strength of stochastic discriminator augmentations according to an overfitting heuristic during training.

  • Results

    Across several datasets, good results are possible with only a few thousand images, often matching StyleGAN2 with an order of magnitude fewer images; CIFAR-10 FID improves from 5.59 to 2.42.

  • Takeaways & Limitations

    Adaptive discriminator augmentation reliably stabilizes GAN training and improves result quality when training data is scarce.

  • Takeaways & Limitations

    Augmentation is not a substitute for real data, so large, high-quality training sets should be collected before relying on augmentation to fill gaps.

Abstract

from arXiv · show

Training generative adversarial networks (GAN) using too little data typically leads to discriminator overfitting, causing training to diverge. We propose an adaptive discriminator augmentation mechanism that significantly stabilizes training in limited data regimes. The approach does not require changes to loss functions or network architectures, and is applicable both when training from scratch and when fine-tuning an existing GAN on another dataset. We demonstrate, on several datasets, that good results are now possible using only a few thousand training images, often matching StyleGAN2 results with an order of magnitude fewer images. We expect this to open up new application domains for GANs. We also find that the widely used CIFAR-10 is, in fact, a limited data benchmark, and improve the record FID from 5.59 to 2.42.

1 Introduction

Limited data causes discriminator overfitting and divergent GAN training, while ordinary augmentation can leak distortions into generated images. The paper introduces adaptive augmentation to prevent overfitting without leakage and reports strong results with few images.

  • Small datasets make the discriminator overfit, rendering its feedback to the generator meaningless and causing training to diverge.
  • Standard dataset augmentation can increase classifier invariance, but GANs may learn to generate the augmented distribution instead.
  • The proposed approach uses diverse augmentations and adaptive control to prevent discriminator overfitting without leaking augmentations into generated images.
  • A few thousand images often match StyleGAN2 results with an order of magnitude fewer images.
  • FID improves from 5.59 to 2.42 on CIFAR-10.

2 Overfitting in GANs

GAN training on small datasets deteriorates as the discriminator overfits, while augmentation-based alternatives risk leaking transformations into generated images. The paper studies leakage-safe stochastic discriminator augmentation and its adaptive control across data regimes.

  • Overfitting in GANs: StyleGAN2 training eventually sees rising FID on small FFHQ subsets as discriminator outputs for real and generated images drift apart.Separate validation accuracy also drops, further evidencing overfitting.
  • Overfitting in GANs: Balanced consistency regularization adds discriminator consistency terms but can still permit augmentation leakage into generated images.
  • Stochastic discriminator augmentation: Stochastic discriminator augmentation evaluates both discriminator and generator training using augmented images, without adding separate consistency-regularization loss terms.
  • Designing augmentations that do not leak: Stochastic augmentations can avoid leakage when skipped with non-zero probability, whereas rotation at excessive probability can produce randomly oriented images.For the tested rotation, correct orientation was maintained below approximately p = 0.85; color-augmentation leaks were unlikely below p = 0.8.
  • Our augmentation pipeline: The pipeline contains 18 transformations across six categories, with each transformation independently applied or skipped using the shared probability p.
  • Our augmentation pipeline: With 2k training images, pixel blitting and geometric transforms provide most of the benefit, while stronger augmentation reduces overfitting but slows convergence.
  • Our augmentation pipeline: The best augmentation probability depends heavily on dataset size, motivating adaptive control instead of fixed-probability tuning.

3 Adaptive discriminator augmentation

ADA dynamically adjusts augmentation strength using an overfitting heuristic, aiming to keep discriminator overfitting at a chosen level. It stabilizes training across dataset sizes by preserving useful discriminator feedback and detailed generator gradients.

  • Overfitting measurement: The heuristics measure discriminator behavior over minibatch averages, with N = 4 corresponding to 256 images.The approach uses outputs on training, validation, and generated images to quantify overfitting.
  • Adaptive control: ADA adjusts augmentation probability p every four minibatches according to an overfitting heuristic, increasing or decreasing it toward a target.The adjustment step is chosen so p can rise from 0 to 1 sufficiently quickly, and p is clamped at zero.
  • Adaptive control: The authors compare validation-based rv with training-set-based rt, selecting rt with a target value of 0.6 for subsequent tests.Both heuristics prevent overfitting and outperform the best fixed augmentation probability found by grid search.
  • Adaptive control: With a 2k training set, ADA applies augmentations almost always near the end of training, while fixed augmentation is too strong initially and too weak later.FID begins deteriorating after p ≈0.5 in this extreme case, indicating a practical limit associated with leaky augmentations.
  • Training behavior: Using ADA, training converges regardless of FFHQ training-set size, overfitting no longer occurs, and the discriminator continues providing detailed gradients to the generator.Without augmentation, discriminator gradients become simplistic as attention narrows to a handful of features.

4 Evaluation

The evaluation compares ADA with baselines and alternatives across training from scratch, transfer learning, and several small datasets. ADA substantially improves limited-data performance, avoids augmentation leakage, and remains effective in transfer settings.

  • Training from scratch: ADA substantially improves FID across FFHQ and LSUN CAT as training-set size decreases, while bCR is effective mainly when data scarcity is less severe.The comparison spans multiple training-set sizes and reports FID distributions over runs.
  • Training from scratch: bCR’s integer xy-translations leak into generated images and blur their average, whereas ADA’s diverse augmentations do not leak.The qualitative comparison uses networks trained on a 5k FFHQ subset.
  • Training from scratch: ADA outperforms the wider set of tested alternatives, including PA-GAN, WGAN-GP, zCR, auxiliary rotations, spectral normalization, shallow mapping, and adaptive dropout.The authors report extensive parameter tuning for the comparison methods.
  • Transfer learning: Transfer learning performs better than training from scratch, with success depending primarily on source-dataset diversity rather than subject similarity.FFHQ transfers equally well from CELEBA-HQ or the more diverse LSUN DOG, while LSUN CAT requires comparable diversity from LSUN DOG.
  • Transfer learning: With small target datasets, ADA almost completely prevents the divergence seen during baseline transfer learning, while Freeze-D alone does not.Freeze-D provides a small but reliable improvement when combined with ADA.
  • Small datasets: ADA achieves a CIFAR-10 FID of 2.42 versus 5.59 for the prior state of the art and an IS of 10.24 versus 9.58 in the class-conditional setting.The authors note that CIFAR-specific architecture tuning significantly affected results and that FID is biased for small datasets.

5 Conclusions

The paper concludes that ADA stabilizes GAN training and improves result quality when data is scarce, while emphasizing that augmentation cannot replace collecting real data. It also reports negligible per-model energy impact and notes comparability issues with parallel-work evaluations.

  • Conclusions: ADA reliably stabilizes training and substantially improves result quality when training data is in short supply.The conclusion frames this as the central demonstrated outcome.
  • Conclusions: Augmentation is not a substitute for real data, so the authors recommend collecting a large, high-quality dataset before filling gaps with augmentation.This is presented as a scope boundary for using ADA.
  • Practical considerations: Enabling ADA has a negligible effect on the energy consumption of training a single model.The paper’s total project consumption was 325 MWh, or 135 single-GPU years, mostly from comparisons and parameter sweeps.
  • Comparison scope: Direct comparison with parallel augmentation papers is difficult because their CIFAR-10 FID-10k evaluations use different generated and validation-image counts.The authors therefore state that those FID-10k values are not directly comparable with Figure 11b FIDs.

Broader impact

The work enables high-quality generative image models with substantially less data, primarily addressing how much data generative models need. Its applied contribution is improved efficiency rather than fundamentally new capabilities, with limited direct impact on broader computer-vision debates.

  • Broader impact: Generative models are foundational tools for learning computational recipes that generate complex data from examples.They also support computer-vision and graphics applications as priors and regularizers.
  • Broader impact: The methods enable high-quality generative image models to be trained with significantly less data than existing approaches.The paper frames this as a contribution to understanding how much data generative models need to learn relevant relationships.
  • Broader impact: The work primarily contributes to efficiency and does not introduce fundamental new capabilities.The authors therefore expect limited effects on broader discussions around surveillance, authenticity, and privacy.
  • Broader impact: The algorithm does not provide direct controls for high-level attributes or direct modification of existing images.The authors expect advances from this work and others to improve such models over time.
  • Broader impact: The contributions lower the barrier to applying GAN-type models with custom image collections in applied research fields.The paper gives biological-specimen modeling as an example of a field that may benefit.

A Additional results

Additional experiments show that ADA produces convincing images across several limited-data datasets and degrades more gracefully as training data decreases. Comparisons and ablations further identify augmentation choices associated with improved results and document leakage risks of alternative approaches.

  • Additional results: ADA yields excellent results across METFACES, BRECAHAD, and AFHQ CAT, DOG, and WILD, while non-ADA StyleGAN2 suffers discriminator-overfitting-related quality loss.The figures include randomly selected generated images and corresponding real training images; both transfer learning and training from scratch are represented.
  • Additional results: With ADA, image quality and diversity remain reasonable on FFHQ subsets with 5k training images, whereas non-augmented methods degrade noticeably below 30k images.The comparison includes BigGAN, StyleGAN2, and the authors’ baseline.
  • Additional results: Approximately 50% of BigGAN training runs failed to converge, while StyleGAN2 runs produced nearly identical FID values across runs.The comparison reports median, minimum, and maximum FID as a function of training-set size.
  • Additional results: ±45° rotations clearly illustrate that bCR can leak similarly transformed images into generation, making its effect fundamentally similar to dataset augmentation.Either approach may be used when such leaks are acceptable.
  • Additional results: Blit + Geom + Color produced the best results with a 2k training set and remained competitive with larger training sets.Figure 21 evaluates individual and cumulative augmentation categories as augmentation probability increases.
  • Additional results: The augmentation pipeline contains 18 transformations spanning geometric, color, filtering, and corruption categories, designed to be non-leaking, diverse, and high-quality.The pipeline is implemented on the GPU.

B.1 Geometric and color transformations

The pipeline applies geometric and color transformations through composed homogeneous matrices, while implementation details address interpolation, borders, and transformation diversity. Its theoretical design relies on augmentation operators that do not leak the target distribution, although the mathematical conditions may be intricate in practice.

  • Geometric and color transformations: Pixel blitting uses x-flips, 90° rotations, and integer translations accumulated in a homogeneous 3 × 3 matrix G.The matrix maps input pixel coordinates to output coordinates, with the image origin at its center.
  • Geometric and color transformations: Geometric transformations include isotropic and anisotropic scaling, arbitrary rotation, and fractional translation, with scale parameters sampled log-normally.Rotations are split before and after anisotropic scaling so it can act in directions beyond coordinate axes.
  • Geometric and color transformations: The combined geometric transformation uses dynamically reflected padding, higher-resolution processing, bilinear interpolation, and downsampling to reduce border and resampling artifacts.The implementation selects wavelet filters to balance resampling quality and computational cost.
  • Geometric and color transformations: Color transformations collect brightness, contrast, saturation, luma-flip, and hue-rotation parameters in a homogeneous 4 × 4 matrix applied per pixel.The output color vector is obtained by multiplying this matrix by the input RGB vector augmented with a homogeneous coordinate.
  • Geometric and color transformations: The transformation pipeline includes additional filtering, additive RGB noise, and cutout operations alongside geometric and color transformations.The appendix figure illustrates individual transformations at several parameter percentiles.
  • Non-leaking augmentations: Invertible augmentation operators do not leak because matching augmented distributions then guarantees matching underlying distributions.The paper defines leakage as the existence of x ≠ y satisfying T x = T y, and notes that the practical mathematical conditions can be intricate.

C.4.2 Transformation group augmentations

Transformation-group augmentations are non-leaking when their stochastic operator is invertible, which is characterized by a nonzero Fourier transform of the mixture weights. Zero patterns, uniform mixtures, and interactions with dataset symmetries can permit leaks.

  • Transformation groups: Flips, translations, rotations, scalings, shears, and color transformations can form transformation groups whose stochastic mixtures are non-leaking under suitable weights.The result applies to finitely generated abelian groups and requires a non-degenerate mixture distribution.
  • Invertibility: An augmentation operator is invertible when an inverse weighting vector exists such that the product of the inverse and augmentation operators equals the identity.For cyclic groups, the inverse weights are obtained by reciprocal Fourier coefficients when those coefficients are nonzero.
  • Invertibility: The inverse exists only when every Fourier coefficient of the augmentation probability weights is nonzero.This condition prevents the augmentation operator from having a null space that can support leaked transformations.
  • Leak conditions: Uniform or periodic group mixtures create Fourier zeros and can cause the generator to learn transformed distributions rather than the true distribution.Leaks may involve group elements, subgroup elements, or mixtures conditional on properties of the data distribution.
  • Leak conditions: Failure of invertibility can be harmless for distributions already invariant to the leaked transformation, but this may change after augmentations are composed.The paper gives upright-face data as an example where a rotation leak would be harmless, while partial invariances introduced upstream can alter that conclusion.
  • Scope: Invertibility is sufficient but not always necessary for non-leaking behavior, particularly for non-compact groups with restricted null spaces.For continuous shifts, the paper hypothesizes that some technically non-invertible operators may not leak, while near-zero spectrum values could still permit contrived leaks.

C.5.1 Conditioning

The conditioning analysis distinguishes exact invertibility from practical numerical stability. Safe augmentation probabilities must avoid configurations where the operator becomes singular or poorly conditioned, while implementation details can affect whether transformations behave as groups.

  • Conditioning: A technically invertible augmentation operator can still fail numerically when it is sufficiently close to a non-invertible configuration.The condition number is infinite for singular matrices and large when inversion is unstable in practice.
  • Conditioning: Regulating augmentation probabilities away from p = 1 provides a safety margin against exact non-invertibility.For many augmentations, uniform sampling over compact group elements corresponds to the problematic p = 1 configuration.
  • Conditioning: For finite-group transformations applied with probability p, the condition number is 1/(1 − p), which diverges as p approaches one.The smallest singular value is 1 − p and the largest is 1 in the stated finite-state example.
  • Implementation boundary: Naive image-grid implementations may violate exact group composition because repeated resampling and boundary effects alter transformations.The implementation accumulates affine transformations before applying them and uses reflection padding to mitigate these issues.

D Implementation details

The implementation builds on the official StyleGAN2 TensorFlow code while retaining most architectural and regularization details. Experiments use GPU clusters and standard generative-image evaluation metrics.

  • Software and architecture: The techniques were implemented on top of the official StyleGAN2 TensorFlow implementation with most network and regularization details unchanged.The retained components include weight demodulation, path-length and lazy regularization, style mixing, filtering, equalized learning rates, and minibatch standard deviation.
  • Compute and evaluation: Experiments ran on a cluster of NVIDIA DGX-1 systems with Tesla V100 GPUs using TensorFlow 1.14.0, PyTorch 1.1.0, CUDA 10.0, and cuDNN 7.6.3.The official pretrained Inception network was used to compute FID, KID, and Inception score.
  • Configuration: Figure 24 lists the experiment-specific hyperparameters alongside the original StyleGAN2 configuration.The figure serves as the implementation reference for comparing settings across experiments.

D.1 Hyperparameters and training configurations

Training configurations vary by dataset, resolution, and batch size, while evaluation and regularization settings follow explicit procedures. Dataset augmentation is generally avoided except where symmetry makes horizontal flips appropriate.

  • Training configurations: Training usually continues until the discriminator has seen 25M real images, while CIFAR-10 uses 2 GPUs and 100M images.Runs generally use 8 GPUs and minibatch size 64, reduced to 32 for METFACES to avoid GPU-memory exhaustion.
  • Dataset augmentation: Dataset augmentation is omitted for FFHQ, LSUN CAT, and CIFAR-10 except in specified cases, while symmetric datasets enable horizontal flips.The authors avoid leaky augmentation on CIFAR-10 because it could unfairly inflate benchmark results.
  • Network capacity: Network capacity follows the original StyleGAN2 configuration for high-resolution datasets, with 512 feature maps in CIFAR-10 and reduced capacity for some 256 × 256 experiments.The reductions facilitate extensive sweeps over dataset sizes.
  • Optimization: Learning rates are selected by grid search, favoring the highest rate that avoids instability, and larger minibatches permit slightly higher rates.Generator moving-average decay is parameterized with respect to minibatches rather than individual images.
  • Regularization: The optimal R1 regularization weight γ varies from 0.01 to 10 and appears to scale almost linearly with dataset resolution.A proposed initial guess is γ0 = 0.0002 · N/M, but the authors recommend dataset-specific experimentation.
  • Numerical precision: Mixed-precision FP16/FP32 training is used throughout nearly all experiments, storing trainable parameters in FP32 while evaluating the networks in FP16.This uses Volta Tensor Cores but requires attention to FP16’s narrower numerical range.

D.2 Comparison methods

The paper compares several discriminator regularization, consistency, architectural, and transfer-learning methods against its baseline, while also testing implementation-specific alternatives. These comparisons identify mapping-network depth and adaptive dropout as practical design choices.

  • PA-GAN adds an auxiliary checksum task whose bit count increases when overfitting is detected, using the rt heuristic with target 0.95.
  • Comparison methods include spectral normalization, Freeze-D, WGAN-GP, balanced and latent consistency regularization, PA-GAN, auxiliary rotations, and adaptive dropout.The methods target discriminator regularization, consistency, transfer learning, auxiliary tasks, or overfitting control.
  • The BigGAN comparison uses an unconditional implementation with self-modulation instead of class-conditional BatchNorm and without class projection in the discriminator.
  • Reducing mapping-network depth from 8 to 2 produced slightly better results than depth 8, while further reduction was consistently inferior.The authors therefore identify depth 2 as a good default choice.
  • Adaptive dropout applies multiplicative Gaussian dropout throughout the discriminator and dynamically adjusts its standard deviation using the rt heuristic with target 0.6.The resulting p is used directly as the value of σ.

E Energy consumption

The project reports computation in GPU-years and electricity consumption, emphasizing the substantial resource cost of exploration and extensive training sweeps. ADA itself is reported not to significantly change the cost of training a single model.

  • Computational effort is reported as single-GPU years using a Volta-class GPU, despite varying use of NVIDIA DGX-1 systems across project stages.
  • Approximately 300 megawatt hours of electricity were consumed by the entire project.Almost half of the total energy was spent exploring and shaping ideas before paper production.
  • Most post-exploration computation was directed toward extensive sweeps shown in the paper’s figures.
  • Figure 25 presents computational effort and electricity consumption data for the project in GPU-years on a single NVIDIA V100 GPU.The project would have required approximately 135 years on one GPU.
Loading 2006.06676v2…