PaperlayerView exact revision on arXiv

Source-linked AI summary

Differentiable Augmentation for Data-Efficient GAN Training

Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, Song Han

arXiv:2006.10738v4cs.CVcs.GRcs.LG

TL;DR

GAN performance deteriorates with limited training data, largely because the discriminator memorizes the training set, while direct augmentation can alter the target distribution. DiffAugment applies the same differentiable augmentations to real and fake samples for both networks, achieving consistent gains across architectures and settings, including FID 6.80 with IS 100.8 on ImageNet 128×128.

  • Problem

    Limited GAN training data causes severe performance degradation, while collecting large datasets can require substantial effort, cost, or impossible-to-obtain examples.

  • Method

    DiffAugment applies the same differentiable augmentations to real and fake images during both generator and discriminator training.

  • Results

    FID 6.80 with IS 100.8 is achieved on ImageNet 128×128, with consistent gains across GAN architectures, datasets, and loss functions.

  • Takeaways & Limitations

    DiffAugment supports data-efficient GAN training, including top CIFAR-10 and CIFAR-100 performance with 20% data and high-quality generation from 100 images without pre-training.

Abstract

from arXiv · show

The performance of generative adversarial networks (GANs) heavily deteriorates given a limited amount of training data. This is mainly because the discriminator is memorizing the exact training set. To combat it, we propose Differentiable Augmentation (DiffAugment), a simple method that improves the data efficiency of GANs by imposing various types of differentiable augmentations on both real and fake samples. Previous attempts to directly augment the training data manipulate the distribution of real images, yielding little benefit; DiffAugment enables us to adopt the differentiable augmentation for the generated samples, effectively stabilizes training, and leads to better convergence. Experiments demonstrate consistent gains of our method over a variety of GAN architectures and loss functions for both unconditional and class-conditional generation. With DiffAugment, we achieve a state-of-the-art FID of 6.80 with an IS of 100.8 on ImageNet 128x128 and 2-4x reductions of FID given 1,000 images on FFHQ and LSUN. Furthermore, with only 20% training data, we can match the top performance on CIFAR-10 and CIFAR-100. Finally, our method can generate high-fidelity images using only 100 images without pre-training, while being on par with existing transfer learning algorithms. Code is available at https://github.com/mit-han-lab/data-efficient-gans.

1 Introduction

Limited training data causes GAN performance to deteriorate because discriminators memorize the training set, while conventional augmentation can alter the target distribution or disrupt training. DiffAugment addresses these issues by applying the same differentiable augmentations to real and fake images, improving data efficiency across settings.

  • Motivation: Limited data is a fundamental GAN challenge because collecting large, diverse, high-quality datasets is costly or sometimes impossible.Examples include rare species and images of specific people or landmarks.
  • Motivation: With 10% of CIFAR-10 data, BigGAN’s FID rises soon after training begins, then the model collapses as discriminator training accuracy saturates and validation accuracy falls.The pattern indicates discriminator memorization of the exact training set and failure to generalize.
  • Motivation: Augmenting only real images shifts the target distribution, while augmenting real and fake images can disrupt the balance between generator and discriminator objectives.Observed consequences include artifacts from masked or unnaturally colored regions and poor convergence.
  • Contribution: DiffAugment applies the same differentiable augmentation to real and fake images during both generator and discriminator training.The method propagates gradients through augmentation, regularizes the discriminator without manipulating the target distribution, and maintains training balance.
  • Results: FID reaches 6.80 with IS 100.8 on ImageNet 128×128, while FID falls 2–4× for 1,000-image FFHQ and LSUN settings.The paper also reports matching top CIFAR-10 and CIFAR-100 performance with 20% training data and competitive 100-image generation without pre-training.

2 Related Work

Prior GAN research emphasizes stability, architectures, and training strategies, while data efficiency remains less studied. The paper distinguishes DiffAugment from standard regularization and classifier augmentation because GANs must learn the data distribution itself.

  • Generative Adversarial Networks: Earlier GAN work improved objectives, architectures, and training strategies, but paid less attention to data efficiency under limited data and labels.The paper studies the more challenging setting where both data and labels are limited.
  • Regularization for GANs: GAN regularization methods penalize sudden local changes in discriminator outputs, whereas DiffAugment encourages discriminator performance under varied augmentations.The paper presents augmentation as a complementary perspective to existing regularization techniques.
  • Data Augmentation: Classifier augmentations preserve labels, but applying them directly to generative models alters the distribution they are intended to learn.DiffAugment is introduced as a strategy to address this distributional concern.

3 Method

GANs use a generator to map latent vectors into samples and a discriminator to distinguish those generated samples from real observations. Training alternates optimization of discriminator and generator losses.

  • GAN training: The generator G maps an input latent vector z, typically sampled from a Gaussian distribution, to an output G(z).This defines the generator’s sample-production role in the GAN pipeline.
  • GAN training: The discriminator D distinguishes generated samples G(z) from real observations x.Its role is to separate the generator’s outputs from the target data observations.
  • GAN training: Standard GAN training alternates optimization of the discriminator loss LD and generator loss LG using loss functions fD and fG.The equation and prose describe this alternating optimization framework.

D D(T(G(z)))

DiffAugment addresses discriminator overfitting by applying the same differentiable augmentation to real and generated samples throughout GAN training. This preserves gradient flow to the generator while avoiding the distribution shift and optimization imbalance caused by one-sided alternatives.

  • Discriminator memorization widens the training–validation accuracy gap, produces uninformative gradients, and destabilizes GAN training.
  • Augment reals only: Augmenting real images only makes the generator learn the transformed distribution, introducing artifacts and distribution shift.Reported artifacts include unnatural color and cutout holes, while stronger transformations significantly worsen performance.
  • Augment D only: Augmenting discriminator inputs only creates unbalanced optimization: D recognizes augmented images above 90% accuracy but recognizes unaugmented generated images below 10%.The generator then receives no useful information from the discriminator and training diverges.
  • DiffAugment: DiffAugment applies the same differentiable transformation to real samples, fake samples, and generator training, allowing gradients to pass through the augmentation to G.The transformation must be differentiable with respect to its input; the same random function is used, but random seeds need not match.
  • DiffAugment: Stronger DiffAugment policies can reduce the discriminator’s training–validation accuracy gap and improve convergence, with Translation, Cutout, and Color used individually or in composition.On CIFAR-10, composing Cutout and Translation further boosts BigGAN, while Color combinations remain robust under the strongest policy.
  • Evaluation: ImageNet 128×128 evaluation measures IS and FID on 50k samples against the validation distribution, selecting each method’s best-FID snapshot over three runs.

4 Experiments

Experiments across GAN architectures, datasets, supervision settings, and data regimes show that DiffAugment consistently improves generation, especially when training data is limited. Analysis indicates its gains persist beyond reducing model capacity or strengthening regularization.

  • ImageNet: ImageNet experiments show significant gains under 25% data, where the baseline undergoes early collapse, while DiffAugment advances state-of-the-art FID and IS with 100% data.
  • Limited-data benchmarks: With 1k, 5k, 10k, or 30k images, DiffAugment improves StyleGAN2 on FFHQ and LSUN-Cat and is on par with ADA.The fixed policy is Color + Translation + Cutout, applied without hyperparameter changes.
  • Benchmark coverage: DiffAugment improves BigGAN, CR-BigGAN, and StyleGAN2 across architectures, regularizations, and loss functions without hyperparameter changes.The reported experiments cover ImageNet, CIFAR-10, CIFAR-100, FFHQ, LSUN-Cat, and low-shot datasets.
  • Low-shot generation: Using 100 images without pre-training, DiffAugment achieves results on par with transfer learning methods trained with tens of thousands of images, except on 100-shot Obama.The exception reflects better generalization from pre-training on human faces.
  • Model size: At one-quarter BigGAN channel capacity, DiffAugment reaches FID 21.57 versus the baseline’s minimum FID 29.02, and its advantage grows with model size.
  • Regularization: Increasing StyleGAN2’s R1 strength to γ = 10^3 improves baseline FID from 34.05 to 26.87, but the best baseline remains 1.8× worse than DiffAugment.At γ = 10^4, performance deteriorates after reaching minimum FID 29.14.
  • Augmentation choices: Rotation, Gaussian-noise, and geometry-transform policies all outperform the baseline, while Color + Translation + Cutout is the strongest reported policy.

5 Conclusion

The conclusion presents DiffAugment as a data-efficient GAN training method whose benefits extend across architectures, supervision settings, objectives, and datasets. It emphasizes joint augmentation of real and fake samples and differentiability as central design choices.

  • DiffAugment applies augmentation to both real and fake samples to prevent discriminator overfitting, while differentiability enables training of both generator and discriminator.
  • Extensive experiments show benefits across StyleGAN2 and BigGAN, multiple supervision settings and objectives, and ImageNet, CIFAR, FFHQ, LSUN, and 100-shot datasets.
  • The method is especially effective when limited data is available.

Broader Impact

The paper frames data-efficient GAN training as a way to make generative modeling more accessible when abundant data is unavailable. Limited data may result from rare incidents, privacy concerns, or historical visual records.

  • DiffAugment aims to make generative modeling accessible to users and fields without abundant data.
  • Rare incidents, privacy concerns, and historical visual data are identified as reasons that real-world datasets may be limited.

A.1 ImageNet Experiments

The ImageNet appendix describes implementation choices and qualitative comparisons for BigGAN and StyleGAN2, alongside CIFAR results and a comparison with CR-BigGAN. DiffAugment improves image quality in limited-data settings, while reported comparisons depend on evaluation and implementation details.

  • ImageNet implementation: The BigGAN reimplementation uses random scaling, center cropping, and horizontal flips; random cropping produced worse IS.
  • BigGAN comparisons: DiffAugment improves BigGAN sample quality at both 25% and 100% data, with a clearer visual difference at 25%.
  • CR-BigGAN comparison: CR-BigGAN reports FID 6.66 versus DiffAugment’s 6.80 with 100% data, but the comparison lacks available code, pre-trained models, and reported IS for CR-BigGAN.
  • CIFAR results: On CIFAR-100, 20% data roughly matches StyleGAN2’s FID and exceeds its IS.
  • CIFAR analysis: With 10% CIFAR-10 data, StyleGAN2 plus DiffAugment reaches IS 8.84 versus 7.75 for the baseline across an R1-regularization sweep.
  • StyleGAN2 comparisons: DiffAugment considerably improves StyleGAN2 image quality when limited data is available.

A.3 CIFAR-10 and CIFAR-100 Experiments

The experiments apply DiffAugment to BigGAN, CR-BigGAN, and StyleGAN2 on CIFAR, largely preserving baseline settings while documenting regularization choices. Using 20% of the data, the method matches top unconditional-generation performance on CIFAR-10 and CIFAR-100.

  • Experimental setup: CIFAR experiments replicate BigGAN and CR-BigGAN with batch size 50, four discriminator steps per generator step, and learning rate 2 × 10^-4 for both networks.Models run on 2 GPUs for up to 250k iterations on CIFAR-10 and 500k on CIFAR-100.
  • Experimental setup: DiffAugment is applied to BigGAN, CR-BigGAN, and StyleGAN2 without changing their baseline settings.The experiments also document interactions with gradient penalties and consistency regularization.
  • Results: 20% data matches the top unconditional-generation performance on CIFAR-10 and CIFAR-100.The paper refers readers to Figure 10 and Tables 6–7 for the quantitative results.

Appendix B Evaluation Metrics

The evaluation uses official TensorFlow implementations of Inception v3 to measure FID and IS across methods and datasets. For ImageNet and CIFAR datasets, generated-sample counts follow the Compare GAN evaluation setting.

  • Metrics: FID and IS are measured with the official TensorFlow Inception v3 model for all methods and datasets.
  • Evaluation protocol: ImageNet, CIFAR-10, and CIFAR-100 use as many generated samples as real images in the validation set.This setting is inherited from the Compare GAN codebase.

Appendix C 100-Shot Generation Benchmark

The 100-shot benchmark uses manually filtered and cropped Internet images and compares low-data generation across several datasets and methods. The appendix includes qualitative, interpolation, and nearest-neighbor evaluations, along with revisions across preprint versions.

  • Dataset construction: The 100-shot datasets are collected from the Internet and manually filtered and cropped before use.The full datasets are stated to be available through the paper’s accompanying materials.
  • Version history: Later preprint versions added StyleGAN2 results, reran low-shot baselines with batch size 16, added landmark interpolation results, reran MineGAN models, and analyzed DiffAugment choices.Another revision noted a different FID evaluation protocol used by concurrent works.
  • Qualitative comparisons: On FFHQ and LSUN-cat with 1k, 5k, 10k, and 30k training images, DiffAugment is reported to consistently outperform StyleGAN2 baselines across data percentages.
  • Overfitting analysis: Style-space interpolation on six 100-shot datasets suggests little overfitting without pre-training.The datasets include Obama, grumpy cat, panda, the Bridge of Sighs, the Medici Fountain, and the Temple of Heaven.
  • Overfitting analysis: Pixel-space and LPIPS nearest-neighbor tests compare generated images with horizontally flipped training images and report that generated images differ from the training set.These tests are used to assess whether the model simply memorizes training examples.
Loading 2006.10738v4…