PaperlayerView exact revision on arXiv

Source-linked AI summary

Regularizing Generative Adversarial Networks under Limited Data

Hung-Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, Weilong Yang

arXiv:2104.03310v1cs.LGcs.CV

TL;DR

GANs rely on abundant training data, creating a challenge when only limited data are available. The paper introduces discriminator regularization linked theoretically to LeCam-divergence and finds improved generalization and training stability, including when combined with augmentation. These results support state-of-the-art performance on limited ImageNet data.

  • Problem

    GAN performance depends heavily on large, diverse training datasets, but training with limited data causes generalization and stability challenges.

  • Method

    The paper regularizes discriminator predictions using moving-average anchors and theoretically connects the regularized WGAN objective to LeCam-divergence.

  • Results

    The method improves generalization and stabilizes GAN training under limited data while complementing data augmentation, including for limited ImageNet.

  • Takeaways & Limitations

    Regularization and data augmentation can be combined to enhance GAN performance under limited-data settings.

  • Takeaways & Limitations

    The theoretical result assumes a fixed generator and converged stationary anchor, and requires the regularization weight to satisfy λ < 1 2α.

Abstract

from arXiv · show

Recent years have witnessed the rapid progress of generative adversarial networks (GANs). However, the success of the GAN models hinges on a large amount of training data. This work proposes a regularization approach for training robust GAN models on limited data. We theoretically show a connection between the regularized loss and an f-divergence called LeCam-divergence, which we find is more robust under limited training data. Extensive experiments on several benchmark datasets demonstrate that the proposed regularization scheme 1) improves the generalization performance and stabilizes the learning dynamics of GAN models under limited training data, and 2) complements the recent data augmentation methods. These properties facilitate training GAN models to achieve state-of-the-art performance when only limited training data of the ImageNet benchmark is available.

1. Introduction

GANs achieve high-fidelity image synthesis but depend heavily on large, diverse datasets, and performance degrades with limited data. The paper introduces discriminator regularization that improves generalization, stabilizes training, and complements augmentation under data scarcity.

  • GAN success depends heavily on large, diverse training datasets, which are expensive or cumbersome to collect.
  • Limited training data substantially degrades BigGAN performance and motivates robust class-conditional GAN learning on small ImageNet subsets.
  • The proposed method regularizes discriminator predictions with moving-average historical predictions to improve GAN generalization on limited data.
  • The regularized WGAN objective is theoretically connected to LeCam-divergence, which the paper finds more robust under limited training data.
  • Experiments show improved generalization, more stable learning dynamics, and complementarity with data augmentation, achieving state-of-the-art performance on limited ImageNet data.

2. Related Work

Prior GAN research uses divergence-based losses, integral probability metrics, augmentation, and regularization to improve modeling or training. Limited-data GAN methods emphasize data diversity, while this paper's overview positions anchors as part of its regularized discriminator-loss procedure.

  • Generative adversarial networks: GAN losses have been developed from f-divergence objectives, including JS-divergence for saturated GANs and χ2-divergence for LSGAN.
  • Generative adversarial networks: WGAN frameworks instead use integral probability metrics, and the paper designs regularization applicable to different GAN loss functions under limited data.
  • Learning GANs on limited training data: Limited-data GAN training is challenging because scarcity causes unstable dynamics, degraded image fidelity, and memorization of training examples.
  • Regularization for GANs: Existing GAN regularization methods mainly target convergence stability or mode collapse through techniques such as noise, gradient penalties, and spectral normalization.
  • Regularization for GANs: Figure 2 shows anchors tracking discriminator predictions before those moving averages are used to compute the regularized discriminator loss.

3. Methodology

The paper regularizes discriminator predictions to improve GAN robustness with limited data, connecting the resulting objective to the LeCam-divergence. This divergence is motivated as a robust, bounded alternative whose empirical behavior improves generalization and training stability.

  • Regularization scheme: The method regulates discriminator predictions with moving-average anchors rather than relying solely on data augmentation.The regularization is applied across GAN losses and is intended to improve generalization on out-of-sample data.
  • Connection to LeCam-divergence: Under mild assumptions, the regularized WGAN objective minimizes a weighted LeCam-divergence between data and generated distributions.The analysis uses a single-anchor simplified regularizer, while experiments use two anchors and apply the term to hinge and non-saturated losses.
  • Robustness under limited data: For extreme P(x)/Q(x) values caused by limited-data estimation errors, LeCam-divergence provides more robust f-function values.The comparison plots the input ratio on the x-axis and the f-function value on the y-axis.
  • Properties of LeCam-divergence: LeCam-divergence is symmetric, bounded between 0 and 2, and reaches zero exactly when the data and generated distributions match.It is also expressed as a symmetric version of χ2-divergence and is related by inequalities to JS-divergence and total variation.
  • Empirical evaluation: Experiments report improved BigGAN generalization and stabilized limited-data training, including CIFAR-10 FID curves and CIFAR quantitative comparisons.The method is also reported to complement existing data augmentation approaches.

4. Experimental Results

Experiments across CIFAR, ImageNet, and StyleGAN2 show that the proposed regularization improves GAN performance and stability under limited data, while complementing data augmentation. Ablations indicate that its benefits depend on a reasonable regularization strength and increase with model capacity.

  • Experimental Setup: The experiments evaluate BigGAN and StyleGAN2 on CIFAR, ImageNet, and StyleGAN datasets using FID and IS metrics.Results include comparisons with GAN losses, regularization methods, and differentiable data augmentation.
  • Results on CIFAR-10 and CIFAR-100: The proposed method improves BigGAN generalization on limited data and achieves the lowest final FID while stabilizing training dynamics.Without the method, FID scores deteriorate later in training; the proposed method reaches the lowest FID at iteration 100K.
  • Comparison to Data Augmentation: The proposed regularization improves data augmentation methods, especially under limited data, and remains complementary to DA and ADA.The combination achieves state-of-the-art performance on limited ImageNet training data, while augmentation can degrade performance when training images are sufficiently diverse.
  • Comparison to State-of-the-art on ImageNet: Under 25% ImageNet data, the method reduces FID by 3.75 and lowers performance variance from 2.59 to 1.73.The reported results use three training runs and indicate improved resistance to scarce training data and more stable training.
  • Results on StyleGAN2: StyleGAN2 performance improves with and without data augmentation, showing that the regularization applies alongside different GAN losses and existing regularizers.StyleGAN2 uses the non-saturated GAN loss and GP-0 by default.
  • Analysis and Ablation Studies: Regularization weights above 0.5 degrade performance, whereas weights from 0.1 to 0.5 are effective in the reported CIFAR-10 study.The analysis links larger weights to negative divergence values.
  • Analysis and Ablation Studies: The method constrains discriminator behavior by penalizing differences between real and generated predictions, preventing their rapid divergence under limited data.This keeps predictions in a particular range and supports convergence toward a stationary point.
  • Analysis and Ablation Studies: The method’s improvement increases with model size because overfitting is more severe for higher-capacity generator and discriminator models.The model-size study progressively halves channels in both networks.

5. Conclusion and Future Work

The paper presents a discriminator regularization method for GANs trained with limited data. Experiments show improved performance, especially in limited-data settings, and further gains when combined with data augmentation.

  • Conclusion: The proposed discriminator regularization method trains more robust GAN models under limited data.The method imposes a regularization loss during discriminator training.
  • Conclusion and Future Work: The method improves GAN performance, especially under limited data, and can combine with data augmentation to further enhance results.The paper identifies conditional GAN tasks and robust learning with noisy large-scale data as future directions.

A.1. Overview

The supplementary document provides theoretical justification, implementation details, and additional low-shot image-generation experiments.

  • Overview: The supplementary document covers a proof of Proposition 1, implementation details, and experiments using only hundreds of images.The additional experiments concern low-shot image generation.

A.2. Theoretical Analysis

The analysis connects the regularized GAN objective to the LeCam-divergence under a stationary-anchor assumption, while experiments examine discriminator stability and practical modifications.

  • Theoretical connection: Under a fixed generator and converged positive anchor, the regularized WGAN objective yields a generator objective weighted by the LeCam-divergence.The analysis uses a single-anchor regularizer and assumes the anchor converges to a stationary value.
  • Theoretical connection: The regularization weight must satisfy λ < 1/(2α); for α = 1, the analysis requires λ < 0.5.This bound follows from the non-negativity of the divergence.
  • Interpretation: The theoretical analysis motivates the method, but the experiments modify the analyzed setup to optimize empirical performance.The paper explicitly notes that theoretical and practical GAN losses can differ.
  • Practical modifications: The practical method uses two discriminator anchors and extends regularization to hinge and non-saturated GAN losses.These modifications are presented as departures from the simplified single-anchor analysis.
  • Empirical support: Without regularization, discriminator predictions for real and generated images diverge rapidly under limited data, whereas the proposed method keeps them within -1 and +1.This behavior is visualized for BigGAN on CIFAR-10.

A.3. Implementation Details

The implementation evaluates the regularization scheme across GAN backbones and datasets, using dataset-specific hyperparameters and supplementary ablations and reproduction checks.

  • Additional evaluations: The study supplements the main results with CIFAR IS scores, EMA ablations, WGAN comparisons, and qualitative comparisons at 25% ImageNet data.The cited tables and figure report evaluation protocols or additional comparisons rather than complete numerical results here.
  • Exponential moving average: The exponential moving average uses α variables for discriminator prediction tracking, with decay factor γ fixed to 0.99 in all experiments.The variables correspond to αR and αF.
  • CIFAR-10 and CIFAR-100: CIFAR experiments set λ = 0.3, use batch size 50, learning rate 2e−4 for both networks, four discriminator updates per generator step, and translation plus cutout augmentation.The settings follow the baseline implementation referenced in the paper.
  • ImageNet: ImageNet experiments use BigGAN with batch size 2048, separate learning rates for discriminator and generator, two discriminator updates per generator step, and λ = 0.01.Random scaling and horizontal flipping are used for preprocessing.
  • Reproduction checks: Most previous results are quantitatively comparable, but some reported StyleGAN and ADA results on small FFHQ datasets differ from the original papers.The reproduction discussion identifies differences for the 5k and 1k FFHQ settings and ADA on 1k FFHQ.

A.4.1 CIFAR-10 and CIFAR-100

The CIFAR and low-shot experiments compare the proposed regularization with alternative GAN backbones, augmentation, and transfer-learning strategies under data scarcity.

  • CIFAR experiments: The proposed approach improves WGAN performance on CIFAR-10, but the WGAN backbone is inferior to BigGAN and more sensitive to hyperparameter settings.Consequently, the experiments use BigGAN for the main CIFAR and ImageNet studies.
  • Qualitative results: Qualitative comparisons show that BigGAN with the proposed method synthesizes more realistic images than baseline models with and without data augmentation.The additional comparisons are reported for the 25% ImageNet setting.
  • Low-shot generation: In low-shot generation, the method combines regularization with data augmentation when training on datasets containing only 100, 160, or 389 images.The evaluated datasets include Obama, Grumpy cat, Panda, Cat, and Dog.
  • Low-shot generation: The combined regularization and augmentation approach performs favorably against transfer-learning techniques pretrained on 70,000 images.The comparison concerns quantitative low-shot FID results.
  • Low-shot generation: The low-shot setting addresses training GANs with only a few dozen images and contrasts with adaptation strategies that modify or fine-tune pretrained GANs.The cited passage frames this as an extreme data-scarcity scenario.
Loading 2104.03310v1…