Source-linked AI summary
Image Augmentations for GAN Training
Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, Han Zhang
TL;DR
Image augmentation's potential for GAN image synthesis had not been systematically studied, leaving choices about transformations, target images, and loss integration unclear. This paper evaluates augmentation strategies for vanilla and regularized GANs, finding that augmenting both real and generated images improves generation and that regularization can further improve results, including state-of-the-art conditional CIFAR-10 generation.
Problem
The impact of image augmentation strategies on GAN image synthesis had not been systematically studied, leaving unclear which transformations to use, which images to augment, and how to integrate augmentations with losses.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Data augmentations have been widely studied to improve the accuracy and robustness of classifiers. However, the potential of image augmentation in improving GAN models for image synthesis has not been thoroughly investigated in previous studies. In this work, we systematically study the effectiveness of various existing augmentation techniques for GAN training in a variety of settings. We provide insights and guidelines on how to augment images for both vanilla GANs and GANs with regularizations, improving the fidelity of the generated images substantially. Surprisingly, we find that vanilla GANs attain generation quality on par with recent state-of-the-art results if we use augmentations on both real and generated images. When this GAN training is combined with other augmentation-based regularization techniques, such as contrastive loss and consistency regularization, the augmentations further improve the quality of generated images. We provide new state-of-the-art results for conditional generation on CIFAR-10 with both consistency loss and contrastive loss as additional regularizations.
1 Introduction
Image augmentation is well established for recognition, but its role in GAN image synthesis remained insufficiently studied to guide choices about which images to augment or how. This paper systematically evaluates augmentation strategies for vanilla and regularized GANs, finding that augmenting both real and generated images improves generation and that added regularization can further improve results.
Motivation
- The study systematically evaluates common image transformations in GANs, addressing an evidence gap that left augmentation choices, targets, and loss integration unclear.Prior work had not systematically studied augmentation strategies for GAN image generation.Source: §1 · pp. 1–2
- Augmenting only real images can lead the discriminator to treat augmentation artifacts as part of the image distribution, encouraging the generator to produce those artifacts.Earlier GAN approaches therefore generally used only random crop and flip, while prior work had explored consistency regularization and augmentation of both real and generated samples.Source: §1 · pp. 1–2
Contributions
- Augmenting both real and generated images improves vanilla GANs, while consistency regularization and contrastive loss further improve generation and support new state-of-the-art conditional CIFAR-10 results.The reported conditional CIFAR-10 FID improves from 9.21 to 8.30 when both consistency and contrastive losses are applied on top of the best augmentation.Source: §1 · p. 2
- The authors aim to provide empirical guidance on augmentation efficacy and robustness for future GAN research and practice.Their contributions also include analysis of augmentation and regularization combinations.Source: §1 · p. 2
2 Augmentations and Experiment Settings
The experiments assess individual augmentations across two GAN settings on CIFAR-10, using FID to compare generated images. The study evaluates vanilla models first, then models with consistency regularization or contrastive loss.
Image Augmentations
- The study evaluates 10 basic and 3 advanced image augmentation operations individually, varying augmentation strength from weakest to strongest rather than chaining operations.Each operation is applied to an image normalized to pixel values in [0, 1].Source: §2 · pp. 2–3
Data and Evaluation
- Experiments use CIFAR-10, with 50K training images and 10K evaluation images, and report FID on 10K generated samples against 10K test images.FID is lower when better, though the paper notes it does not always imply better image quality; runs use different random seeds and report the top 15% trained models.Source: §2 · p. 3
GAN Architectures and Training
- The study tests SNDCGAN for unconditional generation and BigGAN for conditional generation, then evaluates vanilla models and models with consistency regularization or contrastive loss.SNDCGAN trains for 200K steps with batch size 64; BigGAN trains for 100K steps with batch size 256, and both use hinge loss.Source: §2 · p. 3
3 Effect of Image Augmentations for Vanilla GAN
For vanilla GANs, augmenting only real images worsens FID, whereas augmenting both real and generated images substantially improves it. Spatial transformations are more effective, and increased distribution overlap alone does not guarantee better generation.
3.1 Augmenting Only Real Images Does Not Help with GAN Training
- Augmenting only real images worsens SNDCGAN FID at every tested augmentation strength; even minimal translation yields FID 31.03 versus the unaugmented baseline of 24.73.FID increases monotonically as augmentation strength increases; the authors suggest generated augmentation artifacts contribute to the poorer FID.Source: Figure 2 · p. 3, §3.1 · p. 4
3.2 Augmenting Both Real and Fake Images Improves GANs Consistently
- Applying the same augmentation to real and generated images before the discriminator substantially improves GAN performance, and retaining gradients through augmented generated images lets them train the generator.This setup has the discriminator distinguish augmented real images from augmented fake images.Source: §3 · p. 3, §3.2 · p. 4
- BigGAN reaches FID 11.03 with both real and fake images augmented, compared with FID 11.48 for CR-GAN [42]; SNDCGAN reaches FID 18.94, comparable to FID 18.72 for CR-GAN [42].The SNDCGAN result uses translation strength 0.1.Source: Figure 3 · p. 4, §3.2 · p. 5
- Spatial transformations such as translation and zoom improve GAN performance more than visual transformations such as brightness, colorness, and mixup.The figure caption also identifies augmentations that move images off the natural data manifold, such as InstanceNoise, as unhelpful.Source: Figure 3 · p. 4, §3.2 · p. 5
3.3 Augmentations Increase the Support Overlap between Real and Fake Distributions
- Augmentations lower FID between transformed real and fake images, indicating greater support overlap, but greater overlap does not always improve generated-image quality.The authors hypothesize that some augmentation types or high strengths move images far from the natural image distribution; they leave theoretical justification for future work.Source: §3.3 · p. 5, Figure 4 · p. 5
4 Effect of Image Augmentations for Consistency Regularized GANs
Consistency regularization on augmentations of real and fake images further improves GAN generation. Choosing augmentation type and strength carefully yields results that outperform reported prior results, with spatial transforms generally more effective than visual transforms.
Consistency Regularization
- Consistency regularization applied to augmentations of both real and fake images further boosts generation performance.The experiments apply Balanced Consistency Regularization (BCR) to the augmented images.Source: §4 · pp. 5–6
Results
- BigGAN reaches FID 8.65 with translation strength 0.4, outperforming the FID 9.21 reported by Zhao et al. [46].SNDCGAN's best result is FID 14.72 with zoomout strength 0.4, versus FID 15.87 reported by Zhao et al. [46].Source: §4 · p. 6, Figure 5 · p. 6
Augmentation Choices
- Spatial transforms, including zoomout, zoomin, translation, cutout, and cutmix, are more effective with BCR than visual transforms; InstanceNoise does not improve generation performance.The paper notes that spatial transforms retain major image content while introducing spatial variation.Source: §4 · p. 6
5 Effect of Images Augmentations for GANs with Contrastive Loss
The section applies contrastive loss to GAN training with augmented real and fake images, then studies its interaction with augmentation choice and consistency regularization. Combining contrastive loss and BCR yields a reported state-of-the-art FID of 8.30; Cntr-GAN also favors stronger spatial augmentations.
5 Effect of Images Augmentations for GANs with Contrastive Loss
- Cntr-GAN applies contrastive loss to two random augmented copies of both real and fake images, drawing same-image views together and pushing different representations apart.The loss regularizes the discriminator; augmented views of each image form positive pairs, while other examples in the concatenated batch are negatives.Source: §5 · p. 7
- Cntr-GAN outperforms unaugmented BigGAN but remains inferior to BCR-GAN under the default SimCLR augmentations.The comparison is reported in Table 1.Source: §5 · p. 7, Table 1 · p. 7
- FID = 8.30 is reported as a new state-of-the-art result for BigGAN regularized with both CntrLoss and BCR.The reported settings are λCntr = 0.1 and λBCR = 5; Table 1 compares vanilla BigGAN with augmentation-based regularizations.Source: Table 1 · p. 7, §5 · p. 7
- Cntr-GAN favors stronger augmentations than BCR-GAN, with spatial transforms outperforming visual transforms; adjusted SimCLR cropping/resizing at strength 0.3 reaches FID 11.87.Most tested augmentations reach their best performance around strength 0.5.Source: §5 · p. 7
6 Discussion
The discussion examines augmentation effects beyond headline scores, including artifacts, strength annealing, and transform composition. It reports that annealing reduces augmentation effects without changing the relative efficacy of transforms, while a translation-plus-brightness composition reaches FID 8.42.
6 Discussion
- Figure 6 marks unaugmented BigGAN's baseline FID as 14.73 and reports that Cntr-GAN consistently outperforms vanilla GAN, with a preference for spatial augmentations.The figure varies cropping-resizing strength in the default SimCLR augmentation.Source: Figure 6 · p. 7
- For vanilla GANs, augmenting both real and fake images reduces generation artifacts compared with augmenting only real images; contrastive loss and consistency regularization further improve generation quality.The discussion describes qualitative examples of artifacts under different augmentation settings.Source: §6 · p. 8
- Annealing augmentation strength reduces its effect without changing the relative efficacy of different augmentations.Augmentations that help GAN training lose some of their gains with annealing, while harmful augmentations likewise lose some of their negative effect.Source: §6 · p. 8
- FID 8.42 is reported for chained translation and brightness, compared with FID 8.58 for translation alone.The authors say this preliminary result suggests the combination is dominated by the more effective translation and leave automatic composition search for future work.Source: §6 · p. 8
7 Related Work
The related work contrasts augmentation's extensive use in visual recognition and learning settings with its more limited exploration for GAN image generation. It reviews GAN augmentation strategies and regularization approaches that motivate this study.
7 Related Work
- Image augmentation has been studied in recognition tasks including classification [13] [7], detection [31] [47], and segmentation [5, 15], as well as semi-supervised and unsupervised learning.Strategies range from crops and mirroring to geometric distortions, color jittering, and combinations of image statistics.Source: §7 · p. 8
- GAN augmentation research remained limited, with mainstream DCGAN [30], BigGAN [4], and StyleGAN2 [20] using cropping and horizontal flipping as their exclusive strategies.Other cited directions include noise-based stabilization [35], adding noise to both real and fake images [1], and augmentation-based regularization [6] [42] [46].Source: §7 · p. 8
8 Conclusion
The conclusion emphasizes that augmenting both real and generated samples is important for realistic GAN generation, and that consistency regularization can further improve performance. The appendices specify augmentation operations and training procedures for BCR-GAN and Cntr-GAN, with qualitative examples illustrating artifact reduction.
8 Conclusion
- The authors conclude that augmenting both real images and generated samples is critical for realistic samples, and that consistency regularization can further boost performance and outperform contrastive loss.They also report state-of-the-art image-generation performance by combining contrastive loss and consistency loss.Source: §8 · p. 8
B Augmentations
- The appendix defines spatial transforms including ZoomIn and ZoomOut, translations, and image-combination operations such as CutMix and MixUp.It also specifies visual changes such as brightness, color-channel shifts, noise, and CutOut masking, alongside the SimCLR augmentation recipe.Source: B Augmentations · p. 12
C BCR-GAN: GAN with Balanced Consistency Regularization
- BCR-GAN augments real and fake images separately and penalizes changes in discriminator outputs between each image and its augmented version.The algorithm adds this balanced consistency loss to the discriminator objective.Source: C BCR-GAN: GAN with Balanced… · p. 13
D Cntr-GAN: GAN with Contrastive Loss
- Cntr-GAN applies contrastive loss to projected discriminator representations of two augmented views each of real and fake images.The algorithm uses a projection head and combines the real-image and fake-image contrastive losses in the discriminator update.Source: D Cntr-GAN: GAN with Contrastive Loss · p. 13
E Generation Artifacts
- Qualitative examples report fewer cutout, brightness, blueness, and zoom-in artifacts when vanilla GANs augment both real and fake images rather than only real images.The examples also report further quality improvement with contrastive loss and consistency regularization.Source: Figure 7 · p. 14, Figure 8 · p. 15, Figure 9 · p. 16
F.1 BigGAN with Only Real Images Augmented
With CntrLoss and BCR, BigGAN reaches a new state-of-the-art FID of 8.30; augmenting only real images is not helpful for vanilla GAN. Both findings qualify how augmentation performs across these settings.
F.1 BigGAN with Only Real Images Augmented
- Only augmenting real images is not helpful for vanilla GAN, consistent with the comparison shown for BigGAN.Source: §F.1 · p. 18, Figure 11 · p. 18
F.2 Interaction between CntrLoss and BCR
- FID = 8.30 is a new state of the art for BigGAN regularized with CntrLoss (λCntr = 0.1) and BCR (λBCR = 5).The authors attribute the added benefit of CntrLoss to better learned representations alongside BCR's consistency loss on discriminator logits.Source: Figure 12 · p. 18, §F.2 · p. 18
F.3 Annealing Augmentation Strength during Training
Annealing augmentation strength reduces its effect without changing the relative efficacy of different augmentations. In a BCR-GAN exploration, combining translation and brightness achieved a lower best FID than translation alone.
F.3 Annealing Augmentation Strength during Training
- Annealing augmentation strength reduces its effect without changing the relative efficacy of different augmentations.Augmentations that improve GAN training alleviate their improvements with annealing, and vice versa.Source: §F.3 · p. 19, Figure 13 · p. 19
F.4 Exploration on Chain of Augmentations
- FID = 8.42 with chained translation and brightness is lower than the best FID = 8.58 with translation alone in BCR-GAN training.The authors suggest the combination is dominated by the more effective translation and leave automatic composition search to future work.Source: §F.4 · p. 19, Figure 14 · p. 19
G Model Details
- The SNDCGAN and BigGAN architectures are presented, with further details referred to prior work.Source: G Model Details · p. 20