PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis

Bingchen Liu, Yizhe Zhu, Kunpeng Song, Ahmed Elgammal

arXiv:2101.04775v1cs.CVcs.AI

TL;DR

The paper addresses high-fidelity GAN synthesis with few training images and limited computation, conditions that make training vulnerable to overfitting and mode collapse. It builds a lightweight GAN using skip-layer channel-wise excitation and self-supervised discriminator regularization. Across thirteen diverse datasets, the proposed techniques significantly boost synthesis performance and improve training stability under these constraints.

  • Problem

    High-fidelity GAN synthesis typically requires extensive training data and computation, while few-shot, high-resolution settings expose models to overfitting and mode collapse.

  • Method

    The model combines a skip-layer channel-wise excitation module with a self-supervised discriminator encoder and decoder, using auto-encoding only to regularize the discriminator.

  • Results

    On thirteen diverse datasets, SLE and self-supervised discriminator regularization significantly boost GAN synthesis performance under sub-hundred-image and limited-computing conditions.

  • Takeaways & Limitations

    The techniques require minor changes to a vanilla GAN and enhance its practicality through a desirable plug-and-play property.

Abstract

from arXiv · show

Training Generative Adversarial Networks (GAN) on high-fidelity images usually requires large-scale GPU-clusters and a vast number of training images. In this paper, we study the few-shot image synthesis task for GAN with minimum computing cost. We propose a light-weight GAN structure that gains superior quality on 1024*1024 resolution. Notably, the model converges from scratch with just a few hours of training on a single RTX-2080 GPU, and has a consistent performance, even with less than 100 training samples. Two technique designs constitute our work, a skip-layer channel-wise excitation module and a self-supervised discriminator trained as a feature-encoder. With thirteen datasets covering a wide variety of image domains (The datasets and code are available at: https://github.com/odegeasslbc/FastGAN-pytorch), we show our model's superior performance compared to the state-of-the-art StyleGAN2, when data and computing budget are limited.

1 INTRODUCTION

The paper targets high-resolution GAN synthesis when training data and computing are scarce, where overfitting and mode collapse threaten convergence. It introduces SLE and self-supervised discriminator training within a computationally efficient model.

  • High-fidelity GAN applications are constrained by expensive computation and large training-data requirements, especially for small image sets and low budgets.
  • Few-shot settings may offer fewer than 100 images, while transfer learning lacks guaranteed compatible pretraining data and can worsen fine-tuning performance.
  • The proposed SLE module uses low-scale activations to revise high-scale channel responses, supporting robust gradient flow and faster training.
  • The self-supervised discriminator is trained as a feature encoder with an extra decoder to produce more comprehensive signals for generator training.
  • The resulting computational-efficient GAN is evaluated for robustness across multiple high-fidelity datasets, as demonstrated by the paper’s experiments.

2 RELATED WORKS

Prior work addresses faster, higher-resolution, and more stable GAN training through architectural, objective, regularization, and augmentation strategies. The paper identifies remaining challenges under few-shot, high-resolution, and limited-computing conditions.

  • Speed up the GAN training: Existing speed-up methods reduce computation or shorten optimization, but shortened training time does not necessarily produce advanced synthesis quality.
  • Train GAN on high resolution: High-resolution GANs face rigid generator gradients from increased parameters and sparse target distributions at 1024 × 1024 resolution.
  • Stabilize the GAN training: Mode collapse becomes more challenging with fewer samples and smaller batch sizes because an overfit discriminator may provide meaningless generator gradients.
  • Stabilize the GAN training: Discriminator regularization methods include alternative objectives, gradient regularization, weight normalization, and data augmentation, but their effects degrade quickly with limited batch sizes.
  • Stabilize the GAN training: Prior self-supervised discriminator studies mainly covered limited image domains and resolutions from 32^2 to 128^2 without computing-resource limitations.

3 METHOD

The method combines a compact GAN architecture with skip-layer excitation in the generator and self-supervised auto-encoding regularization in the discriminator. These designs target faster training, stronger gradient flow, and more comprehensive image representations while preserving a lightweight model.

  • Architecture: The model uses a single convolutional layer per resolution and only three channels at high resolutions in both generator and discriminator.These choices make the GAN substantially smaller and faster to train while retaining robustness on small datasets.
  • Skip-layer channel-wise excitation: SLE uses channel-wise multiplication between feature-maps at different resolutions to provide shortcut gradient flow without extra computation.Unlike residual addition, the operation does not require equal spatial dimensions and uses a low-resolution activation to modulate a higher-resolution feature-map.
  • Skip-layer channel-wise excitation: SLE transforms low-resolution activations through pooling, convolutions, nonlinearity, projection, and sigmoid gating before multiplying them with high-resolution features.The output retains the same shape as the high-resolution feature-map.
  • Skip-layer channel-wise excitation: SLE can support channel-wise feature recalibration, long-range gradient flow, and automatic style/content disentanglement in the generator.Replacing the low-resolution input with another synthesized sample can preserve content while changing the generated image’s style.
  • Self-supervised discriminator: The discriminator is trained as an encoder with small decoders using reconstruction loss on real samples, while the generator remains a pure GAN component.The decoders reconstruct images from discriminator features, and the auto-encoding objective regularizes D without involving G.
  • Self-supervised discriminator: Reconstructive training encourages discriminator features to cover overall composition and detailed texture across multiple scales.The implementation uses decoders for feature-maps at two scales and matches reconstructed images to corresponding real-image targets.

4 EXPERIMENT

Experiments evaluate the model across diverse few-shot and larger datasets using FID, LPIPS, computational cost, qualitative comparisons, and self-supervision analyses. The proposed compact architecture and two modules generally improve synthesis quality, training efficiency, and resistance to mode collapse relative to the compared models.

  • Evaluation setup: Experiments span 256 × 256 and 1024 × 1024 datasets across faces, animals, objects, paintings, landscapes, and other image domains.The evaluation includes thirteen datasets with varied content categories and image characteristics.
  • Evaluation setup: The study compares the proposed model with StyleGAN2 and an ablated baseline, while accounting for StyleGAN2’s higher computational cost.The baseline combines spectral normalization, exponential-moving-average optimization, differentiable augmentation, and GLU activations.
  • Ablation and stability: Both SLE and decoding-on-D separately improve performance, with self-supervised decoding making the largest contribution in the ablation study.The authors also report that the proposed model is less likely to mode-collapse and remains stable beyond training durations where the baseline usually collapses.
  • Larger-data evaluation: With more training data, StyleGAN2 performs better under larger models and batch sizes, while the proposed model remains competitive with considerably lower computational cost.The comparison notes that StyleGAN2 has more than twice the parameters and receives more training resources.
  • Qualitative results: The proposed model maintains satisfactory qualitative synthesis under matched batch size and training time, while StyleGAN2 either converges more slowly or suffers mode collapse.Best results reportedly require three hours on Flower, Shell, and Pokemon, and eight hours on the other three datasets discussed.
  • Discriminator self-supervision: Auto-encoding self-supervision for D performs best among the tested strategies, whereas combining it with other settings can substantially reduce performance.The authors hypothesize that reconstruction encourages D to cover more image regions, while classification may focus on limited local cues.

5 CONCLUSION

The paper introduces two lightweight techniques for stabilizing GAN training and improving synthesis quality with fewer than 100 high-fidelity images and limited computation. Across thirteen diverse datasets, SLE and self-supervised discriminator regularization substantially improve performance while requiring only minor changes to a vanilla GAN.

  • Conclusion: SLE and self-supervised discriminator regularization stabilize GAN training and improve synthesis quality with sub-hundred high-fidelity images and limited computation.The techniques are presented as lightweight and plug-and-play additions to vanilla GANs.
  • Conclusion: Across thirteen diverse datasets, the two techniques significantly boost GAN synthesis performance.The conclusion emphasizes the method’s intended practicality for downstream GAN tasks.

A PERFORMANCE BOOST FROM SKIP-LAYER EXCITATION

The SLE module consistently improves baseline FID during training, chiefly by accelerating early generator convergence. Its gains are strongest early and become modest after convergence.

  • SLE provides a constant FID performance boost over the baseline across four 1024 × 1024 datasets.The comparison covers Flower, FFHQ, Shell, and Art-paintings, with FID recorded every 10000 iterations.
  • SLE accelerates GAN convergence, with the clearest effect during the first 20000 iterations.The generator reaches a good point earlier than the baseline, which requires substantially more iterations.
  • SLE slightly improves overall performance while preserving behavior similar to the baseline after early convergence.The training curves remain broadly parallel, indicating that SLE mainly changes convergence speed rather than the final trajectory.
  • SLE is unlikely to make a large difference after a good converged state because it adds few parameters and no explicit regularization or training guidance.On Shell, both SLE and baseline models collapsed after 60000 iterations, while other datasets showed little late-stage FID change.
  • SLE also enables unsupervised style-content disentanglement in a simpler and more cost-efficient way than StyleGAN and StyleGAN2.The paper links this capability to SLE while also reporting faster generator convergence and improved generator performance.

B FEATURE-EXTRACTION PERFORMANCE OF DISCRIMINATOR

The discriminator is evaluated as a feature encoder through auto-encoding, revealing that self-supervision produces more descriptive features and aligns with better synthesis performance. However, the evidence is largely limited to few-shot datasets.

  • Auto-encoding evaluates whether discriminator feature maps retain enough information to reconstruct input images.A decoder reconstructs images from the second-last discriminator activation, making feature coverage measurable through reconstruction quality.
  • GAN training makes discriminator features less meaningful as encodings than those from randomly initialized discriminators.The paper reports this comparison for StyleGAN2 and the baseline using Table 7.
  • Self-supervision produces more descriptive discriminator features than random initialization, while reconstruction is less prone to local-pattern overfitting than contrastive learning.The authors report auto-encoding as the best-performing self-supervision strategy among those tested.
  • Better discriminator feature extraction is associated with better GAN synthesis performance for both StyleGAN2 and the baseline.For the baseline, feature-extraction performance aligns with the corresponding FID scores; self-supervision also extracts more image information than vanilla GAN training.
  • The relationship between discriminator feature extraction and synthesis remains incompletely characterized because most experiments use few-shot datasets.The paper calls for further study on larger-scale datasets while validating self-supervision for enhanced GAN performance on few-shot data.

C STYLE-MIXING ON DIFFERENT RESOLUTIONS

Style-mixing is controlled by swapping SLE feature maps across resolutions, with lower-resolution layers producing the largest style changes. Performance varies by dataset and training resolution.

  • The 1024 × 1024 model has three SLE layers available for feature swapping, while the 256 × 256 model has two.The experiments swap feature maps between generated samples at different resolutions.
  • The strongest style-mixing results occur when feature maps are swapped across all resolutions.The most effective layer for style changes is the 128-resolution layer.
  • At 256 × 256 resolution, lower-resolution SLE layers produce the largest style differences.The paper reports the same lower-resolution effect for the 256 × 256 model.
  • Art-paintings supports control of both coloring and texture, including transfers of flat or pointy brush-stroke styles.The model performs less reliably on FFHQ but performs well on Obama despite using fewer than 100 training images.

D MORE QUALITATIVE COMPARISON

Qualitative comparisons use matched random noises to contrast the proposed model with baselines. The reported examples show greater diversity than baseline outputs on some datasets and slower or collapsed StyleGAN2 training on others.

  • Using the same randomly sampled noises, the proposed model generates diverse images while the baseline shows clear mode collapse on Grumpy-cat and Panda.Both models are trained for 50000 iterations with batch size 8, described as sufficient for convergence.
  • Slimmed StyleGAN2 converges much slower than the proposed model on Pokemon and Shell and mode-collapses on Anime-Face.The comparison uses StyleGAN2 with half the channel numbers.

E NEAREST IMAGES FROM TRAINING SETS

The figures compare synthesized images with their nearest real training examples using LPIPS, including settings with 1000 and 100 training images. The accompanying evaluation reports LPIPS between generated samples and closest real samples, alongside an augmentation-based real-image baseline.

  • With 1000 training images, each synthesized sample is paired with its closest real training image ranked by LPIPS.The synthesized image is shown on the left and the nearest real image on the right; samples are uncurated.
  • With 100 training samples, the model combines learned real-image features into new compositions in uncurated samples.Each synthesized image is compared with the closest real training image ranked by LPIPS.
  • Table 8 presents LPIPS values between synthetic images and their closest real images.
  • The evaluation reports average LPIPS between generated samples and their closest real samples, using real images with random augmentations as the baseline.Experiments use three trials with 100 synthesized samples or real images and report the lowest result; trial standard deviations are usually below 0.005.

F DECODER RESULT

The decoder reconstruction results evaluate the auto-encoding discriminator across datasets at 128 × 128 resolution. They show augmented real inputs, full-image reconstructions, and reconstructions of randomly cropped portions.

  • The decoder reconstructs augmented real images for training the auto-encoding discriminator.
  • For each dataset, the first panel shows augmented real images, followed by full-image and randomly cropped-region reconstructions.
  • All reconstructions are produced at 128 × 128 resolution.
Loading 2101.04775v1…