PaperlayerView exact revision on arXiv

Source-linked AI summary

Projected GANs Converge Faster

Axel Sauer, Kashyap Chitta, Jens Müller, Andreas Geiger

arXiv:2111.01007v1cs.CVcs.LG

TL;DR

GAN training is difficult because jointly learned discriminators require substantial regularization, computation, and tuning. Projected GANs instead use fixed pretrained feature projections with channel and cross-scale mixing, achieving state-of-the-art image synthesis and much faster convergence. The method also improves data efficiency across datasets and resolutions up to 1024^2 pixels.

  • Problem

    GAN discriminators are difficult to train jointly with generators, motivating more stable and efficient approaches to unconditional image synthesis.

  • Method

    Projected GANs map real and generated images into fixed pretrained feature spaces and mix features across channels and resolutions before independent discrimination.

  • Results

    State-of-the-art FID is achieved across benchmark datasets, with Projected FastGAN reaching StyleGAN2’s prior best FID after 1.1 M images instead of 88 M.

  • Takeaways & Limitations

    Projected GANs improve image quality, data efficiency, and convergence speed while avoiding additional regularization and expensive hyperparameter sweeps.

  • Takeaways & Limitations

    Projected GANs sometimes produce floating heads or bland backgrounds on AFHQ and incorrect proportions or artifacts on FFHQ despite state-of-the-art FID.

Abstract

from arXiv · show

Generative Adversarial Networks (GANs) produce high-quality images but are challenging to train. They need careful regularization, vast amounts of compute, and expensive hyper-parameter sweeps. We make significant headway on these issues by projecting generated and real samples into a fixed, pretrained feature space. Motivated by the finding that the discriminator cannot fully exploit features from deeper layers of the pretrained model, we propose a more effective strategy that mixes features across channels and resolutions. Our Projected GAN improves image quality, sample efficiency, and convergence speed. It is further compatible with resolutions of up to one Megapixel and advances the state-of-the-art Fréchet Inception Distance (FID) on twenty-two benchmark datasets. Importantly, Projected GANs match the previously lowest FIDs up to 40 times faster, cutting the wall-clock time from 5 days to less than 3 hours given the same computational resources.

2 Related Work

Prior work uses pretrained representations to guide GANs and explores specialized discriminator designs, but projected GANs target unconditional noise-to-image synthesis with fixed pretrained features and random projections.

  • Pretrained Models for GAN Training: Pretrained-model approaches include GAN transfer, generative control, moment matching, perceptual losses, and other non-adversarial uses.The paper positions its method within work using pretrained representations without adversarial pretraining.
  • Discriminator Design: Standard discriminators usually remain vanilla convolutional networks or mirror the generator, while exceptions use encoder-decoder or multiple-discriminator designs.
  • Discriminator Design: Projected GAN differs from prior encoder-decoder discriminators by combining pretrained features with random projections.

3 Projected GANs

Projected GANs map real and generated images through fixed, differentiable feature projectors and train independent discriminators on multi-scale representations. Random channel and cross-scale mixing is designed to make deeper pretrained features more usable.

  • Projected GANs: Projected GAN training uses fixed differentiable projectors to map real and generated images into discriminator input spaces, with independent discriminators for different projections.Only the generator and discriminators are optimized; the projectors remain fixed.
  • Projected GANs: At convergence, the generator matches the real distribution in the projected feature spaces, including when stochastic augmentations precede deterministic projections.
  • Multi-Scale Discriminators: Four discriminators operate on pretrained features from resolutions 64^2, 32^2, 16^2, and 8^2, with logits aligned at resolution 4^2.
  • Random Projections: Random projections mix features because deeper semantic layers are harder to cover and may let the discriminator disregard parts of the feature space.
  • Random Projections: CCM applies fixed Kaiming-initialized 1×1 convolutions across channels, while CSM adds random 3×3 convolutions and bilinear upsampling across scales.CSM yields a simplified U-Net, and neither strategy trains its random projection weights.
  • Pretrained Feature Networks: The model studies pretrained feature-network choices and uses EfficientNet-Lite1 after finding compact representations effective for projected training.

4 Ablation Study

Ablations show that independent multi-scale discriminators and feature mixing improve projected GAN training, especially for deeper features. Compact pretrained networks are effective and reduce computational overhead.

  • Discriminator configurations: Two independent discriminators improve over a single vanilla RGB discriminator, while adding discriminators only at deep layers hurts performance.Shallow features remain important because they contain most information about the original image.
  • Feature-network architecture: Compact EfficientNets outperform both ResNets and Transformers in the pretrained feature-network study.
  • Feature mixing: CCM moderately decreases feature-space distances, and adding CSM yields further improvements across configurations, especially at deeper layers.
  • Feature mixing: The best ablation configuration combines CCM, CSM, and all four discriminators; perceptual discrimination performs worse than multiple independent discriminators.
  • Feature-network architecture: Smaller feature models produce lower FIDs and reduce computational overhead, while ImageNet accuracy does not correlate with projected-GAN performance.

5 Comparison to State-of-the-Art

Projected GANs improve convergence speed, sample efficiency, and FID across datasets and resolutions, outperforming established baselines while requiring substantially fewer training images.

  • Evaluation Scope: The experiments span datasets from hundreds to millions of images, resolutions from 256^2 to 1024^2, and visual domains including clip-art, paintings, and photographs.The evaluation uses FID between 50k generated and all real images, while convergence is measured by images shown to the discriminator.
  • Convergence Speed: Projected FastGAN reaches StyleGAN2’s best FID after 1.1 M images instead of 88 M, reducing wall-clock time from 5 days to less than 3 hours.Projected training improves both convergence speed and final FID, particularly for FastGAN.
  • Sample Efficiency: Projected GAN significantly improves FID over both baselines on CLEVR subsets containing 1k and 10k images.The comparison evaluates data efficiency under randomly subsampled training sets.
  • Large Datasets: Projected GAN outperforms state-of-the-art models in FID across large datasets, including LSUN-Bedroom, FFHQ, and Cityscapes.On LSUN-Bedroom, it achieves FID 1.52 versus 6.15 for GANsformer.
  • Large Datasets: On LSUN-Church, Projected GAN achieves FID 3.18 after 1.1 M images, compared with StyleGAN2’s prior FID 3.39 after 88 M images.The paper reports similar speed-ups across the other large datasets.

6 Discussion and Future Work

The paper identifies systematic artifacts and architecture-specific limitations, while proposing further study of generator combinations and pretrained feature-space properties.

  • Failure Cases: Projected GAN sometimes produces floating heads and cutout-like animals with blurry or bland backgrounds on AFHQ.The paper links this failure case to image-classification features that may underrepresent backgrounds and composition.
  • Failure Cases: On FFHQ, Projected GAN can generate poor-quality samples with incorrect proportions and artifacts despite achieving state-of-the-art FID.This shows that low FID does not exclude visible sample defects in this setting.
  • Generator Limitations: StyleGAN is more difficult to tune and benefits less from projected training than FastGAN.The authors speculate that combining strengths of both generator architectures could improve projected training.
  • Future Work: The authors identify pretrained feature-space properties and efficient feature networks as open directions for improving projected GAN training.They also note that more efficient generative models can lower computational barriers while increasing risks such as deepfakes.

Appendix A Proofs

The proofs formulate Projected GAN training with fixed feature projections and independent discriminators, showing that the generator matches projected real and generated distributions at the minimax optimum, including under stochastic augmentations.

  • Training formulation: Projected GAN training uses fixed feature projectors and independent discriminators operating on different feature projections.The stochastic proof treats augmentations before projection as part of the projection composition.
  • Stochastic augmentations: The consistency result remains valid when stochastic differentiable augmentations are applied before the fixed feature projections.The augmentation parameters include both application probability and transformation parameters.
  • Optimal discriminator: The optimal discriminator for each projection is determined by the projected real and generated densities when the generator is fixed.The proof derives this optimum from the discriminator’s pointwise maximization objective.
  • Consistency result: The generator minimizes the objective exactly when projected real and generated distributions are equal for every feature projection.This condition is stated for the projected densities induced by each discriminator.

Appendix B Additional Metrics and Datasets

Additional metrics, human judgments, and datasets broadly support Projected GAN’s performance and diversity advantages, while revealing precision trade-offs at high resolution and isolated dataset-specific weaknesses.

  • Additional metrics: Projected GAN preserves its ranking across KID, SwAV-FID, CLIP-FID, VirTex-FID, and SWD, indicating that low FID is not specific to the projection feature space.The low SwAV-FIDs are reported as evidence against correlations between the projection and Inception networks explaining the FID results.
  • Diversity and mode coverage: High recall generally accompanies Projected GAN’s strong FID, KID, and SwAV-FID results, while baselines’ higher precision can coincide with mode collapse.On large datasets, baselines outperform Projected GAN in precision only on FFHQ, Cityscapes, and LSUN Church.
  • High-resolution behavior: At high resolutions, Projected GAN performs slightly worse in precision while achieving better mode coverage, with artifacts observed in some AFHQ-Cat samples.The paper says higher-resolution training warrants closer inspection; Table 6 reports the same precision exception for baselines.
  • Generalization: On small datasets, smooth latent interpolations across random samples suggest that Projected GAN generalizes rather than memorizes training examples.The paper notes that FID and other metrics do not detect small-dataset overfitting well.
  • Human evaluation: Human preference rankings largely agree with metric rankings, although FFHQ is reported as a failure case and AnimalFace may benefit from fewer low-quality samples.The AnimalFace explanation is presented as a hypothesis concerning blurry or compressed images in the dataset.
  • Additional datasets: Projected GAN outperforms the previous state of the art on all nine additional evaluated datasets at 256^2 without changing hyperparameters.The additional datasets include LSUN cat and horse, ADE indoor, Oxford flowers, KITTI-fisheye, STL-10, CUB200, Stanford Dogs, and Stanford Cars.

Appendix C Qualitative Comparisons

Qualitative comparisons present randomly selected, uncurated Projected GAN samples across diverse datasets and resolutions, including images up to 1024^2 pixels.

  • 256^2 comparisons: Projected GAN produces uncurated samples for CLEVR, FFHQ, Cityscapes, LSUN Bedroom, LSUN Church, Art Painting, Landscape, AnimalFace-Dog, Flowers, and Pokemon at 256^2.Each dataset’s samples are selected randomly using one global random seed.
  • 256^2 comparisons: Additional 256^2 qualitative results cover AFHQ-Cat, AFHQ-Dog, and AFHQ-Wild.The figures use random samples selected with a global seed and recommend zooming in for comparison.
  • Evaluation setup: The qualitative evaluation does not apply truncation to any model and uses one global seed per dataset.This standardizes the displayed random sample selection across the qualitative comparisons.
  • 1024^2 comparisons: Art Painting and Pokemon are also shown at 1024^2 resolution.The corresponding figures report randomly selected, uncurated results.

Appendix D Additional Experiments

Additional experiments support fixed pretrained features and channel-and-scale mixing as key design choices, while discriminator logits remain stable during Projected GAN training.

  • Pretrained features: Randomly initializing the feature network produces much higher FID than using pretrained representations.The ablation tests whether the architectural bias of the feature network alone is sufficient.
  • Channel mixing: CCM-Kaiming achieves FID 0.77, improving over Feature Norm 1.27, CCM-rotation 1.27, the RGB baseline 1.0, and pretrained F without projection 1.15.The result supports the hypothesis that sufficient channel mixing is needed.
  • Projection training: Longer denoising autoencoder pretraining of random projections increases FID in the reported ablation.The feature network remains fixed during this pretraining comparison.
  • Discriminator behavior: Projected GAN discriminator logits remain mostly constant during training, unlike the steadily increasing logits of the RGB baseline.The experiment uses up to four discriminators and keeps augmentation probability constant.

Appendix E Implementation Details

This section describes the codebases, hyperparameters, and training configurations used in the experiments.

  • The section details the codebases used for dataset preparation, training, and evaluation, together with hyperparameters and training configurations.

E.1 Code and Compute

Experiments use established codebases and specific hardware/software configurations, while training-speed comparisons are made at 256^2 resolution under controlled resources.

  • Dataset preparation, training, and evaluation build on StyleGAN2-ADA, with official Inception used for FID and KID evaluation.
  • Experiments run on an internal cluster using up to 8 Quadro RTX 6000 or NVIDIA V100 GPUs, PyTorch 1.7.1, and CUDA 11.0.
  • At 256^2 resolution, training times are measured in sec/kimg using 8 Quadro RTX 6000 GPUs, with fair comparison to FastGAN because custom CUDA kernels are unavailable for its generator.StyleGAN2 is fastest overall with mixed precision and custom CUDA kernels; Projected GAN uses no regularization beyond spectral normalization.

E.3 Hyperparameters

The appendix reports training-speed measurements and describes optimization settings, baseline hyperparameter searches, and Projected GAN architecture choices.

  • Training speed is reported in Table 10, while discriminator architectures are reported in Table 11.
  • All models use Adam with β1 = 0, β2 = 0.99, ϵ = 10^-8, and exponential moving averages for generator weights.
  • StyleGAN2-ADA: StyleGAN2-ADA uses standard resolution-specific configurations, with a grid search over γ ∈ {1, 10, 20, 50} for small datasets at 256^2 and 1024^2.The best configuration is continued after an initial 1 M images, and adaptive discriminator augmentation uses a target value of 0.6.
  • FastGAN: FastGAN uses learning rate 0.0002 and hinge loss, while batch size is searched over 8, 16, 32, and 64 because it directly affects performance.A batch size of 16 further improves the results over the default suggested batch size of 8.
  • Projected GAN: Projected GAN uses learning rate 0.0002, batch size 64, hinge loss, and resolution-dependent channel multipliers, with doubled base channel count from 64 to 128.The listed channel multipliers range from 16 at 42 resolution to 0.125 at 10242 resolution.
  • Projected GAN: Projected GAN discriminator architectures apply spectral normalization to all convolutional layers.DownBlocks combine a k = 4, stride-2 convolution with BatchNorm and LeakyReLU slope 0.2.
Loading 2111.01007v1…