PaperlayerView exact revision on arXiv

Source-linked AI summary

StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets

Axel Sauer, Katja Schwarz, Andreas Geiger

arXiv:2202.00273v2cs.LGcs.CV

TL;DR

StyleGAN performs poorly on diverse ImageNet-scale data, leaving open whether its architecture or training strategy is responsible. The paper combines Projected GAN training with StyleGAN3 adaptations, progressive growing, and classifier guidance to create StyleGAN-XL. The resulting model achieves state-of-the-art large-scale synthesis, including 1024^2 ImageNet-scale generation, while supporting inversion and editing.

  • Problem

    StyleGAN’s performance degrades on diverse datasets such as ImageNet, raising whether its restrictive architecture or training strategy limits scaling.

  • Method

    StyleGAN-XL combines Projected GAN training with architectural changes, progressive growing tailored to StyleGAN3, and classifier guidance for ImageNet synthesis.

  • Results

    StyleGAN-XL achieves state-of-the-art large-scale image synthesis and is the first to generate 1024^2 images at ImageNet scale, while enabling inversion and editing beyond narrow image domains.

  • Takeaways & Limitations

    The results show that StyleGAN can be scaled to large, diverse datasets through an appropriate training strategy rather than architectural replacement.

  • Takeaways & Limitations

    StyleGAN-XL is three times larger than StyleGAN3, increasing computational overhead for finetuning, and StyleGAN3-based editing is harder than with StyleGAN2.

Abstract

from arXiv · show

Computer graphics has experienced a recent surge of data-centric approaches for photorealistic and controllable content creation. StyleGAN in particular sets new standards for generative modeling regarding image quality and controllability. However, StyleGAN's performance severely degrades on large unstructured datasets such as ImageNet. StyleGAN was designed for controllability; hence, prior works suspect its restrictive design to be unsuitable for diverse datasets. In contrast, we find the main limiting factor to be the current training strategy. Following the recently introduced Projected GAN paradigm, we leverage powerful neural network priors and a progressive growing strategy to successfully train the latest StyleGAN3 generator on ImageNet. Our final model, StyleGAN-XL, sets a new state-of-the-art on large-scale image synthesis and is the first to generate images at a resolution of $1024^2$ at such a dataset scale. We demonstrate that this model can invert and edit images beyond the narrow domain of portraits or specific object classes.

1 INTRODUCTION

StyleGAN’s restrictive architecture performs poorly on diverse datasets such as ImageNet, raising whether the architecture or training strategy is the main limitation. The paper addresses this with Projected GAN training, architectural changes, progressive growing, and classifier guidance, producing StyleGAN-XL.

  • StyleGAN was designed to disentangle variation factors for improved control and interpolation, but its restrictive architecture performs poorly on complex, diverse datasets such as ImageNet.
  • BigGAN performs strongly on ImageNet but has variable training outcomes and lacks the intermediate latent space needed for GAN-based image editing.Diffusion models are more diverse but substantially slower at inference, while prior GAN-editing methods do not directly apply to them.
  • The paper asks whether StyleGAN’s failures on ImageNet reflect fundamental architectural constraints or an inadequate training strategy.Projected GANs improve training stability, training time, and data efficiency by comparing projected real and generated samples in a fixed pretrained feature space.
  • StyleGAN-XL combines Projected GAN training with architectural changes, a StyleGAN3-specific progressive growing strategy, and classifier guidance to scale StyleGAN to ImageNet.The authors also analyze pretrained feature networks used by Projected GANs and incorporate classifier guidance for additional class information.
  • StyleGAN-XL reaches state-of-the-art large-scale image synthesis, supports inversion and editing across ImageNet classes, and generates 1024^2 images at ImageNet scale.The model also matches prior diffusion-model state-of-the-art performance in a fraction of their training time and uses three times the parameters of standard StyleGAN3.

2 BACKGROUND

The paper builds on StyleGAN3 and Projected GAN components, including latent-style modulation, regularization, progressive growing, feature projection, and multi-scale discrimination. These components are adapted to improve training stability and prevent discriminator focus or mode collapse.

  • The system combines a StyleGAN3 generator with Projected GAN feature projectors and multi-scale discriminators.
  • StyleGAN maps a normally distributed latent code z to a style code w, which modulates synthesis-network convolution kernels to control image generation.StyleGAN3 begins synthesis from Fourier features and applies convolutional layers, nonlinearities, and upsampling.
  • Style mixing regularizes style-based generators by using independently sampled style codes at different synthesis layers, while path length regularization promotes smoother latent-to-image mappings.Path length regularization is associated with fewer artifacts, more predictable training, and better inversion.
  • Progressive growing was introduced for stable high-resolution training, but StyleGAN3 revisits it because its equivariant design prevents texture-sticking artifacts.The paper uses this compatibility to target faster convergence and improved synthesis quality.
  • Projected GAN maps real and generated images into discriminator input spaces through pretrained feature networks and feature-mixing modules.Cross-channel and cross-scale mixing use fixed differentiable random projections to prevent discriminators from focusing on only part of the feature space, which could cause mode collapse.
  • Projected GAN uses independent discriminators on feature projections, with multi-scale feature maps processed by convolutional discriminators whose depth depends on input resolution.Its setup uses spectral normalization and does not add gradient penalties; differentiable augmentation is applied before the pretrained feature network.
  • Projected GAN experiments found that StyleGAN generators can be quickly overpowered by discriminators because style modulation may slow generator adaptation.

3 SCALING STYLEGAN TO IMAGENET

StyleGAN-XL adapts StyleGAN3 for ImageNet through architectural, latent-space, feature-network, progressive-growing, and classifier-guidance changes. These modifications improve training stability, efficiency, sample quality, and high-resolution scaling.

  • Architecture: StyleGAN-XL builds on StyleGAN3-T because StyleGAN3-R produced overly symmetric, kaleidoscope-like images on complex datasets.The model uses translational-equivariant layers for ImageNet.
  • Latent space and conditioning: The method adapts the latent space and class conditioning for Projected GAN training, including low-dimensional latent codes and pretrained class embeddings.These changes target optimization and class control in the class-conditional setting.
  • Progressive growing: Progressive growing doubles resolution stage by stage, trains only newly added layers, and keeps lower-resolution layers fixed to prevent mode collapse.The schedule starts at 16^2, removes two layers and adds seven at each increase, and reaches 39 layers at 1024^2.
  • Progressive growing: At 512^2, progressive growing reaches the prior state-of-the-art FID = 3.85 after 2 V100-days, while retaining relatively strong translation equivariance with EQ-T = 48.Config-C has EQ-T = 55, whereas architectures with aliasing have EQ-T ∼15.
  • Feature networks: Combining EfficientNet and DeiT-base feature networks provides complementary representations for Projected GANs, while adding more networks yields no significant improvement.The feature-network ablation compares CNN and ViT architectures alongside supervised and self-supervised pretraining.
  • Classifier guidance: Classifier guidance significantly improves inception score at higher resolutions, but guidance below 32^2 leads to mode collapse.The generator loss adds a cross-entropy term scaled by λ, with λ = 8 working well empirically.

4 RESULTS

StyleGAN-XL is evaluated on ImageNet for synthesis, inversion, and image manipulation. It achieves strong synthesis results across resolutions while supporting inversion, editing, and style mixing beyond narrow single-domain settings.

  • Image Synthesis: StyleGAN-XL substantially outperforms BigGAN-deep, CDM, and ADM across resolutions in FID, sFID, rFID, and IS.Recall is an exception: StyleGAN-XL’s diversity lies between BigGAN and ADM.
  • Image Synthesis: 2.8 FID is achieved at 1024^2 resolution after training for one V100-day.No baseline comparison is reported at this resolution because of resource constraints.
  • Inversion and Manipulation: StyleGAN-XL achieves average PSNR = 13.5 for 512^2 ImageNet validation inversion, compared with BigGAN’s PSNR = 10.8.Its reconstructions are also reported as semantically closer to target images using FID between reconstructions and targets.
  • Inversion and Manipulation: Combining StyleGAN-XL with PTI enables almost-precise inversion of both ImageNet and out-of-domain images while preserving perceptually smooth generator outputs.PTI fine-tunes the generator around an initial inverted style code with regularization around the pivot.
  • Inversion and Manipulation: StyleGAN-XL supports latent-direction editing, extrapolation, and style mixing that can produce out-of-domain combinations between different classes.Style mixing works best for aligned samples and combines style codes from two inputs across different generator layers.

5 LIMITATIONS AND FUTURE WORK

The paper identifies model-size and editability trade-offs, while demonstrating high-resolution synthesis and several editing capabilities. Future work includes reducing computational overhead and improving controllability.

  • Future Work: The model achieves high-resolution ImageNet synthesis and supports inversion, interpolation, style mixing, and language-guided manipulation across diverse classes.The figures illustrate smooth interpolations, low-distortion inversion, cross-class style mixing, and directions such as smile, no stripes, and big eyes.
  • Architectural Limitations: StyleGAN-XL is three times larger than StyleGAN3, increasing computational overhead when used to initialize finetuning.The authors suggest GAN distillation as a possible way to trade performance for model size.
  • Architectural Limitations: StyleGAN-XL is harder to edit than StyleGAN2 through W, although StyleSpace techniques such as StyleMC perform better in their experiments.The reduced semantic controllability is associated with StyleGAN3’s emphasis on equivariance.

A PREPROCESSING IMAGENET

ImageNet preprocessing addresses its limited native resolution by applying super-resolution, while the resulting upsampling stack is designed to reduce computational cost and support later finetuning.

  • Preprocessing: 469×387 is ImageNet’s mean resolution, creating an initial challenge for high-resolution training.The authors preprocess the full dataset to address the lack of high-resolution data.
  • Preprocessing: SwinIR-Large is used for real-world image super-resolution during preprocessing.The approach is compared with generating 256^2 samples and then applying SwinIR.
  • Preprocessing: The upsampling stack is 60 times faster than SwinIR and preserves StyleGAN-XL weights for initialization on other high-resolution datasets.This design avoids the computational overhead of applying SwinIR after generation.

B CLASSES OF UNALIGNED HUMANS

Unaligned human-containing ImageNet classes remain challenging because realistic faces require sensitivity to fine details. ADM produces more convincing human faces than the compared GANs.

  • Classes of Unaligned Humans: ADM generates more convincing human faces than StyleGAN-XL and BigGAN.Both GANs can still synthesize realistic faces.
  • Classes of Unaligned Humans: The main challenge is that ImageNet is unstructured and its human images are not aligned.This setting differs from datasets where faces share a consistent alignment.
  • Classes of Unaligned Humans: Classes containing details to which human observers are more sensitive are particularly challenging.The paper illustrates these cases with examples in Fig. 7.

C INFERENCE SPEED

StyleGAN-XL offers a major inference-speed advantage over ADM because GANs generate samples in one forward pass, whereas diffusion models require repeated application.

  • Inference Speed: StyleGAN-XL is several orders of magnitude faster than ADM in the reported inference-speed comparison.The comparison measures a batch-size-one forward pass in V100-seconds.
  • Inference Speed: GANs generate samples in a single forward pass.This contrasts with diffusion models, which must be applied several hundred or thousand times per sample.
  • Inference Speed: The speed gap may narrow as diffusion-model sampling techniques continue to improve.The paper identifies faster diffusion sampling as an active research area.

D RESULTS ON UNIMODAL DATASETS

StyleGAN-XL is applied to unimodal datasets using a configuration that differs from ImageNet, and the reported results cover 1024^2-resolution generation and inversion. The model achieves state-of-the-art performance on both unimodal datasets.

  • D RESULTS ON UNIMODAL DATASETS: For big and small unimodal datasets, StyleGAN-XL uses ten layers at the lowest stage and adds two layers per resolution stage.This configuration differs from the ImageNet setup; classifier guidance is not used.
  • D RESULTS ON UNIMODAL DATASETS: Table 5 evaluates inversion by comparing model inversions with their reconstruction targets.The supplied table caption identifies the evaluation relationship but does not provide metric values.
  • D RESULTS ON UNIMODAL DATASETS: At 1024^2 resolution, StyleGAN-XL achieves state-of-the-art performance on both reported unimodal datasets.The results are reported in Table 4.

E ADDITIONAL QUALITATIVE RESULTS

Additional qualitative results examine class interpolation, samples from two unimodal datasets, and comparisons with BigGAN and ADM across ImageNet classes.

  • E ADDITIONAL QUALITATIVE RESULTS: StyleGAN-XL produces interpolations between samples from different classes.These interpolations are shown in Fig. 8.
  • E ADDITIONAL QUALITATIVE RESULTS: The qualitative results include StyleGAN-XL samples on FFHQ and Pokemon at 1024^2 resolution.The samples are shown in Figs. 10 and 11.
  • E ADDITIONAL QUALITATIVE RESULTS: ImageNet comparisons place BigGAN, ADM, and StyleGAN-XL side by side across multiple classes without truncation or classifier guidance.Images with the largest VGG16 logits are shown as a proxy for individual image quality.

F IMPLEMENTATION DETAILS

The implementation details cover inversion, training and evaluation protocols, progressive layer configuration, and qualitative visualizations of StyleGAN-XL outputs and comparisons.

  • F IMPLEMENTATION DETAILS: Inversion uses latent optimization in W after estimating an average style code and sampling class labels from a pretrained classifier’s logits.The class label is unknown for the target image, so classifier logits provide the class-sampling distribution.
  • F IMPLEMENTATION DETAILS: StyleGAN3 ImageNet training uses the official implementation, evaluates convergence through 10 million discriminator-seen images, and selects the best FID from three random seeds.The setup uses eight GPUs, total batch size 256, and γ=0.256.
  • F IMPLEMENTATION DETAILS: Evaluation uses the official StyleGAN3 codebase, excludes x-flip data amplification, and computes comparison metrics from released ADM and BigGAN samples where applicable.Baseline values in Table 2 are reported from Dhariwal and Nichol (2021).
  • F IMPLEMENTATION DETAILS: Progressive growing starts at 16^2 with 11 layers, then replaces two layers and adds seven at 32^2 before continuing toward 1024^2.Layer specifications follow Karras et al. (2021).
  • F IMPLEMENTATION DETAILS: Additional figures show class interpolations, inversion examples, FFHQ and Pokemon samples, and qualitative ImageNet comparisons with BigGAN and ADM.The ImageNet comparisons cover multiple named classes and use the layouts described in Figs. 12–14.
  • F IMPLEMENTATION DETAILS: The 1024^2 StyleGAN-XL design contains 39 layers whose cutoff and stopband frequencies follow geometric progression under sampling constraints.Fig. 15 visualizes cutoff, minimum acceptable stopband frequency, sampling rate, and actual stopband across layers.
Loading 2202.00273v2…