PaperlayerView exact revision on arXiv

Source-linked AI summary

A Style-Based Generator Architecture for Generative Adversarial Networks

Tero Karras, Samuli Laine, Timo Aila

arXiv:1812.04948v3cs.NEcs.LGstat.ML

TL;DR

GAN generators remained difficult to interpret and lacked quantitative ways to compare latent-space interpolation. This paper introduces a style-based generator with new evaluation metrics, reporting better quality, disentanglement, and interpolation properties than traditional architectures.

  • Problem

    GAN generators remained black boxes, while latent-space properties were poorly understood and interpolation lacked quantitative comparison methods.

  • Method

    The paper redesigns the generator around layerwise style control and injected noise, and introduces perceptual path length and linear separability metrics for interpolation and disentanglement.

  • Results

    The style-based design is reported as superior to traditional GAN generators on established quality metrics, with better separation of attributes and stochastic effects and greater intermediate-latent linearity.

  • Takeaways & Limitations

    The architecture offers more controllable GAN synthesis while providing metrics for studying latent-space disentanglement and interpolation.

  • Takeaways & Limitations

    Directly shaping the intermediate latent space during training remains a direction for future work.

Abstract

from arXiv · show

We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.

NVIDIA · 1. Introduction

The paper introduces a style-based GAN generator that exposes scale-specific control, separates high-level attributes from stochastic variation, and improves latent-space disentanglement. It also presents FFHQ, a higher-quality, more varied human-face dataset released with code and pretrained networks.

  • 1. Introduction: GAN image resolution and quality have improved rapidly, but generators remain black boxes with poorly understood stochastic features and latent spaces.
  • 1. Introduction: The redesigned generator begins from a learned constant and adjusts image styles at each convolution layer using the latent code.Noise is injected directly into the network to model stochastic variation.
  • 1. Introduction: The architecture separates high-level attributes such as pose and identity from stochastic details such as freckles and hair.
  • 1. Introduction: Scale-specific style control enables intuitive mixing and interpolation operations in generated images.
  • 1. Introduction: The method leaves the discriminator and loss function unchanged, making the architectural contribution orthogonal to debates over GAN losses, regularization, and hyperparameters.
  • NVIDIA: The intermediate latent space can be disentangled because it is not constrained to follow the training-data probability density, unlike the input latent space.
  • 1. Introduction: FFHQ is introduced as a publicly available human-face dataset with higher quality and wider variation than existing high-resolution datasets.The authors also release source code and pretrained networks.

2. Style-based generator

The style-based generator replaces direct latent injection with a learned constant, an intermediate mapping network, and layer-wise style control through AdaIN. Explicit per-layer noise inputs add stochastic detail, while the resulting design improves FID and supports finer control through mixing regularization.

  • Architecture: The generator maps z through f: Z → W and uses w to control synthesis from a learned constant rather than an input layer.The mapping network produces w ∈ W, which controls the generator through layer-wise style parameters.
  • Architecture: AdaIN normalizes each feature map, then scales and biases it using style components derived from w at every convolution layer.The style vector’s dimensionality is twice the number of feature maps on that layer.
  • Stochastic detail: Dedicated single-channel Gaussian-noise images are added after each convolution, with learned per-feature scaling, to provide stochastic detail directly.A separate noise image is fed to each layer of the synthesis network and broadcast across feature maps.
  • Experimental validation: Noise inputs improve results further, while mixing regularization decorrelates neighboring styles and enables more fine-grained control over generated imagery.These are configurations E and F, respectively, in the reported generator comparisons.
  • Experimental validation: Almost 20%: style-based generator (E) improves FIDs over traditional generator (B), showing that the redesign improves image quality rather than compromising it.The comparison uses FID, for which lower is better, across CELEBA-HQ and FFHQ evaluations.

3. Properties of the style-based generator

The style-based generator localizes control across synthesis scales: styles govern coherent, meaningful image attributes, while per-pixel noise produces localized stochastic variation. Style mixing regularization further encourages this separation and improves tolerance to latent-code mixing at test time.

  • Style-based control: Styles control localized, scale-specific aspects of synthesis, with different style subsets governing meaningful high-level image attributes.The mapping network samples styles, while the synthesis network generates images from their collection.
  • Style-based control: AdaIN normalizes each channel before applying style-dependent scales and biases, allowing each style to control one convolution before being overridden.The style changes feature importance for the subsequent convolution independently of the original channel statistics.
  • Style mixing: Style mixing regularization switches between two latent codes at a random synthesis point, encouraging styles to localize during training.At test time, mixing multiple latents yields significantly improved FIDs, indicating better tolerance to adverse latent mixing.
  • Stochastic variation: Different noise realizations preserve overall composition and identity while changing localized stochastic details such as hair placement, silhouettes, backgrounds, and eye reflections.Noise is injected per pixel after each convolution, and its effects can be varied by layer subset.
  • Style and noise separation: Styles affect complete feature maps coherently across the image, whereas independently added pixel noise is suited to spatially varying stochastic effects.This separation supports coherent control of pose, lighting, and background style without using noise for spatially inconsistent decisions.

4. Disentanglement studies

The section argues that the style-based generator produces a less entangled intermediate latent space W by adapting the mapping from Z, and introduces perceptual path length and separability metrics that require no encoder. Experiments find W more perceptually linear and consistently more separable than Z, with mapping networks generally improving these properties and image quality.

  • Motivation: Disentanglement aims to organize latent space into linear subspaces controlling individual variation factors, but typical data distributions and latent sampling prevent complete separation.Artificial disentanglement datasets can hide this problem by uniformly tabulating all combinations of predetermined factors.
  • Intermediate latent space: The mapping f(z) induces W’s sampling density and can adapt W to make variation factors more linear, which should ease realistic image synthesis.The authors therefore expect unsupervised training to yield a less entangled W.
  • Perceptual path length: The perceptual path-length metric measures image changes along latent interpolations using weighted VGG16 distances, with shorter paths indicating greater perceptual linearity.For W, interpolation uses linear interpolation; measurements crop images to faces and average over 100,000 samples.
  • Perceptual path length: W has substantially shorter full-path lengths than Z for the style-based generator with noise inputs, indicating that W is perceptually more linear than Z.The authors note that this comparison is slightly biased in favor of Z because W may contain regions outside the input manifold that reconstruct poorly.
  • Separability: The separability metric fits linear SVMs to predict binary image attributes from latent points, and W is consistently better separable than Z, suggesting less entanglement.Increasing mapping-network depth improves separability and image quality in W; mapping networks also improve FID, separability, and path length in both architectures.

5. Conclusion

The authors conclude that style-based generator designs outperform traditional GAN generators on established quality metrics. They highlight disentanglement and intermediate-latent-space linearity as promising directions for understanding and controlling GAN synthesis, including direct training-time shaping.

  • Conclusion: Style-based generator designs are concluded to be superior to traditional GAN architectures across established quality metrics.The conclusion bases this assessment on the authors’ results and parallel work by Chen et al.
  • Conclusion: Separating high-level attributes from stochastic effects and improving intermediate latent-space linearity may advance GAN synthesis understanding and controllability.
  • Conclusion: Average path length and possibly linear separability metrics could serve as training regularizers, while directly shaping the intermediate latent space remains future work.

A. The FFHQ dataset

The Flickr-Faces-HQ (FFHQ) dataset contains 70,000 high-quality 1024^2 human-face images, offering substantially broader demographic, background, and accessory variation than CELEBA-HQ. It was built from Flickr images using automated alignment, cropping, filtering, and permissive-license selection, with Mechanical Turk cleanup, and released publicly.

  • Dataset characteristics: FFHQ contains 70,000 high-quality human-face images at 1024^2 resolution, with greater variation in age, ethnicity, backgrounds, and accessories than CELEBA-HQ.Its accessory coverage includes eyeglasses, sunglasses, hats, and similar items.
  • Dataset construction: The images were crawled from Flickr, automatically aligned and cropped, restricted to permissive licenses, and pruned using automatic filters.Mechanical Turk was used to remove occasional statues, paintings, and photos of photos.
  • Dataset release: The FFHQ dataset was made publicly available at https://github.com/NVlabs/ffhq-dataset.The release follows the filtering and manual cleanup process described for the collected images.

B. Truncation trick in W

The truncation trick improves average image quality by sampling a shrunken region of W, at the cost of some variation; it scales each latent deviation toward W’s center of mass and works reliably without changing the loss.

  • Motivation: Truncating the sampling space improves average image quality, although it sacrifices some variation because low-density training-data regions are difficult to learn.The problem of poorly represented low-density regions is common across generative modeling techniques.
  • Method: The method computes W’s center of mass as w̄ = E_z∼P(z)[f(z)] and transforms each latent vector using w′ = w̄ + ψ(w − w̄), with ψ < 1.For FFHQ, the center represents an average face.
  • Method: Truncation in W works reliably even without changing the loss, unlike prior observations that only some networks accommodate truncation despite orthogonal regularization.The comparison refers to Brock et al.’s observation about truncation behavior in other settings.

C. Hyperparameters and training details

The training setup largely inherits Progressive GANs, while the improved baseline changes sampling, progressive-growing initialization, and FFHQ loss and training duration. The style-based generator uses an 8-layer, 512-dimensional mapping network, and separability classifiers are trained independently with fixed architecture and optimization settings.

  • Original configuration: The original configuration inherits the Progressive GAN discriminator, resolution-dependent minibatch sizes, Adam hyperparameters, and generator exponential moving average.Mirror augmentation is enabled for CelebA-HQ and FFHQ but disabled for LSUN.
  • Improved baseline: The improved baseline replaces nearest-neighbor sampling with bilinear sampling using a separable 2nd order binomial lowpass filter.The filter is applied after each upsampling layer and before each downsampling layer in both networks.
  • Improved baseline: For FFHQ, the improved baseline switches to non-saturating loss with R1 regularization using γ = 10 and extends training from 12M to 25M images.The longer training reflects FID scores continuing to decrease considerably longer with R1 than with WGAN-GP.
  • Style-based generator: The style-based generator uses leaky ReLU with α = 0.2, equalized learning rate, and an 8-layer mapping network whose z and w activations have dimensionality 512.Its convolutional layers use the same feature-map counts as Karras et al.; increasing mapping-network depth tends to destabilize training at high learning rates.
  • Evaluation classifiers: The separability classifiers use the discriminator architecture without minibatch standard deviation, learning rate 10^-3, minibatch size 8, Adam, and training length of 150,000 images.They are trained independently of the generators, with the same 40 classifiers used across generators for the CelebA attributes.

D. Training convergence

With R1 regularization, FID continues to decrease slowly during training, motivating an increase in training duration from 12M to 25M images. At full 10242 resolution, perceptual path lengths slowly rise during training.

  • D. Training convergence: FID slowly decreases throughout training with R1 regularization, motivating an increase in training time from 12M images to 25M images.This behavior is shown for configurations B and F trained on FFHQ.
  • D. Training convergence: At full 10242 resolution, perceptual path lengths slowly rise as training progresses.The passage reports this alongside the continuing FID decrease.

E. Other datasets

On LSUN BEDROOM, CARS, and CATS, the style-based generator produced varied images with scale-specific control and dataset-dependent stochastic effects. Reported FIDs were 2.65 for BEDROOM and 3.27 for CARS, while BEDROOM results were limited by compression artifacts in the training data.

  • LSUN datasets: The generator produced uncurated LSUN BEDROOM, CARS, and CATS results using the truncation trick with ψ = 0.7.The trick was used for resolutions 4^2–32^2.
  • Quantitative results: 2.65 was the FID for 50K BEDROOM images, while 3.27 was the FID for 50K CAR images.BEDROOM was evaluated at 256^2; CARS at 512 × 384.
  • Style and stochastic effects: In BEDROOM, coarse styles controlled camera viewpoint, middle styles selected furniture, and fine styles controlled colors and material details.CARS showed roughly similar style effects.
  • Style and stochastic effects: Stochastic variation primarily affected fabrics in BEDROOM, backgrounds and headlamps in CARS, and fur, backgrounds, and paw positioning in CATS.Car wheels did not appear to rotate based on stochastic inputs.
  • Training and limitations: Training lasted 70M images for BEDROOM and CATS and 46M for CARS, with BEDROOM results affected by severe compression artifacts from low-quality training data.Higher-quality CARS data enabled 512 × 384 resolution instead of 256^2.
Loading 1812.04948v3…