Analyzing and Improving the Image Quality of StyleGAN

Tero Karras; Samuli Laine; Miika Aittala; Janne Hellsten; Jaakko Lehtinen; Timo Aila

Source-linked AI summary

Analyzing and Improving the Image Quality of StyleGAN

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila

arXiv:1912.04958v2Dec 3, 2019cs.CVcs.LGcs.NEeess.IVstat.ML

TL;DR

StyleGAN produces high-quality images but exhibits characteristic artifacts and existing metrics miss some aspects of image quality. This paper redesigns normalization and training, improving StyleGAN2’s image quality and advancing the state of the art across several datasets.

Problem
StyleGAN exhibits characteristic artifacts, while FID and Precision and Recall miss some aspects of overall image quality.
Method
The paper redesigns generator normalization and network training, and regularizes synthesis mappings to improve image quality and smoothness.
Results
StyleGAN2 fixes several image-quality issues, improves quality across several datasets, and advances the state of the art.
Takeaways & Limitations
The improved model delivers higher-quality unconditional image synthesis and makes generated images easier to attribute to their source.
Takeaways & Limitations
FID and Precision and Recall do not accurately capture all aspects of image quality, particularly shape-related quality.

Abstract

from arXiv · show

The style-based GAN architecture (StyleGAN) yields state-of-the-art results in data-driven unconditional generative image modeling. We expose and analyze several of its characteristic artifacts, and propose changes in both model architecture and training methods to address them. In particular, we redesign the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images. In addition to improving image quality, this path length regularizer yields the additional benefit that the generator becomes significantly easier to invert. This makes it possible to reliably attribute a generated image to a particular network. We furthermore visualize how well the generator utilizes its output resolution, and identify a capacity problem, motivating us to train larger models for additional quality improvements. Overall, our improved model redefines the state of the art in unconditional image modeling, both in terms of existing distribution quality metrics as well as perceived image quality.

NVIDIA · 1. Introduction

The paper targets StyleGAN, then the state-of-the-art method for high-resolution image synthesis, by addressing its characteristic artifacts and further improving image quality. StyleGAN’s distinctive design maps z to w and uses affine-generated styles with AdaIN to control synthesis layers.

1. Introduction: StyleGAN is identified as the current state-of-the-art method for high-resolution image synthesis.It has been shown to work reliably across a variety of datasets.
1. Introduction: The paper focuses on fixing StyleGAN’s characteristic artifacts.Artifact removal is presented as a central motivation for the work.
1. Introduction: The work also aims to improve the quality of StyleGAN’s generated results further.This goal follows rapid progress in the resolution and quality of generative images.
1. Introduction: StyleGAN’s distinguishing feature is an unconventional generator architecture.Unlike architectures that feed z only to the network’s beginning, StyleGAN transforms it before synthesis.
1. Introduction: A mapping network f transforms the input latent code z ∈ Z into an intermediate latent code w ∈ W.This intermediate representation is used to control the synthesis network.
1. Introduction: Affine transforms convert the intermediate code into styles that control synthesis-network layers through adaptive instance normalization (AdaIN).The passage also introduces stochastic variation as an additional component of the architecture.

NVIDIA

The section identifies characteristic StyleGAN artifacts and addresses them through redesigned generator normalization and path-length regularization. These changes improve image quality and make generated images easier to attribute to their source network through latent-space projection.

Artifact analysis: StyleGAN’s characteristic artifacts arise from architectural and training issues that the paper addresses with targeted modifications.The paper identifies two artifact causes and proposes changes intended to eliminate them.
Artifact analysis: Redesigning generator normalization removes blob-like artifacts caused by a design flaw in the original architecture.The normalization redesign is presented as the solution to the first identified artifact source.
Path-length regularization: Regularizing the synthesis network for smooth latent-to-image mappings improves quality, while less frequent regularization preserves effectiveness and reduces computational expense.The regularizer is motivated by perceptual path length, whose interpolations correlate with shape consistency and stability.
Artifact analysis: Instance normalization causes systemic water droplet-like artifacts that appear in all feature maps starting at 64×64 resolution.The artifacts may not always be obvious in generated images, but the internal activation problem is consistently present.
Inversion and attribution: Projection into latent space W works significantly better with the path-length-regularized StyleGAN2 generator, simplifying attribution of generated images to their source network.The improvement is specifically reported relative to the original StyleGAN.

2. Removing normalization artifacts

StyleGAN’s characteristic droplet artifacts arise from AdaIN destroying relative feature magnitudes, allowing localized spikes to bypass normalization. Weight demodulation removes these artifacts while preserving style controllability and largely maintaining FID.

Artifact diagnosis: Droplet-shaped artifacts appear around 64×64 resolution across all feature maps and intensify at higher resolutions.In rare cases, approximately 0.1% of images lack the droplet and become severely corrupted.
Artifact diagnosis: AdaIN can destroy relative feature magnitudes, enabling localized spikes to bypass instance normalization and encode signal-strength information.The hypothesis attributes the artifact to a strong localized spike dominating feature statistics.
Weight demodulation: Weight demodulation replaces instance normalization by adjusting convolution weights according to expected activation statistics, producing a single style-conditioned convolution.The method scales each output feature map by the inverse of its expected standard deviation, with a small ϵ for numerical stability.
Results: The redesigned normalization removes characteristic artifacts while retaining full style controllability.Removing normalization entirely would also remove artifacts but sacrifice scale-specific controls; demodulation avoids that trade-off.
Results: FID remains largely unaffected, while the redesign shifts performance from precision toward recall.The authors argue recall is preferable because truncation can trade recall back into precision, whereas the reverse is not true.

3. Image quality and generator smoothness

Perceptual path length (PPL) correlates with perceived image quality, with lower values indicating smoother latent-to-image mappings and higher-quality images. Path length regularization improves conditioning and tightens PPL variation without collapsing it to zero, though it can trade off against FID on less structured datasets.

Metric limitations: FID and P&R can assign identical scores to generators with markedly different image quality because their feature spaces emphasize texture over shape.FID uses InceptionV3 features and P&R uses VGG-16 features, both trained with texture-biased classifiers.
Perceptual path length: Lower PPL correlates with higher perceived image quality and semantic consistency, whereas high PPL indicates lower-quality images.Per-image PPL is computed from LPIPS distances under small latent-space perturbations, and its distribution can have a long tail.
Path length regularization: The regularizer encourages fixed-size latent steps to produce non-zero, fixed-magnitude image changes by promoting well-conditioned generator Jacobians.Its ideal is achieved when the Jacobian is orthogonal up to a global scale, preserving vector lengths without squeezing dimensions.
Path length regularization: Path length regularization makes models more reliable, easier to explore architecturally and invert, while tightening per-image PPL without pushing its mode to zero.The method can nevertheless involve a tradeoff between FID and PPL on datasets less structured than FFHQ.

4. Progressive growing revisited

Progressive growing stabilizes high-resolution synthesis but introduces phase artifacts, so StyleGAN2 replaces it with skip-generator and residual-discriminator architectures. These designs preserve the shift from coarse to fine details, reveal a highest-resolution capacity problem, and motivate larger models that improve image quality.

4. Progressive growing revisited: Progressive growing can cause phase artifacts in which details such as teeth remain aligned to preferred image locations instead of following pose smoothly.Figure 6 illustrates teeth staying aligned to the camera rather than following the pose.
4. Progressive growing revisited: StyleGAN2 uses a skip generator and residual discriminator without progressive growing, significantly improving FID and PPL.Table 2 evaluates feedforward, skip, and residual architectures without progressive growing; skip generators improve PPL, while residual discriminators benefit FID.
4. Progressive growing revisited: The revised architectures retain progressive growing’s desired behavior by initially emphasizing low-resolution features before shifting attention toward finer details during training.Skip connections allow low-resolution outputs to remain largely unaffected by higher-resolution layers early in training.
4. Progressive growing revisited: The highest-resolution layers fail to dominate late training, indicating that the generator cannot fully utilize the target resolution.The authors infer a capacity problem and test doubling feature maps in the highest-resolution layers.
4. Progressive growing revisited: Doubling high-resolution feature maps increases their contribution and markedly improves FID and Recall, while StyleGAN2 continues to outperform the additional-capacity StyleGAN baseline.The change increases generator parameters by 22% (25M → 30M) and discriminator parameters by 21% (24M →29M).

5. Projection of images to latent space

This section presents projection of images into StyleGAN’s latent space for image manipulation and source attribution. The method optimizes extended per-layer latent codes and stochastic noise, with noise regularization, and StyleGAN2 enables especially accurate projection of generated images.

Projection method: Projection first finds a matching latent code for image manipulation, with prior work favoring a separate w for each generator layer over one common code.This extended latent space produces closer matches, according to the supplied passage.
Projection method: The method injects ramped-down noise during optimization and jointly optimizes stochastic noise inputs while regularizing their multiscale autocorrelations toward unit Gaussian noise.These choices are intended to explore the latent space more comprehensively and prevent noise inputs from carrying coherent signal.
Source attribution: Projection can support attribution by testing whether an analyzed image has a latent code w ∈ W that re-synthesizes it.The section frames attribution as identifying whether a fake image can be reproduced by a specific source generator.
Projection quality: StyleGAN2-generated images can be projected into W almost perfectly, whereas baseline StyleGAN projections often mismatch especially in the backgrounds.Projection quality is measured with LPIPS distance between original and re-synthesized images, using LSUN CAR and FFHQ examples and histograms.

6. Conclusions and future work · A. Image quality

StyleGAN2 fixes several StyleGAN image-quality issues, advances the state of the art across several datasets, and makes generated images easier to attribute to their source. The authors also report faster training and identify future work on path-length regularization and reducing training-data requirements, while the appendix examines metric behavior and artifacts linked to progressive growing.

6. Conclusions and future work: StyleGAN2 fixes several StyleGAN image-quality issues, further improves quality across several datasets, and makes generated images easier to attribute to their source.The quality improvements are sometimes more apparent in motion.
6. Conclusions and future work: StyleGAN2 config E trains 40% faster than original StyleGAN config A at 61 img/s versus 37 images per second at 10242 resolution.The speedup mainly comes from weight demodulation, lazy regularization, and code optimizations.
6. Conclusions and future work: StyleGAN2 config F trains at 31 img/s, making it only slightly more expensive to train than original StyleGAN; training took 9 days for FFHQ and 13 days for LSUN CA.The passage attributes the comparison to larger StyleGAN2 networks.
6. Conclusions and future work: Future work could improve path-length regularization by replacing pixel-space L2 distance with a data-driven feature-space metric.The authors also emphasize reducing training-data requirements, especially where acquiring tens of thousands of samples is infeasible.
A. Image quality: Figure 11 illustrates the quality and diversity achievable in FFHQ, while Figure 12 presents uncurated results for all datasets mentioned in the paper.These large images provide visual examples related to image quality.
A. Image quality: Figure 15 shows a rare failure of the StyleGAN blob artifact that produces a seriously broken image, while Figure 16 shows progressive growing introduces higher-frequency intermediate-layer content that compromises shift invariance.The authors hypothesize that this causes uneven location preferences for details.

B. Implementation details · C. Effects of path length regularization

The implementation retains most StyleGAN details while introducing revised generator components, lazy regularization, and path length regularization across specified configurations. The path length regularizer targets the scale of generator gradients with respect to latent codes and is evaluated using a Jacobian-based formulation.

B. Implementation details: The implementation uses Z and W dimensionality 512, an 8-layer mapping network, a 100× lower mapping-network learning rate, equalized learning rates, leaky ReLU with α = 0.2, and bilinear filtering.These details were kept mostly unchanged from the official TensorFlow implementation corresponding to configuration A in Table 1.
B. Implementation details: Configurations B–F replace the original StyleGAN generator with a revised architecture using normally initialized constant-input components and shared noise scaling across feature maps.All weights use N(0, 1), while biases and noise scaling factors are initialized to zero except specified affine-transformation biases.
B. Implementation details: Configurations C–F apply lazy regularization in separate passes, using k = 16 for the discriminator and k = 8 for the generator.The optimizer state is shared between the main loss and regularization terms, and the regularization term is multiplied by k.
B. Implementation details: Configurations D–F include path length regularization, with target scale a initialized to zero and tracked per GPU using an exponential moving average with βpl = 0.99.The regularizer is averaged over all individual layers of the synthesis network to interact correctly with style mixing regularization.
B. Implementation details: Configurations A–D use progressive growing, whereas E–F disable it and use a fixed learning rate λ = 2 · 10−3.Progressive growing starts at 82 resolution, trains 600k images per resolution, fades in the next resolution for 600k images, and increases learning rate gradually by 3×.
C. Effects of path length regularization: The path length regularizer uses a unit normal random variable y in generated-image space and the generator Jacobian Jw at latent point w.The generated-image dimension is M = 3wh, and the latent-space dimension is L.

C.1. Effect on pointwise Jacobians · C.2. Effect on global properties of generator mapping

The path length prior is approximately minimized when the generator’s Jacobian has equal singular values at every latent point, yielding orthogonality up to a global scale. Such Jacobians preserve curve lengths and imply an isometric latent-to-image embedding, although practical mappings only approach this ideal and still benefit from reduced detours.

C.1. Effect on pointwise Jacobians: The minimizer is a constant multiple of the identity in singular-value space, so orthogonal matrices with suitable global scaling achieve the same pointwise optimum.The radial concentration of high-dimensional normal inputs makes equal singular values optimal.
C.1. Effect on pointwise Jacobians: At high latent dimensionality, the path length prior is minimized when all Jacobian singular values equal a global constant at every latent point.This makes the Jacobian orthogonal up to a globally constant scale.
C.1. Effect on pointwise Jacobians: The global scale of the Jacobians should match the scale already induced by initialization, because otherwise early training may focus on adjusting weight magnitudes instead of enforcing the intended objective.Using the existing global average avoids forcing a specific Jacobian scale.
C.1. Effect on pointwise Jacobians: Path length regularization brings Jacobian singular values closer together in practice, improving conditioning even though exact orthogonality is not reached.The conditioning improvement is correlated with the PPL metric, and the effect is especially pronounced for the more variable Cars dataset.
C.2. Effect on global properties of generator mapping: A generator with everywhere orthogonal Jacobians preserves the lengths of all curves between latent space and image space.The chain rule shows that the Jacobian leaves each curve’s velocity 2-norm unchanged.
C.2. Effect on global properties of generator mapping: Under exact isometry, the generator embeds Euclidean latent space into an image submanifold while mapping latent straight lines to image-manifold geodesics.These geodesics are shortest paths on the embedded manifold.
C.2. Effect on global properties of generator mapping: Exact isometry is not achieved in practice because the training criteria may conflict and an isometric mapping may not express the desired image manifold.Nevertheless, encouraging approximate isometry has desirable consequences.
C.2. Effect on global properties of generator mapping: Approximate isometry discourages unnecessary detours in the generator’s mapping between latent points and images.The supplied passage introduces this consequence but does not provide the remainder of its specific argument.

D. Projection method details

The projection method jointly optimizes a latent code and per-layer noise maps to match a target image, using LPIPS image loss and noise regularization to prevent signal leakage into the noise inputs. Optimization starts from the mapping-network mean with normalized random noise and uses a scheduled Adam procedure.

Projection setup: Projection seeks a latent w and per-layer noise maps ni to reconstruct a target image x; the baseline 1024×1024 generator has 18 noise inputs, while the improved architecture has one fewer.There are two noise inputs per resolution from 4×4 through 1024×1024; the improved model omits noise injection into the learned 4×4 constant.
Optimization procedure: Initialization sets w = µw, estimated from 10 000 random latent codes, and initializes each noise map as ni = N(0, I).During the first 750 iterations, Gaussian noise is added to w during loss evaluation to stabilize optimization.
Optimization procedure: The trainable parameters are all components of w and every noise map ni, optimized for 1000 iterations with Adam using a maximum learning rate of λmax = 0.1.The learning rate ramps up linearly during the first 50 iterations and decays with a cosine schedule during the last 250 iterations.
Loss function: The image-quality term is LPIPS distance between the target and synthesized images, computed after downsampling both to 256×256 resolution.Noise-map regularization is added alongside the image term because directly optimizing noise maps can otherwise place actual signal in them.
Loss function: Noise regularization forms multiscale pyramids for maps larger than 8×8 and penalizes resolution-normalized horizontal and vertical autocorrelations, with α = 105 and post-step normalization.The downsampled maps are used only for regularization and do not participate in synthesis.

E. Results with spectral normalization · F. Energy consumption

StyleGAN2 performs better without spectral normalization, whose generator effect is largely neutralized by weight demodulation. The project consumed approximately 131.61 MWh, with half spent on early exploration and one quarter each on targeted experiments and paper preparation.

E. Results with spectral normalization: Spectral normalization was evaluated alongside or instead of weight demodulation and path length regularization in StyleGAN2.
E. Results with spectral normalization: Adding spectral normalization to the generator is almost a no-op because weight demodulation overrides its scaling except in tRGB layers.SN scales each layer’s weights by 1/σ(w), but Equation 3 overrides this in the main convolutional and affine layers.
E. Results with spectral normalization: Discriminator spectral normalization slightly compromises FID, while enabling it in the generator produces significantly worse results.Disabling subsets of the proposed contributions does not improve the outcome.
F. Energy consumption: The project’s computational effort and electricity consumption were documented in GPU time and energy terms.The computation unit was single-GPU years on a Volta-class GPU, with runs converted by scaling for the number of GPUs used.
F. Energy consumption: Approximately 131.61 MWh of electricity was consumed by the entire project.Estimates used logged job durations, GPU counts, cluster locations, and measured DGX-1 power draw under training.
F. Energy consumption: Approximately half of total energy supported early exploration, while a quarter supported targeted experiments and another quarter supported paper preparation and release.
F. Energy consumption: Training a single FFHQ network using config F took approximately 0.68 MWh, or 0.5% of total project expenditure.Table 5 summarizes the project’s computational effort and electricity consumption; executing the project on one NVIDIA V100 would have taken approximately 51 years.