PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen

arXiv:1710.10196v3cs.NEcs.LGstat.ML

TL;DR

GAN training can suffer from unstable optimization and incomplete variation. This paper progressively grows generator and discriminator resolution, improving convergence while reducing total training time by about a factor of two.

  • Problem

    GAN training can produce unstable gradients when generated and training distributions have little overlap, while preserving variation remains an active evaluation concern.

  • Method

    The method progressively grows both GAN networks from low resolution to higher resolution by adding layers that shift learning toward increasingly fine-scale details.

  • Results

    Progressive growing converges to a considerably better optimum and reduces total training time by about a factor of two.

  • Takeaways & Limitations

    The approach supports stable GAN training at large resolutions while producing generally high-quality results across evaluated image datasets.

  • Takeaways & Limitations

    The results remain short of true photorealism, with shortcomings in semantic sensibility, dataset-dependent constraints, and image micro-structure.

Abstract

from arXiv · show

We describe a new training methodology for generative adversarial networks. The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses. This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2. We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10. Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator. Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation. As an additional contribution, we construct a higher-quality version of the CelebA dataset.

1 INTRODUCTION

The introduction identifies unstable GAN training, especially for high-resolution images, and presents progressive growth of the generator and discriminator as the central solution. It also motivates improved variation, training safeguards, and evaluation on standard datasets plus a higher-quality CELEBA dataset.

  • 1 INTRODUCTION: A GAN uses a generator to produce samples from latent codes and a discriminator to assess whether they match the training distribution.Because both networks are differentiable, the discriminator’s assessment provides gradients for steering the networks.
  • 1 INTRODUCTION: When generated and training distributions have little overlap, distance gradients can point in more-or-less random directions, destabilizing GAN optimization.The introduction notes that least-squares, absolute-deviation, and Wasserstein-based alternatives have been proposed.
  • 1 INTRODUCTION: High-resolution generation worsens this gradient problem because generated images become easier to distinguish, while memory constraints force smaller minibatches.The proposed insight is to grow both networks progressively from low-resolution images and add layers during training.
  • 1 INTRODUCTION: The introduction highlights variation preservation as an important GAN concern and notes that the conventional quality–variation tradeoff has been challenged.It references inception score and other proposed measures of preserved variation.
  • 1 INTRODUCTION: Mode collapse can begin within a dozen minibatches when discriminator overshooting causes exaggerated gradients and escalating signal magnitudes.The authors propose a mechanism to stop this unhealthy competition and describe an initialization modification for more balanced learning speeds across layers.
  • 1 INTRODUCTION: The work evaluates its contributions on CELEBA, LSUN, and CIFAR10 and creates a higher-quality CELEBA version supporting output resolutions up to 1024 × 1024 pixels.The introduction states that the new dataset and full implementation are publicly available.

2 PROGRESSIVE GROWING OF GANS

The method progressively grows synchronized generator and discriminator networks from low resolution, shifting from large-scale structure to fine details. This improves stability and reduces training time while preserving trainability through smooth layer transitions.

  • Method: Progressive training starts at low resolution and adds layers incrementally, first learning large-scale image structure before increasingly fine details.The networks progressively increase image resolution rather than learning all scales simultaneously.
  • Method: The generator and discriminator grow synchronously as mirror-image networks, with existing layers remaining trainable and new layers faded in smoothly.Smooth fading avoids sudden shocks to already well-trained lower-resolution layers.
  • Benefits: Progressive training makes early generation more stable because lower-resolution images contain less class information and fewer modes.Increasing resolution gradually poses a simpler mapping problem than directly generating 1024^2 images.
  • Benefits: 2–6 times faster: comparable result quality is often achieved at this speedup, depending on the final output resolution.Most training iterations occur at lower resolutions.

3 INCREASING VARIATION USING MINIBATCH STANDARD DEVIATION

The section addresses GANs’ tendency to capture only part of the training-data variation with a simplified minibatch-standard-deviation layer. The parameter-free approach summarizes feature variability across minibatches into one replicated feature map, while alternative methods may increase variation more or complement it.

  • Motivation: GANs tend to capture only a subset of the variation in training data, motivating methods that compare statistics across generated and training minibatches.Minibatch discrimination computes feature statistics across images and adds a learned layer near the discriminator’s end.
  • Method: The proposed solution computes per-feature, per-location minibatch standard deviations, averages them into one value, and concatenates the replicated value as an additional feature map.The layer has neither learnable parameters nor new hyperparameters and can be inserted anywhere in the discriminator.
  • Alternatives: Alternative approaches include discriminator unrolling, a generator repelling regularizer, and multiple generators, all targeting increased variation.The repelling regularizer encourages minibatch feature vectors to orthogonalize by adding a generator loss term.
  • Limitations: The authors acknowledge that these alternatives may increase variation more than their solution or may be orthogonal to it.This qualifies the relative scope of the simplified minibatch-standard-deviation approach.

4 NORMALIZATION IN GENERATOR AND DISCRIMINATOR

The section replaces conventional normalization practices with runtime weight scaling and pixelwise feature normalization to control signal magnitudes in GANs. The generator’s unit-length constraint prevents escalation effectively without substantially harming results across most datasets.

  • Motivation: Earlier GAN approaches commonly used batch-normalization variants because unhealthy competition can escalate signal magnitudes, although the authors did not observe covariate shift as an issue.These methods were often applied in the generator and sometimes also in the discriminator.
  • Weight scaling: The authors use trivial N(0, 1) initialization and explicitly scale weights at runtime by the per-layer He-initializer constant.They set ˆwi = wi/c, where c is the per-layer normalization constant from He’s initializer.
  • Pixelwise normalization: The generator normalizes each pixel’s feature vector to unit length after every convolutional layer to prevent signal magnitudes from spiraling out of control.This is implemented as a variant of local response normalization.
  • Pixelwise normalization: 10−8 is used for ϵ in the normalization formula, which prevents signal-magnitude escalation very effectively when needed.The authors report that this heavy-handed constraint generally does not harm the generator and changes results little on most datasets.

5 MULTI-SCALE STATISTICAL SIMILARITY FOR ASSESSING GAN RESULTS

The section proposes a multi-scale statistical similarity metric for comparing GAN results automatically across image collections. It compares local patch distributions from generated and target images across Laplacian-pyramid scales, capturing both image structure and variation.

  • Motivation: Existing MS-SSIM detects large-scale mode collapse but misses reduced variation in colors or textures.The authors motivate automated metrics because comparing many generated images is tedious, difficult, and subjective.
  • Method: The proposed metric compares local patch distributions from generated and target images across Laplacian-pyramid scales starting at 16 × 16 pixels.The pyramid progressively doubles in resolution, and the method is based on the intuition that successful samples match training-set local structure at all scales.
  • Method: Each pyramid level represents a spatial-frequency band; the method samples 16384 images and extracts 128 descriptors per level.Each descriptor is a 7 × 7 pixel neighborhood with 3 color channels, represented as x ∈R7×7×3 = R147.
  • Method: The method normalizes descriptors by each color channel’s mean and standard deviation, then estimates similarity with sliced Wasserstein distance using 512 projections.Sliced Wasserstein distance is an efficiently computable randomized approximation to earthmovers distance.
  • Interpretation: A small Wasserstein distance indicates similar patch distributions, while low- and high-resolution levels reflect large-scale structure and pixel-level attributes, respectively.The finest-level patches encode attributes such as edge sharpness and noise, whereas 16 × 16 patches indicate large-scale image structures.

6 EXPERIMENTS

Experiments evaluate the contributions using SWD and MS-SSIM on CELEBA and LSUN BEDROOM, showing that progressive growing improves convergence, training speed, and image quality. The resulting method supports varied 1024 × 1024 CELEBA-HQ images and achieves 8.80 in unsupervised CIFAR10.

  • Evaluation setup: SWD and MS-SSIM evaluate individual contributions on unsupervised CELEBA and LSUN BEDROOM using WGAN-GP as the baseline.The CELEBA comparison uses 1282 resolution and reports metrics for cumulatively enabled contributions.
  • Progressive growing: Progressive growing produces sharper, more believable images and makes SWD find the generated distribution more similar to the training set.This comparison enables progressive growing on top of the baseline configuration.
  • Progressive growing: About a factor of two reduction in total training time accompanies convergence to a considerably better optimum with progressive growing.The method’s speed advantage increases with output resolution because early networks are shallow and quick to evaluate; throughput equalizes after reaching full resolution.
  • High-resolution results: 30000 CELEBA images at 1024 × 1024 resolution form the high-quality CELEBA-HQ dataset used to demonstrate high-resolution generation.The dataset was created because previously used public GAN datasets were generally limited to 322–4802 resolutions.
  • High-resolution results: 1024 × 1024 CELEBA-HQ outputs are reported as more varied and higher in perceptual quality than earlier megapixel GAN results.The paper also presents selected 2562 results across seven different LSUN categories and describes overall quality as high.

7 DISCUSSION

The results are generally high-quality and training remains stable at large resolutions, but true photorealism is not yet achieved. Remaining challenges include semantic sensibility, dataset-dependent constraints, and image micro-structure, although convincing realism may be within reach in CELEBA-HQ.

  • Limitations and outlook: The method still falls short of true photorealism, with weaknesses in semantic sensibility, dataset-dependent constraints, and image micro-structure.The authors nonetheless suggest that convincing realism may be within reach, especially in CELEBA-HQ.

A NETWORK STRUCTURE AND TRAINING CONFIGURATION … B LEAST-SQUARES GAN (LSGAN) AT 1024 × 1024

The paper specifies progressively grown network configurations from 4 × 4 resolution through full-resolution CELEBA-HQ models, with detailed normalization and optimization choices. It also reports that LSGAN can produce 1024^2 images but is less stable and requires adaptive discriminator noise to prevent training collapse.

  • A.1 1024 × 1024 NETWORKS USED FOR CELEBA-HQ: The generator and discriminator use replicated 3-layer blocks introduced progressively, starting at 4 × 4 resolution and adding layers during training.At full resolution, the generator’s final Conv 1 × 1 and discriminator’s initial Conv 1 × 1 serve as toRGB and fromRGB, respectively.
  • A.1 1024 × 1024 NETWORKS USED FOR CELEBA-HQ: 512-dimensional latent hypersphere vectors generate images represented in [-1,1], with leaky ReLU leakiness 0.2 except for linear final layers.The networks omit batch, layer, and weight normalization, while the generator applies pixelwise feature-vector normalization after each Conv 3 × 3 layer.
  • A.1 1024 × 1024 NETWORKS USED FOR CELEBA-HQ: Adam training uses α = 0.001, β1 = 0, β2 = 0.99, and ϵ = 10^-8 without learning-rate decay or rampdown.Generator visualizations use an exponential running average of weights with decay 0.999, while minibatches are reduced at higher resolutions.
  • A.1 1024 × 1024 NETWORKS USED FOR CELEBA-HQ: The discriminator loss adds a drift penalty L′ = L + ϵdriftEx∈Pr[D(x)^2], using ϵdrift = 0.001 to keep outputs near zero.This term is assigned an extremely small weight.
  • A.2 OTHER NETWORKS: Lower spatial resolutions are obtained by omitting the appropriate number of replicated 3-layer blocks from both networks.This reuses the same progressively structured design across resolutions.
  • A.2 OTHER NETWORKS: The lower-capacity networks halve Conv 3 × 3 feature maps at 16 × 16 and divide them by 4 thereafter, leaving 32 final feature maps.Table 1 and Figure 4 train each resolution and fade in layers for 600k images instead of 800k.
  • A.2 OTHER NETWORKS: The Gulrajani et al. (2017) configuration uses α = 0.0001, β2 = 0.9, ncritic = 5, ϵdrift = 0, and minibatch size 64.It disables progressive resolution, minibatch stddev, and runtime weight scaling, and changes the generator’s activations and normalization.
  • B LEAST-SQUARES GAN (LSGAN) AT 1024 × 1024: LSGAN is generally less stable than WGAN-GP and can lose variation during long runs, although the paper produces 1024^2 images with LSGAN.Adaptive multiplicative Gaussian noise is added to discriminator inputs as a function of discriminator output to prevent uncontrolled spiraling when the dataset is too easy.

C CELEBA-HQ DATASET

The authors construct CELEBA-HQ by processing the original in-the-wild CelebA images into 1024 × 1024 outputs, then ranking them by estimated image quality and retaining the best 30000. The pipeline combines artifact removal, 4x super-resolution, padding and filtering, landmark-based cropping, and high-quality resampling.

  • Dataset construction: 30000 images at 1024 × 1024 comprise the high-quality CELEBA-HQ dataset, created from the original CelebA in-the-wild collection.The source images vary widely in resolution and visual quality, from 43 × 55 to 6732 × 8984 pixels.
  • Processing pipeline: Two pretrained networks remove JPEG artifacts and apply 4x super-resolution to improve each source image’s visual quality.The networks are a convolutional autoencoder and an adversarially trained super-resolution model.
  • Processing pipeline: Mirror padding and Gaussian filtering extend images and create a visually pleasing depth-of-field effect before facial landmarks define the crop region.The crop rectangle is designed to remain consistent for faces viewed from different angles.
  • Quality selection: 202599 images are processed, ranked using a frequency-based quality metric, and reduced to the best 30000 1024 × 1024 images.The metric favors broad, approximately radially symmetric power spectra and penalizes blur and conspicuous directional features.

D CIFAR10 RESULTS · E MNIST-1K DISCRETE MODE TEST WITH CRIPPLED DISCRIMINATOR

The CIFAR10 experiments report an unsupervised inception score of 8.80, while the MNIST-1K test shows that WGAN-GP, the proposed normalization, and minibatch standard-deviation features improve discrete-mode coverage and KL divergence under crippled discriminators.

  • D CIFAR10 RESULTS: CIFAR10 results are evaluated with inception scores reported both as the highest observed score and as statistics across ten random initializations.The second reporting method computes the mean and standard deviation from the highest scores across ten runs.
  • E MNIST-1K DISCRETE MODE TEST WITH CRIPPLED DISCRIMINATOR: The MNIST comparison uses a pre-trained classifier with 0.4% error rate to classify three-channel digits and form numbers in [0, 999].The related setup generates 25,600 images and evaluates discrete-mode coverage and KL divergence.
  • D CIFAR10 RESULTS: 8.80 inception score is achieved on unsupervised CIFAR10 without label conditioning.Figure 9 describes the generated images as non-curated and reports the score for an unsupervised network.
  • D CIFAR10 RESULTS: The CIFAR10 comparison includes DCGAN, Improved GAN, AC-GAN, SGAN, WGAN-GP, and Splitting GAN.Reported inception scores range from 6.58 for DCGAN to 8.87 ± 0.09 for Splitting GAN.
  • E MNIST-1K DISCRETE MODE TEST WITH CRIPPLED DISCRIMINATOR: The MNIST-1K experiment measures how many discrete modes are covered and computes KL divergence between the mode histogram and a uniform distribution.The setup follows a low-capacity generator and severely crippled K/2 and K/4 discriminators, with approximately 2000 and 500 parameters.
  • E MNIST-1K DISCRETE MODE TEST WITH CRIPPLED DISCRIMINATOR: WGAN-GP loss covers substantially more MNIST modes than the original GAN loss and also outperforms unrolled original GAN with the smaller K/4 discriminator.KL divergence favors WGAN-GP even more strongly than the raw mode count.
  • E MNIST-1K DISCRETE MODE TEST WITH CRIPPLED DISCRIMINATOR: Replacing batch normalization with equalized learning rate and pixelwise normalization considerably improves the MNIST result while removing some discriminator parameters.The minibatch stddev layer further improves scores and restores discriminator capacity to within 0.5% of the original.
  • E MNIST-1K DISCRETE MODE TEST WITH CRIPPLED DISCRIMINATOR: Progression provides little benefit on the tiny MNIST images but does not reduce performance.The normalization changes and minibatch stddev layer, rather than progression, account for the reported improvements in this test.

F ADDITIONAL CELEBA-HQ RESULTS

Additional CelebA-HQ results include nearest-neighbor analyses and generated examples, evaluated with mirror augmentation and both SWD and FID metrics.

  • F ADDITIONAL CELEBA-HQ RESULTS: The CelebA-HQ evaluation reports nearest neighbors, additional generated examples, mirror augmentation, and SWD and FID computed from 50K images.These results are presented in Figures 10 and 11; mirror augmentation was enabled for all CelebA and CelebA-HQ tests.

G LSUN RESULTS · H ADDITIONAL IMAGES FOR TABLE 1

The LSUN experiments generated representative images across all 30 categories using category-specific networks and largely constrained training data. Additional figures provide CELEBA-HQ results, LSUN examples with SWD and FID reporting, and larger image collections for non-converged Table 1 setups.

  • G LSUN RESULTS: 30 LSUN categories were each modeled with a separate network using identical parameters.Each category used 100k training images, except BEDROOM and DOG, which used all available data.
  • G LSUN RESULTS: 100k images was considered very limited for most LSUN categories, motivating mirror augmentation except for BEDROOM and DOG.The augmentation was enabled specifically for these tests and excluded the two categories trained on all available data.
  • H ADDITIONAL IMAGES FOR TABLE 1: Figure 18 presents larger collections of generated images for the non-converged setups in Table 1.Training time was intentionally limited so differences between methods would be more visible.
  • H ADDITIONAL IMAGES FOR TABLE 1: CELEBA-HQ generated images were compared with nearest neighbors from the training data using feature-space distances from five VGG layers.Only the highlighted bottom-right crop was used to exclude background and focus on facial-feature matching.
  • H ADDITIONAL IMAGES FOR TABLE 1: Additional 1024×1024 CELEBA-HQ images achieved an average SWD ×103 of 5.44 across levels 1024 through 16.The reported level values were 7.48, 7.24, 6.08, 3.51, 3.55, 3.02, and 7.22.
  • H ADDITIONAL IMAGES FOR TABLE 1: The CELEBA-HQ results had an FID of 7.30 computed from 50K images.The figure also refers to latent-space interpolations shown in the video.
  • H ADDITIONAL IMAGES FOR TABLE 1: Figures 12–17 show 256 × 256 generated images from LSUN categories and report SWD ×103 at levels 256, 128, 64, 32, and 16 alongside FID from 50K images.The average SWD is bolded in each figure.
Loading 1710.10196v3…