PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral Normalization for Generative Adversarial Networks

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, Yuichi Yoshida

arXiv:1802.05957v1cs.LGcs.CVstat.ML

TL;DR

GAN training is unstable, particularly because discriminator behavior can impede generator learning. This paper introduces spectral normalization to stabilize the discriminator, and finds that SN-GANs generate more diverse images with better or comparable inception scores than prior methods.

  • Problem

    GAN training remains unstable because inaccurate discriminator density-ratio estimates can prevent generators from learning target distributions.

  • Method

    The paper introduces spectral normalization, a weight-normalization method that regularizes discriminator layers globally.

  • Results

    SN-GANs generated more diverse images and achieved better or comparable inception scores than previous methods across the reported image-generation experiments.

  • Takeaways & Limitations

    Spectral normalization provides a computationally light, simple discriminator-stabilization technique that performs robustly across reported training settings.

  • Takeaways & Limitations

    For discretely conducted convolutions, the spectral norm depends on stride and padding, differing by a predefined factor K.

Abstract

from arXiv · show

One of the challenges in the study of generative adversarial networks is the instability of its training. In this paper, we propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator. Our new normalization technique is computationally light and easy to incorporate into existing implementations. We tested the efficacy of spectral normalization on CIFAR10, STL-10, and ILSVRC2012 dataset, and we experimentally confirmed that spectrally normalized GANs (SN-GANs) is capable of generating images of better or equal quality relative to the previous training stabilization techniques.

1 INTRODUCTION

The introduction identifies discriminator training control as a persistent source of GAN instability and presents spectral normalization as a simple, computationally light method to stabilize discriminator networks.

  • GAN framework: GANs generate a model distribution intended to mimic a target distribution using a generator and discriminator.The framework has been applied to numerous tasks and datasets.
  • Training challenge: Discriminator performance is difficult to control in high-dimensional spaces, causing inaccurate and unstable density-ratio estimation and failures to learn multimodal target structure.When model and target supports are disjoint, a discriminator can perfectly distinguish them.
  • Proposed method: The paper proposes spectral normalization, a novel weight-normalization method intended to stabilize discriminator training.The introduction frames spectral normalization as the paper’s central contribution.
  • Method properties: Spectral normalization requires tuning only the Lipschitz constant, does not require intensive tuning for satisfactory performance, and adds little computational cost through simple implementation.These properties are presented as favorable characteristics of the proposed normalization method.
  • Evaluation scope: The method functions well even without tuning the Lipschitz constant and is analyzed against weight normalization, weight clipping, and gradient penalty.The study also examines spectral normalization without complementary regularization techniques.

2 METHOD

The method stabilizes GAN discriminator training by constraining its discriminator function to be K-Lipschitz through spectral normalization of each weight matrix. It sets each matrix’s spectral norm to a designated value and estimates that norm efficiently with power iteration.

  • Lipschitz constraint: The discriminator is selected from K-Lipschitz continuous functions, with the Lipschitz norm defined using the ℓ2 distance between arbitrary inputs.This regularity condition is motivated by concerns that the discriminator’s derivative can be unbounded or incomputable.
  • Motivation: Spectral normalization addresses the limitations of input-based regularization by normalizing discriminator weight matrices across the network, including regions outside generator and data supports.The method is introduced to avoid relying only on regularization defined at sampled input examples.
  • Lipschitz constraint: For a linear layer g(h) = Wh, the Lipschitz norm equals the spectral norm σ(W), which is the largest singular value of W.With 1-Lipschitz activations and composition inequalities, layerwise spectral norms provide a bound on the network’s Lipschitz norm.
  • Normalization mechanism: The method normalizes each weight matrix so that its spectral norm satisfies σ(W) = 1, unlike spectral-norm regularization, which only penalizes the norm in the objective.Spectral normalization sets the spectral norm to a designated value rather than adding an explicit penalty term.
  • Implementation: Power iteration estimates each matrix’s spectral norm instead of recomputing it by singular value decomposition, adding very little computational cost.The approach is intended to keep spectral normalization computationally light during training.
  • Optimization effect: The normalization gradient adds an adaptive term that penalizes dominant singular components and prevents the weight column space from concentrating in one direction.The coefficient λ is positive when δ and W̄SNh point in similar directions.

3 SPECTRAL NORMALIZATION VS OTHER REGULARIZATION TECHNIQUES

The section contrasts spectral normalization with weight normalization and weight clipping, arguing that those alternatives impose overly strong constraints. Spectral normalization instead controls only the maximum singular value, preserving feature freedom while enforcing a local 1-Lipschitz constraint.

  • Weight and Frobenius normalization: Weight normalization fixes the sum of squared singular values to d_o, making it equivalent up to scale to Frobenius normalization.It normalizes the ℓ2 norm of each weight-matrix row.
  • Weight and Frobenius normalization: Weight normalization can impose a stronger-than-intended constraint by maximizing ∥W̄_WN h∥2 at √d_o when only the first singular value is nonzero.This concentrates the matrix’s effective behavior in select features.
  • Weight clipping: Weight clipping suffers from the same pitfall of matching the target distribution only at select few features.The passage presents weight clipping as sharing the limitation identified for weight normalization.
  • Spectral normalization: Spectral normalization scales only the maximum singular value, so it is independent of rank and allows the parameter matrix to use as many features as possible under a local 1-Lipschitz constraint.The Lipschitz constant of a linear operator is determined only by its maximum singular value.

4 EXPERIMENTS

Experiments across CIFAR-10, STL-10, and ImageNet evaluated SNGAN against alternative normalization and regularization methods. Spectral normalization was generally robust and competitive, while retaining favorable training behavior and image quality.

  • Performance across settings: SNGAN remained relatively robust under aggressive learning rates and momentum parameters, whereas WGAN-GP failed to train good GANs in those settings on CIFAR-10 and STL-10.Orthonormal regularization performed poorly for one STL-10 setting but slightly exceeded SNGAN under its optimal setting.
  • Performance across settings: On optimal CIFAR-10 and STL-10 settings, SNGAN outperformed almost all contemporaries and improved further with hinge loss, but fell behind orthonormal regularization on STL-10 at equal iteration counts.The comparison used inception scores summarized in Table 2.
  • Generated image quality: Spectral normalization produced clearer and more diverse images than weight normalization, while WGAN-GP failed to train good GANs with high learning rates.The visual comparison was shown in Figure 6.
  • Training stability: Without normalization or with layer normalization, GANs collapsed early, whereas SNGAN generated images and continued improving after orthonormal normalization plateaued around 20K iterations.Orthonormal normalization also produced images, but its inception score plateaued while SNGAN continued improving.

5 CONCLUSION · A THE ALGORITHM OF SPECTRAL NORMALIZATION

The paper proposes spectral normalization to stabilize GAN training and reports more diverse generated examples with better or comparable inception scores. Its algorithm estimates spectral norms through power iteration, recycles singular-vector estimates, normalizes weights, and updates them with SGD.

  • 5 CONCLUSION: Spectral normalization stabilizes GAN training, producing more diverse examples and better or comparable inception scores than prior approaches.It applies global discriminator regularization, unlike WGAN-GP’s local regularization, and may be combined with other methods.
  • A THE ALGORITHM OF SPECTRAL NORMALIZATION: The power method estimates dominant left and right singular vectors from randomly initialized vectors under stated nonmultiplicity and nonorthogonality conditions.The initial vector is randomly initialized for each weight.
  • A THE ALGORITHM OF SPECTRAL NORMALIZATION: The estimated singular-vector pair approximates the spectral norm of each weight matrix.This approximation supplies the normalization quantity used by the algorithm.
  • A THE ALGORITHM OF SPECTRAL NORMALIZATION: Recycling each step’s estimated vector as the next initial vector makes one power-iteration round sufficient for satisfactory experimental performance.This works because SGD changes weights, and therefore their largest singular values, only slightly at each update.
  • A THE ALGORITHM OF SPECTRAL NORMALIZATION: Algorithm 1 initializes one random vector ˜u_l ∈ R^d_l for every layer l = 1, . . . , L.The vectors are sampled from an isotropic distribution.
  • A THE ALGORITHM OF SPECTRAL NORMALIZATION: For each layer and update, the algorithm applies power iteration to the unnormalized weight W_l.This is the first per-update computational step specified in Algorithm 1.
  • A THE ALGORITHM OF SPECTRAL NORMALIZATION: The algorithm calculates the spectrally normalized weight ̄W_SN using the spectral norm.This follows power iteration for each layer’s unnormalized weight.
  • A THE ALGORITHM OF SPECTRAL NORMALIZATION: Finally, it updates each W_l with SGD on mini-batch dataset D_M using learning rate α.The update is performed after calculating the spectrally normalized weight.

B EXPERIMENTAL SETTINGS … B.3 IMAGE GENERATION ON IMAGENET

The experimental settings define Inception score and FID evaluation procedures, compare ResNet and standard CNN architectures, and specify ImageNet preprocessing and conditional normalization. SN-GAN experiments use Adam optimization, with settings varying by dataset and architecture.

  • B.1 PERFORMANCE MEASURES: Inception score evaluates natural-image generation using randomly generated 5000 examples from each trained generator.The score was reported as strongly correlated with subjective human judgments of image quality.
  • B.1 PERFORMANCE MEASURES: FID measures the distance between true and generated image distributions in the final-layer feature space of an Inception model.The distance was computed empirically using 10000 true-distribution samples and 5000 generated samples.
  • B.2 IMAGE GENERATION ON CIFAR-10 AND STL-10: The comparative study tested the recent ResNet architecture of Gulrajani et al. (2017) alongside a standard CNN.Both architectures were included in the additional comparative experiments.
  • B.2 IMAGE GENERATION ON CIFAR-10 AND STL-10: The comparative experiments used Adam with α = 0.0002, β1 = 0, β2 = 0.9, and n_dis = 5.These hyperparameters matched those used by Gulrajani et al. (2017).
  • B.2 IMAGE GENERATION ON CIFAR-10 AND STL-10: For SN-GANs, the generator feature-map dimension was doubled because this modification achieved better results.The change was made in the comparative experiments.
  • B.3 IMAGE GENERATION ON IMAGENET: ImageNet experiments resized images to 128 × 128 pixels and used conditional batch normalization conditioned on labels y ∈ {1, . . . , 1000}.The conditional batch-normalization layer replaced standard batch normalization in conditional GAN generators.
  • B.3 IMAGE GENERATION ON IMAGENET: ImageNet optimization used Adam with the same hyperparameters used for ResNet on CIFAR-10.The supplied passage does not provide the repeated hyperparameter values.

B.4 NETWORK ARCHITECTURES

The experiments use standard CNN models for CIFAR-10 and STL-10, alongside ResNet architectures for CIFAR10, STL-10, and ImageNet. The architectures specify discriminator, generator, and conditional-model modifications, including leaky-ReLU slopes and normalization-layer choices.

  • Standard CNN architectures: Standard CNN models are used for CIFAR-10 and STL-10 image generation, with all leaky-ReLU slopes set to 0.1.These models are described in Table 3.
  • ResNet architectures: The CIFAR10 ResNet architectures follow designs similar to those used by Gulrajani et al. (2017).Table 4 specifies the CIFAR10 ResNet models.
  • ResBlock architecture: In the discriminator, batch-normalization layers are removed from the ResBlock architecture.This modification applies to the discriminator’s ResBlock.
  • Dataset-specific ResNet architectures: The STL-10 and ImageNet experiments use dedicated ResNet architectures; conditional GAN generators replace batch normalization with conditional batch normalization, while projection discriminators follow Miyato and Koyama (2018).The ImageNet description also notes that the projection discriminator uses the same architecture as in Miyato & Koyama (2018).

C APPENDIX RESULTS · C.1 ACCURACY OF SPECTRAL NORMALIZATION

Appendix results show that spectral normalization keeps discriminator-layer spectral norms close to 1 during training. On CIFAR10, deviations are generally small, with only the sixth and seventh convolutional layers showing larger early variation before stabilizing.

  • C.1 ACCURACY OF SPECTRAL NORMALIZATION: Under optimizer setting C, the discriminator layers’ spectral norms do not deviate by more than 0.05 for most of training.This describes the predominant behavior shown in Figure 9.
  • C.1 ACCURACY OF SPECTRAL NORMALIZATION: The sixth and seventh convolutional layers, which have the largest rank, deviate by more than 0.1 at the beginning of training.These are identified as exceptions to the otherwise small deviations.
  • C.1 ACCURACY OF SPECTRAL NORMALIZATION: After several iterations, the spectral norm of the exceptional layers also stabilizes around 1.The passage specifically notes that these layers eventually settle near 1.
  • C APPENDIX RESULTS: Figure 9 tracks the spectral norms of all seven convolutional layers in the standard CNN during CIFAR10 training.The optimizer setting is C in Table 1 throughout training.
  • C.1 ACCURACY OF SPECTRAL NORMALIZATION: The reported spectral norms are measured layer by layer across the discriminator’s seven convolutional layers.Figure 9 presents the complete set of layer-wise norms rather than a single aggregate value.
  • C.1 ACCURACY OF SPECTRAL NORMALIZATION: The accuracy analysis concerns the standard CNN trained on CIFAR10, with spectral norms monitored over the course of training.This combines the figure’s dataset and model description with the accompanying training analysis.

C.2 TRAINING TIME … D SPECTRAL NORMALIZATION VS OTHER REGULARIZATION TECHNIQUES

The merged sections report computational timing, the contrasting sensitivity of weight and spectral normalization to ndis, generated-image comparisons, ImageNet learning curves, and theoretical concerns about alternative discriminator regularizers.

  • C.2 TRAINING TIME: Figure 10 reports computational time for 100 updates using ndis = 5.The passage provides the timing setup but no numerical runtime value.
  • C.3 THE EFFECT OF ndis ON SPECTRAL NORMALIZATION AND WEIGHT NORMALIZATION: Weight normalization deteriorates as ndis increases, whereas spectral normalization does not suffer this unintended effect.The comparison uses setting D while varying only ndis; larger ndis corresponds to more accurate minimax computation.
  • C.3 THE EFFECT OF ndis ON SPECTRAL NORMALIZATION AND WEIGHT NORMALIZATION: Figure 11 presents the effect of ndis on spectral and weight normalization, with shaded regions showing variance across different seeds.All results follow setting D except for the value of ndis.
  • C.4 GENERATED IMAGES ON CIFAR10 WITH GAN-GP, LAYER NORMALIZATION AND BATCH NORMALIZATION: Figure 12 shows generated images from GAN-GP, Layer Norm, and Batch Norm on CIFAR-10.
  • C.5 IMAGE GENERATION ON IMAGENET: Figure 13 shows ImageNet Inception-score learning curves for SN-GANs and GANs with orthonormal regularization in unconditional and conditional settings.The conditional GANs use a projection discriminator.
  • D SPECTRAL NORMALIZATION VS OTHER REGULARIZATION TECHNIQUES: The comparative study argues that weight normalization and weight clipping impose unnecessary restrictions on discriminator search spaces by implicitly favoring low-rank weight matrices.

D.1 WEIGHT NORMALIZATION AND FROBENIUS NORMALIZATION … E REPARAMETRIZATION MOTIVATED BY THE SPECTRAL NORMALIZATION

The paper contrasts spectral normalization with alternative discriminator regularization methods, arguing that it preserves feature capacity while enforcing Lipschitz control and remains computationally practical. It also motivates a learnable-scale reparametrization that trades strict layerwise 1-Lipschitzness for greater model freedom.

  • D.1 WEIGHT NORMALIZATION AND FROBENIUS NORMALIZATION: Weight and Frobenius normalization favor skewed singular-value distributions, which can reduce discriminator representations to approximately low-dimensional feature spaces.Under weight normalization, maximizing output norm can yield a rank-one matrix; Frobenius normalization admits the same argument.
  • D.1 WEIGHT NORMALIZATION AND FROBENIUS NORMALIZATION: Spectral normalization avoids this rank-capacity conflict because the Lipschitz constant depends only on the maximum singular value, not matrix rank.It therefore allows the discriminator to use more feature dimensions while satisfying a local 1-Lipschitz constraint.
  • D.2 WEIGHT CLIPPING: Weight clipping truncates every weight-matrix element to an absolute value bounded by a prescribed constant c ∈ R+, but suffers the same problem as weight and Frobenius normalization.The cited passage characterizes clipping as another regularization technique with analogous capacity limitations.
  • D.3 SINGULAR VALUE CLIPPING AND SINGULAR VALUE CONSTRAINT: Singular-value clipping directly controls the spectral norm but is computationally heavy because it requires singular value decomposition to compute all singular values.A constrained singular-value parametrization is also difficult to train while remaining faithful to its constraint, whereas spectral normalization updates weights at relatively low computational cost.
  • D.4 WGAN WITH GRADIENT PENALTY (WGAN-GP): WGAN-GP imposes a local 1-Lipschitz constraint directly on the discriminator through gradient penalties, but regularizes only sampled points and can become unstable as the generative support changes.The paper also reports that high learning rates can destabilize WGAN-GP, whereas spectral normalization remains stable with aggressive learning rates.
  • D.4 WGAN WITH GRADIENT PENALTY (WGAN-GP): WGAN-GP requires more computation than spectral normalization with single-step power iteration because evaluating ∥∇xD∥2 requires a full forward and backward propagation.The methods are compared for the same number of updates in Figure 10.
  • D.4 WGAN WITH GRADIENT PENALTY (WGAN-GP): The paper proposes combining gradient penalty with spectral-normalization reparametrization, which improves generated-example quality over the WGAN-GP-only baseline.The two methods regularize discriminators by different mechanisms, motivating their combination.
  • E REPARAMETRIZATION MOTIVATED BY THE SPECTRAL NORMALIZATION: A learnable scalar γ in the spectral-normalization reparametrization gives the model more freedom and avoids degeneracy, but relinquishes the layerwise 1-Lipschitz constraint.The resulting Lipschitz condition must be controlled by another method, such as gradient penalty.

E.1 EXPERIMENTS: COMPARISON OF REPARAMETRIZATION WITH DIFFERENT NORMALIZATION METHODS

The addendum compares weight and spectral normalization reparametrizations for training WGAN-GP discriminators using CNN and ResNet-based CNN architectures. Spectral normalization improves inception scores and reduces discriminator overfitting relative to the alternatives.

  • Experimental setup: The experiments compare weight-normalization and spectral-normalization reparametrizations for WGAN-GP discriminator training.The regular CNN matches the architecture used previously, while the ResNet-based CNN follows Gulrajani et al. (2017).
  • Results: Spectral normalization significantly improves inception score over the baseline on the regular CNN and slightly improves it on the ResNet-based CNN.Tables 7 and 8 summarize these results.
  • Learning dynamics: Spectral normalization produces beneficial discriminator learning curves and less training-set overfitting than no reparametrization or weight normalization.Figure 15 reports critic losses and inception scores for the different reparametrization methods.

F THE GRADIENT OF GENERAL NORMALIZATION METHOD

This section formulates the gradient of a general weight-normalization method with respect to the unnormalized weight. It defines normalized weights using a scalar coefficient such as the spectral or Frobenius norm and examines the resulting gradient direction.

  • General normalization: The normalized weight is defined as W̄ := W/N(W), where N(W) is a scalar normalized coefficient such as the spectral or Frobenius norm.This establishes a general reparameterization framework for weight normalization.
  • Empirical comparison: Frobenius normalization produced unavailable inception-score and FID results because training collapsed at an early stage on CIFAR10 without label supervision.The table reports N/A for both metrics under this failure condition.
  • Gradient formulation: The loss derivative is formulated with respect to the unnormalized weight W after transforming a hidden node h by the normalized weight W̄.The formulation uses empirical expectation over the mini-batch.
  • Gradient direction: For Frobenius or spectral normalization, the gradient has a specified geometric point derived from the normalization function N(W).The section explicitly identifies N(W) as either the Frobenius norm ∥W∥F or spectral norm ∥W∥2 in these cases.
Loading 1802.05957v1…