PaperlayerView exact revision on arXiv

Source-linked AI summary

cGANs with Projection Discriminator

Takeru Miyato, Masanori Koyama

arXiv:1802.05637v2cs.LGcs.CVstat.ML

TL;DR

cGAN discriminators commonly concatenate conditional information without explicitly reflecting the assumed conditional probabilistic model. This paper proposes a projection discriminator based on that structure and reports improved conditional image generation and super-resolution results. The authors also identify theoretical uncertainty about how the imposed regularity relates to the generator's implicitly derived conditional distribution.

  • Problem

    Most cGAN discriminators incorporate conditional information by concatenation, while the paper seeks a design that respects the assumed conditional probabilistic model.

  • Method

    The paper models the discriminator using a likelihood-ratio-motivated projection form that takes an inner product between an embedded condition vector and an input feature vector.

  • Results

    The projection discriminator significantly improved conditional image generation on the 1000-class ILSVRC2012 dataset and produced highly discriminative super-resolution images with a single discriminator-generator pair.

  • Takeaways & Limitations

    The results suggest that discriminator form and the design of the distributional metric are important for cGAN performance across conditional image generation and super-resolution.

  • Takeaways & Limitations

    The relationship between the imposed regularity and the generator's implicitly derived conditional distribution remains theoretically unclear.

Abstract

from arXiv · show

We propose a novel, projection based way to incorporate the conditional information into the discriminator of GANs that respects the role of the conditional information in the underlining probabilistic model. This approach is in contrast with most frameworks of conditional GANs used in application today, which use the conditional information by concatenating the (embedded) conditional vector to the feature vectors. With this modification, we were able to significantly improve the quality of the class conditional image generation on ILSVRC2012 (ImageNet) 1000-class image dataset from the current state-of-the-art result, and we achieved this with a single pair of a discriminator and a generator. We were also able to extend the application to super-resolution and succeeded in producing highly discriminative super-resolution images. This new structure also enabled high quality category transformation based on parametric functional transformation of conditional batch normalization layers in the generator.

1 INTRODUCTION

Conditional GANs incorporate conditional information into both generator and discriminator, but common discriminator designs concatenate the condition with inputs or features. The paper proposes a projection-based discriminator motivated by conditional probabilistic structure and reports improved generation quality on ImageNet and super-resolution.

  • GAN and cGAN background: GANs train a generator against a discriminator that evaluates divergence between generated and target distributions.The discriminator is trained alternately with the generator and becomes an increasingly precise critic.
  • GAN and cGAN background: cGANs use conditional information for class-conditional generation, text-to-image synthesis, and image-to-image translation.Their discriminator evaluates pairs of samples x and intended conditions y.
  • Motivation: Most cGAN discriminators incorporate the condition by concatenating an embedded y with the input or a hidden feature vector.The paper contrasts this practice with a structure that reflects the assumed conditional probabilistic model.
  • Motivation: The proposed discriminator is motivated by assuming that the conditional variable given the input follows a discrete or unimodal continuous distribution.The paper identifies this assumption as common in class-conditional image generation and super-resolution.
  • Reported outcomes: An inner product between the embedded condition vector and feature vector yields significantly better ImageNet class-conditional generation and more discriminative super-resolution images.The ImageNet experiment uses the 1000-class ILSVRC2012 dataset and a single discriminator-generator pair.

2 THE ARCHITECTURE OF THE CGAN DISCRIMINATOR WITH A PROBABILISTIC MODEL ASSUMPTIONS

The paper formulates the cGAN discriminator through likelihood-ratio decomposition and models the conditional term with an embedding–feature inner product. This yields the projection discriminator as an alternative to concatenating conditional information.

  • Discriminator formulation: The discriminator is written as D(x, y; θ) := A(f(x, y; θ)), where f combines input x, condition y, parameters θ, and activation A.The formulation uses q for true distributions and p for the generator model.
  • Discriminator formulation: For discrete labels, y can be encoded as a one-hot vector.
  • Conventional alternative: Conventional cGAN designs concatenate y with x at the input layer or with features at a hidden layer.The paper proposes replacing this arbitrary incorporation with a probabilistically motivated alternative.
  • Probabilistic motivation: Assuming simple conditional distributions such as Gaussian or discrete log-linear forms makes it natural to model the likelihood-ratio terms parametrically.The decomposition uses functions f1 for r(y|x) and f2 for r(x).
  • Projection discriminator: The projection discriminator defines f(x, y; θ) as y^T V φ(x; θ_Φ) + ψ(φ(x; θ_Φ); θ_Ψ).V embeds y, φ produces a feature vector from x, and ψ maps the same feature vector to a scalar.

3 MOTIVATION BEHIND THE projection DISCRIMINATOR

The paper derives the projection discriminator from probabilistic assumptions about conditional distributions, introducing labels through an inner product with image features rather than concatenation. This form applies to categorical and unimodal continuous conditional variables, while its effect on generator training remains unclear.

  • Categorical conditional variables: For categorical y, a log-linear model for p(y|x) leads to the discriminator form used in the paper.The derivation assumes regularity conditions and expresses the conditional model using class-specific parameter vectors and image features.
  • Projection structure: The proposed discriminator introduces label information through an inner product between the embedded condition and the feature vector, rather than concatenation.This is the projection structure identified as the paper’s proposed form.
  • Continuous conditional variables: For unimodal continuous p(y|x), the same discriminator form can arise when the conditional distributions are modeled as Gaussians.The derivation uses conditional means parameterized by image features and analyzes the resulting log density ratio.
  • Derivation: Assuming shared precision parameters, the quadratic term can be ignored and the remaining parameter difference expressed through a matrix V, recovering the projection form.The stated condition is Λq = Λp := Λ.
  • Limitation: The regularization effect on generator training is unclear because the implicitly derived p(y|x) may take forms beyond the conditional models considered.The paper identifies this relationship between discriminator function space and generator training as requiring further theoretical work.

4 COMPARISON WITH OTHER METHODS

The paper argues that projection respects frequently occurring conditional probabilistic models, whereas concatenation is comparatively arbitrary. It also contrasts this approach with auxiliary-classifier methods, which may encourage classifier-easy samples and shift the generated conditional distribution.

  • Projection versus concatenation: The projection form is valid for frequently occurring conditional models, while concatenation may include candidate functions without a clear probabilistic basis.The paper expects projection to perform better when the presumed conditional model is not multimodal.
  • Auxiliary classifiers: Auxiliary-classifier methods incorporate labels by adding classifier likelihood terms to the adversarial objective or using a pretrained classifier in sampling.The examples discussed are AC-GANs and Plug and Play Generative Networks.
  • Auxiliary classifiers: Auxiliary classifiers may encourage images that are easy to classify, potentially deviating from the true q(x|y).The paper presents this as a possible consequence of using auxiliary classifiers.

5 EXPERIMENTS

Experiments compare projection against concatenation and AC-GANs for ImageNet class-conditional generation, then evaluate projection on super-resolution and category morphing. Projection achieves lower intra-class FIDs, clearer super-resolution outputs, and meaningful interpolations between substantially different classes, while auxiliary classifier fine-tuning improves appearance at the cost of diversity.

  • Class-conditional image generation: Projection is compared with hidden-layer concatenation and AC-GANs for class-conditional generation on 1,000 ImageNet classes using a single ResNet-based generator–discriminator pair.The experiments use 128×128 images; AC-GANs instead used a pair for each class set of size 10.
  • Class-conditional image generation: Projection outperforms concat throughout ImageNet training and produces lower intra-class FIDs than both compared methods.The reported comparison uses learning curves, inception score, and intra-class FIDs.
  • Class-conditional image generation: AC-GAN samples show mode collapse, whereas projection better matches the target distribution overall through improved intra-class FID behavior.The text notes that AC-GANs can produce recognizable images while losing diversity, and that concat sometimes wins only slightly when it beats projection.
  • Class-conditional image generation: Projection samples with low FID are generally high quality, but some mode collapse remains, especially when classes contain complex and diverse objects.Figure 7 contrasts the five lowest- and highest-FID classes and pairs generated images with original data.
  • Category morphing: Category morphing mixes conditional batch-normalization parameters for two classes and yields meaningful images even when the classes are significantly different.The interpolation uses the same latent vector z for the generated outputs.
  • Super-resolution: Projection super-resolution produces clearer, smoother, and more faithful images than bicubic and concat, with high inception accuracy and MS-SSIM; ensemble sampling improves inception accuracy further.The ensemble averages logits from multiple random z batches and uses 10 seeds, supporting distributional rather than deterministic super-resolution behavior.

6 CONCLUSION

The paper proposes a projection discriminator motivated by probabilistic modeling and reports improved performance across conditional image generation and super-resolution. It also identifies category morphing as an enabled application.

  • The proposed discriminator model is motivated by a commonly occurring family of probabilistic models and imposes a regularity condition on the generator and target distributions.
  • The modification significantly improves performance on conditional image generation and super-resolution tasks.
  • 32x32 to 128x128 super-resolution is evaluated among different methods.
  • The results suggest that discriminator form and distributional-metric design are important factors in trained-generator performance.

A RESULTS OF CLASS CONDITIONAL IMAGE GENERATION ON CIFAR-10 AND CIFAR-100

The experiments compare projection and concatenation discriminator architectures on CIFAR-10 and CIFAR-100, including optimizer hyper-parameter variation. The section reports inception-score comparisons across these models.

  • The CIFAR experiments reuse the discriminator and generator architecture from Miyato et al. (2018) for CIFAR-10.
  • The study varies Adam learning rate α and first-order momentum β1 while holding the other parameter constant.
  • Inception scores are reported for the proposed projection architecture and several concatenation architectures.
  • Performance is compared for class-conditional image generation on CIFAR-10 and CIFAR-100.
  • The ImageNet comparison table reports inception score and intra FIDs using a pretrained classification model.

B OBJECTIVE FUNCTION WITH AN AUXILIARY CLASSIFIER COST

The auxiliary-classifier experiment augments the generator objective with a pretrained classification loss. This improves inception score and visual appearance, but the generated images are easy to classify rather than fully faithful to the target distribution.

  • The generator loss is augmented with an auxiliary classifier loss following the Plug and Play Generative Networks approach.
  • The generator uses the original adversarial loss for 400K updates and the augmented loss for the final 50K updates.
  • The augmented method significantly outperforms PPGNs in inception score.
  • The auxiliary classifier loss appears effective for visual appearance but does not produce a faithful generative model in this experiment.

C MODEL ARCHITECTURES

The model architectures use ResBlocks throughout the experiments, with conditional batch normalization in the generator and specialized conditioning for super-resolution upsampling.

  • ResBlocks are used in all experiments.
  • Generator ResBlocks replace standard batch normalization with conditional batch normalization.
  • The super-resolution generator concatenates random vector z with the embedded low-resolution image vector y before the first convolution in its upsampling ResBlock.
  • Separate model architectures are provided for conditional image generation and super-resolution.

D RESULTS OF CATEGORY MORPHING

The section presents category-transformation results, including transformations from a Lhasa apso to different categories and morphing between categories.

  • Figure 16 shows transformations from a dog, specifically a Lhasa apso, to different categories.
  • Figure 17 shows morphing between different categories.

E MORE RESULTS WITH SUPER-RESOLUTION

The section presents super-resolution results that transform 32×32 images to 128×128 images.

  • Figure 18 presents super-resolution results from 32×32 to 128×128.
Loading 1802.05637v2…