PaperlayerView exact revision on arXiv

Source-linked AI summary

The relativistic discriminator: a key element missing from standard GAN

Alexia Jolicoeur-Martineau

arXiv:1807.00734v3cs.LGcs.AIcs.CRstat.ML

TL;DR

Standard GANs can ignore the fact that each mini-batch contains fake samples, producing unstable training dynamics. The paper introduces relativistic discriminators and reports improved stability and data quality, while noting limited experimental coverage.

  • Problem

    SGAN increases fake-data realism without simultaneously decreasing real-data realism, despite mini-batches containing half fake samples.

  • Method

    The paper generalizes relativistic discrimination to RGANs and average comparisons to RaGANs, with the discriminator comparing real and fake realism.

  • Results

    Relativistic GANs generally produce more stable training and higher-quality data than non-relativistic counterparts, including strong results with gradient penalties and high-resolution images.

  • Takeaways & Limitations

    Relativistic discrimination can improve GAN data quality and stability without additional computational cost, and can complement spectral normalization or gradient penalties.

  • Takeaways & Limitations

    The experiments covered selected loss functions with one random seed, so broader datasets and hyperparameters require further study.

Abstract

from arXiv · show

In standard generative adversarial network (SGAN), the discriminator estimates the probability that the input data is real. The generator is trained to increase the probability that fake data is real. We argue that it should also simultaneously decrease the probability that real data is real because 1) this would account for a priori knowledge that half of the data in the mini-batch is fake, 2) this would be observed with divergence minimization, and 3) in optimal settings, SGAN would be equivalent to integral probability metric (IPM) GANs. We show that this property can be induced by using a relativistic discriminator which estimate the probability that the given real data is more realistic than a randomly sampled fake data. We also present a variant in which the discriminator estimate the probability that the given real data is more realistic than fake data, on average. We generalize both approaches to non-standard GAN loss functions and we refer to them respectively as Relativistic GANs (RGANs) and Relativistic average GANs (RaGANs). We show that IPM-based GANs are a subset of RGANs which use the identity function. Empirically, we observe that 1) RGANs and RaGANs are significantly more stable and generate higher quality data samples than their non-relativistic counterparts, 2) Standard RaGAN with gradient penalty generate data of better quality than WGAN-GP while only requiring a single discriminator update per generator update (reducing the time taken for reaching the state-of-the-art by 400%), and 3) RaGANs are able to generate plausible high resolutions images (256x256) from a very small sample (N=2011), while GAN and LSGAN cannot; these images are of significantly better quality than the ones generated by WGAN-GP and SGAN with spectral normalization.

1 Introduction

GANs train a discriminator to distinguish real from generated data and a generator to fool it. Standard GANs can be unstable, while alternatives improve stability inconsistently and may require more computation.

  • GANs formulate generation as a competition between a discriminator that separates real and fake data and a generator that produces deceptive samples.
  • In SGAN, the discriminator predicts the probability that an input is real, and its optimal loss approximately equals the Jensen–Shannon divergence.
  • SGAN’s saturating generator loss can have zero gradient under perfect classification, whereas its non-saturating loss has a non-zero but volatile gradient.
  • Existing GAN variants have improved stability or sample quality only partially and do not consistently outperform SGAN in large-scale comparisons.

2 Background

GAN objectives can be expressed through scalar transformations of discriminator outputs and generator losses. IPM-based GANs instead use real-valued critics constrained to function classes that prevent unbounded growth.

  • A general GAN formulation uses scalar-to-scalar functions applied to discriminator outputs over real and generated data distributions.P denotes real data, Pz the latent distribution, D(x) the discriminator, and G(z) the generator.
  • Saturating losses alternate maximizing and minimizing a shared objective, while non-saturating losses swap real and fake terms for generator training.
  • SGAN uses cross-entropy with D(x)=sigmoid(C(x)), interpreting the discriminator output as the probability that input data is real.
  • IPMs measure discrepancies over a class F of real-valued functions, and IPM-based GANs use an untransformed critic constrained to prevent loss divergence.

3 Missing property of SGAN

The paper identifies a missing SGAN property: increasing fake realism should reduce real-data realism. Relativistic comparisons are motivated by mini-batch composition, divergence minimization, and links between SGAN and IPM dynamics.

  • 3. Missing property of SGAN: SGAN should decrease D(xr) as D(xf) increases, because discriminator predictions should depend on both real and fake samples.
  • 3.2 Prior knowledge argument: When the generator fools SGAN, classifying both real and fake samples as real conflicts with the prior knowledge that half the mini-batch is fake.
  • 3.3 Divergence argument: Direct JSD minimization would move discriminator outputs for real and fake data toward .50, whereas SGAN increases fake realism without decreasing real realism.
  • 3.4 Gradient argument: In IPMs, real and fake data both contribute to discriminator gradients, but SGAN’s optimal discriminator gradient can ignore real data.
  • 3.4 Gradient argument: Under strong assumptions, SGAN matches IPM dynamics if its discriminator also decreases D(xr) during discriminator updates.

4 Method

The method makes GAN discrimination relativistic by comparing real and fake critic outputs, either pairwise or against the opposing class average. These formulations generalize across GAN losses while retaining linear batch complexity for the average variant.

  • Relativistic discriminator: A relativistic discriminator compares real and fake critic outputs instead of estimating whether each input is independently real.For paired samples, it estimates whether given real data is more realistic than randomly sampled fake data, with a reverse comparison for fake data.
  • Relativistic GANs: RGANs generalize relativistic discrimination to any activation function applied to the critic difference C(xr) − C(xf).The framework treats almost any GAN as capable of using a relativistic discriminator.
  • Relativistic GANs: IPM-based GANs are a special case of RGANs using identity mappings, while generator terms that depend on fake data generally acquire non-zero gradients.In most RGANs, the generator must specify the term g1 because fake data influences it; IPM-based GANs are an exception because they use the identity function.
  • Relativistic average GANs: The average relativistic approach compares an input critic to the average critic of samples from the opposing type.Enumerating every real–fake pairing would increase complexity from O(m) to O(m^2), so that approach is not used.
  • Relativistic average GANs: RaD estimates whether real data is more realistic than fake data on average and has O(m) complexity.This gives the discriminator a more global interpretation while preserving linear complexity in batch size.
  • Relativistic average GANs: The general average formulation is called Relativistic average GAN (RaGAN) and can be combined with any GAN loss function.The paper provides a training algorithm for non-saturating RaGANs.

5 Experiments

Experiments compare relativistic and non-relativistic GAN objectives across stable and unstable CIFAR-10 settings and difficult high-resolution CAT generation. Relativistic objectives generally improve stability and sample quality, with strong results using gradient penalty and at 256×256 resolution.

  • Experimental setup: Experiments compare SGAN, LSGAN, WGAN-GP, HingeGAN, RSGAN, RaSGAN, RaLSGAN, and RaHingeGAN, including relativistic models with WGAN-GP gradient penalty.The comparisons use standard CNN or DCGAN architectures, with stable and unstable training setups.
  • 5.1 Easy/stable experiments: In stable CIFAR-10 experiments, RSGAN and RaSGAN generally outperform SGAN, while RaHingeGAN outperforms HingeGAN and RaLSGAN performs nearly on par with LSGAN.RSGAN-GP outperforms the other tested loss functions with one discriminator update per generator update, reaching FID 25.60 versus 25.5 reported for spectral normalization.
  • 5.3 CIFAR-10: In unstable CIFAR-10 setups, RaLSGAN outperforms LSGAN in all four tested conditions, whereas relativistic SGAN and RaSGAN improve on SGAN in two conditions with small differences.The four conditions vary learning rate, Adam momentum, batch normalization, and activation functions.
  • 5.4 CAT: On CAT, RGANs and RaGANs show lower minimum, maximum, mean, and standard-deviation FID than their non-relativistic counterparts.At 64×64, SGAN and LSGAN exhibit large FID swings, while RaGANs are more stable and achieve lower minimum FID.
  • 5.4 CAT: SGAN cannot converge at 128×128 or larger images, and LSGAN cannot converge at 256×256, while RaGANs generate plausible images with low FID at all tested resolutions.SpectralSGAN and WGAN-GP generate 256×256 images but with poor quality; gradient penalty further reduces FID standard deviation without improving data quality.

6 Conclusion and future work

The paper proposes relativistic discriminators and a generally more stable RaD variant to improve GAN data quality and stability. It reports encouraging results while identifying limited experiments and unresolved mathematical questions as directions for future work.

  • The paper proposes relativistic discriminators as an improvement to standard GANs and generalizes the approach to GAN losses through a more stable RaD variant.The authors also suggest combining relativistic discriminators with tools such as spectral normalization and gradient penalty.
  • The reported results suggest improved data quality and stability without additional computational cost.
  • Future work must clarify the mathematical implications of relativism and evaluate more loss functions across datasets and hyperparameters.Experiments were limited by computational constraints and used only one seed.

A Gradient step

The gradient step specializes the relativistic formulation according to the relationship between the generator and discriminator loss functions. Two cases are given for saturating and non-saturating GANs.

  • When g1(y) = −f1(y) and g2(y) = −f2(y), the formulation corresponds to the saturating GAN.
  • When g1(y) = f2(y) and g2(y) = f1(y), the formulation corresponds to the non-saturating GAN.

C Loss functions used in experiments

The experiments use discriminator losses based on sigmoid cross-entropy and hinge objectives, together with interpolated samples for the distribution P̂x. A relativistic discriminator output compares fake scores with the average real score.

  • The sigmoid discriminator loss uses expected log probabilities for real and fake samples.It is written as D = −Exr∼P [log (sigmoid(C(xr)))] −Exf ∼Q [log (1 −sigmoid(C(xf)))].
  • The hinge discriminator loss combines max-margin penalties for real and fake samples.It is written as D = Exr∼P [max(0, 1 −C(xr))] + Exf ∼Q [max(0, 1 + C(xf))].
  • P̂x is the distribution of interpolated samples formed from real and fake samples using a uniformly distributed interpolation coefficient.The definition is repeated for the experimental loss-function setup.
  • The relativistic discriminator output applies a sigmoid to the fake score minus the expected real score.The expression is ˜D(xf) = sigmoid (C(xf) −Exr∼PC(xr)).

D Architectures

The architectures use convolutional and transposed-convolutional layers with progressively varying channel counts, including reshaping between flattened and spatial representations. The listed discriminator configurations include standard and PACGAN2-based components.

  • The generator-side configurations begin with reshaping 512*4*4 into 512 x 4 x 4 and use transposed convolutions to produce three-channel outputs.Listed transposed-convolution layers include 512->256, 256->128, 128->64, and 64->3 mappings.
  • The discriminator-side configurations use convolutional layers with downsampling strides of 2 and channel mappings such as 3->64, 64->128, 128->256, and 256->512.Several listed layers use 4x4 kernels, stride 2, padding 1, and no bias.
  • Additional configurations expand channel widths to 1024 before reducing them through transposed convolutions to three-channel outputs.The listed mappings include 512->1024, 1024->512, 512->256, 256->128, 128->64, and 64->3.
  • The architecture list includes a discriminator identified as PACGAN2.

E Samples

The sample figures show cat images from relativistic and non-relativistic models across resolutions, with selected outputs tied to FID measurements. The selection procedure favors the lowest recorded FID at regular intervals and, at higher resolutions, includes some especially good-looking examples.

  • Images were selected from the lowest FID recorded every 10k generator iterations, with higher-resolution panels showing some of the nicer-looking cats because of space constraints.The authors note that other cats in the minibatches can look worse.
  • RaLSGAN produces 64x64 cats with FID = 11.97.
  • RaLSGAN produces 128x128 cats with FID = 15.85.
  • At 256x256 resolution, the figures show GAN and LSGAN samples after 5k iterations.
  • The 256x256 relativistic-model figures report FID = 32.11 for RaSGAN and FID = 35.21 for RaLSGAN.
  • The 256x256 comparison figures report FID = 54.73 for SpectralSGAN and FID > 100 for WGAN-GP.
Loading 1807.00734v3…