PaperlayerView exact revision on arXiv

Source-linked AI summary

Demystifying MMD GANs

Mikołaj Bińkowski, Danica J. Sutherland, Michael Arbel, Arthur Gretton

arXiv:1801.01401v5stat.MLcs.LG

TL;DR

The paper addresses uncertainty about gradient bias, kernel choice, and practical training for MMD-based GAN critics. It analyzes MMD and related critics theoretically, adapts Wasserstein-inspired regularization, and evaluates smaller MMD critics and the Kernel Inception Distance. The results show unbiased fixed-critic gradient estimators, bias after learning critics from samples, matching WGAN-GP generator performance with smaller faster critics, and a Cramér-critic zero-loss failure case despite good empirical performance.

  • Problem

    The paper investigates gradient bias, kernel selection, and training and evaluation issues in GANs using MMD and related integral probability metrics.

  • Method

    It analyzes fixed versus learned-critic gradients, characterizes energy-distance kernels, applies Wasserstein-inspired critic regularization, and evaluates MMD GANs with Kernel Inception Distance.

  • Results

    MMD GANs match WGAN-GP generator performance with smaller discriminator networks and faster training, while learned critics produce biased generator gradients relative to optimal critics.

  • Takeaways & Limitations

    MMD GANs combine convolutional representations with kernel critics to obtain simpler training, while Kernel Inception Distance provides an alternative GAN convergence measure.

  • Takeaways & Limitations

    The Cramér critic can assign zero loss to distinct distributions, and finite-sample FID estimates can reverse the true ordering between distribution pairs.

Abstract

from arXiv · show

We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramer GAN critic. Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs. In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance. We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.

1 INTRODUCTION

GANs learn generators that imitate target data by training against a discriminator, but conventional objectives can provide unhelpful gradients when distributions lie on separate manifolds. This motivates IPM-based critics and the MMD GAN, which combines learned convolutional representations with a kernel critic to match WGAN-GP performance using smaller, faster models.

  • GANs train a generator to mimic a target distribution while a discriminator distinguishes generated from target samples.
  • When generated and reference distributions lie on nonintersecting or nearly nonintersecting manifolds, conventional GAN gradients can become uninformative and destabilize training.
  • Integral Probability Metrics measure witness-function expectation discrepancies and can signal proximity between nearby regions of probability mass.
  • MMD GANs adversarially train convolutional features before applying a kernel MMD, while adopting regularization strategies developed for Wasserstein critics.
  • MMD gradients are unbiased for a fixed learned representation but biased relative to optimal-critic gradients when the representation is learned from samples, paralleling WGANs.
  • MMD GANs match WGAN-GP generator performance with smaller discriminator networks, yielding fewer parameters and faster training.

2 LOSSES AND WITNESS FUNCTIONS

The paper formulates MMD as an IPM over an RKHS witness class and explains how feature mappings and critic regularization integrate it into GAN training. It also analyzes kernel selection and shows that the Cramér critic can fail to distinguish certain distributions despite good empirical performance.

  • 2.1 MAXIMUM MEAN DISCREPANCY AND WITNESS FUNCTIONS: MMD is an IPM whose witness functions form the unit ball of an RKHS, with the kernel determining the probability metric.
  • 2.1 MAXIMUM MEAN DISCREPANCY AND WITNESS FUNCTIONS: The MMD witness function compares expected kernel similarities to samples from the generator and reference distributions.
  • 2.1 MAXIMUM MEAN DISCREPANCY AND WITNESS FUNCTIONS: Exponentiated quadratic kernels and their derivatives decay exponentially, creating significant problems in high-dimensional gradient-based representation learning.
  • 2.2 WITNESS FUNCTION AND GRADIENT PENALTIES: MMD GANs apply MMD to convolutional features, equivalently using a kernel transformed by the feature mapping, and can adopt Wasserstein critic-training strategies.
  • 2.2 WITNESS FUNCTION AND GRADIENT PENALTIES: Gradient regularization constrains the critic witness norm to be nearly 1 along random convex combinations of generator and reference samples.
  • 2.3 THE ENERGY DISTANCE AND ASSOCIATED MMD: The energy distance is an MMD with a distance-induced kernel, characteristic when β < 2 and well-defined under a moment condition.
  • 2.3 THE ENERGY DISTANCE AND ASSOCIATED MMD: The Cramér critic replaces an independent reference sample with the origin, a change that can assign zero loss to distinct point-mass distributions.
  • 2.3 THE ENERGY DISTANCE AND ASSOCIATED MMD: Despite this theoretical failure mode, the Cramér critic has shown good empirical performance in prior work and the paper’s experiments.

3 GRADIENT BIAS

The paper distinguishes unbiased gradients for fixed critics from biased gradients caused by learning critics from samples. This distinction applies analogously to WGANs and MMD GANs, while the practical meaning of the respective unbiased objectives remains an open question.

  • Theoretical clarification: Bellemare et al.’s energy-distance gradient argument omitted justification for exchanging expectations and derivatives.The paper proves this exchange under mild assumptions covering common feedforward networks, including convolutions, max pooling, and ReLU activations.
  • Wasserstein GANs: For fixed generator parameters, the WGAN test-set gradient estimator is unbiased, but selecting critic parameters from training samples makes the resulting estimator biased.The bias arises from the learned-critic procedure rather than from evaluating the gradient with a fixed critic.
  • MMD GANs: MMD GANs, including energy-distance variants, have the same structure: MMD^2 estimates are unbiased for fixed critic representations, whereas optimizing the critic from samples yields biased generator gradients.The bias concerns the population divergence involving the optimized critic, not the fixed-representation MMD^2 estimator.
  • Interpretation: The bias vanishes as critic selection improves and is not introduced merely by using a fixed small minibatch size.Instead, the relevant factors are the critic optimization procedure and the total number of samples used to train it.
  • Interpretation: Holding an MMD GAN critic fixed corresponds to a GMMN model, whereas holding a WGAN critic fixed can produce an incorrect generator optimum.The paper suggests that unbiased gradients of MMD^2 may therefore be more meaningful than unbiased gradients of the learned-critic objective, but leaves this for future work.

4 EVALUATION METRICS

The paper reviews limitations of common GAN evaluation metrics and proposes KID as an alternative with an unbiased estimator. It also uses KID-based statistical comparisons to adapt GAN learning rates during training.

  • Evaluation motivation: Quantitative GAN comparisons are difficult, motivating evaluation through sample inspection and numerical metrics.The section discusses Inception score, FID, KID, and an approximate distinct-image diagnostic.
  • Existing metrics: The Inception score favors confident per-image predictions together with a diverse marginal predictive distribution, but its correlation with human judgments is imperfect.It is based on the Inception classifier output p(y | x).
  • Existing metrics: FID compares Gaussian fits to 2048-dimensional Inception activations, but its estimator is biased and can reverse the ordering of models at finite sample sizes.The paper reports that this reversal can occur even for one-dimensional Gaussian distributions and that low estimated variance may conceal the problem.
  • Kernel Inception Distance: KID is the squared MMD between Inception representations and uses a polynomial kernel, avoiding a parametric activation-distribution assumption while providing a simple unbiased estimator.With the cubic kernel, KID compares skewness as well as mean and variance; its computation scales as O(n^2d), so the paper recommends small blocks averaged across estimates.
  • Kernel Inception Distance: On CIFAR-10 train-versus-test comparisons, KID converges quickly toward zero, whereas FID remains substantially positive at n = 2 000 and continues decreasing at n = 10 000.The paper concludes that FID scores should only be compared when the sample count n is the same.
  • Evaluation limitations: The distinct-image diagnostic is difficult to compare reliably because duplicate-image judgments are subjective and may be affected by knowing which models are being evaluated.The paper leaves further exploration of this procedure to future work.
  • Adaptive training: The proposed adaptive schedule compares KID scores from current and earlier iterations using a relative-similarity-test p-value, reducing the learning rate after repeated failures.This replaces hand-tuned GAN learning-rate schedules with a criterion based on statistical comparison rather than an explicit score-change threshold.

5 EXPERIMENTS

Experiments compare MMD GAN variants with WGAN-GP and Cramér GAN across standard image datasets, evaluating kernels, critic sizes, sample quality, and training speed. MMD GANs generally match or exceed competitors while using smaller, faster critics, though performance depends on dataset, kernel, and metric.

  • Experimental setup: MMD GANs were compared with WGAN-GP and Cramér GAN on MNIST, CIFAR-10, LSUN Bedrooms, and CelebA.The experiments used multiple kernels and critic architectures across these four benchmark datasets.
  • CelebA: MMD GAN with the rq* kernel outperforms WGAN-GP and Cramér GAN in KID and FID on CelebA.The CelebA comparison used a higher-resolution setup with a ResNet generator.
  • Training efficiency: 16-filter DCGAN critics typically run about twice as fast as 64-filter critics.Critic size matters more than generator size because critics are updated five times per generator step and evaluated on two batches.
  • MNIST: On MNIST, linear and RBF kernels perform worse early, while RBF never fully catches up with the other models.The distance kernel and WGAN-GP may also converge more slowly than the rq and Cramér GAN models.
  • CIFAR-10: Small-critic MMD GAN models approximately match large-critic WGAN-GP models on CIFAR-10 at substantially reduced computational cost.The comparison reports similar performance despite the smaller critic.
  • Evaluation caveat: Inception scores are not meaningful for LSUN because its domain differs drastically from ImageNet class labels.The LSUN evaluation therefore emphasizes KID and FID for comparing models.
  • LSUN Bedrooms: MMD GANs outperform Cramér GAN and WGAN-GP in KID and FID for LSUN Bedrooms at each critic size.Small critics reduce results for every model, but small-critic MMD GANs still produce reasonably good samples, unlike WGAN-GP in this comparison.

A SCORE FUNCTIONS, DIVERGENCES, AND THE CRAMÉR GAN

This section connects energy distance and Cramér GAN objectives through scoring rules, then analyzes generalized IPM estimators based on data splitting. It shows that critic selection from training samples creates an estimator framework whose behavior differs from fixed-critic estimation and requires independent evaluation data for certain divergence objectives.

  • Score functions: A proper scoring rule gives an expected score that is maximized when the forecast distribution matches the observed distribution.Strict propriety makes equality occur only when the two distributions agree.
  • Energy distance: The energy distance arises from a scoring rule comparing average generator-to-reference distance with average distance between independent generator samples.Taking expectation over reference samples recovers the scoring rule optimized by DISCO Nets.
  • Cramér GAN: Cramér GAN applies the energy distance after mapping samples through a learned function h designed to maximize critic discrimination.The transformed energy distance is expressed as De(P, Q) = S(Q, Q) − S(P, Q).
  • Cramér GAN: Maximizing the divergence requires two independent reference samples Y and Y′ from Q.Without both samples, the terms in the divergence cannot be estimated for discriminator training as specified.
  • Cramér GAN: Optimizing only the average score leaves the reference-distribution variability term uncontrolled and it may explode.The learned representation can disperse reference features broadly, far from the generator support.
  • Estimator bias: The paper argues that natural sample-based estimators of IPM-like distances are biased in nontrivial cases, while fixed-critic estimators can be unbiased.The analysis is developed using a generalized IPM framework for MMD GANs and WGANs.
  • Data-splitting estimators: A data-splitting estimator selects a critic using training samples and evaluates its objective on independent test samples.The estimator is defined by the train-test split, critic-selection procedure, and estimator of the objective functional.
  • Data-splitting estimators: The GAN training process can be viewed as data splitting, with previously seen data forming the training set and the current minibatch forming the test set.This correspondence applies to WGAN-type optimization when critic parameters depend on earlier data.

B.2 ESTIMATOR BIAS

The section shows that sample-based estimators of generalized IPMs are generally downward biased, and that this bias produces biased generator gradients unless it is parameter-independent.

  • Estimator bias: Data-splitting estimators are biased downward unless the learned critic is almost surely optimal.The bias equals the expected suboptimality of the critic selected from training samples.
  • Estimator bias: This bias depends on the training procedure and samples seen, not directly on the test minibatch size.It vanishes as the learned critic converges to its optimum but remains whenever the critic is suboptimal.
  • Estimator bias: No unbiased estimator exists for the generalized IPM under the theorem’s assumptions, including empirical-distribution estimators such as standard MMD and Wasserstein estimators.The proof uses the non-polynomial dependence of the IPM on mixture weights.
  • Scope: The nonexistence proof does not determine whether an unbiased estimator exists for the generalized MMD GAN objective, and squared MMD is a separate case.For squared IPMs, unbiased estimators can exist with at least two samples, as for squared MMD.
  • Gradient bias: Because IPM estimator bias is generally nonconstant, learning the critic from samples almost certainly yields biased gradients for generator parameters.The result distinguishes unbiased gradients with a fixed critic from gradients obtained after critic selection from training data.
  • Gradient bias: A biased objective estimator has unbiased gradients only when its bias is constant across generator parameters.Theorem 4 implies the expected estimator differs from the target by a parameter-independent constant within each connected component.

B.4 WGANS

The WGAN analysis illustrates the general bias results: fixed-critic gradients can be unbiased, while selecting a critic from samples produces biased estimates and gradients.

  • Scope: The bias theorems apply to original WGANs, neural-network distances, and adversarial divergences defined by the attainable critic class.The framework covers both exact Lipschitz critics and critic architectures used in practice.
  • Estimator bias: Sample-based critic selection produces downward-biased WGAN estimates and generally biased gradients.The argument applies when the selected critic is not reliably optimal and the bias is not constant.
  • Example: With P = N(1, 1) and Q = N(0, 1), the true Wasserstein distance is 1, but one-sample critic selection can choose the wrong-sign critic.When the observed training sample ordering reverses, the selected critic can have expected response −1.
  • Example: A selection rule that is correct for one Gaussian pair can fail on a second pair, showing why universal unbiasedness cannot come from stubbornly choosing one critic orientation.The same rule gives −1 for P = N(−1, 1) and Q = N(0, 1), although the distance remains 1.

B.5 MAXIMAL MMD ESTIMATOR

The maximal MMD estimator inherits the same sample-selection bias as WGAN estimators: finite-sample optimization generally underestimates the optimized MMD and biases generator gradients.

  • Estimator construction: MMD GANs estimate η(P, Q) by optimizing a sample-based squared MMD over feature-network parameters.The optimized feature representation acts as the witness-function class in the maximal MMD objective.
  • Asymptotic behavior: As training sample sizes grow, the maximal MMD estimator should converge to η for certain kernels.The cited convergence result also provides rates for those kernels.
  • Gradient bias: Finite-sample optimization generally creates nonconstant bias, so gradients with respect to generator parameters are also biased.The bias arises because the optimized feature parameters are not almost surely perfect.
  • Example: For a linear projection with a linear kernel, η(P, Q) = 1 and is attained only by θ ∈ {(−1, 0), (1, 0)}.Any other unit-norm projection gives a strictly smaller squared MMD.
  • Example: With Gaussian samples and finite training sizes, the estimated projection almost surely misses the optimum, yielding E ˆη(X, Y) < 1.A numerical simulation gives approximately 0.6 when mtr = ntr = 2.

C PROOF OF UNBIASED GRADIENTS

The proof establishes that gradients and expectations can be exchanged for the relevant deep networks outside a measure-zero bad parameter set, despite nondifferentiable activations such as ReLU.

  • Proof strategy: The proof of unbiased gradients proceeds by proving a general gradient–expectation exchange theorem for deep networks.The result is then used to derive the paper’s unbiased-gradient theorem.
  • ReLU difficulty: The standard differentiation condition can fail for ReLU because nondifferentiability locations move with the parameters and may affect a positive-measure input set.For fixed parameters, the example is nondifferentiable at Xθ = −θ1/θ2; over a parameter neighborhood, the exceptional set can have positive measure.
  • Proof strategy: Theorem 5 avoids that condition by exploiting neural-network structure to control average rates of change before applying dominated convergence.This provides the route for exchanging derivatives and expectations without requiring differentiability throughout a parameter neighborhood.
  • Conclusion: The exceptional parameter set ΘP has zero Lebesgue measure under the stated network assumptions.The argument uses Fubini’s theorem and almost-everywhere differentiability in parameter space for fixed inputs.
  • Assumptions: The network class includes affine modules, convolutions, and piecewise-analytic nonlinearities such as ReLU and max pooling.The nonlinear modules are required to be Lipschitz and analytic on pieces defined by analytic inequalities.
  • Assumptions: For ReLU in R2, analyticity domains are four quadrants, while max pooling on two inputs has two half-plane domains.The domains are specified by analytic inequalities involving coordinate signs or coordinate differences.

C.3 MAIN RESULTS

The paper proves conditions under which population expectations and sampled gradients are differentiable and unbiased for WGAN, original GAN, and MMD GAN objectives. It also establishes the corresponding kernel conditions and highlights finite-sample evaluation-ordering failures for FID.

  • General result: Theorem 5 establishes differentiability of θ ↦ E_P[K(h_θ(X))] for almost every parameter value under Assumptions A–D.Its differential is given by E_P[∂_θK(h_θ(X))].
  • Original GANs: Gradients of all original GAN loss functions are unbiased under the stated bounded-output assumptions.The result follows by applying Theorem 5 to the relevant logarithmic network compositions.
  • MMD GANs: MMD GAN gradients are unbiased when the kernel satisfies Assumption E and the generator and critic representation networks satisfy the stated regularity assumptions.The proof combines unbiased U-statistic estimation of MMD² with the differentiability theorem.
  • Kernel conditions: The analyzed dot-product, RBF, rational-quadratic, and distance kernels satisfy the required growth condition with α at most 2.The distance kernel permits 1 ≤ β ≤ 2, with α = β.
  • Evaluation implications: FID estimates can reverse the true ordering of models at practical sample sizes, despite small apparent variance, whereas KID estimators are unbiased in the paper’s comparison.The cited high-dimensional example reports the reversal at m = 50 000 and correct ordering at m = 100 000.

D.3 NON-EXISTENCE OF AN UNBIASED ESTIMATOR

The paper proves that no estimator can be unbiased for FID over broad distribution classes, including classes containing two-component Gaussian mixtures. This limits unbiased FID estimation in practical non-normal settings.

  • Impossibility result: No estimator of FID to an arbitrary fixed normal distribution Q can be unbiased over any class containing two-component Gaussian mixtures.The proof uses the fact that an estimator’s expectation is polynomial in the mixture weight, while the FID expression is not.
  • Impossibility result: The impossibility extends to the two-sample setting where Q is also unknown.The paper states this follows by the same extension used in the theorem.
  • Practical boundary: The conclusion is especially relevant because distributions used for FID in practice are described as non-normal.The paper therefore rules out a practical unbiased FID estimator within the analyzed scope.

E COMPARISON OF EVALUATION METRICS’ RESILIENCE TO NOISE

The paper compares FID, KID, and Inception scores as image disturbances increase on CelebA and CIFAR-10. It uses scaled scores and a train-test evaluation setup that differs slightly from prior work.

  • Experimental setup: As disturbance level α increases, the experiments alter images progressively farther from the reference distribution.Figures 6–11 compare FID, KID, and negative Inception score for CelebA and CIFAR-10 after scaling scores to [0, 1].
  • Experimental setup: The study uses the standard train-test split, with disturbances applied to 160 000 training images and comparisons made against 20 000 test images.This differs slightly from Heusel et al.’s comparison using 50 000 disturbed images and the full 200 000-image dataset.
  • Observed behavior: In this setting, the Inception score is monotonic with increasing noise for more disturbance types than reported by Heusel et al. (2017).The authors also report similar findings beyond this comparison, but the supplied passage does not specify them.

F SAMPLES AND DETAILED RESULTS FOR MNIST AND CIFAR-10

MNIST models achieve broadly reasonable evaluation scores, while CIFAR-10 results show that rational-quadratic MMD GANs can match large-critic WGAN-GP at lower computational cost. Kernel choice affects training behavior and sample quality.

  • MNIST: After 50 000 generator iterations, all MNIST variants achieved reasonable results, with KIDs essentially zero and FIDs near the test-set level.Inception scores were slightly lower, and only the distance-kernel MMD model improved with more top-layer neurons.
  • Training behavior: RBF MMD training more frequently produces extremely blurry outputs that can persist before eventually resolving.The paper attributes this behavior to rapid gradient decay when generated samples are far from reference samples.
  • CIFAR-10: Rational-quadratic MMD GANs with small critics approximately match large-critic WGAN-GP scores at substantially reduced computational cost on CIFAR-10.With small critics, WGAN-GP, Cramér GAN, and the distance kernel performed very poorly.
  • MNIST: Rational-quadratic and Gaussian kernels retain sample quality with reduced discriminator complexity and generate good samples using the standard DCGAN discriminator with critic size 64.These samples are the models listed in Table 3.
  • CIFAR-10: Small-critic WGAN-GP samples are quite poor, whereas large-critic Cramér GAN and rational-quadratic MMD GAN samples are described as having similar quality.Small-critic Cramér GAN samples appear more appealing but show pixel-level issues.
Loading 1801.01401v5…