PaperlayerView exact revision on arXiv

Source-linked AI summary

MMD GAN: Towards Deeper Understanding of Moment Matching Network

Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, Barnabás Póczos

arXiv:1705.08584v3cs.LGcs.AIstat.ML

TL;DR

GMMN’s empirical competitiveness and batch-size efficiency lag behind GANs on challenging datasets. This paper introduces MMD GAN, which replaces GMMN’s fixed Gaussian kernel with an adversarially learned kernel, and reports competitive performance with smaller batches.

  • Problem

    GMMN has weaker empirical performance than GANs on challenging benchmarks and requires larger training batches, limiting its competitiveness and efficiency.

  • Method

    MMD GAN replaces GMMN’s fixed Gaussian kernel with an adversarially learned kernel and trains generator and feature networks through MMD.

  • Results

    MMD GAN is reported to use much smaller batches than GMMN and achieve competitive performance with state-of-the-art GANs on challenging datasets.

  • Takeaways & Limitations

    MMD GAN provides a practical connection between moment matching and GANs, supporting further use of statistical tools in GAN development.

  • Takeaways & Limitations

    The theoretical construction assumes locally Lipschitz generators and injective feature maps, while the empirical study suggests the autoencoder objective is unnecessary for successful training.

Abstract

from arXiv · show

Generative moment matching network (GMMN) is a deep generative model that differs from Generative Adversarial Network (GAN) by replacing the discriminator in GAN with a two-sample test based on kernel maximum mean discrepancy (MMD). Although some theoretical guarantees of MMD have been studied, the empirical performance of GMMN is still not as competitive as that of GAN on challenging and large benchmark datasets. The computational efficiency of GMMN is also less desirable in comparison with GAN, partially due to its requirement for a rather large batch size during the training. In this paper, we propose to improve both the model expressiveness of GMMN and its computational efficiency by introducing adversarial kernel learning techniques, as the replacement of a fixed Gaussian kernel in the original GMMN. The new approach combines the key ideas in both GMMN and GAN, hence we name it MMD GAN. The new distance measure in MMD GAN is a meaningful loss that enjoys the advantage of weak topology and can be optimized via gradient descent with relatively small batch sizes. In our evaluation on multiple benchmark datasets, including MNIST, CIFAR- 10, CelebA and LSUN, the performance of MMD-GAN significantly outperforms GMMN, and is competitive with other representative GAN works.

1 Introduction

GMMN replaces GAN’s auxiliary discriminator with MMD, but its empirical performance and training efficiency remain weaker on challenging datasets. MMD GAN addresses these limitations through adversarially learned kernels.

  • GMMN uses kernel MMD instead of GAN’s auxiliary network to measure distances between data and generated distributions.
  • GMMN has not achieved empirical results comparable with GAN on challenging benchmarks.
  • GMMN requires larger training batches than GAN, reducing its computational efficiency.
  • MMD GAN replaces GMMN’s fixed Gaussian kernel with adversarially learned kernels to improve hypothesis-testing power.
  • MMD GAN introduces theoretical analysis, a practical realization, and feasible-set reduction for faster, more stable training.
  • Experiments report smaller-batch training, success on CIFAR-10, CelebA, and LSUN, and results comparable with other GAN works.

2 GAN, Two-Sample Test and GMMN

GAN measures distribution similarity with a learned auxiliary network, whereas GMMN uses kernel MMD and trains the generator to pass a two-sample test. Adversarial kernel learning strengthens this test while retaining a continuous, differentiable objective with meaningful distributional convergence.

  • GAN trains a generator from a base distribution and an auxiliary network to estimate the distance between real and generated distributions.
  • MMD is a kernel-based two-sample test for measuring the distance between two distributions from finite samples.
  • With a characteristic kernel, zero MMD is equivalent to equality of the two distributions.
  • GMMN trains the generator with a fixed Gaussian characteristic kernel rather than an additional discriminator.
  • Finite-sample MMD testing rejects equality only when the estimated distance exceeds a threshold, so weak kernels can reduce distinguishability.
  • MMD GAN adversarially selects characteristic kernels through injective feature transformations, maximizing MMD between real and generated distributions.
  • The optimized learned-kernel objective is continuous and almost everywhere differentiable, and converges to zero exactly under the stated distributional convergence conditions.

3 MMD GAN

MMD GAN implements adversarial kernel learning with neural generator and feature networks, using constraints and feasible-set reduction to support practical optimization. Its feature-space view connects the method to GMMN.

  • The implementation uses neural networks for the generator and feature map, with weight clipping or gradient penalties to bound gradients.
  • An autoencoder approximates the inverse of the injective feature map required by the theoretical construction.
  • The empirical study suggests the autoencoder objective is unnecessary for successful GAN training, despite the injectivity requirement in the theorem.
  • MMD GAN optimizes generator and feature networks in a GAN-like min–max formulation, but the feature network defines an adversarial MMD kernel rather than a classifier.
  • The algorithm alternates feature-network updates and generator updates using minibatches, RMSProp, and clipping of feature parameters.
  • MMD GAN performs the kernel two-sample test in a learned feature space, with GMMN as the special case using the identity mapping.
  • Feasible-set reduction preserves the optimal solution, while its relaxed penalty makes training faster and more stable in practice.

4 Related Works

The paper situates MMD GAN among Wasserstein, autoencoder-based, and moment-matching approaches. It identifies WGAN as a special case under particular kernel and output-dimension choices.

  • Under a linear kernel and one-dimensional feature output, MMD GAN recovers the WGAN objective.
  • WGAN can be interpreted as first-order moment matching, whereas Gaussian-kernel MMD GAN targets infinitely many moments through its kernel expansion.
  • The paper states that neural networks may lack sufficient capacity to approximate Wasserstein distance and reports benefits from matching higher-order moments.
  • Unlike EBGAN, which separates reconstruction objectives for real and generated samples, MMD GAN uses autoencoders to approximate invertible functions by reconstructing both.
  • Other related methods use linear-kernel latent-feature matching or empirical MMD variance, whereas MMD GAN introduces adversarial training of the kernel.

5 Experiment

Experiments compare MMD GAN with GMMN variants and WGAN across image-generation benchmarks, evaluating sample quality, inception scores, stability, computation time, and alternative Lipschitz constraints.

  • Qualitative Analysis: Increasing GMMN’s batch size to 1024 improves image quality but remains inferior to MMD GAN with B = 64.This comparison is presented as evidence that MMD GAN can be trained more efficiently with smaller batches.
  • Comparisons with GANs: MMD GAN generates smoother MNIST contours than WGAN while both methods avoid the mode-collapse problems described for GAN training.The comparison also covers CelebA and LSUN bedroom samples.
  • Quantitative Analysis: MMD GAN’s inception score exceeds those of other GAN techniques except DFM on CIFAR-10.The score is computed on 50K generated samples to measure sample quality and diversity.
  • Stability of MMD GAN: Lower MMD distance tracks improving sample quality during training, and the model converges shortly after tens of thousands of generator iterations on CelebA.The authors also observe stable qualitative structure across iterations for samples generated from the same noise vectors.
  • Computation Time: At B = 64, WGAN and MMD GAN require 0.268 and 0.676 seconds per generator iteration, respectively; at B = 1024, they require 4.431 and 8.565 seconds.The analysis attributes MMD GAN’s quadratic complexity to kernel-matrix computation but reports that GPU parallelization makes runtime nearly linear for modest batch sizes.
  • Better Lipschitz Approximation and Necessity of Auto-Encoder: Preliminary experiments indicate satisfactory MMD GAN results with gradient-penalty Lipschitz constraints and without auto-encoder reconstruction loss.The study also drops the injectivity requirement and calls for more thorough comparisons of constraint approximations.

6 Discussion

The discussion positions MMD GAN as a practical bridge between moment matching and GANs, combining adversarially learned kernels with theoretical and empirical advantages. It also identifies extensions and open questions, including polynomial kernels and the practical role of injectivity.

  • MMD GAN uses adversarially learned kernels, can train with much smaller batches than GMMN, and achieves competitive performance with state-of-the-art GANs.The paper presents it as a practical step connecting moment matching networks and GANs.
  • WGAN and MMD GAN are connected through first-order and infinite-order moment matching.The discussion describes WGAN as a special case of MMD GAN under certain conditions.
  • Polynomial kernels could extend MMD GAN to finite-order moment matching, motivated by strong domain-adaptation performance at approximately five moments.The cited passage identifies this as a possible extension rather than a demonstrated result in this paper.
  • Although injectivity is required for theoretical guarantees, the authors observe it is not mandatory in practice and conjecture neural networks often learn injective mappings.The injectivity observation is presented as an empirical finding and future-work question.

A.1 Proof of Theorem 3

Under a local Lipschitz assumption on the generator composition, the learned-kernel MMD objective is continuous everywhere and differentiable almost everywhere. The proof bounds changes in MMD using changes in the generated representations and applies standard convergence and differentiability results.

  • For Gaussian kernels, differences between kernel terms are bounded, allowing bounded convergence to establish continuity of the maximized MMD objective.The proof uses the triangle inequality and bounded convergence theorem after bounding each kernel difference by 1.
  • MMD between generated distributions is bounded by twice the expected representation change, yielding a local Lipschitz bound in generator parameters.The representation is hθ = fφ ◦ gθ, and the bound is expressed through local Lipschitz constants L(θ,z).
  • The maximized objective is locally Lipschitz and therefore continuous everywhere, while Rademacher’s theorem gives differentiability almost everywhere.This completes the theorem’s regularity argument for maxφ M_fφ(PX, Pθ).

A.2 Proof of Theorem 4

The theorem’s proof connects convergence in Wasserstein distance with convergence under the learned-kernel MMD objective. It relies on characteristic-kernel inequalities and the boundedness of the Gaussian kernel used in the paper.

  • The argument is explicitly based partly on results from reference [38].The passage identifies the proof’s dependence on prior theoretical results rather than presenting it as entirely self-contained.
  • The proof uses an existing result relating characteristic-kernel MMD to Wasserstein distance.Specifically, the cited inequality states M_k(P,Q) ≤ W(P,Q).
  • The paper’s kernel is k(x,x′) = exp(−∥fφ(x) − fφ(x′)∥2), and it is bounded above by 1.This boundedness is used in the converse direction of the proof.

A.3 Proof of Theorem 5

The supplied discussion relates MMD-based distances to continuity and differentiability properties, while cautioning that a previously proposed discontinuity counterexample uses a space not contained in an RKHS.

  • A sign flip of the scalar discriminator output preserves MMD distance while aligning the expected real-versus-generated ordering.For neural networks with a linear output layer, the sign flip can be implemented by flipping the final-layer weights.
  • Standard MMD is described as continuous and differentiable almost everywhere.The passage says the theorem and proof can be simplified for the standard MMD distance.
  • The cited discontinuity counterexample assumes H = L2, but the paper notes that L2 is not an RKHS.The authors therefore regard that counterexample as inappropriate for the setting under discussion.

B.2 IPM Framework

The IPM framework views probabilistic distances as suprema over function classes, with MMD using an RKHS-defined class. MMD GAN enlarges this class by adversarially learning kernels rather than fixing one kernel.

  • IPM formulation: IPM distances are defined by optimizing the difference between expectations under two distributions over a function class.Changing the function class recovers distances including total variation, Wasserstein distance, and MMD.
  • MMD formulation: For MMD, the function class consists of functions with RKHS norm at most one for the kernel k.
  • GMMN: GMMN uses MMD's analytical representation, avoiding an additional network to estimate the distribution distance.
  • MMD GAN: MMD GAN represents adversarially learned-kernel MMD within the IPM framework.
  • MMD GAN: Adversarial kernel learning yields an IPM with a larger function class than fixed-kernel MMD.The learned kernels satisfy k_i ∈ K for every i.
  • Moment matching: Kernel tricks let MMD match high-order, potentially infinite-order, moments while retaining theoretical guarantees.Directly matching high-order moments requires handling high-order tensors.
Loading 1705.08584v3…