Source-linked AI summary
Training generative neural networks via Maximum Mean Discrepancy optimization
Gintare Karolina Dziugaite, Daniel M. Roy, Zoubin Ghahramani
TL;DR
The paper asks how to train deep generators from i.i.d. samples of an unknown distribution without an explicit density model. It replaces adversarial discrimination with an unbiased empirical MMD objective, proves estimation-error bounds, and reports stronger held-out KDE log density than adversarial nets on MNIST despite visible digit artifacts.
Problem
The paper studies learning deep generative networks from i.i.d. samples of an unknown distribution while generating samples without explicitly representing its density.
Method
MMD nets replace the adversarial discriminator with a kernel two-sample MMD objective, whose empirical estimate is optimized for a neural generator.
Results
MMD Nets achieved a mean held-out MNIST test log density of 315 ± 2 versus 225 ± 2 for adversarial nets, although the generated digits contained many artifacts.
Takeaways & Limitations
MMD provides a closed-form surrogate for the discriminator and may serve as an initialization for more costly procedures.
Takeaways & Limitations
KDE performs poorly in high dimensions, and the RBF kernel may not capture image-specific properties needed for better images.
Abstract
from arXiv · showhide
We consider training a deep neural network to generate samples from an unknown distribution given i.i.d. data. We frame learning as an optimization minimizing a two-sample test statistic---informally speaking, a good generator network produces samples that cause a two-sample test to fail to reject the null hypothesis. As our two-sample test statistic, we use an unbiased estimate of the maximum mean discrepancy, which is the centerpiece of the nonparametric kernel two-sample test proposed by Gretton et al. (2012). We compare to the adversarial nets framework introduced by Goodfellow et al. (2014), in which learning is a two-player game between a generator network and an adversarial discriminator network, both trained to outwit the other. From this perspective, the MMD statistic plays the role of the discriminator. In addition to empirical comparisons, we prove bounds on the generalization error incurred by optimizing the empirical MMD.
1 Introduction
The paper frames generative modeling as learning a deep network that transforms fixed noise into samples matching an unknown data distribution. It introduces MMD nets as an approximation to adversarial learning that replaces the discriminator with a closed-form two-sample statistic and analyzes its estimation error.
- Generative modeling learns a function that transforms fixed noise into outputs whose distribution is close to an unknown data distribution.Given the generator and noise distribution, new samples can be produced without explicitly representing the underlying density.
- Deep neural generators transport input randomness to output randomness, inducing a distribution rather than performing classification or prediction.The paper relates this generator perspective to earlier density networks and decoder interpretations.
- Prior work developed generative models for high-dimensional spaces using restricted Boltzmann machines, deep Boltzmann networks, and neural autoregressive density estimators.
- Adversarial nets train a discriminator to distinguish training from generated samples while the generator learns to confuse it through alternating gradient-based optimization.The resulting procedure is a two-player minimax game requiring iterative updates to both networks.
- MMD nets replace the adversary with a closed-form nonparametric two-sample statistic, and the paper provides empirical illustrations and bounds on empirical-MMD estimation error.
2 Learning to sample as optimization
The paper casts generator learning as minimizing a discrepancy between real and generated distributions, then uses RKHS-based MMD to obtain a tractable adversary. This framework connects adversarial objectives, unbiased empirical estimation, and stochastic optimization, with uniqueness guaranteed under suitable kernels.
- Learning to sample as optimization: A generator family Gθ is learned by minimizing an empirical estimate of a discrepancy between the data distribution and the distribution induced by Gθ applied to fixed noise.
- Adversarial nets: Adversarial nets fit this framework by using discriminators Dφ whose expected log loss measures how well real and generated samples can be distinguished.Optimizing the discriminator maximizes distinguishing ability, while the optimal generator minimizes Jensen–Shannon divergence to the data distribution.
- Adversarial nets: Adversarial training alternates gradient steps on generator and discriminator multilayer perceptrons, with the recommended update balance potentially sensitive to data and network choices.The cited experiments use two discriminator steps for each generator step, while larger ratios can lead to overfitting.
- MMD as an adversary: MMD nets replace discriminator functions with a function class H and minimize the maximum mean discrepancy between P and Gθ(N).When H is an RKHS, the maximization has a closed-form solution, enabling a tractable surrogate for the adversary.
- MMD as an adversary: For Gaussian or Laplace kernels on R^D, MMD is a metric, so the learning objective has a unique minimum when the generated distribution equals P.
- MMD nets: The empirical MMD objective is estimated from independent samples using kernel evaluations, then optimized for a neural generator with minibatch stochastic gradient descent.The algorithm regenerates noise inputs periodically, computes minibatch gradients, and updates generator parameters with a learning rate such as RMSProp.
3 MMD Nets
MMD nets train a generator by minimizing an empirical MMD objective computed from generated samples and training data. The objective is optimized with gradient descent and backpropagation.
- MMD Nets: MMD nets minimize C(Yθ, X), the θ-dependent part of an unbiased MMD estimator for generated samples and training data.Yθ contains outputs Gθ(wj) for noise inputs, while X contains the training samples.
- MMD Nets: Gradient descent solves the minimization, potentially using subsets of the data.
- MMD Nets: Backpropagation computes the generator-parameter derivatives needed to optimize the MMD objective.The gradient depends on partial derivatives of the generator with respect to its parameters.
4 Generalization bounds for MMD
The paper analyzes the estimation error caused by optimizing empirical MMD rather than population MMD. Under complexity and kernel assumptions, it proves bounds that remain valid for continuous, high-dimensional generator parameter spaces.
- 4 Generalization bounds for MMD: Empirical MMD minimizers may poorly minimize population MMD because the empirical objective contains Monte Carlo error.
- 4 Generalization bounds for MMD: The analysis bounds the difference between empirical optimization and the corresponding population objective over generator parameters θ.The parameter space Θ defines generators Gθ and their output distributions pθ.
- 4 Generalization bounds for MMD: The bounds use fat-shattering dimension to capture the complexity of the relevant function classes.The paper defines fatε(F) as the largest size of a set ε-shattered by F.
- 4 Generalization bounds for MMD: Theorem 1 assumes a kernel bounded by one and gives an estimation-error bound involving r(p1, γ1, M), r(p2, γ2, M−1), and 12M−1.The complexity assumptions bound fat-shattering dimensions by γ1ε−p1 and γ2ε−p2, with constants depending only on p1 and p2.
- 4 Generalization bounds for MMD: For a simpler bound, the observation space must be bounded and finite-dimensional, while the kernel must be Lipschitz continuous and translation invariant.For an RBF kernel, the bound loosens as the length scale shrinks.
5 Empirical evaluation
The paper evaluates MMD-based generator training on synthetic data, MNIST, and TFD using kernel-based objectives and density evaluation. MMD training achieves strong held-out density scores, although generated images retain visible artifacts and density estimation is problematic for high-dimensional data.
- 5 Empirical evaluation: MMD optimization is evaluated on a synthetic example, MNIST digits, and the Toronto Face Dataset using held-out kernel-density performance.
- 5 Empirical evaluation: Under an RBF kernel and Gaussian generator, the synthetic experiment uses N = 200 data samples and M = 50 fixed-seed noise inputs.
- 5 Empirical evaluation: The MNIST generator uses a fully connected three-hidden-layer sigmoidal network, with RBF, rational quadratic, and Laplacian kernels evaluated.The RBF kernel performed best in the evaluated parameter ranges.
- 5.2 MNIST digits: The MNIST figure reports no appreciable change in later MMD learning iterations and shows generated digits before and after further empirical-MMD optimization.
- 5.2 MNIST digits: 315 ± 2 mean log density on held-out MNIST data compares with 225 ± 2 reported for adversarial nets, despite visible generated-digit artifacts.The MMD model was trained for 1,000,000 iterations using minibatches of 500 and resampling generated digits every 300 iterations.
- 5. Empirical evaluation: The authors suggest that kernel-density estimation performs poorly in high dimensions and that image-specific kernels might improve generated images.They relate the MMD objective to a difference of kernel-density estimates, unlike the adversarial-network evaluation connection described here.
- 5.3 Toronto face dataset: For TFD, the MMD network reaches mean held-out log density 2283 ± 39 versus 2057 ± 26 reported for adversarial nets.The MMD samples remain clearly distinguishable from the training dataset, indicating limitations of kernel-density evaluation on these high-dimensional data.
6 Conclusion
The conclusion presents MMD as a closed-form surrogate for the adversarial discriminator, while noting that MMD Nets can achieve strong KDE scores despite visibly discrepant samples. It also suggests using MMD Nets to initialize more costly procedures.
- MMD offers a closed-form surrogate for the discriminator in the adversarial nets framework.
- MMD Nets is worth considering as an initialization for more costly procedures because it achieves this performance quickly.
A Proofs
The proofs establish estimation and approximation guarantees for empirical MMD optimization under bounded-kernel and complexity assumptions. In the nonparametric limit, the approximation error is zero when the data distribution lies in the generator family and the kernel is characteristic.
- Theorem 5 gives an estimation-error bound for a finite generator parameter set with probability at least 1 − (T + 1)δε.
- The general proof controls uniform deviations over generator parameters using bounded kernels, concentration inequalities, and function-class complexity assumptions.The proof explicitly uses McDiarmid’s inequality and fat-shattering-dimension conditions.
- Zero approximation error is obtained in the nonparametric limit.This follows when pdata belongs to the generator family and the RKHS uses a characteristic kernel.