Source-linked AI summary
Generative Moment Matching Networks
Yujia Li, Kevin Swersky, Richard Zemel
TL;DR
Learning deep generative models requires methods that can capture data structure without relying on difficult training procedures. The paper uses MMD to train feedforward generators and combines them with autoencoders to model code spaces. On MNIST and the Toronto Face Database, GMMN+AE achieves superior performance compared with other approaches.
Problem
Learning deep generative models requires a method that can generate samples while avoiding the difficult minimax optimization used by GANs.
Method
GMMNs minimize an MMD loss with backpropagation, while GMMN+AE models autoencoder codes and decodes generated codes into data-space samples.
Results
GMMN+AE achieves superior performance to other approaches on MNIST and the Toronto Face Database.
Takeaways & Limitations
The framework provides a simple, efficient way to train deep generative models and can discover the implicit data manifold on the evaluated datasets.
Takeaways & Limitations
The posterior distribution over latent variables is not readily available from a GMMN.
Abstract
from arXiv · showhide
We consider the problem of learning deep generative models from data. We formulate a method that generates an independent sample via a single feedforward pass through a multilayer perceptron, as in the recently proposed generative adversarial networks (Goodfellow et al., 2014). Training a generative adversarial network, however, requires careful optimization of a difficult minimax program. Instead, we utilize a technique from statistical hypothesis testing known as maximum mean discrepancy (MMD), which leads to a simple objective that can be interpreted as matching all orders of statistics between a dataset and samples from the model, and can be trained by backpropagation. We further boost the performance of this approach by combining our generative network with an auto-encoder network, using MMD to learn to generate codes that can then be decoded to produce samples. We show that the combination of these techniques yields excellent generative models compared to baseline approaches as measured on MNIST and the Toronto Face Database.
1. Introduction
The paper introduces GMMNs, feedforward generative networks trained with MMD rather than GAN-style minimax optimization. Combining GMMNs with autoencoders and evaluating on MNIST and TFD yields improved results over comparable baselines, including GANs.
- GMMNs generate independent samples through a deterministic feedforward network from a simple prior.This avoids the expensive MCMC procedures required by some alternative generative models.
- MMD provides a straightforward backpropagation objective that matches all moments of model and empirical data distributions.The kernel trick represents MMD as a simple loss, while minibatch stochastic gradient descent supports efficient training on large datasets.
- The GMMN+AE model learns a GMMN in an autoencoder’s code space and decodes generated codes into data-space samples.This leverages representations learned by the autoencoder as the basis for comparing data and model distributions.
- MNIST and the Toronto Face Dataset show improved results over comparable baselines, including GANs.The authors describe the framework as effective and efficient for producing generative models.
2. Maximum Mean Discrepancy
MMD tests whether two sample sets come from the same distribution by comparing their statistics. With suitable feature maps and kernels, it measures discrepancies across increasingly high-order statistics without explicitly computing them.
- MMD compares statistics from two sample sets to assess whether their generating distributions are equal.Similar statistics indicate that the samples are likely to come from the same distribution.
- The MMD measure computes the mean squared difference between statistics of the two sample sets.
- Using the identity feature map matches sample means, while other feature maps can match higher-order moments.
- The kernel trick rewrites MMD using only inner products between feature vectors.This enables the method to use kernel computations instead of explicitly representing the feature map.
- For universal kernels, MMD is asymptotically zero if and only if the two generating distributions are equal.The Gaussian kernel is given as an example of a universal kernel.
- A universal-kernel feature expansion contains infinitely many terms, making MMD equivalent to minimizing distances between all distributional moments.
3. Related Work
The paper situates GMMNs among deep generative models, emphasizing trade-offs in sampling, likelihood computation, variable ordering, and latent-variable inference. Related approaches include graphical models, autoregressive models, autoencoder-based MCMC methods, variational networks, and earlier feedforward generators.
- Deep generative models aim to learn latent manifolds that support smooth transformations and can disentangle factors of variation.
- Boltzmann machines have difficult training, evaluation, and sampling because of typically intractable partition functions and reliance on expensive MCMC.
- Fully visible directed models enable efficient likelihood calculation, gradient learning, and sampling but impose variable orderings and sequential computation.These constraints can be unnatural for images and prevent exploiting parallel computing methods.
- Autoencoder-based density models use contraction or denoising criteria to generate Markov chains through repeated encoding perturbations followed by decoding.
- Deep variational networks add a neural network to approximate latent posteriors and train with a variational lower bound on model log-likelihood.
- Earlier feedforward generative models mapped prior samples toward data space but associated an extra distribution with outputs and used sampling for learning and inference.
4. Generative Moment Matching Networks
GMMNs learn a deterministic mapping from an easy-to-sample prior to data samples and train it by minimizing MMD instead of a GAN-style minimax objective. The GMMN+AE variant models auto-encoder codes, while practical choices such as minibatches, mixed kernels, and square-root loss support training.
- Generative Moment Matching Networks: GMMNs map samples from a simple prior deterministically through a neural network to generate samples in the data space.The prior and network jointly define the model distribution, and generation requires one prior sample followed by a feedforward pass.
- Generative Moment Matching Networks: MMD replaces the difficult GAN minimax training objective with a differentiable loss that compares model and data distributions.The loss can be optimized with backpropagation through the generative network.
- Auto-Encoder Code Space Networks: Auto-encoder code spaces provide controllable dimensionality and can make MMD estimation more suitable for high-dimensional visual data.The motivation is that the amount of data required for a reliable MMD estimator grows with dimensionality.
- Auto-Encoder Code Space Networks: GMMN+AE trains an auto-encoder first, freezes its weights, and then trains a GMMN to match generated codes to encoded data codes.Samples are produced by decoding codes generated by the GMMN.
- Practical Considerations: Mixtures of multiple Gaussian kernels are used because kernel bandwidth strongly affects MMD efficiency and is difficult to set optimally.The authors report that simple bandwidth values and mixtures of five or more kernels were sufficient in most experiments.
- Practical Considerations: Square-root MMD loss improves behavior near zero by avoiding the flattening of squared MMD and maintaining larger gradients.Its gradient includes a factor that adapts the effective learning rate when the loss and its gradient become small.
- Practical Considerations: Minibatch optimization makes MMD training practical despite the objective’s quadratic scaling with dataset size.Each update uses a small subset of data and an equal number of generated samples.
5. Experiments
Experiments evaluate GMMNs on MNIST and TFD using Parzen-window test likelihoods and qualitative sample analyses. The GMMN+AE generally provides the strongest results and produces smooth, realistic samples while reducing concerns about memorization.
- Experimental setup: MNIST and TFD experiments compare GMMNs in input and auto-encoder code spaces against established generative-model baselines.The baselines include DBN, Stacked CAE, Deep GSN, and GANs.
- Experimental setup: The auto-encoders were trained separately, then fixed while GMMNs matched generated-code statistics to data-code statistics before decoding samples.Generated codes were passed through the decoder to return to the input data space.
- Quantitative evaluation: GMMN is competitive with other approaches, while GMMN+AE significantly outperforms them on test-set log-likelihood.Test-set log-likelihoods were estimated with a Gaussian Parzen window fitted to 10,000 generated samples.
- Qualitative evaluation: GMMN+AE produces the most visually appealing samples, consistent with its Parzen-window log-likelihood estimates.The decoder can correct code-space noise because perturbations correspond to smooth transformations along the data manifold.
- Qualitative evaluation: Nearest-neighbor comparisons suggest that generated samples are not merely copied training examples under Euclidean pixel-wise distance.Figure 2 places model samples above their corresponding nearest training-set neighbors for the relevant panels.
- Qualitative evaluation: Interpolations in the GMMN+AE latent space are mostly smooth and yield realistic-looking projections, including changes in pose, expression, lighting, gender, and facial hair on TFD.The experiment linearly interpolated between five points in uniform space and projected them into data space.
6. Conclusion and Future Work
The paper presents GMMNs as a simple MMD-based framework for deep generative modeling, including an autoencoder-based code-space extension. It reports strong results on MNIST and TFD while identifying computational, training, inference, and scaling directions for future work.
- Conclusion: GMMNs optimize maximum mean discrepancy with a kernel-based objective, avoiding explicit moment computation while supporting minibatch stochastic-gradient training.The approach matches model and data distributions through their moment statistics without explicitly calculating those moments.
- Conclusion: The GMMN+AE model applies MMD to autoencoder codes and decodes generated codes into samples in the original data space.This combines distribution matching with representations learned by an autoencoder.
- Conclusion: GMMN+AE achieves superior performance compared to other approaches on MNIST and the Toronto Face Database, and demonstrates discovery of the data's implicit manifold.The paper also visualizes manifold exploration through interpolations between generated points.
- Future Work: Future work includes linear-time MMD estimators and random features to reduce the quadratic cost associated with minibatch size.These alternatives are proposed as ways to speed up training.
- Future Work: Jointly training the autoencoder and GMMN may produce codes better suited to both reconstruction and generation.The current approach treats the two models separately.
- Future Work: GMMNs make sampling easy, but the posterior distribution over latent variables is not readily available.The paper suggests learning a neural network to predict latent vectors from samples as one possible inference strategy.
- Future Work: Autoencoder-based GMMNs may extend to more complex datasets, including high-resolution color images with convolutional autoencoders.This is presented as a possible application rather than a demonstrated result.