Source-linked AI summary
Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy
Danica J. Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, Arthur Gretton
TL;DR
The paper addresses how to compare high-dimensional distributions and evaluate generative-model samples when density estimates and visual or classifier judgments are limited. It optimizes MMD test power through learned kernels and features, yielding a more discriminative and interpretable comparison of model and reference data. The method also includes efficient null-quantile estimation and is subject to an equal-sample assumption in the presented analysis.
Problem
High-dimensional distribution evaluation is difficult because density estimates are unreliable and human judgments may miss under-dispersed samples from generative models.
Method
The method maximizes the estimated power of a quadratic-time MMD two-sample test by learning kernel and feature representations, with null quantiles estimated by permutation.
Results
The optimized MMD gives greater test power than a standard RBF kernel and outperforms simply maximizing MMD across a variety of synthetic problem settings.
Takeaways & Limitations
Optimized MMD provides an interpretable way to identify how generative-model and reference distributions differ, including through witness-function extrema and learned feature relevance.
Takeaways & Limitations
The analysis assumes equal numbers of samples from the two distributions.
Abstract
from arXiv · showhide
We propose a method to optimize the representation and distinguishability of samples from two probability distributions, by maximizing the estimated power of a statistical test based on the maximum mean discrepancy (MMD). This optimized MMD is applied to the setting of unsupervised learning by generative adversarial networks (GAN), in which a model attempts to generate realistic samples, and a discriminator attempts to tell these apart from data samples. In this context, the MMD may be used in two roles: first, as a discriminator, either directly on the samples, or on features of the samples. Second, the MMD can be used to evaluate the performance of a generative model, by testing the model's samples against a reference data set. In the latter role, the optimized MMD is particularly helpful, as it gives an interpretable indication of how the model and data distributions differ, even in cases where individual model samples are not easily distinguished either by eye or by classifier.
1 INTRODUCTION
The paper motivates optimized MMD as a way to compare distributions and evaluate or train generative models when conventional divergences, density estimates, and human judgments are difficult to interpret in high dimensions.
- Motivation: High-dimensional distribution comparison is needed for testing, learning, and assessing the quality of samples from complex generative models.GANs provide feedback through an interleaved generator–discriminator game, but the choice of divergence affects the compromises made by an imperfect generator.
- Motivation: Different f-divergences produce different mode-seeking or support-covering behaviors, which become difficult to anticipate or visualize in high dimensions.Some divergences emphasize individual modes, whereas others encourage coverage of the data distribution, potentially including low-probability regions.
- MMD in GANs: MMD is an integral probability metric that can distinguish generated and reference samples directly or through learned features.Prior GAN approaches used MMD either on images themselves or on autoencoder features, with feature-based methods reporting better visual quality but more difficult interpretation.
- Evaluation: Density-based evaluation is unreliable in high dimensions, while human evaluation may miss under-dispersed samples.These limitations motivate a distributional test against reference data rather than relying only on likelihood estimates or visual inspection.
- Contribution: The proposed method chooses features and kernels by maximizing the estimated power of a quadratic-time MMD test, rather than merely maximizing MMD.The authors also develop an efficient method for estimating null quantiles and report that explicit test-power optimization is more powerful on artificial data.
- Interpretability: The optimized MMD supports interpretable evaluation by identifying where model and data probability mass differs most.An ARD kernel can identify meaningful output coordinates, and witness-function extrema provide representative model and data samples.
2 MAXIMIZING TEST POWER OF A QUADRATIC MMD TEST
The method optimizes kernel and feature representations for the power of a quadratic-time MMD two-sample test, while preserving test validity through held-out testing and estimating null thresholds by permutation.
- MMD test: The MMD compares distributions through expectations of a kernel, and characteristic kernels make it zero exactly when the distributions are equal.Finite-sample test power still varies across characteristic kernels, motivating kernel selection.
- MMD test: The test rejects H0: P = Q when the scaled unbiased MMD statistic exceeds a threshold cα.The threshold is chosen for a specified false-rejection probability α.
- Null calibration: Because the null distribution depends on the unknown distribution, the threshold is estimated from repeated permutations of the pooled samples.The resulting data-dependent quantile is obtained by evaluating the statistic after randomly repartitioning X ∪ Y.
- Power optimization: Under the alternative, test power depends on both the MMD and its asymptotic variance, so the method optimizes their power-relevant ratio.For small samples or distributions close to the null, both terms must be considered rather than optimizing MMD alone.
- Power optimization: As sample size grows, the method can asymptotically optimize power by maximizing the kernel t-statistic and its empirical estimator.The estimator is differentiable with respect to kernel parameters and can therefore be optimized directly.
- Validity: The learned kernel must be selected on training data and evaluated on separate testing data to maintain hypothesis-test validity.This separation prevents kernel selection from invalidating the final test.
- Kernel families: A learned feature map can be composed with a standard kernel, allowing both representation and kernel parameters to be selected by gradient optimization.The composition remains a kernel, and injective feature maps with characteristic kernels preserve the characteristic property.
- Variance estimation: The variance estimator is differentiable and incorporates second-order terms while correcting small sources of bias.Its computation is expressed through sums involving kernel matrices.
3 EFFICIENT IMPLEMENTATION OF PERMUTATION TESTS FOR [
The section presents implementation strategies that make permutation-based MMD null-distribution sampling efficient, especially by avoiding costly kernel-matrix copying and improving memory access. Experiments show the optimized permutation test is substantially faster than spectral and naive permutation implementations, with favorable scaling for larger problems.
- Implementation optimizations: The optimized permutation approach avoids copying the kernel matrix by using permutation indices and an inverse map for sequential matrix traversal.The inverse-map strategy improves cache behavior by enabling hardware prefetching and reducing CPU cache misses.
- Experimental setup: The experiment compares 200 null-distribution samples on m = 2000 two-dimensional samples from matched-moment Gaussian and Laplace distributions.Results are averaged over 30 runs, with negligible variance across runs.
- Runtime comparison: For larger problems, the permutation implementation scales as O(m^2), compared with O(m^3) for the spectral test, making it an order of magnitude faster.This comparison uses all 24 available system threads.
- Runtime comparison: The implementation is faster on a single thread and saturates more slowly as the number of threads increases for m = 2000.The thread-scaling comparison is shown in Figure 1 (left).
- Runtime comparison: At m = 2000, the optimized implementation takes 12s, versus 87s for Shogun spectral, 182s for naive Python permutation, and 381s for Matlab spectral.These non-parallelized timings exclude kernel computation.
4 EXPERIMENTS
The experiments evaluate optimized MMD for bandwidth selection, model criticism, and GAN training. Across these settings, maximizing estimated test power identifies distributional differences more reliably and can expose subtle artifacts or mismatches.
- Synthetic data: Maximizing estimated test power outperforms maximizing MMD alone across Blobs problem parameters and performs near-optimally.The median heuristic selects overly large bandwidths, while MMD-only optimization sometimes selects low-power regions.
- Model criticism: 98 of 100 MNIST tests obtained p-values of 0.000, with the remaining two obtaining 0.001, using the optimized ARD-type kernel.The corresponding power was substantially higher than for bandwidths selected by maximizing the t statistic or using the median heuristic.
- Model criticism: The learned kernel weights identify differences along image borders and a vertical center line between model and true MNIST samples.These weights provide an interpretable localization of where the distributions differ.
- Model criticism: The witness function indicates slight overproduction of /-like digits and underproduction of vertical 1s, despite classifier-based digit proportions showing no general deficit of 1s.Such proportion differences would be difficult for human observers to detect.
- GAN criterion: MMD is also evaluated as a GAN training criterion using direct image kernels and feature kernels, with t-GMMN minimizing the estimated kernel-specific test statistic.Feature matching uses a mixture of RBF kernels to match full feature distributions rather than only their means.
A VARIANCE OF THE PAIRWISE MMD ESTIMATOR
The appendix notes publication errors and directs readers to a later reference for the corrected treatment. It specifically identifies an equivalence between the appendix estimator and an estimator in Sutherland (2019).
- A VARIANCE OF THE PAIRWISE MMD ESTIMATOR: The publication version of the appendix contained small mistakes.This is an explicit correction notice for the appendix.
- A VARIANCE OF THE PAIRWISE MMD ESTIMATOR: Readers are directed to Sutherland (2019) instead of the publication version of this appendix.The cited later document is the recommended reference for the corrected material.
- A VARIANCE OF THE PAIRWISE MMD ESTIMATOR: Estimator (5) in the appendix is equivalent to estimator (4) in Sutherland (2019).The equivalence is stated specifically for these two estimators.