On GANs and GMMs

Eitan Richardson; Yair Weiss

Source-linked AI summary

On GANs and GMMs

Eitan Richardson, Yair Weiss

arXiv:1805.12462v2May 31, 2018cs.CVcs.LG

TL;DR

The paper investigates whether GANs learn useful statistical models of high-dimensional images beyond producing realistic samples. It proposes a bin-proportion evaluation method and compares GANs with full-image GMMs, finding that GMMs capture the full distribution and support efficient inference, though their samples are less sharp.

Problem
The paper asks whether GANs learn useful statistical models of images despite reported mode collapse and difficult inference beyond sample generation.
Method
The paper proposes comparing reference and generated sample proportions in predetermined bins and compares GANs with GMMs trained on full-sized images.
Results
GMMs generate realistic but less sharp samples than GANs while capturing the full distribution and enabling efficient inference with explicit statistical structure.
Takeaways & Limitations
The results motivate statistical image models that retain efficient inference and accurate structural representation even without producing the sharpest images.
Takeaways & Limitations
The paper does not claim GMMs are the ultimate solution for full-image modeling, and their samples remain less sharp than GAN samples.

Abstract

from arXiv · show

A longstanding problem in machine learning is to find unsupervised methods that can learn the statistical structure of high dimensional signals. In recent years, GANs have gained much attention as a possible solution to the problem, and in particular have shown the ability to generate remarkably realistic high resolution sampled images. At the same time, many authors have pointed out that GANs may fail to model the full distribution ("mode collapse") and that using the learned models for anything other than generating samples may be very difficult. In this paper, we examine the utility of GANs in learning statistical models of images by comparing them to perhaps the simplest statistical model, the Gaussian Mixture Model. First, we present a simple method to evaluate generative models based on relative proportions of samples that fall into predetermined bins. Unlike previous automatic methods for evaluating models, our method does not rely on an additional neural network nor does it require approximating intractable computations. Second, we compare the performance of GANs to GMMs trained on the same datasets. While GMMs have previously been shown to be successful in modeling small patches of images, we show how to train them on full sized images despite the high dimensionality. Our results show that GMMs can generate realistic samples (although less sharp than those of GANs) but also capture the full distribution, which GANs fail to do. Furthermore, GMMs allow efficient inference and explicit representation of the underlying statistical structure. Finally, we discuss how GMMs can be used to generate sharp images.

1 Introduction

The paper asks whether GANs learn useful statistical models rather than merely producing realistic images, comparing them with GMMs and motivating a distribution-focused evaluation method. It addresses concerns about mode collapse, weak evaluation, and difficult inference while examining full-image modeling.

GANs generate remarkably realistic high-resolution samples but are criticized for mode collapse and difficulty modeling the entire data distribution.
Existing GAN evaluation methods rely on learned representations or diversity measures that may miss image properties or fail to assess whether the true data distribution is captured.
Applying log likelihood or Wasserstein distance to GANs is problematic because full probability modeling and high-dimensional computations are intractable or require approximations.
The paper compares GANs with Gaussian Mixture Models trained on the same datasets to study their usefulness as statistical image models.
The proposed evaluation method compares relative proportions of generated and reference samples in predetermined bins without an additional neural network or intractable computations.

2 A New Evaluation Method

The proposed NDB method evaluates whether generated and reference samples have matching bin proportions, using Voronoi cells formed from training data and statistical tests across bins. It operates directly on pixels and measures distribution matching rather than diversity alone.

NDB compares the numbers of reference and generated samples in each bin, treating differences as a two-sample test for Bernoulli variables.
Voronoi bins are constructed by K-means clustering of training samples, and generated samples are assigned to their nearest L2 centroid.
The method reports the number of statistically different bins, with NDB divided by K expected to equal the significance level when both samples come from the same distribution.
Unlike IS and FID, NDB operates directly on image pixels, making it domain agnostic and sensitive to image properties omitted by task-specific representations.
Unlike MS-SSIM and Birthday Paradox methods, NDB measures similarity between data and generated distributions rather than only general sample diversity.
NDB can reveal mode collapse while capturing semantic and physical image properties, although pixel-space L2 distance may not always be meaningful.

3 Full Image Gaussian Mixture Model

The paper trains full-image Gaussian mixture models using Mixture of Factor Analyzers, addressing the dimensionality and distributional complexity of full-image modeling. Low-rank covariance structure and matrix identities make likelihood computation linear in image dimension.

Motivation: Full-image GMMs face both quadratic memory and computation from covariance matrices and potentially exponentially many components as dimension grows.For 64 × 64 color images, a single covariance matrix has 7.5 × 10^7 free parameters.
Model: The model uses Mixture of Factor Analyzers, representing each component with a low-dimensional latent vector and isotropic or diagonal noise.Factor Analyzers allow higher noise variance in specific pixels through diagonal covariance.
Model: A single component has d(l +2) free parameters, while a K-component mixture has K[d(l +2)+1] parameters.Here d is the data dimension and l is the latent dimension.
Likelihood: The log-likelihood of a set of N data points is formed by summing mixture probabilities across N observations and K components.The mixture coefficients are π_i, and each component uses parameters μ_i and Σ_i.
Efficient computation: Woodbury inversion and the matrix determinant lemma avoid storing or directly manipulating d × d covariance inverses and determinants.These identities reduce the required operations to expressions involving the diagonal noise and the lower-dimensional latent structure.
Efficient computation: The resulting log-likelihood computation has complexity linear in image dimension d, enabling efficient training on full-image datasets.The model is optimized with stochastic gradient descent on GPUs rather than EM.

4 Experiments

Experiments compare MFA-based Gaussian mixture models with GANs and VAEs on CelebA, MNIST, and SVHN, evaluating sample quality, distribution coverage, latent structure, and inference. MFA models generate realistic and diverse samples, capture bin proportions better than most GANs, and support explicit latent manipulation, outlier detection, and in-painting, although GAN samples are sharper.

Experimental setup: MFA models were compared with DCGAN, BEGAN, WGAN, WGAN-GP, VAE, and VAE-DFC on CelebA, SVHN, and MNIST.CelebA images were aligned, cropped, and resized to 64×64 pixels.
Sample quality: MFA samples were realistic and diverse but less sharp than GAN samples, and their test likelihood and reconstruction quality converged with relatively few components.Figure 4 shows samples trained on CelebA, MNIST, and SVHN.
Distribution evaluation: MFA achieved the best lowest NDB score for every tested number of components, remaining better than most GANs under alternative initializations.The evaluation used 20,000 generated samples, and results were consistent across runs and clustering choices.
Distribution evaluation: BEGAN and DCGAN showed clear mode collapse, while WGAN showed some bin-proportion distortion and WGAN-GP appeared to reduce it.The binning method exposed missing prototype categories, including semantic and physical image properties.
Latent structure: MFA latent components encoded interpretable image changes, with directions affecting facial hair, glasses, illumination, hair style, digit angle, and strokes.The latent variable controls combinations of learned column-vectors added to the component mean, and sampled linear combinations produce realistic images.
Inference: MFA provided closed-form outlier detection and in-painting without optimization or retraining, whereas GAN-based likelihood and in-painting require special-purpose approximations.Figure 7 shows low-likelihood CelebA images and reconstructions of missing regions from partially observed images.

5 Generating Sharp Images with GMMs

The paper explores two ways to make GMM samples sharper: increasing components or adding a pix2pix refinement model, while testing whether adversarial training changes the sharpness–distribution trade-off.

More components: Increasing the number of GMM components by 20 produces GAN-like sharpness of -4.0 but clearly overfits the training data.This motivates seeking sharper samples without simply increasing model complexity.
MFA+pix2pix: A pix2pix GAN is trained to refine MFA samples by adding fine details without modifying their global structure.Training pairs are formed by projecting each training image onto its most likely MFA component subspace and learning to recover the original image.
MFA+pix2pix: MFA+pix2pix samples reach sharpness -3.5, similar to the original dataset, but their NDB scores become worse.The refinement improves visual detail while reducing fidelity to the evaluated distribution.
Adversarial GMM: The adversarial GMM replaces the WGAN-GP generator with a mixture-based generator while leaving the discriminator and training procedure unchanged.Each component generates an output in parallel, and a one-hot mixture variable selects one component output.
Adversarial GMM: An adversarially trained GMM produces sharp, realistic samples with sharpness -3.8, slightly better than WGAN-GP.However, its NDB evaluation indicates mode collapse, and its data likelihood is much worse than with traditional maximum-likelihood training.

6 Conclusion

The conclusion compares GMMs and GANs as models of full images, emphasizing GMMs’ distributional coverage and inference advantages despite lower sample sharpness. It also cautions that GMMs are not presented as a final solution.

Main findings: GMMs can be efficiently trained on full-image datasets and generate realistic, though less sharp, samples than GANs.The comparison concerns models trained on the same datasets commonly used for GANs.
Main findings: Unlike GANs, GMMs capture the underlying image distribution and provide an explicit representation of its statistical structure.The conclusion presents this as the central advantage of GMMs in the comparison.
Scope: The authors do not claim that GMMs are the ultimate solution for modeling full images.They instead motivate more elaborate models that retain efficient inference and accurate statistical representation, even if samples are less visually attractive.
Evaluation: The NDB method and its Jensen-Shannon alternative evaluate how sample proportions across data-derived bins compare between a reference dataset and a model.The bins are constructed using K-means clustering of training data, while JS divergence provides a soft alternative when sample counts are sufficiently high.

A.2 MFA Training

The MFA training procedure uses K-means-based initialization, gradient-based likelihood optimization, and hierarchical decomposition for large mixtures; the trained model also supports closed-form image inference and sharpness measurement.

MFA training: By default, MFA components are initialized with K-means followed by FactorAnalysis performed separately within each cluster.Alternative initializations include random image subspaces and K-subspaces.
MFA training: MFA training uses Adam with learning rate 0.0001 to optimize negative log likelihood on mini-batches of 256 samples.Gradients with respect to model parameters are computed automatically by TensorFlow.
MFA training: Hierarchical training reduces time and memory requirements by first training Kroot components and then splitting them into sub-components based on assigned sample counts.Larger components receive more sub-components.
MFA inference: For in-painting, the model completes an unseen image by computing the most probable hidden part conditioned on observed pixels.The target is argmaxx1 P(X1 | X2 = x2), where X1 is hidden and X2 is observed.
Sharpness measurement: The sharpness score is the difference between log energies of high-pass filtered and original images, averaged over 2000 images.The score is invariant to multiplying all pixel values by a constant or adding a constant.

B Interpretation of the NDB Bins

NDB bins group images into data-derived clusters whose membership reflects combinations of semantic, physical, and photometric properties. The figures compare highly populated and sparsely populated bins across datasets.

Bin examples: The NDB figures show training images assigned to different bins for CelebA, MNIST, and SVHN.For each dataset, the displayed groups include the largest and smallest of 200 bins.
Interpretation: The bins implicitly correspond to joint combinations of semantic, physical, and photometric image properties.CelebA examples include glasses, hairstyle, hats, pose, skin shade, and image contrast.
Interpretation: A reliable generative model should represent the joint distribution of these observed-pixel properties rather than only deep semantic features.The paper connects this requirement to the risk that representation-based objectives may ignore properties treated as invariant by the underlying classifier.
CelebA: In the CelebA figures, the largest 15 bins are contrasted with the smallest 15 bins among 200 K-means bins.Figure 10 places a bin centroid first in each row, followed by random training samples from that bin.
MNIST and SVHN: The MNIST and SVHN figures likewise compare the largest and smallest 15 bins out of 200.These figures follow the same NDB K-means bin visualization used for CelebA.

C Additional NDB Results

Figures 14 and 15 compare binning proportions for CelebA across model evaluations and multiple bin counts, showing stable MFA proportions but substantial WGAN distortions.

Across 100, 200, and 300 bins, MFA samples remain similar to the reference training distribution and test samples.The same trained MFA and WGAN models are evaluated at each bin count.
WGAN exhibits significant distortions in its binning proportions across the evaluated bin counts.
Figure 15 shows bin-assignment histograms for 20,000 random CelebA test samples and samples from evaluated models at K=200.The histograms are presented in pairs for clarity.

D Additional MFA Samples

Figures 16–18 provide additional, non-cherry-picked random samples from MFA models trained on CelebA, MNIST, and SVHN.

Additional random samples are shown from the MFA model trained on CelebA.The samples are presented as further model outputs rather than selected examples.
Additional random samples are shown from the MFA model trained on MNIST.
Additional random samples are shown from the MFA model trained on SVHN.

E Internal Representation of the MFA Model

The paper visualizes MFA components, their learned directions and noise, and how representation quality changes with model size. It also contrasts MFA’s component-based structure with GAN transformations and illustrates MFA+pix2pix image refinement.

Internal representations: For a toy dataset in R2, MFA uses an internal component-based representation, while GANs learn an elaborate nonlinear transformation from latent space to data space.The GAN representation is illustrated by sampling z on a grid.
Learned MFA components: CelebA MFA components are visualized through mean images, noise variance, and rows of the learned scale matrix A.For each row, the displayed images are µ + A(i), 0.5 + A(i), and µ − A(i).
Representation quality versus model size: Increasing the number of MFA components rapidly improves both test-set log likelihood and reconstruction quality.Figure 21 plots test-set log likelihood and random test-image reconstructions for models with different component counts.
MFA+pix2pix: The MFA+pix2pix model maps MFA-generated images XA → XB toward the data manifold through a learned image transformation.MFA components occupy subspaces with added noise, while pix2pix performs the refinement step.