Distribution Matching in Variational Inference

1 INTRODUCTION

• The paper asks whether GAN-based hybrids can address VAE limitations, including failures in marginal matching and difficulties with explicit distributions.It contrasts conditional distribution matching in VAEs with marginal matching in VAE-GAN hybrids and compares explicit with implicit distributions.Source: §1 · p. 1
• VAEs fail to match marginal distributions in latent and visible space, and stronger explicit model distributions and posteriors do not resolve the problem.The paper reports this pattern across datasets, models, and latent dimensionalities.Source: §1 · p. 1
• VAE-GAN hybrids are harder than VAEs to scale, evaluate, and use for inference, and do not improve sample generation quality over GANs.The paper attributes these constraints to their use of classifier probabilities.Source: §1 · p. 2

2 CHALLENGES WITH VAES

• The ideal visible-space objective matches the model distribution pθ(x) to the data distribution p∗(x), while variational inference pursues this indirectly through conditional matching in latent space.Variational inference uses a tractable variational distribution and maximizes the evidence lower bound (ELBO).Source: §2 · p. 2
• Minimizing the expected KL divergence between qη(z|x) and p(z) minimizes a lower bound on KL[qη(z)||p(z)], connecting conditional training to marginal latent matching.The integral in Equation (5) is nonnegative mutual information.Source: §2 · p. 2
• The section identifies failures to match qη(z) to p(z) and pθ(x) to p∗(x), associated with conditional matching and explicit distributions.It proposes exploring marginal matching and implicit models as ways to address these failures.Source: §2 · p. 2
• The marginal divergence KL[qη(z)||p(z)] is presented as a VAE performance metric because it captures posterior matching and data-distribution learning together.The section situates this analysis alongside rate-distortion work that does not examine implicit distributions or alternative distribution-matching approaches.Source: §2 · p. 2

3.1 Effect of latent posterior distribution

• Powerful RNVP posteriors do not improve latent marginal matching over simple diagonal Gaussian posteriors in VAEs.The reported comparison uses KL[qη(z)||p(z)] for trained VAEs; the result suggests posterior distributions may not be the largest training bottleneck.Source: §3.1 · p. 3

3.2 Effect of the visible distribution

• KL[qη(z)||p(z)] is 44.7 for a Bernoulli model and 256.7 for the same model with a Quantized Normal visible distribution.The Quantized Normal has greater modeling capacity and improves reconstructions, but the paper reports no corresponding increase in sample quality and a larger sample–reconstruction gap.Source: §3.2 · p. 3

3.3 Low probability posterior samples

• Low-posterior samples reveal prior regions not covered by qη(z), and can look unlike typical data, including thickened lines or strong white backgrounds.The samples' latent locations are scattered through the prior rather than isolated in particular regions.Source: §3.3 · p. 3
• For CIFAR-10 and CelebA, VAEs assign higher likelihood to low-posterior samples than to data, supporting a systematic failure to match latent and visible distributions.On ColorMNIST, by contrast, the model assigns lower likelihood to the low-posterior samples and their nearest neighbors than to data.Source: §3.3 · p. 3

4.1 Distribution matching with density ratios

• The density-ratio trick estimates ratios from samples without requiring explicit distribution forms, enabling GANs to match data and model marginals.In variational inference, it opens the way to implicit posteriors and marginal latent matching, while joint training uses alternating gradient descent.Source: §4.1 · pp. 3–4
• With 160 latents across models, KL[qη(z)||p(z)] is higher on more complex datasets and reflects data complexity.The figure reports marginal KL and mutual information, which together make up the ELBO KL term.Source: Figure 1 · p. 4

Marginal distribution matching.

• Adversarial Autoencoders directly minimize KL[qη(z)||p(z)] through adversarial training, omitting the ELBO mutual-information term.The paper also connects this approach to Wasserstein distance.Source: Marginal distribution matching · p. 5

4.2 Effects of density ratios on evaluation, scalability and latent representations

• Density-ratio estimates underestimate KL[qη(z)||p(z)], leaving VAE-GAN hybrids without a meaningful likelihood bound or convergence measure.The underestimation occurs even when the discriminator is trained to optimality.Source: Figure 4 · p. 5, §4.2 · p. 5
• Classifier-based learning does not reliably scale with data dimensionality; AAEs struggle on ColorMNIST and fail on CelebA with 1,000 or 10,000 latents.The CelebA failure occurs even with a larger discriminator.Source: §4.2 · p. 5
• AAEs learn dense latent representations, whereas VAEs use only a few latents and ignore the rest.The paper attributes this difference to marginal matching removing the conditional posterior KL cost; for Gaussian posteriors, variance collapses to zero.Source: §4.2 · p. 5

5 GANS IN VISIBLE SPACE

• Implicit data models avoid blurry samples and reconstructions associated with common explicit Gaussian or Laplacian likelihoods.The section motivates this as a reason to introduce marginal matching in visible space.Source: §5 · p. 6
• Replacing the likelihood term with a density ratio requires infinitely many discriminators and removes gradients to the parametric model in the naive formulation.The ratio estimator absorbs the model by receiving its samples.Source: §5 · p. 6

5.1 Marginal distribution matching

• The visible-space approach trains a ratio estimator to distinguish data from marginal reconstructions, then uses an adversarial loss to make reconstructions realistic.It can be combined with conditional or marginal latent matching and supports implicit or explicit models.Source: §5.1 · p. 6
• For explicit models, reconstruction matching can regularize GAN training against mode collapse, while adversarial matching can steer reconstructions toward realistic regions.The paper describes these as reciprocal regularizing roles.Source: §5.1 · p. 6
• Applying adversarial loss to both reconstructions and samples lets the decoder produce compelling samples early while retaining inference through qη(z|x).The paper reports improved sample quality over applying adversarial loss only to reconstructions.Source: §5.1 · p. 7

5.2 Joint variants

• VEEGAN uses a reconstructor and a variational lower bound to match joint distributions, but its density-ratio estimate also loses an estimate of the bound.Its objective uses a latent-space reconstruction loss, and decoder gradients follow from an expectation under the model joint distribution.Source: §5.2 · p. 7

• The comparison includes diagonal-Gaussian VAEs, DCGAN and WGAN-GP baselines, AAEs, VEEGAN, and the proposed VGH and VGH++ hybrids.VGH and VGH++ differ in whether the visible-space discriminator is trained on reconstructions or on both data and samples.Source: §6 · p. 7
• Models are trained on ColorMNIST, CelebA, and CIFAR-10, with visual inspection and Inception Score, sample diversity, and an independent Wasserstein critic used for evaluation.The ELBO is excluded because the authors say VAE-GAN hybrids cannot estimate it reliably.Source: §6 · p. 7
• The experiments found no clear benefit from merging variational inference with implicit and marginal distribution matching.Source: §6 · p. 7
• Figure 8 reports CelebA sample diversity and CIFAR-10 Inception scores, while Figure 9 compares negative Wasserstein distance.The diversity measure is judged relative to the test set; higher Inception scores and higher negative Wasserstein distance are better.Source: Figure 8 · p. 8, Figure 9 · p. 8

7 CONCLUSION

• The conclusion frames the experiments as testing whether conditional matching and explicit distributions explain VAE marginal-distribution failures, using existing and new VAE-GAN hybrids to explore alternatives.Source: §7 · p. 8
• VAE-GAN hybrids do not resolve major variational-inference challenges; classifier-based divergence estimates, scaling difficulties, hyperparameter sensitivity, and struggles to outperform GAN sample quality limit their present applicability.Source: §7 · p. 8

A Low posterior VAE samples

• The appendix generates low-posterior samples by ranking prior-sampled latents by estimated marginal posterior probability, then decoding the n lowest-ranked latents.For each prior sample, the algorithm estimates its marginal posterior using posterior probabilities across the dataset.Source: A Low posterior VAE samples · p. 11

A.1 Low posterior samples - model analysis

• Low-posterior samples and their nearest neighbors are atypical; nearby data have higher KL cost, and on CIFAR-10 and CelebA the model assigns low-posterior samples higher likelihood than data.The authors interpret this as the model failing to capture data subtleties and being fooled into assigning high likelihood to samples with low probability under the true distribution.Source: §A.1 · p. 11, Figure 12 · p. 14, Figure 13 · p. 14

B Scaling up latent spaces in VAEs and AAEs

• At latent size 10,000, VAEs scale better than AAEs.Source: B Scaling up latent spaces in VAEs and… · p. 11

C Connection to Wasserstein Autoencoders

• Under stated conditions, optimal-transport and maximum-likelihood objectives yield encoder-decoder criteria, but Wasserstein autoencoders require a Dirac decoder and practitioner-chosen reconstruction cost.Unlike density-ratio approaches, VAEs retain a meaningful quantity to track through the closed-form KL divergence.Source: C Connection to Wasserstein Autoencoders · p. 16

D Learning with density ratios: synthetic data experiments

• The experiments separate fixed-distribution divergence estimation from joint distribution minimization, tracking true KL alongside classifier-estimated divergence during learning.The authors average over 10 runs while controlling hyperparameters, classifier architectures, and Gaussian random seeds.Source: D Learning with density ratios:… · p. 17
• At dimensions 100 and 1,000, a 100-million-parameter classifier fails to provide useful gradients and learning diverges; density-ratio estimates are inaccurate regardless of dimensionality.At dimensions 1 and 10, the model can decrease the true KL divergence.Source: D Learning with density ratios:… · p. 17

E.1 Estimating KL[qη(z)||p(z)] using the Monte Carlo approach

• The appendix estimates marginal KL by Monte Carlo, sampling from the data and conditional posterior and averaging posterior probabilities across the dataset for each latent sample.The authors describe this approach as computationally expensive but the most accurate one.Source: §E.1 · p. 17

E.2 Estimating KL[qη(z)||p(z)] using the density ratio trick

• Density-ratio KL estimates are highly sensitive to hyperparameters, partly because classifier probabilities can be inaccurate and q(z) can be too complex to distinguish.The authors leave exploration of improved probability-estimation methods for future work.Source: §E.2 · p. 17
• More frequent discriminator updates make models converge earlier for d = 1, 10, and 100, but do not improve estimated KL, including at d = 1000.A higher discriminator learning rate makes training more unstable and prevents convergence at d = 100.Source: Figure 19 · p. 18, Figure 20 · p. 19

F VGH++ loss function and pseudocode

• The density-model approach learns q(z) with a fitted density estimator, while VGH and VGH++ train data and code discriminators using cross-entropy losses.The supplied passages identify the VGH++ loss and training procedure but do not specify their full equations or updates.Source: §E.3 · p. 19, F VGH++ loss function and pseudocode · p. 20

E.3 Estimating KL[qη(z)||p(z)] using a density model for q(z)

• GMM, MAF, and autoregressive density models all underestimate the true marginal KL, while LSTMs outperform the other models.The authors interpret the failure to reach the theoretical bound as evidence that qη(z) is complex.Source: §E.3 · p. 19

G The effect of the visible distribution in VAE training

• Using a Bernoulli visible distribution spreads mass around each pixel like a unit-variance Gaussian, limiting its ability to specialize for good reconstructions.The authors state that QuantizedNormal has higher modeling capacity and produces better reconstructions.Source: G The effect of the visible… · p. 20

H Tracking the variational lower bound - VEEGAN

• VEEGAN's bound does not increase reliably during training, so variational hybrids do not supply a quantity for assessing convergence.The bound uses an implicit p(x|z) built by Fθ and estimates KL with the density-ratio trick.Source: H Tracking the variational lower bound… · p. 20

I Real data evaluation metrics

• The evaluation uses multiple metrics because no universal generative-model metric captures overfitting, sample quality, and diversity together.The independent Wasserstein critic assesses sample quality and can assess overfitting when trained on validation data; Inception Score does not capture within-class mode collapse or overfitting.Source: I Real data evaluation metrics · pp. 20–21

J AdversarialVB results

• The AdversarialVB results are presented in Figure 24, but the supplied passages do not report a specific outcome.Source: J AdversarialVB results · p. 21

K Training details: hyperparameters and network architectures

• Models use a fixed learning rate, Adam with β1 = 0.5 and β2 = 0.9, batch size 64, and batch normalization.ColorMNIST models are trained for 100000 iterations, while CelebA and CIFAR-10 models are trained for 200000 iterations.Source: K Training details: hyperparameters… · p. 21

Updates

• WGAN-GP uses five discriminator updates per generator update; other models use equal update counts for their components.Source: Updates · p. 22

Choice of latent prior

• All models use a univariate normal prior.Source: Choice of latent prior · p. 22

K.1 Network architectures

• The compared models share generator and discriminator architectures and controlled latent counts; the encoder is convolutional, and the default VAE uses Bernoulli visible and Gaussian latent distributions.The code discriminator is a three-layer MLP with 1000 units per layer; 5000 units did not substantially improve results, and strong code-discriminator gradients can affect encoder reconstruction.Source: §K.1 · p. 22

M Model samples for real data experiments

• GMM, MAF, and LSTM density models underestimate the true marginal KL, while LSTMs outperform the other models.Source: Figure 21 · p. 24
• 44.7 for the Bernoulli model and 256.7 for the QuantizedNormal model are the reported KL values for the displayed VAE comparison.The figure caption reports a reconstruction–sample quality trade-off across hyperparameters.Source: Figure 22 · p. 24
• VEEGAN training curves do not show the expected increase in the bound, although the models perform comparably with state of the art.The plotted numbers are not directly comparable because VEEGAN's bound is not obtained on observed data.Source: Figure 23 · p. 25

L Reconstructions

• On CelebA and CIFAR-10, the reconstruction figures compare standard VAE, Adversarial Autoencoder, and VGH++ outputs against data.Source: L Reconstructions · p. 23, Figure 25 · p. 26, Figure 26 · p. 27

M Model samples for real data experiments

• The sample figures cover CelebA, CIFAR-10, and ColorMNIST.Source: M Model samples for real data… · p. 23, Figure 27 · p. 28, Figure 28 · p. 29