On Unifying Deep Generative Models

1 Introduction

• Deep generative models build distributions over layered variables, and the classic wake-sleep algorithm explored minimizing posterior and approximation KL divergences in opposite directions.The paper situates its connection in this earlier deep-generative-model learning framework.Source: §1 · p. 1
• GAN and VAE research had largely developed as distinct paradigms, leaving their principled connection unclear.GANs were framed around generator-discriminator equilibrium, while VAEs maximized a variational lower bound of data log-likelihood.Source: §1 · p. 1
• The formulation interprets GAN sample generation as posterior inference and connects GANs and VAEs through opposite-direction KL divergences, extending wake-sleep learning phases.It also interprets VAEs as containing a degenerated adversarial mechanism that blocks generated samples during training.Source: §1 · pp. 1–2
• The unified interpretation offers a tool for analyzing a broad class of GAN- and VAE-based variants, including InfoGAN, joint VAE/GAN models, and adversarial domain adaptation.The cited examples include models for hidden-representation inference, improved generation and reduced mode missing, and adaptation traditionally framed discriminatively.Source: §1 · p. 2
• The paper transfers techniques across GANs and VAEs, deriving IWGAN from importance-weighted VAE work and activating a VAE discriminator to use real and generated examples.The authors report that imported techniques are generally applicable to base models and variants, yielding consistently better performance.Source: §1 · p. 2

2 Related Work

• Wake-sleep uses separate wake and sleep phases to minimize KL divergences in opposite directions, whereas VAEs jointly optimize generative and inference models and importance-weighted VAEs tighten the lower bound.Prior VAE work also added an extended sleep procedure to exploit generated samples for learning.Source: §2 · p. 2
• GANs train a discriminator to separate real from generated samples and a generator to confuse it; the paper analyzes the widely used objective that maximizes the discriminator's probability of recognizing generated samples as real.The passage notes this objective shares the original objective's optimal solution and is used in practice in light of vanishing gradients with a strong discriminator.Source: §2 · p. 2
• Prior work combined GANs and VAEs for generation, generalized GAN objectives to other f-divergences, and applied adversarial methods to domain adaptation and Bayesian inference.The paper develops a general formulation intended to reveal connections across these approaches.Source: §2 · p. 3
• Compared with its conference version, this paper broadens the framework to more GAN and VAE variants and adds a more complete, consistent view and discussion of symmetric generation and inference.The presentation was also reorganized to clarify the theory development.Source: §2 · p. 3

3.1 Adversarial Domain Adaptation (ADA)

• ADA learns domain-invariant features by making source and target domains indistinguishable, and its formulation provides the framework in which GANs appear as a special case.For GANs, generated samples form one domain and real data the other.Source: §3.1 · p. 3, §3 · p. 3, §3.2 · p. 4

3.2.1 New Interpretation of GANs

• Under the new view, GAN generator learning minimizes KL from its inference distribution toward the posterior, offset by a negative JSD that pushes generated and real distributions apart.The JSD arises from the prior and is upper-bounded by the KL, but the analysis cautions that it should not be treated as negligible.Source: §3.2.1 · p. 6
• The asymmetric KL drives generated distributions toward discriminator-weighted mixtures, favoring large modes and sharp samples while missing smaller modes; this result does not require an optimal discriminator.When the discriminator gives uniform guesses or cannot distinguish matched real and generated distributions, the KL and JSD gradients cancel and generator learning stops.Source: Figure 2 · p. 5, §3.2.1 · p. 6

3.2.2 InfoGAN / 3.2.3 Adversarial Autoencoder (AAE) and CycleGAN

• InfoGAN adds code inference; AAE (Makhzani et al., 2015) swaps data and code roles, while combining AAE and InfoGAN objectives recovers bidirectional cycleGAN translation (Zhu et al., 2017).In InfoGAN, the added conditional qη(z|x, y) models code recovery; the two objectives train the opposing translations in cycleGAN.Source: §3.2.2 · pp. 6–7, §3.2.3 · p. 7

3.3 Variational Autoencoders (VAEs)

• VAEs and InfoGAN form a symmetric pair by swapping generative and inference processes; with a perfect discriminator, fake samples contribute only a constant loss and cannot aid VAE learning.The VAE formulation also reverses the GAN KL direction and uses a degenerated adversarial discriminator.Source: §3.3 · pp. 7–8

3.4 Connecting the Two Families of GANs and VAEs / 3.4.1 VAE/GAN Joint Models

• Opposite KL directions yield GAN mode-seeking sharpness versus VAE mode coverage and blur, motivating joint objectives; prior work adds GAN losses to VAEs for sharpness and VAE losses to GANs against missing modes.The framework also suggests activating the otherwise degenerated VAE adversary so generated samples can contribute to learning.Source: §3.4 · p. 9, §3.4.1 · p. 9

3.5 Connecting to Wake Sleep Algorithm (WS)

• The inverted latent-visible treatments of GANs and VAEs connect their learning to the complementary wake and sleep phases, which optimize generative and inference models, respectively.Source: §3 · p. 3, §3.4 · p. 9, §3.5 · p. 9

3.4.2 Implicit Variational Inference

• GANs’ implicit variational distribution avoids assuming a particular family such as Gaussians, while VAEs additionally constrain that distribution toward a prior, often through an adversarial loss as in AAE.Source: §3.4.2 · p. 9

3.5 Connecting to Wake Sleep Algorithm (WS)

• Wake-sleep trains the generator using real data and inferred codes in the wake phase, then updates the inference model using generated samples in the sleep phase.VAEs generalize the wake phase by also optimizing q_eta, with prior regularization on code z.Source: §3.5 · p. 10
• GANs generalize the sleep phase by optimizing p_theta to reconstruct reversed y, while InfoGAN further extends reconstruction to code z.The sleep-phase objective corresponds to optimizing discriminator q_phi to reconstruct y from sample x.Source: §3.5 · p. 10

3.6 The Relation Graph

• The unified formulation connects GANs and VAEs to variational EM and wake-sleep algorithms, encompassing a broad range of model variants and extensions.The relation graph presents a unified statistical view of deep generative modeling.Source: §3.6 · p. 10
• Treating GAN sample generation as posterior inference makes inference and generation a symmetric pair, represented by the triangular relation in Figure 4.The paper identifies this symmetric treatment as a key idea in its formulation.Source: §3.6 · p. 10

4.1 Importance Weighted GANs (IWGAN)

• IW-GAN applies importance weighting from IWAE to optimize a tighter lower bound on marginal likelihood, using a k-sample estimate with tractability approximations.The update adds importance weights to the generator learning rule.Source: §4.1 · pp. 10, 12
• IW-GAN assigns higher weights to more realistic samples that fool the discriminator, with negligible added computation from evaluating each sample's weight.The discriminator is trained as in standard GANs.Source: §4.1 · p. 12

4.2 Adversary Activated VAEs (AAVAE)

• AA-VAE activates a discriminator so generated samples that resemble real data and fool it can contribute to training, while weighting real examples emphasizes harder cases.A real example's larger weight indicates it is harder for the generator to simulate.Source: §4.2 · p. 12

5.1 Importance Weighted GANs

• Vanilla and class-conditional GANs use DCGAN settings and are evaluated on MNIST, SVHN, and CIFAR10, with inception scores or conditional-generation accuracy.The IW hyperparameters are not tuned.Source: §5.1 · p. 13
• IW extensions consistently improve the base vanilla GAN and conditional GAN models in the reported evaluations.Table 2 reports inception scores for GANs and classification accuracy for conditional GANs.Source: §5.1 · p. 13

5.2 Adversary Activated VAEs

• AA is tested with vanilla, class-conditional, and semi-supervised VAEs on MNIST, varying the amount of real training data and adding an equal number of generated samples per real batch.Source: §5.2 · p. 13
• Activating the adversarial mechanism in VAEs generally yields larger improvements with smaller sets of real training data; AA-SVAE also improves accuracy over the base semi-supervised VAE.Source: §5.2 · p. 13

• Treating GAN sample generation as posterior inference is a key part of the proposed connection between GANs and VAEs.Source: §6 · p. 13
• The framework treats visible and latent variables, and generation and inference, as symmetric pairs rather than requiring a sharp boundary between them.It treats GAN generation space x as latent, relating generation to variational inference and adversarial domain adaptation.Source: §6 · p. 13
• Empirical data distributions over visible variables are typically easy to sample from but have intractable likelihoods, whereas latent priors are usually explicit and likelihood-evaluable.Adversarial loss and related tools can bridge this difference, including by replacing an intractable KL loss in implicit variational inference.Source: §6 · pp. 13–14
• Visible-space complexity and simpler latent spaces guide the choice of distribution-matching tools; VAEs use KL divergence, while adversarial autoencoders use adversarial loss.Source: §6 · p. 14
• The same symmetry extends to data-label pairs: conditional generators create data from labels, while discriminative approaches can generate labels from data.With black-box mappings between the spaces, the paper regards the two approaches as essentially the same.Source: §6 · p. 14

• The proof obtains the desired result by differentiating Eq. (24) with respect to θ at θ0 and substituting the last equation of Eq. (29).Source: Proof · p. 17

B Proof of JSD Upper Bound in Lemma 1

• The proof establishes that the JSD term is upper bounded by the KL term.This is the stated result of Lemma 1.Source: B Proof of JSD Upper Bound in Lemma 1 · p. 17

C Schematic Graphical Models and AAE/PM/CycleGAN

• Swapping the code and data variables in InfoGAN’s graphical model yields the AAE model and its original objectives.The derivation is also presented as an example of translating a graphical model into mathematical objectives.Source: C Schematic Graphical Models and… · p. 18, Figure 6 · p. 18
• In the conventional AAE view, the discriminator distinguishes prior-sampled codes from codes inferred from observed data, while the encoder and generator reconstruct data and fool the discriminator.The discriminator predicts whether a code came from the prior or was inferred from an observed example.Source: C Schematic Graphical Models and… · p. 19
• AAE resembles Predictability Minimization because its discriminator can be viewed as a special form of code predictor.Source: C Schematic Graphical Models and… · p. 19

D Proof of Lemme 2

• Combining the reconstruction and prior-regularization terms recovers the conventional VAE objective.Source: D Proof of Lemme 2 · p. 19

E Importance Weighted GANs (IWGAN)

• Importance weighting, as in IWAE, gives a tighter lower bound for GANs, with the original bound recovered at k = 1.Source: E Importance Weighted GANs (IWGAN) · p. 20
• For k > 1, IWGAN follows standard GANs by removing the second gradient term even though that term is not removed by the derived bound.The passage identifies the removed term as the k = 1 gradient of the JSD.Source: E Importance Weighted GANs (IWGAN) · p. 20

F Adversary Activated VAEs (AAVAE)

• Adversary Activated VAEs replace the perfect discriminator with a learnable one, adaptively incorporating fake samples into model learning.The proposed objective is symmetric to InfoGAN’s objective with an additional KL prior regularization.Source: F Adversary Activated VAEs (AAVAE) · p. 21

G Experiments

• Vanilla GANs and their importance-weighted extensions use the DCGAN architecture and are evaluated on MNIST, SVHN, and CIFAR10.The importance-weighted extensions’ hyperparameters are not tuned.Source: §G.1 · p. 21
• Vanilla GAN sample quality is measured by inception scores, using evaluation networks with test-set scores of 9.09 on MNIST, 6.55 on SVHN, and 8.77 on CIFAR10.Source: §G.1 · p. 21
• Conditional GANs are evaluated by generated-label accuracy, with classifier test accuracy of 0.990 on MNIST and 0.902 on SVHN.The evaluation compares classifier predictions for generated samples with their conditioning labels.Source: §G.1 · p. 21

G.2 Adversary Activated VAEs

• The experiments apply adversary activation to vanilla VAEs, class-conditional VAEs, and semi-supervised VAEs on MNIST.The generators, inference networks, discriminators, and SVAE classifier use architectures described in the GAN experiments, with sigmoid output activations for Bernoulli pixel means.Source: §G.2 · p. 21
• The test lower bound is evaluated across different numbers of real training examples, with an equal number of generated fake samples per real-data minibatch.Source: §G.2 · p. 21
• Discriminator-output temperature is cross-validated over {1, 1.5, 3, 5}; smoothing is generally helpful, and other extension hyperparameters are not tuned.Table 4 reports average SVAE and AA-SVAE classification accuracy and standard deviations over five runs on the MNIST test set, varying the number of real labeled examples.Source: §G.2 · p. 21, Table 4 · p. 22