Source-linked AI summary
An Introduction to Deep Generative Modeling
Lars Ruthotto, Eldad Haber
TL;DR
Deep generative models promise likelihood estimation and sample generation but remain difficult to design, train, and evaluate. This paper gives a mathematical introduction to normalizing flows, variational autoencoders, and generative adversarial networks, comparing their supported assumptions and challenges. Its discussion identifies approach-specific advantages alongside limitations such as unstable GAN optimization and limited incorporation of domain knowledge.
Problem
Designing and training deep generative models remains difficult because distribution identification, similarity measurement, latent inference, and architecture selection pose mathematical challenges.
Method
The paper develops a concise mathematical framework for normalizing flows, variational autoencoders, and generative adversarial networks, illustrating them with numerical examples and optimal-transport connections.
Results
Normalizing flows directly compute likelihoods under invertibility, VAEs handle non-invertible generators and arbitrary latent dimensions, and VAEs were found less complicated to train than GANs.
Takeaways & Limitations
The three approaches make different assumptions and expose distinct trade-offs in invertibility, latent dimensionality, likelihood evaluation, and training stability.
Takeaways & Limitations
GAN effectiveness is almost impossible to predict a priori because generator and discriminator expressiveness is limited by parameterization, while existing approaches lack domain-specific knowledge.
Abstract
from arXiv · showhide
Deep generative models (DGM) are neural networks with many hidden layers trained to approximate complicated, high-dimensional probability distributions using a large number of samples. When trained successfully, we can use the DGMs to estimate the likelihood of each observation and to create new samples from the underlying distribution. Developing DGMs has become one of the most hotly researched fields in artificial intelligence in recent years. The literature on DGMs has become vast and is growing rapidly. Some advances have even reached the public sphere, for example, the recent successes in generating realistic-looking images, voices, or movies; so-called deep fakes. Despite these successes, several mathematical and practical issues limit the broader use of DGMs: given a specific dataset, it remains challenging to design and train a DGM and even more challenging to find out why a particular model is or is not effective. To help advance the theoretical understanding of DGMs, we introduce DGMs and provide a concise mathematical framework for modeling the three most popular approaches: normalizing flows (NF), variational autoencoders (VAE), and generative adversarial networks (GAN). We illustrate the advantages and disadvantages of these basic approaches using numerical experiments. Our goal is to enable and motivate the reader to contribute to this proliferating research area. Our presentation also emphasizes relations between generative modeling and optimal transport.
1 Motivation
Deep generative modeling has broad applications but remains mathematically and practically difficult to design, train, and evaluate. This paper offers a mathematical introduction to three major approaches while highlighting open challenges and connections to applied mathematics.
- Recent applications, including deep fakes and scientific uses, have increased research activity while also raising societal, legal, and technical challenges.
- The paper prioritizes mathematical accessibility and clarity over generality while introducing three approaches and exposing open challenges rather than providing a comprehensive review.
- Deep generative models approximate complicated, high-dimensional probability distributions and can estimate likelihoods or generate new samples when successfully trained.
- Training is ill-posed because finite samples cannot uniquely identify a probability distribution, making performance highly dependent on architecture, objectives, regularization, and algorithms.
- Training also requires comparing generated and observed distributions or inverting the generator, both of which present distinct challenges.
- Latent-space dimension is difficult and important to choose: underestimation harms approximation, whereas overestimation can make the generator non-injective.
2 Mathematical Formulation and Examples
The paper formulates generative modeling as learning a complicated distribution from a simpler latent distribution, then uses examples to show how dimensionality, invertibility, and posterior inference shape training. It introduces normalizing flows and variational autoencoders as distinct solutions to these challenges.
- 2.1 General Set Up: Generative modeling learns an intractable distribution X over R^n from samples by mapping a tractable latent distribution Z in R^q through a generator.
- 2.1 General Set Up: The latent distribution is typically Gaussian, but its dimension q may differ from data dimension n because data can lie on an unknown lower-dimensional manifold.
- 2.1 General Set Up: Sampling z and computing g(z) generates new data, while likelihood evaluation generally requires an intractable high-dimensional marginalization.
- 2.1 General Set Up: Neural-network generators make high-dimensional function approximation feasible, but selecting an effective architecture remains critical and lacks theoretical guidelines.
- 2.2 Testbed Examples: The moons experiment uses two disjoint half-moon clusters and a two-dimensional latent space, providing a visual testbed for comparing generative-modeling approaches.
- 2.2 Testbed Examples: MNIST supplies 28×28 grayscale digit images, with 60,000 training and 10,000 test examples, and an unknown intrinsic dimension below 784 pixels.
- 2.2 Testbed Examples: The MNIST example trains a model from a two-dimensional standard normal distribution to realistic-looking images despite the unknown intrinsic dimension.
- 2.3 Training the Generator: A High-level Overview: Variational autoencoders handle non-invertible generators and arbitrary latent dimensions using a second network that approximates the posterior and yields a likelihood lower bound.
3 Finite and Continuous Normalizing Flows
Normalizing flows model data through invertible transformations, enabling likelihood evaluation and sampling when both directions are computationally tractable. Finite flows trade expressive layers against tractable Jacobians, while continuous flows use transport-cost regularization to obtain nearly straight characteristics and accurate inverses.
- 3 Finite and Continuous Normalizing Flows: Normalizing flows assume a diffeomorphic, orientation-preserving generator with equal latent and data dimensions, enabling likelihood computation through the change-of-variables formula.The orientation-preserving condition requires det ∇g(z) > 0.
- 3.1 Maximum Likelihood Training: Maximum likelihood training minimizes negative log-likelihood, which is equivalent to minimizing KL(pX || pθ) using samples from X despite the unknown true density.The KL divergence is asymmetric, and optimizing the reverse direction is intractable because it requires the true density.
- 3.1 Maximum Likelihood Training: KL asymmetry encourages generator support to cover observed samples but can permit generated samples with small likelihood under the true distribution.This follows because the objective penalizes pX(x) ≫ pθ(x) more strongly than pX(x) ≪ pθ(x).
- 3.2 Finite Normalizing Flows: Finite flows concatenate transformations with tractable Jacobian determinants, but efficient sampling additionally requires efficient forward evaluation alongside inversion.The central design trade-off is expressive layers versus tractable Jacobians and comparable forward and inverse costs.
- 3.2 Finite Normalizing Flows: Real NVP transforms standard-normal samples toward the moons data, but its smoothness makes the two half-moons appear connected.Alternating layers limit each transformation to one coordinate, while higher-dimensional applications require sufficient depth for full coupling.
- 3.3 Continuous Normalizing Flows: Continuous normalizing flows regularized by transport costs produce meaningful samples and density estimates while making characteristics nearly straight and inverse error nearly machine-precision.The regularized formulation is viewed as a relaxed dynamic optimal-transport problem, and its solution has straight characteristics.
4 Variational Autoencoders
Variational autoencoders replace an intractable posterior with a tractable, learned approximation and optimize the resulting evidence lower bound. The approach balances reconstruction quality against latent-space regularity, but Gaussian approximations can be inaccurate and produce blurred or incorrect samples.
- Posterior approximation: VAEs use a tractable neural-network posterior approximation eψ(z|x) because the true posterior pθ(z|x) is generally intractable for nonlinear, non-invertible generators.The approximate posterior is probabilistic and can be sampled from and evaluated efficiently.
- Evidence lower bound: The ELBO replaces direct likelihood maximization with a tractable surrogate that lower-bounds the evidence by dropping the non-negative KL divergence to the true posterior.The bound becomes tight only when the approximate and true posteriors coincide.
- Limitations: The tightness of the ELBO cannot be determined in practice because both the true posterior pθ(z|x) and evidence pθ(x) are intractable.More expressive non-Gaussian approximate posteriors, including continuous normalizing flows, are suggested as a possible improvement.
- Objective: ELBO minimization combines expected reconstruction error with a KL regularizer that biases approximate posteriors toward the latent distribution.The reconstruction term addresses data-space error, while the regularizer favors latent samples near the prior distribution.
- Objective: Reconstruction and latent regularity create a conflict: optimizing reconstruction alone yields non-normal latent samples and can make prior-sampled generations low quality.The likelihood function is used to balance these competing objectives, although Bernoulli likelihoods offer no obvious balancing parameter.
- Numerical examples: In MNIST experiments, Gaussian approximate posteriors differed substantially from true posteriors, causing substantial reconstruction error for some images and blurred generated digits overall.The approximate posteriors also failed to overlap collectively with a Gaussian, while some latent class clustering emerged without labels.
5 Generative Adversarial Networks
GANs train a generator against a discriminator that distinguishes real from generated samples, but optimization, mode collapse, and evaluation remain difficult. In the MNIST experiments, WGAN outperforms DCGAN by the multivariate ε test, while generated images can still contain recognizable failures.
- GAN formulation: GANs compare generated and data-space distributions without using sample likelihoods, training generator weights against a distance between gθ(Z) and X.This likelihood-free formulation differs from normalizing flows and VAEs.
- GAN formulation: The discriminator is a binary classifier trained to predict whether an example came from the training data or the generator.It aims for dφ(x) ≈ 1 on data samples and dφ(x) ≈ 0 on generated samples.
- Optimization: GAN training alternates discriminator and generator updates toward a Nash equilibrium, requiring a balance because an optimal discriminator can leave the generator with near-zero gradients.Insufficient discriminator training also makes generator updates challenging.
- Failure modes: Mode collapse can map nearly all latent samples to one data point, and less extreme forms may be almost impossible to detect from finitely many generated samples.Proposed heuristics include minibatch-statistics distance terms and one-sided label smoothing.
- Numerical comparison: GAN samples can be sharp and realistic, yet many generated MNIST images contain no handwritten digit and are readily recognized as fakes.The statistical test may disagree with human assessment, and GANs lack likelihood or latent-space information.
- Numerical comparison: WGAN outperforms DCGAN on the multivariate ε test under both random and VAE initialization.VAE initialization produces a quicker and overall larger reduction in this metric.
6 Discussion of the three main approaches
The three approaches trade off likelihood access, latent-space flexibility, and training difficulty. Their practical effectiveness depends on assumptions, hyperparameter choices, and unresolved challenges in sampling and distribution comparison.
- Normalizing flows: Normalizing flows directly optimize data likelihood and monitor latent representations, while optimal-transport regularization can improve training accuracy and efficiency.These benefits rely on invertible, smooth generators and matching intrinsic and ambient dimensions.
- Normalizing flows: Normalizing flows are limited when their smooth invertibility and dimensionality assumptions are violated, making poor results difficult to diagnose.Suboptimal performance may reflect training, modeling, or violated assumptions and can motivate VAE or GAN methods.
- Variational autoencoders: VAEs handle non-invertible generators and arbitrary latent dimensions, while reconstruction error can provide information about whether the latent space is large enough.Their objective supplies a likelihood lower bound that tightens as the approximate posterior approaches the generator-implied posterior.
- Variational autoencoders: VAE training is less complicated than GAN training because it minimizes a loss and permits monitoring through latent-space and reconstruction diagnostics.GANs instead require solving a saddle-point problem, while VAEs require training an additional approximate-posterior network.
- Variational autoencoders: VAEs may sample from a latent distribution unlike the approximate posterior used during training, potentially giving the generator inputs outside its training distribution.In the example, the sampled inputs were generally not normally distributed despite KL-divergence penalization.
- Generative adversarial networks: GANs compare generated and dataset samples through a discriminator rather than estimating likelihoods or latent variables, but training remains difficult.WGAN performed slightly better than binary-classification GAN in the example, while high-dimensional Wasserstein approximation and Lipschitz enforcement remain challenging.
7 Outlook
Future work centers on making high-dimensional distribution comparison more reliable and efficient while narrowing the gap between theory and practice. The paper also identifies limited incorporation of domain-specific knowledge as a significant limitation, especially for scientific applications.
- Future research: Reliable and efficient comparison of complicated, high-dimensional probability distributions is a central future challenge for deep generative modeling.The paper connects progress on this problem with improving training reliability and reducing computational costs.
- Future research: Closing the gap between theory and practice is identified as critical for improving DGM training reliability and reducing computational costs.The paper highlights VAE sampling and WGAN Lipschitz enforcement as demonstrated examples of these challenges.
- Future research: Most existing approaches use black-box neural-network generators, leaving limited support for incorporating domain-specific knowledge in scientific applications.The paper characterizes this lack as a significant limitation.