Source-linked AI summary
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
Sam Bond-Taylor, Adam Leach, Yang Long, Chris G. Willcocks
TL;DR
Deep generative modelling lacks a single approach that simultaneously resolves trade-offs in runtime, diversity, density evaluation, and architectural restrictions. This survey compares major model families and hybrids, explains their design choices and connections, and reviews advances showing that the gap between approaches is shrinking while complexity and runtime remain important constraints.
Problem
Research on deep generative models is fragmented across interconnected approaches with different trade-offs in runtime, diversity, density evaluation, and architectural restrictions.
Method
The paper conducts a broad comparative survey of energy-based models, VAEs, GANs, autoregressive models, normalizing flows, and hybrid approaches, emphasizing modelling decisions and recent implementations.
Results
The gap between GANs and other approaches is shrinking, while improved variational bounds and data augmentation benefit multiple model families.
Takeaways & Limitations
Hybrid models can balance extremes across generative-model families, but added model complexity hinders broader adoption.
Takeaways & Limitations
Coupling and autoregressive flows have restricted triangular Jacobians, limiting interactions among inputs and motivating permutations, convolutions, and related extensions.
Abstract
from arXiv · showhide
Deep generative models are a class of techniques that train deep neural networks to model the distribution of training samples. Research has fragmented into various interconnected approaches, each of which make trade-offs including run-time, diversity, and architectural restrictions. In particular, this compendium covers energy-based models, variational autoencoders, generative adversarial networks, autoregressive models, normalizing flows, in addition to numerous hybrid approaches. These techniques are compared and contrasted, explaining the premises behind each and how they are interrelated, while reviewing current state-of-the-art advances and implementations.
1 INTRODUCTION
This survey introduces deep generative modelling as learning data distributions without supervision, then compares interconnected model families and their trade-offs. It emphasizes applications, image models, and current advances while explaining how modelling choices affect density, speed, quality, and architecture.
- Motivation: Deep generative models learn samples from the training-data distribution without supervision, supporting applications across images, video, audio, text, reinforcement learning, graphics, and medicine.The paper motivates unsupervised learning through lower data-collection cost than labelled learning.
- Comparative framework: Generative models trade off execution time, architectural restrictions, density tractability, and proxy objectives, with direct likelihood sometimes producing worse sample quality.The survey notes that training-speed rankings depend on available compute and year, so such comparisons are indicative.
- Related learning paradigms: Generative modelling overlaps with self-supervised learning because both can learn without supervision, although self-supervised models are not necessarily generative.Examples of self-supervised objectives include auxiliary classification, masking, and contrastive losses.
- Scope: The survey compares energy-based models, VAEs, GANs, autoregressive models, and normalizing flows, while treating hybrids in the most relevant sections or across multiple sections.It also explains the modelling decisions behind each family and their interrelationships.
- Comparative framework: Table 1 compares model families using speed, parameter efficiency, sample quality, diversity, high-resolution scalability, FID, and NLL in BPD on CIFAR-10.The star ratings are defined separately in Table 2.
2 ENERGY-BASED MODELS
Energy-based models define distributions through energy functions, but normalization and sampling create scalability challenges. The section follows architectural restrictions and alternative sampling or score-based methods that address these difficulties, while retaining important trade-offs.
- Energy-based formulation: Energy-based models assign low energy to realistic data and high energy to unrealistic data, expressing probability densities through an energy function.This formulation avoids requiring a prior assumption but leaves normalization difficult in most models.
- Training: Contrastive divergence lowers energies for data samples and raises them for model samples, whose negative examples are obtained through MCMC.The intractable denominator motivates this proxy objective.
- Boltzmann machines: Fully connected Boltzmann machines require Gibbs sampling to find equilibrium states, taking exponential time in the number of hidden units and limiting scalability.Restricted connectivity in RBMs permits exact hidden-unit calculation and parallelized negative sampling, often requiring one step from data initialization.
- Sampling: Stochastic gradient Langevin dynamics makes high-dimensional EBM sampling more practical, but sampling still requires many steps and finite chains may remain far from the model distribution.Persistent contrastive divergence uses replay buffers, while short-run MCMC can use as few as 100 update steps but may not sample the correct distribution.
- Score matching: Score matching avoids the intractable density denominator by matching data and model log-density derivatives, although the data score is usually unavailable.Denoising score matching estimates it from corrupted samples across noise levels, enabling Langevin-based generation.
- Diffusion models: Diffusion models progressively add noise until data are approximately normal, then train a reverse process to remove noise by optimizing a re-weighted ELBO.Inference cost has motivated skipped-step schedules and computation-budget methods.
- Limitations and hybrids: EBMs can provide powerful representations but may exhibit high-variance training, long training and sampling times, and incomplete support of the data space.Hybrid approaches are discussed as responses to these issues.
3 VARIATIONAL AUTOENCODERS
VAEs make latent-variable likelihood optimisation tractable by amortizing an approximate posterior and optimising the ELBO, but posterior limitations can produce blurry samples and motivate richer priors and hierarchical designs.
- Variational inference: VAEs replace intractable posterior inference with a feedforward approximation qφ(z|x), enabling scalable variational inference through the ELBO.The ELBO combines a reconstruction term with KL regularisation, and reparameterization permits gradient backpropagation through stochastic sampling.
- Variational inference: A diagonal Gaussian posterior uses ˜z = µ + σ ⊙ϵ with ϵ ∼ N(0, I), making the KL term analytically integrable and gradient variance manageable.This prior is simple to sample from while retaining a tractable regularisation term.
- Limitations: On complex datasets such as natural images, VAE samples can be unrealistic and blurry because limited posterior approximation may map multiple data points to the same encoding.The resulting averaging is associated with an overly simple Gaussian posterior and MSE reconstruction effects.
- Richer priors: Richer priors, including aggregate-posterior approximations such as VampPrior, can produce multimodal latent representations and reduce inactive latent variables.Other approaches learn complex priors through variational Gaussian processes, energy-based models, or score matching.
- Hierarchical designs: Hierarchical VAEs introduce depthwise dependencies among latent variables, while skip connections can enable parallel inference and separate global from local details.Ladder VAEs use bidirectional inference with bottom-up features and top-down latent processing.
- Limitations: Autoregressive decoders model complex output dependencies but increase runtime and can cause posterior collapse when the decoder is sufficiently powerful.Posterior collapse reduces encoder–decoder gradients by making KL minimisation easy.
4 GENERATIVE ADVERSARIAL NETWORKS
GANs learn through an adversarial discriminator–generator game whose objective is related to Jensen–Shannon divergence, while alternative losses and regularisation address unstable training, vanishing gradients, and mode collapse.
- Adversarial formulation: GANs use a discriminator D to distinguish real from generated samples and a generator G trained to make its samples appear real.The two networks are trained adversarially, with D labelling real and fake samples and G minimising the resulting generator objective.
- Adversarial formulation: With sufficient capacity, the GAN objective corresponds to Jensen–Shannon divergence and permits the generator to recover the data distribution.The symmetric divergence is described as better behaved when both distributions are small than the asymmetric KL divergence used in maximum-likelihood models.
- Training challenges: GAN training is difficult because discriminator improvement can eliminate generator gradients, whereas a weak discriminator provides uninformative gradients; mode collapse is another problem.Non-cooperation can prevent convergence and produce oscillations of increasing amplitude.
- Alternative objectives: Wasserstein distance provides linear gradients that address vanishing gradients and can improve stability, but enforcing 1-Lipschitz continuity introduces optimisation challenges.Weight clipping can invalidate gradients, while gradient penalties depend heavily on generative-distribution support.
- Stabilisation: Spectral normalisation improves sample quality, supports datasets with thousands of classes, and reduces the discriminator updates needed to balance GAN training.Its use is tied to enforcing the discriminator’s 1-Lipschitz constraint in WGANs.
- Data augmentation: Applying random augmentations to all discriminator inputs can improve sample quality and stabilise training, including on datasets containing only 100 samples.The cited explanations include preventing discriminator overfitting and increasing overlap between generated and data supports.
5 AUTOREGRESSIVE LIKELIHOOD MODELS
Autoregressive models factorize data distributions into ordered conditional probabilities, enabling direct likelihood training but requiring sequential sampling. Architectural advances improve receptive fields and parallel density estimation, while ordering, scaling, and output-distribution choices remain important constraints.
- Core formulation: Autoregressive models use the chain rule to represent a joint distribution as a product of conditional probabilities.The model estimates each p(x_i|x_1:i−1) and trains by minimizing negative log-likelihood.
- Limitations: Sequential sampling can be exceedingly slow on high-dimensional data, and the fixed variable ordering may affect performance.Ordering is clear for some modalities such as text and audio but less obvious for images.
- Masked architectures: NADE and RNADE enforce autoregressive dependencies with masks, while MADE uses a fixed mask that supports parallel density estimation.NADE targets binary data, whereas RNADE models real-valued conditionals with mixture distributions.
- Architectures: RNNs, causal convolutions, and attention extend receptive fields for sequential modeling, with dilation and self-attention addressing long-range dependencies.RNNs can forget information, while dilated convolutions increase receptive fields and self-attention references previous inputs without recurrence.
- Architectures: Self-attention offers broad context but its attention matrix grows quadratically with data dimension, motivating local, sparse, and linear alternatives.Linear approaches cache φ(K)^T V for reuse across queries.
- Modelling choices: Autoregressive models use modality-specific output distributions and orderings, but assumptions such as mixture logistics can restrict expressiveness for high-frequency signals.Image models commonly use raster-scan or zig-zag orderings, while audio may require companding, bitwise conditioning, or mixture distributions.
6 NORMALIZING FLOWS
Normalizing flows construct complex distributions by composing invertible transformations of simple ones, combining exact likelihood evaluation with efficient parallelization and useful latent spaces. Their main trade-offs involve invertibility, Jacobian computation, parameter efficiency, and the different parallelism profiles of flow variants.
- Core formulation: Normalizing flows transform a simple distribution through a chain of invertible maps to construct arbitrarily complex densities.The change-of-variables rule tracks the transformed density through successive functions.
- Design constraints: Each flow transformation must be expressive, efficiently invertible, and equipped with a tractable Jacobian determinant, but flows are typically less parameter efficient.Invertibility also forces input and output dimensions to match unless multiscale designs factor out variables.
- Coupling flows: Coupling flows split inputs and transform one part using the other, producing triangular Jacobians whose determinants are efficient to compute.Elementwise bijections make the Jacobian triangular, while stacking layers increases expressivity.
- Autoregressive flows: Autoregressive flows increase flexibility, with MAF favoring parallel density estimation and IAF favoring parallel sampling.A single autoregressive flow is described as a universal approximator, whereas coupling layers are efficient in both directions.
- Probability Density Distillation: Probability density distillation trains an IAF student to mimic a pretrained autoregressive teacher through KL divergence, enabling fully parallel student computation.The student need not use its inverse function during this procedure.
6.2 Convolutional
Convolutional flow designs address restricted interactions in coupling and autoregressive flows by modifying channel mixing and convolutional structure.
- 6.2 Convolutional: Coupling and autoregressive flows have restricted triangular Jacobians, preventing all inputs from interacting directly.Permutations, 1×1 convolutions, larger kernels, and alternative causal structures are proposed to broaden interactions.
6.3 Residual Flows
Residual flows build invertible transformations from residual blocks and can provide dense Jacobians, but inversion and determinant estimation introduce computational constraints. Lipschitz restrictions ensure invertibility while limiting the learned function class, motivating implicit-flow alternatives.
- 6.3 Residual Flows: Invertible residual networks stack residual blocks, and residual-form Jacobians permit determinant computation using the matrix determinant lemma.Planar flows use a single-neuron bottleneck, while Sylvester flows increase representation ability by generalizing the construction.
- 6.3.1 Matrix Determinant Lemma: Planar flows are invertible under simple conditions but have difficult inverses, making them practical primarily for density estimation.Sylvester flows remove the severe bottleneck and allow greater representation ability.
- 6.3.2 Lipschitz Constrained: Restricting the Lipschitz constant below 1 guarantees residual-block invertibility, but the inverse requires fixed-point iteration.Residual flows have dense Jacobians, allowing interactions that coupling layers cannot represent directly.
- 6.3.2 Lipschitz Constrained: Strong Lipschitz assumptions severely restrict the learned function class, with an N-layer residual flow network at most 2N-Lipschitz.Implicit flows address this constraint by composing a residual flow with its inverse and modeling arbitrary Lipschitz transformations.
6.4 Surjective and Stochastic Layers
Surjective and stochastic layers relax the topological and architectural constraints of invertible flows, while discrete extensions trade Jacobian requirements for expressivity limits.
- Surjective layers bypass invertible-flow topology constraints by using stochastic mappings or transformations that discard information.These approaches address the difficulty of mapping unimodal distributions to multimodal ones with diffeomorphisms.
- Augmenting inputs with dimensions enables smoother transformations, with factoring out dimensions equivalent to a multi-scale architecture.
- RAD partitions data space into disjoint regions and applies a separate bijection to each, while CIFs provide a continuous partition equivalent to infinitely many flows.
- VAEs and DDPMs exemplify stochastic layers, and MCMC-sampled energy models can be interleaved with bijectors in normalizing flows.
- Discrete flows omit Jacobian determinants because discrete spaces have no volume, but this restricts them to permuting probability values.
6.6 Continuous Time Flows
Continuous-time flows represent transformations with neural ODEs, enabling invertibility, parameter efficiency, and adaptive computation while introducing solver and topology challenges.
- Neural ODE flows define hidden-state transformations continuously through a Lipschitz neural network and an ordinary differential equation.
- ODE solvers transform input noise into data, while reverse integration provides invertibility and the framework supports parameter efficiency and adaptive computation.
- The adjoint sensitivity method trains ODE-based flows by solving an augmented ODE backward, with automatic differentiation costing roughly as much as evaluating the network.
- FFJORD computes continuous-time flows with ODE solvers and can recover the true posterior distribution in the infinitesimal-flow limit.
- Neural ODE trajectories cannot cross, so additional dimensions can address their topological limitations; a p-dimensional space can be approximated in (2p + 1) dimensions.
- ODE solvers may require many network evaluations when dynamics are stiff or rapidly changing, motivating regularisation for straighter trajectories.
7 EVALUATION METRICS
Generative-model evaluation remains difficult because qualitative inspection is subjective, likelihood comparisons are uneven across model classes, and common metrics have important weaknesses.
- Qualitative sample comparison is subjective and time-consuming, while validation log-likelihood is difficult to compare with implicit-likelihood models.
- Log-likelihood measures diversity well but does not correlate strongly with sample quality.
- Inception Score combines low per-sample label entropy with high class-distribution entropy, but a model producing one image per class can score perfectly.
- Memorization can trivially satisfy evaluation metrics, while feature-based and discriminator-based tests may be poorly suited to non-natural data or biased toward adversarial models.
8 APPLICATIONS
Generative models apply across modalities, but model families have differing practical strengths; implicit representations instead model data as continuous coordinate-to-value signals.
- Autoregressive models are popular for text and audio, whereas VAEs face posterior collapse, GANs struggle with discrete data and mode collapse, and some flows support parallel synthesis.
- Implicit representations treat data as continuous signals by mapping coordinates to data values rather than using discrete arrays.
- Implicit Gradient Origin Networks concatenate latent vectors with coordinates and pass them through an implicit network to form a latent-variable model.
- Implicit GANs map latent variables to the weights of an implicit function using a hypernetwork.
- Implicit networks model data continuously, permitting arbitrarily high resolutions.
9 CONCLUSION
Recent advances are narrowing the gap between generative-model approaches while exposing persistent trade-offs in quality, efficiency, complexity, and scalability. Cross-model improvements, attention, and implicit networks point toward broader capabilities but do not eliminate adoption barriers.
- GANs retain strong sample quality, but other approaches are closing the gap as mode collapse diminishes and training objectives simplify.
- GANs remain hindered by large parameter counts and slow run-times, while hybrid models balance extremes at the cost of added complexity.
- Improved variational bounds benefit VAEs, diffusion models, and surjective flows, illustrating how advances in one generative-model field can transfer to others.
- Attention supports scaling to high-dimensional data by learning long-range dependencies, with linear attention aimed at still higher resolutions.
- Implicit networks offer a direction for efficient synthesis at arbitrarily high resolution.