PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Diffusion Models: A Unified Perspective

Calvin Luo

arXiv:2208.11970v1cs.LGcs.CV

TL;DR

The paper addresses how diffusion models can be understood across variational and score-based perspectives. It derives Variational Diffusion Models from Markovian hierarchical VAEs, proves equivalent prediction objectives, connects score learning to the variational view, and covers guidance for conditional distributions. The resulting framework unifies these interpretations while identifying limitations of score estimation and guidance-related diversity trade-offs.

  • Problem

    The paper seeks to review and unify variational and score-based understandings of diffusion models, including how they learn unconditional and conditional distributions.

  • Method

    The paper derives Variational Diffusion Models as Markovian Hierarchical Variational Autoencoders, analyzes ELBO optimization and equivalent prediction targets, and connects score-based modeling through Tweedie's Formula and guidance.

  • Results

    Optimizing a VDM is equivalent to training a neural network to predict the source input, source noise, or score function at arbitrary noise levels.

  • Takeaways & Limitations

    Diffusion models can be understood through a common framework linking variational likelihood modeling, score learning, and conditional generation via guidance.

  • Takeaways & Limitations

    Vanilla score matching can be inaccurate in low-density regions, while guidance increases conditioning control at the cost of sample diversity.

Abstract

from arXiv · show

Diffusion models have shown incredible capabilities as generative models; indeed, they power the current state-of-the-art models on text-conditioned image generation such as Imagen and DALL-E 2. In this work we review, demystify, and unify the understanding of diffusion models across both variational and score-based perspectives. We first derive Variational Diffusion Models (VDM) as a special case of a Markovian Hierarchical Variational Autoencoder, where three key assumptions enable tractable computation and scalable optimization of the ELBO. We then prove that optimizing a VDM boils down to learning a neural network to predict one of three potential objectives: the original source input from any arbitrary noisification of it, the original source noise from any arbitrarily noisified input, or the score function of a noisified input at any arbitrary noise level. We then dive deeper into what it means to learn the score function, and connect the variational perspective of a diffusion model explicitly with the Score-based Generative Modeling perspective through Tweedie's Formula. Lastly, we cover how to learn a conditional distribution using diffusion models via guidance.

Introduction: Generative Models

Generative models learn data distributions to generate new samples, while different model families emphasize sampling, likelihood, or energy functions. Diffusion models unify likelihood-based and score-based interpretations and explain their mathematical relationship.

  • Generative models learn an approximation to the true data distribution and can then generate new samples from it.Some formulations also evaluate the likelihood of observed or sampled data.
  • GANs learn a complex distribution through an adversarially learned sampling procedure.
  • Likelihood-based models seek to assign high likelihood to observed data, including autoregressive models, normalizing flows, and VAEs.
  • Energy-based models learn a flexible energy function and normalize it to obtain a distribution.
  • Diffusion models have both likelihood-based and score-based interpretations, and the paper reviews their mathematics to make their operation understandable.Score-based models learn the score of an energy-based model as a neural network rather than learning the energy function itself.

Background: ELBO, VAE, and Hierarchical VAE

Variational models introduce latent variables to represent observations and optimize the Evidence Lower Bound (ELBO) when direct likelihood computation is intractable. VAEs use encoder–decoder structure and reparameterization, while Markovian hierarchical VAEs restrict latent transitions to a Markov chain.

  • Latent variables provide hidden representations that can generate observed data and may offer compression or semantically meaningful structure.
  • Evidence Lower Bound: The ELBO provides a tractable proxy for maximizing data likelihood when marginalizing latent variables or accessing the true posterior is intractable.With sufficient parameterization and optimization, the ELBO can become equivalent to the evidence.
  • Evidence Lower Bound: The evidence equals the ELBO plus a non-negative KL divergence between the approximate and true posteriors, proving the ELBO is a lower bound.The KL term also explains why maximizing the ELBO encourages the approximate posterior to match the true posterior.
  • Variational Autoencoders: A VAE learns an encoder distribution over latents and a decoder that converts latent vectors into observations.The encoder and decoder are optimized jointly, commonly with Gaussian posterior and prior choices.
  • Variational Autoencoders: The reparameterization trick expresses stochastic Gaussian sampling as deterministic transformation of noise, enabling gradient-based optimization.A standard Gaussian sample is scaled by the target standard deviation and shifted by the target mean.
  • Hierarchical Variational Autoencoders: A Markovian hierarchical VAE models generation as a chain in which each latent transition conditions only on the previous latent.This structure can be viewed as stacking VAEs recursively across hierarchical levels.

Variational Diffusion Models

A Variational Diffusion Model is a constrained Markovian hierarchical VAE that progressively noisifies data into Gaussian noise and learns reverse transitions for generation. Its ELBO admits a lower-variance optimization and three equivalent prediction objectives: the original image, source noise, or the score at a chosen noise level.

  • Variational Diffusion Models: A VDM is a Markovian Hierarchical VAE with data-dimensional latents, fixed linear-Gaussian encoders, a Gaussian final latent, and Markovian transitions.The encoder structure is predefined rather than learned, while the latent schedule is arranged so the final distribution is standard Gaussian.
  • Variational Diffusion Models: The forward process steadily corrupts an input image with Gaussian noise until it becomes equivalent to pure Gaussian noise at timestep T.x0 denotes the data, xT pure Gaussian noise, and xt an intermediate noisy version.
  • Variational Diffusion Models: VDM generation samples Gaussian noise from p(xT) and iteratively applies learned denoising transitions pθ(xt−1|xt) for T steps to produce x0.Only the reverse conditionals are learned because the forward encoder distributions are fixed Gaussians.
  • Variational Diffusion Models: The VDM ELBO contains reconstruction, prior-matching, and consistency terms, with consistency training matching reverse denoising distributions to corresponding forward Gaussian corruptions.The prior-matching term effectively becomes zero when the final latent distribution is sufficiently Gaussian, while the consistency term dominates optimization cost across timesteps.
  • Variational Diffusion Models: A lower-variance ELBO formulation uses expectations over at most one random variable at a time, improving on the higher-variance Monte Carlo estimate of the original consistency objective.The original consistency estimate samples two random variables per timestep and sums T−1 terms, which can produce high variance for large T.
  • Three Equivalent Interpretations: VDM training has three equivalent objectives: predicting x0, predicting source noise ϵ0, or predicting the score ∇log p(xt) at an arbitrary noise level.Timesteps can be sampled stochastically while minimizing the prediction error against the corresponding ground-truth target; empirical work has sometimes favored noise prediction.

Score-based Generative Models

Score-based generative modeling represents distributions through score functions and generates samples by following those scores with Langevin dynamics. The paper connects this perspective to variational diffusion models and shows how multilevel Gaussian noise addresses key score-matching and sampling problems.

  • Score representation: Score-based models learn the score ∇log p(x) rather than an energy function or normalized density.This avoids explicitly modeling the normalization constant.
  • Langevin dynamics: The score function is a vector field pointing toward higher-likelihood regions, which Langevin dynamics follows to generate samples.Random initialization and an added noise term make the trajectories stochastic and help preserve diversity.
  • Score-matching limitations: Vanilla score matching is problematic on low-dimensional data manifolds and provides inaccurate learning signals in low-density regions.These issues are especially consequential when sampling begins from random noise.
  • Score-matching limitations: Langevin dynamics may fail to mix across disjoint modes, and the score alone does not preserve mixture weights.Different modes can be reached from one initialization, but their sampling frequencies need not match the true distribution.
  • Noise-conditioned modeling: Adding multiple Gaussian noise levels simultaneously removes manifold support constraints, improves low-density training signal, and preserves mixing coefficients in intermediate distributions.A neural network is trained across all noise levels, and annealed Langevin dynamics samples from high to low noise.
  • Unified perspective: Variational diffusion models and score-based generative models share equivalent training objectives and iterative sampling procedures.The connection is established through the score-function formulation and extends naturally toward continuous-time SDEs.

Guidance

Conditional diffusion models incorporate conditioning information at each transition and can predict conditioned inputs, noise, or scores. Guidance adjusts how strongly sampling follows the conditioning signal, trading sample diversity or classifier cost against control.

  • Conditional diffusion: Conditional diffusion models add arbitrary conditioning information y alongside timestep information at every transition.Examples include text encodings for image-text generation and low-resolution images for super-resolution.
  • Conditional diffusion: The conditional model can predict the original input, source noise, or conditional score using networks that also receive y.The score-based form estimates ∇log p(x_t|y) at arbitrary noise levels.
  • Guidance: Vanilla conditional training may downplay conditioning information, motivating guidance to increase explicit control at the cost of sample diversity.Classifier Guidance and Classifier-Free Guidance are the two principal forms described.
  • Classifier Guidance: Classifier Guidance combines an unconditional score with the gradient of a noisy classifier and scales that classifier gradient with γ.γ = 0 ignores conditioning, while larger γ values enforce stronger adherence and reduce diversity.
  • Classifier Guidance: Classifier Guidance requires a separately trained classifier capable of handling arbitrarily noisy inputs.Because standard pretrained classifiers are not optimized for this setting, the classifier must be trained alongside the diffusion model.
  • Classifier-Free Guidance: Classifier-Free Guidance replaces the separate classifier with unconditional and conditional diffusion models, whose relative influence is controlled by γ.Both models can be learned together by randomly dropping conditioning information and querying the shared model with fixed values.

Closing

The paper concludes that diffusion models unify variational and score-based views while achieving strong generative capabilities. It also identifies limits in interpretability and sampling efficiency, and connects diffusion success to the potential of deep hierarchical VAEs.

  • Closing: The paper derives VDMs from Markovian hierarchical VAEs and unifies source-input, source-noise, and score prediction objectives.It also connects variational diffusion explicitly to score-based modeling and covers conditional generation.
  • Closing: Diffusion models have demonstrated strong generative capabilities, including powering state-of-the-art text-conditioned image-generation models such as Imagen and DALL-E 2.The conclusion presents this success alongside the elegance of the underlying mathematics.
  • Limitations: VDMs do not produce interpretable latent variables because their fixed linear-Gaussian encoders restrict intermediate latents to noisy versions of the input.These latents also retain the original input dimensionality, limiting meaningful compressed structure.
  • Limitations: Sampling is expensive because generating a sample requires iterating through many denoising timesteps until the latent is reconstructed.The timestep count must be large enough for the terminal latent to become Gaussian noise.
  • Closing: Diffusion success highlights hierarchical VAEs as powerful generative models even with trivial encoders, fixed latent dimensions, and Markovian transitions.The paper suggests that more general deep HVAEs could learn complex encoders and semantically meaningful latent spaces.
Loading 2208.11970v1…