PaperlayerView exact revision on arXiv

Source-linked AI summary

The Principles of Diffusion Models

Chieh-Hsin Lai, Yang Song, Dongjun Kim, Yuki Mitsufuji, Stefano Ermon

arXiv:2510.21890v3cs.LGcs.AIcs.GR

TL;DR

Diffusion models need a unified account of how reverse processes transform simple reference samples into data and how iterative sampling can be accelerated. This book organizes the subject around variational, score-based, and flow-based perspectives, showing that these views identify different objects for reversing the same probability path while highlighting solver-based speed–accuracy tradeoffs.

  • Problem

    Diffusion models require a unified understanding of the objects learned to reverse probability paths and the computational cost of iterative sampling.

    Source: §4.3.2 · p. 121, §12.2.3 · p. 425

  • Method

    The book organizes reverse modeling through variational, score-based, and flow-based perspectives that identify the unknown object needed to reverse the probability path.

    Source: §12.2.3 · p. 425

  • Results

    The three perspectives carry over to discrete diffusion models but are not mutually exclusive schools or fixed paper categories.

    Source: §12.2.3 · p. 425

  • Takeaways & Limitations

    Diffusion sampling faces a practical tradeoff between computational cost and numerical accuracy, motivating the use of accelerated numerical solvers.

    Source: §4.3.2 · p. 121, §10.1 · p. 351

    Classifier guidance requires training the classifier to operate across all noise levels.

    Source: §8.2.3 · p. 259

Abstract

from arXiv · show

This book presents the core principles that have guided the development of diffusion models, tracing their origins and showing how diverse formulations arise from shared mathematical ideas. Diffusion modeling starts by defining a forward process that gradually corrupts data into noise, linking the data distribution to a simple prior through a continuum of intermediate distributions. The goal is to learn a reverse process that transforms noise back into data while recovering the same intermediates. We describe three complementary views. The variational view, inspired by variational autoencoders, sees diffusion as learning to remove noise step by step. The score-based view, rooted in energy-based modeling, learns the gradient of the evolving data distribution, indicating how to nudge samples toward more likely regions. The flow-based view, related to normalizing flows, treats generation as following a smooth path that moves samples from noise to data under a learned velocity field. These perspectives share a common backbone: a time-dependent velocity field whose flow transports a simple prior to the data. Sampling then amounts to solving a differential equation that evolves noise into data along a continuous trajectory. On this foundation, the book discusses guidance for controllable generation, efficient numerical solvers, and diffusion-motivated flow-map models that learn direct mappings between arbitrary times. It provides a conceptual and mathematically grounded understanding of diffusion models for readers with basic deep-learning knowledge. Supplementary materials for the book are available at the book website: https://the-principles-of-diffusion-models.github.io/

C Behind the Scenes of Diffusion Models:

This section includes supplementary material on learning fast diffusion-based generators and an optional discussion of Elucidating Diffusion Model (EDM). It also lists the work’s authors and notes that an affiliation reflects the institution at the time of the work.

Supplementary material

  • The supplementary material includes a theoretical supplement on learning fast diffusion-based generators and an optional section on Elucidating Diffusion Model (EDM).The listed sections appear as D.5 and D.6, on pages 499 and 503, respectively.Source: C Behind the Scenes of Diffusion Models · p. 5

Authors

Affiliations

Preface and Roadmap

The book establishes a principled foundation for diffusion models by explaining their essential concepts, complementary formulations, and relationships, rather than surveying every variant or application. It also examines efficiency, controllability, and diffusion-inspired generative models, equipping readers to interpret, apply, and extend the field.

Scope and audience

  • The book presents essential diffusion concepts and formulations as a stable foundation for navigating broader model research without surveying every variant or domain-specific application.It targets readers with basic deep-learning or deep-generative-modeling knowledge who want understanding beyond surface familiarity.Source: Preface and Roadmap · pp. 9–10

Core perspectives

  • Diffusion models generate data through gradual time-dependent refinement, understood through complementary variational, score-based, and flow-based perspectives.The book traces these formulations’ origins, clarifies their relationships, and connects intuitive explanations with rigorous mathematics.Source: Preface and Roadmap · p. 9

Extensions and roadmap

  • The book builds on these foundations to address more efficient sampling, greater control over generation, and standalone generative models inspired by diffusion principles.These extensions support applying existing models confidently and pursuing new research directions.Source: Preface and Roadmap · p. 10

Roadmap of This Book

The book develops diffusion models from foundational principles through unified variational, score-based, and flow-based perspectives, then turns to controllable and efficient generation. It concludes by extending these principles beyond continuous spaces while defining the book’s scope and limitations.

Parts A & B: Foundations of Diffusion Models

  • The book traces diffusion models to core principles and organizes their foundations around variational, score-based, and flow-based perspectives that describe related transformations from noise to data.The score-based view uses gradients of log density and SDE/ODE formulations, while the flow-based view uses velocity fields to transport distributions, including between arbitrary source and target distributions.Source: Roadmap of This Book · pp. 10–12

Part B: Core Perspectives on Diffusion Models

  • The book unifies these perspectives through shared temporal evolution of probability distributions, connecting conditioning-based regression, the Fokker–Planck equation, continuity equations, optimal transport, and Schrödinger bridges.The optimal-transport material is optional for readers focused on practical aspects but supplies rigorous mathematical background and links to classical literature.Source: Roadmap of This Book · pp. 12–13

Part C: Sampling from Diffusion Models

  • Sampling is presented as iterative differential-equation solving: guidance steers the time-dependent velocity field toward user objectives, while numerical solvers reduce integration steps and cost without retraining.The book emphasizes the trade-off between coarse-to-fine control and slow iterative integration.Source: Roadmap of This Book · pp. 13, 15

Part D: Learning Fast Generative Models

  • Fast generators are developed through distillation, which trains a student to imitate a slow teacher with few or even one integration steps, and flow-map learning, which directly learns noise-to-data or anytime-to-anytime solution maps.Flow-map learning does not rely on a teacher model.Source: Roadmap of This Book · p. 15

Epilogue: Beyond Diffusion on Continuous State Spaces

  • The epilogue generalizes diffusion as the evolution and reversal of probability mass under prescribed forward corruption, covering both continuous equations and discrete transition-kernel, Markov-chain, and master-equation formulations.The variational, score-based, and flow-based perspectives also extend to discrete data such as text and protein sequences.Source: Roadmap of This Book · p. 15

What This Book Covers and What It Does Not

  • The book aims to provide a durable common template for understanding and designing diffusion methods, but it is not an exhaustive survey of architectures, empirical comparisons, applications, deployment, or engineering.Its stated focus is continuous-time transport, stochastic and deterministic flows, guidance, numerical solvers, distillation, and flow-map models.Source: Roadmap of This Book · p. 16

Notations

The notation defines conventions for arrays and parameters, calculus operators, probability quantities, and recurring distributional shorthand. It also specifies how random variables, conditional expectations, implicit time indices, and Gaussian perturbations are interpreted.

Numbers and Arrays

  • Vectors, matrices, transposes, traces, identity matrices, diagonal matrices, and parameter statuses are defined for array-based notation.a denotes a column vector, A a matrix, A⊤ the transpose, Tr(A) the trace, ID an identity matrix of size D × D, and diag(a) a diagonal matrix; ϕ and θ are learnable parameters, while ϕ×, θ× are trained fixed parameters and ϕ∗, θ∗ are optimal parameters.Source: Numbers and Arrays · p. 17

Calculus

  • Calculus notation distinguishes partial and total derivatives, gradients, Jacobians, divergence, Hessians, and integrals by their operators and shapes.The gradient of a scalar is a column in R^D, the Jacobian of F : R^n → R^m has shape m × n, and the Hessian of f : R^D → R has shape D × D.Source: Calculus · p. 18

Probability and Information Theory

  • Probability notation covers densities, discrete and conditional probabilities, expectations, variance, covariance, KL divergence, Gaussian distributions, and source, target, and prior distributions.p(x) denotes a density or distribution over a continuous vector, while D_KL(p ∥ q) denotes divergence from q to p; Gaussian notation includes N(x; µ, Σ) and standard-normal samples ϵ ∼ N(0, I).Source: Probability and Information Theory · p. 18
  • The same symbol may denote a random vector and its realization, while p(x) denotes a density’s functional form and conditional densities are interpreted according to which variable is fixed.This convention keeps notation compact, and p(x|y) is read as a density in x when y is fixed or as a function of y when x is fixed.Source: Probability and Information Theory · p. 19
  • Conditional expectations such as E[f(x)|z] are functions of z, whereas conditioning on a realized value is written E[f(x)|Z = z].The distinction separates a function z 7→ E[f(x)|z] from its evaluation at a fixed z and is equivalent to integration under the conditional distribution.Source: Probability and Information Theory · p. 19
  • When relevant time indices are clear from context, they may be left implicit: E[x_s|x_t] abbreviates E[x_s|x_t, s, t], and p(x_s|x_t) denotes the conditional distribution with times understood.For fixed t, the Gaussian perturbation kernel is equivalently written as an identity in distribution with ϵ independent of x_0; d= means the two sides have the same probability law.Source: Probability and Information Theory · p. 19

Deep Generative Modeling

This section introduces deep generative modeling as distribution matching for generating realistic and controllable samples, covering explicit and implicit strategies, normalization challenges, and major model families. It also situates diffusion models within variational, score-based, and flow-based perspectives, drawing on connections to VAEs, EBMs, GANs, and normalizing flows.

1.1 What is Deep Generative Modeling?

  • DGMs learn an approximate distribution pϕ from examples drawn from an unknown pdata, targeting realistic novel samples and fine-grained controllable generation.After training, generation samples from the learned model distribution; quality is assessed by similarity to data statistics and task-specific or perceptual metrics.Source: Deep Generative Modeling · p. 21, §1.1 · p. 22

1.1.1 Mathematical Setup

  • With finite samples, DGMs minimize a discrepancy between pdata and pϕ; forward KL equals maximum likelihood and penalizes completely missing data regions through mode covering.Forward KL averages over the data distribution, while Fisher divergence instead aligns score fields and is invariant to normalization constants.Source: §1.1.1 · pp. 22–25
  • Different divergence choices encode different notions of distributional closeness: f-divergences compare density ratios, whereas Wasserstein distance uses sample-space geometry and remains meaningful for nonoverlapping supports.These choices induce distinct learning behavior and are revisited throughout the monograph.Source: §1.1.1 · pp. 25–26

1.1.2 Challenges in Modeling Distributions

  • Neural-network densities must be nonnegative and normalized, but computing the partition function is generally intractable in high dimensions, motivating alternative generative-model families.Exponentiating an energy ensures positivity, while dividing by its integral enforces normalization; the latter integral creates the central computational obstacle.Source: §1.1.2 · pp. 26–27

1.2 Prominent Deep Generative Models

  • Energy-based models define probability through an energy function, but partition-function intractability complicates training; diffusion models instead use log-density gradients to avoid computing it.EBMs assign lower energy to more probable data points and typically train by maximizing data log-likelihood.Source: §1.2 · p. 28
  • Autoregressive models factorize joint distributions into normalized conditionals, enabling exact likelihood training but requiring sequential sampling that becomes costly for long sequences.VAEs use latent variables, an encoder, and decoder trained with the ELBO, but can suffer limited sharpness and decoder latent-variable neglect.Source: §1.2 · pp. 28–30

Normalizing Flows.

  • Normalizing flows learn bijections between simple latent and complex data distributions, using change of variables and tractable Jacobian determinants for exact maximum-likelihood computation.Classic normalizing flows use invertible transformations, while NODEs represent the transformation with an ordinary differential equation.Source: Normalizing Flows · p. 30
  • Classic flow-based models face bijectivity constraints in normalizing flows, ODE-solving overhead in NODEs, and scaling challenges for high-dimensional data.Source: Normalizing Flows · p. 30
  • GANs implicitly minimize the Jensen–Shannon divergence, while f-GANs generalize adversarial training to a family of f-divergences.GANs avoid explicit likelihood estimation but use unstable min-max training, often requiring careful architectures and engineering.Source: Normalizing Flows · pp. 30–31

1.3 Taxonomy of Modeling

  • Deep generative models divide into explicit models with tractable or approximately tractable densities and implicit models defined only through sampling procedures.Explicit examples include ARs, NFs, VAEs, and DMs; implicit models use x = Gϕ(z) with z ∼pprior.Source: §1.3 · p. 32
  • Diffusion models connect variational training to VAEs, score matching to EBMs, and continuous-time transformations to NFs.These three paradigms provide the conceptual foundation for understanding modern diffusion-based generative modeling.Source: §1.3 · p. 32

1.4 Closing Remarks

  • The monograph develops diffusion models through three perspectives: VAEs lead to DDPMs, EBMs and score matching lead to NCSN and Score SDE, and NFs lead to Flow Matching.Together, these origins provide a coherent framework for interpreting diverse diffusion formulations and their unifying principles.Source: §1.4 · p. 33

Foundations and Perspectives · A First Operational View of Diffusion Model Training and Sampling · Message

The section introduces diffusion models through three complementary perspectives and an operational training-and-sampling procedure: learn clean-signal predictions from corrupted data, then repeatedly refine fresh noise toward a sample.

of Diffusion Models

  • Diffusion models are presented through variational, score-based, and flow-based views, connected respectively to variational autoencoders, energy-based models, and normalizing flows.Source: of Diffusion Models · p. 36

Message

  • Training corrupts clean data at a chosen noise level by mixing signal and independent Gaussian noise, with α_t and σ_t controlling their contributions.The resulting noise dial moves from nearly clean data toward increasingly noise-dominated observations.Source: Message · p. 37
  • The corruption schedule defines distributions p_t that progress from data toward a simple, typically Gaussian prior, forming snapshots across noise levels.The accumulated corruption can also correspond to many smaller sequential steps in DDPMs and score SDEs.Source: Message · pp. 37–38

A First Operational View of Diffusion Model Training and Sampling

  • Training samples data, noise levels, and Gaussian noise, then fits a network to predict the hidden clean sample from only the noisy observation and its noise level using squared error.Under this loss, the optimal prediction is E[x_0|x_t,t], averaging plausible clean samples when corruption is ambiguous.Source: Message · pp. 38–39

Message

  • Generation starts from fresh prior noise rather than a hidden corrupted datum, uses the network’s current clean-signal prediction to infer residual noise, and remixes both at progressively lower noise levels.The loop repeats from t = T to t = 0, updating the prediction after each step.Source: Message · pp. 39–40
  • Repeated refinement is necessary because high-noise predictions average over many plausible samples, whereas lower-noise states let successive predictions sharpen into detailed structure.Each update blends the current state toward what the network presently predicts as clean, so the final sample reflects a sequence of revised guesses.Source: Message · pp. 40–41
  • The operational template is the foundation for later mathematical treatments: latent-variable DDPMs, score-based denoising, continuous-time reverse dynamics, and transport along a continuous path.Across these views, training learns from noisy observations while generation transforms fresh noise over decreasing noise levels.Source: Message · p. 41

Variational Perspective: From VAEs to DDPMs

This section develops the variational perspective from VAEs through hierarchical VAEs to DDPMs, explaining how latent-variable generation is trained through variational bounds. It covers encoder–decoder constructions, ELBO trade-offs, Gaussian reconstruction, and diffusion’s fixed forward noising with learned stepwise reverse denoising.

Variational Perspective: From VAEs to DDPMs

  • DDPMs extend the latent-variable template by fixing the encoder as a forward process that gradually maps data to noise while learning a decoder that reverses it through successive denoising steps.VAEs, hierarchical VAEs, and diffusion models therefore share a variational-bound foundation, while deeper flat VAEs alone cannot resolve unimodal posterior limitations or posterior collapse.Source: Variational Perspective: From VAEs to… · p. 42, §2.1.5 · pp. 51–52

2.1 Variational Autoencoder

  • VAEs impose probabilistic structure on latent codes, turning autoencoders into generative models that can produce novel realistic data.The stochastic encoder maps x to z, while the decoder reconstructs x from z; sampling draws z from a simple prior and decodes it.Source: §2.1 · p. 44, Figure 2.1 · p. 44

2.1.1 Probabilistic Encoder and Decoder

  • A VAE uses a tractable encoder qθ(z|x) because exact posterior inference and maximum-likelihood training require intractable integration over latent variables.The decoder generates observations from latent variables sampled from a standard Gaussian prior, typically z ∼pprior := N(0, I).Source: §2.1.1 · pp. 44–45

Theorem 2.1.1: Evidence Lower Bound (ELBO)

  • The ELBO provides a computable lower bound on log likelihood, decomposing into reconstruction and latent-KL terms that balance faithful recovery with coherent prior-based sampling.The reconstruction term recovers the familiar autoencoder loss under Gaussian assumptions, while the KL term regularizes qθ(z|x) toward a simple Gaussian prior.Source: §2.1.2 · p. 45, Theorem 2.1.1: Evidence Lower Bound… · p. 46
  • Maximizing the ELBO reduces inference error because the gap between log likelihood and the ELBO is the non-negative divergence between approximate and true posteriors.The joint-distribution comparison separates true modeling error from inference error and explains why the ELBO is a valid lower bound.Source: Theorem 2.1.1: Evidence Lower Bound… · p. 47

2.1.3 Gaussian VAE

  • With Gaussian encoder and decoder distributions, VAE training reduces to minimizing a regularized reconstruction loss because the KL term has a closed-form expression.The decoder uses fixed variance, and the reconstruction term simplifies under the Gaussian assumption.Source: §2.1.3 · pp. 47–48

2.1.4 Drawbacks of Standard VAE

  • Standard VAEs can generate blurry outputs when overlapping latent regions correspond to distinct inputs, because the optimal Gaussian decoder averages conflicting modes.This limitation is specific to the common simple-Gaussian-decoder, squared-reconstruction-error setup rather than every VAE architecture.Source: §2.1.4 · pp. 48–49

2.1.5 (Optional) From Standard VAE to Hierarchical VAEs

  • Hierarchical VAEs stack latent variables across abstraction levels, enabling progressive generation from coarse to fine structure and yielding a hierarchical ELBO.Their ELBO includes reconstruction, adjacent-layer matching, and top-level prior-regularization terms.Source: §2.1.5 · pp. 50–51
  • Diffusion models extend layered VAE structure while avoiding hierarchical VAEs’ training weaknesses by fixing encoding and learning only the generative reversal, improving stability and output quality.Hierarchical VAEs can suffer from weak higher-level signals, unstable joint optimization, and overly powerful conditionals that suppress latent utility.Source: §2.1.5 · p. 53

2.2 Variational Perspective: DDPM

  • DDPMs use a fixed Gaussian forward process and a learned reverse process that denoises samples incrementally from noise.Each denoising step is more manageable than generating a complete sample from scratch, yielding stability and expressive power.Source: §2.2 · p. 54

2.2.1 Forward Process (Fixed Encoder)

  • The fixed forward chain progressively adds Gaussian noise to data until it converges to the isotropic prior pprior := N(0, I), using a predetermined increasing noise schedule.The process supports direct sampling of noisy states through recursively defined perturbation distributions.Source: §2.2.1 · pp. 54–56

2.2.2 Reverse Denoising Process (Learnable Decoder)

  • The learnable reverse chain starts from xL ∼pprior and sequentially approximates unknown transitions p(xi−1|xi) to recover a coherent data sample x0.The reverse process is Markovian, but the oracle transition is unknown and must be approximated.Source: §2.2.2 · p. 56, Figure 2.5 · p. 56

Question 2.2.1 … Sampling and Inference.

This section develops diffusion sampling and inference from sequential reverse denoising and score-based methods to continuous-time reverse SDEs and probability-flow ODEs. It connects these approaches with diffusion training, Gaussian transition dynamics, and flow-based velocity-field frameworks, while addressing computational limitations and faster or deterministic sampling formulations.

Question 2.2.1

  • Conditioning on the clean sample x0 converts the intractable marginal reverse kernel into a closed-form Gaussian conditional, while minimizing conditional and marginal KL divergences has the same optimum.The equivalence enables DDPM to train against tractable conditional targets instead of directly modeling the complicated marginal mixture.Source: Figure 2.6 · p. 57, Question 2.2.1 · p. 58

Lemma 2.2.2: Reverse Conditional Transition Kernel

2.2.3 Modeling of Reverse Transition Kernel pϕ(xi−1|xi)

  • DDPM models each reverse transition pϕ(xi−1|xi) as Gaussian with learnable mean µϕ(·, i) and fixed positive variance, yielding a closed-form KL-based training objective.Because the forward marginal pi(xi|x0) is directly sampleable in closed form, each loss term needs only a clean sample, timestep, and corrupted sample.Source: §2.2.3 · pp. 59–60

2.2.4 Practical Choices of Predictions and Loss

  • The commonly used ϵ-prediction parameterization reduces training to squared ℓ2 noise prediction, and its simplified loss shares the same population-optimal predictor as the weighted objective.The weighting changes optimization emphasis across noise levels and can affect practical performance despite leaving the ideal predictor unchanged.Source: §2.2.4 · pp. 60–61

Theorem 2.2.3: DDPM’s ELBO

  • DDPM training is an ELBO on the data log-density whose fixed forward noising chain decomposes into per-step KL, prior, and reconstruction terms that become efficiently sampled denoising regressions.The linear–Gaussian forward process and closed-form conditional targets, rather than the absence of per-level KL terms, provide the key tractability.Source: Theorem 2.2.3: DDPM’s ELBO · pp. 62–63, §2.2.5 · p. 62

2.2.6 Sampling

  • DDPM sampling starts from xL ∼ N(0, I) and recursively samples xi−1 for i = L, L−1, …, 1 until x0 is obtained as the final clean generated sample.At each step, the model estimates the clean data from xi and re-noises it at the reduced level i−1.Source: §2.2.6 · pp. 63–64
  • Around 1,000 sequential denoising steps are typically required in the original DDPM formulation, making sampling inherently slow; early steps establish global structure and later steps add detail.At high noise levels, the optimal denoiser predicts an average clean sample, which can produce blurry estimates before progressive refinement.Source: §2.2.6 · pp. 65–66
  • Small β_i steps make reverse transitions approximately Gaussian, but DDPM sampling requires O(L) sequential neural-network evaluations, slowing generation and preventing parallelization.Large β_i can produce multimodal or strongly non-Gaussian reverse distributions that a single Gaussian cannot represent; continuous-time numerical strategies are proposed later to address this bottleneck.Source: §2.2.6 · p. 66

2.3 Closing Remarks

  • The variational account frames DDPMs as fixed forward noising plus learned reverse denoising, with an ELBO decomposing training into tractable conditional denoising tasks and leading toward a unified continuous-time framework.The following chapter shifts from transition probabilities to score learning and develops the connection between DDPM noise prediction and score functions.Source: §2.3 · p. 67

Score-Based Perspective: From EBMs to NCSN

  • The score function guides samples toward higher-probability regions, enabling generation without computing an intractable normalization constant and motivating progressive denoising across noise-perturbed distributions.This score-based view connects energy-based modeling to NCSNs and explains its equivalence with DDPM noise prediction.Source: §2.3 · p. 67, Score-Based Perspective: From EBMs to… · p. 68

3.1.2 Motivation: What Is the Score?

  • Scores bypass the partition function because ∇x log p(x) = ∇x log ˜p(x), while fully characterizing the distribution up to normalization and remaining as expressive as modeling p(x).For EBMs, the model score is −∇xEϕ(x), so score-based objectives avoid directly computing Zϕ.Source: §3.1.2 · pp. 71–72, §3.1.3 · p. 72

3.1.4 Langevin Sampling with Score Functions

  • Langevin dynamics combines score-directed updates with Gaussian noise, allowing exploration beyond local minima; under standard regularity assumptions, its distribution converges exponentially fast to pϕ as t →∞.The target distribution pϕ is stationary, and the continuous-time Langevin SDE is the limit of the discrete Euler–Maruyama updates.Source: §3.1.4 · p. 74
  • In high-dimensional or multimodal distributions, Langevin sampling can mix poorly because efficiency depends on step size, noise scale, and many iterations, potentially missing distributional diversity.These limitations motivate more structured and guided sampling methods than local stochastic exploration.Source: §3.1.4 · p. 75

3.2. From Energy-Based to Score-Based Generative Models

  • Score matching trains a neural network vector field to match the data score, and integration by parts produces an equivalent objective using only model derivatives and data samples.This shift avoids partition-function computation and eliminates the need to sample from the model during training, though energy-based formulations can require costly second derivatives.Source: §3.1.3 · p. 73, §3.2 · p. 76, §3.2.1 · pp. 76–77

Proposition 3.2.1: Hyvärinen’s Tractable Form of SM

  • Hyvärinen’s tractable score-matching objective has minimizer s∗(·) = ∇x log p(·), recovering the true data score without requiring direct access to it.The objective differs from the original formulation only by a constant independent of ϕ.Source: Proposition 3.2.1: Hyvärinen’s… · p. 77, §3.2.1 · p. 77
  • Hyvärinen’s tractable score-matching objective can be trained from observed data alone, combining a score-magnitude penalty with a divergence term whose optimum is the data score.When the score is a gradient field, negative-definite curvature makes stationary points locally attracting maxima of log density; negative trace alone can still indicate a saddle.Source: Proposition 3.2.1: Hyvärinen’s… · p. 77, §3.2 · p. 78, §3.2.2 · p. 78

3.2.2 Sampling with Langevin Dynamics

  • The learned score replaces the oracle score in Langevin dynamics, whose discrete Euler–Maruyama recursion converges to the continuous formulation as the step size becomes small.The same framework motivates score-based generative models as stochastic-process methods for transforming noise toward data.Source: §3.2.2 · p. 78, §3.2.3 · p. 79

3.3.1 Motivation

  • Sliced score matching estimates the Jacobian trace with random projections and JVP/VJP operations, but DSM is presented as more robust for manifold-supported data because sliced methods can be unstable, high-variance, and locally underconstrained.The trace becomes increasingly expensive with dimension, whereas averaging K probes gives an unbiased estimator with variance O(1/K).Source: §3.3.1 · p. 80

3.3.2 Training

  • Denoising score matching uses conditional noise injection to replace the intractable data score with a closed-form regression target, while remaining equivalent to ordinary score matching up to a parameter-independent constant.For Gaussian corruption, the learned score targets a smoothed data distribution, with larger noise capturing coarse structure and smaller noise preserving finer details.Source: §3.3.2 · pp. 81–82, Insight 3.3.1: Conditioning Technique · p. 83

3.3.3 Sampling

  • At sufficiently small noise, Langevin dynamics using the learned score produces samples approximating pdata, while noise injection gives well-defined gradients and improves traversal of low-density regions.Perturbing data yields a full-support distribution, making the score defined throughout RD and smoothing sparse regions between modes.Source: §3.3.3 · p. 84

3.3.4 Why DSM is Denoising: Tweedie’s Formula

  • Tweedie’s formula identifies the posterior-mean denoiser with a σ2-sized score ascent step, establishing the central link between DSM score prediction and denoising.This relationship extends through exponential-family log-normalizers, whose higher derivatives encode posterior cumulants and, in the Gaussian case, higher derivatives of the noisy log density.Source: Lemma 3.3.2: Tweedie’s Formula · pp. 85–86

3.3.5 (Optional) Why DSM is Denoising: SURE

  • SURE can be minimized using only noisy observations while remaining equivalent in expectation to minimizing true MSE and Bayes risk.The SURE-optimal denoiser coincides with the Bayes estimator, and its score parameterization is equivalent to Hyvärinen score matching up to an additive constant.Source: §3.3.5 · pp. 87–88

3.3.6 (Optional) Generalized Score Matching

  • Generalized score matching replaces the gradient operator with a linear operator, yielding tractable sample-based objectives whose complete operators identify the target distribution.The framework recovers classical score matching for L = ∇, supports denoising and higher-order targets, and extends to discrete settings including language modeling.Source: §3.3.6 · pp. 88–90

3.4.1 Motivation

  • Training at a single noise level struggles with multimodal exploration at low noise, loses detail at high noise, and can converge slowly or fail in high dimensions.These limitations motivate multiple noise levels, beginning with coarse exploration and progressively refining samples toward fine details.Source: §3.4.1 · p. 91

3.4.2 Training

  • NCSN trains one noise-conditional score network across a ladder of Gaussian noise scales, recovering the true score at each level through a weighted DSM objective.The ladder spans scales that preserve fine details at low noise and sufficiently smooth the distribution at high noise.Source: §3.4.2 · pp. 92–93
  • DDPM noise prediction and NCSN score prediction contain the same information, with noise prediction equivalent to score prediction up to the known factor 1/σi.This equivalence follows from Tweedie’s formula and connects the Bayes-optimal DDPM noise predictor to the true noisy score.Source: §3.4.2 · p. 93

3.4. Multi-Noise Levels of Denoising Score Matching (NCSN)

  • Annealed Langevin dynamics samples by initializing at high noise and sequentially applying Langevin updates through progressively smaller noise levels until final low-noise refinement.Each level uses the previous level’s output as initialization, moving from a simple starting distribution toward increasingly less-smoothed data distributions.Source: §3.4 · pp. 94–95

3.4.3 Sampling

  • NCSN sampling costs O(LK) sequential network passes because local score accuracy requires small steps while high-dimensional MCMC mixes slowly.The sequential dependence of updates and expensive score-network evaluations makes annealed sampling computationally slow.Source: §3.4 · p. 95

3.5 Summary: A Comparative View of NCSN and DDPM

  • NCSN and DDPM share a dense-discretization bottleneck, often requiring hundreds or thousands of iterations, motivating efficient samplers, distillation, and direct few-step generators.The section identifies later chapters as addressing these routes to faster diffusion generation.Source: §3.5 · p. 96, Question 3.5.1 · p. 96

Diffusion Models Today: Score SDE Framework

  • Generation can use either a stochastic reverse-time SDE or deterministic probability flow ODE, which share marginal distributions despite different sample trajectories.These differential-equation formulations support standard and advanced numerical solvers for improving sampling efficiency and accuracy.Source: Diffusion Models Today: Score SDE… · p. 99

4.1 Score SDE: Its Principles

  • Score SDE unifies DDPM and NCSN as continuous-time stochastic or ordinary differential equations, with a time-dependent score guiding reversal across noise levels.The score evolves from sharply targeting individual modes at low noise to pointing toward the global center at high noise.Source: §3.6 · p. 97, §4.1 · p. 100

4.1.1 Motivation: From Discrete to Continuous-Time Processes

  • As the discrete step size approaches zero, common noise-injection updates converge to a continuous-time SDE with drift, diffusion, and Wiener-process noise.This provides the continuous-time limit connecting discrete diffusion procedures to stochastic differential equations.Source: §4.1.1 · p. 101
  • Once the forward drift and diffusion are specified, the reverse process has a single unknown term—the time-dependent score—so score matching enables sampling by numerical integration.The learned score determines the reverse dynamics that transport terminal noise back toward the data distribution.Source: Diffusion Models Today: Score SDE… · p. 98, §4.1.1 · pp. 102–103

4.1.2 Forward-Time SDEs: From Data to Noise

  • The forward SDE progressively corrupts data into noise, and suitable coefficients make its terminal marginal converge to a simple prior, typically Gaussian.For affine drift, the perturbation kernel is Gaussian and permits simulation-free sampling of noisy states; both NCSN and DDPM fit this setting.Source: §4.1.2 · pp. 103–105

4.1.3 Reverse-Time Stochastic Process for Generation

  • Because stochastic trajectories are not individually reversible, the distribution over trajectories is reversed through a well-defined reverse-time SDE integrated from t = T to t = 0.The reverse process starts from prior samples and evolves them backward to generate data-like samples.Source: Figure 4.4 · p. 105, §4.1.3 · p. 106
  • The reverse-time SDE combines score-driven drift with controlled noise, guiding trajectories toward high-density regions while preserving the forward process’s marginal evolution.The noise is stronger early in reverse time and weakens as samples approach the data manifold.Source: §4.1.3 · pp. 106–107
  • Generation solves the learned reverse-time SDE from x_T ∼ p_prior to t = 0, where Anderson (1982) guarantees matching forward and reverse marginals when p_prior ≈ p_T.The unknown score is approximated by a neural network trained through score matching.Source: §4.1.3 · pp. 107–108

Is it necessary to sample using the SDE in Equation (4.1.6)

  • The Probability Flow ODE is a deterministic process with the same marginal distributions as the forward SDE, providing an alternative to stochastic reverse-time sampling.It is derived by selecting an ODE drift that preserves the forward SDE’s marginal densities rather than simply deleting the diffusion term.Source: Is it necessary to sample using the… · pp. 108–109
  • The PF-ODE can be integrated in either time direction and benefits from established ODE solvers, whereas SDE sampling generally admits only forward-time integration.Practical generation still requires numerical integration because the solution is not available in closed form.Source: Is it necessary to sample using the… · pp. 108–109

4.2 Matching Marginals in Forward/Reverse-Time SDEs and PF-ODE

Theorem 4.2.1: Marginal Alignment via the Fokker–Planck Equation

  • Theorem 4.2.1 shows that deterministic PF-ODE and stochastic reverse-time SDE dynamics can share every marginal distribution despite differing at the sample-trajectory level.The PF-ODE can run forward from p_0 or backward from p_T along the same marginals, while infinitely many conditional processes may realize a fixed marginal.Source: Figure 4.5 · p. 111, Theorem 4.2.1: Marginal Alignment via… · pp. 112–114

4.2.2 A Computable Example: Evolutions of Gaussian Dynamics

  • In the Gaussian example, closed-form score expressions make both reverse-time SDE and PF-ODE derivations possible using elementary calculus, and the reverse transition has Gaussian variance 1/A and mean B/A.The Gaussian calculation recovers the general reverse-time SDE and constructively verifies coincident PF-ODE, forward-SDE, and reverse-SDE marginals.Source: §4.2.2 · pp. 114–116
  • For Gaussian dynamics, matching mean and variance makes the linear ODE share the forward SDE’s Gaussian marginals at every time.Affine maps preserve Gaussianity, and the ODE’s mean and variance satisfy the same scalar evolution equations as the forward SDE.Source: §4.2.2 · p. 117

Proposition 4.3.1: Minimizer of DSM

4.3.2 Sampling and Inference

  • The learned score proxy enables tractable reverse-time SDE and PF-ODE inference from prior samples, producing distributions that approximate the data distribution when score estimation is accurate.SDE accuracy depends on prior mismatch, training error, and numerical discretization error; the PF-ODE process is deterministic.Source: §4.3.2 · pp. 119–121
  • Sampling diffusion models is equivalent to solving a probability-flow ODE or reverse-time SDE, with iterative numerical solvers explaining their computational cost.The PF-ODE additionally supports accelerated solver techniques, inversion for latent representations, and likelihood evaluation through continuous change of variables.Source: §4.3.2 · pp. 121–122
  • The PF-ODE can be solved forward to encode data into noisy latent representations and used to evaluate likelihoods by tracking log-density changes along trajectories.These capabilities support controllable generation applications including image translation and editing.Source: §4.3.2 · p. 122

4.1 VE SDE

  • VE SDEs use zero drift and increasing diffusion variance, with NCSN providing a discretized instance based on a geometric variance schedule.The prior is selected using the VE perturbation kernel and the terminal variance behavior.Source: §4.4.1 · p. 123, §4.4 · p. 124

4.4.2 VP SDE

  • VP SDEs use a nonnegative time-dependent noise schedule and standard normal prior, with DDPM interpreted as their discretization.Their Gaussian perturbation kernels have known means and covariances, allowing direct computation of the conditional score.Source: §4.4.2 · p. 124

4.4.3 (Optional) How Is the Perturbation Kernel pt(xt|x0) Derived?

  • Linear SDE drift preserves Gaussian conditional transitions even for non-Gaussian data, with moments governed by solvable ODEs and isotropic covariance under independent Wiener coordinates.The derivation assumes finite initial mean and variance; isotropic covariance justification is deferred to Section C.1.5 and relies on Itô calculus.Source: §4.4.3 · p. 125

4.5 A Direct View Through Accumulated Perturbation Kernels

  • Accumulated perturbation kernels directly expose signal and noise over time, are equivalent to linear SDE descriptions for admissible affine schedules, and connect to Flow Matching probability paths.The affine form includes familiar linear Gaussian interpolation, while validity requires the implied diffusion variance to remain nonnegative.Source: §4.5 · pp. 127–128, Figure 4.8 · p. 127

4.5.1 Connection to Score SDE

  • For valid affine schedules, specifying α_t and σ_t in p_t(x_t|x_0) is mathematically equivalent to specifying the linear SDE coefficients f(t) and g(t).The signal-to-noise ratio cannot increase with time, and valid schedules must satisfy g^2(t) ≥ 0.Source: §4.5.1 · pp. 128, 130
  • Exact finite-time forgetting of x_0 cannot occur with bounded f and g; practical formulations instead tolerate negligible residual dependence or use infinite-time limits.Achieving α_T = 0 requires the drift contraction to become singular near T.Source: §4.5.1 · pp. 129–130

4.5.2 Connection to Variational-Based Diffusion Model

  • The accumulated kernel provides a more interpretable DDPM formulation, while Bayes’ rule yields continuous-time reverse conditionals from the forward kernels.These reverse conditionals support tractable training targets and efficient sampling, and extend the discrete derivation to arbitrary continuous times.Source: §4.5.2 · p. 130, Lemma 4.5.2: Reverse Conditional… · p. 131

Lemma 4.5.2: Reverse Conditional Transition Kernels

  • The generalized reverse model replaces the clean signal with an x-predictor, reducing the diffusion ELBO’s KL term to a weighted regression loss; sampling then uses the parameterized reverse kernel.An equivalent ϵ-prediction formulation is also valid.Source: Lemma 4.5.2: Reverse Conditional… · pp. 131–132

4.6 (Optional) Fokker–Planck Equation and Reverse-Time SDEs via Marginalization and Bayes’ Rule

  • The optional Fokker–Planck discussion uses marginalization, change of variables, Taylor–Gaussian smoothing, and the ∆t → 0 limit to explain marginal evolution.The presentation is heuristic rather than a mathematically rigorous proof.Source: §4.6 · pp. 133–134, §4.6.1 · p. 133

4.6.2 Why Does Reverse-Time SDE Take The Form?

  • Bayes’ rule and a leading-order Taylor expansion explain the reverse-time transition, whose continuous-time limit produces the reverse-time SDE with a score-function term.The derivation is explicitly heuristic because second-order spatial terms are generally O_p(∆t).Source: §4.6.2 · pp. 134–135, §4.6 · p. 136

4.7 Closing Remarks

  • The reverse-time SDE depends on the score function, while the deterministic PF-ODE follows the same marginal densities, making sampling numerical integration of a learned vector field.Both DDPM and NCSN arise as SDE discretizations, and the PF-ODE provides a deterministic counterpart to their stochastic dynamics.Source: §4.7 · p. 137

Flow-Based Perspective: From NFs to Flow Matching

  • For the same diffusion path, Score SDEs obtain the PF-ODE velocity from a learned score, whereas Flow Matching learns that velocity directly and can avoid Gaussian endpoints.Both approaches recover an intractable marginal quantity by averaging tractable conditional quantities.Source: Flow-Based Perspective: From NFs to… · pp. 138–140, Figure 5.1 · p. 139
  • Flow Matching extends conditional transport beyond Gaussian diffusion to general source and target distributions when conditional paths are efficiently sampleable and their velocities are tractable.This retains simulation-free training through tractable conditional paths and velocities.Source: Flow-Based Perspective: From NFs to… · pp. 139–140

5.1 Flow-Based Models: Normalizing Flows and Neural ODEs

  • Normalizing Flows transform a simple prior into data through invertible maps with exact likelihoods, while Neural ODEs extend this transport continuously in time through a smooth vector field.Normalizing Flows balance expressivity and Jacobian cost using sequences of invertible mappings, whereas Neural ODEs arise as the continuous-depth limit.Source: §5.1 · p. 141, §5.1.1 · pp. 141–142, §5.1.2 · p. 144

5.1.1 Normalizing Flows

  • Normalizing Flows sample by drawing from the prior and applying the learned invertible map, with likelihood evaluation based on the change-of-variables formula.General Jacobian computation can scale as O(D^3), motivating structured transformations with efficiently computable Jacobians.Source: §5.1.1 · p. 142, Sampling and Inference · p. 143

Sampling and Inference.

  • Sampling in discrete and continuous normalizing flows differs between composing finite invertible maps and integrating trajectories of a time-dependent velocity field.The continuous formulation represents transport through integral curves of dx(t)/dt = vϕ(x(t), t).Source: Figure 5.4 · p. 144

5.1.2 Neural ODEs

  • Neural ODEs learn a velocity field that transports a prior at t = 0 to the data distribution at t = T, while continuous-time change of variables enables likelihood evaluation and maximum-likelihood training.The induced marginal distributions evolve according to the continuity equation, connecting Neural ODEs to PF-ODEs.Source: §5.1.2 · pp. 145–146

Lemma 5.1.1: Instantaneous Change of Variables

  • Under sufficient regularity, the log-density along an ODE trajectory follows from the continuity equation and the chain rule.This result is the instantaneous change-of-variables identity for a continuous random process evolving under an ODE.Source: Lemma 5.1.1: Instantaneous Change of… · pp. 146–147
  • NODEs learn a parameterized velocity field vϕ under maximum likelihood, evolving prior latents through an ODE flow toward the terminal distribution.The formulation is the continuous-time analogue of discrete normalizing flows, replacing sums with integrals and log-determinants with traces.Source: Lemma 5.1.1: Instantaneous Change of… · p. 147
  • Exact log-likelihoods follow by reverse-integrating the change-of-variables equation from z(T)=x and adding the tractable prior term log pprior(z(0)).This supports exact likelihood-based training and evaluation for neural ODEs.Source: Lemma 5.1.1: Instantaneous Change of… · p. 147
  • Adjoint sensitivity computes gradients with O(1) memory through an auxiliary ODE, but NODE training remains expensive because numerical integration is required at each step.The memory-efficient gradient method does not remove the computational cost of repeated integration.Source: Lemma 5.1.1: Instantaneous Change of… · p. 147
  • A trained NODE samples by drawing x(0) from the prior and numerically integrating forward, with x(T) approximating a sample from pdata.The terminal state is obtained through numerical solvers.Source: Lemma 5.1.1: Instantaneous Change of… · p. 148
  • Hutchinson’s estimator avoids constructing the full Jacobian and makes high-dimensional likelihood evaluation practical, but produces stochastic rather than exact divergence estimates.The randomness comes from estimating the trace using sampled vectors.Source: Lemma 5.1.1: Instantaneous Change of… · p. 148

Question 5.1.1 · Question 5.2.1

This section develops Flow Matching as a simulation-free framework for learning continuous transport, covering its conditioning-based formulation, diffusion equivalence, continuity-equation guarantees, and Gaussian and conditional probability paths between general source and target distributions. It also examines complementary probability-path and flow-first constructions, the distinction between conditional interpolations and marginal trajectories, and rectification methods that preserve marginals while producing straighter, more easily integrated paths.

Question 5.1.1

  • Flow Matching learns the velocity field directly with a simulation-free regression objective, requiring ODE integration only during generation.This avoids the repeated ODE solves required by likelihood-based NODE training.Source: Question 5.1.1 · p. 148, §5.2 · p. 149

5.2 Flow Matching Framework

  • FM generalizes Gaussian-prior diffusion to continuous transport between arbitrary source psrc and target ptgt distributions through a time-dependent velocity field.The learned ODE transports samples along a prescribed probability path whose marginals interpolate between the endpoints.Source: §5.2 · p. 149, §5.2.2 · p. 153

5.2.1 Lesson from Score-Based Methods

  • Score-based diffusion first learns tractable conditional scores and converts them into a PF-ODE velocity, motivating Flow Matching’s direct conditional-to-marginal velocity formulation.The PF-ODE reproduces the evolving marginal densities and recovers pdata at t = 0 when initialized from the prior at t = 1.Source: §5.2.1 · pp. 151–152

5.2.2 Flow Matching Framework

  • Conditional Flow Matching learns an intractable marginal velocity by regressing on tractable conditional velocities, with the same population minimizer as the ideal objective.Simulation-free training requires easily sampling xt from the conditional path and evaluating its conditional velocity as the regression target.Source: §5.2.2 · pp. 155–156
  • Conditional flow matching replaces the intractable marginal velocity target with tractable conditional velocities while preserving the optimal marginal velocity field.Averaging the conditional training objective over data yields the same optimum as marginal flow matching, extending the conditioning principle shared by variational, score-based, and flow-based approaches.Source: §5.2.2 · pp. 156–157

5.2.3 Comparison of Diffusion Models, General Flow Matching, and NODEs

  • Gaussian flow matching is essentially equivalent to the corresponding diffusion formulation when the Gaussian schedule satisfies the diffusion-SDE compatibility condition.Flow matching also provides a simulation-free alternative to continuous normalizing flows by regressing a prescribed velocity field instead of solving ODEs during maximum-likelihood training.Source: §5.2.3 · p. 158

Theorem 5.2.2: Mass Conservation Criterion

  • The flow-induced density matches a prescribed density path if and only if the velocity and density satisfy the continuity equation.This mass-conservation criterion verifies whether an ODE flow transports the source distribution along the intended path and thereby enables transport between source and target distributions.Source: Theorem 5.2.2: Mass Conservation… · p. 159, Question 5.2.1 · p. 159

Proposition 5.2.3: Marginal VF Generates Given Marginal Density

5.3 Constructing Probability Paths and Velocities Between Distributions

  • Flow matching constructs tractable conditional paths and velocities, requiring efficient conditional sampling and closed-form conditional velocities for valid training.The framework supports both probability-path-first and flow-first constructions, enabling simulation-free flows between arbitrary source and target distributions.Source: §5.3 · pp. 162–163

5.3.1 A Key Special Case: Marginal pt(xt) and Velocity vt(xt) in the Gaussian-to-Gaussian Bridge

  • For Gaussian source and target distributions, the interpolated Gaussian path has an analytically characterized velocity field that satisfies the continuity equation.This Gaussian-to-Gaussian bridge serves as the template for constructing conditional paths and velocities in the general marginal setting.Source: §5.3.1 · p. 163, Proposition 5.3.1: Closed-Form… · p. 164

Proposition 5.3.1: Closed-Form Velocity Field for Gaussian Density Path

  • Fixing an invertible flow map uniquely determines its velocity, whereas specifying only a density path generally leaves the velocity non-unique.The closed-form Gaussian construction extends to latent-conditioned paths and velocities.Source: Proposition 5.3.1: Closed-Form… · pp. 164–165

5.3.2 Conditional Probability-Path-First Construction of vt(·|z) and pt(·|z)

  • The probability-path-first conditional construction supports two-sided endpoint conditioning or one-sided conditioning, using linear interpolation with a fixed small variance or its deterministic limit.With fixed conditional variance, endpoint marginals become Gaussian smoothings of the source and target distributions, approaching the desired endpoints as σ →0; when σ = 0, the path deterministically interpolates between x0 and x1.Source: §5.3.2 · pp. 165–166
  • The one-sided probability-path-first construction recovers p0 = N(0, I) and p1 = pdata, while its unique marginal-velocity minimizer equals the two-sided CFM minimizer.The one-sided loss regresses to the conditional expectation of the two-sided target given xt.Source: §5.3.2 · pp. 166–167
  • Gaussian FM connects flow matching and diffusion: with a Gaussian conditional path and linear schedule, the objectives are equivalent formulations, and the loss is simple and competitive at scale.The target marginal need not be Gaussian, and diffusion paths form a subclass of Gaussian paths.Source: §5.3.2 · pp. 167–168

5.3.3 Conditional Flow-First Construction of vt(·|z) and pt(·|z)

  • Flow-first construction specifies a conditional flow map for arbitrary endpoints, directly yielding a closed-form regression velocity and analytic, simulation-free targets, including geometry-structured spaces.Affine conditional flows recover the usual one-sided and two-sided CFM targets as special cases.Source: §5.3.3 · pp. 168, 170

5.3.4 Probability-Path-First vs. Flow-First Construction

  • Probability-path-first selects a compatible velocity from a conditional density path, whereas flow-first specifies particle trajectories and recovers the density by pushforward; the two need not select the same velocity.Path-first is natural for closed-form conditional velocities, while flow-first suits strong structural priors on trajectories.Source: §5.3.4 · pp. 170–171

5.3.5 Conditional Straightness Does Not Imply Marginal-Flow Straightness

  • Straight conditional interpolations do not imply straight marginal ODE trajectories because the endpoint-pair posterior changes with state and time; therefore, one Euler step is not generally exact.Rectified Flow changes the endpoint coupling so the induced flow can become progressively closer to straight.Source: §5.3.5 · p. 172

Example: A Simple 1D Counterexample

  • The one-dimensional Gaussian counterexample explicitly shows that affine conditional paths can induce non-affine marginal trajectories, with Figure 5.10 displaying the gap between exact ODE curves and straight-line interpolations.The example uses independent coupling with σ0 = 2 and σ1 = 3.Source: Example: A Simple 1D Counterexample · pp. 173–174, Figure 5.10 · p. 174

5.4 (Optional) Properties of the Canonical Affine Flow

  • The canonical affine flow supports Reflow, an iterative refinement scheme that progressively straightens paths between distributions while preserving their marginals.Rectification changes arbitrary endpoint pairings into model-guided dependent couplings by transporting source samples with a pretrained-model ODE.Source: §5.4 · p. 175, §5.4.1 · p. 175

5.4.1 Rectifying Flows: From Noisy Guesses to Structured Pairings

  • Rectifying flows use a pretrained diffusion-model drift to transport each source point to a model-generated endpoint, creating a coherent dependent coupling for a new conditional interpolation.This addresses curved marginal dynamics that can be difficult to integrate accurately with few numerical steps.Source: §5.4.1 · pp. 175–176
  • If the learned drift matches an oracle reference-path drift, Rectify preserves all intermediate marginals and transports the source marginal to the target.Repeated Reflow can progressively produce marginal transport that is easier to integrate with few numerical steps.Source: §5.4.1 · p. 176

5.4.2 Reflow: Iteratively Straightening Flows

  • Reflow repeatedly applies Rectify: it refits a velocity field on the current coupling, generates a new ODE-induced coupling, and yields progressively more stable transport paths.Independent pairings can create meandering trajectories that increase discretization error and simulation variance.Source: §5.4.2 · pp. 177–178

5.4.3 Properties of Reflow

Proposition 5.4.2: Reflow Straightens the Stochastic Path

  • The FM or RF formulation with linear interpolation kernels and Reflow provides a simpler training objective and a practical method for refining stochastic couplings (Liu, Gong, et al., 2022; Liu, 2022).The chapter refers to Theorem 3.7 of Liu, Gong, et al. (2022) for theoretical details.Source: Proposition 5.4.2: Reflow Straightens… · p. 179

Example: Straight Couplings Need Not Be Optimal

5.5 Closing Remarks

  • Flow Matching avoids costly ODE simulations during training by regressing against a known conditional velocity while learning deterministic transport from a Gaussian prior to data.It predefines a probability path and velocity field satisfying the continuity equation.Source: §5.5 · p. 181
  • The chapter previews a unified framework in which variational, score-based, and flow-based perspectives are mathematically equivalent and governed by the Fokker–Planck equation.These perspectives are presented as different descriptions of the same core generative principle.Source: §5.5 · p. 181

A Unified and Systematic Lens on Diffusion Models … Sampling of Diffusion Models

This section develops a unified lens on diffusion models by relating variational, score-based, and flow-based formulations through shared paths, training objectives, density dynamics, and sampling procedures. It further examines their transport interpretations, connecting diffusion processes and deterministic flows with optimal transport and Schrödinger bridges while clarifying the distinctions and conditions involved.

A Unified and Systematic Lens on Diffusion Models

  • Variational, score-based, and flow-based diffusion share a forward marginal path and a learned time-varying vector field transporting a simple prior to data.The Fokker–Planck equation constrains their common density evolution, while parameterizations and training objectives distinguish the viewpoints.Source: A Unified and Systematic Lens on… · pp. 182–183

6.1 Conditional Tricks: The Secret Sauce of Diffusion Models

  • Conditioning on x0 converts intractable marginal targets into tractable Gaussian conditional objectives that are gradient-equivalent and share the same minimizers up to a parameter-independent constant.This applies across variational, score-based, and flow-based objectives, assuming the same model class.Source: §6.1 · pp. 184–185

Example: Closed-Form Oracle Targets on a Finite Dataset

  • On a finite dataset, all three oracle targets become explicit posterior-weighted averages of closed-form conditional quantities.Perfectly fitting these empirical targets and sampling without error would reproduce the empirical training distribution, although practical capacity, optimization, and numerical errors can prevent memorization.Source: Example: Closed-Form Oracle Targets on… · pp. 187–188

6.2 A Roadmap for Elucidating Training Losses in Diffusion Models

  • The chapter organizes its training-loss analysis by extending three objectives to four parameterizations and then separating the common objective structure underlying their equivalence results.The roadmap leads from the four parameterizations to a framework for disentangling diffusion objectives.Source: §6.2 · p. 189

6.2.1 Four Common Parameterizations in Diffusion Models

  • Noise, clean-data, score, and velocity prediction all estimate conditional expectations of their respective quantities given the noisy observation x_t.These correspond to the variational, score-based, and flow-based modeling perspectives.Source: §6.2.1 · pp. 189–190

6.2.2 Disentangling the Training Objective of Diffusion Models

  • Diffusion training objectives disentangle into noise schedule, prediction target, time weighting, and time distribution choices, while affine schedules can be related by time reparameterization and spatial rescaling.The affine path x_t = α_t x_0 + σ_tϵ can be converted to canonical linear or trigonometric schedules.Source: §6.2.2 · pp. 190–191
  • Diffusion training choices reshape effective weighting over noise levels: prediction–loss conversions introduce time-dependent factors, while time sampling and explicit loss weighting jointly determine ptime(t)ω(t).Importance sampling can preserve a fixed objective while changing only the Monte Carlo estimator, whereas changing the objective alters effective training emphasis.Source: §6.2.2 · pp. 192, 193, 195
  • When the effective log-SNR weighting w(Λ) is non-increasing, the weighted diffusion objective becomes, up to parameter-independent terms, an average negative ELBO over Gaussian-perturbed data.Monotonicity makes the weighting over ELBOs nonnegative across corruption levels.Source: §6.2.2 · p. 194

6.3 Equivalence in Diffusion Models

  • At the oracle level, clean-data, noise, score, and velocity predictions contain the same information and can be converted algebraically, although finite networks may behave differently across parameterizations.These choices can alter represented functions, optimization, noise-level weighting, and empirical sampling behavior.Source: §6.3 · p. 196, Proposition 6.3.1: Equivalence of… · p. 197

6.3.1 Four Prediction Types Are Equivalent

  • Squared-error objectives in different prediction types share the same pointwise oracle minimizer up to positive time-dependent factors, which control relative noise-level weighting.Thus, equivalent oracle targets do not imply identical learned models or objectives.Source: §6.3 · p. 196, Proposition 6.3.1: Equivalence of… · p. 198

6.3 Equivalence in Diffusion Models

  • Different deterministic and stochastic dynamics can share the same marginal density path even when their sample trajectories differ, because the density evolution follows the corresponding continuity or Fokker–Planck equation.The section connects these equivalent dynamics to sampling through the PF-ODE and its discretization.Source: §6.3 · p. 196

Proposition 6.3.1: Equivalence of Parametrizations

  • The four optimal parameterizations are equivalent oracle views: given xt and the forward schedule, any one determines the others, including the PF-ODE velocity.A trained model can convert one chosen network output into the other parameterizations, but direct prediction and loss evaluation remain separate design choices.Source: Proposition 6.3.1: Equivalence of… · pp. 197–198

Proposition 6.3.2: PF-ODE in Different Parameterizations

  • The PF-ODE has equivalent score, noise, denoised, and velocity forms, but parameterization affects vector-field stiffness, discretization error, optimization, and solver suitability.The text notes that ε or x prediction often aligns with fast ODE solvers, while x or v prediction can support few-evaluation generators.Source: §6.3.2 · p. 199, Proposition 6.3.2: PF-ODE in Different… · p. 199

6.3.3 All Affine Flows Are Equivalent

  • Any affine interpolation can be expressed as a time-reparameterized and rescaled version of the canonical flow used in FM (Lipman et al., 2022) and RF (Liu, 2022).The canonical interpolation has constant velocity ε − x0, but that simplicity is not essential.Source: §6.3.3 · p. 200
  • Affine flows are equivalent up to scalar rescaling and, when the transformed time is continuous and strictly monotone, time reparameterization.For x_t = α_tx_0 + σ_tϵ, the canonical representation uses c(t) = α_t + σ_t and τ(t) = σ_t/(α_t + σ_t).Source: §6.3.3 · p. 200, Proposition 6.3.3: Equivalence of… · pp. 201–202

Proposition 6.3.3: Equivalence of Affine Flows

  • Score, noise, clean, and velocity training objectives are theoretically equivalent up to time-dependent weights, with velocity requiring an affine head conversion involving x_t.The conversions are determined by the schedule coefficients (α_t, σ_t).Source: Proposition 6.3.3: Equivalence of… · pp. 202–203

6.3.4 (Optional) Analysis of Parametrizations and the Canonical Flow

  • Within the affine interpolation family, the linear schedule minimizes conditional velocity energy, but marginal oracle-velocity energy is minimized only when traversal speed accounts for position-dependent velocity magnitude.The energy-minimizing schedule moves more slowly where the oracle velocity is larger and faster where it is smaller, so ζ(t) = t is not universally optimal for the marginalized velocity.Source: §6.3.4 · pp. 205–206
  • The canonical linear schedule with v-prediction gives constant-scale regression targets, removes schedule-curvature terms, and provides the velocity directly for PF-ODE integration.It avoids endpoint collapse or blow-up from the regression signal under controlled gradient factors, while a single network evaluation supplies the quantity used by a standard ODE solver.Source: §6.3.4 · pp. 204, 207

6.4 Beneath It All: The Fokker–Planck Equation

  • The variational, score-based, and flow-based perspectives arise from the continuity/Fokker–Planck equation: discrete variational kernels consistently discretize SDE dynamics, while ODE flows and SDEs can share the same density path.A velocity field defining a generative ODE can be transformed into a stochastic counterpart, yielding a continuous family from deterministic flows to stochastic sampling with the same data-distribution path.Source: §6.4 · pp. 208–209, §6.4.1 · p. 209

6.4.1 Connection of Flow-Based Approach and Score SDE

  • Reverse-time SDEs and the PF-ODE generate the same prescribed one-time marginal densities, so integrating the ODE backward from the prior yields data samples.The equivalence follows because both dynamics satisfy the same Fokker–Planck evolution for the density path.Source: Question 6.4.1 · p. 209, §6.4.1 · p. 209, §6.4 · p. 210

6.4. Beneath It All: The Fokker–Planck Equation

  • At the oracle level, arbitrary γ(t) changes stochastic trajectories but cancels from the Fokker–Planck equation, preserving the same marginal path across PF-ODE and reverse-time SDEs.The score s*(x, t) = ∇x log pt(x) links the velocity and reverse-time SDE descriptions; Ma et al. (2024) also study choices of γ(t) that reduce the KL gap.Source: §6.4 · pp. 210–211

6.5. Closing Remarks

  • The unified theory identifies conditioning as the shared device that makes marginal training objectives tractable and the Fokker–Planck equation as the universal law governing density evolution.Variational, score-based, and flow-based approaches therefore construct dynamics whose marginals follow a common probability path.Source: §6.5 · p. 212
  • Oracle noise, clean-data, score, and velocity predictions are algebraically interchangeable, but finite networks, loss weighting, and numerical solvers can make their practical training and sampling behavior differ.The continuous theories are equivalent at the density level, while implementations may approximate targets and dynamics differently.Source: §6.5 · p. 212

(Optional) Diffusion Models and Optimal Transport

  • Diffusion models can be viewed as stochastic SDE processes or deterministic PF-ODE flows, but only the stochastic view directly aligns with entropic optimal transport; the PF-ODE has no general known transport objective.A standard diffusion forward/reverse SDE does not generally solve a prescribed two-endpoint Schrödinger bridge, while the PF-ODE does not generally coincide with the quadratic-cost optimal transport map.Source: (Optional) Diffusion Models and… · pp. 214–215

7.1 Prologue of Distribution-to-Distribution Translation

  • Distribution-to-distribution translation seeks cost-efficient mappings between arbitrary source and target distributions, motivating optimal transport and its entropic Schrödinger-bridge variant.One-endpoint, two-endpoint, and flow-matching diffusion methods provide practical translation mechanisms, while this section focuses on their mathematical foundations.Source: §7.1 · p. 216

Question 7.1.1

  • The section distinguishes OT’s deterministic continuity-equation flow from Schrödinger bridges’ controlled Fokker–Planck diffusion and uses this taxonomy to assess diffusion models’ claims of optimality.Its central question is whether diffusion connects p_data and p_prior, or arbitrary endpoints, through an optimal transport formulation.Source: Question 7.1.2 · pp. 216–217, Question 7.1.1 · p. 216

7.2 Taxonomy of the Problem Setups

  • Classical optimal transport selects a coupling between source and target distributions that minimizes an assigned transport cost, with quadratic cost yielding the Wasserstein-2 problem.A coupling has the source and target distributions as marginals, and a deterministic map is a special coupling when it pushes the source distribution to the target.Source: §7.2.1 · pp. 218–219

7.2.1 Optimal Transport (OT)

  • Optimal transport transforms psrc into ptgt via the Monge map, while its Benamou–Brenier formulation finds a smooth velocity field minimizing kinetic energy along continuous distribution paths.For quadratic cost, McCann’s displacement interpolation moves mass along straight lines with constant velocity.Source: §7.2.1 · pp. 220–221

7.2.2 Entropy-Regularized Optimal Transport (EOT)

Why Entropy Regularizer Helps?

  • Entropy regularization favors diffuse couplings, and as ε →0 the regularized plan approaches a possibly singular classical OT coupling.Increasing ε produces more diffuse plans, whereas decreasing ε concentrates transport toward classical OT behavior.Source: Why Entropy Regularizer Helps? · p. 223

7.2.3 Schrödinger Bridge (SB)

  • Schrödinger bridges find the trajectory law closest in KL divergence to a reference diffusion while matching arbitrary endpoint marginals, equivalently minimizing expected control energy over admissible drifts.The optimal forward drift correction g^2∇x log Ψ and reverse correction −g^2∇x log bΨ steer the reference process while preserving the same optimal path law and intermediate marginals.Source: §7.2.3 · pp. 224–228
  • Directly solving the coupled PDEs for Schrödinger-bridge generation is difficult, motivating score-SDE, surrogate-bound, and simulation-free alternatives.These obstacles persist even in low-dimensional settings.Source: §7.2.3 · p. 228

7.2.4 Global Pushforwards and Local Dynamics: An OT Analogy for DGMs

  • Unlike GANs and VAEs, which learn global pushforward maps, diffusion models learn time-dependent local vector fields that continuously transport a prior toward data.This dynamic view parallels the relationship between optimal transport and Schrödinger-bridge formulations.Source: §7.2.4 · p. 228

7.3 Relationship of Variant Optimal Transport Formulations

  • Stochastic-control and path-space KL formulations of SBε are equivalent, its endpoint projection yields EOTε, EOTε converges to static OT, and rescaled SB converges to dynamic OT as ε →0.These relationships organize stochastic-control, entropy-regularized, and classical optimal-transport frameworks.Source: §7.3 · pp. 229–230

7.3.1 SB and EOT are (Dual) Equivalent

  • SB and EOT are equivalent up to constants: optimal SB paths induce optimal endpoint couplings, while EOT couplings combined with reference-conditioned paths recover the optimal SB process.For general references, transition likelihoods determine endpoint costs, with unlikely transitions receiving larger costs.Source: §7.3.1 · pp. 231–232

7.3.2 EOTε is Reduced to OT as ε →0

  • As ε →0, EOTε optimal values converge to OT values, limiting entropic plans are OT-optimal, and plans converge to the unique OT optimizer when it exists.Because EOTε admits efficient Sinkhorn solutions, it provides a tractable proxy for classical OT under the stated assumptions.Source: §7.3.2 · pp. 232–233

7.3.3 SBε is Reduced to OT as ε →0

  • As ε →0, appropriately rescaled SB converges to the Benamou–Brenier dynamic OT problem, with marginals converging under suitable assumptions and uniquely minimizing dynamics converging to the OT solution.The diffusion term vanishes, formally reducing the controlled SDE and Fokker–Planck equation to deterministic transport and continuity equations.Source: §7.3.3 · pp. 234–235

7.4.1 Diffusion models as a Special Case of Schrödinger Bridges

  • Diffusion models arise as the zero-control limit of Schrödinger bridges when Ψ(x, t) ≡1, yielding the reference process forward and Anderson’s reverse-time SDE backward.Full SB can bridge arbitrary endpoints using learned potentials, whereas diffusion models typically fix an approximately Gaussian prior endpoint and learn only the reverse score.Source: §7.4.1 · p. 236

7.4.2 Diffusion Models as Schrödinger Half-Bridges

  • Diffusion models are Schrödinger half-bridges because they enforce only one endpoint constraint, unlike full bridges that match both prescribed source and target distributions.The half-bridge viewpoint naturally applies when the endpoints are the prior and data distributions.Source: §7.4.2 · p. 237
  • Diffusion models can be viewed as Schrödinger half-bridges, matching one endpoint while remaining close to a reference process; alternating forward and backward half-bridges approximates the full bridge.The forward half-bridge fixes psrc, the backward half-bridge fixes the terminal distribution, and iterative combination approximates the full Schrödinger bridge.Source: §7.4.2 · p. 237

7.4. Is Diffusion Model’s SDE Optimal Solution to SB Problem?

  • Standard diffusion forward processes generally do not match the prescribed Gaussian prior exactly at finite time, whereas Schrödinger bridges enforce pT = pprior through an additional control drift.At finite T, the diffusion terminal marginal still depends on pdata; the Schrödinger Bridge framework achieves exact matching regardless of p0 = pdata.Source: §7.4.2 · p. 237, §7.4 · p. 238
  • Diffusion Schrödinger Bridge (DSB) (De Bortoli et al., 2021) uses iterative proportional fitting to alternate endpoint matching until convergence to a bridge satisfying both marginals.The limiting process satisfies P*0 = pdata and P*T = pprior, with convergence proved under mild conditions, but repeated score-matching projections are computationally demanding.Source: §7.4 · pp. 238–239

7.5.1 PF-ODE Flow Is Generally Not Optimal Transport

  • PF-ODE flow maps are not generally quadratic-cost optimal transport maps, including for Gaussian endpoints and for broader initial-distribution classes.Lavenant and Santambrogio (2022) construct an initial distribution for which some intermediate residual map is non-optimal; the contradiction uses failure of a required Hessian-Jacobian commutation condition.Source: §7.5.1 · pp. 240, 241, 243

Question 7.5.1

  • Independent-coupling linear interpolation does not recover the OT map, although coupling a linear path with source and target distributions yields a practical upper bound on OT cost.The question’s answer is no, but the linear path gives the tightest upper bound within the conditional-flow family for a fixed coupling.Source: Question 7.5.1 · p. 244
  • Within conditional flows with fixed endpoint coupling, canonical linear interpolation minimizes the path-energy upper bound on quadratic-cost OT kinetic energy.This result follows from the conditional-flow restriction and yields the tightest version of the bound; Liu (2022) and Lipman et al. (2024) provide the practical formulation.Source: Question 7.5.1 · pp. 244–245
  • Rectified-flow kernels minimize an upper bound on true kinetic energy for any chosen coupling, but this conditional optimality does not ensure marginal global optimality.The straight-line paths can be retained; inefficiency may instead arise from the initial independent pairing, motivating Reflow’s cost-non-increasing iterations (Liu, Gong, et al., 2022).Source: Question 7.5.1 · p. 245

Question 7.5.2

  • Potential rectification does not generally recover optimal transport: disconnected intermediate supports admit nonoptimal fixed points, while Gaussian endpoints admit arbitrarily small loss with arbitrarily large deviation.These counterexamples from Hertrich et al. (2025) limit rectified flows as reliable optimal-transport solvers, although they may still produce strong generative models.Source: Question 7.5.2 · p. 247

7.6. Closing Remarks

  • Diffusion PF-ODEs define deterministic couplings, but generally not quadratic-cost optimal transport maps; learned diffusion flows instead provide flexible high-dimensional transport alongside classical constructions.Classical alternatives include monotone rearrangement, Brenier maps, Knothe–Rosenblatt rearrangements, and Dacorogna–Moser transports, each requiring structural assumptions or substantial analytical machinery.Source: §7.6 · p. 248

Sampling of Diffusion Models

  • Sampling generates data by solving the learned velocity-field ODE backward from t = T to 0, starting with x(T) sampled from the prior.The same formulation supports integrating backward more generally from a later time s to an earlier time t, with s > t.Source: Sampling of Diffusion Models · p. 250

Guidance and Controllable Generation

This section presents guidance and controllable generation as methods for steering diffusion trajectories through conditional-score decomposition, classifier-based and classifier-free guidance, and training-free objectives. It covers conditioning strength, task-guided sampling in data and noise space, and preference alignment through preference tilting and Diffusion-DPO.

Guidance and Controllable Generation

  • Conditional generation steers an unconditional model or learns p0(x|c), while guidance decomposes the conditional score into an unconditional direction plus a condition-directed term.The framework distinguishes control, which meets the condition, from alignment, which also meets human preferences, and treats preference incorporation within the same guidance framework.Source: Guidance and Controllable Generation · p. 251

8.1 Prologue

  • Exact conditional reverse dynamics add ∇xt log pt(c|xt) to the unconditional score, whereas classifier, classifier-free, and training-free methods approximate this guidance direction differently.Classifier guidance learns the term directly; classifier-free guidance uses conditional–unconditional score differences; training-free methods use task-specific surrogate directions because the relevant posterior average is generally intractable.Source: §8.1 · pp. 253–255
  • Diffusion-DPO fine-tunes conditional diffusion models from pairwise preferences, learning the preference tilt without a separate reward model, unlike the reward-model and reinforcement-learning stages of RLHF.Preference strength is represented by a scalar rating and a reweighting parameter, while control alone can produce on-condition but off-preference samples.Source: §8.1 · p. 256

8.2.1 Foundation of Classifier Guidance

  • Classifier guidance applies Bayes’ decomposition by training a time-conditional classifier pψ(c|xt,t) whose input gradient supplies the condition-directed score correction.At guidance scale ω = 1, the oracle field recovers the exact conditional score; general scales have a fixed-time tempered interpretation rather than necessarily defining the sampler’s marginal path.Source: §8.2.1 · p. 257

8.2. Classifier Guidance

  • Classifier guidance uses ω > 1 to amplify conditional fidelity, typically trading away diversity, while weaker guidance increases diversity but weakens conditioning.At inference, the learned classifier gradient is added to the unconditional score and scaled by ω to steer samples toward condition c.Source: §8.2 · p. 258, §8.2.2 · p. 258

8.2.3 Advantages and Limitations

  • Classifier guidance works with any pretrained unconditional diffusion model but requires a separately trained classifier that is costly across noise levels, must handle severe corruption, and may misalign with the diffusion distribution.The classifier is trained only for sampling-time steering and must generalize to noisy inputs across the diffusion path.Source: §8.2.3 · p. 259

8.3.1 Foundation of Classifier-Free Guidance

  • Classifier-free guidance removes the separate classifier by expressing the guidance direction through the difference between conditional and unconditional scores, with ω interpolating or extrapolating between them.For 0 ≤ ω ≤ 1, the field interpolates between conditional and unconditional scores; for ω > 1, it extrapolates beyond the conditional score in the conditional direction.Source: §8.3.1 · p. 260, Figure 8.2 · p. 260
  • Classifier-free guidance interpolates unconditional and conditional scores: ω = 0 ignores conditioning, ω = 1 gives the conditional score, and ω > 1 strengthens alignment but typically reduces diversity.With exact scores, the guided field matches the corresponding algebraic formulation; for ω ≠ 1, its fixed-time tilted-density interpretation differs from the marginals generated by guided dynamics.Source: §8.3.1 · pp. 260–261

8.3.2 Training and Sampling of CFG

  • CFG jointly trains conditional and unconditional predictions in one model by randomly replacing the condition with a null token, while introducing guidance weight ω only during sampling.The null condition is used with user-specified probability p_uncond, and sampling combines the learned conditional and unconditional score estimates in the reverse-time SDE or probability-flow ODE.Source: §8.3.2 · p. 261

8.3. Classifier-Free Guidance

  • CFG enables controllable generation through a simple post-training trade-off between conditioning strength and sample diversity without a separately trained classifier or unconditional diffusion model.Source: §8.3 · p. 262

8.4 Training-Free Guidance

  • Training-free guidance alters sampling dynamics of a fixed pre-trained diffusion model using an external condition, loss, or measurement objective, adding a condition-dependent correction without retraining.Task losses can modify either the predicted clean sample or the score; examples include perceptual distances and feature-space distances or negative similarities from representations such as CLIP.Source: §8.4 · p. 263

8.4.1 Conceptual Framework for Training-Free Guidance

  • Training-free guidance distinguishes data-space corrections to the predicted clean sample from noise-space corrections to the score used in the sampling update.The framework assumes score and x-prediction parameterizations, with analogous treatment possible for ϵ-prediction.Source: §8.4.1 · pp. 263–264, §8.4 · p. 263

Prologue, … Coordinate Choice Is Part of the Numerical Method

This section develops training-free guidance, preference alignment, and acceleration for diffusion models, framing sampling through PF-ODE integration and related iterative restoration methods. It examines solver design, parameterization, coordinate choice, schedule-aware updates, DDIM’s deterministic and stochastic variants, and the connection between DDIM trajectories and conditional flow matching.

Prologue,

  • Guidance can modify data estimates or sampling scores, but the resulting dynamics need not produce the intended marginal distributions across noise levels.Data-space guidance takes loss-decreasing steps on clean estimates, whereas noise-space guidance differentiates through the x-prediction and can be computationally costly.Source: Prologue · pp. 264–265

8.4.2 What Distribution Does Guided Sampling Produce?

  • Exact classifier guidance with weight 1 recovers the conditional marginal path, whereas generic losses and schedules may violate cross-time Fokker–Planck consistency even with exact scores and ODE integration.A fixed-time tilted density does not by itself define the marginals generated by integrating the guided dynamics; ω ≠ 1 alone does not imply inconsistency.Source: §8.4.2 · pp. 265–266

8.4.3 Examples of Training-Free Approaches to Inverse Problems

  • Pre-trained diffusion models solve inverse problems by combining a learned clean-data prior with measurement information introduced during sampling through consistency corrections or likelihood-score approximations.ILVR enforces low-frequency measurement consistency in sample space, while DPS approximates the likelihood score using the denoised posterior mean under a peaked-posterior assumption.Source: §8.4.3 · pp. 267–268
  • InDI learns gradual restoration from paired corrupted and clean data, predicting the clean target from intermediate states; its continuous-time limit is a residual-flow ODE rather than one-shot prediction.The Bayes-optimal squared-error predictor is F*(x_t,t)=E[x_0|x_t], and refinement proceeds iteratively from the corrupted observation.Source: §8.4.3 · pp. 269–270
  • Under Gaussian denoising, InDI’s residual-flow ODE reduces exactly to the diffusion probability-flow ODE, giving diffusion an interpretation as supervised denoiser learning followed by continuous-time refinement.The equivalence uses x_t=α_t x_0+σ_t ε with α_t=1 and σ_t=tσ, together with Tweedie’s formula.Source: §8.4.3 · pp. 270–271

8.5 From Reinforcement Learning to Direct Preference Optimization for Model Alignment

  • DPO is introduced as a more streamlined and stable alternative to RLHF, avoiding explicit reward modeling and reinforcement learning; Diffusion-DPO extends it to diffusion models.RLHF is described as unstable, computationally expensive, dependent on multiple large models, and vulnerable to reward-model misspecification, motivating direct optimization on preference data.Source: §8.5 · p. 272, §8.5.1 · p. 272, Question 8.5.1 · p. 272

Question 8.5.1

  • DPO directly fits preference pairs with a logistic loss, preserving KL regularization toward a reference policy while avoiding explicit reward modeling and reinforcement-learning rollouts.Its implicit reward is the reference-normalized log-probability ratio; constants cancel in pairwise comparisons, producing a stable classification-style objective.Source: Question 8.5.1 · p. 272, §8.5.3 · pp. 276–277

8.5.2 RLHF: Bradley-Terry View

  • RLHF trains a reward model from preference pairs and then optimizes a policy toward higher reward under a KL penalty that keeps it close to a pretrained reference.The Bradley–Terry formulation is shift-invariant and increases the winner–loser score gap; the reward objective can also be interpreted as matching empirical human win probabilities.Source: §8.5.2 · pp. 273–275

8.5.4 Diffusion-DPO

  • Diffusion-DPO moves preference alignment from endpoint likelihoods to reverse-process paths because diffusion likelihood evaluation is computationally intensive and trajectory differentiation can produce unstable gradients.The pathwise objective retains the endpoint reward while regularizing latent denoising trajectories against the reference process.Source: §8.5.4 · p. 277
  • The exact pathwise Diffusion-DPO objective is impractical because endpoint conditioning creates an intractable diffusion-bridge posterior, nested θ-dependent expectations yield high-variance gradients, and long chains require costly backpropagation.Computing O(T) log-densities for T ∼102–103 across winner and loser paths is memory- and compute-heavy.Source: §8.5.4 · pp. 278–279, §3 · pp. 279–280

3. Long Chains, Large Sums, and Expensive Backpropagation. In Rπθ(c, x0:T ), computing

  • Diffusion-DPO replaces intractable endpoint-conditioned path expectations with a tractable single-step DPO-style loss whose margin is a denoising-error difference.Jensen’s inequality and the reverse-process Markov property reduce path-level preference optimization to per-step contributions, while forward-process posteriors provide tractable transition estimates.Source: §3 · pp. 279–281

8.6. Closing Remarks

  • The chapter closes by presenting DPO and Diffusion-DPO as direct, stable alternatives for human-preference alignment, while identifying iterative sampling latency and computational cost as the next challenge.The following chapter will use the ODE interpretation of sampling to study numerical solvers that reduce the required number of steps.Source: §8.6 · p. 282

Numerical Solvers for Fast Diffusion Sampling

  • Diffusion sampling is slow because naive reverse-time integration may require hundreds or thousands of neural-network evaluations, motivating training-free numerical solvers.Training-based alternatives use distillation or direct few-step training, while this chapter keeps the pretrained model fixed.Source: Numerical Solvers for Fast Diffusion… · p. 283, §9.1.1 · p. 285

Message

  • Training-free fast sampling is largely a numerical integration problem: solvers approximate the PF-ODE trajectory with as few model evaluations as possible.The chapter organizes named samplers as different approximations to the same PF-ODE trajectory.Source: Message · p. 285, Question 9.1.1 · p. 286

9.1.1 Fast Sampling as Numerical Integration

  • The empirical PF-ODE generates samples by integrating backward from prior noise, discretizing the reverse-time interval so the final iterate approximates the learned flow map.The unknown PF-ODE integral is the common target of training-free solvers.Source: Message · pp. 285–286, Question 9.1.1 · p. 286

Question 9.1.1

  • Solver design separates prediction parameterization, time parameterization, and numerical approximation, while single-step, multistep, and parallel fixed-point methods differ in how they obtain model evaluations.Sampling cost is measured by function evaluations; multistep methods can often advance with one new evaluation after initialization.Source: Question 9.1.1 · pp. 286–287

9.1.2 Design of Diffusion-Specific Solvers

  • Diffusion-specific solvers retain analytically known noise-schedule terms while numerically approximating only neural predictions, enabling semilinear and exponential-integrator formulations.Factored updates freeze the neural prediction but preserve schedule dependence, unlike plain Euler, while using one neural evaluation per step.Source: §9.1.2 · pp. 288, 290, 291, 292
  • Score, noise, clean-data, and velocity parameterizations represent the same PF-ODE when algebraically converted from one model, but direct score prediction can become poorly scaled near small noise levels.If separate networks are trained for different targets, empirical PF-ODE differences can reflect both model approximation and numerical discretization.Source: §9.1.2 · pp. 288–291
  • Diffusion-specific solvers separate analytically known schedule-dependent dynamics from neural-prediction variation, reducing discretization error to the latter’s trajectory dependence.This principle motivates the solver designs developed later in the chapter.Source: §9.1.2 · p. 292

9.1.3 Roadmap from Numerical Ideas to Diffusion Samplers

  • The roadmap organizes samplers by classical numerical ideas: DDIM uses first-step predictions, DPM-Solver and DPM-Solver++ use higher-order evaluations or reused predictions, and ParaDiGMS jointly refines multiple times.Diffusion-specific structure includes prediction parameterizations, known schedule coefficients, and alternative time parameterizations.Source: §9.1.3 · p. 293

9.2 DDIM

Corollary 9.2.1: Equivalent Forms of the Deterministic DDIM Update

  • DDIM keeps a consistent start-time clean/noise pair fixed, so algebraically equivalent noise and clean-data parameterizations yield the same finite-step update.The induced trajectory recombines the fixed estimates using known coefficients ατ and στ without additional model evaluations.Source: Corollary 9.2.1: Equivalent Forms of… · pp. 297–298
  • Directly freezing the converted score or complete velocity generally gives a different first-order method because those quantities vary with time along the DDIM trajectory.Thus finite-step freezing does not generally commute with time-dependent prediction conversion.Source: Corollary 9.2.1: Equivalent Forms of… · pp. 298–299

9.2.2 Relation to Euler-Type Methods

  • DDIM and plain Euler agree to first order but differ by O((∆s)2) over finite steps, except when ατ and στ are affine, because DDIM preserves finite schedule evolution analytically.DDIM can nevertheless be viewed as Euler after a schedule-aligned change of coordinates; plain Euler holds the complete PF-ODE velocity fixed.Source: §9.2.2 · pp. 300–301
  • DDIM, exponential Euler, and plain Euler are globally first-order, one-evaluation methods whose accuracy depends on the quantity held fixed, so no universal error ordering exists.DDIM is exact when the neural noise prediction is constant along the exact trajectory and is especially suited to slowly varying predictions with appreciably changing diffusion schedules.Source: §9.2.2 · p. 303

Coordinate Choice Is Part of the Numerical Method

  • DDIM is exactly explicit Euler after normalizing state by α and using the schedule-aligned noise-to-signal ratio ρ = σ/α as time, showing that coordinate choice is part of the numerical method.The same construction under clean-data prediction uses state normalization by σ and time coordinate α/σ; at first order, DPM-Solver-1 is exactly DDIM.Source: §9.2.2 · p. 304, Coordinate Choice Is Part of the… · pp. 305–306

9.2.3 (Optional) The Original Variational Perspective on DDIM

  • DDIM exploits the fact that denoising marginals do not uniquely determine temporal couplings, allowing skipped-time transitions and a continuum from DDPM posterior sampling to deterministic DDIM.The free transition-noise coefficient preserves each target marginal; setting it to zero yields deterministic DDIM, while the DDPM posterior is one particular marginal-consistent choice.Source: §9.2.3 · pp. 306–309
  • The conditional Gaussian trajectory is exact for fixed (x0, ϵ), and retaining the same ϵ yields an exact finite-time map between states.The map follows by eliminating ϵ from xs = αsx0 + σsϵ, while remaining conditioned on x0.Source: §9.2.3 · p. 310
  • Because x0 is unknown during generation, substituting its start-time model prediction into the conditional map gives the practical deterministic DDIM update.The conditioned map itself is not yet a sampler because it assumes access to x0.Source: §9.2.3 · p. 310
  • Differentiating xt = αtx0 + σtϵ defines the conditional instantaneous velocity for the Gaussian probability path used in conditional flow matching.The velocity can also be expressed as a function of the current state.Source: §9.2.3 · p. 311
  • Averaging the conditional velocity over posterior uncertainty about x0 at the current state produces v∗(x, t), exactly the oracle marginal PF-ODE velocity.The posterior conditional mean satisfies x∗(x, t) = E[x0|xt = x].Source: §9.2.3 · p. 311

DDIM and CFM · Approximating ˆϵ(k)

This section connects DDIM, CFM, DEIS, and DPM-Solver through conditional Gaussian paths and log-SNR time, deriving updates that approximate neural-prediction variation. It shows how DDIM emerges as a first-order case while higher-order multistep and Runge-Kutta-style schemes improve approximation through staged evaluations and log-SNR-based timestep placement.

DDIM and CFM

  • DDIM and CFM are exact views of the same conditional Gaussian path, but a finite DDIM step is not generally the exact marginal PF-ODE flow because DDIM freezes the step-start prediction.The marginal PF-ODE velocity changes continuously with the evolving state, whereas DDIM keeps the prediction inferred at the beginning of the step fixed.Source: DDIM and CFM · p. 311

9.3 DEIS

  • DEIS is an Adams–Bashforth-type multistep method that reuses previous model predictions without additional within-step evaluations, while retaining the known diffusion-dependent weight analytically.Its polynomial extrapolates recent prediction variation into the next step, improving on DDIM’s constant approximation; very large degrees can be numerically sensitive, so small degrees are preferred.Source: §9.3 · p. 312, Example: Special Cases of Lagrange… · p. 317

9.3.1 Background: Polynomial Extrapolation

  • The Lagrange construction interpolates stored model outputs exactly at their anchor times and extrapolates the resulting polynomial beyond the latest anchor to predict the next solver interval.With n+1 anchors, a degree-at-most-n polynomial provides the simplest curve matching all observed values; in DEIS, those anchors are neural predictions from completed sampling steps.Source: §9.3.1 · pp. 312–314

9.3.2 DEIS: Lagrange Polynomial Approximation of the PF-ODE Integral

  • DEIS approximates the unknown neural prediction over each reverse-time interval with a degree-n Lagrange polynomial built from past outputs, then substitutes it into the factored exponential-integrator update.The method uses all available anchors during warm start, increasing the degree from constant to linear, quadratic, and eventually the target degree as history accumulates.Source: §9.3.2 · pp. 314, 316
  • Under smoothness and bounded-step-ratio assumptions, full-history degree-n DEIS has formal order n + 1 with one-step local defect O(hn+2).During warm start, degree i−1 yields local defect O(hi+1) and formal order i; the order reaches n+1 after n+1 anchors, but global order additionally requires stability and accurate starting values.Source: §9.3.2 · p. 315, Example: Special Cases of Lagrange… · p. 317

9.3.3 DDIM = AB-DEIS-0

  • AB-DEIS-0 and deterministic DDIM give exactly the same finite-step update, while higher-degree DEIS extends this degree-zero, first-order method using past model evaluations.DDIM holds the neural prediction fixed; higher-degree DEIS replaces it with polynomial extrapolation from additional past evaluations while retaining the analytic diffusion treatment.Source: §9.3.3 · p. 318

9.4 DPM-Solver

  • DPM-Solver develops higher-order diffusion solvers by approximating neural-prediction variation in half-log signal-to-noise-ratio time rather than treating it as constant.The log-SNR coordinate is related to DDIM’s schedule-aligned clock by ρt = e−λt and converts the model-dependent term into an exponentially weighted integral.Source: §9.4 · p. 319

Proposition 9.4.1: Exponentially Weighted Exact Solution

  • In log-SNR time, locally Taylor-approximating the neural prediction makes the resulting exponentially weighted terms analytically integrable, yielding explicit higher-order update coefficients.This retains known schedule dependence analytically and concentrates numerical approximation on the neural prediction.Source: Proposition 9.4.1: Exponentially… · p. 321
  • Uniform spacing in λ redistributes numerical resolution according to signal-to-noise progress, producing smaller t-steps near diffusion endpoints and larger steps near the trajectory middle.For the linear schedule (αt, σt) = (1 −t, t), the induced grid has finer spacing at small noise and coarser spacing at large noise.Source: Proposition 9.4.1: Exponentially… · p. 322

9.4.2 Estimating the Integral with Taylor Expansion

  • Taylor expansion reduces the numerical problem to approximating total derivatives of the neural prediction along the empirical PF-ODE trajectory, with analytically computable log-SNR coefficients.DPM-Solver avoids explicit network differentiation by estimating derivatives from additional model evaluations, while truncation yields formal higher-order updates.Source: §9.4.2 · pp. 323–324

Example: DPM-Solver-1

  • DPM-Solver-1 results from dropping the O(h2) remainder, and this update is exactly the deterministic DDIM step.The first-order construction therefore provides the baseline from which higher-order DPM-Solver variants improve.Source: Example: DPM-Solver-1 · p. 324

9.4.3 Implementation of DPM-Solver-n

  • DPM-Solver-n implementations precompute log-SNR-dependent coefficients and approximate higher-order trajectory derivatives using additional evaluations, commonly through finite differences.This avoids the computational cost of exact higher-order derivatives.Source: §9.4.3 · p. 324, Example: DPM-Solver-1 · p. 324

Approximating ˆϵ(k)

  • DPM-Solver-2 uses two model evaluations and achieves second-order accuracy for any fixed γ ∈(0, 1].The choice of γ changes the stage location and error constant without changing the formal second-order accuracy.Source: Approximating ˆϵ(k) · p. 326

9.4.4 DDIM = DPM-Solver-1

  • For any fixed diffusion schedule, deterministic DDIM and DPM-Solver-1 produce exactly the same finite-step update.They are two derivations of the same first-order approximation; log-SNR becomes consequential only beyond first order.Source: §9.4.4 · pp. 327–328

9.4.5 Relation of DPM-Solver to Runge-Kutta-Type Solvers

  • Higher-order DPM-Solvers follow Runge-Kutta staging but retain known diffusion-dependent factors analytically, applying additional evaluations only to the learned noise-prediction variation.Compared with exponential midpoint RK2, DPM-Solver-2 also retains σλ analytically and agrees through second order.Source: §9.4.5 · pp. 328, 329, 331

DPM-Solver and Runge–Kutta Methods … Question 11.1.1

This section develops high-order, parallel, and distilled methods for accelerating diffusion sampling, including DPM-Solver variants, Runge–Kutta and multistep schemes, Picard iteration, and progressive step halving. It also examines flow-map models and training objectives for one- or few-step generation, with and without pretrained teachers.

DPM-Solver and Runge–Kutta Methods

  • DPM-Solver combines within-step Runge–Kutta staging with analytically retained diffusion-dependent weights; DPM-Solver-1 is exactly DDIM.Higher-order variants use additional within-step evaluations to capture neural-prediction variation, but are not identical to finite-step classical Runge–Kutta formulas.Source: DPM-Solver and Runge–Kutta Methods · p. 331

9.5.1 From DPM-Solver to DPM-Solver++ for Guided Sampling

  • DPM-Solver++ targets strong-guidance instability by using data prediction, enabling data-space corrections, and by recycling model evaluations for smaller effective steps under a fixed NFE budget.Strong guidance can amplify outputs and derivatives, narrow high-order solvers’ convergence regions, and push predictions outside the training-data range.Source: §9.5.1 · p. 332

9.5.2 DPM-Solver++’s Methodology

  • DPM-Solver++ preserves analytic schedule-dependent evolution while approximating only the data prediction in log-SNR time, yielding single-step 2S and multistep 2M second-order realizations.Without thresholding, data-prediction conversion is algebraically equivalent to the same empirical PF-ODE; thresholding is a separate model-output modification.Source: §9.5.2 · pp. 332–333

9.5.3 DPM-Solver++ Single-Step by Taylor Expansion

  • DPM-Solver++(2S) reduces to data-prediction DDIM at first order and achieves second-order accuracy by adding a midpoint model evaluation within each step.The midpoint two-stage update differs from the direct second-order Taylor or finite-difference construction by O(h^3).Source: §9.5.3 · p. 334

9.5.4 DPM-Solver++ Multistep by Recycling History

  • DPM-Solver++(2M) estimates local data-prediction variation from previous completed steps, requiring one new model evaluation per step after warm start while preserving second-order accuracy.Under smoothness and bounded step-ratio assumptions, the practical simplification changes the update by O(h^3).Source: §9.5.4 · pp. 335, 337

9.5.5 Relation of DPM-Solver++(2M) to DEIS

  • DPM-Solver++(2M) is the data-prediction/log-SNR analogue of degree-one DEIS extrapolation, although its practical coefficients differ while agreeing through second order.Its multistep information comes from previous predictions, unlike 2S, which uses a new intermediate stage.Source: §9.5.5 · pp. 339–340

9.5.6 Closing Remark

  • DPM-Solver-v3 generalizes solver representation using precomputed model statistics, while predictor–corrector methods combine numerical prediction with score-based refinement.The section presents these methods as broader examples of combining classical numerical ideas with diffusion-specific PF-ODE structure.Source: §9.5.6 · pp. 340–341

9.6 (Optional) ParaDiGMs

  • ParaDiGMs replace sequential time-stepping with Picard iterations that update all time nodes simultaneously, enabling parallel computation across the trajectory.The preceding time-stepping approach evolves sequentially from prior time T toward an arbitrary intermediate time t.Source: Figure 9.3 · p. 342, §9.6 · p. 342, §9.6.1 · p. 342

9.6.1 From Time-Stepping to Time-Parallel Solver

  • ParaDiGMS trades computational resources for faster simulation by parallelizing integration across time through classical Picard iteration.Its central strategy reorganizes time integration rather than changing the underlying generative trajectory.Source: §9.6.1 · p. 342

9.6.2 Methodology of ParaDiGMS

  • ParaDiGMS treats the solution trajectory as a fixed point of an integral-map update, repeatedly evaluating velocity along the previous trajectory and updating all time points.On a decreasing time grid, drift evaluations are independent across nodes and cumulative sums can use parallel prefix scans.Source: §9.6.2 · pp. 343–344
  • Sliding windows anchor cumulative updates at the current left boundary and advance to the first node exceeding its local Picard tolerance, never skipping unconverged nodes.Newly exposed nodes use constant extrapolation by default, with linear or polynomial guesses also possible.Source: §9.6.2 · pp. 344–345

9.6.3 Relation to Time-Stepping Solvers

  • Exact fixed-point convergence recovers the chosen sequential solver’s discrete trajectory, while finite stopping tolerance adds fixed-point iteration error to the base solver’s discretization error.The framework can use Euler, DDIM, DPM-Solver, or higher-order rules while retaining parallel drift evaluation and prefix-sum accumulation.Source: §9.6.3 · pp. 345, 347

9.7. Closing Remarks

  • Solver developments exploit known diffusion structure to reduce sampling from hundreds or thousands of model evaluations to tens, but remain fundamentally iterative.The text then motivates training-based methods that learn one- or few-step generators directly from an ODE-defined flow.Source: §9.7 · p. 348

Distillation-Based Methods for Fast Sampling

  • Distillation trains a fast student from a slow, pre-trained diffusion teacher, compressing hundreds of sampling steps into a few-step or single-step generator.Its two paradigms align either the teacher’s output distribution or its state-to-state flow map; Progressive Distillation replaces two teacher steps with one student step.Source: Distillation-Based Methods for Fast… · p. 350, §10.1 · p. 351

10.1.1 Distribution Level Distillation

  • Distribution-level distillation trains a one-step generator Gθ(z) whose induced distribution pθ(ˆx) approximates the data distribution pdata(x), typically by minimizing a statistical divergence.The supplied passage introduces this objective but does not specify the divergence form.Source: §10.1.1 · pp. 351–352

10.1.2 Flow Map Level Distillation

  • Flow-map distillation learns ΨT→0(xT) from prior noise, converting iterative PF-ODE sampling—which can require at least 10–20 NFE even in DPM-Solver—into one-step generation.Trajectory distillation directly approximates the numerical solver’s instance-level solution map, but full ODE integration during training is expensive and global T-to-0 mapping can reduce controllability.Source: Question 10.1.1 · pp. 352–353, §10.1.2 · p. 352

10.3 Progressive Distillation

  • Progressive Distillation (PD) halves the sampling transitions by matching two consecutive teacher steps, then recursively repeats this on coarser grids until typically 1–4 steps remain.Each student transition covers two teacher transitions, preserving the overall time horizon while reducing the step count N → N/2 → N/4 → · · ·.Source: Question 10.1.1 · pp. 353–354, §10.3 · p. 359

Question 10.1.1

  • KD and PD fit a unified flow-map loss: KD supervises only the endpoint pair (T, 0), whereas PD enforces local consistency across nearby time pairs using teacher or solver approximations.The oracle flow map generally lacks a closed form, so practical methods depend on numerical solvers, surrogate targets, or teacher quality.Source: Question 10.1.1 · p. 354

10.2 Distribution-Based Distillation

  • Distribution-based distillation aligns a one-step generator’s forward-noised marginals with the teacher’s, using KL or related divergences whose gradients can be estimated through student and target score functions.DMD (Yin et al., 2024b; Yin et al., 2024a), VSD (Lu and Song, 2024), and SiD share this principle.Source: §10.1.1 · p. 352, §10.2 · p. 355

10.2.1 Formulation of VSD as a Representative Approach

  • At population optimum, VSD matches the generator distribution to the data distribution, while its alternating score-training solution implies matching noised marginals and hence pdata under Gaussian-noising injectivity.These equalities describe the ideal population solution, not a guarantee for finite learned networks or alternating optimization.Source: §10.2.1 · p. 355, §10.2.2 · p. 357

10.2 Distribution-Based Distillation

  • VSD’s gradient emerges from differentiating the KL objective: the score mismatch between target and generator distributions is propagated through the generator’s parameter sensitivity.The derivation uses the score-function identity and the reparameterization ˆxt = αtGθ(z) + σtϵ.Source: §10.2 · p. 356

10.2.2 Training Pipeline of VSD

  • VSD alternates auxiliary score estimation on generator samples with generator updates using the pretrained teacher as a proxy for the intractable target score.The auxiliary score is trained by denoising score matching, while the teacher approximates the data-distribution score on synthetic noised samples.Source: §10.2.2 · pp. 356–357

10.2.3 Beyond KL: General Divergences and Why KL Is Special

  • General f-divergences retain the same score-mismatch direction but reweight density-ratio regimes; reverse KL is convenient because its density-ratio factor disappears, avoiding an additional ratio estimator.Forward KL instead emphasizes regions where model density is small relative to target density.Source: §10.2 · p. 358

10.3.1 Distillation Operation in PD

  • PD trains a single student transition to reproduce the endpoint of two adjacent teacher transitions, using a closed-form pseudo-clean target for DDIM.The target is obtained by inverting the one-step DDIM update so its output matches the teacher’s two-step composition.Source: §10.3.1 · p. 360, Question 10.3.1 · pp. 360–361

10.3.2 Entire Training Pipeline of PD and Its Sampling

  • Each PD round replaces the teacher with a student that traverses the same time horizon using N/2 transitions, then repeats halving on progressively coarser grids.Sampling follows skip-2, skip-4, and later larger jumps while preserving the start and end times.Source: §10.3.2 · p. 362

10.3.3 Additional Discussion: Local Semigroup Matching and the Possibility of Generalized Solvers

  • PD is a local semigroup-matching procedure: a student one-step map matches a teacher composition of short adjacent fragments rather than a full rollout to time 0.This provides a local surrogate for the global flow-map regression objective and requires only short teacher fragments.Source: §10.3.3 · pp. 362–363
  • The distillation principle extends beyond DDIM when coarse updates are invertible, but complicated solvers may require endpoint matching and fewer time steps need not reduce neural function evaluations.For stochastic samplers, pathwise matching additionally requires fixed or coupled random draws; otherwise the target is generally distributional.Source: §10.3.3 · pp. 363–364

10.3.4 PD with Guidance

  • For classifier-free guidance, a first stage trains one network conditioned on guidance weight ω, after which PD repeatedly halves its sampling steps.The two-stage pipeline was demonstrated in both pixel and latent spaces, including Stable Diffusion.Source: §10.3.4 · p. 364

Question 11.0.1

  • The chapter shifts from teacher-based acceleration toward standalone flow-map models that directly approximate ODE solution maps, enabling long trajectories to be traversed in one or a few network evaluations.These models aim to transport the prior to the data distribution while preserving intermediate diffusion marginals, without relying on a pretrained model.Source: Question 11.0.1 · pp. 366–367, §11.1 · p. 368

11.1. Prologue

  • Diffusion models define a continuous-time forward corruption process and learn reverse-time transport that restores the desired marginal distributions while enabling intermediate-step steering.Source: §11.1 · p. 368

Question 11.1.1

  • The central question is whether Gθ(·, s, t) can learn the flow map Ψs→t(·) without pre-trained models while maintaining high-fidelity generation.Source: Question 11.1.1 · p. 368
  • A unified objective trains flow-map models to approximate the oracle transformation between arbitrary times, with exact recovery at the oracle-loss optimum; approaches divide into distillation and training from scratch.Source: Question 11.1.1 · pp. 368–370
  • Consistency Models (Song et al., 2023) learn the special flow map Ψs→0, replacing its unavailable oracle target with a stop-gradient estimate from an earlier point on the same trajectory.Source: Question 11.1.1 · p. 370
  • CTM learns arbitrary-time maps using intermediate stop-gradient targets, whereas MF (Geng et al., 2025a) models average drift over intervals and uses self-referential velocity objectives.Both methods support distillation from pre-trained diffusion models or self-contained training from scratch; MF's scratch strategy uses one-point conditional velocity from the forward corruption process.Source: Question 11.1.1 · pp. 370–371

Relationship Between CTM and MF. … B.2.2 Derivation of the Continuity Equation from Conservation Laws

This section develops diffusion models as probability-transport methods across continuous and discrete state spaces, connecting consistency-based flow maps, CTM, and Mean Flow with learned reverse dynamics. It establishes the underlying ODE, continuity-equation, Fokker–Planck, and numerical-solver foundations for trajectories, density evolution, and sampling.

11.2 Special Flow Map: Consistency Model in Discrete Time

  • Consistency models approximate the special flow map Ψs→0, whose semigroup property implies that composing transitions through an intermediate time equals transitioning directly.Along a trajectory, global, self, and local consistency are equivalent formulations of mapping every state to the same terminal point.Source: §11.2 · pp. 373–375

11.2.1 Discrete-Time Approximations for Learning a Consistency Function

  • Consistency Distillation uses a pretrained diffusion model and a one-step DDIM update, whereas Consistency Training estimates the score from paired data and trains fully from scratch.Stop-gradient targets prevent collapse to trivial constant predictions; the one-point score estimator is conditionally unbiased, and the resulting teacher-free loss differs from the consistency loss by O(∆s^2) under stated assumptions.Source: §11.2.1 · pp. 375–377
  • Under smoothness and moment assumptions, the teacher-free approximation satisfies LCT = LCM + O(∆s^2), because the first-order Taylor error cancels after conditioning on xs.The oracle DDIM update equals the conditional mean, while the one-point pathwise update uses the same (x0, ϵ) pair.Source: §11.2.1 · pp. 377–378

11.2.2 Sampling with Consistency Model

  • A trained consistency model generates samples either in one function evaluation from prior noise or through multiple noise-injection and denoising jumps at earlier time points.The multi-step procedure alternates consistency-model evaluations with Gaussian noise injection along a decreasing sequence of times.Source: §11.2.2 · p. 379

11.3 Special Flow Map: Consistency Model in Continuous Time

  • The continuous-time formulation defines the flow map for all times rather than a fixed grid, reducing discretization dependence and enforcing consistency globally along the underlying dynamics.This formulation is motivated by the invariance of the consistency function along PF-ODE trajectories.Source: §11.3 · p. 380, §11.3.1 · p. 380

11.3.1 Continuous-Time Consistency Model

  • The continuous-time consistency condition follows from the chain rule along PF-ODE trajectories, with velocity estimated either from a pretrained diffusion model or a direct one-point approximation.Although a residual objective is possible, practical methods use teacher-based consistency objectives, and the discrete formulation is refined through ∆sN →0.Source: §11.3.1 · p. 380

Proposition 11.3.1: Continuous-Time Consistency Training

  • At θ = θ−, the discrete consistency loss behaves up to O(∆s^2) like an inner product between the student update and the tangent change of the stop-gradient target.The zeroth-order discrepancy vanishes at the comparison point, yielding the stated scaled-gradient identity.Source: Proposition 11.3.1: Continuous-Time… · p. 381
  • Continuous-time consistency training uses a pseudo-objective based on the total derivative of the consistency function along the oracle trajectory.The derivative is not merely a partial time derivative, and stop-gradient treats θ− as constant under ∇θ.Source: Proposition 11.3.1: Continuous-Time… · p. 381

11.3.2 Training Continuous-Time Consistency Model

  • Continuous-time consistency models can use either a pretrained diffusion velocity for distillation or a one-point conditional estimate for training from scratch.After training, sampling follows the same procedure as in the discrete-time case; the methods distinguish continuous- and discrete-time objectives.Source: §11.3.2 · pp. 382–383

11.3.3 (Optional) Practical Considerations of Continuous-Time CT

  • Continuous-time CT requires stabilization because unbounded tangent derivatives can cause exploding gradients, motivating trigonometric schedules, EDM-like parameterization, bounded time embeddings, adaptive weighting, and tailored time sampling.The adaptive weighting normalizes expected rescaled loss separately at each time, reducing variance across times and improving training stability; the parameterization also enforces fθ(x, 0) ≡ x.Source: §11.3.3 · pp. 383–386

11.4 General Flow Map: Consistency Trajectory Model

  • CTM learns a general flow map Ψs→t by approximating PF-ODE solutions, extending the EDM setup from fixed schedules to arbitrary linear Gaussian forward kernels in v-prediction form.Its practical setup begins with EDM’s αt = 1 and σt = t formulation and uses a single-step Euler-inspired parameterization.Source: §11.4 · p. 388

11.4.1 CTM Parametrization for Flexible Transition Learning

  • CTM parameterizes the oracle flow map as a residual correction whose learned function fits the general consistency-mapping framework and inherently satisfies the initial condition.The same parameterization supports finite s-to-t transitions and infinitesimal transitions, with the diagonal encoding instantaneous velocity and enabling both long jumps and local diffusion dynamics.Source: §11.4.1 · pp. 389–390

11.4.2 Consistency Loss in CTM

  • CTM’s consistency loss is introduced to approximate the oracle solution map while supporting both distillation and training from scratch.The loss enforces the semigroup property for learned transitions.Source: §11.4.2 · p. 390, §11.4.1 · p. 390
  • CTM enforces the flow-map semigroup property by matching a direct student transition to a teacher–student composition, interpolating between global and local supervision.With a pretrained teacher, the target uses a numerical PF-ODE solver; without one, CTM replaces it with a self-induced teacher, whose accuracy depends on recovering local diagonal dynamics.Source: §11.4.2 · pp. 391–392

11.4.3 Auxiliary Losses in CTM

  • The CTM objective combines consistency and diffusion losses with an optional GAN term, using regression losses for stability and adversarial training for finer detail.The diffusion-model loss directly supervises local trajectory behavior, while the GAN term encourages distributional matching and sharper samples.Source: §11.4.3 · pp. 394–395

11.4.4 Flexible Sampling with CTM

  • CTM learns Ψs→t for arbitrary s > t, supporting anytime-to-anytime transitions, γ-controlled stochasticity, ODE-based solvers, exact likelihoods, and diffusion samplers.For γ ∈[0, 1), sampling uses the general flow map, whereas γ = 1 reduces to CM-style multistep sampling; γ = 0 avoids numerical-solver discretization errors.Source: §11.4.4 · pp. 395–397
  • CTM analysis attributes degraded long multistep sampling to accumulated mismatches from repeated neural jumps, while γ = 0 removes temporal overlap and yields improved stability as steps increase.The supplied analysis states that CM-style multistep sampling degrades when M ≥4 and that γ = 0 provides a favorable speed–quality trade-off.Source: §11.4.4 · pp. 396–397

11.5.1 Modeling and Training of Mean Flow

  • Mean Flow (MF) learns the average drift over [t, s] under flow matching, replacing an intractable oracle with a stop-gradient surrogate derived from differentiating the flow trajectory.MF can approximate the oracle velocity through either distillation from a pretrained diffusion model or an unbiased paired data estimator, requiring a Jacobian–vector product.Source: §11.5.1 · pp. 398–399

11.5.2 Sampling of Mean Flow

  • A trained MF model approximates Ψs→t directly, enabling one-step, multistep, and γ-controlled stochastic sampling from prior noise.Sequential multistep generation uses a time grid, while γ-sampling injects controllable stochasticity as in CTM.Source: §11.5.2 · p. 399

11.5.3 Relationship between CTM and MF

  • CTM and MF use different parameterizations of the same oracle flow map Ψs→t, with their squared oracle regression errors related by a time-dependent weighting (Hu et al., 2025).Source: §11.5.3 · p. 400
  • Both CTM and MF learn direct mappings between arbitrary time steps, supporting γ-sampling, diffusion guidance, exact likelihood computation, and accelerated higher-order inference.Their compatibility with diffusion inference follows because both recover the instantaneous diffusion drift in the infinitesimal limit, unlike specialized maps Ψs→0 in the CM family.Source: §11.5.3 · p. 402
  • CTM and MF encode the same oracle flow map, with equivalent squared-error regression losses up to known time-dependent scaling but different practical targets and optimization procedures.CTM uses an integral flow-map target requiring an ODE during training, whereas MF uses a tractable differential target based on finite differences.Source: §11.5.3 · pp. 400, 402
  • Flow-map learning replaces repeated numerical updates with one or a few learned jumps, shifting computational difficulty from sampling into learning a global finite-time transition.Success depends on map parameterization, tractable targets, time-pair sampling and weighting, and optimization stabilization.Source: §11.5.3 · pp. 403, 407
  • The semigroup, Lagrangian, and Eulerian perspectives are equivalent descriptions of an exact ODE flow under standard existence, uniqueness, and smoothness assumptions.The semigroup view splits trajectories into finite intervals, the Lagrangian view moves the ending time, and the Eulerian view moves the starting point while keeping the endpoint fixed.Source: §11.5.3 · pp. 403–405

A Unifying Principle and the Road Ahead

  • The book’s unifying perspective identifies probability transport and the change-of-variable principle as the backbone connecting diffusion models and fast generators.These methods begin from a simple reference distribution, evolve probability laws under prescribed dynamics, and use an appropriate law-evolution equation to describe that evolution.Source: §12.1 · p. 411

12.1 The Density-Transport Backbone

  • Diffusion formulations differ in their forward process, law-evolution equation, and generative parameterization, but all track probability mass under prescribed dynamics.Deterministic maps yield continuity-equation transport, while adding noise yields Fokker–Planck dynamics; both preserve total probability mass.Source: §12.1 · p. 413

12.2 Beyond Continuous States: Probability Evolution on Discrete Spaces

  • Finite-state diffusion replaces continuous trajectories and gradients with probability transfers between discrete states governed by transition kernels or jump rates.The evolving probability vector obeys the master equation, also called the Kolmogorov forward equation.Source: §12.2 · p. 415, Question 12.2.1 · p. 416

Question 12.2.1

  • Discrete and continuous models follow the same recipe: choose a forward corruption process, characterize its probability path, and learn a reverse transport mechanism.Reverse transport is represented by Gaussian conditionals, scores, or velocity fields in continuous settings, and by transition probabilities, ratios, or jump rates in finite spaces.Source: Question 12.2.1 · p. 416

12.2.1 Discrete-Time Transitions

  • A finite-state process represents each category as a one-hot state and evolves its marginal probability vector through row-stochastic transition matrices.For sequences, forward corruption may factorize across positions, while the learned reverse model can use global context through a Transformer or graph neural network.Source: §12.2.1 · pp. 416, 417, 419, 420
  • The discrete reverse kernel is obtained by Bayes’ rule, weighting each possible source state by its forward transition probability and current marginal mass, then normalizing by total arriving mass.This reverses a single forward step whenever the observed next-state marginal is positive.Source: §12.2.1 · p. 420
  • Discrete reverse transitions are Bayesian inverses that depend on the unavailable marginal distribution, not matrix inverses of the forward transition matrices.Forward transitions can merge probability mass and therefore need not be invertible; reverse modeling learns the missing marginal information.Source: §12.2.1 · p. 420

12.2.2 Continuous-Time Limit: The Master Equation

  • As discrete steps become infinitesimal, a Markov chain becomes a CTMC whose rate matrix generates the master equation, expressing probability change as inflow minus outflow.The generator has nonnegative off-diagonal jump rates and rows summing to zero, enforcing probability conservation.Source: §12.2.2 · pp. 421–422
  • The CTMC master equation is the finite-state analogue of the continuity equation: both govern conserved probability flux, but CTMC mass jumps along allowed graph edges rather than neighboring spatial locations.The rate matrix plays the structural role of a velocity field while the geometry changes from continuous space to a transition graph.Source: §12.2.2 · pp. 422–423

12.2.3 Three Perspectives on Reverse Modeling

  • The variational perspective learns categorical reverse kernels by matching tractable forward posteriors conditioned on clean data, while clean-state prediction is only one possible parametrization.Discrete ELBO terms are finite sums over states, and alternative direct-logit or structured parametrizations remain possible.Source: §12.2.3 · pp. 425–427
  • The score/ratio perspective replaces Euclidean scores with local probability ratios along allowed jumps, which determine reverse rates and can be learned before simulating the reverse chain or CTMC.Time reversal uses the forward rate multiplied by a marginal-probability ratio, subject to positive marginals and the forward transition support.Source: §12.2.3 · pp. 427–429
  • Continuous flow matching learns tractable conditional velocity targets whose squared-error population minimizer equals the intractable marginal velocity, avoiding explicit marginal-density computation.Discrete flow matching analogously trains on conditional transitions or jump rates, recovering a marginal transport field through conditional expectation.Source: §12.2.3 · pp. 429–430

12.2.4 What Is Similar and What Is Genuinely Different?

  • Discrete diffusion is not a literal continuous analogue: sample paths are pure jumps, reverse dynamics require Bayesian time reversal, and forward transition structure directly shapes reverse parameterization and sampling.Discrete generation simulates categorical reverse transitions or CTMC jumps rather than numerically integrating ODEs or SDEs.Source: §12.2.4 · pp. 431–432
  • Probability transport is the shared organizing principle: forward corruption evolves a law, and generation learns a coherent reverse mechanism, with continuous and discrete spaces differing mainly in their mathematical objects.Continuous models use densities, gradients, velocity fields, and ODE/SDE dynamics; discrete models use probability vectors, transition kernels, jump rates, ratios, and probability currents.Source: §12.2.4 · p. 433, §12.3 · p. 434

12.3 Closing Remarks of the Book

  • The book’s closing perspective treats diffusion as constructing generators from prescribed forward processes, with variational, score-based, flow-based, and direct flow-map formulations sharing this transport viewpoint.Different state spaces require different reverse representations, but methods are coherent when the forward process, law evolution, and learned generative mechanism fit together.Source: §12.3 · p. 434

Crash Course on Differential Equations

  • The differential-equation crash course supplies prerequisites on ODEs, SDEs, PDEs, trajectories, and numerical approximation for later diffusion analysis.It highlights Euler, Runge–Kutta, Adams–Bashforth, Picard iteration, and Euler–Maruyama as representative deterministic and stochastic solvers.Source: Crash Course on Differential Equations · p. 436

A.1.1 Intuition of Ordinary Differential Equation

  • An ODE specifies how a state follows a time-dependent vector field, so solving it means tracing a trajectory whose tangent matches the local drift.The initial state determines which trajectory is followed among the paths permitted by the same velocity field.Source: §A.1.1 · p. 438, Figure A.1 · p. 438

Question A.1.1

Theorem A.1.1: Local Existence and Uniqueness

  • Under continuity and Lipschitz conditions in the state, every initial condition has a unique local ODE solution on an interval around the initial time.Picard–Lindelöf iteration constructs the solution, with convergence guaranteed under the Lipschitz condition.Source: Theorem A.1.1: Local Existence and… · pp. 439–440
  • Under the stated assumptions, an ODE with a time-dependent velocity field uniquely determines the trajectory from its initial condition.Applying these theorems to diffusion probability-flow ODEs may require additional assumptions on the score function ∇x log pt(x).Source: Theorem A.1.1: Local Existence and… · pp. 440–441

Corollary A.1.1: Non-Intersection of Solutions

A.1.3 Exponential Integration Factor

  • Exponential integration factors solve the linear component of semilinear ODEs exactly while leaving only the nonlinear residual for numerical discretization.This structure appears in diffusion probability-flow ODEs and makes step sizes depend on nonlinear dynamics rather than potentially large linear coefficients.Source: §A.1.3 · pp. 442–444

A.1.4 Numerical Solvers of Ordinary Differential Equations

  • Numerical ODE solvers approximate continuous trajectories on discrete time grids, trading smaller, more informative steps against additional velocity evaluations.Time-stepping methods include single-step and multistep schemes, while parallel-in-time methods update tentative states through fixed-point constructions such as Picard iteration.Source: §A.1.4 · pp. 444–445

Mental Picture

  • Diffusion sampling integrates the same ODE solver formulas backward on a decreasing time grid by treating the step size as signed.This reversal does not require new ODE solver families, but it differs from reversing an SDE, which requires additional analysis.Source: Mental Picture · p. 446

A.1.5 Representative Numerical Solvers

  • Euler is first-order with global error O(h), Runge–Kutta methods gain second-order accuracy through within-step stages, and AB2 achieves global error O(h2) by reusing prior-step velocities.AB2 requires initialization by another solver, while Karras et al. (2022) use Heun’s method directly for diffusion ODE solving.Source: §A.1.5 · pp. 446–448
  • Picard iteration refines an entire candidate ODE trajectory and converges locally to the solution under standard Picard–Lindelöf conditions, while enabling independent evaluations across time points.Its trajectory-wide updates open the possibility of parallel-in-time computation.Source: §A.1.5 · p. 448

A.2.1 From ODEs to SDEs: An Intuitive Introduction

  • As Δt →0, Euler discretization converges to the exact ODE solution under standard regularity conditions, while adding scaled random perturbations yields continuous-time stochastic dynamics.The stochastic extension uses a diffusion coefficient and scaling that keeps perturbations finite in the limit.Source: §A.2.1 · p. 449

A.2.2 Further Explanation of Equation (A.2.3)

  • SDEs are interpreted through Itô integrals because Brownian paths are almost surely nowhere differentiable, so differential notation is formal rather than classical.The drift is a classical integral, whereas the stochastic term uses left-endpoint evaluation in the Itô construction.Source: §A.2 · p. 450, §A.2.2 · pp. 450–451

A.2.3 A Numerical Solver for SDE.

  • General SDEs usually require numerical solvers: Euler–Maruyama approximates drift over Δt and simulates stochastic noise with Gaussian increments, while suitable conditions ensure a unique strong solution.Linear SDEs are identified as a later exception admitting a closed-form solution.Source: §A.2.3 · p. 452

Fokker–Planck

  • The appendix frames continuity and Fokker–Planck equations as continuous-time change-of-variable laws: they describe density evolution while conserving probability mass and distinguish particle motion from density evolution.It also contrasts Lagrangian and Eulerian transport views and notes that identical marginal densities can arise from different sample paths in an SDE and its probability-flow ODE.Source: Fokker–Planck · p. 453

B.1 Change-of-Variable Formula: From Deterministic Maps to Stochastic Flows

  • A sequence of smooth bijections transforms densities through Jacobian volume changes, and infinitely many small updates converge to a deterministic flow map governed by a time-dependent velocity field.This continuous-time limit provides the bridge from discrete normalizing-flow transformations to the continuity equation.Source: §B.1.1 · pp. 455, 457, §B.1.2 · p. 458

B.1.2 Continuous-Time Limit: Continuity Equation

  • The continuity equation expresses conservation of probability mass: velocity transports density, while the divergence term captures local expansion or compression and preserves total probability.It is the continuous-time analogue of the change-of-variables formula and follows from the pushforward induced by the flow.Source: §B.1.2 · pp. 458–459
  • A prescribed velocity field uniquely determines particle trajectories and the resulting density evolution through the continuity equation, while a density path alone generally admits multiple velocity fields.A density path is realizable only if some velocity field and flow map satisfy the continuity equation.Source: §B.1.2 · pp. 460–461

B.1.3 Stochastic Processes: Fokker-Planck Equation

  • The Fokker–Planck equation adds diffusion from stochastic noise to drift transport, while a probability-flow ODE reproduces the SDE’s one-time marginal densities.The SDE and probability-flow ODE generally differ in transition distributions, temporal correlations, and sample paths despite matching densities at each time.Source: §B.1.3 · pp. 461–462

B.2 Intuition of the Continuity Equation

  • The physical control-volume interpretation derives continuity from boundary flux: accumulation equals negative outward flux, and positive divergence decreases density while negative divergence increases it.For transported quantities, the flux is j(x,t) = p(x,t)v(x,t).Source: §B.2.1 · p. 463, §B.2 · p. 464, §B.2.2 · p. 464

B.2.2 Derivation of the Continuity Equation from Conservation Laws

  • The coordinate-free derivation integrates accumulation over an arbitrary control volume, converts boundary flux using the divergence theorem, and obtains the pointwise continuity equation.Because the control volume is arbitrary, the resulting integrand must vanish pointwise.Source: §B.2.2 · pp. 464–465
  • At the particle level, density decreases when neighboring particles spread apart and increases when they contract, linking individual velocity dynamics to whole-density evolution.The divergence expansion separates change along moving particles from deformation of the surrounding flow.Source: §B.2.2 · p. 465

B.3 (Optional) Wasserstein Gradient Flows as Distribution-Level Training … C.2.1 Girsanov’s Theorem as a Bridge Between Likelihood Training and Score Matching

This section develops distributional and trajectory-level foundations for diffusion-model training, introducing Wasserstein gradient flows, Itô calculus, Fokker–Planck dynamics, linear SDEs, and Girsanov’s theorem. It uses these tools to connect diffusion derivations, likelihood training, score matching, and the temporal coupling of states through path and transition laws.

B.3 (Optional) Wasserstein Gradient Flows as Distribution-Level Training

  • Wasserstein gradient flow treats training as moving the model distribution toward pdata through particle velocities, then approximating that ideal motion with finite-dimensional neural-parameter updates.The continuity equation describes distributional evolution, while neural training projects desired particle motion onto movements realizable by changing ϕ.Source: §B.3 · pp. 466–468

Closing Remarks.

  • The continuity equation provides a shared taxonomy for diffusion, likelihood-based, distribution-matching, and score-mismatch methods, which choose different discrepancies, velocities, and parameter updates to drive pϕ toward pdata.This viewpoint extends continuity-equation reasoning beyond diffusion dynamics to training-time evolution of model distributions.Source: Closing Remarks · pp. 470–471

Itô’s Calculus and Girsanov’s Theorem

  • Itô’s formula supplies the stochastic chain rule, while Girsanov’s theorem compares path probabilities under altered drifts and links likelihood-weighted score matching to path-space KL divergence and data likelihood.Together, these tools support standard diffusion derivations including the Fokker–Planck equation, reverse-time SDEs, training objectives, and likelihood relations.Source: Itô’s Calculus and Girsanov’s Theorem · p. 472

C.1 Itô’s Formula: The Chain Rule for Random Processes

  • Because Wiener processes are classically nondifferentiable, ordinary calculus does not directly apply to stochastic processes, motivating Itô’s calculus for stochastic integrals.The section positions Itô calculus as the mathematical foundation for differentiating functions of SDE-driven trajectories.Source: §C.1 · p. 474

C.1.1 Motivation: Why Do We Need a Special Chain Rule?

  • The ordinary chain rule fails for SDEs because Brownian increments have nonvanishing quadratic variation, so second-order terms in dwt contribute instead of disappearing.A Brownian-motion example shows that the missing term changes expected quadratic functions linearly over time.Source: §C.1.1 · pp. 474–475

C.1.2 Deriving 1D Itô’s Formula from Taylor Expansion

  • The derivation begins by revisiting the deterministic chain rule through a Taylor expansion for a scalar, time-varying process before extending the reasoning to stochastic trajectories.The setup treats dyt as a small increment and expands a scalar function h(yt, t).Source: §C.1.2 · p. 475
  • The one-dimensional Itô formula follows from a stochastic Taylor expansion that retains second-order increments, with Brownian scaling making cross terms negligible and (dwt)2 = dt.This establishes the stochastic chain rule used in later derivations.Source: §C.1.2 · pp. 475–476

C.1.3 Itô’s Formula: The Chain Rule for SDEs

  • The multidimensional Itô formula extends the one-dimensional chain rule to scalar functions of vector-valued SDEs and includes the Hessian correction term.The same framework supports stochastic product calculations for vector processes.Source: §C.1.3 · pp. 476–477

C.1.4 Itô’s Formula’s Application: Derivation of Fokker-Planck Equation

  • Applying Itô’s formula to a diffusion SDE derives the Fokker–Planck equation governing the time evolution of its probability density.The derivation applies the formula to a smooth test function, takes expectations, and uses integration by parts before exploiting arbitrariness of the test function.Source: §C.1.4 · pp. 478–479

C.1.5 Itô’s Formula Application: Closed-Form Solution of a Linear SDE

  • An integrating-factor construction combined with Itô’s formula provides an explicit solution to a vector-valued linear SDE.The resulting conditional law is Gaussian because the stochastic-integral term is Gaussian, and it is characterized by its conditional mean and covariance.Source: §C.1.5 · pp. 479–480
  • The linear-SDE analysis shows that the conditional covariance is isotropic and recovers the corresponding mean and covariance evolution equations.These moment equations are also derived directly from the SDE using conditional expectations and Itô’s product rule.Source: §C.1.5 · p. 481

Question C.2.1

  • The setup compares the likelihood of one observed trajectory under alternative SDEs whose deterministic drift differs while their initial distribution and diffusion coefficient remain fixed.Girsanov’s theorem quantifies how changing the drift changes the relative weighting assigned to the same trajectory.Source: Question C.2.1 · p. 482

C.2.1 Girsanov’s Theorem as a Bridge Between Likelihood Training and Score Matching

  • Girsanov’s theorem gives an exponential Radon–Nikodym likelihood ratio between path measures with the same initial distribution and diffusion coefficient but different drifts.The ratio contains an Itô integral and a Riemann integral, enabling comparisons of stochastic dynamics and path-space likelihood objectives.Source: Question C.2.1 · pp. 482–483
  • The forward and reverse-time diffusion SDEs induce path distributions P and Pϕ, with the reverse process parameterized by a learnable score function sϕ(xt, t).The section distinguishes these trajectory laws from the marginal distributions observed at individual times.Source: §C.2.1 · p. 483
  • A stronger trajectory-level notion models the probability law over entire trajectories.Source: §C.2.1 · p. 483
  • This trajectory-level formulation captures coupling between states at different times.Source: §C.2.1 · p. 483
  • It includes both transition laws and temporal dependencies across the trajectory.Source: §C.2.1 · p. 483

C.2. Change-of-Variable For Measures: Girsanov’s Theorem in Diffusion Models479 … D.2.4 Stein’s Identity and Surrogate SURE Objective

The section connects diffusion objectives across path-space, variational, and score-based views, showing how KL decompositions, ELBOs, score matching, Tweedie’s formula, and Stein’s identity yield tractable training principles. It also relates weighted score error to reverse path-law and data-likelihood discrepancies, while deriving SURE as an observable surrogate for conditional MSE.

C.2. Change-of-Variable For Measures: Girsanov’s Theorem in Diffusion Models479

  • Matching the full path law implies matching every marginal, but identical marginals do not determine the coupling or trajectories, as illustrated by the SDE and PF-ODE.The exact reverse-time SDE reproduces the forward diffusion’s full time-reversed path law under standard regularity, exact terminal initialization, and the true score.Source: §C.2 · p. 484
  • Likelihood-weighted score matching minimizes reverse path-law KL up to terminal-distribution mismatch, which also upper-bounds data-side KL and expected negative log-likelihood.The terminal term vanishes when pT = pprior, while other time weightings generally lack this same path-space KL interpretation.Source: §C.2 · pp. 484–485

D.1.1 Theorem 2.2.1: Equivalence Between Marginal and Conditional KL Minimization

  • Theorem 2.2.1 decomposes joint KL into marginal and conditional contributions, establishing equivalence of marginal and conditional KL minimization when the parameterization is sufficiently expressive.The optimality argument uses that KL is minimized at p* = p and the law of total probability.Source: §D.1.1 · p. 487

D.1.2 Theorem 2.2.3: ELBO of Diffusion Model

  • The ELBO proof introduces the forward noising distribution and applies Jensen’s inequality to the marginal log-likelihood under the joint generative factorization.The supplied derivation presents the proof setup and the joint distribution factorization.Source: §D.1.2 · p. 487, §D.1 · p. 488

D.1. Variational Perspective

  • The variational perspective derives diffusion training objectives by applying Jensen’s inequality to the forward-noising and reverse-process factorizations, then grouping the resulting ELBO terms.The decomposition relies on the Markov structure of the forward process.Source: §D.1 · p. 488

D.2.1 Proposition 3.2.1: Tractable Score Matching via Integration by Parts

  • Integration by parts converts intractable score matching into a tractable objective differing from the original loss by a distribution-dependent constant.The derivation requires suitable differentiability, support, and boundary-decay conditions.Source: §D.2.1 · p. 490

D.2.2 Theorem 3.3.1: Equivalence Between SM and DSM Minimization

  • Denoising score matching and marginal score matching differ only by a parameter-independent constant, so they have the same minimizers.The first two cross terms cancel, while the remaining term depends only on pdata(x) and pσ(˜x|x).Source: §D.2.2 · p. 491

D.2.3 Lemma 3.3.2: Tweedie’s Formula

  • Tweedie’s formula expresses the score of a time-dependent Gaussian-perturbed marginal through the posterior expectation of the clean variable, enabling score-based denoising relations.The proof differentiates the Gaussian conditional density, integrates over the posterior, and divides by the marginal density.Source: §D.2.3 · pp. 491–492

D.2.5 Theorem 4.2.1: Marginal Alignment via Fokker–Planck … D.6.2 A Common EDM Special Case: αt = 1, σt = t

This section establishes theoretical identities connecting forward diffusion, PF-ODE, and reverse-time SDE dynamics, including score, flow-density, parametrization, and distillation formulations. It also presents the EDM unit-variance parameterization and its αt = 1, σt = t special case, where the network transitions from an identity map at low noise to a scaled residual predictor at high noise.

D.2.5 Theorem 4.2.1: Marginal Alignment via Fokker–Planck

  • The PF-ODE and reverse-time SDE reproduce the forward SDE’s marginals in corresponding time order under uniqueness and regularity assumptions.The PF-ODE matches p_t forward and backward, while the reverse-time SDE follows the same marginals in reverse order.Source: §D.2.5 · pp. 496–497

D.2.6 Proposition 4.3.1: Minimizer of SM and DSM

  • The minimizer of the score-matching and denoising-score-matching objective is the conditional expected score, which equals ∇x_t log p_t(x_t) under suitable regularity.The proof uses conditional least-squares minimization, Bayes’ rule, and exchanging gradients with integrals.Source: §D.2.6 · pp. 497–498

D.2.7 Closed-Form Score Function of a Gaussian

  • For a multivariate Gaussian with mean µ and invertible covariance Σ, the score is obtained by differentiating its log density with respect to the observation.The derivation proceeds from the Gaussian density through its log-density and gradient.Source: §D.2.7 · p. 498, Lemma D.2.1: Score of Gaussian · pp. 498–499

D.3.1 Lemma 5.1.1: Instantaneous Change of Variables

  • Instantaneous change of variables yields the continuous-time log-density evolution along ODE trajectories, equivalently derived from the continuity equation.The result follows from both a small-step normalizing-flow expansion and trajectory-wise continuity-equation analysis.Source: §D.3.1 · p. 500

D.3.2 Theorem 5.2.2: Mass Conservation Criterion

  • A continuously differentiable flow pushes forward the initial density and satisfies the continuity equation; under uniqueness, this flow-induced density equals the target density.The weak formulation relaxes regularity requirements and is standard in PDE theory and numerical analysis.Source: §D.3.2 · pp. 500–501, §D.3 · p. 502

D.3. Flow-Based Perspective

  • The flow-based perspective proves equivalence by showing that the pushforward and candidate densities share the continuity equation and initial condition.Uniqueness via characteristics completes the equivalence for sufficiently smooth, globally Lipschitz velocity fields.Source: §D.3 · p. 502

D.4.1 Proposition 6.3.1: Equivalence of Parametrizations

  • Noise-, clean-, and velocity-prediction objectives have equivalent optimal estimators because their optima are algebraically related to the same score function.The equivalence uses denoising score matching, Tweedie’s formula, and the optimal conditional estimators for ε, x, and velocity.Source: §D.4.1 · p. 503

D.5.1 Knowledge Distillation Loss as an Instance of the General Framework Equation (10.1.4)

  • Knowledge-distillation loss fits the unified flow-map framework through oracle-flow change of variables and the semigroup property, while score mismatch supplies a data-space gradient mapped to parameters by the generator Jacobian.The transport interpretation identifies gradient descent as updating parameters opposite the pulled-back score-mismatch signal.Source: §D.5.1 · p. 504

D.5.2 Theorem 11.2.1: CM Equals CT up to O(∆s2)

  • Theorem 11.2.1’s proof uses Tweedie’s formula, shared-noise coupling, conditional means, and a second-order expansion to analyze the DDIM and consistency-map relationship.The derivation conditions on the sampled time, expands around the conditional mean, and uses the vanishing conditional residual mean together with smooth-scheduler differences of O(∆s).Source: §D.5.2 · p. 505, §D.5 · p. 506

D.5.3 Proposition 11.3.1: Continuous-Time Limit of the CT Gradient

  • The continuous-time CT-gradient proof expands the oracle flow map and differentiates with the target network stop-gradient, proving the identity at θ = θ−.The comparison point treats the target network as the stop-gradient copy of the online network, so no gradient passes through fθ−.Source: §D.5.3 · p. 507

D.6.1 Criteria for Neural Network xϕ Design

  • EDM parameterizes x-prediction with a time-dependent skip connection and scaled residual, normalizing inputs and targets to unit variance while minimizing error amplification through cskip.The selected output scaling gives cout(0) = 0 and cout(t) → σd at sufficiently large σt, yielding xϕ(xt, 0) ≈ xt and xϕ(xt, t) ≈ σd Fϕ(·).Source: §D.6 · p. 508, §D.6.1 · p. 509

D.6.2 A Common EDM Special Case: αt = 1, σt = t

  • With αt = 1 and σt = t, the forward kernel is N(xt; x0, t2I), while the exact terminal marginal is pdata ∗ N(0, T2I) and is commonly approximated by N(0, T2I) for large T.The approximation defines the sampling prior pprior(xT) := N(xT; 0, T2I).Source: §D.6.2 · pp. 510–511
  • In this EDM special case, the parameterization smoothly interpolates from an identity map at t ≈ 0 to a scaled residual predictor on standardized inputs when t ≫ 0.At low noise, cskip ≈ 1 and cout ≈ 0; at high noise, cskip ≈ 0 and cout ≈ σd, so the output is determined by the learned denoising function.Source: §D.6.2 · p. 511

Related papers

Loading 2510.21890v3…