PaperlayerView exact revision on arXiv

Source-linked AI summary

Score-Based Generative Modeling through Stochastic Differential Equations

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, Ben Poole

arXiv:2011.13456v2cs.LGstat.ML

TL;DR

Score-based and diffusion models use related but distinct noise-corruption and reversal procedures, motivating a unified formulation for broader sampling and modeling capabilities. This paper develops an SDE-based framework with reverse-time sampling, predictor-corrector and ODE methods, achieving state-of-the-art CIFAR-10 scores and high-fidelity 1024 x 1024 generation.

  • Problem

    Existing score-based and diffusion models use related but distinct sequential corruption and reversal formulations, motivating a unified framework for new sampling methods and capabilities.

  • Method

    The framework models continuous-time diffusion with SDEs, estimates time-dependent scores using neural networks, and reverses the process numerically for sampling.

  • Results

    The model achieves CIFAR-10 Inception score 9.89 and FID score 2.20, plus high-fidelity generation of 1024 x 1024 images from a score-based model.

  • Takeaways & Limitations

    The SDE framework unifies prior approaches while enabling flexible sampling, exact likelihood computation, uniquely identifiable encoding, and conditional generation without retraining.

  • Takeaways & Limitations

    The proposed sampling approaches remain slower than GANs on the same datasets, and their sampler choices introduce additional hyperparameters.

Abstract

from arXiv · show

Creating noise from data is easy; creating data from noise is generative modeling. We present a stochastic differential equation (SDE) that smoothly transforms a complex data distribution to a known prior distribution by slowly injecting noise, and a corresponding reverse-time SDE that transforms the prior distribution back into the data distribution by slowly removing the noise. Crucially, the reverse-time SDE depends only on the time-dependent gradient field (\aka, score) of the perturbed data distribution. By leveraging advances in score-based generative modeling, we can accurately estimate these scores with neural networks, and use numerical SDE solvers to generate samples. We show that this framework encapsulates previous approaches in score-based generative modeling and diffusion probabilistic modeling, allowing for new sampling procedures and new modeling capabilities. In particular, we introduce a predictor-corrector framework to correct errors in the evolution of the discretized reverse-time SDE. We also derive an equivalent neural ODE that samples from the same distribution as the SDE, but additionally enables exact likelihood computation, and improved sampling efficiency. In addition, we provide a new way to solve inverse problems with score-based models, as demonstrated with experiments on class-conditional generation, image inpainting, and colorization. Combined with multiple architectural improvements, we achieve record-breaking performance for unconditional image generation on CIFAR-10 with an Inception score of 9.89 and FID of 2.20, a competitive likelihood of 2.99 bits/dim, and demonstrate high fidelity generation of 1024 x 1024 images for the first time from a score-based generative model.

1 INTRODUCTION

The paper presents a unified continuous-time SDE framework that transforms data into noise and reverses this process using time-dependent scores for generation. It enables flexible sampling, likelihood computation, controllable generation, and a unified view of prior score-based methods.

  • Reverse-time generation: A reverse-time SDE enables generation when the score, ∇x log p_t(x), is known at each intermediate time step.The score is the gradient of the log probability density with respect to the data.
  • Unified SDE framework: The framework generalizes prior score-based generative approaches by modeling a continuum of distributions evolving over time through a prescribed, data-independent diffusion SDE.The forward process progressively diffuses data into random noise, while the reverse process molds noise into data for sampling.
  • Flexible sampling and likelihood computation: General-purpose SDE solvers support reverse-time sampling, while predictor-corrector samplers combine numerical solvers with score-based MCMC methods.The proposed special samplers include Langevin MCMC and HMC, alongside deterministic samplers based on the probability flow ODE.
  • Controllable generation: Conditional reverse-time SDEs enable class-conditional generation, image inpainting, colorization, and other inverse problems from one unconditional score-based model without retraining.The conditional scores can be efficiently estimated from unconditional scores.
  • Unified framework: SMLD and DDPM can be represented as discretizations of two separate SDEs within the framework, enabling their unified analysis and tuning.The framework also permits new architectures and sampling algorithms for improving score-based generative models.

2 BACKGROUND

The background describes perturbing data across increasing noise scales and estimating the scores of the resulting distributions. It reviews Langevin-based sampling and diffusion Markov chains, whose optimal denoising-score objectives recover perturbed-distribution scores.

  • Score-based modeling: A Gaussian perturbation kernel defines perturbed data distributions across noise scales σ_min = σ_1 < … < σ_N = σ_max, with the smallest scale approximating the data distribution.The perturbed distribution integrates the data distribution against the Gaussian kernel; σ_max is chosen as a high-noise endpoint.
  • Score-based modeling: Noise Conditional Score Networks are trained with weighted denoising score-matching objectives to estimate ∇_x log p_σ(x) at each noise scale.With sufficient data and model capacity, the optimal model matches the perturbed distribution’s score almost everywhere.
  • Score-based modeling: Langevin sampling traverses noise scales from σ_N to σ_1, and converges to an exact sample from p_σmin(x) ≈ p_data(x) as M → ∞ and ε_i → 0 under regularity conditions.The procedure initializes at the largest-noise distribution and applies M Langevin steps at each scale.
  • Diffusion probabilistic modeling: Diffusion models construct a Markov chain with transitions p(x_i | x_i−1) = N(x_i; √(1−β_i)x_i−1, β_iI), choosing scales so x_N is approximately N(0, I).The marginal perturbation is p_αi(ẋ) = ∫ p_data(x)p_αi(ẋ | x)dx, with α_i = ∏_{j=1}^i(1−β_j).
  • Diffusion probabilistic modeling: The diffusion reverse chain is parameterized by a Gaussian model and trained with a re-weighted ELBO, enabling ancestral sampling from the learned reverse transitions.Its objective is a weighted sum of denoising score-matching objectives, so the optimal model matches ∇_x log p_αi(x).

3 SCORE-BASED GENERATIVE MODELING WITH SDES

This section formulates score-based generative modeling as a continuous-time diffusion process: an SDE transforms data into a tractable prior, while a score-dependent reverse-time SDE generates data from that prior. It introduces score estimation by continuous score matching and identifies VE, VP, and sub-VP SDEs as key instances underlying prior noise-perturbation methods.

  • SDE formulation: The framework generalizes finitely many noise scales to infinitely many scales, with perturbed data distributions evolving according to an SDE as noise intensifies.This continuous formulation extends the noise-perturbation idea used by previous methods.
  • SDE formulation: The forward SDE maps the data distribution p_0 at t=0 to a tractable prior distribution p_T at t=T, using drift and diffusion coefficients.The process is defined by dx = f(x,t)dt + g(t)dw, with w the standard Wiener process; the scalar diffusion coefficient is assumed independent of x for presentation.
  • Reverse-time SDE: The reverse-time SDE generates samples from p_0 by running backward from p_T and requires the score ∇_x log p_t(x) at every time.Once the marginal scores are known, the reverse diffusion process can be derived and simulated.
  • Score estimation: A time-dependent score model estimates ∇_x log p_t(x) through continuous score matching, whose optimum equals the true score with sufficient data and model capacity.Denoising score matching is used in the stated objective, while sliced and finite-difference score matching are also applicable.
  • SDE instances: SMLD and DDPM noise perturbations are discretizations of the VE and VP SDEs, respectively, while the proposed sub-VP SDE has variance bounded by VP and performs particularly well on likelihoods.VE has exploding variance as t → ∞, whereas VP maintains variance one when the initial distribution has unit variance; VE, VP, and sub-VP perturbation kernels are Gaussian and available in closed form.

4 SOLVING THE REVERSE SDE

The section develops numerical methods for sampling from the reverse-time SDE, including general solvers, reverse diffusion, and predictor-corrector samplers. It also introduces a probability flow ODE that enables exact likelihood computation, latent manipulation, efficient sampling, and uniquely identifiable encodings.

  • Reverse-time SDE solvers: A trained time-dependent score model defines the reverse-time SDE, which can be simulated with numerical SDE solvers to generate samples.Applicable methods include Euler–Maruyama, stochastic Runge–Kutta, and other discretizations.
  • Reverse-time SDE solvers: Reverse diffusion samplers discretize the reverse-time SDE like the forward process, avoiding the non-trivial derivation of ancestral sampling rules for new SDEs.Ancestral sampling in DDPM is a special discretization of the reverse-time VP SDE.
  • Predictor-corrector sampling: Predictor-corrector samplers combine a numerical SDE predictor with score-based MCMC correction, and PC samplers surpass predictor-only samplers under comparable computation with sufficient corrector steps.The approach is compatible with models trained using either fixed noise scales or continuous objectives.
  • Probability flow ODE: The probability flow ODE has the same marginal probability densities as the SDE and, with score estimates, becomes a neural ODE for deterministic sampling.It supports exact likelihood computation through the instantaneous change of variables formula and can encode data into latent representations for image editing.
  • Probability flow ODE: Probability flow ODE encodings are uniquely identifiable under sufficient training data, model capacity, and optimization accuracy because the forward SDE has no trainable parameters.ODE sampling with fixed or black-box solvers can produce competitive or high-quality samples while trading off accuracy and efficiency.
  • Architectural improvements: The NCSN++ architecture achieves an FID of 2.45 on CIFAR-10 with PC samplers, while DDPM++ achieves 2.78.These are the reported results for the optimal VE and VP SDE architectures trained with discrete objectives.

5 CONTROLLABLE GENERATION

The framework supports controllable generation by conditioning the reverse-time SDE on an observation y, enabling sampling from conditional data distributions and a broad family of inverse problems. The paper demonstrates this approach for class-conditional generation, image imputation, and colorization.

  • Conditional generation: The continuous framework can generate samples from p_0(x|y) when the forward conditional distribution p_t(y|x) is known.The required conditional score can be learned with a separate model or estimated using heuristics and domain knowledge.
  • Conditional generation: Conditional generation samples from p_0(x|y) by starting from p_T(x|y) and solving a conditional reverse-time SDE.The reverse dynamics include both the unconditional score and the conditional gradient ∇_x log p_t(y|x).
  • Class-conditional generation: Class-conditional sampling trains a time-dependent classifier p_t(y|x) using data generated by the tractable forward SDE.Training pairs are formed by perturbing dataset samples according to p_0t(x_t|x_0).
  • Image imputation and colorization: Imputation samples incomplete data points with an unconditional model, while colorization applies imputation after decoupling known dimensions with an orthogonal linear transformation.Both tasks are framed as conditional sampling, with colorization differing because its known dimensions are coupled.

6 CONCLUSION · APPENDIX · A THE FRAMEWORK FOR MORE GENERAL SDES

The paper presents an SDE-based score-modeling framework that clarifies existing methods and enables new sampling, likelihood, representation, and conditional-generation capabilities. Its appendices extend the framework to state-dependent matrix-valued diffusion coefficients, with corresponding reverse-time, probability-flow, and conditional formulations, while noting computational and sampling limitations.

  • 6 CONCLUSION: The framework unifies score-based generative modeling and enables new sampling algorithms, exact likelihood computation, identifiable encoding, latent-code manipulation, and conditional generation.These capabilities are presented as outcomes of the proposed SDE-based framework.
  • 6 CONCLUSION: Sampling remains slower than GANs on the same datasets, and the range of available score-based samplers introduces additional hyperparameters.Combining stable score-model learning with fast implicit-model sampling is identified as an important research direction.
  • APPENDIX: The appendices provide additional details, derivations, and results, including general SDEs, VE, VP, and sub-VP SDEs, practitioner guidance, and probability-flow formulations.Appendix A discusses matrix-valued state-dependent diffusion coefficients; Appendices B–D cover specific SDEs and probability flow.
  • A THE FRAMEWORK FOR MORE GENERAL SDES: The general framework considers Itô SDEs of the form dx = f(x,t)dt + G(x,t)dw, with drift f: R^d → R^d and diffusion G: R^d → R^(d×d).This extends the main-text setting in which the diffusion coefficient is independent of x.
  • A THE FRAMEWORK FOR MORE GENERAL SDES: For state-dependent diffusion, the reverse-time SDE includes the divergence of G(x,t)G(x,t)^T and the score term multiplied by G(x,t)G(x,t)^T.The reverse-time formulation is attributed to Anderson (1982).
  • A THE FRAMEWORK FOR MORE GENERAL SDES: The corresponding probability-flow ODE contains one-half of the diffusion-divergence term and one-half of the score term, with signs and coefficients determined by the general SDE.This formulation is linked to the paper’s earlier probability-flow equation and Appendix D.1.
  • A THE FRAMEWORK FOR MORE GENERAL SDES: Conditional generation under the general SDE adds the conditional score ∇_x log p_t(y | x) to the reverse-time dynamics.The conditional reverse-time SDE is given as the generalization of the earlier conditional formulation.
  • A THE FRAMEWORK FOR MORE GENERAL SDES: Non-affine drift or diffusion can make transition kernels unavailable in closed form, hindering denoising score matching; alternative score-matching methods and SDE simulation address this difficulty.The text gives sliced score matching as an example and notes that the SDE can be simulated to train the time-dependent score model.

B VE, VP AND SUB-VP SDES

The section establishes that SMLD and DDPM noise perturbations are discretizations of VE and VP SDEs, respectively. It also introduces sub-VP SDEs, whose variance is lower than VP while retaining the same mean and suitable Gaussian perturbation behavior.

  • VE and VP SDEs: SMLD and DDPM noise perturbations discretize the VE and VP SDEs, respectively.The VE SDE arises as the continuous-time limit of SMLD’s Markov chain, while DDPM’s chain converges to the VP SDE.
  • Sub-VP SDE: The sub-VP SDE is introduced as a VP-inspired modification that can improve sample quality and likelihoods.The passage describes sub-VP SDEs as a new SDE family motivated by the VP SDE.
  • VE and VP SDEs: VE variance explodes over time, whereas VP variance remains bounded and stays unit when the initial variance is unit.This distinction motivates the names Variance Exploding and Variance Preserving.
  • Sub-VP SDE: Sub-VP and VP SDEs have the same mean, while sub-VP variance is no greater than corresponding VP variance.With shared β(s) and equal initial covariance, Σ_sub-VP(t) ≤ Σ_VP(t) for all t ≥ 0.
  • Sub-VP SDE: Under suitable conditions, sub-VP SDEs converge to standard Gaussian like VP SDEs, and all three SDE families have Gaussian perturbation kernels.Their affine drift coefficients enable efficient training with the score-matching objective.

C SDES IN THE WILD · D PROBABILITY FLOW ODE

The paper instantiates VE, VP, and sub-VP SDEs whose discretizations recover SMLD and DDPM models, while addressing endpoint discontinuities and numerical instabilities through small positive time cutoffs. Empirically, the continuous perturbation kernels closely match their discrete counterparts, and the cutoff ε affects likelihood and sampling quality.

  • C SDES IN THE WILD: SMLD uses geometrically spaced noise scales with σmin = 0.01, while the continuous limit begins at σmin for positive time.The discrete noise schedule sets σmin to 0.01; the continuous formulation has a discontinuity at t = 0 because σ(0) = 0 but σ(0+) = σmin.
  • C SDES IN THE WILD: The VE SDE provides a continuous generalization of SMLD, with its perturbation kernel derived from the corresponding SDE transition.The paper presents the VE SDE and derives its perturbation kernel from the general transition equation.
  • C SDES IN THE WILD: Because the SMLD noise schedule is undefined at t = 0, VE computations use t ∈ [ε, 1] with ε = 10^-5 in experiments.The issue arises from σ(0) differing from its right-hand limit, making the VE SDE undefined at the endpoint.
  • C SDES IN THE WILD: The VP SDE uses a linear continuous noise schedule with βmin = 0.1 and βmax = 20, matching the settings of Ho et al. (2020).This schedule is obtained as the continuous limit of the discrete VP/DDPM schedule.
  • C SDES IN THE WILD: DDPM’s corresponding VP SDE avoids discontinuity but requires t ∈ [ε, 1] because the perturbed variance vanishes as t approaches zero.Sampling uses ε = 10^-3 to match DDPM variance, whereas training and likelihood computation use ε = 10^-5 for better empirical results.
  • C SDES IN THE WILD: N = 1000 discrete noise scales yield perturbation kernels that align well with the continuous VE and VP SDE generalizations.The comparison uses variances for SMLD and scaling factors plus variances for DDPM; the figure reports an almost exact match.
  • C SDES IN THE WILD: Sub-VP SDEs reuse the VP β(t) schedule and the same numerical interval [ε, 1].The paper gives a distinct sub-VP perturbation kernel while retaining the VP schedule and computational restriction.
  • C SDES IN THE WILD: Smaller ε generally improves likelihood, while an appropriate sampling ε improves Inception scores and FIDs without visibly changing samples to human observers.This dependence is reported across all SDEs for likelihood and specifically for sampling quality metrics.

D.1 DERIVATION … D.5 UNIQUELY IDENTIFIABLE ENCODING

The appendix derives the probability flow ODE, shows that it preserves the SDE’s marginal densities, and explains likelihood computation and deterministic sampling. It also reports sampling limitations and evidence that latent encodings remain consistent across architectures and training runs.

  • D.1 DERIVATION: The probability flow ODE is derived from the SDE’s Fokker–Planck equation and shown to induce the same marginal probability density as the original SDE.The transformed process has zero diffusion and therefore becomes an ODE.
  • D.2 LIKELIHOOD COMPUTATION: The probability flow ODE enables log-likelihood computation through the instantaneous change of variables formula and numerical integration of its trajectory.The trajectory x(t) is obtained by solving the probability flow ODE, while the score is replaced by a time-dependent score-based model.
  • D.2 LIKELIHOOD COMPUTATION: The Skilling–Hutchinson trace estimator provides an efficient unbiased estimate of the ODE divergence using reverse-mode automatic differentiation.Its vector-Jacobian product costs approximately as much as evaluating the ODE drift, and averaging can reduce error arbitrarily.
  • D.3 PROBABILITY FLOW SAMPLING: Probability flow sampling integrates the ODE backward from a prior sample using a deterministic iteration rule conditioned on the iteration number.Unlike reverse diffusion or ancestral samplers, it adds no randomness after the initial prior sample.
  • D.3 PROBABILITY FLOW SAMPLING: Probability flow sampling provides specialized iteration rules for both SMLD and DDPM models based on their score-based models and noise schedules.The appendix gives separate update forms for SMLD and DDPM models.
  • D.4 SAMPLING WITH BLACK-BOX ODE SOLVERS: Probability flow ODE solvers sample faster but typically produce worse FID scores than SDE solvers without a corrector, especially for VE SDEs on high-dimensional data.The observed quality depends on the SDE choice and applies to both the appendix discretization and black-box ODE solvers.
  • D.5 UNIQUELY IDENTIFIABLE ENCODING: For the same CIFAR-10 inputs, two VE-SDE models with different architectures and training runs produce latent encodings that are close in every dimension.The comparison uses Model A with 4 layers per resolution and Model B with 8 layers per resolution across 16 images.

E REVERSE DIFFUSION SAMPLING · F ANCESTRAL SAMPLING FOR SMLD MODELS

The paper discretizes the reverse-time SDE into reverse diffusion samplers for VE and VP SDEs, unifying DDPM ancestral sampling as an alternative discretization of the reverse-time VP SDE. It also adapts ancestral sampling to SMLD models through sequential noise scales and a learned reverse transition parameterization.

  • E REVERSE DIFFUSION SAMPLING: The reverse-time SDE is discretized using a fixed time schedule and Gaussian noise z_i ~ N(0, I) at each iteration.The discretization is written as x_i+1 = x_i + f_i(x_i) + G_i z_i, with the schedule absorbed into f_i and G_i.
  • E REVERSE DIFFUSION SAMPLING: The reverse discretization uses the trained score-based model s_θ*(x_i, i), conditioned on the iteration number.The reverse update has the corresponding functional form for stepping backward through the discretized SDE.
  • E REVERSE DIFFUSION SAMPLING: Reverse diffusion samplers arise by applying this discretization to the VE and VP SDEs as the predictor components of Algorithms 2 and 3.These methods are named reverse diffusion samplers because they use the discretization strategy of Eq. (46).
  • E REVERSE DIFFUSION SAMPLING: The DDPM ancestral sampler matches its reverse diffusion counterpart as β_i approaches zero, making it another discretization of the same reverse-time SDE.Because β_i = β̄_i Δt, this correspondence occurs as Δt approaches zero and unifies DDPM sampling with the continuous reverse-time VP SDE.
  • F ANCESTRAL SAMPLING FOR SMLD MODELS: SMLD ancestral sampling uses ordered noise scales σ_1 < σ_2 < ¨ ¨ ¨ < σ_N to form a Markov chain x_0 → x_1 → ¨ ¨ ¨ → x_N.The chain is generated by perturbing a data point sequentially with these noise scales.
  • F ANCESTRAL SAMPLING FOR SMLD MODELS: The SMLD reverse transition is parameterized as a Gaussian kernel p_θ(x_i−1 | x_i) with learned mean μ_θ(x_i, i) and variance τ_i^2.The derivation follows the DDPM variational objective and uses x_i(x_0, z) = x_0 + σ_i z.
  • F ANCESTRAL SAMPLING FOR SMLD MODELS: The score model s_θ(x_i, i) estimates z/σ_i, and the resulting Eq. (47) is called ancestral sampling for SMLD models.The method uses z_i sampled from N(0, I) and the optimal score-model parameter θ*.

G PREDICTOR-CORRECTOR SAMPLERS · H ARCHITECTURE IMPROVEMENTS · H.1 SETTINGS FOR ARCHITECTURE EXPLORATION

The paper combines reverse-SDE predictors with score-based MCMC correctors, while architecture improvements and task-specific settings strengthen sample quality, likelihood, and high-resolution generation. It also identifies different optimal EMA rates for VE and VP perturbations.

  • G PREDICTOR-CORRECTOR SAMPLERS: Predictor-corrector sampling alternates a numerical reverse-time SDE solver with a score-based MCMC corrector.The paper instantiates this framework with reverse diffusion and annealed Langevin dynamics for VE and VP SDEs.
  • G PREDICTOR-CORRECTOR SAMPLERS: The corrector step sizes use Gaussian-noise and score-model norms together with a signal-to-noise ratio r.The corrector algorithms re-frame annealed Langevin dynamics with modifications intended to improve interpretability and empirical performance.
  • G PREDICTOR-CORRECTOR SAMPLERS: Denoising is important because small residual noise can significantly worsen FID, especially for SMLD models without a final denoising step.DDPM sampling includes denoising, whereas the described SMLD setup does not.
  • G PREDICTOR-CORRECTOR SAMPLERS: Predictor-corrector methods perform on par or better than predictor-only and corrector-only samplers across interpolation strategies.Figure 9 further indicates that samples are best when computation is split between predictor and corrector.
  • H ARCHITECTURE IMPROVEMENTS: Architecture exploration produced new state-of-the-art CIFAR-10 sample quality and likelihood, plus the first high-fidelity 1024 x 1024 samples from score-based models.The authors open-source code and checkpoints for these models.
  • H.1 SETTINGS FOR ARCHITECTURE EXPLORATION: The explored architecture components include FIR anti-aliasing, 1/sqrt(2) skip-connection rescaling, BigGAN residual blocks, four residual blocks per resolution, and StyleGAN-2 progressive architectures.The progressive designs vary input and output paths between skip and residual forms.
  • H.1 SETTINGS FOR ARCHITECTURE EXPLORATION: 0.999 is the EMA rate used for VE models, while 0.9999 is used for VP models because each performs better for its respective perturbation type.For architecture exploration, FIDs are computed on 50k samples using a 1000-step PC sampler; VE uses one corrector step, whereas VP omits it.

H.2 RESULTS ON CIFAR-10 … I.3 COLORIZATION

The paper improves score-based generation on CIFAR-10 through architectural and continuous-training choices, achieving strong FID and likelihood results, and extends the framework to high-resolution synthesis and controllable generation tasks. These tasks include class-conditional sampling, imputation, and colorization via conditional reverse-time SDEs and transformed-space diffusion.

  • H.2 RESULTS ON CIFAR-10: Additional architecture components improve VE-SDE performance on both CIFAR-10 and CelebA on average, although no consistently best progressive-growing configuration is identified.The comparison varies other components freely to assess each component’s importance.
  • H.2 RESULTS ON CIFAR-10: 2.45 FID is achieved by basic NCSN++ on CIFAR-10, compared with 2.78 for basic DDPM++, using the lowest FID during training.For VE-SDE model comparisons, FID is instead averaged over checkpoints after 0.5M iterations.
  • H.2 RESULTS ON CIFAR-10: 2.41 FID is achieved by DDPM++ cont. (deep) with both VP and sub-VP SDEs, while sub-VP training reaches 2.99 bits/dim.Continuous objectives further improve FID values, and DDPM++ cont. improves VP-SDE FID from 2.78 to 2.55.
  • H.3 HIGH RESOLUTION IMAGES: 1024 x 1024 CelebA-HQ generation is demonstrated with a modified NCSN++ model trained using the continuous objective and a 2000-step predictor-corrector sampler.The model was trained for around 2.4M iterations with batch size 8 and EMA rate 0.9999.
  • I CONTROLLABLE GENERATION: Conditional reverse-time SDE sampling uses the conditional score, which decomposes into the unconditional data score plus the gradient of the log-likelihood of the condition.This formulation subsumes the conditional reverse-time SDE as a special case and supports the paper’s sampling methods.
  • I.1 CLASS-CONDITIONAL SAMPLING: Class-conditional sampling trains a time-dependent classifier on noise-perturbed CIFAR-10 examples and combines it with an unconditional NCSN++ model.The classifier is a Wide-ResNet-28-10 conditioned on log σ_i through random Fourier features and trained with cross-entropy losses across noise scales.
  • I.2 IMPUTATION: Imputation samples unknown dimensions conditioned on known dimensions by defining a diffusion process over the unknown components and approximating its conditional score.The approximation uses samples of the known dimensions from a typically tractable conditional distribution; extended inpainting results are provided in Figs. 14 and 15.
  • I.3 COLORIZATION: Colorization decouples grayscale and color channels with an orthogonal transformation, applies the imputation method to complete the color channels, and transforms back.Orthogonality preserves the standard Wiener process in the transformed space, enabling the same SDE construction; extended results appear in Figs. 16 and 17.

I.4 SOLVING GENERAL INVERSE PROBLEMS

Score-based generative models solve inverse problems by estimating the conditional score and using it in a reverse-time SDE to sample from p(x | y). The method relies on tractable perturbed observations and an approximation that becomes reliable at small or large diffusion times.

  • Problem: Inverse problems seek samples from p(x | y) given a known forward process p(y | x), but direct Bayesian inference is generally non-trivial.Estimating the prior p(x) and applying Bayes’ rule both present practical difficulties.
  • Score-based solution: Score-based models estimate the conditional score ∇x log p_t(x_t | y_t, y) and use it in the reverse-time SDE to sample from p(x | y).The conditional score is constructed from the unconditional score model and a likelihood-gradient term involving a sampled perturbed observation.
  • Assumptions: The approach assumes p(y_t | y) is tractable and p_t(x_t | y_t, y) ≈ p_t(x_t | y).The first distribution can often be derived from the forward process and SDE, while the second approximation is motivated by diffusion-time dependence.
  • Assumptions: For small t, y_t is close to y, while for large t, y has less influence on x_t and approximation errors matter less because they are used early.These conditions support replacing the full conditional score with an approximate expression based on the score model and likelihood term.
Loading 2011.13456v2…