Source-linked AI summary
Conditional Image Generation with Score-Based Diffusion Models
Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, Christian Etmann
TL;DR
Conditional score estimation in score-based diffusion models involves competing approaches with important limitations and approximation trade-offs. The paper compares these approaches theoretically and experimentally, proves consistency for CDE, and introduces CMDE, which interpolates between CDE and CDiffE. CMDE and CDE perform on par and significantly outperform CDiffE, while CMDE achieves better FID scores in some tasks.
Problem
Existing generative approaches have important limitations, while conditional diffusive estimation trades reduced optimization error for approximation error.
Method
The paper systematically compares conditional score estimators, proves CDE consistency, and introduces CMDE through a multi-speed diffusion framework.
Results
CMDE and CDE perform on par and significantly outperform CDiffE; CMDE achieves the best FID scores in inpainting and outperforms CDE on FID in inpainting and super-resolution.
Takeaways & Limitations
Diffusing the condition at an appropriate speed can improve posterior approximation and yield better FID scores, while score-based diffusion performs on par with prior state-of-the-art methods in super-resolution.
Takeaways & Limitations
CDiffE introduces approximation error when replacing the conditional score with one conditioned on a sampled noisy observation.
Abstract
from arXiv · showhide
Score-based diffusion models have emerged as one of the most promising frameworks for deep generative modelling. In this work we conduct a systematic comparison and theoretical analysis of different approaches to learning conditional probability distributions with score-based diffusion models. In particular, we prove results which provide a theoretical justification for one of the most successful estimators of the conditional score. Moreover, we introduce a multi-speed diffusion framework, which leads to a new estimator for the conditional score, performing on par with previous state-of-the-art approaches. Our theoretical and experimental findings are accompanied by an open source library MSDiff which allows for application and further research of multi-speed diffusion models.
1. Introduction
Score-based diffusion models offer attractive capabilities for image generation, motivating a systematic study of conditional score estimation. The paper compares existing approaches, proves consistency for conditional denoising, and introduces multi-speed diffusion and CMDE.
- Motivation: Existing generative-model families trade off computational cost, likelihood estimation, visual quality, architectural flexibility, or training stability.The introduction contrasts autoregressive models, VAEs, normalizing flows, and GANs on these limitations.
- Motivation: Score-based diffusion models combine state-of-the-art image generation and likelihood estimation with better high-resolution time complexity than autoregressive models.They also avoid training instabilities and mode collapse reported for GANs.
- Contributions: The paper systematically reviews and compares conditional score-estimation approaches on super-resolution, inpainting, and edge-to-image translation.These tasks evaluate conditional image generation across three inverse-problem settings.
- Contributions: The authors prove consistency for the conditional denoising estimator, providing theoretical justification for an approach previously used without justification.The result supplies a theoretical foundation for future use of the estimator.
- Contributions: Multi-speed diffusion lets different input-tensor parts diffuse at different speeds, yielding the conditional multi-speed diffusive estimator CMDE.CMDE is presented as a novel conditional-score estimator that unifies previous estimation methods.
- Contributions: The paper accompanies its theoretical and empirical work with the open-source MSDiff library for conditional and multi-speed diffusion research.The code is stated to be released in the near future.
2. Notation
The paper establishes notation for vectors, probability distributions, random-variable realizations, and the joint distribution used in later derivations.
- Indexing vectors: Vectors are written as ordered tuples such as v = (v1, ..., vn) ∈ Rn.The notation also introduces index constraints 1 ≤ i < j < n.
- Probability distributions: A random variable's probability distribution is denoted solely by the name of its density's argument.This convention is used throughout the paper's notation.
- Probability distributions: The symbol xt denotes a realization of the random variable Xt.The lowercase and uppercase forms distinguish a realization from its random variable.
3. Methods
The paper develops and compares estimators for conditional scores in score-based diffusion, including a theoretically justified conditional denoising estimator and the novel CMDE. Multi-speed diffusion interpolates between existing approaches by balancing optimization and approximation errors.
- Conditional score estimation: Score-based diffusion models learn conditional image distributions by estimating ∇x_t ln p(x_t|y) for reverse-time sampling.The paper reviews conditional denoising, conditional diffusive, and multi-speed conditional diffusive estimators.
- Conditional denoising estimator: The conditional denoising estimator is theoretically consistent for the conditional score under the stated technical assumptions.The proof establishes validity of the estimator, and consistency follows as the number of Monte Carlo samples approaches infinity.
- Conditional diffusive estimator: CDiffE diffuses both target and condition, reducing optimization error while introducing approximation error from replacing p(x_t|y) with p(x_t|ŷ_t).The condition is sampled from p(y_t|y), so the learned conditional distribution is generally an approximation to the desired one.
- Conditional multi-speed diffusive estimator: CMDE uses different diffusion speeds for target and condition, interpolating between CDE and CDiffE to balance optimization and approximation errors.Reducing the condition’s diffusion rate brings p(x_t|y_t) closer to p(x_t|y), potentially making optimization harder.
- Multi-speed theory and extensions: CMDE’s objective upper-bounds joint negative log-likelihood with suitable weighting, and its approximation-error bound vanishes as the condition diffusion speed approaches zero.CMDE coincides with CDiffE when σ_y(t)=σ_x(t) and approaches CDE as σ_y(t)→0.
4. Experiments
The experiments compare conditional score-based diffusion approaches across inpainting, super-resolution, and edge-to-image translation, using reconstruction, consistency, FID, and diversity measures.
- Experimental setup: The study evaluates conditional diffusion approaches on inpainting, super-resolution, and edge-to-image translation tasks.The experiments use CelebA and Edges2shoes datasets and compare approaches including CDE, CDiffE, and CMDE.
- Evaluation: FID compares unconditional image and reconstruction distributions, while JFID compares their joint distributions with observations.The evaluation uses 5000 samples and includes diversity through per-pixel reconstruction variability.
- Inpainting: Inpainting uses randomly positioned square masks covering 25% of CelebA images.The conditional score model reconstructs each image from its masked observation.
- Inpainting: CDE and CMDE significantly outperform CDiffE on inpainting, while CMDE achieves the best FID scores.CDE has a small advantage over CMDE in reconstruction error and consistency.
- Super-resolution: 8x super-resolution compresses 160x160 CelebA images to 20x20 observations using bicubic downscaling.CMDE and CDE perform similarly, significantly outperform CDiffE, while CMDE has the smallest reconstruction error and best FID-based distribution capture.
- Edge to image translation: In edge-to-image translation, CDiffE fails to produce samples consistent with the sketch condition, whereas CDE and CMDE are comparable.CDE performs slightly better across all metrics, while CMDE may improve with tuning of σy(t).
5. Comparison with state-of-the-art
The paper compares score-based diffusion approaches with HCFlow for super-resolution and finds a trade-off between per-image reconstruction error and distributional coverage.
- Super-resolution comparison: HCFlow performs marginally better than CDE and CMDE in reconstruction error for super-resolution.The comparison uses HCFlow preprocessing and hyperparameters exactly as specified in its original paper.
- Super-resolution comparison: CDE and CMDE obtain significantly better FID and diversity scores than HCFlow, indicating better distribution coverage.The authors regard FID as the most principled metric because exact per-image reconstruction is generally undesirable for ill-posed inverse problems.
- Super-resolution comparison: CMDE is identified as the most successful approach for approximating the posterior distribution according to FID scores.This conclusion follows the comparison of score-based diffusion methods with HCFlow.
6. Conclusions and future work
The paper establishes theoretical support for conditional denoising and introduces multi-speed diffusion through CMDE. Experiments show competitive conditional generation and improved FID in selected tasks.
- Contributions: The paper proves consistency of the conditional denoising estimator, providing theoretical justification for its use.The result establishes consistency for estimating the conditional score under technical assumptions.
- Contributions: CMDE uses multi-speed diffusion to interpolate between CDE and CDiffE by controlling the condition’s diffusion speed.The framework introduces a novel conditional-score estimator.
- Experimental conclusions: CMDE and CDE perform on par while significantly outperforming CDiffE across the evaluated conditional generation tasks.The difference is especially apparent in edge-to-image translation, where CDiffE produces samples inconsistent with the condition.
- Experimental conclusions: CMDE outperforms CDE on FID scores for inpainting and super-resolution.The authors associate appropriate condition diffusion speed with better posterior approximation and a potentially beneficial optimization landscape.
- Experimental conclusions: Score-based diffusion models perform on par with prior state-of-the-art methods in super-resolution and achieve better posterior approximation by FID.The conclusion emphasizes distributional accuracy rather than only per-image reconstruction.
A. Variance schedule
Discrete training can make conditional score estimation difficult when x and y diffuse at different speeds. VS-CMDE gradually reduces y’s diffusion rate, improving the approximation and outperforming CMDE in discrete training, but its advantage is not consistent with continuous training.
- Variance schedule: Discrete training with σy max = 1 produced a conditional distribution far from the true distribution under the tested configuration.The authors attribute the difficulty to discrete training combined with a large difference between x and y diffusion speeds.
- Variance schedule: VS-CMDE gradually decreases y’s diffusion rate during training to make the optimization problem easier.Its schedule starts with a larger diffusion rate and decreases toward the target maximum over M training iterations.
- Variance schedule: Faster early diffusion of y smooths the joint distribution, helping the score model fit an easier vector field before later refinement.The initial fit serves as initialization for estimating the harder score when y diffuses more slowly.
- Variance schedule: In continuous training, VS-CMDE outperformed vanilla CMDE on Edges2shoes and matched CDE on JFID and LPIPS while producing slightly more diverse samples.CDE outperformed CMDE on FID for Edges2shoes, while performing on par with VS-CMDE.
- Variance schedule: The authors conclude that VS-CMDE is preferable in discrete training, whereas continuous training provides no consistent competitive advantage over CMDE.This conclusion summarizes the reported training-regime comparison.
B.2. Consistency of CDE
The consistency analysis establishes that Monte Carlo training of the conditional denoising estimator converges to the conditional score under compactness, continuity, integrability, and uniqueness assumptions. The broader multi-SDE analysis connects multi-speed diffusion to likelihood-weighted score-model objectives.
- B.2. Consistency of CDE: The proof assumes compact parameter and data spaces, continuity of the objective, an integrable envelope, and a unique parameter realizing the target score.These conditions support uniform convergence and identification of the population minimizer.
- B.2. Consistency of CDE: Under technical assumptions, the conditional denoising estimator is a consistent estimator of ∇x_t ln p(x_t|y) as the Monte Carlo sample count increases.The result follows for minimizers of the Monte Carlo approximation.
- B.3. Likelihood weighting: The generalized reverse-time SDE and Girsanov analysis yield a likelihood-weighted objective for multi-speed and multi-SDE score models.The derivation uses a positive-definite covariance matrix and relates the objective to KL terms and denoising score matching.
- B.3. Likelihood weighting: The multi-speed system represents x and y as a single multidimensional SDE with a block-structured covariance matrix.The diagonal blocks use σx(t) and σy(t), with zero cross-covariance terms.
B.4. Mean square approximation error
The approximation analysis bounds the error from replacing the conditional score with a score based on a diffused condition. Under smoothness, positivity, and compactness assumptions, the bound decreases as the condition’s diffusion variance vanishes.
- B.4. Mean square approximation error: The analysis assumes positive conditional densities and compact data space, alongside regularity conditions used in the supporting lemmas.These assumptions provide boundedness and continuity needed for the error bounds.
- B.4. Mean square approximation error: Gaussian smoothing of the condition converts the relevant conditional density into a convolution with variance σy(t)^2.The proof uses the normal distribution of Y_t conditioned on Y and exchanges differentiation with integration under its assumptions.
- B.4. Mean square approximation error: Under Assumptions 3–5, the mean square approximation error is bounded by a function that decreases to zero as σy(t)^2 decreases to zero.The bound combines terms controlled by Gaussian smoothing and a term proportional to L^2σy(t)^2.
- B.4. Mean square approximation error: Bayes’s rule decomposes the conditional score into a score for the conditional density of y given x_t and the marginal score of x_t.The proof then bounds the resulting summands separately.
C. Architectures and hyperparameters
The experiments use nearly identical neural-network architectures across tasks and estimators to support fair comparisons. The main score model is a 43.5M-parameter DDPM, with task-specific batch sizes and EMA training.
- Architectures and hyperparameters: The same architecture was used across tasks and estimators, with only the final convolution differing to support joint score estimation.The diffusive estimators use six final filters instead of three for CDE, producing a negligible parameter-count difference.
- Architectures and hyperparameters: The DDPM uses channel dimension 96, depth multipliers, two ResNet blocks per scale, and attention in the final three scales.The total parameter count is 43.5M.
- Architectures and hyperparameters: Training uses EMA rate 0.999, the optimizer settings from, batch size 50 for super-resolution and edge-to-image translation, and 100 for inpainting.Batch size therefore varies by task.
D. Extended visual results
The paper provides additional visual samples for super-resolution, inpainting, and edge-to-shoe synthesis in Figures 9–11.
- Additional super-resolution samples are provided in Figure 9.
- Additional inpainting samples are provided in Figure 10.
- Additional edge-to-shoe synthesis samples are provided in Figure 11.