PaperlayerView exact revision on arXiv

Source-linked AI summary

UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models

Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, Jiwen Lu

arXiv:2302.04867v4cs.LGcs.CV

TL;DR

Pre-trained DPM sampling remains expensive, and existing fast samplers often struggle when fewer than 10 evaluations are available. UniPC combines a no-extra-evaluation corrector with an arbitrary-order predictor in a unified framework. It achieves strong unconditional and conditional sampling results, including 3.87 FID on CIFAR10 and 7.51 FID on ImageNet 256 × 256 with 10 evaluations, while remaining behind training-based methods.

  • Problem

    Existing fast samplers provide insufficient sampling quality in applications favoring fewer than 10 function evaluations.

  • Method

    UniPC combines UniC, which improves existing samplers without extra model evaluations, with arbitrary-order UniP predictors in a unified analytical framework.

  • Results

    UniPC performs consistently better than previous methods across unconditional and conditional tasks using pixel-space and latent-space DPMs, achieving 3.87 FID on CIFAR10 and 7.51 FID on ImageNet 256 × 256 with 10 evaluations.

  • Takeaways & Limitations

    UniPC supports fast, few-step DPM sampling and can improve the order of accuracy of off-the-shelf fast samplers through UniC.

  • Takeaways & Limitations

    UniPC still lags behind training-based methods, and closing that gap requires future work.

Abstract

from arXiv · show

Diffusion probabilistic models (DPMs) have demonstrated a very promising ability in high-resolution image synthesis. However, sampling from a pre-trained DPM is time-consuming due to the multiple evaluations of the denoising network, making it more and more important to accelerate the sampling of DPMs. Despite recent progress in designing fast samplers, existing methods still cannot generate satisfying images in many applications where fewer steps (e.g., $<$10) are favored. In this paper, we develop a unified corrector (UniC) that can be applied after any existing DPM sampler to increase the order of accuracy without extra model evaluations, and derive a unified predictor (UniP) that supports arbitrary order as a byproduct. Combining UniP and UniC, we propose a unified predictor-corrector framework called UniPC for the fast sampling of DPMs, which has a unified analytical form for any order and can significantly improve the sampling quality over previous methods, especially in extremely few steps. We evaluate our methods through extensive experiments including both unconditional and conditional sampling using pixel-space and latent-space DPMs. Our UniPC can achieve 3.87 FID on CIFAR10 (unconditional) and 7.51 FID on ImageNet 256$\times$256 (conditional) with only 10 function evaluations. Code is available at https://github.com/wl-zhao/UniPC.

1 Introduction

DPM sampling is costly because denoising requires multiple model evaluations, while existing fast samplers provide limited evidence and quality at fewer than 10 steps. UniPC addresses this with a training-free predictor-corrector framework that improves accuracy without extra evaluations and achieves strong few-step results.

  • Sampling pre-trained DPMs is time-consuming because denoising requires multiple model evaluations.
  • Existing training-free samplers reduce sampling to fewer than 20 steps, but quality below 10 steps remains insufficient for applications requiring efficient plausible samples.
  • UniPC combines UniC, which reuses current model outputs without extra evaluations, with UniP, a family of arbitrary-order predictors.
  • UniPC supports unified arbitrary-order formulations and can improve existing samplers while maintaining inference speed and memory usage similar to DPM-Solver++.
  • 3.87 FID on CIFAR10 and 7.51 FID on ImageNet 256 × 256 are achieved with only 10 function evaluations.

2 Background and Related Work

DPMs gradually add Gaussian noise during training and use a learned noise predictor to define sampling dynamics. Fast samplers avoid retraining by numerically solving diffusion SDEs or ODEs, but existing ODE solvers have limited explicit high-order forms.

  • 2.1 Diffusion Probabilistic Models: DPMs transform data into an approximately normal distribution by gradually adding Gaussian noise.
  • 2.1 Diffusion Probabilistic Models: The model ϵθ(xt, t) predicts noise from noisy data, with parameters learned using a weighted training objective.
  • 2.2 Fast Sampling of DPMs: Training-free fast samplers directly reuse pretrained information, avoiding further training costs and facilitating conditional sampling.
  • 2.2 Fast Sampling of DPMs: Training-free methods accelerate sampling by numerically solving the diffusion SDE or ODE.
  • 2.2 Fast Sampling of DPMs: Existing diffusion-ODE solvers lack explicit analytical forms for orders above 3, limiting directly computable high-order variants.

3 A Unified Predictor-Corrector Solver

UniPC unifies predictor and corrector construction for diffusion ODE sampling. UniC improves an initial estimate using current and previous information, while UniP provides arbitrary-order prediction; together they yield higher-order, reusable samplers for few-step settings.

  • UniPC consists of UniP and UniC, which share one analytical form while supporting arbitrary predictor order and corrector-based refinement.
  • 3.1 The Unified Corrector UniC-p: UniC refines an initial estimate using the current point together with previous points.
  • 3.1 The Unified Corrector UniC-p: The corrector uses the half-log-SNR step size hi and an increasing sequence ri to determine the points used in its update.
  • UniC-p has (p + 1)-th-order accuracy under stated regularity conditions, while UniP-p has p-th-order accuracy.
  • 3.1 The Unified Corrector UniC-p: UniC reuses stored model evaluations, avoiding the extra evaluation that ordinarily doubles predictor-corrector computation.
  • Combining UniP-p with UniC-p produces UniPC-p with (p + 1)-th-order accuracy and convergence order p + 1 under additional conditions.
  • Multistep UniPC reuses previous evaluations and is used by default because it is empirically more efficient with limited model evaluations.
  • UniPC can also adapt to data prediction models and conditional sampling with dynamic thresholding.

4 Experiments

Experiments evaluate UniPC across unconditional and conditional diffusion sampling, including pixel-space and latent-space models, with emphasis on very few function evaluations. Results show improved quality, convergence, diversity, and plug-and-play gains, while performance remains below training-based methods.

  • Unconditional sampling: UniPC consistently improves unconditional sampling quality over DPM-Solver++ on CIFAR10, LSUN Bedroom, and FFHQ, especially with fewer NFE.Compared with DPM-Solver++, FID improves by 6.0 on CIFAR10, 5.9 on LSUN Bedroom, and 8.5 on FFHQ.
  • Conditional sampling: UniPC produces better-quality and faster-converging conditional samples than competing methods under classifier and classifier-free guidance.The conditional experiments use ImageNet 256×256 and Stable Diffusion settings with different guidance configurations.
  • Conditional sampling: UniPC has lower l2 distance to 1000-step DDIM than DPM-Solver++ in Stable Diffusion guided sampling, indicating faster convergence.The convergence error is measured using the l2 distance between each method's results and 1000-step DDIM.
  • Ablations: Both tested B(h) implementations outperform DPM-Solver++, while B1(h) performs better at 5–6 sampling steps and B2(h) catches up as NFE increases.For guided sampling, B1(h) is consistently worse than B2(h).
  • UniC analysis: UniC increases baseline solver order by 1 and improves sampling quality across singlestep and multistep DPM-Solver++ variants.The evaluation measures FID↓ on CIFAR10 across different baseline orders and updating methods.
  • Additional analyses: UniPC generates more diverse CIFAR10 samples than DPM-Solver++ across varied function-evaluation counts and surpasses previous methods by a large margin at 10–25 NFE.Diversity is measured by Inception Score, while the broader comparison uses guided ImageNet 256×256 sampling with guidance scale 8.0.
  • Efficiency and visual quality: UniPC's wall-clock inference time and memory usage are similar to DPM-Solver++, while qualitative results at 7 NFE show more plausible samples than competing methods.DEIS tends to crash at extremely few steps, and DDIM samples are relatively blurry in the cited visualization.
  • Limitations: UniPC still lags behind training-based methods, leaving further work to close the gap between training-free and training-based sampling.This limitation is stated in the paper's limitations and broader-impact discussion.

5 Conclusions

UniPC provides a unified, arbitrary-order predictor-corrector framework for fast DPM sampling and can improve existing samplers without extra model evaluations. Experiments support its effectiveness across sampling settings, while the authors identify further design directions.

  • UniPC unifies UniP and UniC for arbitrary orders and can improve off-the-shelf fast samplers through UniC.
  • UniPC consistently produces better-quality Stable-Diffusion-XL images than DPM-Solver++ at 15 NFE, whose outputs contain visible artifacts.
  • Extensive experiments demonstrate UniPC effectiveness on unconditional and conditional sampling with pixel-space and latent-space pre-trained DPMs.
  • The paper identifies better B(h), more accurate ϵθ(˜xc ti, ti), and improved order schedules as directions for further improvement.

A.2 Adapting UniPC to Data Prediction Model

UniPC is adapted to data prediction models with separate predictor and corrector formulations. Under the stated regularity and step-size assumptions, the resulting UniPC-p achieves order p + 1 accuracy.

  • UniPC-p adapts the framework to data prediction models using half-log-SNR coordinates and previous data points.
  • UniC can correct outputs from any solver for the data prediction model.
  • Under regularity assumption E.3 and nonzero B(h) = O(h), UniPC-p has order of accuracy p + 1.
  • The paper lists separate detailed algorithms for UniC-p and UniP-p in the data prediction setting.

C UniPC with varying coefficients (UniPCv)

UniPCv uses varying coefficients to construct a unified update whose coefficients depend on the selected sequence of previous-point ratios. The method is proved to achieve (p + 1)-th-order accuracy.

  • UniPC-p matches derivative coefficients simultaneously using a vector ap.
  • For the data prediction model, UniC-p and UniP-p are provided as separate algorithmic procedures using buffered predictions.
  • UniPCv is (p + 1)-th order accurate, while C_p is invertible under the stated conditions.
  • The multistep UniPC-p implementation computes derivative differences from buffered model outputs and solves for update coefficients.
  • UniPCv replaces the standard update with formula (14), using Ap = C_p^-1; Ap depends only on the selected {r_i}.

D Order of Convergence

Under regularity conditions, UniP-p has convergence order p, while UniPC-p achieves order p + 1 for noise and data prediction models.

  • Regularity assumptions: The analysis assumes Lipschitz model behavior, step sizes h = O(1/M), nondegenerate signal scales, and sufficiently accurate starting values.The assumptions support stability and multistep convergence analysis.
  • Convergence results: UniPC-p achieves convergence order p + 1, while UniP-p achieves order p under the stated regularity conditions.The result applies to the noise prediction model; analogous data-prediction corollaries are also given.
  • Convergence results: UniC-p can raise the convergence order of general p-order diffusion ODE solvers to p + 1.This includes solvers such as DDIM and DPM-Solver variants within their stated orders.
  • Data prediction model: For the data prediction model, UniP-p has order p and UniC-p has order p + 1 under additional assumptions.These assumptions include Lipschitz continuity, controlled step sizes, bounded signal ratios, and suitable starting values.
  • Analytical construction: The framework uses Taylor expansions and exponentially weighted integrals in half log-SNR λ to derive high-order predictor and corrector updates.The same analytical strategy is developed for both noise and data prediction models.

E.6 Proof of Proposition D.5

The proof establishes order p convergence for UniP-p by bounding propagated errors under Lipschitz and step-size regularity assumptions.

  • Error control: Lipschitz continuity transfers state approximation errors into bounded model-output errors.The bound is expressed using the model Lipschitz constant L.
  • Error propagation: The proof combines local update bounds with controlled step sizes and initialization errors to propagate the global error estimate.The starting values satisfy an O(h^p) bound under the multistep initialization assumption.
  • Proof conclusion: UniP-p has convergence order p under the stated assumptions.The proof controls the maximum accumulated approximation error across sampling steps.

E.7 Proof of Proposition D.6

The proof of Proposition D.6 shows that UniPC-p attains order p + 1 by combining predictor error control with UniC’s higher-order local accuracy.

  • Oracle analysis: The oracle predictor has local error O(h^p+2), which supports the higher-order UniPC convergence result.This bound is combined with the corrector analysis.
  • Error propagation: The argument requires suitable initial errors and uses recursive bounds for both predictor and corrected trajectories.These bounds are established for the initial steps and then iterated.
  • Non-oracle analysis: For non-oracle sampling, the proof separately bounds predictor errors before applying the theorem to obtain the stated order.The initialization satisfies an O(h^p) bound for the first p steps.
  • Proof conclusion: UniPC-p achieves convergence order p + 1 under the proposition’s regularity and initialization assumptions.The proof applies the local truncation error together with recursive error bounds.

F.1 Implementation Details about UniPC

UniPC is implemented by default as a multistep method that reuses prior evaluations, with warm-up steps and no final corrector evaluation for fair cost comparisons.

  • Coefficient computation: For UniP-2 and UniC-1, the coefficient a1 can be set directly to 1/2 without solving an equation.For higher orders, the coefficient vector is computed by inverting the Rp matrix.
  • Warm-up procedure: UniPC requires a warm-up procedure to gather enough data points for high-order updates.This procedure is also used in previous multistep approaches.
  • Evaluation budget: The corrector is omitted after the final predictor because evaluating the current-timestep model output would add an extra function evaluation.This omission is made for fair comparisons.
  • Experiments: The experiments include additional unconditional sampling results on CIFAR10 and FFHQ.These results are reported in Tables 8 and 9.

G.1 More Quantitative Results

Additional experiments compare UniPC and existing samplers on unconditional CIFAR10 and conditional ImageNet 256×256 sampling, including extremely few function evaluations. Results also examine solver variants, guidance scales, and qualitative outputs at 7 NFE.

  • Unconditional Sampling: UniPCv achieves the best sampling quality at 7–9 NFE, but other UniPC variants perform better at 5, 6, and 10 NFE.The results indicate that variant selection depends on the available NFE budget.
  • Conditional Sampling: DPM-Solver++ and UniPC are evaluated against DEIS in additional conditional ImageNet 256×256 sampling results.The results are reported in Table 11.
  • Conditional Sampling: B2(h) significantly outperforms B1(h) in guided sampling, while UniPC achieves better quality with both large and small guidance scales using few steps.The paper suggests that B1(h) = h is too simple for guided sampling.
  • Qualitative Comparisons: Figure 5 compares DDIM, DPM-Solver++, and UniPC on ImageNet 256×256 samples generated with only 7 NFE.The comparison fixes the initial noise across methods and randomly selects classes from ImageNet’s 1000 classes.
Loading 2302.04867v4…