PaperlayerView exact revision on arXiv

Source-linked AI summary

Iterating Consistency Models: Stability, Error Bounds and Noise Schedules

Alessio Spagnoletti, Abdul-Lateef Haji-Ali, Andrés Almansa, Alain Oliviero Durmus, Eric Moulines, Marcelo Pereyra

arXiv:2610.03414v1stat.MLcs.CVcs.LG

TL;DR

Multistep consistency-model samples can improve or degrade as steps are added, with quality highly sensitive to the noise schedule and existing theory offering limited guidance. The paper analyzes sampling as alternating noising and approximate denoising, derives bounds that separate contraction from accumulated error, and reports that measured error profiles closely follow the predicted forms.

  • Problem

    Adding consistency-model sampling steps can improve or degrade sample quality depending sensitively on the schedule, while existing theory does not fully explain this behavior.

    Source: §1 · p. 2

  • Method

    The paper analyzes multistep sampling as alternating noising and approximate denoising, deriving error bounds under stability assumptions and a complementary guarantee with estimable trained-model assumptions.

    Source: §1 · p. 2, §3.3 · p. 7

  • Results

    The estimated contraction and approximation profiles can be measured reliably and closely match the functional forms predicted by the theory.

    Source: §1 · p. 2, §4 · p. 9

  • Takeaways & Limitations

    The bounds distinguish schedule roles: large early noise levels drive contraction, while small late noise levels control residual bias, providing a basis for sampler design.

    Source: §5 · p. 9

    The certificate cannot exploit cancellation between initialization and model errors, so opposite variance errors can add in the certificate despite partly canceling in the law.

    Source: §3.4 · p. 8

Abstract

from arXiv · show

Consistency models (CMs) have become a leading approach for generating high-quality samples in few steps. However, adding steps can improve or degrade sample quality in ways that are highly sensitive to the schedule and that existing theory does not fully explain. To provide accuracy guarantees and guide CM sampler design, we analyze multistep CM sampling as a composition of noising and approximate denoising operators. Under explicit, verifiable stability assumptions, we derive a non-asymptotic error bound that separates contraction of the initialization error from accumulation of approximation error. The bound assigns distinct roles to the schedule: large early noise levels drive contraction, while small late noise levels control the residual bias. As a corollary, we obtain explicit constants for strongly log-concave and semi-log-concave targets. We further establish a complementary guarantee whose assumptions, one-step accuracy and stability, can be estimated for a given trained model. Experiments show that the contraction and approximation profiles entering our bounds can be reliably measured and closely match the predicted functional forms. Together, these results provide a meaningful convergence theory for multi-step CMs and a practical route to sampler design.

1 INTRODUCTION

The introduction frames multistep consistency-model sampling as a trade-off between contraction of sampling error and accumulation of model bias, with quality sensitive to the noise schedule. The paper develops convergence guarantees to explain that trade-off and guide sampler design.

1 INTRODUCTION

  • Consistency models map noisy points along a probability-flow ODE trajectory directly to clean data, and can be distilled from a pretrained diffusion model or trained from scratch.Training approaches include consistency distillation and consistency training.Source: §1 · p. 1
  • Multistep sampling is brittle: added steps may improve or degrade sample quality, which is highly sensitive to the schedule and not well explained by existing theory.Stopping early can sometimes yield better samples than continuing.Source: §1 · p. 2
  • The non-asymptotic error bound separates contraction of initialization error from accumulated approximation error, with large early noise driving contraction and small late noise controlling residual bias.Explicit constants are obtained for strongly log-concave and semi-log-concave targets.Source: §1 · p. 2
  • A complementary guarantee uses one-step accuracy and learned-map stability assumptions that can be estimated for a trained consistency model.Source: §1 · p. 2
  • Experiments find that the contraction and approximation profiles in the bounds can be measured reliably and closely match the predicted functional forms.Source: §1 · p. 2
  • Building on Lyu et al. (2024), the analysis places contraction stability on the exact denoiser, allows general Gaussian noising and schedules, and permits score error to grow as noise approaches zero.The paper notes that noising no longer smooths the target as the noise level tends to zero.Source: §1 · p. 2

2 BACKGROUND

Diffusion models generate samples by reversing a noising process, while consistency models approximate the flow map back to clean data. The section introduces the multistep setting and the paper's bounds for analyzing it.

2 BACKGROUND

  • A diffusion model starts with data corrupted by a time-dependent Gaussian noising process and generates samples by reversing that process.Source: §2 · pp. 2–3
  • The probability-flow ODE preserves the noisy marginals when run backward, yielding a map from time t to clean data at time 0.Source: §2 · p. 3
  • A consistency model approximates this flow map, predicting the time-zero endpoint from a noisy state in one neural function evaluation.The flow map obeys self-consistency along a trajectory and the boundary condition that the time-zero map is the identity.Source: §2 · p. 3
  • Multistep sampling alternates forward noising with reverse transport; in practice, 2–8 iterations may suffice, but additional steps can help or hurt depending on the schedule.Source: §2 · p. 3
  • The paper bounds the distance between the sampler output law and the target, separating initialization-error contraction from approximation-error accumulation.Large early noise levels drive contraction, while small late noise levels control approximation error; constants are explicit for strongly log-concave and semi-log-concave targets.Source: §2 · p. 3

3 MAIN RESULTS

The section develops schedule-dependent multistep error certificates, including stability and approximation profiles and bounds estimable for trained consistency models. Gaussian examples examine certificate behavior: mean errors attain the bounds, while variance errors can reveal conservatism or missed cancellation.

3.1 CONVERGENCE OF MULTISTEP CONSISTENCY MODELS

  • The finite-horizon W2 certificate separates attenuated initialization error from accumulated step-approximation error, and a schedule balance condition yields a geometric bound.For a repeated level with q_t < 1, the accumulated-error bound approaches E_t/(1−q_t); an extra step improves the certificate only when the current bound exceeds that plateau, not necessarily the actual error.Source: §3.1 · p. 4

3.2 NOISE-DEPENDENT STABILITY AND APPROXIMATION ERROR

  • For approximate-score flows, noising is exactly α_t-Lipschitz, while target-curvature and score-error assumptions yield explicit denoising stability and approximation profiles.The strongly log-concave case gives explicit profiles under the stated score-error bound and positive strong-convexity assumptions.Source: §3.2 · pp. 4–5
  • With relaxed curvature, the exact sampling step is contractive at every noise ratio when ℓ≤m, while for ℓ>m sufficiently large noise restores certified contraction but raises the error budget.The theorem extends the strongly log-concave case to curvature defects controlled by ℓ, with contraction determined by the noise ratio.Source: §3.2 · pp. 5–6

3.3 MULTISTEP BOUNDS ESTIMATED FROM A TRAINED MODEL

  • A learned-map recursion replaces exact-step stability with learned-map stability and target error, enabling multistep certificates from estimates for a trained model.The trained-CM theorem bounds target error using cell-wise consistency errors and learned-map Lipschitz constants, without requiring uniformly bounded consistency residuals; numerical solver error must also be included in the appropriate norm.Source: §3.3 · pp. 6–7

3.4 A GAUSSIAN EXAMPLE, AND TIGHTNESS OF CERTIFICATES

  • For a Gaussian target and affine learned map, the certificate is attained for mean errors but overstates variance errors.Source: §3.4 · p. 7
  • Mean errors attain the bounds exactly, so along a decreasing schedule error can first fall and then rise; at a repeated level it approaches a plateau, improving poor but degrading good initializations.With the target variance reproduced, the variance stays exact and the Wasserstein error follows the recurrence with equality at every step.Source: §3.4 · p. 8
  • For variance errors, the certificate misses a compression factor and decays like q^n at a repeated level, while actual error eventually decays like q^2n.The certificate overstates inherited variance error by κ; Figure 1 compares certificate and exact error at a repeated VE level.Source: §3.4 · p. 8, Figure 1 · p. 8
  • When the trained model has the wrong target variance, its map difference is unbounded, so no uniform E_t satisfies A4, though Proposition 3.9 still applies.That proposition uses the learned map and the target error Eπ_t.Source: §3.4 · p. 8
  • The certificate cannot exploit cancellation: opposite-direction initialization and model variance errors partly cancel in the distribution but add in the certificate.At a repeated level, the certificate plateau exceeds limiting error by a factor that approaches one as the offset grows.Source: §3.4 · p. 8

4 NUMERICAL VALIDATION ON STABLE DIFFUSION (SD) 1.5

Experiments on two latent consistency models found that estimated contraction profiles were contractive and closely followed predicted functional forms; the estimated plug-in bound also tracked held-out error.

4 NUMERICAL VALIDATION ON STABLE DIFFUSION (SD) 1.5

  • Validation used a VE LCM distilled from Stable Diffusion 1.5 and a TrigFlow LCM trained on CelebA in a KL-regularized VAE latent space.The profiles were estimated with optimal-transport W2 estimators and compared with Corollary 3.4 predictions for strongly log-concave targets.Source: §4 · pp. 8–9
  • The approximate contraction profiles were contractive and closely followed predicted forms, with fitted m ≈ 0.8 for SD 1.5 and m ≈ 0.5 for CelebA.Independent covariance estimates gave m > 0.6 and m > 0.3, respectively; Figure 2’s analytical curve is a fitted form, not a certificate of global contraction.Source: Figure 2 · p. 9, §4 · p. 9
  • Since Et bounds approximation error uniformly over inputs while Eπ is the approximate target-only one-step error, predicted Et should exceed approximate Eπ.Source: §4 · p. 9, Figure 3 · p. 9
  • The SD 1.5 plug-in one-step W2 bound lay above held-out empirical W2 error and tracked its variation with noise level.The comparison combines estimated SD 1.5 profiles; the reported bound is an upper bound, not an equality.Source: §4 · p. 9

5 CONCLUSION

The paper gives a finite-horizon W2 recursion that separates initialization contraction from approximation-error accumulation, then derives schedule-sensitive bounds and validates predicted profiles empirically.

5 CONCLUSION

  • The finite-horizon W2 recursion separates contraction of initialization error from accumulation of approximation error.Stability is placed on the exact denoiser, making the contraction factor a target property with explicit forms under strong log-concavity and, above a noise threshold, semi-log-concavity.Source: §5 · p. 9
  • Under a balance condition, the recursion yields a sharp geometric bound, and a complementary guarantee extends the analysis to trained consistency models under estimable assumptions.The conclusion identifies one-step accuracy and stability as the assumptions for the trained-model guarantee.Source: §5 · p. 9
  • Experiments corroborate the analysis: estimated contraction and approximation profiles closely follow the predicted functional forms.The stated future direction is optimizing schedules directly from the bound and extending analysis to zero-shot samplers with guidance or rewards.Source: §5 · p. 9

A CURVATURE UNDER GAUSSIAN SMOOTHING AND STABILITY OF THE PROBABILITY FLOW

The appendix establishes trajectory stability through curvature properties of Gaussian-smoothed targets, including strongly log-concave and relaxed weak-curvature settings.

A CURVATURE UNDER GAUSSIAN SMOOTHING AND STABILITY OF THE PROBABILITY FLOW

  • For exact and approximate trajectories, Lemma A.1 bounds their distance at time s by a stability factor times their distance at time t plus an additive error.The lemma applies for 0 ≤ s ≤ t under Theorem 3.3’s assumptions.Source: A CURVATURE UNDER GAUSSIAN SMOOTHING… · p. 13
  • For an m-strongly log-concave target, Gaussian smoothing preserves strong log-concavity with curvature μr = m/(1 + mr^2).The strongly log-concave pathwise case applies this smoothed curvature in Lemma A.1 under Theorem 3.3’s parametrization and flow-existence assumptions.Source: A CURVATURE UNDER GAUSSIAN SMOOTHING… · p. 14
  • A sufficient condition for relaxed strong log-concavity outside a ball is Hessian curvature at least m outside and at least −β inside, yielding ℓ = 2 max{m + β, (m + β)^2ϱ^2}.The segment through the ball has length at most min(u, 2ϱ), which supports the relaxed curvature condition.Source: A CURVATURE UNDER GAUSSIAN SMOOTHING… · p. 15
  • Gaussian smoothing preserves the relaxed outside-a-ball curvature condition with transformed parameters (mr, ℓr), and for r > 0 gives ∇²fr(x) ⪰ (mr − ℓr)Id.The appendix defines fr = −log πr and states that fr is C1 under the relaxed condition and C∞ for positive noise.Source: A CURVATURE UNDER GAUSSIAN SMOOTHING… · p. 15
  • The relaxed curvature bound propagates through the smoothing flow, and its positive-noise Hessian lower bound supplies the curvature used to control trajectory separation.The proof derives a differential inequality for the distance between rescaled trajectories and uses a backward integrating factor.Source: A CURVATURE UNDER GAUSSIAN SMOOTHING… · pp. 13, 15, 16

B PROOFS OF THE MAIN RESULTS

The proofs establish complementary multistep error routes—an unrolled one-step recursion and fixed-point tracking—and extend certificates to trained flows and early stopping.

B PROOFS OF THE MAIN RESULTS

  • The one-step recursion bounds each iterate’s error by a stability-weighted previous error plus the current approximation error, without requiring contraction or ordered noise levels.The proof unrolls this recursion to obtain bound (3).Source: B PROOFS OF THE MAIN RESULTS · p. 16
  • Under the balance condition, the accumulated approximation-error terms decrease geometrically across steps, yielding the refined bound (4).The induction also covers zero error bounds and stability factors.Source: B PROOFS OF THE MAIN RESULTS · p. 16
  • When the learned step is contractive, it has a unique fixed-point law whose distance from the target is bounded by the one-step target defect divided by one minus the contraction factor.At repeated noise levels, the resulting bound plateau certifies the distance to that fixed-point law; the Gaussian example attains it.Source: B PROOFS OF THE MAIN RESULTS · p. 18
  • The fixed-point-tracking bound is no smaller than the unrolled recursion under the stated monotonicity and error-compatibility conditions, but the bounds are unordered for arbitrary error bounds.The proof requires bounds on changes between consecutive sampling operators, uniformly over input laws.Source: B PROOFS OF THE MAIN RESULTS · pp. 17–18
  • The trained-model guarantee derives accuracy and stability certificates from score-approximation and learned-map bounds, and the same estimates extend to grids that stop at a positive noise level.With early stopping, the target becomes the marginal at the stopping level and noising starts there.Source: B PROOFS OF THE MAIN RESULTS · pp. 18–19

C COMPARISON WITH OTHER MULTISTEP BOUNDS

This section compares the paper’s multistep certificates with prior bounds and gives an example where additional stability and uniform accuracy yield a sharper upper bound.

C COMPARISON WITH OTHER MULTISTEP BOUNDS

  • For the OU-restarted sampler of Lyu et al. (2024), the stability thresholds rely on different assumptions, and the compared error budgets do not order the resulting bounds.The paper’s threshold requires curvature control and a uniform approximation bound; the cited method’s threshold uses an affine stability condition and target-marginal error bounds.Source: C COMPARISON WITH OTHER MULTISTEP BOUNDS · p. 20
  • For strongly log-concave targets, the stated two-step recursion uses contraction factors below one at every positive sampling time above δ.The cited corollary supplies an explicit approximation-error bound for this setting.Source: C COMPARISON WITH OTHER MULTISTEP BOUNDS · p. 20
  • For Chen et al. (2025), the paper retains the actual second moment and applies a support bound to strengthen their two-step bound, while noting that consistency error alone does not establish the required uniform error bounds.The additional bounds are verified in the example.Source: C COMPARISON WITH OTHER MULTISTEP BOUNDS · p. 20
  • At κ = 0.01, T = r1 = 2, r2 = 1 and ∆r = 1, the compared certificates are 0.1746 for (30), 0.7659 for (29), and 0.1600 for Proposition 3.9.The comparison concerns upper bounds, not actual sampling errors.Source: C COMPARISON WITH OTHER MULTISTEP BOUNDS · p. 21
  • The example’s two-step certificate is strictly smaller than every decreasing-schedule multistep certificate considered, illustrating a gain from additional stability and uniform accuracy.The stated comparison is between guarantees, not actual sampling errors.Source: C COMPARISON WITH OTHER MULTISTEP BOUNDS · p. 21

D EXPERIMENTAL ESTIMATORS AND IMPLEMENTATION DETAILS

The appendix documents the latent-coordinate experiments and the finite-sample pairwise estimator used to assess learned-step stability.

D EXPERIMENTAL ESTIMATORS AND IMPLEMENTATION DETAILS

  • The SD 1.5 consistency-distillation run uses 1000 epochs on Conceptual 12M and evaluates quantities at 50 VE noise levels from the Karras schedule.The schedule levels are denoted σ_j.Source: D EXPERIMENTAL ESTIMATORS AND… · p. 21

D.1 LATENT COORDINATES AND EXPERIMENTAL SETUP

  • The experiments express noisy samples in normalized latent coordinates, with F_t giving the clean-latent prediction; the SD 1.5 and CelebA setups use distinct latent pools and noise grids.For VE, r = t = σ; for TrigFlow, r = tan t.Source: §D.1 · p. 21

D.2 PAIRWISE STABILITY ESTIMATOR

  • Pairwise stability is estimated from secant ratios between learned denoiser outputs at noisy particles, using a high empirical quantile to probe the upper tail with less sensitivity than the sample maximum.The estimation uses 512 particles and 8000 pairs per level for SD 1.5, and 2048 particles and 12000 pairs per level for CelebA.Source: §D.2 · p. 22

D.3 TARGET-ONLY ERROR ESTIMATORS

The appendix specifies target-only error and one-step OT estimators, then derives an affine Gaussian example in which the theoretical certificates coincide.

D.3 TARGET-ONLY ERROR ESTIMATORS

  • The target-only error profile is a finite-sample plug-in estimate of the distributional target defect, and it does not in general certify the uniform error required by the alternative bound.A uniform bound implies a target-only bound, but the converse need not hold.Source: §D.3 · pp. 22–23

D.5 ONE-STEP EMPIRICAL OT COMPARISON

  • The one-step experiment independently evaluates each VE level from the same Gaussian initialization and compares empirical output-to-target OT distances with a recursive estimate using separately calibrated stability.The distance terms use the same empirical OT estimator.Source: §D.5 · p. 23

E THE GAUSSIAN EXAMPLE: DERIVATIONS AND FURTHER REGIMES

  • The appendix’s Gaussian example specifies an affine learned map whose slope encodes variance error and whose offset displaces the mean.With no offset, the map represents the right mean but the wrong variance when its learned precision differs from the target precision.Source: §E.1 · p. 23, E THE GAUSSIAN EXAMPLE: DERIVATIONS… · p. 23

E.1 MODEL

  • When the learned precision matches the target, the offset is the score-error profile and the learned step is the exact step followed by a shift, so the two proposition certificates coincide.The offset error envelope attains the stated profile in this case.Source: §E.1 · p. 24

E.2 HOW THE TWO ERRORS PROPAGATE

The section separates inherited variance error from error created by a learned step, while tracking mean error exactly. The certificate can overstate variance error because it loses contraction factors and signed cancellation.

Contraction

  • Mean error contracts at exactly q_t, while inherited variance error contracts more strongly by q_t/κ_t.For the variance component, κ_t is at least one, is nonincreasing, and depends on the current standard deviation.Source: §E.2 · p. 24

Error decomposition

  • A learned step splits standard-deviation error into an inherited component and a step-created component, each signed according to whether it makes the law too wide or too narrow.The mean and standard-deviation errors after one learned step from the target form the two entries of the target error.Source: §E.2 · p. 24

Certificate slack

  • The certificate matches mean error exactly but overstates inherited variance error by a bounded per-step factor that compounds across steps.Adding the sizes of signed variance components also misses cancellation between errors in opposite directions.Source: §E.2 · p. 24

Figure 4

  • In the mean-only Figure 4 setting, the certificate and exact error coincide along the illustrated schedules, and repeated-level error approaches its plateau from either side.Stars mark the optimal step count in the left panel.Source: Figure 4 · p. 25

E.3 REGIMES

The regimes show when the certificate is exact, conservative, or unable to provide a uniform bound, depending on whether errors arise in the mean or variance and whether variance errors cancel. The next noise level trades contraction of inherited error against newly introduced bias.

Mean error

  • When only mean error is present, the certificate is exact; along a decreasing schedule the error first falls and then rises, while at a repeated level it approaches the plateau from either side.The repeated-level step improves a poor initialization but degrades a good one.Source: §E.3 · p. 25

Variance error from the initialization

  • With an exact consistency model and only initialization variance error, the certificate contracts by q_t, while the actual variance error contracts faster, by q_t/κ_t.Table 1 organizes regimes by whether initialization and learned-step errors are absent or present.Source: §E.3 · p. 25, Table 1 · p. 25

Variance error from the learned step

  • A learned step with the wrong variance can satisfy A5 and A6 even when its score error is infinite, so Proposition 3.9 applies where Proposition 3.1 does not.In this regime, the certificate overstates inherited variance error and retains a gap at the plateau.Source: §E.3 · p. 25

Both errors

  • When both errors are present, variance error dominates early and mean error late; the certificate becomes tight as the dominant error shifts to the mean.The crossover comes earlier as ω_t grows and later as the initial variance error grows.Source: §E.3 · p. 25

Choosing the next level

  • Without cancellation, larger second-step noise contracts remaining variance error more strongly, but mean bias can create an interior optimum that the certificate may overshoot.For a large mean error, the certificate may recommend a step that does not help; its chosen level costs only a few percent of the error.Source: §E.3 · p. 26

Cancellation

  • Opposite-direction variance errors can nearly cancel in the law while adding in the certificate, so a recommended second step can worsen the actual error.With offset 4η_r, however, a second step helps and following the certificate costs little.Source: Table 1 · p. 25, §E.3 · p. 26

Scope

  • For Gaussian targets, the profiles extend coordinatewise, but the certificate follows the flattest direction while actual error averages across directions.This creates dimension-dependent slack; the model does not establish that these mechanisms occur in the Section 4 experiments.Source: §E.3 · p. 26

F VALIDATING THE ONE-STEP BOUND ON SD 1.5

The section tests whether a one-step bound assembled from separately estimated profiles tracks empirical W2 error for SD 1.5 consistency models. The bound stays above and follows the measured error, but multistep comparisons are not reported because estimator bias obscures them.

Experimental setup

  • The experiment starts from N(0, I), takes one noising–denoising step separately at each VE level, and compares empirical W2 distance with the plug-in one-step bound.The distance uses exact optimal assignment between the resulting latent point clouds and samples from a different target dataset than the one used for calibration.Source: F VALIDATING THE ONE-STEP BOUND ON SD… · pp. 26–27

Validation result

Limitation

  • Multistep bounds are not reported for these models because the two-step bound reaches the statistical-bias level of the empirical W2 estimator.Meaningful comparison would require vastly more samples because empirical W2 converges slowly in high dimensions.Source: F VALIDATING THE ONE-STEP BOUND ON SD… · p. 27
Loading 2610.03414v1…