PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Noise Conditioning Necessary for Denoising Generative Models?

Qiao Sun, Zhicheng Jiang, Hanhong Zhao, Kaiming He

arXiv:2502.13129v2cs.CV

TL;DR

The paper asks whether noise conditioning, widely regarded as critical, is necessary for denoising-based generative models, drawing motivation from blind image denoising. It compares conditioned and unconditioned models and analyzes the resulting errors; most models degrade modestly or sometimes improve without conditioning, and an unconditioned model reaches FID 2.23 on CIFAR-10.

  • Problem

    Noise conditioning is widely regarded as critical, and the paper asks whether blind-denoising ideas extend to denoising-based generative models.

    Source: §1 · p. 1

  • Method

    The authors compare denoising-based generative models with and without noise conditioning, analyze error in regression and iterative sampling, and adapt EDM for noise-unconditional generation.

    Source: §1 · pp. 1–2

  • Results

    Most studied models degrade modestly without noise conditioning, some improve, and the proposed noise-unconditional variant reaches FID 2.23 on CIFAR-10.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The findings motivate reconsidering noise conditioning and exploring generative models not constrained by that design.

    Source: §7 · p. 9

    The analysis assumes the network can fit the effective target and uses a Lipschitz condition that the authors say holds with high probability for an appropriate choice, not exactly on real data.

    Source: §4.4 · p. 5

Abstract

from arXiv · show

It is widely believed that noise conditioning is indispensable for denoising diffusion models to work successfully. This work challenges this belief. Motivated by research on blind image denoising, we investigate a variety of denoising-based generative models in the absence of noise conditioning. To our surprise, most models exhibit graceful degradation, and in some cases, they even perform better without noise conditioning. We provide a theoretical analysis of the error caused by removing noise conditioning and demonstrate that our analysis aligns with empirical observations. We further introduce a noise-unconditional model that achieves a competitive FID of 2.23 on CIFAR-10, significantly narrowing the gap to leading noise-conditional models. We hope our findings will inspire the community to revisit the foundations and formulations of denoising generative models.

1. Introduction

Noise conditioning is widely treated as critical in denoising generative models, but this work asks whether blind-denoising ideas make it unnecessary. It compares conditioned and unconditioned models, analyzes their errors, and introduces an unconditioned EDM-derived variant.

1. Introduction

  • The study is motivated by the possibility that noise levels can be estimated from corrupted natural images, making blind denoising feasible.It asks whether image-denoising research generalizes to denoising-based generative models.Source: §1 · p. 1
  • Most studied models degrade modestly without noise conditioning, some flow-based methods improve, and only one variant fails disastrously.The authors report that their theoretical error bound correlates with unconditional behavior and is orders of magnitude higher for catastrophic failures.Source: §1 · pp. 1–2
  • The analysis examines uncertainty in noise levels, single-step denoising error, and accumulated iterative-sampling error.The resulting bound depends only on noise schedules and the dataset, not training.Source: §1 · pp. 1–2
  • The proposed noise-unconditional EDM-derived model reaches FID 2.23 on CIFAR-10, versus 1.97 FID for noise-conditional EDM.The authors say this significantly narrows the gap to its noise-conditional counterpart.Source: §1 · p. 2

2. Related Work

Related work traces noise conditioning through diffusion and related generative models, while connecting this study to blind image denoising. The paper emphasizes that iterative generation introduces error accumulation beyond classical denoising.

Noise Conditioning

  • Noise conditioning became standard in diffusion, from early time-dependent readouts to NCSN and DDPM and their subsequent derivatives [1], [1], [1].The passage names iDDPM, ADM, and nearly all later derivatives as inheriting the practice.Source: §2 · p. 2
  • DDIM and EDM recast reverse diffusion as ODE sampling, while Flow Matching learns flow fields; noise conditioning remains the de facto choice [1a], [2], [3].Consistency Models also rely on noise conditioning and its implementation details for success [4].Source: §2 · p. 2

Blind Image Denoising

  • Blind image denoising studies noise estimation and denoising without prior noise knowledge, and neural networks have proved effective for these tasks [1], [2], [3], [4], [5], [6], [7], [8].Unlike classical blind denoising, generative models iterate, so errors can accumulate and introduce new challenges.Source: §2 · p. 2

3. Formulation

The formulation expresses denoising models through their noisy inputs, regression targets, schedules, and iterative samplers. It isolates the network’s dependence on the noise level to compare conditioned and unconditioned variants.

3. Formulation

  • The reformulation separates the neural network from the schedules so its behavior with respect to noise conditioning can be analyzed.The schedule functions are method-dependent, and the time step is referred to as the noise level.Source: §3 · p. 2, §3.1 · p. 2

3.1. Denoising Generative Models

  • Training samples clean data, noise, and a time step to form a noisy image, then minimizes a weighted regression loss.The regression target can be epsilon-, x-, or v-prediction, with method-specific schedules.Source: §3.1 · pp. 2–3
  • Table 1 collects the schedule choices used by different models in the reformulation.The schedule functions expose the neural network’s direct output, including for methods whose outputs are preconditioned.Source: Table 1 · p. 3, §3.1 · p. 3, §3.2 · p. 3
  • At inference, a sampler starts from noise and iteratively denoises using a pre-specified sequence of time steps and schedules.The formulation covers first-order samplers such as Langevin sampling and Euler ODE solvers; higher-order samplers can be formulated similarly.Source: §3.1 · p. 3
  • SDE-based solvers can add sampling-time noise, whereas ODE-based solvers do not add it.The sampling-time noise term is zero for ODE-based solvers.Source: §3.1 · p. 3

3.2. Noise Conditional Networks

  • Existing networks receive the noise level as a conditioning input, commonly through a time embedding; this study contrasts NNθ(z|t) with NNθ(z).The schedules are predesigned rather than learned network parameters.Source: §3.2 · p. 3
  • For a given noisy sample z, multiple data-noise-time triplets can imply different regression targets; the effective target averages their possible targets.Figure 2 illustrates this with Flow Matching: different triplets produce the same z but distinct targets.Source: Figure 2 · p. 3

4. Analysis of Noise-Unconditional Models

The analysis characterizes how removing noise conditioning changes regression targets and iterative sampling. It links concentrated noise-level posteriors to small target error and bounds accumulated sampling error under stated assumptions.

4.1. Effective Targets

  • The unconditioned effective target averages regression targets over possible noise levels, whereas the conditioned target depends on both z and t.The conditioned network’s effective target averages over triplets that produce the same z and t.Source: §4.1 · pp. 3–4

4.2. Concentration of Posterior p(t|z)

  • When the posterior p(t|z) is concentrated, the conditioned and unconditioned effective targets can coincide, assuming sufficient network capacity.For high-dimensional data, the paper’s Flow Matching analysis predicts a concentrated posterior; its real-data simulation finds concentration and greater concentration at smaller t*.Source: §4.1 · p. 4, §4.2 · p. 4

4.3. Error of Effective Regression Targets

  • For the analyzed Flow Matching setting, the effective-target error is substantially smaller than the target norm; real-data verification finds it at about 1/10^3 of that norm.The theoretical statement applies under its stated data-dimension conditions and uses σd for per-pixel dataset standard deviation.Source: §4.3 · p. 4

4.4. Accumulated Error in Sampling

  • The accumulated-error bound for iterative sampling is a sum of schedule-dependent terms under assumptions on target error and Lipschitz behavior.The network is assumed capable of fitting the effective target, and the Lipschitz condition is reported to hold with high probability for an appropriate Li choice.Source: §4.4 · p. 5
  • The schedules and dataset suffice to compute the bound without training, and different schedules can yield dramatically different unconditioned behavior.Although the bound implies greater accuracy for the conditional variant, experiments find unconditioned models can outperform it in some cases.Source: §4.4 · p. 5

5. A Noise Unconditional Diffusion Model

The section introduces a diffusion model tailored to work without noise conditioning by modifying EDM’s schedules. The resulting uEDM uses a constant output coefficient, which the authors expect to free the network from modeling a time-dependent scale.

5. A Noise Unconditional Diffusion Model

  • The authors design a noise-unconditional diffusion model by modifying EDM schedules to seek greater robustness without noise conditioning while maintaining competitive performance.They name the model uEDM, short for (noise-)unconditional EDM.Source: §5 · p. 5
  • EDM’s core component is a preconditioned denoiser.The resulting uEDM schedules are provided in Section D.5.Source: §5 · p. 5
  • Setting the coefficient applied to the denoiser network to a constant is expected to free the network from modeling a time-dependent scale.The authors report that this design has a lower error bound than EDM in experiments (Statement 3).Source: §5 · p. 5

6. Experiments

The experiments compare denoising generative models with and without noise conditioning, primarily on class-unconditional CIFAR-10, with additional datasets, tasks, and sampling analyses. Most methods remain functional without conditioning; the analysis links their behavior to an error bound and tests alternatives across stochasticity and conditioning designs.

6.1. Main Observations

  • 2.23 FID on CIFAR-10 is achieved by uEDM without noise conditioning, narrowing the gap to noise-conditional EDM at 1.99 FID in this study.The paper also cites 1.97 FID for EDM reported in Karras et al. (2022).Source: Table 2 · p. 6, §6.1 · p. 6
  • Most variants degrade modestly without conditioning; some flow-based methods improve, consistency models remain functional, and DDIM with an ODE sampler fails at 40.90 FID.The paper summarizes noise conditioning as helpful for quality but not essential for fundamental functionality.Source: §6.1 · p. 6

6.2. Analysis

  • The error bound strongly correlates with observed robustness: DDIM’s catastrophic failure accompanies a bound orders of magnitude higher, while EDM, FM, and uEDM have small bounds.The bound uses model schedules and dataset statistics, so it can be computed without training and may help assess a model before training.Source: §6.2 · pp. 6–7
  • For noise-unconditional DDIM, increasing stochasticity consistently improves FID; the authors hypothesize that independent SDE noise injections promote error cancellation.At λ = 0 the sampler is the ODE case; the experiment fixes sampling at 500 steps.Source: §6.2 · p. 7, Figure 6 · p. 7
  • For iDDPM, EDM, and FM, predicted-noise, jointly trained predictor, and noise-unconditional variants behave similarly, consistent with uncertainty in estimating t.Even the predictor pretrained on true t cannot be perfect because p(t|z) retains uncertainty.Source: §6.2 · p. 8, Figure 7 · p. 8

6.3. Extra Datasets and Tasks.

  • Across ImageNet 32×32, FFHQ 64×64, and class-conditional CIFAR-10, removing conditioning is also effective; FM can improve, while EDM degrades without catastrophic failure.The evaluation also finds almost no degradation from removing conditioning at different classifier-free guidance scales on ImageNet 256×256 SiT.Source: §6.3 · p. 8, §6.4 · p. 8

7. Discussion and Conclusion

The discussion connects noise-unconditional models to single energy functions and flow fields that need not depend on noise level. It also reports almost no degradation under classifier-free guidance and encourages exploring models beyond noise conditioning.

7. Discussion and Conclusion

  • For SiT-B/2 on ImageNet 256×256, removing noise conditioning causes almost no degradation across guidance scales.The figure uses an Euler sampler with 250 steps.Source: Figure 8 · p. 9
  • Classical energy-based models use a single energy function dependent only on x, whereas t-conditioning implies annealed Langevin dynamics over a sequence of energy functions.The study suggests pursuing a single energy function aligns with the classical EBM goal.Source: §7 · p. 9
  • Flow Matching can be more robust to removing t-conditioning because its native formulation estimates a flow field that need not depend on t.The study suggests a single flow field can suffice for these methods.Source: §7 · p. 9
  • The authors encourage exploring generative models not constrained by the predominant noise-conditioning design.Source: §7 · p. 9

Appendices

The appendix describes numerical procedures for evaluating quantities related to the error from removing noise conditioning. On CIFAR-10, theoretical estimates generally align with observed values, with reduced precision for highly noisy inputs.

A. Details of Numerical Experiments

A.1. Computation of Relavent Quantities

  • A uniform grid of 100 t values is used to evaluate integrals, focusing approximation on intervals where p(t)p(z|t) is comparatively large.The two-step procedure is described as reducing computational cost while maintaining low numerical error.Source: §A.1 · p. 14

A.2. Numerical Experiments

  • Table 4 estimates closely match observed variance and effective-target error on CIFAR-10, except near t=1, where the single-data-point approximation is less precise.The estimates remain within the same order of magnitude; Fig. 3 visualizes concentration of p(t|z) on AFHQ-v2.Source: Table 4 · p. 15, §A.2 · p. 15

A.3. Evaluation of the Bound Values

The bound evaluation plots its terms for several models and identifies an EDM phase change around step 50. The appendix qualifies the bound’s strictness and magnitude, and notes that noise-unconditional experiments reuse conditional-model hyperparameters without further tuning.

A.3. Evaluation of the Bound Values

  • EDM’s B_i shows a phase change around i=50, attributed to a non-smooth δ value and hypothesized to occur when the data distribution ceases to resemble a point distribution.Figure 9 visualizes the bound terms A_i and B_i for DDIM, EDM, FM, and uEDM.Source: §A.3 · p. 15, Figure 9 · p. 16
  • The uEDM bound is not mathematically strict because its non-constant weighting conflicts with an assumption used in deriving the effective targets.The authors still treat it as reasonable intuition for the uEDM configuration.Source: §A.3 · p. 17
  • The bound can overestimate error because it assumes stepwise accumulation and because effective-target properties worsen near clean data; neural-network smoothing typically yields smaller real error.Ignoring the last 10 steps gives a more reasonable bound range in the reported estimates.Source: §A.3 · p. 17

B.1. General Experiment Configurations

  • Noise-unconditional models use the same hyperparameters tuned for conditional models, with no further tuning; the authors expect tuning may improve their performance.Source: §B.1 · p. 17

B.2. Special Experiments

Special experiments find that increasing sampler stochasticity reduces FID degradation for noise-unconditional models across two schedules, and test predictor-based and noise-like alternatives. The authors also report reproduction gaps while maintaining that paired comparisons remain meaningful.

B.2. Special Experiments

  • A noise-level predictor has at most 30K parameters, while a separately tested noise-like condition uses an unsupervised intermediate output that may not equal t.Source: §B.2 · pp. 18–19
  • As λ increases, FID degradation for noise-unconditional models becomes smaller with both cosine and linear schedules.Table 6 compares conditional and unconditional inference performance across λ values.Source: §B.2 · p. 18, Table 6 · p. 19
  • The lightweight predictor’s MSE is on the order of 10^-4, corresponding to a mean noise-level error of about 0.01.Source: §B.2 · p. 19

B.3. Our Reimplementation Faithfulness

  • Some reproductions fall short, including iCT and FFHQ-64 EDM; the authors say absolute performance is not the focus and paired comparisons remain meaningful.Source: §B.3 · p. 19
  • The 500-step iDDPM result is better than the 1000-step result reported in its original work, which the authors take as successful reproduction.Source: §B.3 · p. 19

C.1. Proof of the Effective Target

This section proves that regression losses with and without noise conditioning are equivalent, up to a θ-independent constant, to losses using effective targets, and analyzes effective-target error through posterior noise-level variance. It also bounds accumulated sampler error over N steps by comparing conditioned and unconditioned sampling from the same initial noise.

C.1. Proof of the Effective Target

  • With and without noise conditioning, the original regression loss is equivalent to its corresponding effective-target loss up to a θ-independent constant.The conditioned and unconditioned effective targets are given in Eqs. (5) and (7), respectively.Source: §C.1 · pp. 19–20

C.2. Approximation of the Variance of ppt|zq

  • For a fixed noisy image generated at noise level t*, the posterior variance of t scales as O(t*^2/d), and numerical results support this estimate.The section derives an upper bound under minimal technical assumptions and presents the scaling as a practical approximation.Source: §C.2 · p. 21

C.2.1. RIGOROUS UPPER BOUND ON VARIANCE.

  • With one data point, the posterior variance is O(t*^2/d) with high probability; with exponentially many candidates, the same bound extends when the correct clean image can be recovered.The multi-point argument reduces to the single-point analysis after recovering the clean sample.Source: §C.2.1 · pp. 23–24

C.2.2. INTUITIVE DERIVATION OF THE APPROXIMATE CONSTANT

  • An intuitive approximation recovers the variance scaling by reducing to one data point and neglecting the signal coordinate in the large-dimension likelihood.The derivation is restricted to the bulk regime, postpones edge-case analysis, and reports alignment with Statement 1.Source: §C.2.2 · pp. 24–25

C.3. Approximation of Epzq

  • For a single datapoint under the stated Flow Matching setup, Statement 1 provides an approximation to the effective-target mean squared error Epzq.The setup takes x in [-1,1]^d, Gaussian noise, a uniform noise level, and a noisy image generated at t*.Source: §C.3 · p. 25
  • In C.3, the effective-target error E_p(z) equals the squared distance between z and x multiplied by Var(t|z).The posterior variance is approximated by a first-order expansion when it is much smaller than the concentrated mean t*.Source: §C.3 · p. 26

C.4. Bound of Accumulated Error

  • For an N-step sampler initialized with the same noise, the output discrepancy between noise-conditioned and unconditioned sampling is bounded by a sum of per-step products A_iB_i.The proof assumes the same added noise at each sampling step in both processes.Source: §C.4 · p. 26

D. Derivation of Coefficients for Different Denoising Generative Models

The appendix expresses iDDPM, DDIM, EDM, Flow Matching, and uEDM in a unified notation, with model coefficients and derivations summarized in Tables 8 and 9.

D.1. iDDPM

  • iDDPM’s training and sampling formulations are translated into the unified notation, using a cosine diffusion schedule and coefficients summarized in Table 8.The training weighting is 1, and the training-time distribution is uniform over steps 1 through T.Source: §D.1 · p. 27, Table 8 · p. 28

D.2. DDIM

  • DDIM shares DDPM’s training process but uses a linear schedule; its generalized sampler interpolates between iDDPM at λ = 1 and DDIM at λ = 0.The linear schedule demonstrates the generality of the unified scheme, and the resulting DDIM coefficients are summarized in Table 8.Source: §D.2 · p. 29

D.3. EDM

  • EDM’s training objective is rewritten in unified notation, while sampling removes external conditioning through a change of variable and rescales the final output by σ_d.The training formulation absorbs the EDM preconditioning coefficients into the objective for the raw network.Source: §D.3 · pp. 30–31

D.4. Flow Matching

  • Flow Matching is expressed with unified coefficients and sampled by solving its ODE from t = 1 to t = 0 using a first-order Euler update.The update advances each state by the predicted velocity multiplied by the time-step difference.Source: §D.4 · p. 31

D.5. Our uEDM Model in the Formulation

  • uEDM modifies EDM’s c_in and c_out functions while keeping all other configurations unchanged, with the resulting coefficients compared in Table 9.The appendix derives these coefficients by substituting the modified functions into EDM’s general coefficient form.Source: §D.5 · p. 31, Table 9 · p. 32

E. Additional Samples

The appendix adds generated-sample examples across models, datasets, inference settings, and class-conditional generation. One class-conditional FM example is annotated with an FID change from 2.72 to 2.55.

E. Additional Samples

  • CIFAR-10 examples show ICM and ECM samples with one- and two-step inference, pairing conditioned and unconditioned runs using the same noise.Each figure orders samples as one step with t, one step without t, two steps with t, then two steps without t.Source: E. Additional Samples · p. 32, Figure 10 · p. 33, Figure 11 · p. 33
  • ImageNet 32×32 examples show FM samples from Euler and EDM-Heun samplers, pairing conditioned and unconditioned runs with the same noise.The figure orders the samples as Euler with t, Euler without t, Heun with t, then Heun without t.Source: E. Additional Samples · p. 32, Figure 12 · p. 33
  • Class-conditional FM is annotated with an FID change from 2.72 to 2.55.The corresponding samples compare FM and EDM with and without noise conditioning, using the same noise and label.Source: E. Additional Samples · p. 34, Figure 13 · p. 34

Related papers

Loading 2502.13129v2…