PaperlayerView exact revision on arXiv

Source-linked AI summary

Noise Estimation for Generative Diffusion Models

Robin San-Roman, Eliya Nachmani, Lior Wolf

arXiv:2104.02600v2cs.LGcs.CV

TL;DR

Few-step diffusion generation normally requires costly noise-parameter tuning for each step count. The paper learns to estimate noise during inference and dynamically adjust the schedule without changing diffusion-model weights, improving image and speech synthesis under restricted steps.

  • Problem

    High-quality diffusion synthesis with few denoising steps requires costly noise-schedule tuning, often separately for each step count.

    Source: Introduction · p. 1, Related Work · p. 2

  • Method

    A neural network estimates the current noise level from the generated data and dynamically schedules subsequent denoising steps without modifying the diffusion model.

    Source: Introduction · p. 1, Method · p. 3

  • Results

    The method improves restricted-step generation across vision and audio, including better image FID and improved PESQ and STOI for speech.

    Source: Introduction · p. 1

  • Takeaways & Limitations

    Noise estimation provides a generic, on-the-fly alternative to fixed or per-step-count noise-parameter selection for few-step synthesis.

    Source: Conclusions · p. 8

    It remains to be determined whether the same Pθ network works across multiple datasets.

    Source: Conclusions · p. 9

Abstract

from arXiv · show

Generative diffusion models have emerged as leading models in speech and image generation. However, in order to perform well with a small number of denoising steps, a costly tuning of the set of noise parameters is needed. In this work, we present a simple and versatile learning scheme that can step-by-step adjust those noise parameters, for any given number of steps, while the previous work needs to retune for each number separately. Furthermore, without modifying the weights of the diffusion model, we are able to significantly improve the synthesis results, for a small number of steps. Our approach comes at a negligible computation cost.

Introduction

Diffusion models can generate high-quality images and speech, but few-step synthesis traditionally requires costly noise-schedule tuning. The paper proposes conditioning-based noise estimation that dynamically controls denoising and improves restricted-step generation across vision and audio.

Introduction

  • Few-step diffusion synthesis requires costly grid search for a noise schedule that preserves high-fidelity outputs.Source: Introduction · p. 1
  • The proposed method estimates noise schedules from conditioning data rather than fixing denoising steps in advance.A neural network monitors the denoising process and schedules subsequent steps using estimated data noise.Source: Introduction · p. 1
  • The method produces high-fidelity samples with few denoising steps and outperforms prior models using costly per-sample grid search at the same step count.Source: Introduction · p. 1
  • Across vision and audio, the method improves image FID under step restrictions and speech metrics including PESQ and STOI.Source: Introduction · p. 1

Related Work

Prior diffusion and score-based models established strong image, speech, time-series, and other generative results, but high-quality diffusion sampling commonly requires many steps. DDIM accelerates denoising through a non-Markovian process, while later work broadens diffusion applications.

Related Work

  • Diffusion models originally required up to thousands of iterative steps during training and often use about a thousand steps for high-quality generation.Source: Related Work · pp. 1–2
  • Score matching with Langevin dynamics introduced another generative modeling route and achieved state-of-the-art CIFAR-10 results (Song and Ermon 2019).Source: Related Work · p. 1
  • DDPM combined neural diffusion and score matching for image generation, while related work applied diffusion-based score matching to speech and achieved state-of-the-art comparisons with established methods.Source: Related Work · pp. 1–2
  • DDIM accelerates denoising by using a non-Markovian diffusion process, producing faster sampling than other diffusion models.Source: Related Work · p. 2
  • Later diffusion research modeled stochastic differential equations, energy-based generation, probabilistic forecasting, and categorical diffusion across several tasks.Source: Related Work · p. 2

Background

DDPMs learn to reverse a noise-adding Markov process, training a network to estimate injected noise and sampling by iterative denoising from Gaussian noise. The noise schedule controls corruption strength, while inference applies learned updates and can optionally be deterministic.

Background

  • DDPMs learn data-density gradients or injected noise and use iterative dynamics to reverse diffusion from Gaussian noise.Source: Background · pp. 2–3
  • The forward process is a Markov chain that gradually adds Gaussian noise according to schedule β1, . . . βN across latent variables yN through y0.Source: Background · p. 2
  • Training samples a diffusion state, noise level, and Gaussian noise, constructs a noisy latent, and updates εθ to estimate the added noise.Source: Background · pp. 2–3
  • The noise level √¯α determines corruption strength: values near 1 add little noise, while values near 0 add large amounts.Source: Background · p. 2
  • Uniformly sampling √¯α over [0,1] performs poorly because εθ rarely trains on fine-grained, low-noise examples.Source: Background · p. 2
  • Sampling starts from Gaussian noise and repeatedly applies the learned reverse update, with optional Gaussian perturbation at nonfinal steps.Source: Background · p. 3
  • Setting σ̃ = 0 removes added random noise, making the sampling process deterministic and often improving results.Source: Background · p. 3

Method

The method estimates the current noise level during denoising and uses it to adapt subsequent noise parameters, supporting different schedules and DDPM conditioning schemes. It combines a learned estimator with closed-form schedule adjustments and applies these adjustments throughout inference.

Method

  • Estimating noise level with Pθ provides the diffusion network with conditioning tied to the actual generation state, especially when few-step intermediate quality varies widely.At inference, the noise level is unknown, and predefined conditioning is not fully exploited.Source: Method · p. 3

Noise Level Estimation

  • Pθ takes the step-n data yn as input and outputs an estimated noise level ˆα for controlling the generation process.The estimator is trained by sampling noise levels and is particularly optimized for accuracy when noise is low, because final steps determine sample quality.Source: Noise Level Estimation · pp. 3–4, Method · p. 3

Noise Schedule Adjustments

  • Given ˆα and the number of remaining denoising steps, updateNoiseSchedule computes the noise parameters α and β for subsequent sampling.The method assumes the schedule follows either a linear or Fibonacci distribution and needs β0 and the distribution type to determine the remaining parameters.Source: Noise Schedule Adjustments · p. 4

Linear Schedule

  • For a linear schedule, recovering the step size x allows Eq. 15 to generate the β_i values needed for the remaining denoising steps.The derivation uses the approximation log(1 − β_i) ≈ −β_i for β_i typically between 10^-6 and 10^-2.Source: Linear Schedule · p. 4

Fibonacci Schedule

  • For a Fibonacci schedule, a closed-form solution computes the noise schedule (β, α) from the estimated noise level ̄α and β0.The solution is obtained by solving the associated recurrence and linear system.Source: Fibonacci Schedule · p. 4

Conditioning on Discrete Indexes

  • For DDPMs conditioned on a discrete step index, the method estimates the noise level and feeds the model the integer encoding of the interval containing that estimate.This adapts the method from exact noise-level conditioning to the discrete conditioning used by many DDPMs.Source: Conditioning on Discrete Indexes · p. 4

Inference procedure

  • During inference, the schedule is initialized with predefined values and readjusted at selected iteration indexes using Pθ and the deterministic update rule.In the experiments, the adjustment set is U = {1, 2, . . . , N}, so the schedule is updated at every step.Source: Figure 1 · p. 4, Inference procedure · p. 5

Experiments

Experiments evaluate adaptive noise scheduling for speech and image diffusion synthesis with few denoising iterations. Across both domains, the method improves quality against tuned diffusion baselines while retaining practical generation costs.

Experiments

  • The experiments determine few-step noise schedules for state-of-the-art speech and image models, evaluating the approach across both domains.Speech uses six iterations with Fibonacci schedule adjustments; image experiments use CelebA and LSUN datasets with FID evaluation.Source: Experiments · p. 5, Speech Generation · p. 5, Image Generation · p. 5

Speech Generation

  • Large PESQ and STOI gains are obtained over a grid-searched baseline for six-iteration speech generation, despite a small MCD decrease.The speech model uses WaveGrad with a learned noise estimator based on ConvTASNet components.Source: Speech Generation · p. 5

Image Generation

  • For CelebA, the method improves DDIM FID by 163.4 at three iterations with η = 0.0, and the gap remains through 10 iterations.Figure 4 compares adaptive scheduling against DDIM across different η values using the same diffusion model.Source: Image Generation · p. 5, Figure 4 · p. 6
  • On CelebA, the method improves DDPM's FID by 266.03 at 10 iterations and yields a large-margin improvement through 100 iterations.The noise-estimation model uses a VGG11 backbone pretrained on ImageNet.Source: Image Generation · pp. 5–6
  • On LSUN Church and Bedroom, the method improves DDIM results below 10 iterations and exceeds one-iteration distillation scores with six iterations.The comparison covers 256x256 Church and Bedroom generation; qualitative samples also show sharper, more coherent outputs.Source: Image Generation · pp. 6, 8, Figure 3 · p. 6, Figure 7 · p. 7

Speech Generation

  • The noise estimator predicts the noise level within 10^-4 error, with better accuracy near alpha = 1, where precision is critical.This experiment evaluates estimates across alpha values using noisy validation speech signals.Source: Image Generation · p. 8

Conclusions

The method adjusts noise parameters on-the-fly during denoising based on the current noise level, altering subsequent process states. The paper notes that reuse of the same Pθ network across multiple datasets remains future work.

  • On-the-fly noise adjustment alters subsequent denoising states based on the estimated current noise level and is independent of conditioning parameters.Source: Conclusions · p. 8
  • Figure 9 compares Pθ's speech noise-estimation performance with the L1 distance between ground-truth and generated mel spectrograms across perturbed-index values.Source: Figure 9 · p. 9
  • The paper leaves open whether the same Pθ network can be used across multiple datasets.Source: Conclusions · p. 9
Loading 2104.02600v2…