Source-linked AI summary
On the Importance of Noise Scheduling for Diffusion Models
Ting Chen
TL;DR
The paper studies how noise scheduling affects diffusion-model training across tasks and image resolutions. It evaluates schedule functions and input scaling, finding that scaling the input while keeping the schedule fixed works across resolutions and, with RIN, supports state-of-the-art pixel-based high-resolution generation. The approach is evaluated on pixel-based diffusion models, while latent diffusion and classifier guidance are outside the quantitative evaluation.
Problem
The paper asks how noise scheduling affects diffusion-model performance and whether one effective scheduling strategy can work across tasks and image resolutions.
Method
The paper systematically compares parameterized schedule functions and combines a fixed schedule function with input scaling by a factor of b.
Results
Different resolutions require different schedule functions, while input scaling improves performance; at 256×256, FID drops from 4.28 to 3.52, and combining the strategy with RIN enables state-of-the-art high-resolution pixel-based generation.
Takeaways & Limitations
Practitioners should select and tune a suitable noise-scheduling scheme when training diffusion models for a new task or dataset.
Takeaways & Limitations
The scheduling technique is tested only on pixel-based diffusion models, and classifier guidance is not used for quantitative evaluation.
Abstract
from arXiv · showhide
We empirically study the effect of noise scheduling strategies for denoising diffusion generative models. There are three findings: (1) the noise scheduling is crucial for the performance, and the optimal one depends on the task (e.g., image sizes), (2) when increasing the image size, the optimal noise scheduling shifts towards a noisier one (due to increased redundancy in pixels), and (3) simply scaling the input data by a factor of $b$ while keeping the noise schedule function fixed (equivalent to shifting the logSNR by $\log b$) is a good strategy across image sizes. This simple recipe, when combined with recently proposed Recurrent Interface Network (RIN), yields state-of-the-art pixel-based diffusion models for high-resolution images on ImageNet, enabling single-stage, end-to-end generation of diverse and high-fidelity images at 1024$\times$1024 resolution (without upsampling/cascades).
1 Why is noise scheduling important for diffusion models?
Noise scheduling determines which noise levels diffusion models learn to denoise, and its optimal form can change with image resolution. Higher-resolution images contain more pixel redundancy, making denoising easier at the same noise level and potentially leaving some levels undertrained without schedule adjustment.
- How diffusion noise scheduling works: Diffusion models sample a time t and noise an input x0 into xt before training a denoising network to predict noise or clean data.The schedule γ(t) controls the distribution of noise levels presented during training.
- Why scheduling matters across resolutions: As image size increases, denoising at the same noise level becomes simpler because natural images typically contain more redundancy among nearby pixels.Independent per-pixel noise leaves more recoverable signal when redundancy is higher.
- Why scheduling matters across resolutions: The optimal schedule at a smaller resolution may not remain optimal at a higher resolution.Without adjustment, some noise levels may be undertrained.
2 Strategies to adjust noise scheduling
The paper studies two ways to adjust diffusion noise scheduling: directly parameterizing γ(t), or scaling inputs while keeping a schedule fixed, then combines both strategies.
- Strategy overview: The study systematically compares parameterized noise schedules with input scaling as two ways to adjust diffusion training.The schedule determines the noise-level distribution seen by the denoising network.
- 2.1 Strategy 1: changing noise schedule functions: Cosine, sigmoid, and simple linear functions provide alternative continuous noise schedules, with cosine and sigmoid hyper-parameters controlling their shapes.The simple linear schedule is γ(t) = 1−t; the paper notes that selected hyper-parameters skew noise toward noisier levels.
- 2.2 Strategy 2: adjusting input scaling factor: Scaling x0 by b indirectly adjusts noise levels: reducing b increases effective noise under the same γ.Because scaling can change the variance of xt, the method normalizes xt to unit variance before denoising.
- 2.2 Strategy 2: adjusting input scaling factor: Input scaling shifts logSNR vertically without changing its shape, unlike the cosine and sigmoid schedules considered.This offers a distinct way to alter the noise distribution, especially when t is close to 0.
- 2.3 Putting it together: a simple compound noise scheduling strategy: The proposed compound strategy uses a fixed schedule such as γ(t) = 1−t together with an input scaling factor b during training and inference.Variance normalization is applied consistently when enabled, and inference can use a separately chosen continuous-time schedule.
3 Experiments
Experiments evaluate noise schedules and input scaling across ImageNet resolutions, finding resolution-dependent schedule preferences and improved high-resolution pixel generation when combined with RIN.
- 3.1 Setup: Experiments use class-conditional ImageNet generation evaluated with FID and Inception Score on 50K samples generated by 1000-step DDPM.
- 3.2 The effect of strategy 1 (noise schedule functions): Different image resolutions require different noise schedule functions, making the optimal schedule difficult to find because several hyperparameters are involved.
- 3.3 The effect of strategy 2 (input scaling): As resolution increases, the optimal input scaling factor becomes smaller, and tuning input scaling improves 256×256 FID from 4.28 to 3.52.
- 3.3 The effect of strategy 2 (input scaling): Input-scaling adjustments are easier to find than optimal schedule functions because they tune a single scaling factor while keeping the schedule fixed.
- 3.4 The simple compound strategy, combined with RIN, enables state-of-the-art single-stage high-resolution image generation based on pixels: The compound scheduling strategy with RIN enables state-of-the-art high-resolution image generation directly from pixels without latent diffusion.
- 3.5 Visualization of generated samples: Random visualization samples preserve global structure across resolutions, although smaller-scale object parts may be imperfect; high-resolution hyperparameters were not thoroughly tuned.
4 Conclusion
The paper concludes that noise scheduling is important across diffusion tasks and that input scaling is effective across image resolutions. Combined with RIN, this strategy supports single-stage high-resolution image generation.
- Noise scheduling matters for image generation and other tasks such as panoptic segmentation.
- Scaling the input by a constant factor is a simple strategy that works across different image resolutions.
- Combining the scheduling strategy with RIN enables single-stage generation of high-resolution images.