PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Noises Are Created Equally:Diffusion Noise Selection and Optimization

Zipeng Qi, Lichen Bai, Haoyi Xiong, Zeke Xie

arXiv:2407.14041v2cs.CV

TL;DR

The paper asks whether randomly sampled Gaussian noises are equally good, since prior work largely overlooked their effect on generation quality. It proposes selecting noises by inversion stability and optimizing given noises to enhance that stability. The authors report improved human-preference and objective evaluations for both methods, with DrawBench winning rates up to 57% and 72.5%, respectively.

  • Problem

    Prior work largely overlooked how sampled noise affects generation quality and whether noise could be selected or optimized to improve results.

    Source: §1 · p. 1

  • Method

    The paper selects noise by inversion stability and optimizes arbitrary given noise via an inversion-stability objective in noise space.

    Source: §1 · p. 2

  • Results

    Both methods improve representative diffusion models on human preference and objective metrics, with DrawBench human-preference winning rates up to 57% for noise selection and 72.5% for noise optimization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that higher inversion stability often accompanies better generation and use it to motivate both noise selection and noise optimization.

    Source: §6 · p. 9

    Noise selection repeats forward and inversion processing for 100 seeds, while noise optimization repeats forward, inversion, and gradient computation for 100 steps, making both costlier and slower than standard generation.

    Source: §5 · p. 9

Abstract

from arXiv · show

Diffusion models that can generate high-quality data from randomly sampled Gaussian noises have become the mainstream generative method in both academia and industry. Are randomly sampled Gaussian noises equally good for diffusion models? While a large body of works tried to understand and improve diffusion models, previous works overlooked the possibility to select or optimize the sampled noise the possibility of selecting or optimizing sampled noises for improving diffusion models. In this paper, we mainly made three contributions. First, we report that not all noises are created equally for diffusion models. We are the first to hypothesize and empirically observe that the generation quality of diffusion models significantly depend on the noise inversion stability. This naturally provides us a noise selection method according to the inversion stability. Second, we further propose a novel noise optimization method that actively enhances the inversion stability of arbitrary given noises. Our method is the first one that works on noise space to generally improve generated results without fine-tuning diffusion models. Third, our extensive experiments demonstrate that the proposed noise selection and noise optimization methods both significantly improve representative diffusion models, such as SDXL and SDXL-turbo, in terms of human preference and other objective evaluation metrics. For example, the human preference winning rates of noise selection and noise optimization over the baselines can be up to 57% and 72.5%, respectively, on DrawBench.

1 Introduction

The paper shifts attention from model weights and architecture to the sampled noise, asking whether noise can be selected or optimized to improve generation. It proposes both approaches based on inversion stability and reports gains in human preference and objective metrics.

1 Introduction

  • Earlier work focused on model weights and architecture, while the effect of noise on generation quality was largely overlooked despite outputs varying in quality.The paper frames noise selection and optimization as questions about improving generated results.Source: §1 · pp. 1–2
  • Higher inversion stability, measured by cosine similarity between sampled noise ϵ and inverse noise ϵ′, usually corresponds to better generation and motivates selecting stable seeds.The paper gives choosing the highest-stability seed among 100 noise seeds as an example.Source: §1 · p. 2
  • Noise optimization uses gradient descent on an inversion-stability loss with respect to sampled noise, enhancing the stability of arbitrary given noises rather than changing model weights.The authors present it as a noise-space method for improving diffusion-model results.Source: §1 · p. 2
  • Up to 57% and 72.5% DrawBench human-preference winning rates for noise selection and noise optimization, respectively, accompany reported gains on SDXL and SDXL-turbo.The methods are also preferred by HPS v2, with winning rates up to 67% and 88%, respectively.Source: §1 · p. 2, Figure 1 · p. 2

2 Prerequisites

This section introduces the diffusion sampling and noise-inversion processes used by the paper. It also defines fixed points of the denoising-inversion transformation as the ideal case in which a noise is recovered unchanged.

2 Prerequisites

  • Diffusion models generate a sample by denoising Gaussian noise along a reverse path from step T to 1, with the whole chain represented by pθ(x0:T).The model probability pθ describes a sampling transition, and the chain starts from p(xT)=N(xT|0,I).Source: §2 · pp. 2–3
  • DDIM inversion maps a clean sample back to noise along a predefined path, whereas denoising maps sampled noise toward a generated sample.The two paths are not identical: inversion uses predicted noise from the score network, and cumulative numerical differences leave sampled noise ϵ close to, but not identical to, inverse noise ϵ′.Source: §2 · p. 3
  • A fixed point is a noise ϵ satisfying ϵ=F(ϵ) for the denoising-inversion map F, so inversion perfectly recovers the sample generated from that noise.The paper introduces this ideal property as background for its stability discussion.Source: §2 · p. 3

3 Methodology

The methodology uses noise inversion stability to guide two noise-space procedures: selecting a stable sampled noise or optimizing a given noise toward greater stability.

3 Methodology

  • Because exact fixed points of the denoising–inversion transformation are intractable, the method measures noise inversion stability using cosine similarity between a sampled noise and its inverse.Cosine similarity is a simple choice; the authors note that other similarity metrics are possible.Source: §3 · p. 4
  • Among K sampled seeds, noise selection generates and inverts each candidate, then chooses the noise with the highest stability score.The score is based on cosine similarity between the sampled noise and its inverse.Source: §3 · pp. 3–4
  • Noise optimization applies gradient descent to 1 − cos(ϵ, ϵ′) with respect to the sampled noise, holding model weights and ϵ′ constant at each step.The algorithm returns the updated noise after the specified gradient-descent steps.Source: §3 · pp. 3–4

4 Empirical Analysis

The experiments evaluate noise selection and noise optimization for text-to-image generation, comparing stable with unstable noises and optimized with original noises. Across quantitative and qualitative evaluations, both approaches improve results, with optimization also tested across denoising steps and on 3D generation.

4.1 Experimental Settings

  • Experiments use SDXL-turbo with 4 denoising steps and SDXL with 10 steps by default, evaluating generated images with human preference and objective metrics.The evaluation includes HPS v2, AES, PickScore, and ImageReward; the main datasets are DrawBench and Pick-a-Pic.Source: §4.1 · pp. 4–5, §4 · p. 4, §4.2 · p. 5

4.2 The Experiments of Noise Selection

  • HPS v2 scores for stable noises exceed unstable-noise scores by 0.225 on Pick-a-Pic and 0.289 on DrawBench; human wins exceed 56%, and HPS v2 wins reach 65% and 67%.The reported winning rates compare images generated from stable and unstable noises using SDXL-turbo.Source: §4.2 · pp. 5–6, Figure 3 · p. 5, Table 1 · p. 5
  • Stable noises improve color, style, text rendering, object co-occurrence, position, and counting compared with unstable noises.Examples include rendering “diffusion” correctly and correcting object-count errors.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3 The Experiments of Noise Optimization

  • 69% and 72.5% human-preference winning rates and 87% and 88% HPS v2 winning rates favor optimized noises over original noises on Pick-a-Pic and DrawBench, respectively.The study evaluates noise optimization with SDXL.Source: Table 2 · p. 6, §4.3 · p. 7, Figure 5 · p. 7
  • Optimized noises improve color, style, text rendering, object co-occurrence, position, and counting, and can improve human motion and appearance details.One example is a huntress accurately holding the end of an arrow.Source: §4.3 · p. 7, Figure 6 · p. 7, Figure 7 · p. 8
  • Optimization improvements are relatively robust across denoising-step counts and are especially strong with very few steps.The study examines SDXL with 5, 10, 30, and 50 denoising steps.Source: §4.3 · pp. 7–8, Figure 8 · p. 8
  • Noise optimization also improves body details in an example using the SV3D image-to-3D generative model.The authors leave further experimental details and results to Appendix C.Source: §4.3 · p. 8

5 Discussion and Limitations

The discussion contrasts general improvement through noise selection or optimization with earlier noise-editing work focused on layout control. It reports open theoretical and optimization questions, alongside higher computational costs.

Related Work

  • Unlike earlier noise-editing work aimed at layout control, this work targets generally improving diffusion-model results by selecting or optimizing Gaussian noise using stability scores.Figure 9 shows qualitative noise-optimization results for SV3D, with input images shown small and differences highlighted in red.Source: §5 · pp. 8–9, Figure 9 · p. 9

Theoretical Understanding

  • The authors report that they do not yet theoretically understand why inversion stability relates to generation quality or how model performance mathematically depends on stability.They describe theoretical understanding of noise selection and optimization as an important next step.Source: §5 · p. 9

Optimization Strategies

  • The noise-space loss can fail to converge smoothly, and the authors did not carefully tune hyperparameters or use advanced optimizers because of computational costs and limited understanding of its landscape.They characterize the current optimization strategy as far from fully realizing the potential of noise-space algorithms.Source: §5 · p. 9

Computational Costs

  • Both noise selection and optimization cost more time and computation than standard generation, requiring 100 inversion evaluations or 100 gradient-descent steps, respectively.The authors note that optimization may be accelerated with fewer gradient-descent steps.Source: §5 · p. 9

6 Conclusion

The paper reports that higher inversion stability often accompanies better generation and presents noise selection and optimization as ways to use this relationship. Experiments show improvements across multiple result dimensions and diffusion models.

6 Conclusion

  • The proposed noise-selection and noise-optimization methods improve qualitative results, human-preference rates, and objective scores across multiple diffusion models, without fine-tuning model parameters.The authors describe the methods as generally applicable to different diffusion models in a plug-and-play manner.Source: §6 · p. 9
  • Higher inversion stability often accompanies better generated results, motivating noise selection and noise optimization in noise space.The authors characterize their work as an initial step toward understanding and improving diffusion models from the noise-space perspective.Source: §6 · p. 9

A Experimental Settings of Main Experiments

A.1 Datasets and Data Preprocessing

The appendix describes three prompt datasets with different characteristics and records the preprocessing and hyperparameter settings used for the experiments.

A.1 Datasets and Data Preprocessing

  • Pick-a-Pic prompts come from real users, DrawBench prompts are complex and include rare words, and GenEval prompts describe attributes using simple prompts.The main experiment used only the first 100 Pick-a-Pic prompts for faster validation and lower computational cost; DrawBench has about 200 examples and GenEval has 553 prompts across six attributes.Source: §A.1 · p. 13
  • The main experiments used half-precision tensors and flattened noise tensors along the channel dimension to calculate inversion stability.Source: §A.1 · p. 13

A.2 The Hyperparameters:

  • For selection, each prompt used 100 random-seed noises and the highest-scoring stable noise was chosen; optimization used 100 gradient steps with learning rate 100, cosine annealing, and momentum 0.5.Source: §A.2 · p. 13

A.3 Evaluation Metrics

The evaluation appendix describes three learned preference metrics, an aesthetic score, and human pairwise judgments based on several image criteria.

A.3 Evaluation Metrics

  • HPS v2 is calculated by a CLIP model fine-tuned on the diverse HPD v2 human-preference dataset.Source: §A.3 · p. 13
  • AES uses an aesthetic-score predictor built by adding five MLP layers to a frozen CLIP, with scores from 0 to 10 and higher scores indicating better visual quality.Source: §A.3 · p. 13
  • PickScore is calculated by a CLIP model fine-tuned on user-labeled preferences from Pick-a-Pic, while ImageReward is described as an early human-preference model (Xu et al., 2024).Source: §A.3 · p. 14
  • Human evaluators choose the better image pair based on semantic alignment, object appearance and structure, detail richness, aesthetic appeal, and social-sharing preference.The evaluation used a web page shown in Figure 10.Source: §A.3 · p. 14, Figure 10 · p. 14
  • The paper distinguishes AES as primarily assessing visual quality, while HPS v2, PickScore, and ImageReward assess human preference.Source: §A.3 · p. 14

B Supplementary Experimental Results

The supplementary section shows additional optimized-noise generations, then extends the analysis to image-to-3D diffusion. For SV3D, it derives a noise inversion rule distinct from text-to-image inversion.

B Supplementary Experimental Results

C 3D Object Generation

C.1 Methodology

  • For SV3D, the paper derives an image-to-3D noise inversion rule because its inversion differs from text-to-image diffusion.SV3D uses the EDM framework; the derivation describes inversion from ˆx_t to ˆx_t+1 and approximates the prediction at x_t+1 with the prediction at x_t.Source: §C.1 · p. 16

C.2.1 Datasets

The 3D evaluation uses rendered OmniObject3D objects and follows SV3D’s inference configuration. Noise optimization is applied with a specified gradient-descent schedule.

C.2.1 Datasets

  • Thirty OmniObject3D objects were rendered as 84-frame rotating video sequences under white-background ambient lighting.Because SV3D does not disclose its test-set rendering details, pixel-level similarity was challenging to achieve.Source: §C.2.1 · p. 16

C.2.2 The Hyperparameters

  • Inference used 50 steps, cfg 2.5, and Euler sampling; noise optimization used 20 gradient-descent steps at learning rate 1500 and momentum 0.5.Source: §C.2.2 · p. 16

C.3 Performance Evaluation

The authors report that optimized noises improve image-to-3D generation quality, with qualitative results showing smoother, more faithful objects. Evaluation uses perceptual and similarity metrics, while the paper says no societal consequences require specific discussion.

C.3 Performance Evaluation

  • Quality is measured with LPIPS, SSIM, and CLIP-S; PSNR is not applicable because multi-view ground truth is unavailable.Source: §C.3 · p. 16
  • Optimized noises lead to higher image-to-3D generation quality on OmniObject3D static orbits.Table 3 presents the quantitative results; the supplied text does not state the individual metric values.Source: §C.3 · p. 16, Table 3 · p. 17
  • Compared with original-noise results, optimized-noise objects generally have fewer jagged edges, smoother surfaces, and better fidelity.Source: §C.3 · p. 17

Broader Impact

  • The authors state that no potential societal consequences require specific discussion in this paper.Source: Broader Impact · p. 17

Related papers

Loading 2407.14041v2…