Source-linked AI summary
A Noise is Worth Diffusion Guidance
Donghoon Ahn, Jiwon Kang, Sanghyun Lee, Jaewon Min, Minjae Kim, Wooseok Jang, Hyoungwon Cho, Sayak Paul, SeonHwa Kim, Eunju Cha, Kyong Hwan Jin, Seungryong Kim
TL;DR
Diffusion image quality commonly relies on guidance, which increases inference cost, raising the question of whether guidance is necessary. NoiseRefine learns to refine initial Gaussian noise into guidance-free noise. It reports comparable quality to CFG and PAG while sampling about twice as fast as CFG alone, or three times as fast as CFG and PAG.
Problem
High-quality text-to-image diffusion sampling largely relies on guidance methods such as CFG, which double inference cost despite drawbacks including oversaturation and reduced diversity.
Source: §1 · p. 1Method
NoiseRefine learns a mapping from standard Gaussian noise to guidance-free noise and trains by comparing denoised images to mitigate inversion error.
Source: §1 · p. 2Results
Images from refined noise without guidance have quality comparable to CFG and PAG, at about twice the speed of CFG alone or three times the speed of CFG and PAG.
Source: §1 · p. 2Takeaways & Limitations
NoiseRefine is a plug-and-play module for the original diffusion model that replaces guidance with a single noise-refinement forward pass.
Source: §6 · p. 9The MSD gradient approximation is supported under a mild assumption.
Source: §3.2 · p. 5
Abstract
from arXiv · showhide
Diffusion models excel in generating high-quality images. However, current diffusion models struggle to produce reliable images without guidance methods, such as classifier-free guidance (CFG). Are guidance methods truly necessary? Observing that noise obtained via diffusion inversion can reconstruct high-quality images without guidance, we focus on the initial noise of the denoising pipeline. By mapping Gaussian noise to `guidance-free noise', we uncover that small low-magnitude low-frequency components significantly enhance the denoising process, removing the need for guidance and thus improving both inference throughput and memory. Expanding on this, we propose \ours, a novel method that replaces guidance methods with a single refinement of the initial noise. This refined noise enables high-quality image generation without guidance, within the same diffusion pipeline. Our noise-refining model leverages efficient noise-space learning, achieving rapid convergence and strong performance with just 50K text-image pairs. We validate its effectiveness across diverse metrics and analyze how refined noise can eliminate the need for guidance. See our project page: https://cvlab-kaist.github.io/NoiseRefine/.
1. Introduction
Diffusion image quality commonly depends on guidance, motivating a way to replace it by refining the initial noise. NoiseRefine learns to map Gaussian noise to guidance-free noise, using small low-frequency components and efficient training to produce comparable-quality images faster.
1. Introduction
- Guidance methods improve image quality but double inference cost, and CFG remains standard despite drawbacks including oversaturation and reduced diversity.The paper asks whether guidance effects can be replaced with minimal changes to the diffusion pipeline.Source: §1 · p. 1
- Diffusion inversion supplies initial noises that reconstruct high-quality images without guidance, providing examples for constructing the guidance-free noise space.The approach generates high-quality images with guidance and applies inversion to obtain the corresponding initial noises.Source: §1 · p. 2
- The difference between standard and guidance-free noise is concentrated in subtle, low-magnitude, low-frequency components.The analysis reports that paired-noise differences are smaller than differences between random noise instances and mainly correspond to low frequencies.Source: §1 · pp. 2–3
- NoiseRefine shifts its distance loss from noise space to denoised-image space to mitigate inversion error, while MSD avoids costly full denoising-step backpropagation.The paper reports that skipping gradient computation during denoising reduces overhead and accelerates convergence.Source: §1 · p. 2
- Comparable-quality images are generated about twice as fast as with CFG alone, or three times as fast as with CFG and PAG.The approach uses refined noise without guidance during sampling.Source: §1 · p. 2
- NoiseRefine identifies a guidance-free noise space and learns to map standard Gaussian noise into it with a neural network.Its contributions include Multistep Score Distillation (MSD), which detaches gradients during denoising to accelerate convergence.Source: §1 · p. 2
2. Related Work
Related work addresses guidance, inversion, and noise optimization. The paper positions NoiseRefine as learning a direct, prompt-generalizable mapping to guidance-free noise rather than relying on costly iterative optimization or a fixed noise database.
2. Related Work
- Guidance methods improve fidelity but double computational and memory costs because they generate degraded samples at each denoising step.Classifier guidance uses trained classifier gradients, while CFG models an implicit classifier.Source: §2 · p. 3
- DDIM inversion maps images to noise for deterministic reconstruction, while linear approximation can cause artifacts and fixed-point iterations reduce approximation error.Guided inversion requires guidance during sampling to reconstruct the same image, and vice versa.Source: §2 · p. 3
- Existing noise optimization can require many iterations, and a noise database does not generalize to unseen prompts; NoiseRefine instead learns a direct mapping in one inference step.The learned mapping is described as providing generalization to new prompts.Source: §2 · p. 3
3. Method
NoiseRefine learns to map Gaussian initial noise to guidance-free noise, using guided generations and inversion to identify the target space. It trains the mapping through denoised-image distances and multistep score distillation, with dataset filtering to support high-quality unguided generation.
3.1. Guidance-Free Noise Space
- Small differences between Gaussian and inversion noise are concentrated in low-frequency components, suggesting these components can guide high-quality generation without sampling-time guidance.The guidance-free noise space is constructed from inversion noises that reconstruct high-quality guided images without guidance.Source: §3.1 · pp. 3–4
3.2. Learning to Map Guidance-Free Noise Space
- NoiseRefine learns the mapping by minimizing the distance between a guided image and an image denoised from refined noise, avoiding direct learning of potentially erroneous inversion noise.The method uses the same diffusion pipeline to generate the guided target and the unguided prediction.Source: §3.2 · p. 4
- The method uses a lightweight LoRA and residual connection in the noise-refining model to leverage pretrained text-image knowledge and support rapid convergence.The authors motivate the residual connection by the small difference between initial and inversion noise.Source: §3.2 · p. 5
- Multistep score distillation detaches denoising-network gradients during training; the paper reports faster convergence, sharper images, and lower computational costs than full-gradient optimization.The authors also describe MSD as a close approximation to the full-gradient objective.Source: §3.2 · p. 5
3.3. Dataset Construction
- Refined-noise samples generated without guidance show comparable quality to guided samples and consistently better quality than unguided samples from Gaussian noise.The qualitative comparison uses Gaussian noise without guidance, Gaussian noise with sampling guidance, and refined noise without guidance.Source: Figure 8 · p. 6
- The training set combines CFG and PAG generations with varied scales and filters low-quality images using a human preference model.The paper reports that filtering produces superior results compared with training without these dataset enhancements.Source: §3.3 · p. 6
4. Experiment
Experiments on Stable Diffusion 2.1 evaluate refined-noise generation on four text-to-image benchmarks. Refined noise improves over unguided Gaussian noise and reaches quality comparable to guided sampling; the ablations examine denoising steps and training-data filtering.
4.1. Results
- Across reported metrics, refined-noise images consistently improve over unguided Gaussian-noise images and are comparable to guided images; FID is lower than the guidance setting.The study reports human preference, prompt adherence, FID, and IS evaluations.Source: §4.1 · p. 7, Table 1 · p. 7, Table 2 · p. 7
- 45 participants compared 30 image pairs and confirmed comparable visual appeal and prompt alignment for refined-noise and guided images.The pairs were evaluated for visual appeal and prompt alignment.Source: §4.1 · p. 7, Table 4 · p. 7
4.2. Ablation Study
- Increasing denoising steps from N = 5 to N = 10 improves performance, while N ≥10 makes backpropagation costs prohibitive and motivates MSD.The ablation reports FID and IS; the table covers denoising-step count and dataset filtering.Source: Table 3 · p. 7, §4.2 · p. 8
- Filtering out images with aesthetic scores below 6.0 leaves about 25% of generated images and significantly improves generation metrics.The filtering test starts from images generated using 80K MS-COCO prompts.Source: Table 3 · p. 7, §4.2 · p. 8
5. Analysis and Discussion
NoiseRefine adds mostly small, low-frequency components that can establish an image layout early in denoising. The analysis also reports diversity and generalization, and argues that learning in noise space is an efficient alternative to tuning the full pipeline.
5.1. What Does noise refining model Learn?
- Refined noise adds mostly small, low-frequency components, with a distribution resembling the difference between Gaussian and inversion noise.The model learns this pattern without explicit constraints.Source: §5.1 · p. 8
- Condition-dependent low-frequency signals act as initial image layouts, helping form object shapes early and enabling denoising to add details within that layout.Gaussian starts struggle to form coherent layouts, sometimes leaving details misplaced or ambiguous.Source: §5.1 · p. 8, Figure 9 · p. 8, Figure 10 · p. 8
- Generated layouts vary with initial noise, and the analysis reports greater IS diversity than guidance-based methods plus good performance on unseen noise and prompts.Table 1 is cited as evidence of generalization beyond the training data.Source: §5.1 · p. 8
5.2. Why Learn Noise Mapping?
- Learning noise-space mappings is presented as an efficient alternative to tuning the denoising pipeline, using 50K noise and self-generated image pairs.Tuning the diffusion model converged more slowly and had frequent loss explosions; prompt learning is introduced as a related idea of tuning inputs instead of model parameters.Source: §5.2 · pp. 8–9
6. Conclusion
NoiseRefine replaces guidance during diffusion sampling with a single neural-network refinement of the initial noise. The appendix passages also outline DDPM and DDIM processes used in the diffusion pipeline.
6. Conclusion
- NoiseRefine replaces guidance in diffusion sampling with a noise refinement performed by a single neural-network forward pass.The conclusion describes it as a plug-and-play module based on the original diffusion model.Source: §6 · p. 9
A.1. Preliminaries
- DDPM defines a forward Markov chain that adds Gaussian noise and a reverse Markov chain that samples the preceding state.The forward transition follows a variance schedule.Source: §A.1 · p. 12
- DDIM uses a non-Markovian forward process in which each state may depend on the previous state and the original image, and reparameterizes sampling with the noise predictor.The work uses DDIM but states that other schedulers can be used with corresponding parameter changes.Source: §A.1 · pp. 12–13
A.2. Derivations
The derivation states a Lipschitz-continuity assumption for denoising and presents Multistep Score Distillation as an approximation to the full-gradient objective. The supplied passages provide only partial details of the derivations.
A.2. Derivations
- Proposition 1 assumes Lipschitz continuity under distance metric d and asserts a bound for each initial noise xT and its denoised image x0.The constant κ is positive; the displayed bound itself is not included in the supplied passage.Source: §A.2 · p. 13
- MSD approximates the full-gradient objective by detaching gradients at each denoising step, under a mild assumption.The passage states that the two gradients can be approximated, with k ∈ (0, 1) identified as a constant.Source: §A.2 · p. 14
- The derivation approximates the denoising-network Jacobian with an identity matrix, making the resulting gradient proportional to the identity.The proportionality constant is denoted ηt.Source: §A.2 · p. 14
B. More Ablation Studies
The diffusion-model Jacobian is approximately identity except at very small timesteps, where a multiplying constant approaches zero. An ablation finds that using pretrained knowledge outperforms training the same UNet architecture from scratch.
B.1. Diffusion Model Jacobian Approximation
- The diffusion-model Jacobian behaves like the identity except at very small timesteps, where the deviation does not affect the primary analysis.The visualization uses a 500 × 500 section of the full Jacobian matrix.Source: §B.1 · p. 15, Figure 12 · p. 15
- The timestep-dependent constant decreases toward zero as t approaches 0 in DDIM.The passage relates this to timesteps where the Jacobian deviates from identity.Source: §B.1 · p. 15, Figure 13 · p. 15
B.2. Utilizing Pretrained Knowledge of Diffusion Models
- Qualitative and quantitative results indicate that leveraging pretrained knowledge outperforms training the same UNet architecture from scratch.The comparison used filtered MS COCO data for 25K training steps, with other settings held constant.Source: §B.2 · p. 16, Table 5 · p. 16, Figure 14 · p. 16
C.1. Additional Qualitative Results
Additional qualitative examples show refined-noise generation comparable to guidance. In a user study, the method outperformed the random-Gaussian-noise baseline on all human evaluation criteria.
C.1. Additional Qualitative Results
- Refined-noise images are qualitatively comparable to images generated with guidance on examples drawn from the evaluations in Tables 1 and 2.The supplementary examples are shown in Figures 15–18.Source: §C.1 · p. 17, Figure 15 · p. 18, Figure 16 · p. 19
C.2. User Study
- The method outperformed the random-Gaussian-noise baseline on all human evaluation criteria in a study with 26 participants evaluating 20 image pairs.Participants judged overall image quality and prompt faithfulness; the reported percentages use total selections divided by total responses.Source: §C.2 · p. 17, Table 6 · p. 17
D.1. Implementation Details
The implementation refines Gaussian noise once before unguided denoising, while guided pipelines generate comparison images. The paper also describes training and evaluation settings and reports that learning the noise prompt converges faster and yields higher-quality outputs than fine-tuning the denoising network at equal training steps.
D.1. Implementation Details
- The noise-refining model maps Gaussian noise to refined noise, which the guidance-free pipeline denoises, while guided pipelines generate comparison images using guidance methods.The implementation uses Stable Diffusion 2.1 with LoRA rank 128 and a default 10-step refinement setup.Source: §D.1 · pp. 22–23
D.2. Experimental Details
- Training uses 20K CFG-generated MS COCO images and 30K CFG-and-PAG-generated Pick-a-pic images, filtered by an aesthetics score above 6.0.Evaluation includes Drawbench, HPDv2, Pick-a-pic, and MS-COCO; inference time is averaged over 30K images at 20 steps and batch size 1 on an RTX 3090.Source: §D.2 · p. 23
E.1. Effectiveness of Prompt Learning
- Noise prompts are conditionally generated from Gaussian noise and text, shifting learning to the pipeline input rather than directly training the denoising model.The discussion compares this approach to conditional prompt learning, which optimizes prompts instead of models.Source: §E.1 · p. 24
- Training the noise prompt leads to faster convergence and higher-quality outputs than fine-tuning the denoising network at the same training steps.The comparison fine-tunes a LoRA module with the same rank and layers as the noise-refining model.Source: §E.1 · p. 24
E.2. Why does refined noise help denoising?
Frequency and attention analyses associate refined noise’s guidance-free performance with low-frequency structure and prompt-aligned layouts formed early in denoising. Additional analyses show how prompts interact with refined layouts and compare the method with PAHI.
E.2. Why does refined noise help denoising?
- Replacing only the low-frequency band (0, 0.1) with refined-noise components produces a refined-noise image, with results emerging from a cutoff radius of 0.03.The study uses Fourier-transformed initial and refined noise and varies the frequency replacement band.Source: Figure 20 · p. 24, §E.2 · p. 25
- Even the lower 50% of refined-noise frequency bands can form a recognizable corgi shape, while high-frequency-only inputs fail to generate coherent images.The figure uses DDIM with 20 denoising steps without CFG.Source: §E.2 · p. 25, Figure 22 · p. 25
- When omitted frequency components are reinitialized with Gaussian noise, low-frequency components retain the structure and high-frequency components add details; low-only inputs can reconstruct the image, unlike high-only inputs.Retaining only the lower 10%–20% of frequencies can still reconstruct the original image when the rest is reinitialized.Source: Figure 23 · p. 26
- The authors infer that unguided outputs are poor because denoising fails to form appropriate low-frequency layouts, leaving high-frequency detail to produce unnatural artifacts.They identify why unguided models struggle to form low-frequency components as an open direction for future research.Source: §E.2 · pp. 26–27
- With Gaussian noise, cross-attention fails to form meaningful maps early and remains poorly aligned later, whereas CFG or refined noise yields prompt-aligned maps and refined noise shows object shapes from the first step.The paper relates alignment to coherent layouts and successful outputs.Source: §E.2 · p. 27, Figure 25 · p. 27
E.3. Impact of Initial Noise and Prompt on Generated Image
- When the layout prompt and denoising prompt match or are similar, refined noise retains the layout; an unrelated laptop prompt fails to align, while CFG can disregard the initial lion layout and generate a laptop.With a null denoising prompt, the refined layout can still produce a feline animal.Source: §E.3 · pp. 28–29
- Refined versions of spherical interpolations between Gaussian noises produce semantic image interpolations without guidance, indicating that the model learns a mapping rather than memorizing specific low-frequency signals.The unrefined interpolated noises are also denoised without guidance for comparison.Source: §E.3 · p. 29, Figure 27 · p. 29
E.4. Comparison with a related work
- Compared with PAHI’s prompt-conditioned Gaussian parameter prediction, NoiseRefine targets guidance replacement without a Gaussian-distribution constraint and supports full-step models; the reported comparison favors NoiseRefine.The comparison uses filtered 20K MS COCO images for 25K steps; the paper notes PAHI’s official code was unavailable.Source: §E.4 · p. 30, Figure 28 · p. 30, Table 7 · p. 30
E.5. Robustness to the number of denoising steps and schedulers
NoiseRefine’s refined noise produces reliable outputs across tested denoising-step counts and scheduler settings, despite training with a fixed scheduler and step count.
E.5. Robustness to the number of denoising steps and schedulers
- Refined noise consistently produces reliable outputs across the tested denoising-step counts and scheduler settings.The qualitative experiments use 10, 20, and 50 denoising steps and compare multiple schedulers.Source: §E.5 · p. 30, Figure 29 · p. 31
- The noise-refining model is trained with DDIM and 10 denoising steps, then tested with DPM++ SDE, DPM++ 2M, and EDM.The scheduler comparisons use the prompt “a photo of a cat”.Source: §E.5 · p. 30
- Figure 29 presents results for different schedulers at 10, 20, and 50 denoising steps.The passage characterizes the results as consistent across these configurations.Source: §E.5 · p. 30, Figure 29 · p. 31