Source-linked AI summary
InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization
Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, Di Huang
TL;DR
Text-to-image diffusion models can still produce images that fail to align with prompts, a problem the paper attributes to invalid initial noise. InitNO evaluates and partitions initial noise using attention-based scores, then optimizes it toward valid regions. The authors report improved semantic accuracy over state-of-the-art approaches and describe the method as plug-and-play.
Problem
Text-to-image diffusion models can generate images that remain poorly aligned with prompts, including cases of subject neglect, mixing, and incorrect attribute binding.
Source: §1 · p. 1Method
InitNO partitions the initial latent space using cross-attention response and self-attention conflict scores, then optimizes noise toward valid regions.
Source: §1 · p. 2, §4 · p. 3, §4.1 · p. 4Results
The authors report that InitNO surpasses state-of-the-art approaches in semantic accuracy and improves alignment with specific conditions such as layout-to-image generation.
Source: §1 · p. 2Takeaways & Limitations
The method is presented as a plug-and-play way to integrate training-free controllable generation into existing diffusion models.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Recent strides in the development of diffusion models, exemplified by advancements such as Stable Diffusion, have underscored their remarkable prowess in generating visually compelling images. However, the imperative of achieving a seamless alignment between the generated image and the provided prompt persists as a formidable challenge. This paper traces the root of these difficulties to invalid initial noise, and proposes a solution in the form of Initial Noise Optimization (InitNO), a paradigm that refines this noise. Considering text prompts, not all random noises are effective in synthesizing semantically-faithful images. We design the cross-attention response score and the self-attention conflict score to evaluate the initial noise, bifurcating the initial latent space into valid and invalid sectors. A strategically crafted noise optimization pipeline is developed to guide the initial noise towards valid regions. Our method, validated through rigorous experimentation, shows a commendable proficiency in generating images in strict accordance with text prompts. Our code is available at https://github.com/xiefan-guo/initno.
1. Introduction
The paper attributes persistent prompt-alignment failures in text-to-image diffusion to invalid initial noise. InitNO partitions the initial latent space and optimizes noise toward valid regions, improving semantic accuracy over state-of-the-art approaches.
1. Introduction
- Prompt alignment remains difficult despite large-scale training, with subject neglect, subject mixing, and incorrect attribute binding among the persistent failures.The paper attributes these challenges to invalid initial noise.Source: §1 · p. 1
- Different initial noises under the same prompt can yield markedly different alignment, motivating a division of the initial latent space into valid and invalid regions.Noise from valid regions produces semantically reasonable images.Source: §1 · pp. 1–2
- InitNO partitions the latent space and steers initial noise using cross-attention response and self-attention conflict scores.The method also uses a distribution-alignment loss to keep optimized noise near the standard Gaussian distribution, addressing under- and over-optimization risks.Source: §1 · p. 2
- InitNO outperforms state-of-the-art approaches in semantic accuracy and offers plug-and-play, training-free controllable generation for conditions such as layout-to-image.The introduction also reports improved alignment with specific conditions.Source: §1 · p. 2
2. Related Work
Related work includes architectural approaches that require training new models and training-free methods that modify denoising. InitNO instead optimizes sampled noise in the initial latent space.
2. Related Work
- Some approaches expand network architectures and add large language models, but require training new text-to-image models and do not apply to existing widely used models.Other training-free strategies are also discussed.Source: §2 · p. 2
- Prior training-free methods include combining outputs from multiple diffusion models, adjusting cross-attention, and shifting noisy images during denoising.Per-step noisy-image updates require careful parameter design to balance under- and over-optimization.Source: §2 · pp. 2–3
- Unlike those approaches, InitNO fully optimizes sampled noise in the initial latent space and is described as efficient and orthogonal to existing methods.The paper says this route avoids the under- versus over-optimization trade-off.Source: §2 · p. 3
3. Preliminaries
Stable Diffusion performs text-conditioned diffusion in an autoencoder latent space. Its cross-attention links text tokens to image features, while self-attention relates spatial image features.
3. Preliminaries
- Stable Diffusion encodes an image into a latent code with an encoder and reconstructs it with a decoder.The model operates in the acquired latent space rather than directly in the image domain.Source: §3 · p. 3
- A DDPM denoises latent representations over timesteps, conditioned in Stable Diffusion on a CLIP text embedding; its stated training objective minimizes squared noise-prediction error.At inference, a standard-Gaussian latent is sequentially denoised and decoded into an image.Source: §3 · p. 3
- Cross-attention uses text-derived keys and values with UNet features as queries to relate text tokens to spatial image patches.The attention map gives each token's probability at a spatial patch.Source: §3 · p. 3
- Self-attention uses intermediate image features as keys and values to correlate spatial positions and support globally coherent structures.Its map indexes attention associated with spatial patches.Source: §3 · p. 3
4. INITNO
INITNO combines initial-latent-space partitioning, which identifies valid noise, with a pipeline that steers sampled noise toward those regions.
4.1. Initial latent space partitioning
- Cross-attention response and self-attention conflict scores screen for subject neglect and subject mixing; scores below 0.2 and 0.3, respectively, pass their tests, and both tests define valid noise.The scores are calculated using the initial noise and first denoising step.Source: §4.1 · p. 4
4.2. Noise optimization pipeline
- Unlike stepwise tuning, INITNO adjusts noise in the initial latent space, where it can be constrained to the standard-Gaussian domain; under- or over-optimization risks poor quality or out-of-domain images.Attend-and-Excite modifies the noisy image at each denoising step, while its settings can require case-specific tuning.Source: §4.2 · pp. 4–5
- A joint loss combines cross-attention response, self-attention conflict, and KL distribution-alignment losses to guide noise optimization.The KL term constrains the target Gaussian distribution toward the standard Gaussian.Source: §4.2 · p. 5
- The procedure optimizes Gaussian mean and standard deviation parameters, resampling noise after 50 unsuccessful updates for up to five rounds before choosing the pool noise with the lowest summed scores.The optimization uses Adam with a learning rate of 1 × 10^-2.Source: §4.2 · p. 5
5. Experiments
Experiments evaluate INITNO on two-subject datasets and report stronger prompt alignment, favorable user-study comparisons, and extensions to grounded and complex prompts.
5.1. Experimental settings
- Experiments use Stable Diffusion v1.4 with guidance 7.5 and 50 denoising steps, evaluating three two-subject datasets while noting broader prompt applicability.The datasets are Animal-Animal, Animal-Object, and Object-Object.Source: §5.1 · pp. 5–6
5.2. Qualitative comparison
- Against methods using the same prompts and seeds, INITNO separates concepts, handles varied sampled noises, and binds properties more faithfully.The qualitative comparisons describe reduced subject mixing and neglect, including examples such as a yellow clock.Source: §5.2 · p. 6
5.3. Quantitative comparison
- INITNO outperforms other approaches on the reported CLIP similarities, earns favorable user-study comparisons, and takes 18.93 seconds per image versus SD’s 8.34.The CLIP evaluation reports full-prompt, minimum-object, and text-text similarity; higher scores are better.Source: §5.3 · pp. 6–7, Figure 7 · p. 7
5.4. Ablation study
- Self-attention conflict loss addresses apple–pear mixing, while cross-attention conflict substitution is unsatisfactory and distribution alignment guards against distorted noise.Without distribution constraints, optimized noise may deviate from the standard Gaussian distribution.Source: §5.4 · p. 7
5.5. Grounded Text-to-Image
- When added to BoxDiff, INITNO transfers intermediate noisy-image modifications to the initial latent space, yielding faithful, location-aware images.The grounded generation example uses BoxDiff.Source: §5.5 · pp. 7–8, Figure 11 · p. 8
5.6. More results
- Examples with three or more subjects and intricate attributes show high-fidelity complex scenes, while final cross-attention maps show reasonable allocation alongside semantically consistent results.The maps are visualized for each subject token after denoising.Source: §5.6 · p. 8
6. Conclusion
The conclusion presents INITNO as an attention-guided way to partition and navigate initial noise, with distribution alignment supporting semantically faithful generation.
6. Conclusion
- INITNO partitions the initial latent space using attention-based scores and steers noise through an optimization pipeline with distribution alignment, outperforming existing methods in semantic faithfulness.The approach is presented as a plug-and-play solution for training-free controllable generation in existing diffusion models.Source: §6 · p. 8