Source-linked AI summary
Dense Text-to-Image Generation with Attention Modulation
Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, Jun-Yan Zhu
TL;DR
Dense-caption generation remains difficult because models may omit or blend objects, and text alone offers limited layout control. DenseDiffusion analyzes intermediate attention maps and modulates them during generation without fine-tuning, improving textual and layout fidelity while achieving comparable qualitative layout control to specially trained models.
Problem
Existing text-to-image models often omit or blend objects in dense captions, while text prompts alone provide limited precise scene-layout control.
Method
DenseDiffusion is a training-free method that modulates a pretrained model’s intermediate attention maps according to caption segments and layout masks.
Results
DenseDiffusion improves Stable Diffusion, outperforms several compositional diffusion models across textual, layout, and image-quality evaluations, and achieves comparable qualitative layout control to trained models.
Takeaways & Limitations
Attention modulation enables a pretrained text-to-image model to reflect dense textual and spatial conditions without fine-tuning.
Takeaways & Limitations
Performance depends on Stable Diffusion’s capacity, and fine-grained masks with thin structures remain difficult to follow.
Abstract
from arXiv · showhide
Existing text-to-image diffusion models struggle to synthesize realistic images given dense captions, where each text prompt provides a detailed description for a specific image region. To address this, we propose DenseDiffusion, a training-free method that adapts a pre-trained text-to-image model to handle such dense captions while offering control over the scene layout. We first analyze the relationship between generated images' layouts and the pre-trained model's intermediate attention maps. Next, we develop an attention modulation method that guides objects to appear in specific regions according to layout guidance. Without requiring additional fine-tuning or datasets, we improve image generation performance given dense captions regarding both automatic and human evaluation scores. In addition, we achieve similar-quality visual results with models specifically trained with layout conditions.
1. Introduction
Dense captions often cause text-to-image models to omit or blend objects, while text prompts alone provide limited precise control over scene layout. DenseDiffusion addresses these issues with training-free attention modulation and improves fidelity across textual, layout, and image-quality evaluations.
- Dense captions describe specific image regions, but existing models often omit or blend visual features from different objects.
- Text prompts alone make precise control over generated scene layouts difficult.
- Training or fine-tuning layout-conditioned models is computationally expensive and may require retraining for each new user.
- DenseDiffusion analyzes intermediate self- and cross-attention maps, then modulates them on the fly according to layout conditions without training.
- DenseDiffusion improves Stable Diffusion and outperforms several compositional diffusion models across textual, layout, and image-quality conditions.
2. Related work
Related work spans text-to-image diffusion, training-free compositional methods, and spatial-control systems. DenseDiffusion builds on attention-based editing and modulation while targeting faithful dense-caption and layout conditioning without fine-tuning.
- Text-to-Image Diffusion Models: Text-to-image diffusion models synthesize diverse photorealistic images and support flexible editing through interactions between intermediate image and textual features.
- Text-to-Image Diffusion Models: Prompt-to-Prompt, Plug-and-Play, and pix2pix-zero manipulate or reuse attention and spatial features to preserve structure or align edited generations.
- Compositional Diffusion Methods: Composable Diffusion, MultiDiffusion, Attend-and-Excite, and Structure Diffusion address dense captions with training-free denoising, attention, or noise optimization strategies.
- Compositional Diffusion Methods: Paint-with-words increases cross-attention for segmented objects, but existing compositional methods can fail to reflect text and layout conditions faithfully.
- Image Synthesis with Spatial Control: Spatial-control approaches include data-driven systems, domain-limited conditional GANs, and methods that add spatial modulation to pretrained diffusion models.
- Image Synthesis with Spatial Control: Several concurrent works also investigate training-free spatial control for diffusion models.
3. Method
DenseDiffusion represents captions as region-linked segments and modulates Stable Diffusion’s cross- and self-attention maps during denoising. Its modulation preserves the pretrained model’s behavior while adapting strength to attention ranges, segment areas, and timestep.
- 3. Method: Each segment (c_n, m_n) pairs a nonoverlapping caption phrase with a binary map for one image region.
- 3.1. Preliminaries: Stable Diffusion generates images through successive denoising steps from random noise, using a timestep-dependent noise-prediction network.
- 3.1. Preliminaries: Matched keys consistently receive higher mean and maximum attention than unmatched keys in both cross- and self-attention analyses.
- 3.1. Preliminaries: Attention maps are computed as softmax(QK^T / sqrt(d)), where Q and K are query and key values derived from intermediate and context features.
- 3.2. Layout-guided Attention Modulation: Cross-attention modulation increases scores between image tokens and text tokens assigned to the same segment region.
- 3.2. Layout-guided Attention Modulation: Self-attention modulation increases within-segment communication and decreases communication between different segments to reduce feature mixing.
- 3.2. Layout-guided Attention Modulation: The modulation accounts for original attention ranges, segment areas, and timestep by using Mpos, Mneg, S, and λ_t.
4. Experiments
Experiments evaluate DenseDiffusion on dense-caption fidelity and layout control using automatic metrics, user studies, visual comparisons, and ablations. The method achieves stronger condition fidelity than training-free baselines and comparable or better layout adherence than trained layout-conditioned models.
- Textual Fidelity: DenseDiffusion achieves the best performance on textual-condition automatic metrics and user studies.Table 2 reports the highest results for both automatic evaluation and human preference.
- Layout Fidelity: 63% of AMT participants prefer DenseDiffusion over SD-Pww for fidelity to layout conditions.SD-Pww is the only baseline evaluated with segmentation-map layout control, and DenseDiffusion also achieves substantially higher IoU scores.
- Evaluation Setting: The evaluation compares dense-caption fidelity and layout alignment using CLIP-Score, SOA-I, IoU, and human preference studies.The dataset contains 250 MS-COCO samples with two or more unique objects, generating four images per caption for each baseline.
- Layout Fidelity: DenseDiffusion aligns more closely with input masks than SD-Pww while using the same dense caption and remaining training-free.Among the compared Stable Diffusion-based methods, only SD-Pww and DenseDiffusion support segmentation-map layout control.
- Comparison to Layout-Conditioned Models: DenseDiffusion adheres to layout conditions comparably to SpaText and outperforms MAS in many visual comparisons despite requiring no layout-specific training.MAS and SpaText are trained for layout control, whereas DenseDiffusion and SD-Pww use pre-trained Stable Diffusion models without training.
- Ablation Study: Ablations show that removing attention-modulation components generally reduces metric scores, with cross-attention modulation producing the largest degradation when omitted.Value-range adaptation also improves fidelity, while mask-area adaptation can produce monotonous backgrounds that help segmentation metrics without always satisfying textual conditions.
5. Conclusion
DenseDiffusion is a training-free method for improving dense-caption fidelity and controlling image layout, with results comparable to models specifically trained for layout control. Its effectiveness remains bounded by the base model’s object-generation capacity and coarse attention resolution for thin structures.
- DenseDiffusion improves fidelity to dense captions and enables image layout control without training.
- Comparable layout control is achieved relative to models specifically trained for this task.
- The method depends on Stable Diffusion’s capacity to generate the requested objects.It fails to produce objects that Stable Diffusion cannot generate itself, such as a juggling bear.
- Fine-grained masks with thin structures remain difficult because the attention layers are fairly coarse.The method can fail to follow details such as the shape of leaves.
A. Additional Comparison
Additional qualitative results evaluate concurrent methods on six dense captions, combining MS-COCO examples with captions curated by the authors.
- Six dense captions are used for additional qualitative comparison.Three captions come from MS-COCO, while three are curated by the authors.
A.1. Baselines
The baselines address object omission, mixed object features, and limited spatial control, using Stable Diffusion as their implementation or fine-tuning base.
- The baselines target object omission, mixed visual features, and the lack of spatial controls in pretrained text-to-image models.
- Attend-and-Excite and Universal-Guided-Diffusion modify inference with additional objectives, but Attend-and-Excite does not support layout conditions.Universal-Guided-Diffusion uses an off-the-shelf MobileNetV3 segmentation network for additional guidance.
A.2. Results
Qualitative comparisons show DenseDiffusion more faithfully follows textual and layout conditions than several alternative methods, while other methods exhibit recognizable failure modes.
- DenseDiffusion is more effective than SD-Pww and Structure Diffusion at following textual and layout conditions.Figures 9–11 use curated prompts, while Figures 12–14 use MS-COCO prompts.
- MultiDiffusion reflects objects accurately but often produces collage-like results because its independent denoising processes lack interaction.
- Composable Diffusion, Attend-and-Excite, and Universal-Guided-Diffusion often fail to follow the conditions and tend to generate unrealistic images.
- The qualitative comparisons include methods marked † that receive textual condition only.