Dense Text-to-Image Generation with Attention Modulation

• Dense captions describe specific image regions, but existing models often omit or blend visual features from different objects.Source: §1 · p. 2
• Text prompts alone make precise control over generated scene layouts difficult.Source: §1 · p. 2
• Training or fine-tuning layout-conditioned models is computationally expensive and may require retraining for each new user.Source: §1 · p. 2
• DenseDiffusion analyzes intermediate self- and cross-attention maps, then modulates them on the fly according to layout conditions without training.Source: §1 · p. 2
• DenseDiffusion improves Stable Diffusion and outperforms several compositional diffusion models across textual, layout, and image-quality conditions.Source: §1 · p. 2

Text-to-Image Diffusion Models

• Text-to-image diffusion models synthesize diverse photorealistic images and support flexible editing through interactions between intermediate image and textual features.Source: §2 · p. 2
• Prompt-to-Prompt, Plug-and-Play, and pix2pix-zero manipulate or reuse attention and spatial features to preserve structure or align edited generations.Source: §2 · p. 2

Compositional Diffusion Methods

• Composable Diffusion, MultiDiffusion, Attend-and-Excite, and Structure Diffusion address dense captions with training-free denoising, attention, or noise optimization strategies.Source: §2 · p. 2
• Paint-with-words increases cross-attention for segmented objects, but existing compositional methods can fail to reflect text and layout conditions faithfully.Source: §2 · p. 2

Image Synthesis with Spatial Control

• Spatial-control approaches include data-driven systems, domain-limited conditional GANs, and methods that add spatial modulation to pretrained diffusion models.Source: §2 · pp. 2–3
• Several concurrent works also investigate training-free spatial control for diffusion models.Source: §2 · p. 3

3. Method

• Each segment (c_n, m_n) pairs a nonoverlapping caption phrase with a binary map for one image region.Source: §3 · p. 3

3.1. Preliminaries

• Stable Diffusion generates images through successive denoising steps from random noise, using a timestep-dependent noise-prediction network.Source: §3.1 · p. 3
• Matched keys consistently receive higher mean and maximum attention than unmatched keys in both cross- and self-attention analyses.Source: Table 1 · p. 3
• Attention maps are computed as softmax(QK^T / sqrt(d)), where Q and K are query and key values derived from intermediate and context features.Source: §3.1 · p. 3

3.2. Layout-guided Attention Modulation

• Cross-attention modulation increases scores between image tokens and text tokens assigned to the same segment region.Source: §3.2 · p. 4
• Self-attention modulation increases within-segment communication and decreases communication between different segments to reduce feature mixing.Source: §3.2 · p. 4
• The modulation accounts for original attention ranges, segment areas, and timestep by using Mpos, Mneg, S, and λ_t.Source: §3.2 · pp. 4–5

Textual Fidelity

• DenseDiffusion achieves the best performance on textual-condition automatic metrics and user studies.Table 2 reports the highest results for both automatic evaluation and human preference.Source: Table 2 · p. 5, §4.2 · p. 6

Layout Fidelity

• 63% of AMT participants prefer DenseDiffusion over SD-Pww for fidelity to layout conditions.SD-Pww is the only baseline evaluated with segmentation-map layout control, and DenseDiffusion also achieves substantially higher IoU scores.Source: Table 3 · p. 5, §4.2 · p. 6

Evaluation Setting

• The evaluation compares dense-caption fidelity and layout alignment using CLIP-Score, SOA-I, IoU, and human preference studies.The dataset contains 250 MS-COCO samples with two or more unique objects, generating four images per caption for each baseline.Source: §4.1 · pp. 5–6

Layout Fidelity

• DenseDiffusion aligns more closely with input masks than SD-Pww while using the same dense caption and remaining training-free.Among the compared Stable Diffusion-based methods, only SD-Pww and DenseDiffusion support segmentation-map layout control.Source: Figure 4 · p. 6, §4.2 · p. 6

Comparison to Layout-Conditioned Models

• DenseDiffusion adheres to layout conditions comparably to SpaText and outperforms MAS in many visual comparisons despite requiring no layout-specific training.MAS and SpaText are trained for layout control, whereas DenseDiffusion and SD-Pww use pre-trained Stable Diffusion models without training.Source: Figure 5 · p. 7, §4.2 · p. 7

Ablation Study

• Ablations show that removing attention-modulation components generally reduces metric scores, with cross-attention modulation producing the largest degradation when omitted.Value-range adaptation also improves fidelity, while mask-area adaptation can produce monotonous backgrounds that help segmentation metrics without always satisfying textual conditions.Source: §4.2 · pp. 7–9

• DenseDiffusion improves fidelity to dense captions and enables image layout control without training.Source: §5 · p. 9
• Comparable layout control is achieved relative to models specifically trained for this task.Source: §5 · p. 9
• The method depends on Stable Diffusion’s capacity to generate the requested objects.It fails to produce objects that Stable Diffusion cannot generate itself, such as a juggling bear.Source: §5 · p. 9
• Fine-grained masks with thin structures remain difficult because the attention layers are fairly coarse.The method can fail to follow details such as the shape of leaves.Source: §5 · p. 9

• Six dense captions are used for additional qualitative comparison.Three captions come from MS-COCO, while three are curated by the authors.Source: A. Additional Comparison · p. 12

• The baselines target object omission, mixed visual features, and the lack of spatial controls in pretrained text-to-image models.Source: §A.1 · p. 12
• Attend-and-Excite and Universal-Guided-Diffusion modify inference with additional objectives, but Attend-and-Excite does not support layout conditions.Universal-Guided-Diffusion uses an off-the-shelf MobileNetV3 segmentation network for additional guidance.Source: §A.1 · p. 12

• DenseDiffusion is more effective than SD-Pww and Structure Diffusion at following textual and layout conditions.Figures 9–11 use curated prompts, while Figures 12–14 use MS-COCO prompts.Source: §A.2 · p. 12
• MultiDiffusion reflects objects accurately but often produces collage-like results because its independent denoising processes lack interaction.Source: §A.2 · p. 12
• Composable Diffusion, Attend-and-Excite, and Universal-Guided-Diffusion often fail to follow the conditions and tend to generate unrealistic images.Source: §A.2 · p. 12
• The qualitative comparisons include methods marked † that receive textual condition only.Source: Figure 9 · p. 13, Figure 10 · p. 14, Figure 11 · p. 15