Source-linked AI summary
Grounded Text-to-Image Synthesis with Attention Refocusing
Quynh Phung, Songwei Ge, Jia-Bin Huang
TL;DR
Text-to-image models still struggle to faithfully render prompts containing multiple objects, attributes, and spatial compositions. The paper uses LLM-generated layouts and two attention-refocusing losses during diffusion sampling, improving controllability across DrawBench, HRS, and TIFA. The method remains limited by incorrect or small GPT-4 boxes for many-object prompts and failures on some out-of-distribution layouts.
Problem
Existing text-to-image models often mix, swap, or omit objects, attributes, and spatial compositions in complex prompts.
Method
The method uses LLM-generated bounding-box layouts and two losses to refocus cross- and self-attention during sampling with pretrained diffusion models.
Results
The method significantly and consistently improves over several strong baselines on DrawBench, HRS, and TIFA when GPT-4 boxes are combined with attention-refocusing losses.
Takeaways & Limitations
Attention refocusing can be incorporated into existing text-to-image diffusion models to improve layout-conditioned controllability.
Takeaways & Limitations
For prompts with many objects, GPT-4 may produce incorrect counts or small boxes, and synthesis can fail on some out-of-distribution layouts.
Abstract
from arXiv · showhide
Driven by the scalable diffusion models trained on large-scale datasets, text-to-image synthesis methods have shown compelling results. However, these models still fail to precisely follow the text prompt involving multiple objects, attributes, or spatial compositions. In this paper, we reveal the potential causes in the diffusion model's cross-attention and self-attention layers. We propose two novel losses to refocus attention maps according to a given spatial layout during sampling. Creating the layouts manually requires additional effort and can be tedious. Therefore, we explore using large language models (LLM) to produce these layouts for our method. We conduct extensive experiments on the DrawBench, HRS, and TIFA benchmarks to evaluate our proposed method. We show that our proposed attention refocusing effectively improves the controllability of existing approaches.
1. Introduction
The paper addresses failures in text-to-image models involving multiple objects, attributes, and spatial relationships by grounding synthesis in explicit layouts and refocusing both cross- and self-attention. It combines LLM-generated layouts with attention-based guidance during sampling to improve controllability.
- Motivation: Existing text-to-image models often mix, swap, or omit objects, attributes, and spatial compositions specified in complex prompts.The paper identifies this as a controllability problem despite strong zero-shot capacity and photorealism.
- Motivation: Prior methods manipulate cross-attention maps but commonly overlook self-attention, where pixels with similar features can attend across different objects.The paper motivates addressing both attention mechanisms using explicit spatial layouts.
- Attention refocusing: The proposed method introduces two layout-based losses that refocus cross- and self-attention during sampling without extra training.The approach is designed to align attention with desired regions and prevent attention to irrelevant regions.
- LLM-generated layouts: LLMs, including GPT-4, generate explicit visual layouts such as bounding boxes from text prompts, avoiding the need to create layouts manually.The pipeline uses the generated layouts to ground synthesis with pretrained diffusion models.
- Results: The method significantly and consistently improves over several strong baselines on DrawBench, HRS, and TIFA when GPT-4 bounding boxes are combined with the proposed losses.The paper reports favorable comparisons with state-of-the-art grounded text-to-image methods.
- Method overview: The framework combines text-to-layout generation with grounded text-to-image synthesis using pretrained models and attention-refocusing during sampling.Both phases use off-the-shelf models without extra training.
3. Method
The method grounds text-to-image diffusion sampling in explicit layouts and refocuses both cross-attention and self-attention without extra training. LLM-generated bounding boxes provide layouts, while iterative loss-based updates guide denoising toward the specified regions.
- Overview: The pipeline uses off-the-shelf pretrained models in two phases: text-to-layout prediction and grounded text-to-image generation.LLMs produce visual representations such as bounding boxes from text prompts; no extra training is used.
- Cross-Attention Refocusing (CAR): Cross-attention refocusing uses layout masks to reduce token attention to irrelevant regions and improve correspondence with target boxes.For the “three parrots” example, CAR mitigates incorrect background attention and produces three parrots as desired.
- Self-Attention Refocusing (SAR): Self-attention refocusing reduces attention from pixels inside each box to regions outside that box, helping preserve distinct regional attributes.The loss addresses similar regions, such as a car and chair, whose attributes can otherwise become mixed.
- Sampling with the Attention-Refocusing Losses: CAR and SAR losses modify the noisy sample during denoising through gradient descent, with repeated updates during early denoising steps.After τ updates, the output returns to the diffusion UNet to resume denoising and compute x_t−1.
- Text-to-Layout Prediction: GPT-4 generates object labels and box coordinates from the image-generation prompt to supply the explicit layout.The procedure uses a designed prompt with in-context learning.
4. Experiments
Experiments evaluate attention-refocusing guidance across grounded text-to-image models, datasets, guidance formats, and baselines. The method improves controllability, alignment, and layout fidelity while preserving image quality.
- Evaluation setup: The method is evaluated on HRS, DrawBench, TIFA, COCO2014, and DenseDiffusion-based mask guidance tasks.HRS covers counting, spatial, color, and size compositions; TIFA assesses text-image alignment, while COCO2014 supports FID evaluation.
- Evaluation setup: Attention-based guidance plugs into Stable Diffusion, Attend-and-excite, Layout-guidance, MultiDiffusion, GLIGEN, and ControlNet.The experiments test both bounding-box and segmentation-mask guidance under default configurations.
- Quantitative results: Counting F1 improves by an average of 2%, spatial accuracy by an average of 10%, and size and color accuracy by up to 10.9% on HRS.Layout-guidance shows a 7% counting improvement, while GLIGEN gains around 10% in spatial accuracy and 8% in color accuracy.
- Quantitative results: The attention-guided Stable Diffusion variant outperforms Layout-guidance and MultiDiffusion, while CAR and SAR improve counting and spatial accuracy across TIFA baseline versions.The overall TIFA score also improves without detriment to other categories.
- Quantitative results: Attention-refocusing improves textual alignment without affecting image quality, maintaining FID scores compatible with the original models.The method reports an IoU of 38.97 ± 0.56 with leading SOA-I and CLIP scores for mask guidance.
- Ablation and qualitative results: Adding both CAR and SAR to GLIGEN yields around a 10% accuracy improvement on HRS, while qualitative results reduce spatial, attribute, and object-count errors.The losses also refine ControlNet outputs by reducing attribute mixing and irrelevant objects.
- LLM-generated layouts: GPT-4 outperforms Llama 1, Llama 2, and GPT-3 in three metrics, and the resulting two-stage model surpasses single-stage Stable Diffusion in object relationships and textual alignment.Users can iteratively modify layouts through ChatGPT to generate updated images.
5. Limitation
The framework can fail when GPT-4 miscounts or produces undersized boxes for prompts with many objects. It can also fail when the grounded generator does not follow layouts outside its distribution.
- Limitation: For prompts describing many objects, GPT-4 occasionally produces incorrect counts or small bounding boxes.These errors affect the generated layout before image synthesis.
- Limitation: The grounded text-to-image model sometimes fails to adhere to accurately generated out-of-distribution layouts.Thus, correct layout prediction does not guarantee faithful image generation in every setting.
6. Conclusion
The paper presents attention refocusing to align cross- and self-attention with layouts during sampling, while using LLMs to generate visual layouts from text prompts. The approach is designed for incorporation into existing text-to-image diffusion models and shows favorable performance against state-of-the-art grounded methods.
- Attention refocusing improves the alignment of cross- and self-attention layers given layouts during sampling.
- Large Language Models are explored for generating visual layouts from text prompts.
- The proposed losses can be incorporated into existing text-to-image diffusion models.
- Comprehensive experiments show favorable performance against state-of-the-art grounded text-to-image models.
8. Implementation details
The implementation applies attention-refocusing losses during denoising, uses fixed settings for refinement and smoothing, and generates layouts with GPT-4 prompting. It also compares the resulting two-stage models with single-stage baselines.
- Cross-Attention Refocusing and Self-Attention Refocusing losses are applied to 16 × 16 attention maps during 50 denoising steps.The refinement iteration step varies as denoising progresses.
- A Gaussian kernel with filter size 3 × 3 and standard-deviation value σ = 0.5 is used.
- Layout-guidance, MultiDiffusion, Attend-and-Excite, and GLIGEN use their original-paper default settings as baselines.
- The GPT-4 prompt combines task instructions, in-context exemplars, and a user prompt to produce object bounding boxes and labels.The instruction defines the output format, while exemplars provide additional task context.
- The two-stage text-to-image models are compared with single-stage Stable Diffusion and Attend-and-Excite models in Figure 13.
13 end
The denoising algorithm returns the latent for the next timestep and iterates through the prescribed refinement steps. Its control flow includes a return to Step 1.
- The algorithm includes a control-flow instruction to go to Step 1.
- The denoising procedure is organized as numbered algorithm steps.
- The excerpt places the control-flow instruction at Step 21.
23 end
GPT-4 can generate layouts that capture implicit object relationships, while the method also adapts attention-refocusing losses to segmentation masks and other layout modalities. Additional results include ControlNet applications and inference-time comparisons.
- Comparison of four language models: GPT-4 can reason about implicit object relationships in layout generation.For a squirrel with a leather racket, it places the racket box centrally within the squirrel box, unlike the other compared language models.
- The method adapts the Cross-Attention Refocusing loss to depth maps, segmentation masks, and edge maps.Segmentation masks are converted and associated with individual tokens.
- Comparison of four language models: Figure 12 compares layouts generated by Llama 1, Llama 2, GPT-3, and GPT-4.
- For segmentation masks, attention is optimized over the entire foreground by averaging because the masks provide precise object boundaries.
- The segmentation-mask formulation includes separate foreground and background losses.
- The LSAR formulation for bounding boxes is used, with additional results reported for applying the losses to ControlNet.
11. Additional quantitative result
Additional evaluations report improvements on DrawBench and HRS, including stronger performance than BoxDiff in several categories and faster inference than Layout-guidance.
- Attention-refocusing substantially improves DrawBench spatial accuracy and compares favorably with baselines in counting.
- Around 7%: attention-refocusing surpasses BoxDiff in HRS spatial composition, while also outperforming it in counting and color.
- The proposed losses improve HRS counting baselines and are compatible with Attend-and-excite.
- Attention-refocusing is more effective than Layout-guidance in speed.
12. Additional visual comparison GLIGEN with and without our losses
Visual comparisons show that attention-refocusing refines GLIGEN outputs across counting, spatial, color, and size compositions, including novel compositions and out-of-distribution prompts.
- Benchmark comparison: Across HRS and DrawBench, GLIGEN with the losses outperforms other grounded and text-to-image baselines across four categories.
- Counting: Counting: CAR and SAR losses avoid extra background objects, producing the correct number of objects with GLIGEN.
- Spatial: Spatial: CAR and SAR losses mitigate objects appearing in incorrect bounding boxes.
- Color: Color: attention-refocusing losses refine GLIGEN’s color blending problem.
- Size: Size: the losses generate correct objects inside bounding boxes, improving size accuracy even for out-of-distribution prompts.