PaperlayerView exact revision on arXiv

Source-linked AI summary

Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, Daniel Cohen-Or

arXiv:2301.13826v2cs.CVcs.CLcs.GRcs.LG

TL;DR

Text-to-image diffusion models can omit prompt subjects or bind attributes incorrectly, limiting semantic faithfulness. The paper introduces Attend-and-Excite, an inference-time attention-based latent guidance method, and reports improved faithfulness across prompts while remaining bounded by the pretrained model’s expressive power.

  • Problem

    Text-to-image models do not always faithfully reflect prompt semantics, including failures to generate subjects and correctly bind attributes.

  • Method

    Attend-and-Excite guides denoising latents so cross-attention strengthens every subject token’s activation without additional training or fine-tuning.

  • Results

    Attend-and-Excite alleviates catastrophic neglect and incorrect attribute binding, producing more semantically faithful images than Stable Diffusion and alternative methods.

  • Takeaways & Limitations

    Inference-time latent manipulation can correct semantic failures after a diffusion generator takes a wrong turn.

  • Takeaways & Limitations

    The method is limited by the pretrained generator’s expressive power and may produce out-of-distribution latents for prompts outside its learned textual distribution.

Abstract

from arXiv · show

Recent text-to-image generative models have demonstrated an unparalleled ability to generate diverse and creative imagery guided by a target text prompt. While revolutionary, current state-of-the-art diffusion models may still fail in generating images that fully convey the semantics in the given text prompt. We analyze the publicly available Stable Diffusion model and assess the existence of catastrophic neglect, where the model fails to generate one or more of the subjects from the input prompt. Moreover, we find that in some cases the model also fails to correctly bind attributes (e.g., colors) to their corresponding subjects. To help mitigate these failure cases, we introduce the concept of Generative Semantic Nursing (GSN), where we seek to intervene in the generative process on the fly during inference time to improve the faithfulness of the generated images. Using an attention-based formulation of GSN, dubbed Attend-and-Excite, we guide the model to refine the cross-attention units to attend to all subject tokens in the text prompt and strengthen - or excite - their activations, encouraging the model to generate all subjects described in the text prompt. We compare our approach to alternative approaches and demonstrate that it conveys the desired concepts more faithfully across a range of text prompts.

1 INTRODUCTION

Text-to-image models can produce creative images that fail to faithfully represent every subject and attribute in a prompt. Attend-and-Excite introduces inference-time semantic guidance that strengthens attention to neglected subjects and improves semantic faithfulness.

  • Motivation: State-of-the-art text-to-image models can fail to reflect the target prompt’s semantic meaning despite generating diverse imagery.
  • Semantic Issues: Catastrophic neglect omits prompt subjects, while incorrect attribute binding assigns attributes to the wrong subjects or leaves them unbound.
  • Contribution: Generative Semantic Nursing shifts the latent code during denoising so generated images better incorporate semantic information from the input prompt.
  • Attend-and-Excite: Attend-and-Excite guides cross-attention to make every subject token dominant in some image patch, strengthening neglected-token activations during inference.
  • Results: Attend-and-Excite demonstrates superiority over Stable Diffusion and alternative methods in generating semantically faithful images across varied prompts.

2 RELATED WORK

Prior work addresses semantic flaws in text-to-image generation through compositional generation or prompt-structure manipulation, but important scope and effectiveness limitations remain. The paper focuses on inference-time guidance of a pretrained model.

  • Background: Text-to-image methods have progressed from GANs and autoregressive models to diffusion models, yet faithful alignment with input prompts remains difficult.
  • Composable Diffusion: Composable Diffusion combines outputs for different image components, but often struggles with realistic multi-object compositions and supports only subject conjunctions and negations.
  • StructureDiffusion: StructureDiffusion splits prompts into noun phrases and averages their cross-attention operations, but its outputs often resemble Stable Diffusion and do not meaningfully amend semantic faults.
  • Scope: Object relations and compositions are additional semantic issues that may require models beyond this work’s inference-time guidance of a pretrained generator.

3 PRELIMINARIES

Stable Diffusion generates images by denoising latent representations conditioned on text through cross-attention. Attend-and-Excite relies on these attention maps to guide subject coverage.

  • Latent Diffusion Models: Stable Diffusion encodes images into spatial latent codes, denoises them with a diffusion model, and decodes the final latent into an image.
  • Latent Diffusion Models: At each denoising timestep, the UNet removes noise from a noised latent using the timestep and a text-conditioning encoding.
  • Text-Conditioning Via Cross-Attention: Stable Diffusion applies text guidance through cross-attention layers at spatial resolutions 64, 32, 16, and 8.
  • Text-Conditioning Via Cross-Attention: The attention map A_t assigns each text token a probability for every spatial patch, indicating how much information the token passes to that patch.
  • Text-Conditioning Via Cross-Attention: Attend-and-Excite operates on 16 × 16 attention units because they contain the most semantic information.

4 ATTEND-AND-EXCITE

Attend-and-Excite guides Stable Diffusion during inference by refining cross-attention and latent representations so all subject tokens receive strong activation. Its procedure extracts subject-token maps, optimizes neglected tokens across denoising steps, and uses gradual refinement to support faithful generation.

  • Generative Semantic Nursing: Attend-and-Excite shifts the noised latent at each denoising timestep using gradients of an attention-based loss, requiring subject tokens to receive strong activation.The intervention occurs on the fly during inference without additional training.
  • Cross-Attention Extraction: The method extracts spatial cross-attention maps for subject tokens by aggregating the Stable Diffusion UNet’s 16 × 16 attention layers and heads.The maps indicate each subject token’s influence on image patches, after reweighting that ignores the encoder’s start-of-text token.
  • Smooth Attention Maps: Gaussian smoothing makes a maximally activated patch depend on neighboring patches, reducing solutions based on partial subject-like information rather than a full generated subject.The paper notes that a single high-attention patch can otherwise resemble only part of an object.
  • Attention Optimization: Attend-and-Excite maximizes each subject token’s strongest patch activation, strengthening the most neglected token at a timestep and allowing different tokens to be emphasized at different timesteps.This objective encodes the intuition that each subject should dominate at least one image patch.
  • Iterative Latent Refinement: Iterative latent refinement raises subject-token attention gradually across selected denoising steps, because excessive updates could push the latent out of distribution and produce incoherent images.The implementation uses thresholds of 0.05, 0.5, and 0.8 at iterations 0, 10, and 20, respectively, with a minimum target attention of 0.8.
  • Explainable Image Generators: With Attend-and-Excite, attention maps for both cat and frog are accurately localized, making them more faithful explanations after catastrophic neglect is mitigated.Without mitigation, the neglected subject’s map highlights irrelevant regions and no longer faithfully represents localization.

5 RESULTS

Attend-and-Excite is evaluated on prompts designed to test subject neglect and attribute binding, using qualitative comparisons, CLIP-based metrics, and a user study. Across these evaluations, it more faithfully represents the prompt than the alternatives, including on complex prompts and multiple semantic metrics.

  • Evaluation Setup: The benchmark uses three prompt subsets testing two subjects, animal-object combinations, and two colored objects, with varied attributes matched to subject tokens.The authors note that the method also applies beyond these conjunction-and-color prompts, including prompts with more subjects and complex attributes.
  • Qualitative Comparisons: Qualitatively, Attend-and-Excite more faithfully generates all subjects and binds colors correctly than Stable Diffusion, StructureDiffusion, and Composable Diffusion.The alternatives either omit subjects, mix them, or assign colors to the wrong objects; Attend-and-Excite also handles complex prompts and subject interactions.
  • Text-Image Similarities: Attend-and-Excite outperforms all baselines across all three subsets for both Full Prompt Similarity and Minimum Object Similarity.Minimum Object Similarity specifically evaluates the most neglected subject, while Full Prompt Similarity uses the complete text prompt.
  • Text-Image Similarities: At least 7% separates Attend-and-Excite from both Stable Diffusion and StructureDiffusion on Minimum Object Similarity across all test cases.StructureDiffusion scores are similar to, and slightly lower than, Stable Diffusion; Composable Diffusion can appear closer because mixed objects may score highly against both subjects.
  • Text-Text Similarities: Attend-and-Excite outperforms every alternative by at least 4.7% across all three subsets on CLIP text-text similarity.The text-text metric also identifies Composable Diffusion as the lowest-performing approach, consistent with its subject-mixing behavior.
  • User Study: In a 65-respondent user study, Attend-and-Excite receives the highest preference across Animal-Animal, Animal-Object, and Object-Object subsets.The reported preferences are 90.70%, 77.64%, and 77.16%, respectively, using 10 prompts and four shared random seeds per subset.

6 LIMITATIONS

Attend-and-Excite remains constrained by Stable Diffusion’s learned distribution and expressive power. It can produce out-of-distribution or less realistic results for unfamiliar prompts and unnatural subject combinations, while broader semantic challenges remain.

  • Stable Diffusion’s expressive power limits Attend-and-Excite because the method applies no additional training.Out-of-distribution prompts may produce latents and images that do not correspond to the text.
  • Unnatural subject combinations may yield less realistic generated images.The paper attributes this to such combinations falling outside Stable Diffusion’s learned distribution for real images.
  • Complex object compositions remain additional challenges beyond the two semantic issues addressed.Examples include relations such as “riding on,” “in front of,” and “beneath.”
  • Attend-and-Excite has not been explored for negation, although the paper suggests demanding low attention for the negated subject as a possible approach.

7 CONCLUSIONS

The paper introduces Generative Semantic Nursing and its Attend-and-Excite formulation to manipulate diffusion latents during denoising so subject tokens receive attention. It reports on-the-fly mitigation of two core semantic issues without additional training, while describing token selection and runtime considerations.

  • 7 CONCLUSIONS: Generative Semantic Nursing manipulates latents during denoising to improve semantic faithfulness in a pre-trained text-to-image diffusion model.
  • 7 CONCLUSIONS: Attend-and-Excite encourages every subject token to be attended to by some image patch during generation.
  • 7 CONCLUSIONS: Users can select tokens to strengthen, including nouns, background settings, or other prompt tokens.
  • 7 CONCLUSIONS: Multi-token subjects may require identifying the dominant token from attention maps before applying Attend-and-Excite.
  • 7 CONCLUSIONS: Runtime increases from approximately 5.6 seconds for Stable Diffusion to approximately 9.7 seconds without iterative refinement and ∼15.4 seconds for challenging prompts with refinement.

A.4 Quantitative Evaluations

The quantitative evaluation uses constructed subsets of animals, objects, and colors, with CLIP- and BLIP-based processing choices documented for reproducible comparisons. Cross-attention visualizations examine subject-token behavior with and without Gaussian smoothing.

  • Datasets: Three evaluation subsets use animals, objects, and colors to construct prompts for quantitative experiments.The prompt lists are intended to facilitate future comparisons and evaluations.
  • CLIP-based Metrics: CLIP ViT-B/16 measures embedding-based similarities, while BLIP generates captions for the evaluated images.
  • CLIP-based Metrics: Input prompts are embedded by averaging CLIP embeddings across 80 prompt templates.
  • CLIP-based Metrics: Aggregated metrics omit seeds whose method returned black images because NSFW content was discovered and removed.
  • Qualitative Analysis: Cross-attention visualizations compare Attend-and-Excite with and without Gaussian smoothing for prompts involving a rabbit with a crown and a dog and squirrel.

B ABLATION STUDY

The ablation study finds that Gaussian smoothing and iterative latent refinement jointly improve semantic faithfulness and image quality. Refinement is selectively useful, while modifying late denoising steps adds artifacts without significant semantic change.

  • Component Ablations: Removing iterative refinement can eliminate a prompted subject, such as the crown in the first ablation row.
  • Component Ablations: Removing Gaussian smoothing can eliminate a prompted subject, such as the gray backpack in the third ablation row.
  • Component Ablations: Applying both Gaussian smoothing and iterative latent refinement produces more semantically faithful, higher-quality generations across three subsets.
  • Component Ablations: Iterative refinement has little influence when Stable Diffusion already generates both subjects successfully.
  • Experimental Design: Ablation comparisons use the same three seeds across variants, while qualitative comparisons use shared seeds for Prompt-to-Prompt and Attend-and-Excite.
  • Early Stopping: Stopping latent modification after 25 denoising steps avoids late updates that add artifacts without significant semantic change.

C.1 Comparison to Image Editing Methods

The section compares Attend-and-Excite with Prompt-to-Prompt attention re-weighting and evaluates both semantic faithfulness and generation quality across prompt subsets.

  • Comparison to Prompt-to-Prompt: Prompt-to-Prompt scales selected tokens’ spatial attention maps, whereas Attend-and-Excite targets neglected subject tokens during generation.Prompt-to-Prompt uses fixed token-specific scaling; the proposed method addresses whether every subject receives attention.
  • Qualitative comparison: Prompt-to-Prompt often leaves subjects neglected and attributes incorrectly bound, while Attend-and-Excite produces more semantically accurate results under matched seeds and prompts.Examples include a missing clock and a green bowl replacing the requested yellow bowl.
  • Quantitative comparison: At least 5% higher Minimum Object Similarity and at least 3% higher BLIP-based text-text similarity were reported for Attend-and-Excite over Prompt-to-Prompt.The CLIP-based comparison spans Animal-Animal, Object-Object, and Animal-Object subsets; the BLIP comparison covers all three subsets.
  • Evaluation design: The evaluation uses conjunctions of two subjects so relevant baselines can be compared and neglect can be analyzed separately for each subject.The prompt design supports per-subject analysis of semantic neglect.
  • Evaluation design: Additional experiments evaluate complex prompts with three or more subjects, challenging attribute bindings, and background subjects using 40 prompts and 64 shared random seeds.The comparisons include StructureDiffusion and Stable Diffusion.

C.3 Evaluation of Generation Quality

The authors evaluate whether Attend-and-Excite preserves image quality and the quality of subjects that were not neglected while strengthening neglected subjects.

  • Perceived image quality: Early stopping at denoising step 25 helps maintain image quality by restricting intervention to early denoising steps.Final timesteps proceed without intervention.
  • Subject quality: Maximum Object Similarity evaluates whether strengthening neglected subjects harms the quality of non-neglected subjects.The metric measures CLIP similarity for the least neglected subject independently of the full text.
  • Subject quality: Stable Diffusion can receive a slightly higher Maximum Object Similarity because neglect leaves only one subject to match against its sub-prompt.The authors expect this difference when Attend-and-Excite generates both requested subjects.
  • Subject quality: The Maximum Object Similarity gap is described as small relative to the improvement in Minimum Object Similarity.This comparison motivates assessing both neglected and non-neglected subjects.

C.4 Additional Qualitative Results

Additional experiments provide uncurated comparisons across prompts, baselines, styles, and subject combinations, while visualizations inspect attention changes and failure modes.

  • Additional Stable Diffusion results: Figures 15 and 16 show uncurated Stable Diffusion results before and after Attend-and-Excite using eight seeds without cherry-picking.The comparisons use the same generation setup across conditions.
  • Additional Stable Diffusion results: Figures 17 and 18 extend Stable Diffusion comparisons to complex prompts, styles, uncommon subject-setting combinations, and complex attributes.The optimized subject tokens are highlighted in the corresponding comparisons.
  • Baseline comparisons: Figure 19 adds comparisons on prompts from the StructureDiffusion paper using the same seeds.These results broaden the qualitative prompt set beyond the paper’s primary examples.
  • Baseline comparisons: Composable Diffusion can achieve high CLIP image-text similarity through subject mixture, producing a single hybrid object containing multiple input subjects.Additional qualitative results illustrate this failure mode.
  • Attention visualization: Figure 24 visualizes cross-attention maps for subject tokens before and after Attend-and-Excite and supports using attention as an explanation.The visualization compares the attention behavior associated with the subject tokens.
Loading 2301.13826v2…