PaperlayerView exact revision on arXiv

Source-linked AI summary

Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment

Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, Gal Chechik

arXiv:2306.08877v3cs.CLcs.CV

TL;DR

Text-conditioned image generators can incorrectly associate attributes with entities, motivating SynGen’s syntax-guided inference-time intervention. SynGen aligns cross-attention maps for related modifiers and nouns while separating unrelated words, and human evaluations report improvements across three datasets, including a challenging new set.

  • Problem

    Text-conditioned diffusion models often produce images where modifiers fail to influence the visual attributes of their grammatically related entity-nouns.

  • Method

    SynGen parses prompts for entity-noun–modifier bindings and optimizes a cross-attention loss during inference to align related maps and separate unrelated ones.

  • Results

    Over 100% improvement across all three datasets over the previous state of the art was reported for SynGen.

  • Takeaways & Limitations

    The results highlight that incorporating linguistic information during inference can improve faithfulness in text-conditioned image generation.

  • Takeaways & Limitations

    Effectiveness depends on parser quality, and performance degrades as the number of depicted attributes increases.

Abstract

from arXiv · show

Text-conditioned image generation models often generate incorrect associations between entities and their visual attributes. This reflects an impaired mapping between linguistic binding of entities and modifiers in the prompt and visual binding of the corresponding elements in the generated image. As one notable example, a query like "a pink sunflower and a yellow flamingo" may incorrectly produce an image of a yellow sunflower and a pink flamingo. To remedy this issue, we propose SynGen, an approach which first syntactically analyses the prompt to identify entities and their modifiers, and then uses a novel loss function that encourages the cross-attention maps to agree with the linguistic binding reflected by the syntax. Specifically, we encourage large overlap between attention maps of entities and their modifiers, and small overlap with other entities and modifier words. The loss is optimized during inference, without retraining or fine-tuning the model. Human evaluation on three datasets, including one new and challenging set, demonstrate significant improvements of SynGen compared with current state of the art methods. This work highlights how making use of sentence structure during inference can efficiently and substantially improve the faithfulness of text-to-image generation.

1 Introduction

Text-conditioned diffusion models can misbind modifiers to entity-nouns, reducing faithfulness to prompts. SynGen uses syntactic information and cross-attention alignment during inference, improving binding across three evaluated datasets.

  • 1 Introduction: Modifier-entity binding failures cause generated attributes to mismatch the nouns they grammatically modify.Examples include swapping pink and yellow attributes between a sunflower and flamingo, leaking attributes to unmentioned regions, or neglecting attributes.
  • 1 Introduction: SynGen steers cross-attention maps so visual bindings follow the linguistic bindings identified in the prompt.The method targets the correspondence between prompt terms and image pixels associated with those terms.
  • 1 Introduction: The method introduces a loss that brings grammatically related modifier-noun attention maps together while separating them from unrelated words.It intervenes in the latent code during generation without compromising generated-image quality, according to the authors.
  • 1 Introduction: Human evaluation on three datasets found significant performance improvements, sometimes doubling accuracy.The datasets include natural compositional prompts, replicated prior-work prompts, and a new challenging set with varied modifiers and entity-nouns.

2 Syntax-Guided Generation

SynGen extracts syntactic bindings between entity-nouns and modifiers, then uses attention-map losses to guide diffusion denoising toward those bindings. Its intervention aligns related maps, separates unrelated maps, and updates latents during inference.

  • 2 Syntax-Guided Generation: SynGen parses prompts to identify top-level entity-nouns and recursively collect their syntactic modifiers.The modifier set includes adjectival modification, compounds, nominal and adverbial modification, adjectival complements, and modifier coordination.
  • 2.2 Controlling generation with language-driven cross-attention losses: Attention maps initially overlap across entities and modifiers, then separate according to syntactic bindings during denoising and remain separated after intervention stops.Figure 3 illustrates this evolution for “a red crown and a golden strawberry.”
  • 2.2 Controlling generation with language-driven cross-attention losses: The positive loss minimizes attention-map distance between each modifier and its corresponding entity-noun.The intended effect is greater spatial overlap between maps for linguistically related words.
  • 2.2 Controlling generation with language-driven cross-attention losses: The negative loss separates attention maps for modifier-noun pairs from words that are grammatically unrelated to those pairs.Unmatched words are defined by excluding the words in each noun-modifier set from the full prompt.
  • 2.2 Controlling generation with language-driven cross-attention losses: The combined loss uses symmetric Kullback-Leibler divergence to measure distance between normalized attention maps.The final objective combines the positive and negative terms.
  • 2.2 Controlling generation with language-driven cross-attention losses: Unlike an earlier attention-based test-time method, SynGen optimizes pairwise relations among linguistically related words rather than maximizing the smallest selected-token map.This distinction aligns the diffusion process with the prompt’s linguistic structure.
  • 2.2 Controlling generation with language-driven cross-attention losses: During denoising, SynGen updates the latent representation with gradient steps that reduce the attention-map loss.The method intervenes during the first 25 of 50 denoising steps; fewer steps corrected binding less effectively, while more steps produced blurred images.

3 Experiments

The experiments evaluate SynGen across three datasets, including a new challenging benchmark, using human judgments of concept separation and visual appeal alongside fine-grained binding metrics. The evaluation also documents dataset construction choices and notes that the automatic metric has limited quality.

  • Datasets: ABC-6K contains natural compositional prompts with at least two color words modifying different entities, plus counterparts with swapped modifier positions.The evaluation randomly samples 600 prompts from this benchmark.
  • Datasets: DVMP challenges binding with up to three modifiers per entity and includes colors, material patterns, design patterns, and noun modifiers.Its modifiers are selected to be visually verifiable and semantically coherent.
  • Human Evaluation: Human evaluation presents one prompt with four generated images to three raters, who select the best image or indicate equal quality; majority decisions are reported.The evaluation measures concept separation by prompt matching and visual appeal, with image order randomized to reduce bias.
  • Fine-grained Evaluation: Fine-grained evaluation measures Proper Binding, Improper Binding, and Entity Neglect from human judgments of attribute-object mappings and entity depiction.Proper and Improper Binding are ratios over attributes, while Entity Neglect is the complement of the depicted-entity ratio.

4 Results

SynGen achieves the strongest reported quantitative results across three datasets and improves fine-grained binding, while qualitative analyses show that baselines exhibit multiple attribute and entity-related failure modes.

  • Quantitative Results: SynGen ranks first across all three datasets, sometimes achieving double the approval rate of the second-ranked A&E baseline.The comparison covers concept separation and visual appeal, with concept separation directly measuring semantic leakage.
  • Fine-grained Evaluation: SynGen outperforms all models by a landslide on proper and improper binding and matches state-of-the-art performance on entity neglect.Entity neglect is not directly targeted by SynGen.
  • Qualitative Analysis: Baselines show semantic leakage when modifiers transfer between prompt entities or spread to unmentioned image regions.Examples include pink color transferring from a clock to a brown chair and attributes appearing in backgrounds or on unmentioned plants.
  • Qualitative Analysis: Baselines also neglect attributes, cast noun modifiers as standalone entities, and misinterpret words such as “baby” and “fire” as separate objects.SynGen correctly binds “baby” to rabbit in the cited example.
  • Qualitative Analysis: Other methods exhibit entity entanglement by assigning objects their typical attributes instead of the attributes specified in the prompt.For example, baselines generate a black-and-white dog rather than associating blue with the dog.
  • Ablation: The ablation study identifies complementary roles for positive and negative loss terms in controlling attribution and leakage errors.Removing either term produces semantic leakage, while removing the positive term also causes failed attribution binding.

5 Related Work

Prior work addresses prompt-to-image relations, entity omission, semantic leakage, and syntax-based control, but SynGen focuses specifically on modifier–entity binding through attention-map optimization.

  • Semantic Leakage: Earlier methods improve relations between prompt terms and visual components but do not specifically target modifier–entity binding.SynGen formulates distances between attention patterns of grammatically related and unrelated words.
  • Attention-based Interventions: Attention-based interventions motivated SynGen because cross-attention determines entity spatial layout and is implicated in entity omission and semantic failures.Attend-and-Excite targets entity omission by encouraging noun tokens to attend to corresponding image regions.
  • Syntax-based Generation: Structured Diffusion parses noun and modifier phrases, encodes them separately, and intervenes on attention patterns to address missing entities and semantic leakage.The paper reports that implicitly influencing attention is preferable in its experiments.
  • Concurrent Work: A concurrent approach trains a layout predictor to associate noun-related pixels with CLIP phrase representations, whereas SynGen intervenes during generation without that predictor.The cited comparison characterizes the approaches as similar in spirit but differing in their intervention mechanism.

6 Limitations

SynGen remains constrained by prompt complexity, parser quality, and generation speed, although its degradation with more attributes is less pronounced than for other methods.

  • Attribute Complexity: SynGen performance degrades as the number of depicted attributes increases, although the decline is less pronounced than for previous methods.The paper attributes this partly to excessive negative loss and difficulty forming a cohesive narrative for cluttered images.
  • Parser Dependence: When the parser fails to extract the required syntactic relations, SynGen operates essentially like Stable Diffusion.The method’s effectiveness is therefore intrinsically tied to parser quality.
  • Efficiency: SynGen takes longer than Stable Diffusion and is slightly slower than Attend-and-Excite when prompts contain modifiers.The timing comparison is reported in Appendix A.

7 Conclusions

SynGen addresses improper binding by aligning syntax-related entities and modifiers in cross-attention while separating unrelated words, and improves performance across three datasets.

  • Conclusion: SynGen targets improper binding, where objects and attributes are incorrectly matched to entity-nouns and modifiers.The method uses an inference-time intervention rather than retraining the diffusion model.
  • Conclusion: Its loss encourages overlapping cross-attention maps for syntax-related modifiers and entity-nouns while discouraging overlap among unrelated prompt words.The evaluation includes DVMP, a challenge dataset designed to expose difficult improper-binding cases.
  • Conclusion: SynGen demonstrates improvement of over 100% across all three datasets compared with the previous state of the art.The conclusion links the results to using linguistic structure during denoising for faithful text-to-image generation.

A Implementation Details

SynGen uses inference-time latent updates guided by cross-attention maps, with documented runtime, model, parser, and hyperparameter choices.

  • Efficiency: SynGen takes 9.76 seconds per image, compared with 4 seconds for Stable Diffusion and 8.8 seconds for Attend-and-Excite.The timing used 50 randomly sampled images from the A&E, DVMP, and ABC-6K datasets.
  • Hyperparameters: Experiments used 50 diffusion steps, guidance scale 7.5, scale-factor 20, and 25 latent update steps.
  • Parser: The implementation uses spaCy with the out-of-the-box en_core_web_trf model for syntactic parsing.
  • Cross-attention maps: Cross-attention maps connect D2 latent patches to N prompt tokens, indicating which tokens are relevant for generating each patch.
  • Cross-attention maps: SynGen represents each token’s attention over patches by transposing and row-normalizing the patch-to-token attention matrix.

B.1 Further Investigation of the Positive and Negative Loss Terms

Ablations and parameter studies show that SynGen’s positive and negative loss terms complement each other, while intervention strength and duration require balancing separation against image quality.

  • Loss terms: Both positive and negative loss terms are needed for proper binding: removing the negative term increases missing objects and apparent improper binding.
  • Loss terms: Removing the positive term produces fuzzier object separation, incomplete objects, unfulfilled attributes, and semantic leakage.
  • Loss terms: The two loss terms have complementary effects on object count, binding, and image coherence.
  • Intervention steps: Intervening for too few timesteps leaves semantic leakage or incomplete images, whereas all 50 timesteps increase blurred images.
  • Scale factor: Greater scale-factor values strengthen concept separation only up to a point, after which generations become blurred or lose visual appeal.
  • Prompt complexity: As modifier count increases, the gap between SynGen and baselines widens, although repeated modifiers reduce performance across all methods.

D.1 Qualitative analysis by number of modifiers

Additional analyses examine modifier-count, repetition, entanglement, baseline, and failure-case behavior, revealing stronger comparative gains with more modifiers alongside recurring generation failures.

  • Modifier count: In the DVMP challenge set, prompts contain 2 to 6 modifiers, and error frequency tends to rise as modifier count increases.
  • Modifier count: SynGen’s semantic leakage does not increase faster than baselines as modifier count grows, but it tends to generate too many entities.
  • Baseline comparison: Spatial-Temporal Diffusion consistently misses at least one prompted entity in the reported 18-image comparison and also exhibits semantic leakage.
  • Qualitative analyses: SynGen’s qualitative comparisons include prompts with recurring terms, entangled entities, and more than two modifiers.
  • SynGen failures: With many modifiers and entities, an excessively large negative loss can push latents outside the decoder’s learned distribution, producing blurred or incoherent images.
  • SynGen failures: SynGen can omit objects, merge entities, or omit attributes, especially when many modifiers refer to the same entity.
  • SynGen failures: SynGen may generate multiple instances of an entity instead of the specified count, such as multiple strawberries for “a strawberry.”

G.1 Additional Details on Human Evaluation Experiments

The evaluation combines crowdsourced human comparisons, standardized datasets and prompt categories, and an automatic phrase-to-image metric whose agreement with human judgments is limited.

  • Human evaluation: Human evaluators compared image generations using multiple criteria, with full results reported in Table 5.
  • Human evaluation: Raters needed at least 5,000 approved HITs, an approval rate above 98%, and a perfect qualification exam; compensation was $10 hourly.
  • Datasets: The DVMP dataset lists entities and modifiers, including 13 colors, across objects, animals, and fruit categories.
  • Evaluation criteria: The concept-separation criterion asks whether an image matches the prompt, while visual appeal is evaluated separately.
  • Automatic evaluation: CLIP-based whole-prompt similarity can score swapped attributes similarly to correctly bound attributes because it fails to encode syntactic bindings.
  • Automatic evaluation: The phrase-segmented CLIP metric agrees with majority human selection 43.5% of the time, versus 25% for random selection.
Loading 2306.08877v3…