PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis

Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, William Yang Wang

arXiv:2212.05032v3cs.CVcs.CL

TL;DR

Existing text-to-image diffusion models still struggle to bind attributes to the correct objects and compose multiple objects coherently. StructureDiffusion incorporates parsed language structures into cross-attention guidance without training samples, improving attribute binding and compositionality while maintaining image quality and diversity.

  • Problem

    Text-to-image models still struggle to bind attributes correctly and compose multiple objects coherently in complex scenes.

  • Method

    StructureDiffusion parses hierarchical language structures and uses separately encoded text spans to guide cross-attention representations without additional training samples.

  • Results

    The method achieves more accurate attribute binding and compositionality while maintaining overall image quality and diversity.

  • Takeaways & Limitations

    Structured language representations provide an efficient way to improve compositional text-to-image generation using frozen diffusion models.

  • Takeaways & Limitations

    Cross-attention computation costs increase with the number of noun phrases.

Abstract

from arXiv · show

Large-scale diffusion models have achieved state-of-the-art results on text-to-image synthesis (T2I) tasks. Despite their ability to generate high-quality yet creative images, we observe that attribution-binding and compositional capabilities are still considered major challenging issues, especially when involving multiple objects. In this work, we improve the compositional skills of T2I models, specifically more accurate attribute binding and better image compositions. To do this, we incorporate linguistic structures with the diffusion guidance process based on the controllable properties of manipulating cross-attention layers in diffusion-based T2I models. We observe that keys and values in cross-attention layers have strong semantic meanings associated with object layouts and content. Therefore, we can better preserve the compositional semantics in the generated image by manipulating the cross-attention representations based on linguistic insights. Built upon Stable Diffusion, a SOTA T2I model, our structured cross-attention design is efficient that requires no additional training samples. We achieve better compositional skills in qualitative and quantitative results, leading to a 5-8% advantage in head-to-head user comparison studies. Lastly, we conduct an in-depth analysis to reveal potential causes of incorrect image compositions and justify the properties of cross-attention layers in the generation process.

1 INTRODUCTION

The paper targets compositional failures in text-to-image synthesis, especially incorrect attribute binding and missing or confused objects. It proposes efficient, training-free structured cross-attention guidance using linguistic prompt structures, and evaluates improvements with a new benchmark and extensive analysis.

  • Motivation: Composing multiple objects coherently requires T2I models to understand both the full prompt and its individual linguistic concepts.The model must combine multiple concepts while generating all visual concepts specified by the text.
  • Problem: Attribute binding remains a fundamental problem, with models potentially swapping attributes between objects in prompts such as “a brown bench in front of a white building.”Such failures can produce a white bench and a brown building, possibly because of training-set bias or imprecise language understanding.
  • Method: The method uses free attribute-object relation pairs from sentence parse trees to provide a data-efficient alternative for improving compositionality.Structured prompt representations are combined with diffusion guidance rather than relying on additional training data.
  • Method: Structured cross-attention guidance extracts hierarchical text spans and separately encodes entities and visual concepts to disentangle attribute-object pairs.This replaces reliance on a single sequence of text embeddings for guidance and is developed in Stable Diffusion.
  • Contributions: The proposed approach is training-free, requires no additional training samples, improves attribute binding and compositionality, and introduces the Attribute Binding Contrast set (ABC-6K) benchmark.Extensive experiments and analysis also investigate causes of incorrect attribute binding and identify directions for improving faithfulness and compositionality.

2 DIFFUSION MODELS & STRUCTURED GUIDANCE

The section introduces StructureDiffusion, a training-free method that incorporates hierarchical language structures into Stable Diffusion’s cross-attention layers. It uses full-prompt attention maps for layouts and structured concept representations for semantics, while addressing object omission in conjunctions.

  • Stable Diffusion: Stable Diffusion uses an autoencoder and diffusion model to generate images by decoding lower-resolution latent maps from Gaussian-noise inputs.The autoencoder encodes images into latent maps for diffusion training and decodes diffusion outputs during inference.
  • CLIP Text Encoder: CLIP embedding contextualization can cause incorrect attribute binding because later tokens blend with preceding token semantics under causal attention masks.This contextualization is identified as a potential source of binding errors in prompts containing multiple objects.
  • Cross Attention Controls: Cross-attention maps control generated-image layout and structure, whereas value vectors carry semantics into attended regions, enabling separate control of composition and content.The method builds on this presumed disentanglement between attention maps and values.
  • Structured Guidance: The method extracts hierarchical noun phrases or scene-graph concepts, encodes each separately with frozen CLIP, and combines their value projections with full-prompt attention maps.Concept embeddings are realigned with the prompt, projected into values, and multiplied by the prompt-derived attention maps to form a conjunction of concepts.
  • Structured Guidance: The guidance requires no training or additional data and preserves layout because attention maps remain computed from full-prompt queries and keys.A variant additionally computes concept-specific attention maps to address Stable Diffusion’s tendency to omit objects in conjunctions.

3 EXPERIMENT

The experiments evaluate StructureDiffusion on attribute-binding, concept-conjunction, and general MSCOCO prompts using human and automatic measures. It improves compositional alignment and fidelity while preserving general image quality and diversity.

  • Datasets: ABC-6K contains 6.4K MSCOCO captions, or 3.2K contrastive pairs, with at least two color words modifying different objects.CC-500 adds challenging concept-conjunction prompts, while 10K randomly sampled MSCOCO captions test generalization.
  • Evaluation Metrics: Human evaluations compare Stable Diffusion and StructureDiffusion for text-image alignment and image fidelity, supplemented by automatic composition evaluation.For fidelity, annotators judge which image is more realistic and natural regardless of the text.
  • ABC-6K Results: 42% of ABC-6K comparisons favored StructureDiffusion, 7% higher than losses, while 22% of comparisons were tied.Qualitative gains include fewer missing objects, better-matched colors, and improved attributes or details.
  • General Prompts: StructureDiffusion achieved 40.9 IS, 17.9 FID and 72.3 R-Precision on general MSCOCO prompts, compared with Stable Diffusion’s 39.9 IS, 18.0 FID and 72.2 R-Precision.The comparable scores indicate maintained image fidelity and diversity for general prompts.

4 ABLATION STUDY

The ablation study shows that token realignment is important for preserving image quality, whereas StructureDiffusion’s computational cost grows with the number of noun phrases. It also motivates reducing potentially unnecessary padding-token computation by studying alternative embedding patterns.

  • Token realignment: ∼2 IS / FID degradation (37.5 IS, 19.8 FID) occurs when noun-phrase spans are naively expanded instead of realigned, relative to re-alignment or Stable Diffusion.Realignment ensures each token’s value vector corresponds to the correct attention map.
  • Computational cost: StructureDiffusion’s cross-attention computation cost increases with the number of noun phrases.The limitation arises because attention computation is performed for each noun phrase.
  • Padding embeddings: Because Stable Diffusion pads CLIP text-encoder sequences to 77 tokens, the study examines four patterns of token embeddings to assess whether all padding tokens are necessary.The authors note that most attention maps are computed from padding embeddings.

5 RELATED WORK

This section reviews diffusion-based text-to-image synthesis, compositional generation methods, and diffusion guidance techniques. It highlights latent diffusion and Stable Diffusion, layout or scene-graph conditioning, and classifier-free guidance with cross-attention properties.

  • Text-to-Image Synthesis: Diffusion models generate high-quality images with a more stable training process, while latent diffusion generates continuous feature maps in an autoencoder’s latent space.Stable Diffusion uses a similar architecture and is trained on large-scale image-text datasets with a fixed CLIP text encoder.
  • Compositional Generation: Compositional generation aims to help text-to-image models understand and disentangle basic concepts, using layouts or scene graphs to strengthen weak text conditions.Prior work also explores controllable generation and concept conjunctions.
  • Diffusion Guidance: Classifier-free guidance jointly trains a diffusion model on conditional and unconditional inputs, enabling flexible and improved conditional synthesis in many large-scale models.Related work also identifies unique properties of cross-attention maps in Imagen.

6 CONCLUSION

The paper proposes a training-free method for compositional text-to-image generation that addresses diffusion models’ difficulty with compositional synthesis and incorrect attribute binding. Its approach incorporates language structures into cross-attention through structured diffusion guidance.

  • The method targets compositional image-synthesis failures in existing large-scale text-to-image diffusion models, especially incorrect object–attribute binding.
  • Structured diffusion guidance incorporates language structures into the cross-attention layers of the generation process.
  • The approach proposes two simple techniques to align structured encoding with cross-attention.

ETHICAL STATEMENT · A RELATED WORK

The paper reports compensated English-language Mechanical Turk evaluation and situates its approach within text-to-image synthesis, structured vision-language representations, and diffusion guidance. Related work spans GAN-based, autoregressive, and diffusion models, alongside methods using shared structures or guidance signals across modalities.

  • ETHICAL STATEMENT: ETHICAL STATEMENT: Evaluation uses Amazon Mechanical Turk comparison tasks organized as batches of HITs.Workers were selected from English-speaking countries because the task required understanding English prompts.
  • ETHICAL STATEMENT: ETHICAL STATEMENT: Each HIT takes approximately 15–30 seconds, costs $0.15, and yields an hourly payment of $18.These figures describe compensation for each submitted HIT.
  • A RELATED WORK: A RELATED WORK: Text-to-image synthesis research includes GAN-based, autoregressive, and diffusion models.The passage cites representative work across all three model families.
  • A RELATED WORK: A RELATED WORK: XMC-GAN is a one-stage GAN using contrastive losses across image-image, image-text, and region-token pairs.Zhang et al. (2021) proposes this multi-loss design.
  • A RELATED WORK: A RELATED WORK: Vision-language research seeks shared structures between language and vision to unify the modalities.Prior work includes semantic parsing, visual-semantic embedding, and joint-structure learning approaches.
  • A RELATED WORK: A RELATED WORK: Semantic parsing structures have been used to facilitate embedding learning in visual-semantic frameworks.Wu et al. (2019) incorporates semantic-parsing structure into visual-semantic embedding.
  • A RELATED WORK: A RELATED WORK: Diffusion guidance can use classifier gradients to convert an unconditional diffusion model into a class-conditional one.Dhariwal & Nichol (2021) back-propagate classification loss into image space.
  • A RELATED WORK: A RELATED WORK: Text-to-image diffusion guidance has also used noisy CLIP similarity to measure alignment with text.Liu et al. (2021b) and Nichol et al. (2021) apply noisy CLIP models using cosine similarity.

B IMPLEMENTATION DETAILS

The experiments build on Stable Diffusion v1.4 with controlled sampling and guidance settings, while avoiding hand-crafted prompt additions and using the Stanza Library.

  • The method is implemented on Stable Diffusion v1.4 throughout the experiments.
  • Comparisons use the same initial Gaussian map, 50 diffusion steps, and PLMS sampling by fixing the seed.
  • The guidance scale is fixed at 7.5, and cross-attention key-value matrices are equally weighted unless specified otherwise.
  • The text input excludes hand-crafted additions such as “a photo of,” and the experiments use the Stanza Library.

C VISUALIZATION OF ATTENTION MAPS · D ABLATION STUDY

Visualization of cross-attention maps supports the method’s assumptions about stable spatial attention distributions and semantically meaningful values. The method also accelerates interpretable attention formation and reduces attribute color leakage through structured prompt parsing and alignment.

  • C VISUALIZATION OF ATTENTION MAPS: Cross-attention maps from Stable Diffusion and the proposed method show similar spatial distributions and highlights throughout diffusion.This supports the assumption that the attention map M_t remains unchanged despite multiple values in each cross-attention layer.
  • C VISUALIZATION OF ATTENTION MAPS: The proposed method accelerates interpretable attention formation for both “green” and “clock” tokens.Figure 9 tracks these token maps across diffusion timestamps from step 50 to step 1.
  • C VISUALIZATION OF ATTENTION MAPS: Values in cross-attention maps represent rich textual semantics mapped into image-space contents.The visualizations justify the claim that values carry content-related semantics.
  • C VISUALIZATION OF ATTENTION MAPS: The method separately parses and aligns “A long narrow yellow kitchen” and “black and white floor tiles” to enhance their attribute semantics.This structured encoding forms V and strengthens the meanings of “yellow” and “black and white” separately.
  • C VISUALIZATION OF ATTENTION MAPS: Separately enhancing “yellow” and “black and white” mitigates yellow blending into black-and-white regions.The passage attributes the disappearance of color leakage to these operations.
  • C VISUALIZATION OF ATTENTION MAPS: Using multiple Keys, the proposed method successfully rectifies image layouts for the token “sheep.”Figure 10 compares attention maps for the sheep token across different methods.

D.1 A CASE STUDY OF ATTRIBUTE BINDING · D.2 COMPARISON OF PARSERS

The case study identifies contextualized token embeddings and inaccurate cross-attention maps as two causes of incorrect attribute binding. Parser comparison shows scene graphs extract more spans than constituency parsing for Examples 1–4, while both agree for Example 0.

  • D.1 A CASE STUDY OF ATTRIBUTE BINDING: Two root causes of incorrect attribute binding are contextualized token embeddings caused by causal attention masks and inaccurate attention maps.The case study examines both mechanisms using prompt encoding and cross-attention visualizations.
  • D.1 A CASE STUDY OF ATTRIBUTE BINDING: The visualized attention maps show bird semantics mistakenly attending to the bear’s mouth, producing an unnatural beak-like shape.Figure 11 presents this as a potential root cause of incorrect attribute binding.
  • D.1 A CASE STUDY OF ATTRIBUTE BINDING: Replacing “red” with its embedding from “a green bag and a red apple” produces an unnatural solid green region in the generated apple image.The experiment uses the same Gaussian noise initialization and attributes the contamination to semantics from the preceding “green” token.
  • D.1 A CASE STUDY OF ATTRIBUTE BINDING: Cross-attention maps for “bird” highlight the bird’s bottom-left location but also show saliency around the bear’s mouth.The maps are averaged across attention heads from both downsample and upsampling blocks.
  • D.2 COMPARISON OF PARSERS: The parser comparison evaluates constituency parsing and scene graph parsing for extracting text spans and generating images.Table 3 compares the spans extracted by the two parsers.
  • D.2 COMPARISON OF PARSERS: Example 0 yields the same extracted spans from both parsers for CC-500 prompts, whereas Examples 1–4 yield more spans with the scene graph parser.The passage notes that concepts in the middle of the sentence appear more often in these spans.

E LIMITATIONS & FUTURE WORK · F ADDITIONAL RESULTS

The method is limited by imperfect external parsing, ungroundable style descriptions, and evaluation filtering that may overstate improvements. Additional results show performance across datasets, attributes beyond color, and style variations without negative style effects.

  • E LIMITATIONS & FUTURE WORK: The method depends on an imperfect external constituency parser, currently Stanza, which could be replaced by a more advanced learning-based parser.This limitation concerns the parsing function used by the method.
  • E LIMITATIONS & FUTURE WORK: Style descriptions such as “in Van Gogh style” may be parsed as separate noun phrases that cannot be grounded in image space.The method mainly focuses on compositional text-to-image synthesis and does not fully address style descriptions.
  • E LIMITATIONS & FUTURE WORK: For CC-500 prompts, constituency and scene graph parsers produce the same results, whereas general prompts yield more scene-graph text spans with repeated middle concepts.The parser comparison identifies different span-generation behavior on general prompts.
  • E LIMITATIONS & FUTURE WORK: 20% of the most similar image pairs were filtered from Table 1 for evaluation efficiency, so improvements may be compromised on the full generated-image set.StructureDiffusion tends to generate images similar to Stable Diffusion, motivating the filtering.
  • F ADDITIONAL RESULTS: Qualitative results cover CC-500, ABC-6K, and attributes beyond colors, including shape, size, and materials.These results are presented in Figures 13–15.
  • F ADDITIONAL RESULTS: Across combinations of style descriptions appended to “an astronaut riding a horse,” the method has no negative effects on image style relative to Stable Diffusion.In Figure 16, “base” refers to Stable Diffusion.
Loading 2212.05032v3…