PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models

Long Lian, Boyi Li, Adam Yala, Trevor Darrell

arXiv:2305.13655v3cs.CV

TL;DR

Complex prompts expose weaknesses in text-to-image diffusion models, especially for numeracy, spatial reasoning, and related prompt-understanding capabilities. LMD addresses this with a training-free two-stage pipeline that uses an LLM-generated layout and a layout-grounded diffusion controller, doubling average generation accuracy across four tasks while supporting iterative specification and unsupported prompt languages.

  • Problem

    Text-to-image diffusion models struggle to accurately follow complex prompts requiring capabilities such as numeracy, negation, attribute binding, and spatial reasoning.

  • Method

    LMD uses a training-free two-stage pipeline in which an LLM generates captioned layouts and a controller guides an off-the-shelf diffusion model using those layouts.

  • Results

    LMD outperforms its base diffusion model and several strong baselines, doubling average generation accuracy across four tasks.

  • Takeaways & Limitations

    LMD supports multi-round scene specification and generation from prompts in languages unsupported by the base diffusion model without additional training.

  • Takeaways & Limitations

    Because the models are used off-the-shelf, the LLM may generate layouts ambiguous to the diffusion model, including underspecified viewpoints.

Abstract

from arXiv · show

Recent advancements in text-to-image diffusion models have yielded impressive results in generating realistic and diverse images. However, these models still struggle with complex prompts, such as those that involve numeracy and spatial reasoning. This work proposes to enhance prompt understanding capabilities in diffusion models. Our method leverages a pretrained large language model (LLM) for grounded generation in a novel two-stage process. In the first stage, the LLM generates a scene layout that comprises captioned bounding boxes from a given prompt describing the desired image. In the second stage, a novel controller guides an off-the-shelf diffusion model for layout-grounded image generation. Both stages utilize existing pretrained models without additional model parameter optimization. Our method significantly outperforms the base diffusion model and several strong baselines in accurately generating images according to prompts that require various capabilities, doubling the generation accuracy across four tasks on average. Furthermore, our method enables instruction-based multi-round scene specification and can handle prompts in languages not supported by the underlying diffusion model. We anticipate that our method will unleash users' creativity by accurately following more complex prompts. Our code, demo, and benchmark are available at: https://llm-grounded-diffusion.github.io

1 Introduction

Text-to-image diffusion models struggle with complex prompts, while LMD addresses this through a training-free, two-stage LLM-grounded generation process and reports higher accuracy across four tasks.

  • Motivation: Diffusion models often misinterpret numeracy, negation, spatial relationships, and attribute binding in complex prompts.These limitations include failing to generate a specified number of objects or correctly associate attributes with objects.
  • Approach: LMD avoids training a new diffusion model by using a pretrained LLM to provide grounding for prompt understanding.The method is explicitly training-free and uses frozen pretrained models.
  • Approach: The first stage converts a user prompt into captioned bounding boxes, a background caption, and an optional negative prompt.The LLM is adapted as a text-grounded layout generator through in-context learning.
  • Approach: The second stage uses a novel controller to steer an existing diffusion model toward the LLM-generated instance-level layout.Unlike region-semantic control, the controller targets precise object instances in designated regions.
  • Capabilities: LMD supports multi-round scene specification and prompts in languages unsupported by the base diffusion model without additional training.Users can provide follow-up instructions and clarifications to update subsequent generation.
  • Results: LMD doubles average generation accuracy across four tasks and outperforms its base diffusion model and several recent baselines.The paper also introduces a benchmark for evaluating prompt understanding in text-to-image models.

2 Related Work

Prior work studies text-to-image generation, LLM-based visual grounding, spatial conditioning, and instruction-based editing; LMD combines these directions around instance-level layout grounding.

  • Text-to-image diffusion models: Text-to-image diffusion models produce high-quality images but remain limited on attribute binding and spatial reasoning.These challenges motivate methods that improve understanding of complex textual prompts.
  • LLMs for visual grounding: Existing LLM-based layout prediction can be constrained to a closed set of annotated object classes.The cited approach therefore cannot generate layouts for objects outside its training-set classes.
  • LLMs for visual grounding: LMD uses pretrained LLM knowledge to generate layouts through prompting with in-context examples rather than external annotations.The related-work passage states that high-quality layout generation is present in pretrained LLM weights.
  • Spatially-conditioned image generation: Spatially conditioned generation methods synthesize images from priors such as poses, segmentation maps, strokes, layouts, or scene graphs.The literature includes both pre-diffusion layout-to-image systems and diffusion-era controllers.
  • Spatially-conditioned image generation: Recent training-free region-control methods emphasize semantic regions, whereas LMD focuses on controlling individual object instances and their counts.The distinction concerns instance-level grounding rather than only region semantics.
  • Instruction-based editing: Instruction-based image-editing systems modify image content through external editing models, while LMD edits the scene layout instead of raw pixels.Layout editing supports instructions such as swapping or moving objects.

3 LLM-grounded Diffusion

LMD uses a two-stage, training-free pipeline: an LLM converts prompts into captioned layouts, and a controller guides an existing diffusion model to generate layout-grounded images. Instance-level masked latents and layout updates support precise object placement, multi-round scene specification, and prompts in unsupported languages.

  • Overview: LMD generates images in two stages: text-grounded layout generation followed by layout-grounded image generation.The method uses an LLM for the first stage and an existing diffusion model for the second.
  • 3.1 LLM-based Layout Generation: The LLM represents each foreground object with a captioned bounding box and adds background and optional negative prompts.Bounding-box coordinates use the (x, y, width, height) format, and each object instance receives a separate box.
  • 3.2 Layout-grounded Stable Diffusion: LMD addresses the instance indistinguishability that prevents prior regional methods from precisely controlling object counts within semantic regions.Processing one foreground box at a time enables precise placement and attribute binding for each instance.
  • 3.2 Layout-grounded Stable Diffusion: The controller generates masked latents for individual boxes and composes them as priors to guide diffusion toward the specified layout.Cross-attention is strengthened inside each box and attenuated outside it, while the composed latents guide overall generation.
  • 3.2 Layout-grounded Stable Diffusion: The controller conditions diffusion to generate one instance at each masked location while preserving natural foreground-background composition.The same initial noise latent is shared across boxes to encourage globally coherent viewpoint, style, and lighting.
  • 3.3 Additional Capabilities of LMD: LMD supports multi-round scene specification and prompts in languages unsupported by the underlying diffusion model without additional training.Updating layouts enables iterative object and attribute edits, while the LLM can convert non-English prompts into English layouts.

4 Evaluation

The evaluation compares LMD with diffusion and LLM-based baselines across qualitative and benchmark settings. LMD improves prompt-following accuracy through layout-grounded generation, while retaining gains across diffusion models and evaluation methods.

  • Qualitative Comparison: Both VisualChatGPT and GILL lack accurate prompt following for three of the four tasks LMD solves.
  • LMD qualitatively improves prompt following over Stable Diffusion by generating images aligned with LLM-produced layouts.
  • Proposed benchmark: LMD uses a benchmark covering negation, generative numeracy, attribute binding, and spatial reasoning, with 400 composed prompts evaluated by object detection.
  • Proposed benchmark: 1.3× to 3.6×: LMD improves generation accuracy over Stable Diffusion across four tasks, doubling accuracy on average.LMD exceeds twice Stable Diffusion accuracy on spatial relationships and negation; LMD+ adds further improvements with GLIGEN.
  • Ablation Study: The layout-grounded controller provides stronger instance-level grounding than training-free semantic methods and surpasses GLIGEN on generative numeracy, attribute binding, and spatial reasoning.Integrating GLIGEN as LMD+ yields further improvements without additional training.
  • Ablation Study: Around 2× performance boost: LMD maintains its gains when switching the base model from SDv1.5 to SDv2.1 without hyperparameter tuning.Table 3 also states that additional hyperparameter tuning could potentially improve performance further.
  • T2I-CompBench: LMD improves all four T2I-CompBench tasks when applied to either SDv1 or SDv2.This evaluation additionally uses VQA models for color, shape, and texture, and UniDet for spatial evaluation.
  • 88.18% vs 10.90% for SD: LMD+ achieves higher accuracy on the first question, while scoring 35.45% vs 31.81% for SD on the second.The results indicate prompt alignment without degradation of naturalness or coherency.

5 Discussions

Because the pipeline uses off-the-shelf models, LLM-generated layouts can be ambiguous to the diffusion model. Iterative user feedback can correct such ambiguities, but the method also inherits base-model biases and may benefit more from in-context objects.

  • Off-the-shelf models can produce LLM layouts whose viewpoint assumptions are ambiguous to the diffusion model.A top-down layout may yield a side-view image, causing objects such as apples to miss their intended support.
  • An additional request specifying viewpoint and object sizes enables the LLM to adjust the subsequent layout and improve prompt alignment in a second round.
  • LMD inherits biases from its base diffusion model, and in-context examples may produce better layouts for objects they explicitly mention.

6 Summary

LMD enhances text-to-image prompt understanding with a training-free, two-stage process that combines LLM-generated layouts with layout-grounded image generation. It uses pretrained components to guide an existing diffusion model without parameter optimization.

  • LMD uses a training-free two-stage generation process to improve text-to-image diffusion models' prompt understanding.The method combines LLM-based text-grounded layout generation with layout-grounded image generation.
  • LMD's reported results show stronger prompt following than the base model and strong baselines, while also supporting broader language and multi-round scene specification.These capabilities are presented as extensions of the prompt-understanding method.
  • The layout-grounded generation pipeline operates in latent space, repeatedly denoising latent variables before decoding the final latent into an image.The underlying framework encodes images into latents, applies diffusion and reverse denoising, then decodes z0.
  • Stable Diffusion conditions denoising through text features passed into cross-attention layers of the diffusion U-Net.The text encoder produces token features that affect the U-Net output through cross-attention.

C Additional features and use cases from instruction-based scene specification

Instruction-based scene specification lets users iteratively revise layouts through follow-up requests while preserving generation consistency. It also supports contextual dialogue and open-ended suggestions without additional training.

  • C Additional features and use cases from instruction-based scene specification: LMD accepts follow-up instruction requests after an initial prompt and regenerates images from the updated layout.The layout is revised through multi-round interaction rather than changing only the original request.
  • C Additional features and use cases from instruction-based scene specification: The demonstrated additional use cases are supported without additional training.
  • C Additional features and use cases from instruction-based scene specification: Users can try different adjustments on the same generation while preserving the overall image style and layout.This supports fine-grained content creation.
  • C Additional features and use cases from instruction-based scene specification: The LLM can respond to open-ended requests and suggest objects or edits that could improve the scene.
  • C Additional features and use cases from instruction-based scene specification: Context-aware instruction parsing enables more natural dialogue than methods that process a single instruction without context.
  • C Additional features and use cases from instruction-based scene specification: Supported interactions include asking what objects could make a scene lively, undoing the last edit, and replacing an object with a small pond.

D.1 Text-to-layout stage

The text-to-layout stage depends on the LLM's ability to produce accurate layouts from prompts. Larger and stronger LLMs generally improve layout generation, while open-source models can also produce high-quality layouts.

  • All evaluated LLMs almost perfectly follow prompt requirements when generating layouts, making layout-to-image generation the reported bottleneck.
  • GPT-4 improves layout and subsequent image-generation results compared with GPT-3.5-turbo.
  • High-quality layout generation is not limited to proprietary LLMs, with Llama2-based StableBeluga2 and Mixtral also evaluated.
  • The ablations evaluate the effects of hyperparameters ω and λ on the method.
  • Performance is relatively stable across ω and λ, with ω = 4 optimal for both LMD and LMD+ in the reported settings.The authors report that λ = 3 outperforms the default λ = 2, suggesting further gains from tuning.
  • The tested LLM sizes directly concern stage 1 layout generation rather than the full image-generation pipeline.
  • Larger LLMs offer much better layout-generation capabilities than smaller models.

D.2 Layout-to-image stage

The layout-to-image stage is evaluated through hyperparameter ablations and combinations with alternative layout-to-image methods. Results support the effectiveness and relative robustness of the proposed stage.

  • Varying ω: ω = 4 already achieves optimal performance among the searched values for both LMD and LMD+.The method is relatively stable across hyperparameter selections, despite no prior hyperparameter search.
  • Varying λ: λ = 3 performs better than the default λ = 2, indicating further improvement through hyperparameter tuning.
  • Ablation results on T2I-CompBench: On T2I-CompBench, the method surpasses Stable Diffusion and variants combining its stage 1 with previous layout-to-image methods.This ablation attributes the gains to the proposed layout-to-image stage rather than only the text-to-layout stage.
  • Generated-layout distribution: With one similar in-context example, the LLM generates layouts tailored to queried objects instead of copying the example's box distribution.The qualitative analysis suggests that in-context references do not require heavy prompt engineering to avoid overly similar layouts.

F Additional visualizations

Additional visualizations compare LMD with Stable Diffusion v1.5 and with LLM-based image-generation systems. They illustrate stronger complex-prompt following and more durable multi-round scene specification for LMD.

  • Stable Diffusion v1.5 comparison: LMD more accurately follows complex prompts than Stable Diffusion v1.5 in the shared qualitative comparisons.The comparison uses the same text prompts as Fig. 1.
  • Multi-round scene specification: VisualChatGPT and GILL show image degradation beginning with the second request, whereas LMD handles several sequential requests without degradation.The comparison concerns multi-round scene specification for image generation.

H Details for SDXL integration

LMD transfers to SDXL with only marginal pipeline modifications because its training-free design does not require additional training. The implementation accounts for SDXL's two text encoders and supports alternative upsampling.

  • Applying LMD to SDXL: LMD is applicable to SDXL 1.0 without additional training because the method is training-free.SDXL 1.0 uses a U-Net module three times larger than those in previous Stable Diffusion models.
  • Implementation: Applying LMD to SDXL requires attention control across both text encoders, with the remainder of the procedure largely unchanged.Only marginal modifications to the LMD pipeline are needed.
  • SDXL refiner: An alternative uses standard-resolution LMD denoising followed by a few SDXL-refiner upsampling steps to reach resolutions such as 1024 × 1024.Most generation remains in the standard-resolution latent space.

I Generating images from languages not supported by the underlying diffusion model

LMD supports prompts in languages unsupported by the underlying diffusion model by producing English layouts through the LLM. This requires no diffusion-model adaptation.

  • Language handling: LMD generates images from unsupported-language prompts by asking the LLM to output layouts in English.The demonstrated examples include Korean and Chinese prompts.
  • SDXL comparison: The training-free pipeline can be applied to SDXL, whose direct text-prompt generation may still fail to accurately follow the requested object arrangement.The SDXL example compares direct generation with LMD for a gray cat and an orange dog on grass.
  • Language handling: The method keeps the in-context layout output in English while translating only the corresponding prompt input to a non-English language.The underlying diffusion model continues receiving an English layout, so no adaptation is needed.

J Details for text-to-image benchmarks

The benchmarks evaluate negation, generative numeracy, attribute assignment, and spatial relationships using controlled prompts and layout-based criteria. Experiments use fixed prompt templates, specified pretrained models, and default denoising and guidance settings.

  • Benchmark design: For negation and numeracy, a layout is correct when the generated object count matches the prompt, with counts ranging from 1 to 5.Negation prompts require omitting a specified object, while numeracy prompts specify an object count.
  • Benchmark design: Attribute assignment tests whether two objects receive their specified colors, using modifiers randomly selected from ten colors.The color list includes red, orange, yellow, green, blue, purple, pink, brown, black, white, and gray.
  • Benchmark design: The evaluation covers negation, generative numeracy, attribute assignment, and spatial relationship tasks using controlled text prompts.Each task uses 100 prompts, for 400 prompts in total.
  • Benchmark design: Spatial evaluation places two objects at opposite locations and checks whether their layout bounding-box coordinates exactly match the prompt.Locations are chosen from left, right, top, and bottom.
  • Implementation: LMD uses Stable Diffusion v1.5 by default, while LMD+ uses GLIGEN based on Stable Diffusion v1.4 without additional training or adaptation.The default process uses 50 denoising steps and latent composition during the first 25 steps.
  • Implementation: The appendix provides the full LLM layout-generation prompt and fixed in-context examples in Tables K.1 and K.2.LLM completion begins from “Objects:”.
Loading 2305.13655v3…