PaperlayerView exact revision on arXiv

Source-linked AI summary

LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation

Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, Tat-Seng Chua

arXiv:2308.05095v2cs.CVcs.AI

TL;DR

Complex-scene text-to-image models can misrepresent spatial, action, and numeric relations, while existing fine-grained guidance often requires manual input. LayoutLLM-T2I automatically induces coarse layouts with an LLM and uses layout-guided diffusion for image synthesis, achieving new state-of-the-art results on T2I benchmarks, particularly in complex-scene and zero-shot settings.

  • Problem

    Existing text-to-image models struggle with faithful spatial, action, and numeric relations in complex scenes, while fine-grained guidance can require manual user input.

  • Method

    LayoutLLM-T2I uses LLM-based in-context learning to induce coarse layouts, then injects layout and semantic relation information into a layout-guided diffusion model.

  • Results

    The proposed model achieves new state-of-the-art results on T2I benchmarks and improves generation particularly in complex-scene and zero-shot settings.

  • Takeaways & Limitations

    Automatically generated layout guidance supports high-faithfulness text-to-image synthesis without requiring manual guidance.

  • Takeaways & Limitations

    The study mainly considers fully automatic solutions without human efforts, unlike methods that combine Stable Diffusion with additional human guidance.

Abstract

from arXiv · show

In the text-to-image generation field, recent remarkable progress in Stable Diffusion makes it possible to generate rich kinds of novel photorealistic images. However, current models still face misalignment issues (e.g., problematic spatial relation understanding and numeration failure) in complex natural scenes, which impedes the high-faithfulness text-to-image generation. Although recent efforts have been made to improve controllability by giving fine-grained guidance (e.g., sketch and scribbles), this issue has not been fundamentally tackled since users have to provide such guidance information manually. In this work, we strive to synthesize high-fidelity images that are semantically aligned with a given textual prompt without any guidance. Toward this end, we propose a coarse-to-fine paradigm to achieve layout planning and image generation. Concretely, we first generate the coarse-grained layout conditioned on a given textual prompt via in-context learning based on Large Language Models. Afterward, we propose a fine-grained object-interaction diffusion method to synthesize high-faithfulness images conditioned on the prompt and the automatically generated layout. Extensive experiments demonstrate that our proposed method outperforms the state-of-the-art models in terms of layout and image generation. Our code and settings are available at https://layoutllm-t2i.github.io.

1 INTRODUCTION

Complex-scene text-to-image generation remains difficult because models struggle with spatial relations, actions, numeration, and object interactions. LayoutLLM-T2I addresses this through automatic LLM-based layout planning followed by layout-guided diffusion, achieving new state-of-the-art results.

  • 1 INTRODUCTION: Existing diffusion models perform well on many T2I tasks but still struggle to synthesize faithful images in complex scenes.The cited issues include spatial confusion, action ambiguity, and numeration failure.
  • 1 INTRODUCTION: High-faithful synthesis requires understanding scene layout and delicate object interactions before rendering fine-grained image details.The paper motivates this as a coarse-to-fine process analogous to first sketching objects and relations, then completing details.
  • 1 INTRODUCTION: Unlike approaches requiring manually supplied guidance, the proposed solution aims to generate layouts and images automatically.The paper distinguishes its fully automatic setting from methods using additional human guidance.
  • 1 INTRODUCTION: The framework first induces a coarse layout from the prompt with an LLM, then injects organized entities and semantic relations into Stable Diffusion.Its two stages are text-to-layout induction and layout-guided text-to-image generation.
  • 1 INTRODUCTION: LayoutLLM-T2I achieves new state-of-the-art results on T2I benchmarks, especially for complex scenes and zero-shot settings.The experiments also support the effectiveness of feedback-based layout sampling and layout-guided feature integration.

2 RELATED WORK

Related work spans text-to-image generation, spatially guided synthesis, and layout-to-image generation. LayoutLLM-T2I differs by using language models to plan layouts for complex text-guided image generation.

  • 2 RELATED WORK: Text-to-image research has progressed from GANs, VAEs, and autoregressive models toward diffusion models for realistic image synthesis.Diffusion methods provide a major foundation for current T2I systems.
  • 2 RELATED WORK: Complex prompts remain challenging because they combine multiple objects with spatial, action-based, and numeric relations.These relations require faithful correspondence between language and scene structure.
  • 2 RELATED WORK: Prior methods improve object-relation modeling using segmentation features or spatial conditioning during visual synthesis.Such methods provide additional structural information to improve faithfulness.
  • 2 RELATED WORK: Layout-to-image methods use layouts as inputs, with recent diffusion approaches producing more reliable image generation.The paper instead focuses on deriving layout guidance from text for T2I.

3 PRELIMINARY ON LATENT DIFFUSION

Stable Diffusion performs diffusion in a low-dimensional latent space encoded by a hierarchical VAE. A UNet iteratively denoises text-conditioned latents, which are then decoded into images.

  • 3 PRELIMINARY ON LATENT DIFFUSION: Stable Diffusion operates in a low-dimensional latent space rather than directly in image space, improving computational efficiency.A VAE encoder maps an image I to a spatial latent code Z.
  • 3 PRELIMINARY ON LATENT DIFFUSION: The denoising network is typically a UNet containing convolution, self-attention, and cross-attention layers.These components support the model’s latent-space denoising process.
  • 3 PRELIMINARY ON LATENT DIFFUSION: At each timestep, the diffusion model denoises the latent conditioned on the text prompt, starting from Gaussian noise.The process iteratively transforms Z_T into Z_0.
  • 3 PRELIMINARY ON LATENT DIFFUSION: After denoising, the VAE decoder maps the final latent back into image space to generate the output image.The final latent is Z_0.

4 METHODOLOGY

LayoutLLM-T2I uses a coarse-to-fine pipeline that first induces layouts from prompts with LLM in-context learning, then generates images with layout-guided diffusion and relation-aware interactions.

  • 4 Methodology: The framework first infers a coarse-grained layout from the textual prompt and then synthesizes the final image conditioned on both prompt and layout.The two stages are text-to-layout induction and layout-guided text-to-image generation.
  • 4.1 Text-to-Layout Induction: Feedback-based sampler learning selects informative in-context examples for LLM layout planning using layout and image rewards.The policy is optimized with Monte Carlo reward estimation and REINFORCE, encouraging layouts that support reasonable, aesthetic images.
  • 4.1 Text-to-Layout Induction: The sampler represents candidate texts in a latent layout-embedding space and uses similarity-based policy sampling before querying the LLM for a layout.The LLM receives the prompt, selected examples, and instruction to produce the induced layout.
  • 4.2 Layout-guided Image Generation: Layout guidance is encoded by combining Fourier-mapped bounding-box coordinates with object labels, then injected into the diffusion model through an adapter.The adapter balances quality and controllability while preserving the pretrained diffusion model's original components.
  • 4.2 Layout-guided Image Generation: Relation-aware generation selects visual object features from bounding-box masks and applies attention to integrate interactions among objects, layout, text, and image tokens.This addresses the limitation that generic self-attention does not explicitly model intact visual objects and their relations.

5 EXPERIMENTS

Experiments on COCO 2014 evaluate layout planning and layout-guided image generation across standard and relation-focused settings. LayoutLLM-T2I achieves the strongest reported performance, with feedback-based sampling and relation-aware generation contributing to its gains.

  • LayoutLLM-T2I substantially outperforms compared baselines in overall layout and image generation, achieving state-of-the-art results especially on pair-wise relevance metrics.
  • 5.2.1 Text-to-Layout Generation: Across numerical, spatial, semantic, mixed, and null prompts, the method achieves the best performance under most layout metrics and surpasses baselines on complex and abstract cases.Mixed prompts contain multiple relations or numerical descriptions, while null prompts require abstract reasoning without explicit relation keywords.
  • 5.2.2 Layout-guided Text-to-Image Generation: In layout-guided image generation, LayoutTrans performs worst, several diffusion and transformer baselines show similar results, and the proposed method has a substantial advantage.The reported similarity among several baselines is described as reflecting comparable layout-generation capabilities.
  • 5.3.1 Ablation Study: Feedback-based sampling outperforms random and nearest-neighbor sampling on mIoU and LaySim, indicating that semantic similarity alone does not fully capture layout planning.Nearest-neighbor sampling improves over random sampling, but the proposed feedback-based strategy performs best on the reported layout metrics.
  • 5.3.1 Ablation Study: Layout-planning performance improves as in-context shots increase from zero to four, then largely saturates after three shots, while zero-shot performance remains competitive with recent baselines.The experiments compare zero-shot and few-shot settings with 2, 3, 4, and 5 examples.
  • 5.3.2 Case Study: Qualitative results across five COCO 2014 subsets show that the proposed method can reason about object distributions and depict their relations in generated images.The subsets cover numerical, spatial, semantic, complex, and abstract layouts.

6 CONCLUSION

The paper addresses cross-modal text-guided image generation by targeting weaknesses in layout planning, numerical reasoning, spatial relations, semantic relations, complex layouts, and abstract imagination. It uses feedback-based LLM prompting and relation-aware interaction, with experiments validating the proposed model.

  • The proposed model targets five weaknesses in existing text-to-image generation: numerical reasoning, spatial relations, semantic relations, complex layout planning, and abstract imagination.
  • Feedback-based learning performs in-context layout planning, while a relation-aware interaction module promotes image generation.
  • Extensive experiments on a constructed test set validate the effectiveness and superiority of the proposed model.

B.1 Detailed Implementation Settings

The framework uses a two-stage implementation: scene-graph triplets and CLIP embeddings support the first stage, while reinforcement learning selects informative in-context examples in the second.

  • Stage 1: Scene-graph parsing extracts up to 10 subject-predicate-object triplets per caption and embeds them with CLIP’s textual branch.The first stage also uses triplet embeddings and intermediate UNet representations as inputs.
  • Stage 2: A linear 128-neuron layer learns layout-level similarities over CLIP semantic embeddings to implement the example-selection function.
  • Stage 2: Reinforcement learning optimizes the feedback-based sampler using layout-level reward implemented with mIoU.
  • Baselines: Baseline models are retrained on COCO 2014 with added cross-attention layers to accept complete free-form natural-language conditions.

B.2 Detailed Test Set Construction

The test set is constructed from COCO 2014 validation captions by filtering numeral, spatial, semantic, and null relation subsets, then sampling large subsets down to 200 instances.

  • Filtering: The construction begins by filtering COCO 2014 validation captions with keyword lists and POS-based detection of notional verbs.The resulting primary subsets target numerical, spatial, and semantic relations.
  • Second Filtering: Exclusive numerical and semantic subsets remove samples that also belong to the other relation categories, while the spatial subset is formed analogously.
  • Null Dataset: The null subset removes numerical, spatial, and semantic samples from the total dataset to exclude captions containing explicit relation keywords.
  • Sampling: Datasets exceeding 200 instances are randomly reduced to 200 samples, and Table 4 reports sample counts, average boxes, and average caption lengths.

B.3 Detailed Layout and Image Evaluation

Evaluation measures layout and image generation with multiple metrics, while in-context prompts specify normalized object bounding boxes and provide shot-based examples for layout generation.

  • Layout Evaluation: Layout evaluation uses FID and LaySim, with LaySim comparing generated and gold layouts through weighted maximum matching of bounding boxes.The supplied passages describe FID as computed from discriminative layout features and LaySim as a layout-similarity measure.
  • Layout Evaluation: Free-label predictions are mapped to the closest COCO class using CLIP textual similarities before computing mIoU and LaySim.
  • Image Evaluation: Image evaluation also includes FID, which measures the Fréchet distance between synthetic-image and reference-image distributions.
  • In-Context Examples: Shot-based examples demonstrate layouts for counted objects, spatial containment, kitchen scenes, and refrigerator contents.
  • Prompt Format: The prompt format represents each object as a bounding box [x, y, w, h], with coordinates and box endpoints constrained between 0 and 1.
  • In-Context Examples: Additional examples provide layouts for kitchen items, appliances, food, condiments, and a person with ski equipment.

C EXPERIMENTAL RESULTS

Feedback Sampling generally outperforms nearest-neighbor and random sampling across most categories and evaluation metrics, while random sampling performs worst in most cases.

  • Impact of In-context Example Sampling: Feedback Sampling generally outperforms Nearest Neighbor Sampling and Random Sampling across most categories and evaluation metrics.The comparison covers the full test set and five categories.
  • Impact of In-context Example Sampling: Random Sampling produces the worst layouts in most cases, especially on the numerical subset.
  • Impact of In-context Example Sampling: Nearest Neighbor Sampling performs better than Random Sampling in the reported comparisons.

C.1 More Examples

Qualitative comparisons show that the proposed method improves textual faithfulness over Stable Diffusion and achieves high numerical, semantic, and spatial fidelity against layout-guided baselines.

  • Stable Diffusion vs. Ours: The proposed method improves textual faithfulness over Stable Diffusion in qualitative comparisons across numerical, spatial, semantic, mixed, and null tests.Figure 10 compares Stable Diffusion with the proposed method on these five test subsets.
  • Layout-guided Generation Baselines vs. Ours: Against layout-guided baselines, the proposed method generates images with high numerical, semantic, and spatial fidelities.Figures 11 and 12 compare ground truth, ground-truth layouts with generated images, LayoutDM results, and the proposed results.
  • Impact of In-context Example Sampling: Figure 8 evaluates in-context example sampling with mIoU, LaySim, and Sim (I-T) on the full test set and five categories.
  • Impact of In-context Example Sampling: Figure 9 compares different shot numbers using FID, mIoU, and LaySim for layout performance.
  • Layout-guided Generation Baselines vs. Ours: Figure 11 presents qualitative results for numerical, mixed, and null test sets using ground truth, ground-truth layout outputs, LayoutDM, and the proposed method.
  • Layout-guided Generation Baselines vs. Ours: Figure 12 presents qualitative results for spatial and semantic test sets using the same four-way comparison.
Loading 2308.05095v2…