PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-correcting LLM-controlled Diffusion Models

Tsung-Han Wu, Long Lian, Joseph E. Gonzalez, Boyi Li, Trevor Darrell

arXiv:2311.16090v1cs.CV

TL;DR

Text-to-image diffusion models still struggle to follow complex prompts, while open-loop generation does not check alignment after producing an image. SLD uses LLM-guided assessment and correction to improve alignment, with reported gains on LMD and when correcting DALL-E 3 generations.

  • Problem

    Text-to-image diffusion models often struggle with complex prompts involving numeracy, spatial relationships, and attribute binding, while open-loop generation lacks a robust mechanism to ensure alignment with user expectations.

    Source: §1 · pp. 1–2

  • Method

    SLD uses LLM-driven object detection and LLM-controlled assessment and correction to iteratively identify and rectify image errors without additional base-model training.

    Source: §1 · p. 2

  • Results

    SLD surpasses LMD+ by 9.0% on the LMD benchmark and yields 26.5% performance gains over DALL-E 3 images before self-correction.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework can correct a majority of incorrect generations, particularly in numeracy, attribute binding, and spatial relationships, and can be adapted for image editing by changing LLM instructions.

    Source: §1 · p. 2

    For complex hair removal, SAM-based region selection also removed the person's face, and the base diffusion model then failed to produce a natural composition.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Text-to-image generation has witnessed significant progress with the advent of diffusion models. Despite the ability to generate photorealistic images, current text-to-image diffusion models still often struggle to accurately interpret and follow complex input text prompts. In contrast to existing models that aim to generate images only with their best effort, we introduce Self-correcting LLM-controlled Diffusion (SLD). SLD is a framework that generates an image from the input prompt, assesses its alignment with the prompt, and performs self-corrections on the inaccuracies in the generated image. Steered by an LLM controller, SLD turns text-to-image generation into an iterative closed-loop process, ensuring correctness in the resulting image. SLD is not only training-free but can also be seamlessly integrated with diffusion models behind API access, such as DALL-E 3, to further boost the performance of state-of-the-art diffusion models. Experimental results show that our approach can rectify a majority of incorrect generations, particularly in generative numeracy, attribute binding, and spatial relationships. Furthermore, by simply adjusting the instructions to the LLM, SLD can perform image editing tasks, bridging the gap between text-to-image generation and image editing pipelines. We will make our code available for future research and applications.

1. Introduction

Complex prompts remain difficult for text-to-image diffusion models, and open-loop generation lacks a mechanism to check whether the result matches the prompt. SLD adds LLM-guided assessment and correction to generation, improving benchmark results and supporting image editing.

1. Introduction

  • Complex prompts involving numeracy, spatial relationships, and attribute binding remain challenging for state-of-the-art open-source and proprietary diffusion models.Scaling model sizes and training data has not eliminated these difficulties.Source: §1 · p. 1
  • SLD turns single-round generation into a closed loop that uses LLM-driven object detection and LLM-controlled assessment and correction to identify and rectify errors.The controller can suggest adding, moving, or removing objects, which a base diffusion model implements through latent-space composition.Source: §1 · p. 2
  • SLD requires no additional training of the base diffusion model and can correct images initially generated by proprietary models such as DALL-E 3 through APIs.The approach is also described as avoiding external human annotation.Source: §1 · p. 2
  • SLD surpasses LMD+ by 9.0% on the LMD benchmark and yields 26.5% performance gains over DALL-E 3 images before self-correction.LMD+ is described as a strong baseline that already uses an LLM in image generation.Source: §1 · p. 2
  • SLD can correct a majority of incorrect generations, particularly in numeracy, attribute binding, and spatial relationships, and can be adapted for image editing by changing LLM instructions.The paper presents generation and object-level editing as tasks supported by a unified pipeline.Source: §1 · p. 2

2. Related Work

Prior work controls diffusion generation with additional inputs or LLM-interpreted layouts, while image-editing methods still face challenges with precise object manipulation. SLD combines detectors and LLMs in a closed loop for generation and prompt-driven editing.

2.1. Text-to-Image Diffusion Models

  • Earlier diffusion work incorporated inputs such as keypoints and bounding boxes, while later LLM-based methods interpreted prompts and configured initial layouts.These approaches are described as often generating in one open-loop iteration without guaranteeing prompt alignment.Source: §2.1 · pp. 2–3
  • SLD is presented as the first closed-loop diffusion generation method, using detectors and LLMs for iterative checking and correction across numeracy, attribute binding, and spatial reasoning.The approach is described as applicable to models including DALL-E 3.Source: §2.1 · p. 3

2.2. Diffusion-based Image Editing

  • Existing editing methods include global style transformations and local edits, but often struggle with precise object manipulation and spatial edits such as resizing or repositioning.Self-Guidance still requires users to provide coordinates for moving or repositioning objects.Source: §2.2 · p. 3
  • SLD places detectors and LLMs in the editing loop, enabling prompt-driven object-level edits such as adding, replacing, moving, and modifying attributes.The paper also lists swapping among its editing operations.Source: §2.2 · p. 3

3. Self-correcting LLM-controlled Diffusion

SLD combines LLM-driven object detection with LLM-controlled assessment and correction in a self-correction pipeline. It applies latent-space edits, repeats correction until the layout stabilizes or a round limit is reached, and adapts to image editing through changed LLM instructions.

3.1. LLM-driven Object Detection

  • The LLM parser extracts key object details from the prompt, and an open-vocabulary detector uses them to locate objects and produce current bounding boxes.The parser includes associated quantities and attributes, while the detector can also count non-attributed objects when quantity constraints require it.Source: §3.1 · p. 4

3.2. LLM-controlled Analysis and Correction

  • The LLM controller compares detected boxes with the prompt and proposes corrected boxes, which SLD converts into addition, deletion, repositioning, and attribute-modification operations.Object IDs preserve correspondence between boxes of the same object type before and after correction.Source: §3.2 · p. 4

3.2.1 Latent Operations for Training-Free Image Correction

  • SLD applies edits in latent space: it composites newly added or modified objects, reinitializes removed regions with Gaussian noise, then allows a final forward diffusion process to refine the image.For repositioning, resizing occurs in image space to preserve quality; early forward-diffusion steps freeze unaffected regions before later steps allow everything to change.Source: §3.2.1 · p. 5, Figure 4 · p. 5

3.2.2 Termination of the Self-Correction Process

  • Correction repeats until the controller proposes the current layout or the maximum number of rounds is reached; later rounds can improve correctness beyond the often-effective first round.The described correctness guarantee is qualified by the accuracy of the detector and LLM controller.Source: §3.2.2 · pp. 5–6

3.3. Unified text-to-image generation and editing

  • Changing the LLM instructions adapts SLD to editing, where the controller proposes updated boxes and latent operations for precise object-level changes.The stated editing capabilities include object repositioning, addition, and deletion with exact control.Source: §3.3 · p. 6

4. Experiments

Experiments report improved generation correctness across complex-prompt tasks and demonstrate object-level editing capabilities. Multi-round correction improves on a single round, while complex object shapes can challenge region selection.

4.1. Comparison with Image Generation Methods

  • On numeracy tasks, SLD with LMD+ reaches up to 98% accuracy, while SLD improves DALL-E 3 performance by over 20%; it also improves attribute binding and spatial reasoning.For attribute binding, SLD improves DALL-E 3 and LMD+ by 6% and 14%, respectively; for spatial reasoning, the gains are 6% and 12%, respectively.Source: §4.1 · p. 7

4.2. Application to Image Editing

  • SLD performs object-level edits such as replacing objects while preserving surrounding content, whereas InstructPix2Pix is confined to global transformations and DiffEdit can fail to locate objects accurately.The examples also show edits spanning object counts, attributes, positions, and sizes, including enlarging a cup by exactly 1.25×.Source: §4.2 · p. 7, Figure 6 · p. 7, Figure 7 · p. 8

4.3. Discussion

  • Multi-round correction consistently outperforms a single round, although the first round rectifies the majority of errors and is the most effective round.The second round still yields improvements across all four tasks.Source: Table 2 · p. 8, §4.3 · p. 8
  • Complex object shapes can make SAM’s region selection remove adjacent parts unintentionally, such as a person’s face when the target is hair.The authors identify better region selection as a needed improvement.Source: §4.3 · p. 8, Figure 8 · p. 8

5. Conclusion

The conclusion presents SLD as a detector- and LLM-based self-correction framework for text-to-image alignment. It reports compatibility with multiple generative models and fine-grained image-editing capabilities.

5. Conclusion

  • SLD uses detectors and LLMs in a self-correction system intended to enhance text-to-image alignment.Source: §5 · p. 8
  • The framework is reported to set a new state of the art on an image-generation benchmark and to be compatible with models including DALL-E 3.Source: §5 · p. 8
  • SLD extends to image editing, offering fine-grained object-level manipulation that the conclusion says surpasses existing methods.Source: §5 · p. 8

A. More Comparisons with Image Editing Methods

SLD handles fine-grained image edits that prior diffusion-based methods struggle with, including object deletion and addition.

A. More Comparisons with Image Editing Methods

B. Comparison between LMMs and Detector-LLM Combinations

The paper investigates LMM-based image assessment for object-level editing recommendations, but reports that GPT-4V miscounts characters and struggles to provide precise bounding boxes.

B. Comparison between LMMs and Detector-LLM Combinations

  • The LMM setup assesses generated images alongside user prompts to provide object-level editing recommendations.The authors investigate this approach with GPT-4V and LLaVA.Source: B. Comparison between LMMs and… · p. 11
  • GPT-4V miscounts the dwarfs in a DALL-E 3 image, identifying seven rather than five.This example illustrates an image-assessment error.Source: B. Comparison between LMMs and… · p. 11
  • GPT-4V also struggles to define precise bounding-box coordinates, motivating the authors' detector–LLM combination for assessment and editing suggestions.They identify integration of advanced LMMs as a direction for future research.Source: B. Comparison between LMMs and… · p. 11

C. Comprehensive Image Generation Results

SLD improves generation accuracy across detector choices and supports image-editing operations, including selective object removal.

C. Comprehensive Image Generation Results

  • Compared with InstructPix2Pix and DiffEdit, SLD handles object deletion and addition, while DiffEdit may identify an object yet fail to produce a plausible background.The figure contrasts these editing examples.Source: Figure 9 · p. 11
  • OWL-ViT v2's localization of seagulls enables their selective removal to align with the user's prompt.The figure caption highlights detection of all seagulls in the image.Source: Figure 10 · p. 11
  • SLD enhances overall accuracy over baseline text-to-image generators regardless of which detector is used in its self-correction pipeline.The paper attributes marginally reduced gains with the alternate detector to its relatively inferior detection capabilities and differences in object recognition; accuracy gains remain especially evident in numeracy and spatial reasoning.Source: C. Comprehensive Image Generation… · pp. 11–12, Table 3 · p. 12

D. Our Prompts and Instructions to the LLM

The LLM parser identifies prompt objects, while the controller proposes bounding-box adjustments; explicit chain-of-thought guidance improved suggestion accuracy.

D. Our Prompts and Instructions to the LLM

Loading 2311.16090v1…