Source-linked AI summary
Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs
Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, Bin Cui
TL;DR
Diffusion models struggle with complex prompts requiring multiple objects, attributes, and relationships, while existing approaches face spatial-guidance and training-cost limitations. RPG addresses this gap with a training-free multimodal-LLM framework that recaptions and plans subprompts for regional diffusion, and reports superior text-guided generation and editing with broad architectural compatibility.
Problem
Diffusion models often struggle to compose objects with different attributes and relationships, while layout- or attention-based methods provide rough spatial guidance and feedback-based methods require additional training costs.
Method
RPG uses multimodal LLMs as recaptioners and chain-of-thought planners, assigns descriptive subprompts to complementary regions, and applies regional diffusion in a training-free generation and editing framework.
Results
RPG reports superior text-guided image generation and editing ability in comparisons with methods including SDXL, DALL-E 3, and InstructPix2Pix.
Takeaways & Limitations
RPG unifies text-guided generation and editing while generalizing across MLLM architectures and diffusion backbones.
Takeaways & Limitations
Existing layout- or attention-based methods remain limited by rough spatial guidance and difficulty handling overlapped objects, while feedback-based alternatives require high-quality feedback and additional training costs.
Abstract
from arXiv · showhide
Diffusion models have exhibit exceptional performance in text-to-image generation and editing. However, existing methods often face challenges when handling complex text prompts that involve multiple objects with multiple attributes and relationships. In this paper, we propose a brand new training-free text-to-image generation/editing framework, namely Recaption, Plan and Generate (RPG), harnessing the powerful chain-of-thought reasoning ability of multimodal LLMs to enhance the compositionality of text-to-image diffusion models. Our approach employs the MLLM as a global planner to decompose the process of generating complex images into multiple simpler generation tasks within subregions. We propose complementary regional diffusion to enable region-wise compositional generation. Furthermore, we integrate text-guided image generation and editing within the proposed RPG in a closed-loop fashion, thereby enhancing generalization ability. Extensive experiments demonstrate our RPG outperforms state-of-the-art text-to-image diffusion models, including DALL-E 3 and SDXL, particularly in multi-category object composition and text-image semantic alignment. Notably, our RPG framework exhibits wide compatibility with various MLLM architectures (e.g., MiniGPT-4) and diffusion backbones (e.g., ControlNet). Our code is available at: https://github.com/YangLing0818/RPG-DiffusionMaster
1. Introduction
RPG is a training-free framework that uses multimodal LLMs to recaption and plan complex prompts, then performs complementary regional diffusion for compositional generation and editing. The framework is reported to improve text-guided generation and editing relative to prior methods while supporting multiple MLLM architectures and diffusion backbones.
- Motivation: Complex prompts involving multiple objects, attributes, and relationships remain difficult for diffusion models to follow accurately.Existing layout- or attention-based methods provide rough spatial guidance and struggle with overlapped objects, while feedback-based methods require high-quality feedback and additional training costs.
- Method: RPG uses multimodal LLMs for recaptioning and chain-of-thought planning, decomposing complex prompts into descriptive subprompts assigned to complementary image regions.The planner partitions image space into subregions and generates rationales and instructions for subsequent image composition.
- Method: Complementary regional diffusion independently generates content within designated rectangles and merges it spatially, while contour-based regional diffusion supports precise editing.The regional approach is designed to address conflicting overlapped image contents and inconsistent regions targeted for modification.
- Framework: RPG unifies text-guided image generation and editing in a closed-loop framework.The framework uses multimodal feedback and progressive self-refinement to connect generation and editing workflows.
- Generalization: RPG generalizes across MLLM architectures and diffusion backbones, including MiniGPT-4 and ControlNet.The framework can incorporate additional conditions such as pose, depth, and canny edges through ControlNet.
- Results: Qualitative and quantitative comparisons report superior text-guided generation and editing ability against methods including SDXL, DALL-E 3, and InstructPix2Pix.Figure 2 specifically highlights RPG’s ability to convey intricate and compositional text prompts in generated images.
2. Method
RPG combines multimodal recaptioning, chain-of-thought region planning, and regional diffusion to decompose complex text-to-image generation into coordinated subregion tasks. It also extends the workflow to image editing through multimodal feedback, contour-based planning, and closed-loop refinement.
- Multimodal Recaptioning and Planning: RPG decomposes complex prompts into a base prompt and descriptive subprompts before allocating them to complementary spatial regions.The framework uses multimodal LLMs for recaptioning and multimodal chain-of-thought planning.
- Text-Guided Image Editing: RPG extends text-to-image generation to editing by analyzing source–target semantic discrepancies, generating edit plans and contours, and applying closed-loop self-refinement.The editing workflow combines contour-based editing with complementary regional diffusion.
- Multimodal Recaptioning: Text-to-text recaptioning adds fine-grained details to subprompts, improving generated-image fidelity and reducing prompt–image semantic discrepancy.
- Chain-of-Thought Planning: Chain-of-thought planning divides image space into complementary regions and assigns each augmented subprompt to a specific region.Region planning uses split ratios and multimodal in-context examples to produce meaningful subregions.
3. In-context examples
RPG uses multimodal LLMs to recaption prompts and images, plan complementary regions, and coordinate regional generation with editing in a closed loop.
- Regional diffusion: Complementary regional diffusion generates subregion content independently, resizes and concatenates the resulting latents, and uses base latents to improve coherence.The method also supports contour-based regional diffusion for targeted editing and generalizes across diffusion backbones.
- Recaptioning: RPG decomposes complex prompts into base and descriptive subprompts, then uses multimodal feedback to identify image–prompt discrepancies.The framework compares key entities, numeric accuracy, attribute binding, and object relationships before planning edits.
- Planning: CoT planning assigns recaptioned subprompts to complementary image regions and produces rationales and instructions for composition or editing.Editing plans include addition, deletion, and modification operations.
- Image editing: RPG extends regional diffusion to editing by masking target contours and applying diffusion-based inpainting according to planned operations.This supports addition, deletion, and modification of inconsistent regions.
- Closed-loop refinement: The workflow supports multi-round closed-loop refinement, combining recaptioning, CoT planning, contour editing, and regional generation.MLLM feedback progressively refines alignment with the target prompt.
3. Experiments
Experiments evaluate RPG across compositional generation, regional diffusion, model generalization, and image editing. Results report strong performance across composition scenarios and improved editing precision.
- Text-to-image generation: RPG is evaluated on attribute binding, numeric accuracy, and complex relationship scenarios against DALL-E 3, SDXL, and LMD+.The study uses T2I-CompBench to assess text-image alignment for general and compositional generation.
- Text-to-image generation: RPG achieves the best reported performance across the evaluated compositional and general text-to-image settings, including comparisons with ConPreDiff.The authors attribute gains to CoT planning and complementary regional diffusion.
- Hierarchical regional diffusion: Increasing the number of regional-diffusion hierarchies improves text-to-image generation quality.The authors present hierarchical splitting as a way to handle increasingly compositional generation tasks.
- Generalization: RPG generalizes to multiple multimodal LLMs, diffusion backbones, and ControlNet, although architecture choices influence generation results.ControlNet integration improves compositionality, image fidelity, and textual semantic alignment.
- Image editing: RPG produces more precise text-guided edits than Prompt2Prompt, InstructPix2Pix, and MasaCtrl.The framework also supports multi-round self-refinement and achieves satisfying editing results within 3 rounds.
4. Model Analysis
Ablations isolate the effects of recaptioning, CoT planning, and the base prompt ratio. Each component addresses a different aspect of compositional image generation.
- Recaptioning: Without recaptioning, RPG tends to ignore key words, whereas denser recaptioned descriptions produce more delicate and precise images.The ablation attributes improved detail to more informative descriptions of key words.
- CoT planning: Without CoT planning, the model fails to parse and convey complex relationships; CoT planning improves fine-grained attribute and relationship expression.The planned composition is described as more realistic.
- Base prompt: A proper base-prompt ratio benefits conjunction across subregions, while an excessive ratio causes confusion between base and regional prompts.The base latent is used to improve coherence, but its contribution requires balancing.
5. Related Work
Related work spans text-guided diffusion, compositional generation, and multimodal LLMs for image tasks. Existing approaches add controls, manipulate attention, or use image feedback, but involve limitations in guidance or training cost.
- Text-guided diffusion: Text-guided diffusion models improve image quality and text alignment, yet complex prompts involving multiple attributes and relationships remain challenging.The difficulty persists despite advances such as SDXL, ContextDiff, and DALL-E 3.
- Compositional diffusion generation: Compositional-generation methods introduce trainable spatial modules, conditional adapters, layouts, or inference-time latent and attention manipulation.These approaches provide spatial or semantic control but can add training and inference costs.
- Open limitations: Existing layout- or attention-based methods provide rough spatial guidance and struggle with overlapped objects, while feedback-based methods require high-quality feedback and additional training.These limitations frame the motivation for a training-free compositional framework.
- Multimodal LLMs for image generation: Multimodal LLM research extends language-model abilities to image understanding, reasoning, and synthesis, including collaboration with diffusion models.This line of work motivates using multimodal reasoning for image generation.
6. Conclusion
RPG is presented as a training-free framework that uses multimodal LLMs to control diffusion models for complex or compositional text-to-image generation. It combines complementary regional diffusion with closed-loop text-guided generation and editing, and is designed to generalize across MLLM architectures and diffusion backbones.
- RPG uses multimodal LLMs to master diffusion models for complex or compositional text-to-image generation.
- Complementary regional diffusion collaborates with an MLLM-based recaptioner and planner within RPG.
- RPG unifies text-guided image generation and editing through a closed-loop approach.
- The framework is capable of generalizing to any MLLM architectures and diffusion backbones.