Source-linked AI summary
GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editing
Rongyao Fang, Chengqi Duan, Kun Wang, Linjiang Huang, Hao Li, Shilin Yan, Hao Tian, Xingyu Zeng, Rui Zhao, Jifeng Dai, Xihui Liu, Hongsheng Li
TL;DR
Existing image generation and editing methods commonly process prompts without explicit reasoning about visual composition and operations. GoT introduces semantic-spatial reasoning chains before image synthesis, supported by large-scale datasets and an end-to-end MLLM-diffusion framework with SSGM. The framework reports significant improvements across generation and editing and enables interactive modification of reasoning steps.
Problem
Current visual generation methods lack explicit reasoning about complex semantic relationships, spatial arrangements, and editing operations.
Method
GoT uses MLLMs to generate semantic-spatial reasoning chains and guides an end-to-end diffusion model through a Semantic-Spatial Guidance Module.
Results
GoT demonstrates significant improvements in text-to-image generation quality and image-editing accuracy, with highest reported scores on several evaluated benchmarks.
Takeaways & Limitations
Users can interactively adjust generated images by directly modifying the explicit reasoning process.
Abstract
from arXiv · showhide
Current image generation and editing methods primarily process textual prompts as direct inputs without reasoning about visual composition and explicit operations. We present Generation Chain-of-Thought (GoT), a novel paradigm that enables generation and editing through an explicit language reasoning process before outputting images. This approach transforms conventional text-to-image generation and editing into a reasoning-guided framework that analyzes semantic relationships and spatial arrangements. We define the formulation of GoT and construct large-scale GoT datasets containing over 9M samples with detailed reasoning chains capturing semantic-spatial relationships. To leverage the advantages of GoT, we implement a unified framework that integrates Qwen2.5-VL for reasoning chain generation with an end-to-end diffusion model enhanced by our novel Semantic-Spatial Guidance Module. Experiments show our GoT framework achieves excellent performance on both generation and editing tasks, with significant improvements over baselines. Additionally, our approach enables interactive visual generation, allowing users to explicitly modify reasoning steps for precise image adjustments. GoT pioneers a new direction for reasoning-driven visual generation and editing, producing images that better align with human intent. To facilitate future research, we make our datasets, code, and pretrained models publicly available at https://github.com/rongyaofang/GoT.
1. Introduction
Existing visual generation methods map prompts to images without explicit reasoning about complex spatial relationships, while MLLMs offer capabilities that are not fully integrated into generation. GoT addresses this gap with semantic-spatial reasoning chains, large-scale data, and an end-to-end reasoning-guided framework.
- Current diffusion-based systems process textual prompts without explicit reasoning, limiting performance on scenes with precise spatial arrangements and object interactions.
- GoT introduces step-by-step natural-language reasoning before image output, integrating semantic relationships with spatial information for generation and editing.
- The GoT datasets contain 8.4M text-to-image samples and 920K image-editing examples with semantic-spatial reasoning annotations.
- A unified end-to-end framework combines MLLM-generated reasoning and visual tokens with a Semantic-Spatial Guidance Module that conditions diffusion generation.
- GoT reports significant improvements in text-to-image quality and image-editing accuracy, while enabling users to modify reasoning steps for interactive control.
2. Related Work
Diffusion models advanced visual synthesis, while LLMs and MLLMs introduced reasoning, grounding, and multimodal understanding. Related work explored prompt interpretation, unified modeling, layout planning, and editing, but these capabilities remained incompletely integrated.
- Diffusion models progressed from early visual synthesis systems to latent-space and larger-scale models supporting photorealistic, controllable, and instruction-based generation.
- Chain-of-thought extended reasoning capabilities from LLMs to MLLMs, including visual grounding and analysis of object relationships.
- Prior methods used LLMs for prompt interpretation or combined multimodal understanding with generation, but did not establish that strong reasoning benefits generation.
- Layout-guided approaches improved spatial control using bounding boxes, staged planning, recurrent refinement, or specialized editing models.
3. Generation Chain-of-Thought (GoT)
GoT reframes visual generation and editing as reasoning-guided processes that explicitly model semantic relationships and spatial configurations. It decomposes generation and manipulation into structured steps intended to improve compositional control and interpretability.
- Humans reason about object relationships and spatial arrangements, whereas most current models process prompts without explicit reasoning for complex scenes.
- GoT shifts visual generation from direct mapping to multimodal reasoning that jointly represents semantic relationships and spatial configurations.
- Text-to-image GoT chains specify element coordinates, enabling arrangement of scene constituents while maintaining their semantic relationships.
- For editing, GoT analyzes the scene, plans sequential modifications with precise coordinates, and describes the resulting image.
- GoT decomposes complex instructions into explicit steps, providing intermediate reasoning and unifying generation and editing in one system.
- The approach requires both detailed semantic-spatial reasoning data and a visual generation model capable of accepting chain inputs.
4. GoT Dataset: Semantic-Spatial Reasoning Chains for Visual Generation and Editing
The GoT dataset uses LLMs and MLLMs to create semantic-spatial reasoning chains for generation and editing. Task-specific pipelines combine descriptions, entity extraction, grounding, localization, and transformation reasoning across large source datasets.
- Each GoT sample contains task instructions, reasoning-chain annotations, and corresponding images for generation or editing.
- The text-to-image pipeline combines Qwen2-VL descriptions, Qwen2.5 entity extraction, and Qwen2-VL object grounding to produce semantic and spatial annotations.
- The editing pipeline describes source and target images, localizes editing regions with bounding boxes, and synthesizes coherent reasoning chains using Qwen2.5.
- Multi-turn editing annotations capture step-by-step sequences with multiple spatial coordinates and transformation descriptions.
- The text-to-image data combines LAHR, JourneyDB, and FLUX.1-generated images, while editing data derives from OmniEdit and SEED-Edit-Multiturn.
- Creating the datasets required 100 NVIDIA A100 GPUs operating for over a month.
5. GoT Framework: Reasoning-guided Visual Generation and Editing
GoT combines an MLLM reasoning engine with an end-to-end diffusion model to guide visual generation and editing using semantic and spatial information. Its multi-guidance design incorporates semantic, spatial, and reference-image signals during diffusion.
- Semantic-Spatial MLLM Design: Qwen2.5-VL-3B generates GoT reasoning chains and visual tokens for unified generation and editing tasks.For editing, it processes reference images; for both tasks, it captures attributes, relationships, modifications, and bounding boxes.
- Multi-Guidance Strategy: The Semantic-Spatial Guidance Module conditions an SDXL-based diffusion model with semantic, spatial, and reference-image guidance.Semantic embeddings enter cross-attention, spatial features come from color-coded coordinate masks, and reference features come from encoded images.
- Multi-Guidance Strategy: During diffusion, classifier-free guidance combines semantic, spatial, and reference-image signals with independently controlled guidance scales.The noisy latent is combined with null conditioning and the three guidance types; training samples conditioning combinations while excluding the fully conditioned case.
- Training Procedure: GoT uses 60,000 pretraining steps followed by 10,000 finetuning steps across multiple GoT datasets.LoRA updates the Qwen2.5-VL decoder while the SDXL-based diffusion module is fully optimized end-to-end.
- Text-to-Image Generation: Text-to-image examples show GoT planning object placement for compositional prompts with multiple objects, attributes, relationships, and spatial positions.The reported samples are described as coherent, aesthetically pleasing, and aligned with the input captions.
6. Experiments
Experiments evaluate GoT on text-to-image generation, interactive generation, image editing, and framework ablations. Results show strong overall benchmark performance, controllable interaction, qualitative editing strengths, and positive contributions from reasoning, SSGM, and extended pretraining.
- Text-to-Image Generation: 0.64 overall score is the highest on GenEval, with GoT particularly strong on single-object, counting, and color tasks.The corresponding category scores are 0.99, 0.67, and 0.85.
- Text-to-Image Generation: GoT surpasses Janus and JanusFlow in overall performance despite those systems performing better on specific spatial tasks.The comparison indicates more consistent performance across diverse generation requirements.
- Text-to-Image Generation: GoT generates coherent, aesthetically pleasing compositions containing multiple objects, attributes, relationships, and relative spatial positions.The qualitative samples emphasize effective planning of object placement.
- Interactive Generation: Users can modify descriptions and bounding-box positions to perform object replacement, position adjustment, and attribute modification while preserving scene coherence.The interactive examples provide control without requiring specialized expertise.
- Image Editing: 0.864 CLIP-I and 0.276 CLIP-T are GoT’s highest Emu-Edit scores, while GPT-4o evaluation reaches 0.533 on ImagenHub and 0.561 on Reason-Edit.On Reason-Edit, GoT ranks second to SmartEdit’s 0.572; the framework uses 2.8B+3B parameters.
- Image Editing: GoT qualitatively handles indirect object localization, complex spatial instructions, delicate additions, and multi-step editing operations.Red bounding boxes indicate coordinates predicted by the MLLM.
- Ablation Study: Adding GoT reasoning chains strengthens semantic guidance, while SSGM mainly improves image editing by providing more accurate spatial control.In GenEval, SSGM notably affects position but yields only a minor overall gain.
- Ablation Study: The full model with GoT reasoning, SSGM, and 60,000-step pretraining achieves the highest ablation scores, with every added component contributing positively.The study evaluates GenEval overall and ImagenHub GPT-4o evaluation.
7. Conclusion
The conclusion presents GoT as a reasoning-guided paradigm for visual generation and editing that integrates explicit semantic-spatial chains with visual synthesis. It reports benchmark performance alongside interactive control through modifiable reasoning chains.
- Conclusion: GoT integrates MLLM reasoning into visual generation through explicit semantic-spatial reasoning chains.The paradigm replaces direct mapping with a process offering precise spatial control over object relationships and arrangements.
- Conclusion: GoT combines 9M+ training examples, the Semantic-Spatial Guidance Module, and end-to-end training to support generation and editing.The conclusion also identifies modifiable reasoning chains as the basis for interactive control.
8. Training Details
Training uses separate pretraining and fine-tuning stages with specified datasets, schedules, optimizers, and parameter-efficient adaptation settings.
- Pretraining: Pretraining runs for 60,000 steps on LAHR-GoT, JourneyDB-GoT, and OmniEdit-GoT with cosine scheduling and a maximum learning rate of 1 × 10−4.The schedule includes 500 warmup steps.
- Fine-Tuning: Fine-tuning runs for 10,000 steps on FLUX-GoT, OmniEdit-GoT, and SEED-Edit-MultiTurn-GoT with a maximum learning rate of 5 × 10−5.This phase uses 200 warmup steps.
- Optimization: Both stages use Adam with β1 = 0.9, β2 = 0.98, ϵ = 1 × 10−6, weight decay 0.05, and batch size 128.
- Parameter-Efficient Adaptation: The LLM uses LoRA with r = 32, alpha 32, and dropout 0.05, while diffusion uses a noise offset of 0.1.
9. Visualization Results
Additional visualizations provide image-editing examples with corresponding GoT content and further interactive-generation samples. A separate visualization analyzes the Multi-Guidance Strategy’s hyperparameters.
- Additional Examples: Figure 7 shows image-editing examples together with the corresponding GoT content, while Figure 8 provides further interactive-generation examples.
- Hyperparameter Analysis: Figure 9 analyzes how Multi-Guidance Strategy hyperparameter selection affects generated images.The hyperparameter definitions are given in Section 5.3.
10. GoT Format and Examples
The GoT format structures visual generation and editing as explicit semantic-spatial plans. It represents objects, relationships, locations, and sequential editing operations to guide coherent transformations.
- Text-to-image generation: GoT structures text-to-image data as detailed plans that decompose image components and their spatial relationships.Location information is appended to key objects to improve spatial understanding.
- Single-turn editing: Single-turn editing records the source image, target object, editing operation, and resulting image as a step-by-step transformation.
- Multi-turn editing: Multi-turn editing decomposes an instruction into consecutive commands, applying the single-turn structure at each step.The process begins with source-image description and ends with a description of the edited result.
- Editing representation: Positional information is appended to editing objects to enhance spatial comprehension.
11. Prompts for Evaluation and Dataset Construction
The appendix documents prompts and examples used to evaluate image editing and construct GoT datasets for generation and editing. These materials cover automated scoring, detailed visual annotation, grounding, editing operations, and multi-turn reasoning.
- 11.1. Prompts for Evaluating Image Editing Performance: GPT-4o evaluates image editing using the average minimum of two scores for each sample.The evaluation uses GPT-4o-2024-11-20.
- 11.2. Prompts for Text-to-Image Data Construction: Text-to-image data preparation uses prompts for detailed recaptioning, object identification, and grounding.These prompts support construction of structured semantic-spatial annotations.
- 11.3. Prompts for Image Editing Data Construction: Image-editing data preparation uses prompts for edited-object parsing, image description, cropped-object description, and reinstruction.The appendix also shows in-context assembly of GoT prompts for editing data.
- Image-editing examples: Single-turn examples specify source content, an edited region, the requested operation, and the resulting image description.Examples include adding a person, changing an object’s color, and removing an object while preserving the surrounding scene.
- Additional visualizations: Additional examples illustrate interactive generation and hyper-parameter selection for the framework’s multi-guidance strategy.
- Dataset examples: Dataset examples include text-to-image collections and single-turn or multi-turn editing collections.The figures identify FLUX-GoT, JourneyDB-GoT, LAHR-GoT, OmniEdit-GoT, and SEED-Edit-Multiturn-GoT.
- Image-editing examples: Multi-step examples sequence several localized edits, such as recoloring clothing, removing birds, replacing a background, and adding a car.Each step identifies a target region and describes the intended transformation.