PaperlayerView exact revision on arXiv

Source-linked AI summary

GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing

Zhenyu Wang, Aoxue Li, Zhenguo Li, Xihui Liu

arXiv:2407.05600v2cs.CV

TL;DR

Existing image-generation and editing models struggle with complex instructions, diverse requirements, and reliable verification, while individual models specialize in only some tasks. GenArtist addresses this by coordinating multiple tools with an MLLM agent that decomposes problems, plans and verifies operations, and supplies missing positional inputs. It reports more than 7% improvement over DALL-E 3 on T2I-CompBench, state-of-the-art MagicBrush performance, and limitations from the underlying tools and localization models.

  • Problem

    Existing models struggle with complex generation and editing problems, lack autonomous verification and correction, and cannot reliably satisfy diverse requirements with a single specialized model.

  • Method

    GenArtist uses an MLLM agent to select and execute integrated generation and editing tools through decomposition, tree-structured planning, step-by-step verification, self-correction, and position-input completion.

  • Results

    GenArtist improves over DALL-E 3 by more than 7% on T2I-CompBench and achieves state-of-the-art performance on MagicBrush.

  • Takeaways & Limitations

    The experiments support GenArtist as a unified system for diverse image-generation and editing tasks with enhanced controllability and reliable outputs.

  • Takeaways & Limitations

    Failures can still arise from limitations in editing-tool resolution and localization-tool accuracy.

Abstract

from arXiv · show

Despite the success achieved by existing image generation and editing methods, current models still struggle with complex problems including intricate text prompts, and the absence of verification and self-correction mechanisms makes the generated images unreliable. Meanwhile, a single model tends to specialize in particular tasks and possess the corresponding capabilities, making it inadequate for fulfilling all user requirements. We propose GenArtist, a unified image generation and editing system, coordinated by a multimodal large language model (MLLM) agent. We integrate a comprehensive range of existing models into the tool library and utilize the agent for tool selection and execution. For a complex problem, the MLLM agent decomposes it into simpler sub-problems and constructs a tree structure to systematically plan the procedure of generation, editing, and self-correction with step-by-step verification. By automatically generating missing position-related inputs and incorporating position information, the appropriate tool can be effectively employed to address each sub-problem. Experiments demonstrate that GenArtist can perform various generation and editing tasks, achieving state-of-the-art performance and surpassing existing models such as SDXL and DALL-E 3, as can be seen in Fig. 1. Project page is https://zhenyuw16.github.io/GenArtist_page.

1 Introduction

GenArtist addresses diverse and complex image-generation and editing requirements by coordinating multiple tools with an MLLM agent that decomposes tasks, plans verification, and supplies missing positional inputs. It reports improved performance over existing systems, including more than 7% over DALL-E 3 on T2I-CompBench and state-of-the-art results on MagicBrush.

  • Motivation and contribution: Complex prompts and editing instructions are decomposed into simpler sub-problems, improving the reliability of model execution.The agent extracts objects and background elements from generation prompts and breaks editing instructions into single actions.
  • Motivation and contribution: A planning tree assigns operations to nodes, alternatives to sibling nodes, and verification after each operation, enabling generation, editing, and self-correction.This structure coordinates execution while checking whether operations succeed.
  • Motivation and contribution: Auxiliary tools automatically provide missing position-related inputs, compensating for the position insensitivity of existing MLLMs during tool execution.Position information is incorporated through detection models so appropriate object-level tools can be used.
  • Motivation and contribution: GenArtist coordinates generation and editing tools through an MLLM agent, providing a unified system for diverse image tasks.The agent serves as the system’s coordinating “brain” and selects suitable tools for different requirements.
  • Results: More than 7% improvement over DALL-E 3 on T2I-CompBench accompanies state-of-the-art performance on the MagicBrush image-editing benchmark.The experiments support GenArtist’s effectiveness across text-to-image generation and image editing.

2 Related Work

Prior work has advanced general and specialized image generation and editing, while multimodal language models have enabled tool-using agents. GenArtist builds on these directions by using an MLLM agent to coordinate a unified generation-and-editing workflow.

  • GenArtist’s relation to prior work: GenArtist applies an MLLM agent to decompose tasks, construct a verified planning tree, invoke generation and editing tools, and provide missing positional values.This coordination combines capabilities from both tool libraries within one system.
  • Image generation and editing: Existing image-generation and editing research includes general models and specialized methods for customization, text rendering, exemplar-based editing, and person-focused generation.The related work describes both broad diffusion-based systems and task-specific adaptations.
  • AI agents: MLLMs extend language-model agents with visual processing, enabling tool use for complex multimodal tasks.The related work positions MLLMs as a basis for agents that execute tasks through external tools.

3 Method

GenArtist uses an MLLM agent to decompose image tasks, construct and traverse a planning tree with verification and self-correction, and select tools from extensible generation and editing libraries. Auxiliary tools and position information address missing or imprecise spatial inputs before execution.

  • System overview: The MLLM agent decomposes complicated generation and editing tasks, constructs a planning tree, verifies results, and invokes tools from generation and editing libraries.An auxiliary library supplies missing position-related values.
  • Planning tree: Generation, editing, and alternative-tool operations are represented as tree nodes, with sibling tools offering different solutions to the same action.Generation nodes can receive adaptively created editing subtrees for self-correction.
  • Planning and verification: Execution traverses the tree in preorder, verifying each tool result, proceeding after success and backtracking to sibling alternatives after failure.This traversal supports coordinated planning and recovery from unsuccessful operations.
  • Planning and verification: Verification checks object content, attributes, positions, relationships, and background, while poor aesthetic quality can trigger regeneration with another tool or random seed.Human feedback can also be integrated during quality assessment.
  • Tool library: The tool library is extensible and includes generation, editing, and auxiliary tools, with tool descriptions and required inputs guiding agent selection.New models can be added to the current tool library.
  • Position-aware tool execution: Auxiliary tools compensate for missing positional inputs, while detector outputs provide spatial information that helps the MLLM select suitable tools.This addresses the limited spatial sensitivity of MLLMs and supports position-dependent operations.

4 Experiments

GenArtist is evaluated on complex image generation and editing benchmarks, ablations, visual planning examples, and failure cases. Results show strong benchmark performance, while verification, tool selection, tree planning, and position-aware execution improve reliability.

  • 4.2 Comparison with Image Editing Methods: GenArtist achieves superior MagicBrush editing results in both single-turn and multi-turn settings compared with prior global- and instruction-guided methods.The multi-turn setting evaluates iterative edits applied to previous source images.
  • 4.3 Ablation Study: A 3% average improvement from chain-based correction, followed by further gains from tree-based handling of failed editing tools, supports verification and tree planning.The ablation also reports that agent tool selection outperforms the evaluated generation tools.
  • 4.3 Ablation Study: Adding position information significantly improves the spatial and complex aspects of T2I-CompBench over tool selection without position information.Without position information, performance shows only a slight improvement over tool selection results.
  • 4.3 Ablation Study: The planning-tree examples show iterative tool selection, verification, failed edits, and subsequent corrective operations for multi-object generation and editing.The system may switch tools when an attempted edit fails or produces an undesirable result.
  • 4.4 Error Case Analysis: Errors remain when editing tools lack fine-resolution ability or localization tools inaccurately identify the target region.The authors suggest more powerful tools or human feedback during verification as possible remedies.

5 Conclusion

The conclusion presents GenArtist as a unified image generation and editing system coordinated by an MLLM agent. It attributes the system’s broad task coverage and reported state-of-the-art performance to decomposition, tree planning, and external tool execution.

  • 5 Conclusion: GenArtist coordinates image generation and editing through an MLLM agent that decomposes problems, plans with a tree structure, and invokes external tools.The conclusion describes the agent as the system’s coordinating “brain.”
  • 5 Conclusion: The reported experiments indicate that GenArtist handles complex generation and editing problems, achieves state-of-the-art performance, and supports a wide range of generation tasks.The authors frame the system as an important step toward autonomous agents.

Appendix

The appendix adds quantitative comparisons and visual results to provide a more comprehensive comparison with existing state-of-the-art methods.

  • Appendix: Additional quantitative comparisons and visual results are included to compare GenArtist more comprehensively with existing state-of-the-art methods.The appendix supplements the main experimental presentation.

A More Quantitative Experiments

Using the officially old-version T2I-CompBench evaluation code, GenArtist consistently improves performance over existing text-to-image and compositional methods.

  • Table 6 compares GenArtist with existing text-to-image generation models and compositional methods using the official old-version code.
  • Over 7% higher attribute-binding performance than DALL-E 3 is reported for GenArtist under the officially old-version T2I-CompBench metric.The comparison uses the same old-version evaluation method used in many existing reports.
  • Up to 12% improvement is reported for shape-related attributes compared with DALL-E 3.

B More Qualitative Experiments

GenArtist’s qualitative experiments show that its agent-and-tool framework handles demanding generation and editing requirements through decomposition, verification, position-aware execution, and diverse operations.

  • Comparative visualized results on image generation: GenArtist achieves superior image-generation results for attribute binding, numeric accuracy, position accuracy, and complex relationships through verification, iterative correction, and position-aware tool execution.The comparisons include SDXL, LMD+, RPG, PixArt-α, Playground, Midjourney, and DALL-E 3.
  • Comparative visualized results on image editing: GenArtist handles highly specific and reasoning-based editing instructions that existing methods often struggle to satisfy.Examples include modifying a particular pizza, changing a vase’s color and placement, and determining which person to remove.
  • Comparative visualized results on image editing: GenArtist decomposes complex editing instructions into single-step operations and supports addition, removal, and attribute editing through integrated tools.This enables multi-step instructions and diverse editing operations in the qualitative comparisons.
  • Visualized results about various tasks and user instructions: GenArtist addresses diverse user requirements, including layout- or pose-conditioned generation and customization-related generation, by integrating multiple tools.The visual examples cover varied tasks and user instructions.
  • Visualization for the step-by-step process: The step-by-step process begins with a suitable generation tool and continues with additional generation or editing tools until the image matches the requirements.The procedure is illustrated separately for image generation and image editing in Figures 10 and 11.
Loading 2407.05600v2…