PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

Jiahao Zhao, Xiaomin Yu, Zhongxiang Sun, Fengwei Teng, Chengwei Qin, Xiaobin Hu, Jun Xu, Shuicheng Yan

arXiv:2608.04436v1cs.CV

TL;DR

Open-world image generation remains unreliable for requests requiring complex reasoning, external knowledge, or factual constraints. ToolArtist unifies reasoning, tool use, and native image generation under one policy, achieving the best non-proprietary WorldGenBench-Humanities average KCS score of 22.10 versus 21.76 for Qwen-Image.

  • Problem

    Existing image-generation systems provide limited agent control, leaving reasoning, tool use, and image generation uncoordinated for complex open-world requests.

  • Method

    ToolArtist post-trains a Unified Multimodal Model to jointly orchestrate reasoning, external search tools, and native image generation under one policy.

  • Results

    22.10 KCS was the best non-proprietary WorldGenBench-Humanities average, versus 21.76 for Qwen-Image, while ToolArtist also outperformed partially agentic approaches.

  • Takeaways & Limitations

    The released training data and complete SFT/RL infrastructure support future research on agentic image generation.

Abstract

from arXiv · show

Text-to-image (T2I) models can produce visually compelling images, yet they remain limited on open-world tasks that require complex semantic understanding, multi-step reasoning, and the integration of external world knowledge. Existing efforts introduce agent capabilities into image generation, but they either prescribe a fixed workflow or place only a subset of the open-world image generation process under agent control. Consequently, reasoning, tool invocation, and image generation are not coordinated by a single policy. We propose ToolArtist, a fully agentic image generation model obtained by post-training a Unified Multimodal Model (UMM). ToolArtist dynamically orchestrates reasoning, external tool use, and native image generation within one unified policy. During Supervised Fine-Tuning (SFT), we equip a teacher agent with search tools alongside an image-generation tool. We then convert the collected trajectories into a UMM compatible format, where the image-generation tool is concealed while the resulting generated images are retained. During Reinforcement Learning (RL), we develop an agentic RL infrastructure for UMMs and introduce Reason-Act-Draw GRPO (RAD-GRPO), which uses complementary intent and quality rewards to jointly optimize the model. Experiments show that placing the entire open-world image-generation process under an agent policy consistently outperforms approaches with fixed pipelines or only partially agent-controlled components. We release the training data and the complete post-training infrastructure.

1 Introduction

ToolArtist addresses the limitations of open-world image generation by unifying reasoning, external tool use, and native image generation under a single UMM policy. Its SFT and RL strategy enables fully agentic trajectories, and experiments report improvements over partially agentic approaches while releasing the supporting data and infrastructure.

  • Motivation: T2I models produce visually compelling images but often fail on open-world requests requiring complex reasoning, long-tail knowledge, factual constraints, or time-sensitive information.Such outputs can be visually plausible yet factually erroneous.
  • Limitations of Prior Work: Existing agentic methods either use fixed pipelines or delegate final synthesis to an external generator, leaving image generation outside the model’s freely chosen actions.These paradigms therefore control only part of the open-world generation decision process.
  • ToolArtist: ToolArtist unifies autonomous reasoning, external tool utilization, and native visual generation under a single Unified Multimodal Model policy.Given a request, it can decide whether and how to invoke tools, incorporate textual and visual evidence, and generate or revise images natively.
  • Post-training: The post-training strategy converts teacher-agent trajectories with concealed image-generation tools into UMM-compatible data and uses RAD-GRPO with intent and quality rewards during RL.The released dataset contains 7k high-quality SFT trajectories, alongside complete SFT and RL infrastructure.
  • Results and Resources: On WISE and WorldGenBench-Humanities, ToolArtist outperforms open-source UMMs and image-generation methods with only partial agentic capabilities.The authors release the training data and complete post-training infrastructure for agentic image generation research.

2 Preliminary

Conventional text-to-image generation assumes a closed world, whereas ToolArtist uses a unified multimodal model as a policy that reasons, searches, and natively draws through variable-length interactions with the open-world environment.

  • Closed-World T2I: Conventional T2I generation directly samples an image from a prompt-conditioned distribution, assuming required information is stated in the instruction or stored in the generator.This closed-world assumption breaks down when generation depends on external information acquired from the world.
  • Unified Multimodal Model: Emu3.5 represents language and images as tokens in one multimodal sequence, enabling ToolArtist to process observations and generate images within a shared autoregressive context.Instructions, reasoning, and tool calls are textual tokens, while input, retrieved, and generated images are visual-token spans.
  • Agentic Interaction: At each interaction round, the UMM conditions on the complete multimodal history, produces reasoning, and either calls TextSearch or ImageSearch or generates an image.Reasoning identifies missing knowledge, evaluates evidence, and determines whether to search or draw next.
  • Native Image Generation: Native drawing produces both an executable generation caption and visual tokens within the UMM, while the generated image remains in history for possible further inspection and correction.Unlike a prompt-optimization agent, ToolArtist does not delegate image generation to an external image generator.
  • Agentic Interaction: ToolArtist’s occurrence, ordering, and number of search and drawing actions are determined by evolving context rather than a fixed workflow.The policy terminates only when it decides that the latest drawing action is the final answer.
  • Agentic Image Generation Masking: Masking excludes user instructions and environment observations from the policy objective while directly optimizing the UMM’s reasoning, searching, and drawing behaviors.Reasoning and action tokens remain policy-supported, whereas conditioning and returned observations remain available only as context.

3 Training

ToolArtist is post-trained in two stages: SFT converts teacher-agent rollouts into unified multimodal supervision, while RAD-GRPO optimizes the complete reasoning, tool-use, and native image-generation policy with intent and quality feedback.

  • SFT: SFT uses teacher-agent multi-turn rollouts with text search, image search, and external image generation, then converts them into UMM-compatible training samples.The converter retains search evidence while rewriting external image-generation outputs into native multimodal generation supervision.
  • SFT: Unlike a fixed search-then-draw pipeline, rollout trajectories can alternate reasoning, tool calls, and further searches until reaching a final generation intent.The resulting trajectory contains reasoning, tool calls, tool responses, and the externally generated image.
  • SFT: SFT supervises only policy-generated textual and visual spans, teaching one multimodal trajectory that reasons, gathers evidence, captions the target, and generates image tokens natively.Conditioning prefixes and environment-returned observations are excluded from the objective.
  • RL with RAD-GRPO: RAD-GRPO directly optimizes complete open-world inference trajectories, whose reasoning, search, and visual-generation actions may be freely interleaved.SFT provides initialization, but RAD-GRPO addresses its limitation in discovering better agentic image-generation strategies.
  • RL with RAD-GRPO: RAD-GRPO combines intent and quality rewards: intent evaluates the final generation caption, while quality evaluates whether the decoded image satisfies the request and matches that caption.Quality uses faithfulness, visual correctness, text accuracy, and aesthetics, weighted 0.1, 0.4, 0.4, and 0.1, respectively.

4 Experiments

The experiments evaluate open-world image generation with WISE and WorldGenBench-Humanities, showing that the proposed model achieves strong knowledge-oriented performance. It reaches 0.79 overall on WISE and the best non-proprietary average KCS on WorldGenBench-Humanities.

  • Benchmarks: WISE measures world-knowledge integration across 1,000 prompts spanning 25 subdomains, grouped into cultural common sense, spatio-temporal reasoning, and natural science.Reported categories include Cultural, Time, Space, Biology, Physics, Chemistry, and Overall.
  • Benchmarks: WorldGenBench-Humanities evaluates reasoning-driven image generation across 244 countries and regions using 732 prompts and prompt-specific Knowledge Checklist Scores.The split is organized by continent and checks expected semantic points such as culturally appropriate clothing and local architecture.
  • Results: 0.79 overall: the model outperforms prior agentic image-generation models on WISE.Frontier proprietary image models remain ahead in several WISE categories, particularly Time and Space.
  • Results: 0.81 on Physics and 0.79 on Chemistry: the model is competitive on WISE’s knowledge-heavy natural-science categories.These results contrast with the stronger proprietary performance on several other WISE categories.
  • Results: 22.10 average KCS: the model achieves the best non-proprietary score on WorldGenBench-Humanities, compared with 21.76 for the sThe supplied passage truncates the identity of the comparison model after “the s”.

5 Analysis

The analysis shows that RAD-GRPO improves reward while gradually concentrating policy behavior, and that source-aware image summaries substantially improve semantic image-search performance. Qualitative evaluation and corpus statistics further illustrate grounded generation and diverse, multi-turn training trajectories.

  • Reward Dynamics: RAD-GRPO’s moving-average reward rises overall, increasing rapidly around Steps 10–20 before fluctuating between about 0.37 and 0.40.Raw reward fluctuates because it combines online search, Caption Reward, and Image Reward.
  • Policy Entropy Dynamics: Actor generation entropy decreases from approximately 12.03 to a stable range of about 11.87–11.91 during the latter half of training.The passage associates this decrease with concentration on higher-reward reasoning, search, and generation behaviors, without sudden instability.
  • Image Search Ablation: Source-aware image summaries improve WISE Overall from 0.61 to 0.79, helping prevent visually plausible but semantically wrong reference selection.The ablation removes source summaries while keeping the rest of the inference framework unchanged.
  • Qualitative Case Study: On the Socotra resin-collection case, our RL checkpoint obtains KCS 0.625 versus 0.000 for Unify-Agent and 0.250 for GenSearcher, satisfying five checklist points.The result includes traditional resin-harvesting implements, a clay-like container, and other cultural and geographic cues described in the passage.
  • SFT Corpus Analysis: The retained SFT corpus contains 7,132 trajectories, with mean and median input lengths of 20.5k and 20.2k tokens and mean and median tool calls both equal to 4.0.Its largest topic category is Geo/Architecture at 18.9%, and the corpus mainly comprises multi-turn search-and-generation trajectories with retrieved evidence and visual context.

6 Related Work

Related work spans unified multimodal models, tool-using agents for image generation, and reinforcement learning methods for visual generation. These approaches combine multimodal understanding, external evidence or tools, iterative generation, and group-relative optimization in different ways.

  • Unified Multimodal Model: Unified multimodal models combine visual understanding and generation within a shared model, unlike specialized generators that do not express reasoning, observations, and synthesis as one policy.Janus-Pro separates visual understanding and generation pathways while retaining a shared autoregressive backbone.
  • Tool-Using Agents and Agentic Image Generation: Agentic image-generation systems build on ReAct by adding external evidence, planning, iterative refinement, or tool orchestration to synthesis workflows.Unify-Agent uses a structured pipeline covering prompt understanding, multimodal evidence search, grounded recaptioning, and final synthesis.
  • Reinforcement Learning for Visual Generation: GRPO enables outcome-driven reasoning optimization by estimating relative advantages from groups of sampled outputs without a learned critic.UniGRPO jointly optimizes reasoning and image synthesis in one reasoning-then-generation round, while interleaved GRPO targets multimodal text–image outputs.

A Appendix

The appendix supplies additional implementation details and qualitative evidence for ToolArtist’s post-training procedure and analyses. It includes complete agent prompts and tool protocols, plus four additional WorldGenBench-Humanities cases.

  • The appendix adds implementation details and qualitative evidence for the post-training procedure and analyses presented in the main paper.
  • Section A.1: Section A.1 reports complete prompts for teacher-agent data synthesis and ToolArtist inference, including tool definitions and interaction protocols.
  • Section A.1: Section A.1 also specifies the native image-generation format used in the reported interactions.
  • Section A.2: Section A.2 extends the main-text comparison with four additional WorldGenBench-Humanities cases spanning different historical contexts.

A.1 Prompt Displays

The teacher-agent prompt defines a retrieval-augmented image-generation workflow that searches for textual information and reference images, synthesizes a visual prompt, and invokes image generation. It also requires multimodal review and iterative refinement when the generated result is unsatisfactory.

  • Agent workflow: The agent proactively searches web information and reference images, compiles a high-quality image-generation prompt, and calls the image-generation tool.The workflow places information retrieval, prompt construction, and image generation within one agent procedure.
  • Retrieval and references: For uncertain details, the prompt prioritizes text_search for textual descriptions, then image_search for reference images that the agent judges for quality and relevance.The agent selects suitable returned images rather than using references indiscriminately.
  • Prompt construction: Before drawing, the agent distills retrieved information into a visual description covering subject, scene, style, composition, color, material, action, and lighting.The prompt explicitly forbids dumping raw search results directly into the generation prompt.
  • Iterative refinement: After generation, the agent inspects the returned image and, if unsatisfied, adjusts the prompt or reference images and calls draw again until the result is satisfactory.Multiple rounds of tool calls are allowed, so generation is explicitly iterative rather than limited to one attempt.
  • Reference formatting: Reference images in draw prompts must use fixed indexed tokens such as [IMAGE1] and cannot be identified with natural-language expressions.The tokens correspond to the order of images supplied to the tool.

A.2 Comparisons

The comparison section presents four top-ranked WorldGenBench-Humanities cases where the RL checkpoint outperforms Unify-Agent and GenSearcher, including a historical Whitechapel scenario. Each figure compares Ours, Unify-Agent, and GenSearcher in that order.

  • A.2 Comparisons: Four top-ranked WorldGenBench-Humanities cases are selected where the RL checkpoint outperforms both Unify-Agent and GenSearcher.These cases are used for the comparisons.
  • A.2 Comparisons: Each figure compares Ours, Unify-Agent, and GenSearcher in that order.The ordering is consistent across the selected comparison figures.
  • A.2 Comparisons: Figure 7 presents a WorldGenBench-Humanities case about a working woman in Whitechapel, East London, in 1889.The figure includes the full evaluation prompt for this historical scenario.

WorldGenBench-Humanities Prompt

WorldGenBench-Humanities evaluates whether generated images combine historically specific social, material, and environmental cues across four open-world scenes. The RL checkpoint outperforms Unify-Agent and GenSearcher on the reported Whitechapel, Antofagasta, and Moorea cases while satisfying multiple checklist points.

  • Whitechapel case: Whitechapel: The RL checkpoint obtains KCS 0.600 versus 0.300 for Unify-Agent and 0.300 for GenSearcher, satisfying six checklist points.The prompt combines Victorian East End social cues with cobblestones, gaslights, coal smog, street occupations, police patrols, and soot-darkened facades.
  • Whitechapel case: Whitechapel: The model better grounds the scene in patched working-class dress, coal-smog atmosphere, and soot-darkened Victorian facades while preserving shared street, housing, and gaslight cues.
  • Antofagasta case: Antofagasta: The RL checkpoint obtains KCS 0.500 versus 0.300 for Unify-Agent and 0.200 for GenSearcher, satisfying five checklist points.The prompt requires port infrastructure and regional historical grounding, including iron cranes, labor clothing, steam transport, saltpeter bags, colonial buildings, desert surroundings, fog, and merchant-bank signage.
  • Antofagasta case: Antofagasta: The model better grounds the port scene in period labor clothing, colonial masonry harbor buildings, a distant steam freighter, desert surroundings, and morning fog.
  • Moorea case: Moorea: The RL checkpoint obtains KCS 0.400 versus 0.100 for Unify-Agent and 0.000 for GenSearcher, satisfying four checklist points.The prompt requires agricultural activity and Polynesian island grounding, including volcanic terrain, noni fruit, a rattan basket, tropical trees, orchard fences, storage sheds, and a Pacific view.
  • Moorea case: Moorea: The satisfied points include scattered ripe white noni fruit, a local flower garland, and a distant Pacific Ocean view, while several traditional and landscape cues remain missing.Missed points include traditional floral dress, a woven rattan basket, breadfruit and banana trees, stone fences, red laterite soil, and traditional storage sheds.
  • Santo Domingo case: Santo Domingo: The prompt combines traditional sugar production with the city’s historical setting, requiring a wooden cane press, colonial walls, vats, portraits, steam, and Caribbean-style exteriors.The passage also lists bagasse, arched doors and windows, bamboo sieves, and barrels as required details.
  • Santo Domingo case: Santo Domingo: The model better grounds the workshop in a wooden cane press, earthenware vats, visible sugar-cooker steam, and Caribbean-style exterior buildings.

A.3 Additional Trajectory Cases

This section presents an additional trajectory case using the same display format as the main-text case study. It omits the constant system prompt and shows extracted turns, retained observations, and the final generation output.

  • Trajectory presentation: The additional case follows the main-text case study’s display format, with the constant system prompt omitted.The presentation covers each remaining turn from the original trajectory.
  • Trajectory presentation: Reasoning is shown without <think> tags, while tool calls appear as tool names with query lists instead of raw JSON/XML wrappers.These formatting choices apply to content extracted from the original trajectory.
  • Trajectory presentation: Tool observations are retained, and the final generation caption and image appear at the end.The displayed sequence includes both intermediate observations and the concluding generation output.

A.4 SFT Data Examples

The SFT examples show agent trajectories that search for authoritative text and image references before invoking image generation, producing factually constrained technical and fictional scenes. The rollouts preserve reasoning, tool calls, references, prompts, and generated-image checks in a unified sequence.

  • Rollout format: SFT rollouts begin with the user request and preserve original text while replacing image placeholders with actual reference and generated images.This provides a UMM-compatible record of the multimodal interaction.
  • WGS 84 vs GRS 80: The WGS 84–GRS 80 example searches authoritative sources for exact semi-minor axes, the maintaining organization, and comparison-diagram references before drawing.The agent combines text search with image references to satisfy both factual and visual requirements.
  • WGS 84 vs GRS 80: The generated technical diagram lists WGS84 at 6356752.3142 m, GRS80 at 6356752.3141 m, and the shared semi-major axis at 6378137.0 m.The prompt specifies exaggerating the 0.105 mm semi-minor-axis difference for visibility while noting that it is not to scale.
  • Sigewinne and Bailu: The Sigewinne–Bailu example first verifies that Sigewinne’s official Vision is Hydro, then retrieves character, symbol, nursery, and potion image references before drawing.The final scene uses those references in a whimsical underwater nursery with a medical chart displaying Hydro.
  • Sigewinne and Bailu: The resulting Sigewinne–Bailu image is judged consistent with the requested anime style, correctly depicting both characters and showing Hydro on the medical chart.The rollout concludes that the generated image meets all requirements.
Loading 2608.04436v1…