PaperlayerView exact revision on arXiv

Source-linked AI summary

CanvasAgent: Enabling Complex Image Creation and Editing via Visual Tool Orchestration

Hairui Zhu, Yiying Yang, Tengjin Weng, Ziyu Lu, Xiao Yao, Xiaoyang Ye, Lin Ma, Wenhao Jiang

arXiv:2607.05465v1cs.CVcs.AI

TL;DR

Complex image-editing requests require coordinated multi-step tool use beyond a single model call. CanvasAgent learns this orchestration from CanvasCraft and SFT+RL training, reaching 0.821 overall reward and 0.849 trajectory score.

  • Problem

    Complex image creation and editing requires coordinating heterogeneous operations as executable trajectories rather than using a single prompt-to-image or instruction-to-image call.

  • Method

    CanvasAgent learns multi-turn visual tool orchestration from CanvasCraft’s annotated SFT trajectories and RL task specifications using a two-stage SFT+GRPO framework with hybrid rewards.

  • Results

    0.821 overall reward, 0.869 alignment, 0.762 aesthetics, and 0.849 trajectory score make SFT+RL the best training strategy across all reported metrics.

  • Takeaways & Limitations

    The results support executable multi-step tool trajectories as a basis for improving final image quality and trajectory reliability in complex visual workflows.

  • Takeaways & Limitations

    CanvasAgent uses a fixed set of 11 tools, relies on an external MLLM judge, and requires real tool execution during RL rollout.

Abstract

from arXiv · show

Complex image creation and editing often require more than a single generation or editing model. A user request may involve synthesizing images, localizing objects, segmenting regions, editing selected content, compositing intermediate assets, reading text, and enhancing the final result. Such tasks shift multimodal agents from perception-augmented reasoning to manipulation-centered visual creation, where tools must actively transform visual states rather than merely inspect them. However, existing multimodal tool-use agents are mostly optimized for perception, search, or domain-specific editing, and lack large-scale supervision for executable image-creation trajectories. In this paper, we introduce CanvasCraft, a large-scale multimodal tool-use dataset for complex image creation and editing, and \textbf{CanvasAgent}, a tool-augmented multimodal agent that learns to orchestrate heterogeneous visual tools through multi-turn interaction. CanvasCraft contains 140K fully annotated executable trajectories and 10K RL task specifications. CanvasAgent is first trained with SFT to learn executable reasoning-action trajectories, and is then optimized with GRPO using a hybrid reward that combines outcome- and process-level signals. During rollout, CanvasAgent inspects intermediate results, tracks visual assets, and adapts tool decisions to the evolving visual state. Experiments evaluate both final image quality and trajectory behavior, demonstrating the effectiveness of CanvasAgent and the proposed dataset for complex multi-tool image creation workflows.

1 Introduction

Complex image creation and editing require long-horizon, visually grounded, stateful orchestration of heterogeneous tools rather than a single model call. The paper introduces CanvasCraft and CanvasAgent to learn and execute these multi-turn workflows through annotated trajectories, reinforcement learning, and hybrid rewards.

  • Motivation: Practical image requests often chain generation, localization, segmentation, regional editing, compositing, OCR, cropping, and enhancement beyond a single model call.These operations may involve synthesizing scenes, modifying selected regions, adding content, and improving resolution.
  • Problem formulation: Complex workflows are long-horizon, visually grounded, and stateful because later actions depend on inspected intermediate results and correctly selected visual assets.Multiple images, masks, crops, extracted objects, and edited variants may coexist during execution.
  • Research gap: Existing multimodal tool-use agents mainly target understanding, search, perception, or reasoning, leaving complex image-creation workflows insufficiently addressed.Prior systems cover parts of multi-step reasoning and editing but not the full manipulation-centered setting.
  • CanvasCraft: CanvasCraft provides fully annotated execution trajectories for supervised learning and task specifications with expected tool sets for reinforcement learning.The SFT subset includes reasoning, tool calls, parameters, outputs, intermediate artifacts, and final images, while the RL subset supports reinforcement-learning optimization.
  • CanvasAgent: CanvasAgent uses two-stage SFT+GRPO training and a hybrid reward to orchestrate heterogeneous visual tools while optimizing final-image quality and tool-use processes.The reward combines image-prompt alignment, visual or aesthetic quality, reasoning and trajectory validity, rule-based executability checks, and efficiency penalties.

2 Related Work

Prior work has advanced tool-augmented multimodal agents, individual image-generation and editing operations, and multi-step photo-retouching systems. CanvasAgent addresses the remaining need for open-ended image workflows by learning tool-orchestration policies with SFT, GRPO, and hybrid outcome/process rewards.

  • Tool-Augmented Multimodal Agents: Tool-augmented multimodal agents extend language and multimodal models with visual foundation models or expert modules for planning, execution, and programmatic visual reasoning.ReAct formalizes a reason-action-observation pattern, while recent systems add active visual inspection, search, and executable reasoning.
  • Image Editing and Creation Workflows: Image-generation and editing models provide strong individual operations but typically execute one instruction per model call without explicitly managing multi-step tool dependencies or intermediate visual assets.Latent diffusion enables high-quality synthesis, while instruction-guided editors improve controllable editing, generation, and text rendering.
  • Image Editing and Creation Workflows: Photo-retouching agents coordinate editing over multiple steps, but their environments primarily target photo retouching rather than open-ended image creation and editing workflows.JarvisArt controls Lightroom operations, and JarvisEvo studies a self-evolving edit-evaluate-reflect loop.
  • Learning Tool-Orchestration Policies: Supervised fine-tuning teaches tool invocation schemas and reasoning-action formats, whereas reinforcement learning optimizes tool-use policies beyond static demonstrations.Prior reinforcement-learning approaches include PPO and GRPO for search, visual reasoning, and tool-use systems.
  • Learning Tool-Orchestration Policies: CanvasAgent uses a two-stage SFT+GRPO framework and a hybrid reward to optimize visual reasoning, image quality, trajectory validity, and robust tool-use behavior.The optimization problem requires selecting tools, setting parameters, tracking intermediate assets, and judging whether the current visual result is sufficient.

3 Method

Section 3 presents CanvasCraft and the two-stage SFT+RL training framework for CanvasAgent. During execution, the agent reasons over visual states and intermediate assets to adaptively orchestrate tool calls using evolving visual feedback.

  • Data construction: CanvasCraft comprises CanvasCraft-SFT for fully annotated multi-step trajectories and CanvasCraft-RL for diverse task-level specifications used in reinforcement learning.The dataset is constructed for complex image creation and editing.
  • Training framework: CanvasAgent is trained in two stages, combining supervised fine-tuning with reinforcement learning.The framework is described as SFT+RL training.
  • Execution: During execution, CanvasAgent perceives the current visual state, reasons over intermediate assets, and plans subsequent tool calls.These decisions are guided by user instructions and evolving visual feedback, enabling adaptive multi-tool orchestration.

3.1 CanvasCraft Dataset and Construction

CanvasCraft is a diverse dataset for complex visual tool orchestration, combining fully annotated SFT trajectories with weakly supervised RL task specifications. Its construction covers heterogeneous tools, multi-step dependencies, and varying reasoning, trajectory-length, and tool-diversity demands.

  • Tool Set: The unified toolkit comprises 11 heterogeneous visual tools spanning generation, editing, grounding, segmentation, extraction, compositing, cropping, OCR, rotation, flipping, and super-resolution.Each tool exposes structured inputs and outputs through a low-level JSON-style schema for multi-step composition.
  • CanvasCraft-SFT: CanvasCraft-SFT records step-level reasoning, tool calls, structured parameters, intermediate assets, and final outputs from executable multi-step trajectories.This supervision teaches valid tool invocation, asset referencing, and cross-tool dependencies, ranging from single-tool operations to complex multi-turn workflows.
  • CanvasCraft-RL: CanvasCraft-RL represents each task with an instruction, optional input images, and an expected tool set, leaving tool ordering, parameterization, verification, and stopping strategies for rollout exploration.Tasks are organized by reasoning difficulty, trajectory length, and tool diversity, with emphasis on medium and hard levels.
  • Complementary Supervision: The two subsets provide complementary supervision: SFT teaches executable tool-use patterns through complete trajectories, while RL enables flexible planning and dynamic multi-tool orchestration under task-level supervision.This division encourages task decomposition and state-dependent planning during reinforcement learning.
  • Dataset Overview: CanvasCraft contains 140K SFT trajectories, 10K RL task specifications, and a manually curated 250-sample evaluation benchmark.The dataset supports supervised tool-use bootstrapping and reinforcement-learning policy optimization.

3.2 Two-Stage Training Framework

CanvasAgent uses a two-stage SFT+GRPO framework: SFT bootstraps executable tool-use behavior from expert trajectories, while GRPO optimizes multi-tool planning through relative-reward exploration under weak supervision.

  • Framework Overview: The two-stage design combines trajectory-imitation reliability with reinforcement-learning policy-exploration flexibility for multi-tool planning.SFT provides stable tool-use behavior, and GRPO optimizes rollout-level planning using a task-specific hybrid reward.
  • SFT Stage: SFT trains valid reasoning-action formats, JSON-style tool calls, parameter generation, cross-tool dependencies, and intermediate visual-asset references from complete expert trajectories.This initialization stabilizes reinforcement learning by reducing invalid tool calls, unstable rollouts, and sparse-reward problems.
  • GRPO Stage: GRPO samples multiple executable rollouts per task and updates the policy by relative rewards without requiring an additional value model.The method encourages exploration of alternative task decompositions and tool-use strategies under weak supervision.

3.3 Hybrid Reward Design

The hybrid reward evaluates both final-image quality and execution trajectories for difficult multi-step visual tasks. It combines semantic, perceptual, procedural, symbolic, and efficiency signals to provide rich supervision while reducing reward hacking.

  • Reward composition: The reward combines outcome and process scores as R(τ) = 0.3 · Ralign(τ) + 0.1 · Raes(τ) + 0.2 · Rtraj(τ) + 0.4 · Rrule(τ).Outcome scores use expert LLM-as-judge evaluators, while process scoring evaluates trajectory behavior.
  • Outcome scores: Alignment scores instruction satisfaction across requested objects, attributes, actions, relations, colors, text, quantities, and edited regions, while aesthetic scores assess perceptual image quality.Aesthetic evaluation covers composition, color, lighting, clarity, texture naturalness, style consistency, artifacts, and blur, independently of alignment and trajectory quality.
  • Process scores: Trajectory scores judge task decomposition, tool selection, dependency validity, and avoidance of irrelevant operations without directly judging the final image.The trajectory and expected tool set are provided to the judge as inputs.
  • Rule-based reward: Rule-based rewards combine format, action, and efficiency signals to enforce executable reasoning-action protocols and compatibility with the evolving visual asset state.They check structured syntax, executable tool calls, valid arguments and asset references, operation-specific constraints, successful execution, and state updates.
  • Efficiency control: Efficiency penalties discourage failed, repeated, overly long, excessively costly, or incomplete tool trajectories, encouraging concise and task-relevant orchestration.The penalties cover Perror, Prepeat, Plength, Pcost, and Pmiss, preventing score gains from blindly invoking more tools.
  • Overall design: Together, the hybrid reward balances semantic alignment, visual aesthetics, reasoning validity, rule adherence, and execution efficiency while reducing reward hacking.It combines semantic, perceptual, procedural, and symbolic constraints for robust multi-step visual creation.

4 Experiments

Experiments on 250 CanvasCraft-RL samples show that SFT establishes executable tool use, while SFT+RL with hybrid rewards substantially improves image quality and trajectory behavior. Qualitative and human evaluations further support closed-loop orchestration and strong instruction satisfaction.

  • Evaluation setup: The evaluation uses 250 CanvasCraft-RL samples and reports Overall Reward, Alignment Score, Aesthetic Score, Trajectory Score, Rule-based Score, and Trajectory Length.The samples contain user instructions, optional input images, and reference tool sets used only for reward computation.
  • Overall results: CanvasAgent (SFT) raises Overall Reward from 0.426 to 0.557 and Trajectory Score from 0.092 to 0.576 versus Qwen3-VL-8B-Instruct, but averages 1.320 tool calls against 3.592 expected.SFT teaches structured reasoning–action formats and executable trajectories but initially underuses the tool set.
  • Overall results: 0.821 Overall Reward, 0.869 Alignment Score, 0.849 Trajectory Score, and 0.785 Rule-based Score are achieved by CanvasAgent (SFT+RL), with 5.436 average tool calls.Compared with SFT, RL increases tool calls from 1.320 to 5.436, encouraging richer multi-step visual manipulation.
  • Ablation study: SFT+RL performs best across training strategies, reaching 0.821 Overall Reward, 0.869 Alignment, 0.762 Aesthetics, and 0.849 Trajectory Score.RL from scratch reaches 0.604 Overall Reward but lowers Alignment and Aesthetics to 0.472 and 0.666, indicating unstable exploration without SFT initialization.
  • Ablation study: The full hybrid reward balances outcomes and process: removing outcome reward leaves Trajectory Score at 0.907 but reduces Alignment and Aesthetics to 0.320 and 0.565, while removing process reward yields 0.379 Overall Reward and 0.357 Trajectory Score.During RL, tool calls first increase and then stabilize or decrease, suggesting a shift from exploration toward efficient orchestration.
  • Qualitative analysis: A representative multi-image editing case shows CanvasAgent managing intermediate assets and using grounding and OCR observations to guide subsequent editing and compositing decisions.The trajectory combines perception, extraction, editing, OCR, and compositing rather than producing the final image in one step.
  • Human evaluation: CanvasAgent achieves the best human-evaluation scores across Task Alignment, Key Details Alignment, and Aesthetic Quality on 12 evaluation samples.These dimensions correspond to the judge-based measures of instruction satisfaction, semantic/text correctness, and aesthetics.

5 Conclusion

CanvasAgent frames complex image creation and editing as executable multi-step tool orchestration rather than a single generation call, supported by CanvasCraft’s large-scale supervision. The paper also identifies fixed tooling, external judging, and costly RL execution as limitations while outlining future extensions.

  • CanvasAgent decomposes open-ended visual requests into executable multi-step tool trajectories instead of treating image generation as a single black-box call.
  • CanvasCraft provides 140K SFT trajectories and 10K curated RL task specifications spanning editing, composition, perception, and verification behaviors.
  • Limitations and future work: CanvasAgent currently uses 11 fixed tools, relies on an external MLLM judge, and requires real tool execution during RL rollout.
  • Limitations and future work: Future work will study dynamic tool discovery, learned or self-evaluation rewards, efficient rollouts, user feedback, self-improvement, and video creation and editing.

A LLM-as-Judge Prompts Details

The appendix specifies separate LLM-as-judge prompts for alignment, aesthetic quality, and trajectory reasonableness, with each evaluator restricted to its designated criterion. It also notes that CanvasAgent’s controllable visual creation benefits must be balanced against potential misuse through responsible release and dataset safeguards.

  • A.1 Prompt for Alignment Score: The alignment judge evaluates only whether the final output satisfies the user’s visual request, considering semantic correctness, requested elements, spatial relations, text, preservation, and visual fidelity.It receives the task prompt, available input and output images, and the final response or trajectory only as auxiliary context.
  • A.1 Prompt for Alignment Score: Alignment scoring uses a 0.0–1.0 rubric ranging from unrelated or empty outputs to complete satisfaction of important constraints, with partial and semantically weak results assigned lower bands.The judge is instructed to be conservative and choose the lower band when uncertain.
  • A.2 Prompt for Aesthetic Score: The aesthetic judge scores only the final image’s visual quality, including composition, color and lighting, clarity, natural boundaries, stylistic coherence, and visible artifacts.Semantic correctness and trajectory quality are explicitly evaluated elsewhere.
  • A.3 Prompt for Trajectory Score: The trajectory judge evaluates whether the process is reasonable through logical tool selection, valid dependencies, correct use of prior outputs, concise reasoning, and verification before termination.It penalizes irrelevant tools, hallucinated IDs, invalid dependencies, blind repetition, and unsupported claims; tool order need not be mandatory when an equivalent route works.
  • A.3 Prompt for Trajectory Score: A single-tool-call trajectory must receive 0.0 because it does not demonstrate verification, reassessment, refinement, or meaningful multi-step tool use.Weak follow-up checks, missing expected tools, premature completion, ignored errors, and unsupported success claims also trigger conservative score caps.
  • A.5 Broader Impacts: CanvasAgent may reduce manual effort in design, prototyping, and image-editing workflows by decomposing complex instructions into interpretable tool-use trajectories.The appendix also warns that stronger visual generation and editing could enable deceptive or harmful content, motivating responsible release, dataset filtering, and documentation of intended uses and limitations.
Loading 2607.05465v1…