Source-linked AI summary
Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents
Shudong Liu, Dongyang Chen, Enci Zhang, Jinwei Liang, Zheng Ma, Lewei Lu
TL;DR
Existing agent benchmarks rarely test whether visual evidence can be converted into precise actions whose parameters directly determine outcomes. EASEL addresses this gap with closed-loop reconstruction and semantic drawing tasks, plus trajectory supervision, and finds broad instability in current multimodal agents while EASEL-9B improves over its base model.
Problem
Existing tool-mediated benchmarks rarely require precise coupling between visual evidence and execution parameters, leaving dexterous visual tool use underexplored.
Method
EASEL benchmarks closed-loop reference reconstruction and instruction-conditioned annotation, handwriting, and path planning, and trains EASEL-9B with two-stage trajectory supervision.
Results
Current agents show early saturation, post-peak degradation, and sharp semantic-task boundaries, while EASEL-9B improves over the base model by a relative 6.3%.
Takeaways & Limitations
EASEL exposes closed-loop visual-action failures that answer-only QA and high-level tool-use benchmarks can obscure.
Takeaways & Limitations
EASEL trades breadth for control by isolating visual goals, parameterized actions, and feedback on a 2D canvas.
Abstract
from arXiv · showhide
Evaluation is shifting from static QA toward agentic settings where models act through external tools. We identify a critical yet underexplored capability within this space - dexterous visual tool use: fine-grained, closed-loop parameterized visual action in which models infer tool parameters from visual evidence, and those parameters directly govern the final result. Existing benchmarks cover web navigation, GUI operation, and software engineering, but rarely target this coupling between visual evidence and execution precision. We propose EASEL, a benchmark evaluating a controlled instance of dexterous visual tool use that adopts reference-guided visual reconstruction as its primary proxy task: the agent incrementally paints a canvas to match a reference image. EASEL additionally includes semantic tasks spanning region annotation, handwriting, and path planning. We further provide EASEL-Data, a 440k-sample two-stage curriculum dataset for trajectory supervision, and EASEL-9B to investigate its effect on this capability. Evaluation of 25 models reveals that current multimodal agents systematically struggle on EASEL. Reconstruction similarity bottlenecks at low levels (0.40-0.54), while trajectory diagnostics expose severe closed-loop instability - models typically saturate early or degrade post-peak. Semantic tasks reveal sharp capability boundaries in precision annotation and path planning. EASEL-9B, trained on EASEL-Data, surpasses the base model by a relative 6.3%, ranking third among all evaluated models.
1 Introduction
EASEL targets dexterous visual tool use, requiring agents to convert fine-grained visual evidence into parameterized actions whose effects are refined through feedback. It benchmarks this capability with reconstruction and semantic tasks, and studies trajectory supervision for closed-loop action.
- Existing tool-use benchmarks generally tolerate spatial imprecision, so precise visual execution is rarely required.
- EASEL defines dexterous visual tool use as fine-grained, closed-loop parameterized action in which visual cues determine tool parameters and updated states guide refinement.
- Painting provides a scalable proxy because each stroke visibly changes the canvas, exposes parameter errors, and requires no manual annotation.
- EASEL evaluates reference-guided reconstruction alongside instruction-conditioned region annotation, handwriting, and path-planning tasks.
- The benchmark contains 110 reconstruction samples and 5 instruction-conditioned semantic tasks, with up to 11,392 interactions per full evaluation.
- EASEL-Data provides 440k next-action supervision samples through a two-stage curriculum, while evaluation identifies early saturation, post-peak degradation, and semantic capability boundaries.
2 Related Work
Prior visual-agent work supports understanding, navigation, editing, or task-specific stroke generation, but generally does not require general-purpose agents to refine parameterized actions toward an explicit visual goal. EASEL distinguishes itself by making action precision directly determine the visual result.
- Agent benchmarks cover software engineering, web and GUI navigation, and long-horizon visual tool use, while GUI grounding usually targets high-tolerance coordinates.
- Image-based agent research includes visual search, region annotation, drawable workspaces, program invocation, and segmentation-capable models.
- Visual editing and design benchmarks connect language-guided models to image-editing systems or real design software, but EASEL makes parameter precision directly determine result quality.
- Neural painting and stroke-rendering methods optimize reconstruction with task-specific models, whereas EASEL uses reconstruction diagnostically for general-purpose agents under one tool interface.
- Unlike related drawing systems, EASEL focuses on closed-loop refinement toward an explicit visual goal.
3 EASEL
EASEL is a closed-loop drawing benchmark in which agents reconstruct references or satisfy semantic instructions through parameterized strokes. It evaluates both final outputs and intermediate trajectories, using task-specific semantic metrics and trajectory diagnostics.
- Benchmark tasks: EASEL reconstruction repeatedly pairs a reference image and current canvas with a brush_stroke action that updates the canvas toward the reference.
- Benchmark tasks: Semantic tasks cover region outlining, handwriting, and maze paths, with fixed brush appearance and optional undo or submit actions.
- Benchmark construction: The benchmark includes 110 reconstruction samples and 5 semantic tasks across five categories.
- Benchmark construction: The reconstruction categories vary geometric structure, color distribution, and semantic prior strength, while semantic tasks include two outline tasks, handwriting, and two maze tasks.
- Evaluation: Result Quality measures final visual satisfaction, while Trajectory Quality diagnoses how intermediate feedback is used across the rollout.
- Evaluation: Reconstruction similarity combines SSIM, normalized RGB L1 distance, and Edge IoU, with SSIM receiving primary weight.
- Evaluation: Trajectory diagnostics include Similarity@50%, Trajectory AUC, Best Similarity, and Final-Best Gap.
- Evaluation: Semantic scores use Dice for outlines, positional letter matching for handwriting, and normalized route progress for maze tasks.
4 EASEL-Data
EASEL-Data converts reference images and generated painting trajectories into next-action supervision through a two-stage curriculum. The resulting pipeline supports training for coarse reconstruction first, followed by mid-to-late refinement.
- Two-stage curriculum: EASEL-Data is constructed through reference preparation, vector trajectory generation, and two-stage SFT construction.The pipeline is summarized as three stages.
- Reference and trajectory pools: 11k reference images span procedurally generated geometry and glyphs, COCO crops, and Tiny ImageNet natural scenes.The pool covers visual complexity from simple geometry to complex natural images.
- Reference and trajectory pools: Each reference image receives a 250-step brush_stroke trajectory generated by a stroke-based painting policy.A budget-aligned variant uses ⌈B/5⌉ policy steps with five raster strokes each.
- Next-action supervision: Next-action supervision pairs the reference and replayed current canvas with the next stroke predicted by the model.Each target is one observed continuation from the rendering policy.
- Two-stage curriculum: C1 contains 308k early-reconstruction samples, while C2 contains 132k light-completion samples spanning the full trajectory at decreasing density.C1 emphasizes blank-canvas coverage and coarse structure and color; C2 covers mid-to-late refinement.
5 Experiments
Experiments evaluate 25 multimodal agents on reconstruction, trajectory behavior, and semantic tasks. Results show low and stratified reconstruction performance, early closed-loop failure patterns, measurable curriculum gains, and sharper boundaries on semantic tasks.
- Evaluation setup: 25 multimodal agents are evaluated, including closed-source APIs and open-source models ranging from 4B to 90B parameters.The set covers proprietary and open-source model families across the current performance frontier.
- Trajectory diagnostics: Most closed-source models reach their effective peak within the first 10% of the step budget, then either plateau or degrade.Plateau models stop making meaningful changes, whereas degrading models continue acting while reducing final quality.
- Standard reconstruction: The data policy reaches 0.665 under the budget-aligned setting and 0.715 in the data-native setting, exceeding every evaluated agent.Uniform fills score zero on Edge IoU, while the data policy provides attainable-score anchors.
- Standard reconstruction: Gemini 3.1 Pro achieves the best Final Similarity (0.535), followed by Claude Opus 4.7 (0.472), while open-source models cluster in the 0.40–0.45 range.EASEL-9B reaches 0.459, ranking third overall and outperforming Qwen3.5-9B (0.432) by +0.027.
- Trajectory diagnostics: The trajectory gap between top closed-source models and the data policy indicates that sustained feedback-driven correction remains the main bottleneck.The reported gap is framed as a limitation in long-horizon correction rather than initial perception.
- Trajectory supervision: EASEL-9B reaches 0.456 with C1-only training and 0.459 with the full C1+C2 curriculum, compared with 0.432 for the base model.C1 provides most of the gain, while C2 adds a further +0.003.
- Semantic task analysis: Semantic tasks expose sharper capability boundaries than reconstruction: top models lead outline tasks and handwriting, GPT-5.5 solves Simple Maze perfectly, and Maze Path remains unsolved.Claude Opus 4.7 fails Circle Outline through persistent draw-undo loops.
- Semantic task analysis: GPT-5.5 demonstrates iterative correction on Simple Maze, submitting after 25 undos with a perfect score.Gemini 3.1 Pro and GPT-5.5 use undo mainly on maze tasks, unlike Claude Opus 4.7’s non-maze undo pattern.
6 Discussion
EASEL isolates fine-grained visual manipulation in a controllable 2D canvas while probing breadth across semantic tasks. Training on EASEL-style trajectories largely preserves standard perception performance and modestly broadens fine-grained multimodal perception.
- Scope and proxy validity: EASEL trades breadth for control by isolating visual goals, parameterized actions, and closed-loop feedback on a 2D canvas.This protocol supports focused diagnosis of fine-grained manipulation and closed-loop correction.
- Applications: The semantic tasks extend the interface beyond reconstruction to annotation, handwriting, and path planning.The setting uses explicit, editable strokes rather than delegated specialized or generative execution.
- Generalization of trajectory supervision: EASEL-9B matches or modestly improves the base model on standard perception benchmarks after trajectory training.Reported changes include MMVP +0.7%, POPE +2.0%, HallusionBench +0.6%, BLINK spatial reasoning +0%, and counting +1.7%.
7 Conclusion
EASEL benchmarks precision-critical visual actions through reference-guided reconstruction, semantic tasks, and trajectory diagnostics. Current models show early plateaus, post-peak degradation, and limited corrective-feedback use, while EASEL-9B improves over its base model by 6.3% relatively.
- Benchmark: EASEL operationalizes dexterous visual tool use as precision-critical, closed-loop parameterized visual action.Its evaluation combines reference-guided reconstruction, instruction-conditioned semantic tasks, and trajectory diagnostics.
- Findings: Models plateau early, degrade after initial gains, and rarely exploit corrective feedback on EASEL.These trajectory failures are described as weaknesses obscured by answer-only QA and high-level tool-use benchmarks.
- Training outcome: EASEL-9B improves over the base model by a relative 6.3%.The conclusion attributes most of the gain to early-phase training, while long-horizon feedback correction remains limited.