PaperlayerView exact revision on arXiv

Source-linked AI summary

Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents

Shudong Liu, Dongyang Chen, Enci Zhang, Jinwei Liang, Zheng Ma, Lewei Lu

arXiv:2608.25417v1cs.AI

TL;DR

Existing tool-use benchmarks rarely demand precise spatial control, leaving dexterous visual tool use underexplored. EASEL evaluates this capability through reference-guided painting and semantic tasks; across 25 agents it finds early saturation or post-peak degradation, while EASEL-9B improves over its base by a relative 6.3%.

  • Problem

    Existing tool-use benchmarks rarely demand precise spatial control, leaving limited evaluation of agents’ ability to translate visual evidence into precise execution parameters.

    Source: §1 · p. 1

  • Method

    EASEL benchmarks dexterous visual tool use through incremental reference-guided canvas reconstruction and instruction-conditioned annotation, handwriting, and path-planning tasks.

    Source: §1 · p. 2

  • Results

    Across 25 agents, trajectories show early saturation or post-peak degradation and semantic tasks expose sharp capability boundaries; EASEL-9B improves over its base by a relative 6.3%.

    Source: §1 · p. 2, §7 · p. 7

  • Takeaways & Limitations

    The paper concludes that long-horizon feedback correction remains limited despite EASEL-9B’s training gains.

    Source: §7 · p. 7

    EASEL trades breadth for control by using a 2D canvas to isolate visual goals, parameterized actions, and closed-loop feedback.

    Source: §6 · p. 7

Abstract

from arXiv · show

Evaluation is shifting from static QA toward agentic settings where models act through external tools. We identify a critical yet underexplored capability within this space - dexterous visual tool use: fine-grained, closed-loop parameterized visual action in which models infer tool parameters from visual evidence, and those parameters directly govern the final result. Existing benchmarks cover web navigation, GUI operation, and software engineering, but rarely target this coupling between visual evidence and execution precision. We propose EASEL, a benchmark evaluating a controlled instance of dexterous visual tool use that adopts reference-guided visual reconstruction as its primary proxy task: the agent incrementally paints a canvas to match a reference image. EASEL additionally includes semantic tasks spanning region annotation, handwriting, and path planning. We further provide EASEL-Data, a 440k-sample two-stage curriculum dataset for trajectory supervision, and EASEL-9B to investigate its effect on this capability. Evaluation of 25 models reveals that current multimodal agents systematically struggle on EASEL. Reconstruction similarity bottlenecks at low levels (0.40-0.54), while trajectory diagnostics expose severe closed-loop instability - models typically saturate early or degrade post-peak. Semantic tasks reveal sharp capability boundaries in precision annotation and path planning. EASEL-9B, trained on EASEL-Data, surpasses the base model by a relative 6.3%, ranking third among all evaluated models.

1 Introduction

EASEL evaluates dexterous visual tool use through closed-loop visual reconstruction and semantic tasks, and introduces trajectory supervision for training. Evaluation of multimodal agents reveals instability in closed-loop action and capability boundaries on semantic tasks.

Motivation

  • Prior evaluations and visual methods tolerate spatial error, produce understanding without persistent output, or delegate execution; sustained perception in multi-step closed-loop action remains untested.Painting offers a proxy in which each stroke changes the canvas and shapes the next decision, while parameter errors are not absorbed by a tolerant interface or external generator.Source: §1 · pp. 1–2

Capability definition

  • Dexterous visual tool use means inferring explicit parameters from pixel-level cues, directly changing the output, and refining actions from each updated state rather than delegating execution.The challenge is continuously translating fine-grained visual signals into corrective actions, not achieving per-step perfection.Source: §1 · pp. 1–2

EASEL benchmark

  • EASEL evaluates 110 reference-guided reconstructions and five instruction-conditioned semantic tasks, with up to 11,392 interactions per full evaluation.The semantic tasks include annotation, handwriting, and path planning; painting provides visible feedback for the next action, and trajectories can be derived automatically without manual annotation.Source: §1 · p. 2

Trajectory supervision

  • EASEL-Data supplies 440k next-action samples through a two-stage curriculum—early reconstruction (C1), then light completion (C2)—used to train EASEL-9B.The training investigates the effect of trajectory supervision on closed-loop visual action.Source: §1 · p. 2

Evaluation findings

  • Across 25 multimodal agents, evaluation reveals early saturation, post-peak degradation, and semantic-task capability boundaries missed by reconstruction metrics alone.The evaluation considers both result quality and trajectory quality.Source: §1 · p. 2

2 Related Work

Prior work spans interactive agent benchmarks and image-based tools, but typically lacks EASEL’s coupling of precise visual parameters to result quality. EASEL evaluates general-purpose agents through closed-loop reconstruction toward an explicit visual goal.

2.1 Agent Evaluation and Visual Tool Use

2.2 Parameterized Drawing and Generation

  • Learning to Paint and Paint Transformer optimize task-specific stroke sequences for reconstruction, whereas EASEL uses reconstruction diagnostically to evaluate general-purpose agents through a unified tool interface (Huang, Heng, and Zhou 2019; Liu et al. 2021).Draw with Thought produces scientific diagrams as structured code (Cui et al. 2025), while SketchAgent creates sketches from language descriptions (Vinker et al. 2025); neither targets closed-loop refinement toward an explicit visual goal.Source: §2.2 · pp. 2–3

3 EASEL

EASEL evaluates closed-loop visual tool use through reference-guided reconstruction and instruction-guided semantic drawing. Its benchmark specifies task construction, stroke actions, and separate measures for final results and trajectories.

3.1 Task Formulation

  • At each reconstruction step, the agent observes the reference and current canvas, outputs a JSON brush_stroke, and aims to make the final canvas visually match the reference.Semantic tasks use an image and text instruction for outlining, handwriting, or maze-path drawing; maze-wall collisions are rejected as no-ops but consume a step and provide feedback.Source: §3.1 · p. 3

3.2 Benchmark Construction

  • EASEL includes 110 reconstruction samples and five semantic tasks: two region outlines, handwriting “hello,” and simple and harder maze paths.Reconstruction categories vary in geometric structure, color distribution, and semantic-prior strength; semantic tasks use task-image canvases for annotation and mazes, but a blank canvas for handwriting.Source: §3.2 · p. 3

3.3 Evaluation Metrics

  • Reconstruction evaluation uses Final Similarity as its primary ranking metric, alongside trajectory diagnostics; semantic tasks use task-specific scores and log action traces for failure analysis.Semantic scores use Dice for outlines, letter recognition and positional match for handwriting, and normalized route progress for mazes; trajectory diagnostics include midpoint similarity, AUC, best similarity, and Final-Best Gap, which must be interpreted with Final Similarity.Source: §3.3 · pp. 3–4

3.4 Tool Semantics

  • The reconstruction brush_stroke exposes 13 continuous normalized parameters for a quadratic Bézier path and stroke appearance, while semantic tasks fix appearance and predict only six path coordinates.Path and appearance values are interpolated or composited by the renderer; semantic brush strokes use fixed opaque dark-blue, medium-width appearance.Source: §3.1 · p. 3, §3.4 · p. 4

4 EASEL-Data

EASEL-Data is constructed through a three-stage pipeline.

4 EASEL-Data

  • Its construction comprises reference image preparation, vector trajectory generation, and two-stage SFT construction.The pipeline is summarized in Figure 3.Source: §4 · p. 4

4.1 EASEL-9B

EASEL-9B is a two-stage fine-tuning probe designed to diagnose the effect of the EASEL curriculum, not to set an upper bound on painting quality.

4.1 EASEL-9B

  • EASEL-9B fine-tunes Qwen3.5-9B with LoRA, training first on C1 and then on C2 with 15% C1 replay.The stages use learning rates 1 × 10−4 and 5 × 10−5, respectively; the ViT’s last four blocks and merger layers receive full gradient updates. Both stages use bf16, effective batch size 64 across 8 GPUs, and a maximum sequence length of 8192.Source: §4.1 · p. 5

5 Experiments

Across 25 multimodal agents, EASEL exposes weak reconstruction and unstable closed-loop behavior, alongside sharp differences in semantic-task precision and path planning. EASEL-9B improves on its base model, but sustained feedback-driven correction remains limited.

5.1 Standard Reconstruction

  • 0.665 composite similarity (Edge IoU 0.281) budget-aligned and 0.715 (Edge IoU 0.563) data-native data-policy scores exceed every agent, while Gemini’s 0.535 matches mean-color fill at 0.534.The span from uniform fills (Edge IoU 0) to agents and the data policy indicates that the composite metric remains discriminative above current agents’ scores.Source: §5.1 · p. 5
  • 0.535 Final Similarity makes Gemini 3.1 Pro the top agent, ahead of Claude Opus 4.7 (0.472); EASEL-9B scores 0.459, third overall and best non-proprietary.EASEL-9B exceeds its Qwen3.5-9B base model (0.432) by +0.027.Source: §5.1 · p. 5, Table 3 · p. 6
  • Closed-source agents reach their effective peak within the first 10% of steps: some plateau, while others degrade (GPT-5.5 F-B Gap 0.073; AUC 0.440 vs. Final 0.426).Five open-weight models’ F-B Gaps below 0.01 accompany 82.3%–92.4% exact-repeat rates; the bottleneck is sustained feedback-driven correction, not initial perception.Source: Figure 4 · p. 5, §5.1 · pp. 5–6, Table 3 · p. 6
  • 0.456 with C1-only training and 0.459 with the full C1+C2 curriculum improve on Qwen3.5-9B’s 0.432; C1 accounts for most of the gain and C2 adds +0.003.Trained models’ Final-Best Gap rises slightly from the base model’s 0.015 to 0.019–0.020, indicating better local action quality but limited long-horizon correction.Source: §5.1 · p. 6

5.2 Semantic Task Analysis

  • Gemini 3.1 Pro and GPT-5.5 lead both outline tasks and achieve perfect handwriting; Claude Opus 4.7 fails Circle Outline through persistent draw-undo loops.Weaker models produce partial or zero scores, and stable action strategy is a prerequisite for tasks requiring cumulative canvas strokes.Source: §5.2 · p. 6, Table 4 · p. 7
  • GPT-5.5 earns a perfect Simple Maze score after 25 undos, but harder Maze Path remains unsolved; models exhibit distinct undo and invalid-action failure patterns.Gemini 3.1 Pro and GPT-5.5 use undo mostly on maze tasks, whereas Claude Opus 4.7’s many non-maze undos erase useful strokes and Claude Sonnet 4.6 rarely responds to maze constraint violations.Source: §5.2 · pp. 6–7, Table 4 · p. 7

6 Discussion

EASEL’s controlled 2D canvas isolates visual goals, parameterized actions, and closed-loop feedback for diagnosing fine-grained visual manipulation. Its explicit action traces support broader semantic tasks and training methods, while EASEL-9B preserves or modestly improves perception benchmark performance after training.

Scope and proxy validity

  • EASEL’s 2D canvas trades breadth for control, isolating visual goals, parameterized actions, and closed-loop feedback to diagnose fine-grained visual manipulation and correction.Source: §6 · p. 7

Applications

  • EASEL uses explicit, editable strokes; its semantic tasks span annotation, handwriting, and path planning, while action traces and feedback support outcome-based optimization, reinforcement learning, and planning.Medical annotation, CAD sketching, diagram authoring, and embodied manipulation motivate broader relevance because they share visual state observation, structured action prediction, and potential closed-loop feedback.Source: §6 · p. 7

Generalization of trajectory supervision

  • EASEL-9B matches or improves the base model on MMVP (+0.7%), POPE (+2.0%), HallusionBench (+0.6%), and BLINK spatial reasoning (+0%) and counting (+1.7%).The authors suggest closed-loop action training preserves and modestly broadens fine-grained multimodal perception, and propose EASEL-style tasks as a possible training signal with more targeted supervision.Source: §6 · p. 7

7 Conclusion

EASEL operationalizes dexterous visual tool use through reference-guided reconstruction, semantic tasks, and trajectory diagnostics, exposing limitations that answer-only QA and high-level tool-use benchmarks obscure. EASEL-9B improves over its base model by a relative 6.3%, though long-horizon feedback correction remains limited.

7 Conclusion

  • A relative 6.3% improvement over the base model is achieved by EASEL-9B.Early-phase training accounts for most of the gain and fine-grained visual perception improves, while long-horizon feedback correction remains limited.Source: §7 · p. 7
  • EASEL operationalizes dexterous visual tool use as precision-critical, closed-loop parameterized visual action.By combining reference-guided reconstruction, instruction-conditioned semantic tasks, and trajectory diagnostics, it exposes failures obscured by answer-only QA and high-level tool-use benchmarks: models plateau early, degrade after initial gains, and rarely exploit corrective feedback.Source: §7 · p. 7

Related papers

Loading 2608.25417v1…