PaperlayerView exact revision on arXiv

Source-linked AI summary

TOBench: A Task-Oriented Omni-Modal Benchmark for Real-World Tool-Using Agents

Zhiqiang Liu, Wenhui Dong, Yilang Tan, Yuwen Qu, Haochen Yin, Chenyang Si

arXiv:2605.16909v1cs.AI

TL;DR

Existing benchmarks often evaluate tool use, computer use, and multimodal reasoning in isolation, leaving realistic omni-modal workflows insufficiently covered. TOBench introduces an executable benchmark and evaluation harness centered on closed-loop multimodal verification. Across 15 contemporary models, the best model reaches 41.0% overall task success versus a 94.0% human benchmark, while the benchmark remains limited to a subset of real-world omni-modal work and depends on evolving MCP implementations.

  • Problem

    Existing benchmarks often evaluate tool use, computer use, and multimodal reasoning separately, leaving a gap in end-to-end omni-modal tool use for realistic professional workflows.

  • Method

    TOBench combines 100 executable tasks with MCP-based environments and task-specific grounded verifiers that require agents to execute tools, inspect artifacts, and revise outputs.

  • Results

    41.0% overall task success is achieved by the best model, while the human benchmark reaches 94.0%, and hard tasks remain largely unsolved.

  • Takeaways & Limitations

    TOBench provides a practical testbed for studying how agents perceive, act, verify, and self-correct in realistic omni-modal environments.

  • Takeaways & Limitations

    TOBench covers only a subset of real-world omni-modal work, and its behavior may depend on evolving MCP server descriptions, response formats, and implementation details.

Abstract

from arXiv · show

Tool-using agents are increasingly expected to operate across realistic professional workflows, where they must interpret multimodal inputs, coordinate external tools, inspect intermediate artifacts, and revise their actions before producing a final result. Existing benchmarks, however, often evaluate tool use, computer use, and multimodal reasoning in isolation, leaving a gap between benchmark settings and end-to-end omni-modal tool use in the real world. To address this gap, we introduce MM-ToolBench, a benchmark and evaluation harness for task-oriented omni-modal tool use. MM-ToolBench contains 100 executable tasks from two macro task families, Customer Service and Intelligent Creation, covering 20 subcategory slices and supported by 27 MCP servers with 324 tools. The central design of MM-ToolBench is closed-loop multimodal verification: agents must execute tools, inspect rendered or transformed artifacts, and self-correct when outputs fail task-specific requirements. To make such evaluation scalable and verifiable, MM-ToolBench couples MCP-based execution with task-specific grounded evaluators and a semi-automated construction pipeline for scenario discovery, task instantiation, evaluator synthesis, and human audit. Experiments on 15 contemporary agentic models show that MM-ToolBench remains highly challenging: Claude Opus 4.6, commonly regarded as one of the strongest coding-agent models, achieves only 32.0% task success, far below the 94.0% human benchmark. We envision MM-ToolBench as a practical foundation for evaluating and advancing next-generation omni-modal tool-using agents through closed-loop multimodal verification.

1 Introduction

Existing benchmarks often separate tool use, multimodal perception, and computer interaction, while realistic workflows require their coordination with artifact inspection and revision. TOBench addresses this gap with executable tasks and closed-loop verification, but current models remain far below human performance.

  • Real-world workflows require agents to coordinate multimodal perception, tool execution, artifact transformation, and iterative verification over changing workspace states.
  • TOBench contains 100 executable tasks spanning two macro families, 20 subcategory slices, 27 MCP servers, and 324 tools.
  • Closed-loop multimodal verification requires agents to execute tools, inspect rendered or transformed artifacts, and revise behavior when task requirements are unmet.
  • 41.0% task success is achieved by Qwen3.5-Plus, compared with 94.0% for humans, showing that TOBench remains challenging for contemporary agentic models.

2 Related Work

Prior benchmarks expanded tool-use evaluation toward planning, long-horizon execution, MCP ecosystems, GUI grounding, and multimodal interaction. TOBench is positioned as distinct through realistic professional tasks, a unified MCP ecosystem, and iterative artifact inspection with grounded verifiers.

  • Earlier tool-use benchmarks evaluate function calling, planning, multistep execution, reproducibility, and live MCP interaction, but most remain primarily textual.
  • Table 1 compares TOBench with representative benchmarks across scale and capability dimensions, including tasks, domains, tools, MCP integration, cross-scenario operation, and real environments.
  • Multimodal and computer-use benchmarks broaden evaluation toward GUI grounding and multimodal interaction across diverse applications.
  • TOBench differs by targeting realistic professional task completion, using a unified MCP-based ecosystem, and centering iterative artifact inspection with task-specific grounded verifiers.

3 TOBench

TOBench formalizes realistic omni-modal tasks as executable harnesses containing instructions, tools, mutable states, observations, transitions, evaluation criteria, and grounded verification. Its construction combines scenario discovery, task instantiation, multimodal asset curation, and task-specific evaluation across two macro families.

  • Task Formalism: Each task instance specifies a user instruction, professional role, domain rules, multimodal assets, executable MCP environment, and grounded verifier.
  • Task Formalism: The harness models actions as tool use or natural-language interaction and observations as tool outputs, rendered artifacts, and textual feedback.
  • Task Formalism: Tool calls can update files or external state, while rendering or inspection exposes multimodal evidence that can trigger corrective follow-up actions.
  • TOBench Construction Framework: Tasks are constructed from realistic professional scenarios by selecting omni-modal MCP tools, discovering user-centered scenarios, instantiating executable tasks, and curating multimodal assets.
  • Evaluation Pipeline: Task-specific evaluation generates and audits evaluation points and code, then executes grounded evaluators under time-sensitive conditions.
  • TOBench Construction Framework: 67 Customer Service tasks and 33 Intelligent Creation tasks form the benchmark’s two macro families, covering 20 subcategory slices.

4 Evaluation Harness

TOBench evaluates each task through a task-specific executable harness that checks realized trajectories, workspace artifacts, and tool logs against grounded criteria. Its evaluation combines heterogeneous checks, execution-time preprocessing, and manual review of generated evaluation points.

  • Task-level harness: Each task binds an executable environment to a grounded verifier that evaluates the executed trajectory, final workspace snapshot, and tool log.The verifier returns binary outcomes for approved evaluation criteria rather than comparing with a static answer key.
  • Evaluation criteria: Evaluation points cover format constraints, judge-based multimodal constraints, and tool/result constraints.These categories address output structure, multimodal content, and tool-grounded correctness.
  • Execution-time evaluation: Evaluation outcomes may depend on document rendering, image conversion, speech transcription, or re-querying time-sensitive tools.These preprocessing steps allow criteria to inspect realized multimodal artifacts and externally grounded results at execution time.
  • Evaluation criteria: Each task derives its evaluation criteria separately from the user request, role, domain rules, expected outputs, and ground-truth workspace.Task-specific derivation replaces a single benchmark-wide rubric for heterogeneous and partially open-ended tasks.
  • Evaluation criteria: All generated evaluation points are manually reviewed to remove omissions, unsupported assumptions, and duplicate checks.Manual auditing is applied after task-specific evaluation-point generation.
  • Task-level success: A task is counted as solved only when all relevant evaluation points pass.Task-level success uses binary evaluation outcomes and requires every approved criterion to pass.

5 Experiments

TOBench evaluates 15 agentic models on 100 executable tasks and remains challenging across task families and difficulty levels. Its failure analysis highlights execution, multimodal reasoning, and self-verification as central bottlenecks.

  • 5.2 Main Results: 20.00% on Customer Service-Hard and 15.38% on Intelligent Creation-Hard were the best scores on the hardest splits.Performance remained unsaturated even on easy tasks and collapsed on hard splits.
  • 5.2 Main Results: Customer Service stresses grounded retrieval and faithful tool use, whereas Intelligent Creation is especially sensitive to multimodal editing and final-result verification.The two macro families therefore stress different capabilities.
  • 5.3 Error Analysis: Tool Call Error, Tool Parameter Error, Multimodal Capability Deficit, Self-Verification Failure, and Non-Agent Error form the manual failure taxonomy.The taxonomy organizes benchmark failures into five top-level categories.
  • 5.3 Error Analysis: Tool call and parameter errors remain the most pervasive execution bottleneck, often causing failure before high-level reasoning becomes relevant.Examples include choosing the wrong tool, omitting required operations, hallucinating unsupported actions, and passing invalid arguments.
  • 5.3 Error Analysis: Multimodal reasoning errors dominate after basic execution succeeds, including visual extraction, spatial reasoning, temporal localization, and cross-modal evidence alignment.The remaining errors of stronger models also concentrate in multimodal reasoning and final verification.
  • 5.3 Error Analysis: Missing visual verification causes agents to stop after plausible edits or metadata checks, leaving rendered artifacts uninspected.The harness penalizes open-loop completion and rewards closed-loop self-correction, contributing to Intelligent Creation-Hard difficulty.

6 Limitations and Discussion

The benchmark improves realism through broad task coverage, multimodal assets, and MCP-based infrastructure, but remains limited in scope and evaluation controllability. Its mixed evaluators and dependence on evolving software ecosystems introduce potential noise, bias, complexity, and variance.

  • TOBench covers only a subset of real-world omni-modal work despite spanning 100 executable tasks across 20 subcategory slices.
  • Code-based checks can miss subtle failures, while multimodal judges may introduce bias.
  • MCP-centric evaluation improves realism for current ecosystems but may depend on evolving server descriptions, response formats, and implementation details.
  • Realistic software stacks and multimodal assets better reflect user workloads but increase evaluation complexity and variance.

7 Conclusion

TOBench combines realistic professional workflows, MCP-native tool ecosystems, and closed-loop verification of multimodal artifacts. Its evaluation also incorporates standardized MCP-server refinements and modality-specific servers that support visual inspection and self-correction.

  • TOBench emphasizes realistic professional workflows, MCP-native tool ecosystems, and closed-loop verification over multimodal intermediate artifacts.
  • Community MCP resources were refactored with optimized output formats and stronger error handling to provide more instructional feedback during long-horizon tasks.
  • Vision and audio MCP servers support operations such as image editing and document rendering, enabling agents to inspect layouts and formatting for autonomous self-correction.

B.1 Definition of Benchmark Traits

TOBench defines benchmark traits around realistic, multimodal, MCP-connected workflows and evaluates agents through role-based task dialogues. Its agent configuration requires planning, iterative tool use, multimodal inspection, verification, and adherence to workspace and domain constraints.

  • Benchmark traits include task counts, domains, tools, MCP integration, cross-scenario operation, realistic environments, grounded information, fuzzy prompts, and multimodal requirements.
  • TOBench supports sequential and parallel execution; the reported evaluation used a 4-core CPU, 16GB RAM, parallelism factor 2, and required 3–8 hours for all tasks.
  • Tasks use single-turn dialogues with an agent system prompt defining ReAct reasoning and task-specific domain rules.
  • The agent role requires planning, stepwise execution, reflection after tool feedback, outcome verification, and professional document standards.
  • Generation tools must use context-aligned parameters, absolute file paths, and workspace-bounded file operations.
  • The user persona must convey all task details faithfully, avoid unsupported assumptions, and provide complete information in single-turn scenarios.

D Detailed Error Taxonomy

TOBench uses a detailed error taxonomy and task-specific evaluation points to distinguish failure modes and support criterion-level analysis across tasks.

  • Table 4 enumerates fine-grained error subcategories for trajectory annotation and error heatmaps beneath five top-level categories.
  • Table 5 catalogs task-specific evaluation-point types, motivating per-task evaluator synthesis instead of a single benchmark-wide scoring template.

E Extra Analysis for TOBench

TOBench’s task analyses expose distinct failures in fine-grained perception, visual-spatial alignment, temporal video reasoning, cross-modal fusion, and multimodal hallucination. These examples show how agents can produce incorrect artifacts or conclusions despite executing relevant tools.

  • Fine-grained Perception: Gemini 3 Pro recorded 28 chairs instead of the ground-truth 27 in the wedding-layout budgeting task.
  • Visual-Spatial Understanding: Qwen 3.5 Plus produced oversized signatures with landlord-label overlap, tenant misplacement, and footer obstruction despite repeated adjustment attempts.
  • Video Temporal Relationship Understanding: MiniMax M2.5 missed deliberate food throwing and a tip confrontation because fixed 10-second frame extraction failed to capture continuous actions.
  • Cross-modal Information Fusion: Gemini 3.1 Pro captured an unrelated outdoor frame at 40 seconds instead of the work-hours interface linked to the first spoken reason.
  • Multimodal Hallucination: GPT-4o fabricated the Minecraft glitch trigger, reproduction steps, and timestamps after viewing only the first video frame.

E.2.1 Traj 1

The trajectory generates a personalized astronaut image for Leo, builds a one-slide welcome presentation from a template, and verifies the resulting files and slide content. The final presentation includes the requested greeting, images, and saved output.

  • Task requirements: The task requests an astronaut image of Leo and a welcome PPTX for his 10th birthday using a presentation template.The requested design includes a spacesuit, moon setting, spaceship, blue starry background, greeting text, and hotel logo.
  • Image generation: The astronaut image was generated successfully and saved as Leo_astronaut.png.The image was described as showing Leo in a spacesuit on the moon with a spaceship and blue starry sky.
  • Presentation creation: The presentation was created from example.pptx, which contains 1 slide and 11 layouts.The workflow inspected the template and selected slide content for the welcome presentation.
  • Slide content: The slide received the title “Space Explorer” and greeting “Happy 10th Birthday, Astronaut Leo!” plus the astronaut image and hotel logo.The title was added at 32 pt and the greeting at 24 pt; both images were inserted as separate slide elements.
  • Verification: The completed presentation was saved to workspace/welcome_tv.pptx and subsequently checked through text extraction and file listing.The verification reported one slide with text and confirmed the requested image, template, logo, and presentation files were present.

E.2.2 Traj 2

The trajectory analyzes a tutorial video about using pandas to read CSV data, resolving a mismatch between the user’s Excel description and the video’s CSV content. It captures supporting frames and creates tutorial.md with Data Preparation and Code Example sections.

  • Task requirements: The requested tutorial identifies Data Preparation and Code Example as the two main parts and requires supporting images and a markdown file.The requested outputs are tutorial.md, data_preparation.png, and code_example.png.
  • Content analysis: The audio transcription describes importing CSV data into a Jupyter notebook with pandas rather than reading an Excel table.The workflow explicitly compares the transcription with the user’s description and then inspects video frames to resolve the discrepancy.
  • Content analysis: The video frames show a CSV file and notebook in the same folder, followed by pandas code using read_csv and a DataFrame output containing animal records.The code examples include import pandas, pandas.read_csv("animals.csv"), and displaying the resulting data.
  • Supporting frames: The workflow captured data_preparation.png at 2 seconds and code_example.png at 22 seconds for the two tutorial sections.The selected frames depict the shared folder for preparation and the notebook code and output for the example.
  • Deliverables: The tutorial.md file was written with second-level headers, descriptions, and inserted images, then verified as complete.The final checks report both requested sections, saved images in the assets folder, and Markdown image insertion.
Loading 2605.16909v1…