PaperlayerView exact revision on arXiv

Source-linked AI summary

OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents

Hongrui Jia, Jitong Liao, Xi Zhang, Haiyang Xu, Tianbao Xie, Chaoya Jiang, Ming Yan, Si Liu, Wei Ye, Fei Huang

arXiv:2510.24563v2cs.CV

TL;DR

Computer-use evaluations have largely emphasized GUI interaction while overlooking MCP tool invocation. OSWorld-MCP jointly benchmarks tool use, GUI operation, and decision-making; experiments show MCP tools improve agent metrics, while the highest LMM tool invocation rate is 36.3%.

  • Problem

    Existing benchmarks largely neglect external-tool invocation, and comprehensive, equitable benchmarks jointly assessing tool use, GUI operation, and decision-making are lacking.

    Source: §1 · pp. 1–2

  • Method

    OSWorld-MCP extends OSWorld with 158 curated MCP tools across seven applications and jointly evaluates tool invocation, GUI operation, and decision-making, using automated generation and manual validation.

    Source: §1 · p. 2, §3.1 · p. 4, §5 · p. 10

  • Results

    MCP tools improve agent metrics; Claude 4 Sonnet records the highest LMM tool invocation rate, only 36.3%.

    Source: §1 · p. 3, §4.2 · p. 7

  • Takeaways & Limitations

    OSWorld-MCP offers a unified standard for fair comparison of tool-utilization abilities across models.

    Source: §1 · p. 2

    OSWorld-MCP is released solely for research and educational purposes, with use subject to applicable laws, institutional guidelines, and license terms.

    Source: ETHICS STATEMENT · pp. 10–11

Abstract

from arXiv · show

With advances in decision-making and reasoning capabilities, multimodal agents show strong potential in computer application scenarios. Past evaluations have mainly assessed GUI interaction skills, while tool invocation abilities, such as those enabled by the Model Context Protocol (MCP), have been largely overlooked. Comparing agents with integrated tool invocation to those evaluated only on GUI interaction is inherently unfair. We present OSWorld-MCP, the first comprehensive and fair benchmark for assessing computer-use agents' tool invocation, GUI operation, and decision-making abilities in a real-world environment. We design a novel automated code-generation pipeline to create tools and combine them with a curated selection from existing tools. Rigorous manual validation yields 158 high-quality tools (covering 7 common applications), each verified for correct functionality, practical applicability, and versatility. Extensive evaluations of state-of-the-art multimodal agents on OSWorld-MCP show that MCP tools generally improve task success rates (e.g., from 8.3% to 20.4% for OpenAI o3 at 15 steps, from 40.1% to 43.3% for Claude 4 Sonnet at 50 steps), underscoring the importance of assessing tool invocation capabilities. However, even the strongest models have relatively low tool invocation rates, Only 36.3%, indicating room for improvement and highlighting the benchmark's challenge. By explicitly measuring MCP tool usage skills, OSWorld-MCP deepens understanding of multimodal agents and sets a new standard for evaluating performance in complex, tool-assisted environments. Our code, environment, and data are publicly available at https://osworld-mcp.github.io.

1 INTRODUCTION

OSWorld-MCP addresses the lack of fair evaluations that jointly assess GUI operation, tool invocation, and decision-making in real-world computer-use tasks. It integrates MCP tools with GUI actions so agents can choose between them while completing tasks.

1 INTRODUCTION

  • Installing the VS Code autoDocstring extension may take at least four GUI steps, while an MCP tool can do it in one step.The authors argue that comparing agents with autonomous tool invocation against agents evaluated only on GUI interaction is inequitable, motivating an integrated benchmark.Source: §1 · p. 2, Figure 1 · p. 2
  • OSWorld-MCP includes 158 curated MCP tools across 7 applications, applicable to 250 tasks (69%); 153 tasks (42%) require multi-round tool invocations.Claude 4 Sonnet achieves 0 accuracy on tasks requiring four tool rounds with GUI-only operation and 16.7% with MCP tools.Source: §1 · p. 2
  • Agents can choose MCP tools or GUI actions at each step; Tool Invocation Rate (TIR) and Average Completion Steps (ACS) supplement task accuracy.TIR measures the proportion of tasks successfully completed using MCP tools, while ACS measures completion efficiency.Source: §1 · pp. 2–3
  • OpenAI o3’s success rate rises from 8.3% to 20.4% at 15 steps with MCP tools versus GUI-only operation.The experiments also find that MCP tools improve agent accuracy and efficiency overall.Source: §1 · p. 3
  • An automated pipeline using OpenAI o3 produces 72 functional tools, which are combined with existing-server tools and manually curated into 158 verified tools.The pipeline comprises code generation, code filtering, and tool wrapping; manual verification removes redundant and overly task-specific tools.Source: §1 · p. 3
  • Tool invocation rate positively correlates with performance, but agents struggle to compose multiple tools and select the correct tool from a full list.The reported tool invocation rates for multimodal agents remain relatively low, leaving room to improve tool utilization.Source: §1 · p. 3

2 RELATED WORK

Existing multimodal-agent benchmarks focus on GUI task completion and lack a fair framework for jointly evaluating GUI interaction, tool invocation, and decision-making. MCP standardizes tool integration, but existing MCP benchmarks have limitations in coverage, real-time evaluation, or task openness.

2.1 BENCHMARKS FOR MULTIMODAL AGENTS

  • Static benchmarks rely on predefined GUI trajectories, while interactive benchmarks typically offer only GUI actions, leaving tool invocation and alternative action paths unevaluated.Together, these limitations leave no comprehensive, fair framework for assessing tool invocation, GUI interaction, and decision-making.Source: §2.1 · p. 3

2.2 MODEL CONTEXT PROTOCOL

  • MCP provides a standardized, model-agnostic interface for secure tool and resource integration, but existing MCP benchmarks are limited in tool coverage, real-time evaluation, or task openness.The paper notes that some benchmarks cover only a few dozen tools, while others use LLM-based evaluation or define tasks around available tools.Source: §2.2 · pp. 3–4

3 OSWORLD-MCP BENCH

OSWorld-MCP evaluates computer-use agents’ tool invocation, GUI operation, and decision-making in realistic computing environments. It combines generated and curated MCP tools with task analyses and metrics for measuring agent performance and tool-use choices.

3.1 OVERVIEW

  • The benchmark builds on OSWorld’s realistic Ubuntu, Windows, and macOS environments to assess tool invocation, GUI operation, and decisions about pathways and tool selection.OSWorld covers nine applications and 369 real-world tasks; OSWorld-MCP’s analysis excludes eight Google Drive tasks.Source: §3.1 · p. 4, §3.3 · p. 5

3.2 TOOLS GENERATION AND FILTER

  • To address the shortage of practical, non-overlapping MCP tools, the pipeline generated 72 tools from verified code and manually filtered 264 generated or curated tools down to 158.The 264 collected tools included 72 generated tools and 192 curated from existing MCP servers; each retained tool was approved by at least two reviewers.Source: §3.2 · pp. 4–5

3.3 TOOLS ANALYSIS

  • OSWorld-MCP contains 158 manually validated high-quality tools, and its 361 analyzed tasks comprise 250 Tool-Beneficial and 111 Non-Tool-Beneficial tasks.The benchmark’s tools were found relevant to real-world needs, while many tasks can be completed more efficiently through multiple tool invocations.Source: §3.2 · p. 5, §3.3 · p. 6

3.4 METRICS

  • Task Accuracy measures overall task completion, Tool Invocation Rate measures successful tool use on beneficial tasks and successful non-use on non-beneficial tasks, and Average Completion Steps measures steps per task.Together, the metrics assess performance, tool-use decisions, and completion efficiency.Source: §3.4 · p. 6

4 EXPERIMENTS

MCP tools generally improved accuracy and reduced completion steps, but tool invocation remained limited and benefits weakened on tasks requiring combinations of tools. The experiments also show that tool use correlates with task accuracy, while its relationship to efficiency is more complex.

4.1 SETUP

  • Six LMMs and Agent-S2.5 were evaluated across nine applications using GUI-Owl, with 15- and 50-step limits and three-run averages.Agent-S2.5 used OpenAI o3 for generation and UI-TARS-1.5-72B for grounding; OSWorld-MCP let agents choose between GUI operations and application-relevant tools retrieved from the 158-tool set.Source: §4.1 · p. 7, §4.2 · p. 7

4.2 MAIN RESULTS

  • 35.3 (15 steps) and 43.3 (50 steps) made Claude 4 Sonnet the most accurate LMM in OSWorld-MCP; MCP improved accuracy and reduced ACS for all but Qwen2.5-VL.Claude 4 Sonnet had the highest tool invocation rate at 36.3%; Agent-S2.5 accuracy was 42.1 at 15 steps and 49.5 at 50 steps.Source: §4.2 · pp. 7–8, Table 1 · p. 8
  • Across models, TIR positively correlated with accuracy across step limits and task sets, but showed no clear correlation with ACS.The authors suggest correct tool use can improve efficiency, while varying task complexity and additional attempts at higher step limits may obscure that effect.Source: Figure 5 · p. 8, §4.2 · pp. 8–9
  • With more available tools, GUI-only accuracy tended to fall and ACS rise; MCP initially helped when tools were few, but benefits diminished as tasks required tool combinations.GUI plus MCP generally had higher accuracy and lower ACS, yet sometimes lower accuracy than GUI-only at similar ACS; the authors attribute this to the difficulty of combining MCP tools.Source: Figure 5 · p. 8, §4.2 · p. 9
  • At 15 steps, Gemini-2.5-Pro accuracy rose from 7.4 to 20.5 and ACS fell from 13.8 to 11.4; Tool-Beneficial accuracy rose from 6.3 to 24.9.Accuracy changes on Non-Tool-Beneficial Tasks were minor; such tasks could still benefit from tools that aid completion or help rule out irrelevant solution paths.Source: §4.2 · pp. 7–8, Table 2 · p. 9

4.3 ABLATION STUDY

  • Removing RAG reduced overall accuracy from 20.5 to 15.5; on Tool-Beneficial Tasks, accuracy fell from 24.9 to 18.0 and TIR from 20.4 to 12.4.The authors attribute the decline to long tool contexts reducing tool use and accurate invocation; on Non-Tool-Beneficial Tasks, accuracy was unchanged while TIR rose from 8.7 to 9.9.Source: Table 2 · p. 9, §4.3 · p. 10
  • 22.7 versus 20.5 was the overall accuracy after randomizing versus alphabetically ordering tool descriptions.Tool-Beneficial performance was nearly unchanged, while differences were considerable on Non-Tool-Beneficial Tasks; descriptions remained in lexicographical order for consistent evaluation.Source: Table 2 · p. 9, §4.3 · p. 10

4.4 CASE STUDY

  • In a LibreOffice Calc task, Gemini-2.5-Pro used tools to create a sheet and copy a column, then verified the result; without MCP tools, the agent failed to select the column.The task was to copy the “Revenue” column and header to a new sheet named “Sheet2.”Source: §4.4 · p. 10

5 CONCLUSION

OSWorld-MCP jointly evaluates GUI operation and MCP tool invocation in computer-use scenarios, using validated tools within the OSWorld framework. Experiments show tool invocation can improve robustness and efficiency, while exposing trade-offs between usage frequency and overall performance.

5 CONCLUSION

  • Experiments on eight state-of-the-art LMMs show MCP tool invocation can improve robustness and efficiency, with trade-offs between usage frequency and overall performance.The conclusion does not specify the size of these effects.Source: §5 · p. 10
  • OSWorld-MCP jointly evaluates GUI operation skills and MCP tool-invocation capabilities in computer-use scenarios.It is designed as a fair and comprehensive benchmark for Large Multimodal Models within the OSWorld framework.Source: §5 · p. 10
  • An automated pipeline and meticulous manual validation produced a high-quality, diverse MCP tool collection for realistic and balanced evaluation.Source: §5 · p. 10
  • Future work should extend OSWorld-MCP to more complex, dynamic, and collaborative environments and incorporate human-centred evaluation metrics.The conclusion identifies these directions as ways to advance general-purpose, efficient, and trustworthy computer-use agents.Source: §5 · p. 10

ETHICS STATEMENT

OSWorld-MCP uses publicly available software and synthetic interaction records, excludes proprietary, confidential, and personally identifiable information, and is released for research and educational purposes subject to legal, institutional, and license requirements.

ETHICS STATEMENT

  • The dataset uses public OSWorld environments and tasks, validated or open-source MCP tools, and synthetic interaction records, with no proprietary, confidential, or personally identifiable information or human-subject data.The authors state that no human subject data are collected.Source: ETHICS STATEMENT · p. 10
  • The authors release OSWorld-MCP for research and education, ask users to follow applicable laws, institutional guidelines, and license terms, and accept responsibility for harmful content and privacy or security risks.The authors say they bear responsibility for ensuring the dataset is free of harmful or unethical content and its use does not compromise privacy or security.Source: ETHICS STATEMENT · p. 11

A APPENDIX · A.1 MORE CASES

The appendix presents additional tool-call examples to illustrate how tool use can help agents complete diverse, complex tasks more efficiently and accurately.

A.1 MORE CASES

  • The additional cases analyze tool calls as a way for agents to complete diverse, complex tasks more efficiently and accurately.Source: §A.1 · p. 14

A.1.1 CASE: VSCODE SETTINGS

This case demonstrates using an MCP code tool to modify VS Code settings so that `__pycache__` folders are hidden in the Explorer view.

A.1.1 CASE: VSCODE SETTINGS

  • The executed action was `osworld mcp code.add files exclude`.Source: §A.1.1 · p. 14
  • The evaluation confirms that the requested `__pycache__` folder is hidden in the Explorer view.Figure 6 shows the execution process for this VS Code settings task.Source: Figure 6 · p. 14, §A.1.1 · p. 14

A.1.2 CASE: FILESYSTEM OPERATION

The agent corrected a wrong image filename by listing the home directory, then converted the correctly named receipt image to a PDF on the desktop.

A.1.2 CASE: FILESYSTEM OPERATION

  • After conversion failed with /home/user/receipt.jpg, listing the home directory revealed /home/user/OIP.jpg; converting it produced /home/user/Desktop/receipt.pdf.The requested output was receipt.pdf on the desktop.Source: §A.1.2 · p. 15, Figure 7 · p. 15

A.1.3 CASE: SLIDE DUPLICATION

This case shows the agent completing a slide-duplication task after recovering from incorrect target-position parameters. Its workflow used repeated Impress duplication calls and saved the presentation.

A.1.3 CASE: SLIDE DUPLICATION

  • The recorded workflow issued four LibreOffice Impress slide-duplication calls (steps 2, 4, 5, and 6), then saved the presentation at step 7.Source: §A.1.3 · p. 16
  • After incorrect target-position parameters caused failures at steps 2 and 4, the agent duplicated the final slides to positions 25 and 26 and completed the task.The requested sequence was original slide A, original slide B, duplicated slide A, then duplicated slide B.Source: §A.1.3 · p. 16, Figure 8 · p. 17

A.1.4 CASE: SPREADSHEET COPYING · A.2 MCP TOOL LIST · A.3 USE OF LLMS

The appendix demonstrates successful spreadsheet copying with MCP calls, enumerates the complete tool inventory, and states that LLMs assisted only with manuscript preparation.

A.1.4 CASE: SPREADSHEET COPYING

  • The case uses Calc tools to copy A1:A20 to Sheet2 at A1, retrieve workbook information, and switch active sheets during the operation.The listed calls include repeated copy and sheet-switch actions.Source: §A.1.4 · p. 18
  • Figure 9 shows the “Revenue” column copied with its header to “Sheet2” successfully.The task was to copy the column and header to a new sheet named “Sheet2”.Source: §A.1.4 · p. 18, Figure 9 · p. 19

A.2 MCP TOOL LIST

  • 158 MCP tools are enumerated in the complete inventory.The inventory is presented as Table 3.Source: §A.2 · p. 20, Table 3 · p. 20
  • The listed tools span VS Code, Chrome, and LibreOffice Calc, with examples for browser settings and bookmarks and spreadsheet operations.Calc entries include creating charts and pivot tables, retrieving workbook information, and copying cells between sheets.Source: §A.2 · pp. 21–22
  • The inventory also includes Writer and Impress document and presentation tools, alongside OS, VLC, filesystem, and Git tools.Examples include writing text, inserting images, controlling media playback, managing files, and performing Git operations.Source: §A.2 · pp. 23–27

A.3 USE OF LLMS

  • LLMs assisted only with manuscript clarity, coherence, conciseness, phrasing, and academic style; the authors developed the conceptual content, experiments, analysis, and conclusions.Source: §A.3 · p. 27

Related papers

Loading 2510.24563v2…