Source-linked AI summary
OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents
Hongrui Jia, Jitong Liao, Xi Zhang, Haiyang Xu, Tianbao Xie, Chaoya Jiang, Ming Yan, Si Liu, Wei Ye, Fei Huang
TL;DR
Existing benchmarks largely overlook MCP tool invocation, limiting fair comparison of computer-use agents with different tool capabilities. OSWorld-MCP addresses this gap with a curated, realistic benchmark that combines GUI actions and MCP tools, finding improved performance overall but persistently low tool-use rates.
Problem
Most computer-use benchmarks assess GUI interaction while overlooking MCP tool invocation, making comparisons across agents with different tool capabilities inequitable.
Method
OSWorld-MCP combines OSWorld’s realistic environment with 158 manually validated MCP tools and lets agents choose between tool invocation and GUI actions.
Results
MCP tools generally improve task success, while Claude 4 Sonnet attains the highest reported tool invocation rate at 36.3%.
Takeaways & Limitations
Explicitly measuring MCP usage provides a more comprehensive assessment of computer-use agents in complex, tool-assisted environments.
Takeaways & Limitations
The dataset is limited to synthetic interaction records from publicly available software environments and tasks, with no human-subject data.
Abstract
from arXiv · showhide
With advances in decision-making and reasoning capabilities, multimodal agents show strong potential in computer application scenarios. Past evaluations have mainly assessed GUI interaction skills, while tool invocation abilities, such as those enabled by the Model Context Protocol (MCP), have been largely overlooked. Comparing agents with integrated tool invocation to those evaluated only on GUI interaction is inherently unfair. We present OSWorld-MCP, the first comprehensive and fair benchmark for assessing computer-use agents' tool invocation, GUI operation, and decision-making abilities in a real-world environment. We design a novel automated code-generation pipeline to create tools and combine them with a curated selection from existing tools. Rigorous manual validation yields 158 high-quality tools (covering 7 common applications), each verified for correct functionality, practical applicability, and versatility. Extensive evaluations of state-of-the-art multimodal agents on OSWorld-MCP show that MCP tools generally improve task success rates (e.g., from 8.3% to 20.4% for OpenAI o3 at 15 steps, from 40.1% to 43.3% for Claude 4 Sonnet at 50 steps), underscoring the importance of assessing tool invocation capabilities. However, even the strongest models have relatively low tool invocation rates, Only 36.3%, indicating room for improvement and highlighting the benchmark's challenge. By explicitly measuring MCP tool usage skills, OSWorld-MCP deepens understanding of multimodal agents and sets a new standard for evaluating performance in complex, tool-assisted environments. Our code, environment, and data are publicly available at https://osworld-mcp.github.io.
1 INTRODUCTION
OSWorld-MCP addresses the lack of fair benchmarks that jointly assess GUI operation, MCP tool invocation, and decision-making in realistic computer-use tasks. It combines 158 curated tools with GUI actions and evaluates models’ choices between them.
- Existing benchmarks largely overlook external-tool invocation, making comparisons between tool-enabled and GUI-only agents inherently inequitable.The paper identifies a missing integrated standard for evaluating GUI skills, tool use, and decision-making together.
- MCP can complete the VS Code autoDocstring installation in one step, whereas GUI operations may require at least four steps.The comparison motivates evaluating tool use alongside GUI interaction because the two execution paths differ in efficiency and robustness.
- OSWorld-MCP jointly evaluates GUI operation, MCP tool invocation, and decision-making through 158 tools spanning 7 common applications.The benchmark extends OSWorld with a curated tool set and includes distractor tools and tasks requiring multi-round invocation.
- The benchmark allows agents to choose autonomously between MCP tools and direct GUI actions at every task step.This setting assesses both tool selection and the efficiency of hybrid execution paths.
- An automated three-module pipeline generates candidate tools, which are combined with existing MCP tools and manually filtered for functionality and practical relevance.The modules are Code Generation, Code Filter, and Tool Wrap; the pipeline produces 72 functional tools before combination and curation.
2 RELATED WORK
Prior multimodal-agent benchmarks primarily assess GUI-based task completion, while MCP research introduces standardized interfaces for connecting models to external tools and data. OSWorld-MCP builds on these directions to evaluate tool invocation in realistic, interactive environments.
- Existing multimodal-agent benchmarks primarily evaluate GUI-based operations, often using static manually collected datasets.Examples include Mind2Web, WebLinx, and OmniAct.
- MCP provides a standardized, model-agnostic interface for connecting AI applications with external tools, data resources, and workflows.Its plug-and-play design addresses fragmented and highly customized integrations.
3 OSWORLD-MCP BENCH
OSWorld-MCP extends OSWorld with curated MCP tools and evaluates agents’ tool invocation alongside GUI operation and hybrid decision-making. Its construction combines automated generation, manual filtering, and task-level analysis to assess tool relevance and decision efficiency.
- 3.1 OVERVIEW: 158 high-quality MCP tools extend OSWorld across seven common applications for evaluating tool invocation, GUI operation, and decision-making.Agents can choose between MCP tools and direct GUI actions during task execution.
- 3.2 TOOLS GENERATION AND FILTER: The benchmark generates tools through code generation, filtering, and wrapping, then manually removes redundant or overly task-specific tools from the collected set.The pipeline produces 72 functional tools, while reviewers retain only tools judged qualified by both independent reviewers.
- 3.3 TOOLS ANALYSIS: 133 of the 158 tools were manually validated as improving task efficiency, while 25 originated from existing external MCP servers.The tools are distributed across application scenarios and intended for practical real-world use.
- 3.3 TOOLS ANALYSIS: Five multimodal models invoked 131 tools at least once, and the benchmark includes 250 Tool-Beneficial Tasks alongside 111 Non-Tool-Beneficial Tasks.The task split is based on whether at least one available tool can substantially improve execution efficiency.
- 3.4 METRICS: OSWorld-MCP measures task accuracy, Tool Invocation Rate, and Average Completion Steps to capture performance, tool-use decisions, and execution efficiency.TIR reflects whether tools are invoked appropriately, while lower ACS corresponds to more efficient tool selection and decisions.
4 EXPERIMENTS
OSWorld-MCP evaluates multimodal agents under hybrid GUI-and-MCP interaction, showing broad performance gains from tool invocation but weaker benefits as tool-selection complexity increases.
- Main results: 35.3 at 15 steps and 43.3 at 50 steps: Claude 4 Sonnet achieves the highest end-to-end-model accuracy in OSWorld-MCP, while also recording the highest tool invocation rate.Seed-VL1.5 and Claude 4 Sonnet obtain the lowest ACS at 15 and 50 steps, with 10.2 and 20.1 respectively.
- Main results: Higher accuracy and lower ACS occur for all but Qwen2.5-VL after MCP tools are introduced, although Qwen2.5-VL shows a slight accuracy gain with increased ACS.Gemini-2.5-Pro has the largest reported improvement, rising from 7.4 to 20.5 accuracy and reducing ACS from 13.8 to 11.4 at 15 steps.
- Main results: Tool Invocation Rate generally correlates positively with accuracy, while its relationship with ACS is non-linear and depends on task difficulty and model-specific strategies.The positive TIR–accuracy relationship remains stable across step limits and task sets, whereas ACS shows no obvious correlation with TIR.
- Tool complexity: MCP tools generally improve complex-task performance, but benefits diminish for extremely complex tasks requiring multiple-tool combinations.As available tools increase, MCP configurations eventually show falling accuracy and TIR with rising ACS, consistent with more difficult tool selection and composition.
- Ablation study: Removing RAG tool filtering reduces Gemini-2.5-Pro overall accuracy from 20.5 to 15.5 and Tool-beneficial-task accuracy from 24.9 to 18.0.On Tool-beneficial Tasks, TIR also falls from 20.4 to 12.4, while randomly shuffling tool descriptions increases overall accuracy from 20.5 to 22.7.
- Case study: A LibreOffice Calc case study shows Gemini-2.5-Pro using MCP tools to create a sheet, copy a column, and verify the result, whereas the GUI-only agent fails to select the column.The example illustrates MCP tools complementing GUI interaction on a multi-step spreadsheet task.
5 CONCLUSION
OSWorld-MCP jointly evaluates GUI operation and MCP tool invocation for realistic computer-use scenarios, with experiments showing benefits and remaining challenges.
- OSWorld-MCP jointly assesses GUI operation skills and MCP tool-invocation capabilities in computer-use scenarios.
- Experiments on eight state-of-the-art LMMs show that tool invocation can improve robustness and efficiency, while usage frequency may trade off against overall performance.
- Future extensions target more complex, dynamic, and collaborative environments and human-centred evaluation metrics.
ETHICS STATEMENT
OSWorld-MCP uses public software environments and synthetic interaction records, with tools generated, validated, or selected from open-source servers.
- The dataset uses publicly available OSWorld software environments and tasks and contains synthetic multimodal-agent interaction records.
- Its MCP tools are automatically generated and manually validated by the authors or selected from existing open-source MCP servers.
- The dataset excludes proprietary, confidential, and personally identifiable information and is released solely for research and educational purposes.
- Users should comply with applicable laws, institutional guidelines, and license terms while protecting privacy and security.
A.1 MORE CASES
The appendix presents additional tool-call examples showing how tool calls support more efficient and accurate completion of diverse, complex tasks.
- Additional tool-call examples illustrate more efficient and accurate completion of diverse and complex tasks.
A.1.1 CASE: VSCODE SETTINGS
A VS Code case modifies settings to hide pycache folders in the explorer view using an MCP file-exclusion operation.
- The task modifies VS Code settings so all pycache folders are hidden in the explorer view.
- The example executes the osworld mcp code.add files exclude operation as its first listed action.
- The evaluation program confirms that the specific folder is hidden in the explorer view.
A.1.2 CASE: FILESYSTEM OPERATION
The agent used MCP tools to convert a receipt image into a desktop PDF, recovering from an incorrect filename by listing the home directory.
- A.1.2 CASE: FILESYSTEM OPERATION: The initial conversion failed because the specified file or directory did not exist.The agent then listed the home directory to identify the correct image filename.
- A.1.2 CASE: FILESYSTEM OPERATION: The corrected conversion used the image at /home/user/OIP.jpg and wrote /home/user/Desktop/receipt.pdf.
- A.1.2 CASE: FILESYSTEM OPERATION: The agent successfully converted the receipt image to a PDF saved as “receipt.pdf” on the desktop.The successful conversion followed correction of the input filename.
A.1.3 CASE: SLIDE DUPLICATION
The agent duplicated the final two presentation slides into the requested sequence, after correcting failed attempts caused by an incorrect target-position parameter.
- A.1.3 CASE: SLIDE DUPLICATION: The agent ultimately duplicated the final two slides to positions 25 and 26, completing the requested slide sequence.The presentation was then saved.
- A.1.3 CASE: SLIDE DUPLICATION: The initial duplication attempts failed at steps 2 and 4 because the target-position parameter was incorrect.Several subsequent attempts were required before the duplications succeeded.
A.1.4 CASE: SPREADSHEET COPYING
The agent copied the Revenue column with its header from the workbook into a new sheet named Sheet2, using MCP spreadsheet tools.
- A.1.4 CASE: SPREADSHEET COPYING: The spreadsheet workflow used a cross-sheet copy operation followed by switching the active sheet.
- A.1.4 CASE: SPREADSHEET COPYING: The Revenue column was copied successfully to Sheet2.The operation copied range A1:A20 to Sheet2 starting at cell A1.
- A.1.4 CASE: SPREADSHEET COPYING: The broader tool collection includes spreadsheet, presentation, document, operating-system, browser, media, filesystem, and Git operations.