MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents

1 Introduction

• Existing CUA benchmarks mostly target GUI agents, while API-agent evaluation requires app-internal access and verification independent of the GUI.Only a handful of testbeds support API agents; AgentStudio compares interfaces through external output files and does not support MCP.Source: §1 · pp. 1–2
• MCPWorld is a testbed for API, GUI, and hybrid desktop agents built around revisable, source-available white-box applications.This design broadens which app features can be exposed as callable APIs and supports verification through direct monitoring of application behavior.Source: §1 · p. 2
• MCPWorld combines a containerized desktop environment, unified GUI and MCP tools, and three app-specific verification methods.Verification uses dynamic binary instrumentation, targeted code injection, or API-driven state queries to monitor internal changes independently of the agent workflow.Source: §1 · p. 2
• The benchmark contains 201 annotated tasks across 10 desktop applications, ranging from simple configurations to complex multi-step workflows.Tasks include natural-language instructions and fine-grained progress signals; 12 authors with computer science backgrounds developed and annotated them.Source: §1 · p. 2
• 75.12% task completion accuracy was achieved by a Claude Computer Use agent with Claude 3.7 Sonnet in preliminary experiments.The authors describe this as initial evidence of the practical effectiveness of automation leveraging MCP.Source: §1 · p. 2

2.1 Framework Overview

• App-internal signals let MCPWorld evaluate task completion independently of the agent implementation and UI state, supporting fairer comparisons across interaction strategies.The framework loads task configurations and app snapshots, launches the app, and uses registered handlers to assess signals and report task success or failure.Source: §2.1 · pp. 2–3

2.2 Tool-based Observation and Action Space

• The unified tool space supports GUI observations and actions alongside direct MCP function calls, with GUI-only, CLI-only, and multimodal input modes.MCP tools expose predefined APIs, whose availability is limited by each application; GUI tools standardize access to representations such as screenshots and accessibility trees.Source: §2.2 · pp. 3–4, Figure 1 · p. 4

2.3 Environment

• Docker standardizes the operating system and dependencies, while restoring only persistent application data provides task context without restoring a full system image.This approach is designed to reduce setup latency and resource cost while maintaining consistent starting states.Source: §2.3 · p. 4

2.4 Automated Evaluator

• The white-box evaluator hooks semantically meaningful internal events to verify outcomes and intermediate steps, addressing gaps in UI- and output-file-based checks.External UI checks can be fragile, while output-file checks can miss transient in-memory states and intermediate milestones; internal hooks can capture events such as a breakpoint hit and inspect call stacks or variable states.Source: §2.4 · p. 5, Figure 2 · p. 6

3.1 Applications and Tasks

• The benchmark includes 201 realistic tasks across 10 open-source desktop applications, covering simple actions, complex workflows, and varied difficulty levels.Tasks are designed around core app functions and reasonable initial states, then cross-validated for feasibility, clarity, and consistency; most include intermediate milestones.Source: §3.1 · pp. 6–7

3.2 Implementing Task Verification Logic

• Annotators trace app execution to identify definitive completion signals, then implement verification suited to each application’s design.Compiled apps use dynamic instrumentation; introspective-language apps use targeted code injection; exposed APIs, logs, or databases can be queried for intermediate milestones and final goals.Source: §3.2 · pp. 7–8

4.1 Experimental Setup

• Each task was attempted three times with each tool configuration, and the evaluator automatically checked final goals and key steps in the containerized benchmark.The study used the full suite of 10 applications and 201 tasks, with a 300-second limit per attempt.Source: §4.1 · p. 8, §4 · p. 8

4.2 Results

• Hybrid achieved 75.12% Task Success Rate versus GUI-Only's 70.65% and MCP-Only's 53.23%; its 69.63% Key Step Completion Rate was close to GUI-Only's 68.82% and above MCP-Only's 59.78%.These overall values are averaged across all benchmark tasks and runs.Source: §4.2 · pp. 8–9, Table 4 · p. 9

4.3 Analysis

• Limited MCP functionality and coverage largely explain MCP-Only's low success and also limit Hybrid's improvement over the GUI-Only baseline.The authors base this attribution on the failure distribution in Table 5.Source: §4.3 · p. 9, Table 5 · p. 9
• As task complexity increased, success declined by 54.85% for GUI-Only, 23.01% for MCP-Only, and 39.30% for Hybrid; the authors associate MCP with more direct, deterministic control.They describe Hybrid as using MCP when APIs are available and falling back to GUI when needed.Source: §4.3 · p. 9
• For tasks comprising 5–10 steps, success dropped after MCP support was added; the authors preliminarily attribute the decline to interruptions from overly long MCP prompts.The proposed explanation is tentative, not a confirmed cause.Source: §4.3 · p. 9
• The authors report higher task success than prior work, attributing it partly to flexible actions, recovery from errors, and a task set of common everyday tasks.They plan to add more complex tasks, including cross-application interactions.Source: §4.3 · pp. 9–10

5 Limitations

• The preliminary experiments use a single agent framework, limiting comparative coverage across agent architectures.The authors plan to incorporate a broader range of architectures.Source: §5 · p. 10
• Requiring white-box, open-source applications limits the breadth of software included, by design, to support robust and in-depth evaluation.Source: §5 · p. 10
• The current suite needs more apps and tasks, especially complex cross-application workflows and multi-turn interactions; task annotation also requires application-source understanding.The authors report that LLMs can significantly reduce the annotation burden.Source: §5 · p. 10

6 Conclusion

• MCPWorld enables white-box evaluation of GUI, API, and hybrid agents by hooking internal application behavior across approximately 201 tasks and 10 applications.Source: §6 · p. 10
• The preliminary experiments indicate the potential of hybrid approaches but reveal significant challenges for current agents on complex real-world tasks.Source: §6 · p. 10
• Future work will expand the task and application library, add multi-turn and cross-application scenarios, and test more agent architectures.The authors expect these efforts to accelerate development of more capable and reliable agents.Source: §6 · p. 10

A.1 Overall Infrastructure

• MCPWorld provides lightweight XFCE and GPU-accelerated KDE desktop configurations, selectable per task or experiment to suit different application and resource needs.The environments use TurboVNC and KasmVNC, respectively.Source: §A.1 · p. 13

A.2 Details of Tool-based Observation and Action Space

• Agents use a unified tool-based interaction loop: requests follow tool input schemas and return results that the loop passes back to the LLM.Results can include text, screenshots, errors, and optional system feedback.Source: §A.2 · pp. 13–14
• MCP tools are dynamically added to the agent's available tools at initialization, so agents invoke them through the same interface as local tools.Observation and action tools are summarized in Tables 7 and 8.Source: §A.2 · pp. 13–14, Table 7 · p. 13

A.3 Implementation of MCP Client

• MCPClient connects to configured MCP servers, discovers their tools, invokes them, and standardizes tool representations and results for the agent.It also manages session resources and can connect to one or more servers.Source: §A.3 · p. 14
• MCPWorld starts the configured server for each task, enabling the agent to discover and use tools for the relevant application.A server configuration specifies its launch command, arguments, and required environment variables; this declarative setup simplifies adding MCP-enabled applications.Source: §A.3 · pp. 14, 16

B.1 Binding Application-Under-Test

• Open-source applications are compiled from source and dynamically bound into the container, allowing different versions or instrumented builds without rebuilding the base image.This supports in-depth application analysis and white-box evaluation while maintaining a reproducibly configured environment.Source: §B.1 · p. 16

B.2 User Data and Initial State Restoration

• Each task can specify prerequisite data sources and target locations to establish its required application context.These can include user profiles, project files, or other data needed before interaction begins.Source: §B.2 · p. 16
• During task initialization, MCPWorld copies each specified source into its target location, typically using rsync to mirror the source directory.The restoration script processes the task’s configured context-data entries.Source: §B.2 · p. 16
• Restoring task-specific data gives every execution an identical, well-defined starting state and avoids variation from prior interactions or environmental drift.It also lets tasks maintain different initial conditions without interfering with one another.Source: §B.2 · pp. 16–17

C.1 Configuration and Hyperparameters

• The agent used Claude 3.7 Sonnet at temperature 0, the computer_use_20250124 tool version, and a 300-second timeout per task attempt.Temperature 0 was used to minimize response randomness and improve reproducibility.Source: §C.1 · p. 17
• Tool access varied across GUI-only, MCP-only, and hybrid setups, with the hybrid configuration combining GUI, Bash, Edit, and available MCP tools.In specific ablations, Bash and Edit were disabled across the three configurations.Source: §C.1 · p. 17

C.2 Prompt Templates

• The reference agent uses ReAct-style prompting, with system-prompt variants providing operational context and behavioral guidance for different tool configurations.The LLM also receives task instructions and dynamically compiled tool descriptions and input schemas.Source: §C.2 · pp. 17, 19
• The detailed turn-by-turn reasoning and translation of intent into tool calls are primarily managed by Claude 3.7 Sonnet, rather than specified as user-defined prompt templates.This includes selecting MCP tools and filling their parameters from the schemas.Source: §C.2 · p. 19

D Factor Analysis: Impact of BASH Tool Availability

• 75.12% Hybrid, 70.65% GUI-Only, and 53.23% MCP-Only Task Success Rate with Bash enabled; Bash-disabled rates were 65.50%, 67.00%, and 43.00%, respectively.The reported comparisons show higher overall Task Success Rate with Bash in each modality; Figure 4 compares success and key-step completion rates across modalities and Bash settings.Source: D Factor Analysis: Impact of BASH Tool… · p. 19, Table 9 · p. 19, Figure 4 · p. 20
• For hard tasks, Bash-enabled success was 40.00% in MCP-Only and 51.11% in Hybrid, versus 26.67% and 28.89% without Bash.The reported GUI-Only hard-task rates were 35.56% with Bash and 33.33% without it; for easy tasks, GUI-Only success slightly decreased with Bash, from 91.67% to 90.41%.Source: D Factor Analysis: Impact of BASH Tool… · p. 20
• Bash offers system-level flexibility alongside GUI or MCP interactions, but its limited impact on hard GUI-only tasks suggests it may not overcome complex GUI automation challenges without application-specific APIs.The paper notes that tool availability should be considered when comparing performance across benchmarks.Source: D Factor Analysis: Impact of BASH Tool… · p. 20
• Bash can let agents reach final goals without completing every predefined key step, so task success may exceed key-step completion.This divergence is an intentional consequence of allowing tools such as Bash to take shortcuts to the final goal.Source: D Factor Analysis: Impact of BASH Tool… · p. 20

• Table 10 is identified as the complete list of tasks in MCPWorld.Source: Table 10 · p. 21
• The section labels this inventory as the complete task list rather than a partial selection.Source: Table 10 · p. 22
• The task-list format includes fields for template, key steps, and difficulty level.Source: E Complete Task List · p. 30