PaperlayerView exact revision on arXiv

Source-linked AI summary

MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments

Quyu Kong, Xu Zhang, Zhenyu Yang, Nolan Gao, Chen Liu, Panrong Tong, Chenglin Cai, Hanzhang Zhou, Jianan Zhang, Liangyu Chen, Zhidan Liu, Steven Hoi, Yue Wang

arXiv:2512.19432v3cs.CL

TL;DR

Existing mobile-agent benchmarks are saturated and omit realistic application, instruction, and tool-use scenarios. MobileWorld addresses these gaps with a challenging, reproducible benchmark and an extended planner-executor framework, finding 51.7% success for the best agentic framework and 20.9% for the best end-to-end model.

  • Problem

    Existing benchmarks such as AndroidWorld are saturated and omit important applications, long-horizon cross-app workflows, vague instructions, and external tool integration.

  • Method

    MobileWorld combines realistic long-horizon tasks, agent-user interaction and MCP-augmented categories, self-hosted open-source applications, deterministic verification, and a planner-executor framework with extended actions.

  • Results

    51.7% overall success is achieved by the best agentic framework, compared with 20.9% for the best end-to-end model, while performance further degrades on agent-user interaction and MCP-augmented tasks.

  • Takeaways & Limitations

    MobileWorld reveals substantial capability gaps in ambiguity detection, collaborative dialogue, hybrid execution planning, and other requirements of real-world mobile automation.

  • Takeaways & Limitations

    The benchmark’s failure analysis identifies unresolved challenges in long-term memory and state checking, complex logic reasoning, and spatial-temporal context awareness.

Abstract

from arXiv · show

Among existing online mobile-use benchmarks, AndroidWorld has emerged as the dominant benchmark due to its reproducible environment and deterministic evaluation; however, recent agents achieving over 90% success rates indicate its saturation and motivate the need for a more challenging benchmark. In addition, its environment lacks key application categories, such as e-commerce and enterprise communication, and does not reflect realistic mobile-use scenarios characterized by vague user instructions and hybrid tool usage. We introduce MobileWorld, a substantially more challenging benchmark designed to reflect real-world usage through 201 tasks across 20 applications. MobileWorld derives its difficulty from an emphasis on long-horizon, cross-application workflows, requiring nearly twice as many completion steps on average (27.8 vs. 14.3) and featuring a significantly higher proportion of multi-app tasks (62.2% vs. 9.5%) than AndroidWorld. To overcome the limitations of existing environments, MobileWorld achieves a balance between production-grade utility and reproducible evaluation by utilizing open-source alternatives to industry standards (e.g., Mattermost for Slack). This approach enables a fully observable and controlled environment through source code modification and direct backend database access for precise verification. MobileWorld also introduces novel task categories, including agent-user interaction and Model Context Protocol (MCP)-augmented tasks, for evaluating agents in user-aware, hybrid-tool scenarios. To facilitate evaluation, we develop a planner-executor agentic framework with extended action spaces to support user interactions and MCP calls. Our results reveal a sharp performance drop compared to AndroidWorld, with the best agentic framework and end-to-end model achieving 51.7% and 20.9% success rates, respectively, highlighting ample headroom for future research.

1 Introduction

MobileWorld is introduced to address saturation and realism gaps in existing mobile-agent benchmarks, especially AndroidWorld. It raises task complexity through longer, cross-application workflows and adds agent-user interaction scenarios for ambiguous instructions.

  • Motivation: AndroidWorld’s state-of-the-art success rates exceed 90%, limiting its ability to distinguish incremental improvements from genuine breakthroughs.Its reproducible emulator and deterministic evaluation make it widely used, but its performance saturation motivates a harder benchmark.
  • Motivation: Existing benchmarks omit important application categories and usually restrict tasks to short, single-application workflows with fully specified instructions.The identified gaps include e-commerce, enterprise communication, long-horizon planning, cross-application work, and clarification dialogues.
  • Benchmark: MobileWorld is introduced as a substantially more challenging benchmark designed to better reflect real-world mobile usage.
  • Benchmark: 27.8 average completion steps versus 14.3 in AndroidWorld, while 62.2% of tasks are cross-application versus 9.5%.These workflows require long-horizon planning, memory retention, and precise instruction following; the top agentic framework reaches 51.7% success on MobileWorld versus over 90% for the best AndroidWorld agents.
  • New task categories: Agent-user interaction tasks evaluate whether agents handle ambiguous instructions through collaborative dialogue.The task design requires agents to request clarification when information is incomplete.

Agent User Interaction Task Example

MobileWorld extends mobile-agent evaluation beyond GUI-only tasks with user dialogue, external MCP tools, and reproducible verification across realistic applications. Its framework and benchmark expose substantial weaknesses in current agents, especially on hybrid interaction and tool-use scenarios.

  • Agent-user interaction: Agent-user interaction tasks require agents to recognize missing information and proactively ask a simulated user for clarification.The simulated user supplies the needed information through an LLM-based role-play interaction.
  • MCP-augmented tasks: MCP-augmented tasks combine GUI navigation with external tool invocation, such as retrieving a GitHub README before messaging its summary through Mattermost.The agent selects an appropriate GitHub MCP tool, then completes the communication workflow through GUI operations.
  • MCP-augmented tasks: 19.9% of MobileWorld tasks are MCP-augmented, making strategic choice between GUI operations and API-based tools an explicit evaluation target.
  • Evaluation infrastructure: Self-hosted open-source alternatives and source-code modification support deterministic evaluation for applications that otherwise require authentication or expose opaque internal states.Mattermost is used as a Slack alternative, while backend access and controlled environments support precise verification.
  • Empirical findings: 51.7% overall success is achieved by the best agentic framework, while most end-to-end models score below 10% on agent-user interaction and near 0% on MCP-augmented tasks.The framework extends the action space with ask_user and mcp_call to unify GUI manipulation, dialogue, and external tool use.
  • Research challenges: The benchmark identifies open challenges in ambiguity detection, MCP context management, long-term memory and state checking, complex reasoning, and spatial-temporal awareness.
  • Conclusion: MobileWorld aims to provide a rigorous, reproducible environment for advancing agents capable of long-horizon reasoning, active user interaction, and MCP tool use.

2 Related Works

Prior benchmarks established important evaluation settings for GUI agents, but mobile benchmarks remain limited in task complexity, instruction clarity, tool integration, and deterministic verification. MobileWorld addresses these gaps by unifying agent-user interaction and MCP-augmented tasks in a mobile evaluation framework.

  • AndroidWorld provides a reproducible Android environment with 116 programmatic tasks across 20 real-world applications.
  • Existing mobile benchmarks face performance saturation, fully specified instructions, limited external-tool integration, and a realism–verifiability trade-off with commercial applications.
  • Agent-user benchmarks such as τ-bench evaluate dynamic conversations with simulated users and domain-specific API tools, but operate outside mobile GUI environments.
  • MCP benchmarks assess hybrid tool invocation and GUI operation; OSWorld-MCP reports success gains from 8.3% to 20.4% for OpenAI o3, while invocation rates remain 36.3%.
  • MobileWorld is the first mobile benchmark to integrate agent-user interaction and MCP-augmented tasks within one framework for practical mobile-agent deployment.

3 MobileWorld

MobileWorld formalizes mobile-agent tasks in a POMDP and extends benchmark environments with ambiguous instructions, hybrid GUI–MCP workflows, controlled applications, and deterministic verification. Its dataset emphasizes cross-application complexity, interactive tasks, external tools, and backend-based evaluation.

  • 3.1 Task Definition: Mobile GUI tasks are modeled as a POMDP with environment states, observations, UI actions, transitions, and binary task-completion rewards.
  • 3.1 Task Definition: Agent-user interaction tasks deliberately omit critical information, requiring agents to request clarification rather than solve fully specified goals.
  • 3.1 Task Definition: MCP-augmented tasks require agents to combine external tool invocations with standard GUI operations because both interaction modes coexist in realistic applications.
  • 3.2 Environment Development: The containerized environment uses rooted Android devices, self-hosted applications, and snapshots to provide controlled deployment and reproducible starting conditions.
  • 3.2 Environment Development: MobileWorld integrates 61 MCP tools spanning navigation, repositories, documents, financial data, and scholarly literature, enabling capabilities beyond standard GUI operations.
  • 3.2 Environment Development: Deterministic evaluation combines textual matching, backend database queries, local-storage inspection, and other verification methods instead of relying on noisy model-based judging.
  • 3.4 Data Statistics: 62.2% of MobileWorld tasks require multiple apps, 42.3% involve agent-user interaction or MCP augmentation, and 47.3% rely on self-hosted database verification.

4 Experiments

Experiments show that MobileWorld remains difficult for current mobile agents, especially on interaction, MCP, and long-horizon tasks. Results also expose weaknesses in clarification, tool orchestration, memory, temporal grounding, and structured reasoning.

  • Main Results: 51.7% overall success is achieved by the best-performing model, GPT-5 + UI-Ins-7B, versus 90%+ reported on AndroidWorld.The result indicates that MobileWorld remains challenging even for state-of-the-art models.
  • Main Results: 51.7% versus 20.9% overall success separates the best agentic framework from the best end-to-end model, Doubao-1.5-UI-TARS.The reported gap is associated with complex reasoning, cross-app coordination, and support for interaction and MCP tool invocation.
  • Main Results: 62.2% interaction success is achieved by GPT-5, while Doubao-1.5-UI-TARS reaches 32.4% and Qwen3-VL and GELab-Zero remain below 10%.Among agentic frameworks, GPT-5 reaches 51.6% on MCP-augmented tasks, showing that both interaction and tool orchestration remain difficult.
  • In-depth Analysis: 24.2 average completion steps is achieved by Gemini-3-Pro + UI-Ins-7B, but low step counts can reflect early failure rather than efficiency.Doubao-1.5-UI-TARS and GUI-Owl-7B record 20.9 and 20.6 steps alongside lower success rates.
  • In-depth Analysis: 1.11 average queries and 0.40 UIQ make GPT-5’s clarification behavior strongest among the reported models.UIQ remains below interaction success across models, while Doubao-1.5-UI-TARS combines 32.4% interaction success and 1.22 queries with UIQ 0.13.
  • In-depth Analysis: 2.63 average MCP calls lead Gemini-3-Pro, followed by GPT-5 at 2.23 and Claude-4.5-Sonnet at 1.91, while Qwen3-VL’s 2.32∼3.84 calls yield only 0.0∼5.4 MCP success.The passage links successful MCP completion to accurate tool names and arguments, as well as appropriate tool use.

5 Conclusion

MobileWorld evaluates mobile GUI agents with complex workflows, user clarification, and MCP tool use in a reproducible environment. Its framework extends the action space and exposes persistent performance gaps that motivate future research.

  • Conclusion: MobileWorld combines long-horizon GUI workflows, agent-user interaction, and MCP-augmented tasks within a deterministic evaluation environment.The benchmark uses open-source applications and multiple verification methods to support controlled assessment.
  • Conclusion: 51.7% was achieved by the best agentic framework, while the best end-to-end models reached 20.9% success on MobileWorld.These results are substantially lower than performance on existing benchmarks.
  • Conclusion: Performance further degrades on agent-user interaction and MCP-augmented tasks, exposing gaps in ambiguity detection, collaborative dialogue, and hybrid execution planning.The benchmark’s failure analysis identifies context-length limitations for long-term memories and MCP tool management as research directions.
  • Conclusion: The planner-executor framework adds ask_user for clarification and mcp_call for structured external-tool invocation.These actions unify GUI manipulation, multi-turn interaction, and tool use in one closed-loop process.
  • Conclusion: The framework requires precise action selection and supports separate GUI and MCP action formats, with only one tool call allowed per action.Its execution guidance emphasizes exact output formatting, efficient recovery from failed actions, and asking users for missing information.

B.1 APP List

MobileWorld includes applications spanning everyday communication, productivity, navigation, social interaction, and e-commerce scenarios.

  • B.1 APP List: MobileWorld covers communication, productivity, navigation, social interaction, and e-commerce through applications such as Mail, Calendar, Google Map, Mattermost, and Taodian.The application list is paired with the number of associated tasks.

B.2 Open-source APP Environment Construction

The benchmark constructs self-hosted, containerized application environments using open-source implementations and backend snapshots or instrumentation for reproducible automated evaluation.

  • B.2 Open-source APP Environment Construction: Self-hosted backend environments are constructed for key open-source applications to support reproducibility and deterministic evaluation.The implementation section describes strategies for four primary applications.
  • B.2 Open-source APP Environment Construction: Mattermost and Mastodon preserve backend state through snapshots that are restored before each task execution.Their environments include database contents and file or media storage.
  • B.2 Open-source APP Environment Construction: The Mail app adds attachment selection, email search, Share-based composition, and callbacks that persist sending events for evaluation.It is adapted from a pure-frontend React Native Gmail clone.
  • B.2 Open-source APP Environment Construction: Taodian replaces its original backend with a lightweight file-based mock server and instruments checkout interactions for validation.The application also adds curated product catalogs, SMS-based login, and shopping-cart management features.
  • B.2 Open-source APP Environment Construction: The construction approach makes applications self-contained, reproducible, and suitable for automated evaluation while approximating real-world functionality.Figure 9 illustrates a Mastodon poll-editing task involving navigation, menu access, deletion, and text input.

C.1 GUI-Only Task Example: Task Completion

A GUI-only task requires editing a Mastodon poll by removing one option and replacing another through navigation, menu access, deletion, and text input.

  • C.1 GUI-Only Task Example: Task Completion: The task removes the “USA” poll option and changes “Brazil” to “Canada” in Mastodon.The demonstrated procedure navigates to the post, opens the edit menu, deletes the unwanted option, and types the replacement.

C.2 GUI-Only Task Example: Information Retrieval

The information-retrieval example evaluates a GUI agent’s ability to obtain a current fact through Chrome and return it in the requested format. Both execution and validation depend on real-time information.

  • The agent launches Chrome and searches for Beijing’s highest temperature today.It then extracts the current forecast from Google’s dynamic weather widget.
  • Execution and validation are real-time so the answer reflects the day’s actual temperature.
  • The task requires returning only an integer Celsius temperature.The specified output for the example is 11°C.

C.3 Agent-User Interaction Task Example

The agent-user interaction example tests how an agent handles an ambiguous request to remove calendar events. It must clarify the intended deletion before performing a potentially disruptive action.

  • The request to remove a few events from October 20 does not identify which events should be deleted.
  • The agent asks whether to delete only “Meet with Sam” or the entire recurring series.
  • User confirmation is required because deleting the wrong occurrence or series could disrupt the schedule.

C.4 MCP-Augmented Task Example

The MCP-augmented example combines external tool use with mobile GUI actions to retrieve repository data and email it to a specified recipient. The agent must transfer MCP results into a composed message.

  • The agent invokes an MCP tool to fetch the three most recent commits from the google-research/android_world repository.
  • It opens Mail, selects Compose, and fills the recipient, subject, and body with the retrieved commit information.The requested format is “author: commit message.”
  • The task requires combining repository retrieval with email delivery through a mobile interface.
Loading 2512.19432v3…