PaperlayerView exact revision on arXiv

Source-linked AI summary

MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks

Yi Zhu, Xiongwei Wu, Qiyi Wang, Tingyu Qu, Jiajun Liu, Sihan Cao, Long Chen, Weigao Sun, Feida Zhu, Yiran Zhong, Steven Hoi

arXiv:2608.23035v1cs.AI

TL;DR

Existing mobile-agent benchmarks miss either live runtime constraints or high-level planning beyond GUI manipulation. MobilePA-Bench provides a stateful, tool-centric sandbox spanning broad mobile tasks and capability dimensions, and evaluations find that frontier LLMs remain unreliable, with the best overall score reaching 75.52%.

  • Problem

    Existing GUI-centric and static function-calling benchmarks do not jointly evaluate background tool use, long-horizon planning, and live mobile runtime constraints.

  • Method

    MobilePA-Bench evaluates central planners in a stateful sandbox with live backend mutations, structured feedback, evidence-aligned verification, and four capability dimensions.

  • Results

    The best overall score is 75.52%, while models struggle with compound reliability across tool ordering, runtime exceptions, memory, and multi-agent coordination.

  • Takeaways & Limitations

    MobilePA-Bench functions as a diagnostic framework for improving stateful feedback learning, inter-agent communication, and memory-tool grounding.

Abstract

from arXiv · show

As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capability evaluation essential. Yet existing benchmarks fall into two camps, each with a critical blind spot: GUI-centric benchmarks test surface-level screen manipulation while overlooking background tool use and long-horizon planning, whereas static function-calling benchmarks rely on offline API matching that is detached from real runtime constraints. To close this gap, we present \textbf{MobilePA-Bench}, an interactive, stateful, and tool-centric benchmark for evaluating the tool-calling and planning abilities of mobile planning agents. MobilePA-Bench runs on an executable sandbox that maintains live application databases and returns structured feedback, spanning $13$ functional domains and $212$ realistic mobile tools. Beyond basic tool use, it evaluates a central planning agent along three advanced dimensions: \emph{(1)~Sub-agent Collaboration}---decomposing a complex task and delegating specialized work to capable sub-agents; \emph{(2)~Memory Usage}---recalling stored memories, user profiles, and past preferences to resolve implicit requests; and \emph{(3)~Skill Usage}---invoking pre-packaged composite skills instead of planning every step from scratch. Extensive experiments show that current frontier LLMs remain unreliable in mobile settings: performance drops sharply under strict tool ordering, permission limits, and unexpected runtime errors. By pairing an interactive function-calling sandbox with evidence-based verification, MobilePA-Bench serves as both a practical diagnostic benchmark and an interactive foundation for agentic reinforcement learning---accelerating the development of dependable mobile agents.

1. Introduction

MobilePA-Bench addresses the limitations of GUI-centric and static function-calling benchmarks by evaluating mobile planners in complex, stateful workflows. It covers four capabilities and exposes substantial reliability failures in current LLMs.

  • Motivation: Existing benchmarks overlook either real-time OS feedback and exceptions or system-level tool orchestration beyond GUI manipulation.Static function-calling benchmarks use offline matching, while vision-centric benchmarks focus on pixel-level interaction.
  • Motivation: Realistic mobile tasks require memory retrieval, structured API and skill execution, multimodal tool use, and sub-agent delegation.These capabilities must be coordinated across complex workflows such as travel planning.
  • Benchmark: MobilePA-Bench evaluates 1,705 tasks across 13 domains and 212 tools in an interactive, stateful sandbox with environmental friction and structured feedback.The platform includes call dependencies, permission blocks, and dynamic state mutations while decoupling planning from visual parsing overhead.
  • Findings: 75.52% overall is the best-performing model’s score, while current LLMs struggle with strict ordering, runtime exceptions, ambiguous memory retrieval, and multi-agent coordination.These results motivate diagnostic evaluation of dependable mobile planning rather than isolated tool matching.

2. Related Works

Prior mobile-agent benchmarks emphasize GUI interaction or static function matching, leaving live OS dependencies and broader planner capabilities insufficiently tested. MobilePA-Bench unifies stateful execution with memory, skills, and sub-agent collaboration in realistic mobile workflows.

  • GUI-centric benchmarks: GUI-centric benchmarks test screenshot interpretation and pixel-level actions but do not cover the broader mobile ecosystem or central planning.MobilePA-Bench instead delegates fine-grained interface actions to downstream GUI sub-agents.
  • Function-calling benchmarks: Static function-calling benchmarks evaluate isolated tool selection and parameter formatting without executing calls in live, stateful environments.They therefore omit dynamic OS states and real-world OS dependencies described for mobile settings.
  • MobilePA-Bench: MobilePA-Bench executes tools under call dependencies, permission boundaries, and runtime exceptions, enabling assessment of adaptive recovery during execution.The sandbox provides dynamic, real-time feedback rather than offline matching.
  • MobilePA-Bench: MobilePA-Bench embeds sub-agent collaboration, memory usage, and skill orchestration into realistic mobile workflows instead of isolated synthetic tasks.This integrates advanced planning capabilities with basic tool use in one interactive suite.

3.1. Benchmark Scope and Design Axes

MobilePA-Bench models mobile planning as a stateful orchestration loop in which a central planner selects tools, retrieves context, loads skills, and delegates specialized work. Verification is separated from the capability taxonomy and uses evidence-aligned completion criteria.

  • Central planning: The central planner converts natural-language requests into correct, verifiable action sequences in a stateful phone environment.It performs high-level reasoning while invoking APIs, retrieving memory, loading skills, and delegating specialized work.
  • Sub-agent architecture: GUI execution is a downstream sub-agent capability, while the central planner retains decomposition, route selection, and cross-step coordination.Other sub-agents encapsulate visual processing and conditional monitoring.
  • Capability dimensions: The benchmark separates Basic Tool Use, Sub-agent Collaboration, Memory Usage, and Skill Usage as four capability dimensions.The dimensions cover API grounding and recovery, routing and handoffs, memory-grounded disambiguation, and reusable procedures.
  • Evaluation loop: The orchestration loop combines structured tools, modular sub-agents, memory, reusable skills, and environmental feedback from a stateful mobile environment.The environment returns state changes or system errors so the planner can update its next decision.
  • Verification: Verification routes queries to Tool Call, State Change, or Agent Behavior buckets according to the observable evidence that best establishes completion.Execution remains independent of the capability taxonomy.

3.2. Interactive Mobile Execution Environment

MobilePA-Bench models mobile tasks as closed-loop interactions in which a central planner selects structured actions, receives execution feedback, and replans until termination. Its stateful sandbox unifies tool interfaces, persistent database mutations, execution logs, and verification under realistic runtime friction.

  • Task Formulation: A mobile task combines a user intent, initial state, dialogue history, and active candidate actions, with the planner selecting the next action from these inputs.
  • Task Formulation: The sandbox executes each action, updates the environment, returns structured feedback, and supports error-aware replanning until Finish or the maximum step threshold.
  • Action Interfaces and Unified Action Space: All system operations, sub-agent dispatches, memory queries, and skill invocations use structured function schemas with names, descriptions, and JSON argument specifications.
  • Action Interfaces and Unified Action Space: Skill loading dynamically expands the planner’s active action space by adding concrete tool schemas associated with the invoked skill.
  • Stateful Simulation Sandbox: The sandbox integrates tool schemas, executable implementations, and a persistent backend that validates calls, mutates application state, records operations, and returns dynamic feedback.
  • Stateful Simulation Sandbox: Controlled obstacles such as missing parameters, permission blocks, and entity ambiguities force real-time plan repair while enabling deterministic execution and trajectory replay.

3.3. Benchmark Construction and Evaluation

MobilePA-Bench constructs executable tasks around diverse completion semantics and evaluates them with evidence-aligned verification rather than one universal metric. Fixed primary checkers cover exact tool calls, terminal state changes, or observable agent behavior, with capability-specific gates added when required.

  • Benchmark Construction: Benchmark tasks are constructed around four capability dimensions, while verification policies are assigned separately according to each query’s most reliable completion evidence.
  • Evidence-Aligned Task Verification: A single rigid verification metric is unsuitable because mobile tasks may require canonical tool sequences, equivalent state outcomes, or open-ended interaction.
  • Evidence-Aligned Task Verification: MobilePA-Bench assigns tasks to three non-interchangeable query buckets, each paired with a checker matched to its completion semantics.
  • Evidence-Aligned Task Verification: Tool Call tasks require exact tool names, ordering, argument fields, normalized values, and no extraneous side-effect calls.
  • Evidence-Aligned Task Verification: State Change tasks compare terminal database transitions against annotated target deltas while allowing multiple valid execution paths and rejecting unrelated destructive writes.
  • Evidence-Aligned Task Verification: Agent Behavior tasks evaluate observable interaction trajectories for open-ended activities such as delegation, clarification, and recommendations using task-specific rubrics.

3.4. Task Construction Across Capability Dimensions

MobilePA-Bench constructs tasks across four capability dimensions, using realistic mobile workflows, annotated state and action spaces, and dimension-specific verification. The design covers direct tools, delegation, memory retrieval, and reusable skills.

  • The benchmark organizes mobile workflows around Basic Tool Use, Sub-agent Collaboration, Memory Usage, and Skill Usage.These dimensions span foundational execution and three advanced planning capabilities.
  • Basic Tool Use: Basic Tool Use tasks inject reference obfuscation, permission blocks, missing arguments, and state mutations to test execution under dynamic friction.Evaluation verifies deterministic tool calls or equivalent state changes without extraneous or destructive actions.
  • Sub-agent Collaboration: Sub-agent Collaboration tasks require selecting a valid downstream route and supplying the contextual payload needed for specialized execution.Success measures delegation and handoff quality rather than downstream policy execution.
  • Memory Usage: Memory Usage tasks omit explicit preferences, requiring persistent-memory queries and successful use of all required gold memory IDs.The 1,530-query scenario snapshot covers 13 high-level scenarios and 89 level-2 subcategories, separately from the 1,705-task evaluation denominator.
  • Skill Usage: Skill Usage tasks specify a gold skill and require loading it before completing the downstream task under its fixed checker.Joint success is evaluated separately for Skill-Only Routing and Mixed Tool-Skill Routing.

3.5. Benchmark Scoring and Aggregation

The benchmark combines four capability scores into a fixed, full-denominator overall metric. Basic Tool Use receives the largest weight, while the remaining weights reflect advanced coordination, personalization, and skill execution.

  • MobilePA-Bench computes overall performance using immutable full-denominator scoring, counting missing or invalid predictions as failures.
  • Scoreoverall = 0.50 × ScoreBasic + 0.10 × ScoreSubAgent + 0.20 × ScoreMemory + 0.20 × ScoreSkill.
  • Basic Tool Use aggregates five behavioral categories, while Sub-agent Collaboration uses routing-and-handoff success and Memory Usage uses end-to-end success.

4. Experiments

Experiments evaluate 13 models across 1,705 tasks and four capability dimensions using fixed verification criteria. Results show low overall reliability, pronounced memory and coordination weaknesses, fragmented model strengths, and compound failures across capabilities.

  • 1,705 tasks across 212 tools and 13 domains evaluate Basic Tool Use, Sub-agent Collaboration, Memory Usage, and Skill Usage.The complete suite contains 1,040 Basic Tool Use, 89 Sub-agent Collaboration, 376 Memory Usage, and 200 Skill Usage tasks.
  • Main Results: Basic Tool Use peaks at 83.85%, whereas Sub-agent Collaboration ranges from 43.82%–77.53% and Memory Usage from 33.78%–64.63%.
  • Capability Trade-offs: Claude-Opus-5 leads Basic Tool Use and Skill Usage, Qwen-3.8-Max leads Memory Usage, and Gemini-3.1-Pro leads Sub-agent Collaboration.The strongest dimension-specific results are distributed across different models rather than concentrated in one planner.
  • Memory Usage: 64.63% is Qwen-3.8-Max’s Memory Usage score, with the strongest model failing more than one third of memory-bound tasks.Memory success requires both retrieving relevant evidence and applying it correctly downstream.
  • Discussion: Compound reliability, rather than isolated capability absence, is identified as the core bottleneck because realistic workflows combine memory, skills, stateful tools, and delegation.

5. Conclusion

MobilePA-Bench evaluates central planning agents through a stateful, tool-centric orchestration loop with live backend mutations, environmental friction, and evidence-aligned verification. Its results show that dependable mobile intelligence still requires stronger state-aware reasoning, communication, and memory-grounded execution.

  • MobilePA-Bench evaluates central planners that execute APIs, query personalized memory, load composite skills, and delegate specialized subtasks in realistic mobile environments.GUI control is integrated as a modular downstream route within the orchestration loop.
  • The sandbox exposes live backend mutations, environmental friction, and dynamic feedback while applying evidence-aligned verification across three query buckets.
  • 75.52% is the strongest overall weighted score, with failures concentrated in parameter grounding, delegation timing, and personalized-context application.
  • The findings call for unified progress in state-aware reasoning, disciplined inter-agent communication, and memory-grounded execution.
Loading 2608.23035v1…