PaperlayerView exact revision on arXiv

Source-linked AI summary

SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation

Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, Kaiwen Zhou, Rui Shao, Liqiang Nie, Yasheng Wang, Jianye Hao, Jun Wang, Kun Shao

arXiv:2410.15164v3cs.AI

TL;DR

Existing smartphone-agent benchmarks have limited task and agent coverage, and automated success detection can require human intervention or inaccessible data. SPA-BENCH evaluates agents across diverse tasks with a unified framework and automated evaluation; experiments find cross-task performance gaps and practical costs that constrain deployment.

  • Problem

    Existing smartphone-agent benchmarks often focus on English system or Google apps, evaluate few agents, and use success detection that may need human input or inaccessible data.

    Source: §1 · p. 2

  • Method

    SPA-BENCH combines diverse English and Chinese single- and cross-app tasks with an 11-agent framework and automated evaluation of task completion and resource consumption.

    Source: §1 · pp. 2–3

  • Results

    Across tasks, even the best-performing agents completed less than 70% successfully, with agentic-workflow agents outperforming agent-as-a-model agents and cross-app tasks harder than single-app tasks.

    Source: §6.1 · p. 7, §7 · p. 10

  • Takeaways & Limitations

    The findings identify memory, robust error handling, and accurate self-evaluation as areas needing improvement for smartphone agents.

    Source: §7 · p. 10, §6.3 · p. 10

    Some agents’ long execution times and high costs limit their practicality for real-world use.

    Source: §6.2 · p. 9, §7 · p. 10

Abstract

from arXiv · show

Smartphone agents are increasingly important for helping users control devices efficiently, with (Multimodal) Large Language Model (MLLM)-based approaches emerging as key contenders. Fairly comparing these agents is essential but challenging, requiring a varied task scope, the integration of agents with different implementations, and a generalisable evaluation pipeline to assess their strengths and weaknesses. In this paper, we present SPA-Bench, a comprehensive SmartPhone Agent Benchmark designed to evaluate (M)LLM-based agents in an interactive environment that simulates real-world conditions. SPA-Bench offers three key contributions: (1) A diverse set of tasks covering system and third-party apps in both English and Chinese, focusing on features commonly used in daily routines; (2) A plug-and-play framework enabling real-time agent interaction with Android devices, integrating over ten agents with the flexibility to add more; (3) A novel evaluation pipeline that automatically assesses agent performance across multiple dimensions, encompassing seven metrics related to task completion and resource consumption. Our extensive experiments across tasks and agents reveal challenges like interpreting mobile user interfaces, action grounding, memory retention, and execution costs. We propose future research directions to ease these difficulties, moving closer to real-world smartphone agent applications. SPA-Bench is available at https://ai-agents-2030.github.io/SPA-Bench/.

1 INTRODUCTION

Fair comparison of smartphone agents is difficult because benchmarks vary in task scope and evaluation methods, while existing coverage is narrow. SPA-BENCH brings diverse tasks, multiple agents, and automated evaluation together, and reports that workflow agents outperform agent-as-a-model agents despite practical time and cost constraints.

1 INTRODUCTION

  • Figure 1 shows tasks assigned to agents for execution, with execution results passed to evaluation for measuring task completion and resource consumption.Source: Figure 1 · p. 2
  • Earlier benchmarks often focused on English system or Google apps, evaluated fewer than five agents, and relied on success detection needing human input or inaccessible data.Action-based evaluation can face multiple valid action sequences, while state-based methods can struggle to identify the right post-action state.Source: §1 · p. 2
  • SPA-BENCH evaluates agents across English and Chinese apps, single- and cross-app tasks, and third-party apps, integrating 11 agents with automated assessment.Its evaluation covers task completion and resource consumption.Source: §1 · pp. 2–3
  • SPA-BENCH’s experiments found that agentic-workflow agents outperform agent-as-a-model agents, although time and cost constraints limit real-world practicality.The benchmark also analyzes smartphone-agent capabilities and limitations and discusses future research directions.Source: §1 · pp. 2–3

2 RELATED WORK

Prior smartphone-agent work varies in how agents are built and how they observe interfaces, and evaluation approaches face distinct limitations. SPA-BENCH compares these agent types and introduces hybrid evaluation approaches for a broader range of apps and tasks.

2 RELATED WORK

  • Smartphone agents use agent-as-a-model or agentic-workflow designs and observe interfaces through text, vision, or combined inputs.SPA-BENCH evaluates all three observation types.Source: §2 · p. 3
  • Success detection includes human evaluation and automated action-based, state-based, and hybrid methods, each with trade-offs in cost, accuracy, or coverage.Action-based methods face non-unique correct sequences; state-based methods may miss minor actions.Source: §2 · p. 3
  • SPA-BENCH introduces two hybrid approaches for single- and cross-app tasks that support a wider range of apps without hand-crafted validation logic.The paper compares its evaluation against other automated smartphone-agent benchmarks across agents, tasks, and evaluation dimensions.Source: §2 · p. 3

3 SPA-BENCH TASK

SPA-BENCH assembles everyday smartphone tasks across English and Chinese apps, with difficulty structured around task complexity and the number of apps involved. Human annotations provide descriptions, trajectories, and, for single-app tasks, final-state components used in evaluation.

3.1 OVERVIEW

  • The task collection spans 39 English and 29 Chinese apps, with 150 single-app and 20 cross-app tasks for each language, focusing on everyday core functions.The collection includes third-party apps, and its diversity increases complexity through Chinese-language use and intricate interfaces.Source: §3.1 · pp. 3–4
  • Single-app tasks progress through three difficulty levels, while cross-app difficulty depends on using two or three apps and coordinating between them.Single-app levels generally involve fewer than five, fewer than ten, and fewer than fifteen actions; cross-app sets cover six task types.Source: §3.1 · p. 4
  • Figure 2 illustrates simpler tasks as foundations for more complex ones, with shared early trajectories and final screenshots marking key components.Source: Figure 2 · p. 4

3.2 TASK CONSTRUCTION

  • Human annotators created task descriptions and screenshot trajectories with one action between adjacent screenshots; single-app annotations also identify final-state key components.Cross-app annotations include descriptions and trajectories, but omit final-state components because their final states are flexible.Source: §3.2 · p. 4
  • SPA-BENCH contains 300 single-app and 40 cross-app tasks, evenly divided between English and Chinese, and annotated tasks were checked for clarity, trajectory accuracy, component quality, and feasibility.Feasibility checks covered different Android devices, Android versions, and app versions.Source: §3.2 · pp. 4–5

4 AGENT FRAMEWORK

SPA-BENCH uses isolated worker processes to connect agents with Android emulators and supports adding agents with minimal integration. Emulator snapshots support consistent experiment setup, although externally stored app data can require manual cleanup.

4.1 A UNIFIED PLUG-AND-PLAY FRAMEWORK

  • The framework connects worker processes, agents, and Android emulators, passing task descriptions and collecting trajectories and logs while agents act on device observations.Agents are independently isolated, and new agents can be added with minimal integration.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2 SNAPSHOT-BASED EMULATOR FOR CONSISTENT TESTING

  • Snapshots restore emulator state, including installed apps, credentials, and settings, reducing repeated setup; external app data may still require manual intervention after experiments.A 24-core CPU with 64GB RAM can support up to eight emulators or worker processes, depending on agent resource needs.Source: §4.2 · p. 5

5 AUTOMATED EVALUATION PIPELINE

Section 5 defines seven metrics spanning task completion and resource use, then evaluates success through separate automated pipelines for single-app and cross-app tasks.

5.1 METRICS

  • Completion metrics cover task success, step ratio, termination reason, premature termination, and overdue termination.Step ratio is considered only for successful tasks; premature and overdue termination signals apply only to their corresponding termination reasons.Source: §5.1 · pp. 5–6
  • Time spent and API cost measure resource consumption, but apply only to agents using proprietary MLLMs.For locally hosted fine-tuned models, execution time depends heavily on computational resources and there are no external API costs.Source: §5.1 · p. 6

5.2 SUCCESS DETECTION

  • Single-app success detection reaches F1 scores of 0.926 for English tasks and 0.884 for Chinese tasks; cross-app detection reaches 0.845.The single-app scores are compared with human evaluations, and the cross-app method is described as closely aligning with them.Source: §5.2 · pp. 6–7
  • Single-app evaluation first matches annotated key components against execution screenshots, then uses an MLLM with action information for final success determination.The coarse stage filters trajectories irrelevant to the task before fine detection.Source: §5.2 · p. 6, Figure 4 · p. 6
  • Cross-app evaluation splits the full trajectory into app-based segments and checks subtasks sequentially until the final subtask or an earlier failure.The approach evaluates subtasks rather than the entire task at once.Source: §5.2 · pp. 6–7, Figure 5 · p. 6

6 EXPERIMENTS

M3A led overall success rates, while English and single-app tasks were generally easier than Chinese and cross-app tasks. The experiments also identify grounding, memory, termination, error recovery, and execution costs as persistent challenges.

6.1 OVERVIEW OF SUCCESS RATE

  • M3A achieved the highest success rates across all task sets; agents generally did better on English and single-app tasks, while cross-app tasks were harder.Table 2 reports success rates across task types; AutoDroid was tested only on single-app tasks because its framework supports only those tasks.Source: §6.1 · p. 7, Table 2 · p. 7
  • Agents using proprietary models had a significant success-rate advantage over fine-tuned models, whose fixed-UI action optimization often failed in dynamic settings.Visual layout inputs can also be incomplete or too complex, leaving action grounding difficult for end-to-end completion.Source: §6.1 · pp. 7–8
  • On cross-app tasks, M3A completed only 6 of 40, while most other agents completed no more than four tasks across English and Chinese apps.Long sequences require app switching, reasoning, and retaining prior actions; even OdysseyAgent, designed for cross-app tasks, sometimes faltered at transitions.Source: §6.1 · p. 8

6.2 COMPLETION- AND CONSUMPTION-RELATED METRICS

  • Higher success rates correlated with lower mean step ratios, while task termination exposed a trade-off between premature and overdue stopping.M3A's mean step ratio was 0.92, partly reflecting combined actions and shortcuts; SeeAct had the lowest PTR (0.100) but highest OTR (0.276).Source: §6.2 · pp. 8–9
  • Agents without error-handling mechanisms were more prone to failure and early termination after execution errors.Examples include parsing failures when required output phrases are absent and failures when necessary XML files cannot be accessed.Source: §6.2 · p. 9
  • Agentic-workflow systems face cost and speed trade-offs: lower-cost agents may be slow or less successful, while faster, more successful agents can be costly.AutoDroid used $0.008 per step but took 34 seconds per step and had a 0.327 success rate; M3A and T3A cost around $0.10 per step, while MobileAgentV2 took 56.1 seconds per step.Source: §6.2 · p. 9

6.3 KEY INSIGHTS

  • The proposed improvement areas include stronger visual grounding across complex interfaces and more diverse fine-tuning data spanning languages, instructions, and app scenarios.The authors note that a11y-tree and OCR approaches perform relatively well in English, while Chinese interfaces can be more visually complex and dynamic.Source: §6.3 · pp. 9–10
  • The authors call for better memory, reflection, error recovery, task-completion detection, and execution efficiency to address failures in long or dynamic tasks.Suggested mechanisms include memory-augmented or episodic architectures, reflection and recovery loops, and reducing module overhead or input-context size; the experiments also report easier tasks as more successful and include open-ended tests and case studies.Source: §6.3 · pp. 9–10, §6.2 · p. 9

7 CONCLUSION

SPA-BENCH evaluates smartphone agents across diverse tasks, but even the best agents complete less than 70% successfully. The experiments identify performance gaps and practical limits, motivating improvements in memory, error handling, evaluation, and efficiency.

7 CONCLUSION

  • Even the best-performing agents complete less than 70% of tasks successfully.The conclusion reports significant gaps between agentic-workflow and agent-as-a-model agents, particularly in action grounding and generalisation within complex Chinese apps.Source: §7 · p. 10
  • Long execution times and high costs limit the real-world practicality of agents that excel on simpler tasks.Source: §7 · p. 10
  • The conclusion calls for better memory, error handling, self-evaluation, integration of reasoning with UI understanding, and lower execution time and cost.Source: §7 · p. 10
  • Fine-tuned-model agents should be adapted to diverse scenarios and focus on long-sequence decisions rather than isolated actions.Source: §7 · p. 10

A LIMITATION AND FUTURE WORK

The authors plan to broaden SPA-BENCH's task collection, framework support, and evaluation methods. Proposed work also addresses task diversity, emulator feasibility, experimental robustness, and finer-grained success assessment.

A LIMITATION AND FUTURE WORK

  • The task collection is planned to expand in scope and presentation diversity, include tasks easier to test on emulators, and be evaluated through repeated experiments.Examples of presentation diversity include vague task descriptions and varied human tones.Source: A LIMITATION AND FUTURE WORK · p. 14
  • The framework is planned to support more agents, including locally deployable and cloud-based agents, and agents for Android tablets and iOS devices.Source: A LIMITATION AND FUTURE WORK · p. 14
  • Evaluation work will improve single-app success detection, extend cross-app success detection, and replace binary success with a finer-grained task-completion metric.Source: A LIMITATION AND FUTURE WORK · p. 14

B.1 TASK APPS

This section documents SPA-Bench’s bilingual single-app and cross-app task suites, including the single-app English inventory, app-category distribution, and task components and human trajectories. It also describes agent integration, evaluation procedures, and handling of execution errors and device constraints.

B.1 TASK APPS

  • Figure 6 presents app categories for 300 single-app tasks, with separate English and Chinese views and circle sizes proportional to task counts.The supplied figure caption does not give individual app counts.Source: Figure 6 · p. 14, §B.1 · p. 14

B.2 LIST OF TASKS

  • 340 tasks are organized into single-app English, single-app Chinese, cross-app English, and cross-app Chinese categories in Tables 4–7.The supplied passage identifies the corresponding tables but does not provide their full contents.Source: §B.2 · p. 14

B.2.1 SINGLE-APP ENGLISH TASKS

  • Single-app English tasks span app searches and filters, content creation, settings changes, messaging, and multi-step actions such as saving or confirming results.Examples include filtering shopping results, changing device settings, creating notes, and sending WhatsApp messages.Source: §B.2.1 · pp. 15–17

B.2.2 SINGLE-APP CHINESE TASKS

  • Table 5 is identified as containing the single-app Chinese tasks.The supplied passage gives the table caption but no task examples or further details.Source: Table 5 · p. 18
  • Chinese single-app tasks cover everyday actions across services, including telecom, shopping, travel, healthcare, social media, and productivity apps.Examples include searching and filtering products, posting or commenting on social media, booking appointments, and editing documents.Source: §B.2.2 · pp. 19–21

B.2.3 CROSS-APP ENGLISH TASKS

  • Cross-app English tasks span information management, media, social sharing, and web shopping, linking actions across apps such as Chrome, Calendar, Spotify, and eBay.Examples include researching a date and setting a Calendar reminder, or comparing Catan prices on Amazon and Temu before adding the cheaper option to a cart.Source: Table 6 · p. 21, §B.2.3 · p. 22

B.2.4 CROSS-APP CHINESE TASKS

  • Cross-app Chinese tasks link apps for music, search, shopping, maps, travel, and social sharing, including recording prices from Taobao and JD.com in Tencent Docs.Other examples include sharing a route from Meituan through Amap to WeChat Moments and saving a product image from Taobao to Gallery.Source: Table 7 · p. 22, §B.2.4 · p. 22

B.3 EXAMPLE OF KEY COMPONENTS

  • Figures illustrate key components across difficulty levels, human-annotated cross-app task trajectories, and the distribution of human-executed task steps by difficulty and task type.The text points to Figure 7 for key components and Figure 9 for the step distribution; Figure 8 shows English cross-app task examples with trajectories.Source: Figure 7 · p. 23, §B.3 · p. 23, §B.4 · p. 23

C INTEGRATED AGENTS

  • The framework integrates 11 agents with varied models, modalities, and action spaces, adapting implementations where needed for Android interaction and end-to-end execution.Fine-tuned agents primarily trained on offline AITW are tested on real-world tasks requiring precise action grounding and long-sequence execution; UI inputs may include screenshots, XML, accessibility trees, or OCR.Source: C INTEGRATED AGENTS · pp. 23–24, §C.1 · p. 24, Table 8 · p. 25

C.3 LOGS AND ERRORS

  • The framework logs actions, performance metrics, and errors, and automatically reruns tasks affected by unexpected failures to avoid penalizing agents for them.Expected errors include invalid responses or restricted functionality; unexpected errors include network failures, Android malfunctions, and CAPTCHA challenges.Source: §C.3 · p. 25

C.4 SCOPE OF USING ANDROID EMULATOR

  • Some English tasks involving WhatsApp and OneNote, and most Chinese tasks, run on physical Android devices because of app restrictions and emulator compatibility issues.Physical devices replace emulators but do not provide the snapshot functionality described for emulators.Source: §C.4 · p. 25

D.1 COARSE DETECTION: KEY COMPONENT MATCHING

  • Evaluation first matches OCR-extracted text against final-state key components from the last screenshot backward, then uses a GPT-4o-based MLLM evaluator to judge task success when coarse detection succeeds.If no key-component match is found, the task is marked failed and fine detection is skipped; the evaluator receives task descriptions, screenshots, and actions, with actions represented as text or concatenated screenshots.Source: §D.1 · pp. 25–26, §D.2 · p. 26

D.3 APPROACH EVALUATION AND RESULTS

The pipeline combines coarse and fine success detection, with its best-performing modes aligning with human evaluations on English and Chinese tasks. The appendix also details prompt variants and a sequential procedure for evaluating cross-app tasks.

D.3 APPROACH EVALUATION AND RESULTS

  • Coarse detection reduced fine-detection calls and improved F1, particularly on Chinese tasks; reasoning modes showed no significant differences.Adding action data improved decision-making but increased token length and sometimes led to hallucinations.Source: §D.3 · p. 26
  • F1 reached 0.926 on English tasks and 0.884 on Chinese tasks in the best-performing modes, which were used for later evaluations.The selected modes were result-only reasoning with image action for English and reason-and-result with text action for Chinese.Source: §D.3 · p. 26

D.4.1 SYSTEM PROMPT

  • The system prompt judges completion from screenshots, requiring all subtasks, entities, and actions to match while allowing corrected or intermediate completions.It also permits action information when screenshots alone are insufficient and treats dependent subtasks as order-constrained.Source: §D.4.1 · pp. 26–27, §D.4.2 · p. 27

D.4.3 BASE PROMPT

  • Base prompts supply the task description, execution history, screenshots, and optional reasoning or action information, with action details consulted only when needed.Text-action and image-action variants describe the transition from one screenshot to the next; result-only and reason-and-result prompts specify different output formats.Source: §D.4.3 · p. 27, §D.4.4 · p. 28, §D.4.5 · p. 28

D.5 EXAMPLE OF SUCCESS DETECTION

  • The M3A Airbnb example passes both coarse and fine detection because all four annotated key components match OCR-extracted text in the final screenshot.Figure 10 illustrates the Airbnb Level 2 task.Source: Figure 10 · p. 29, §D.5 · p. 29

E.1 SUBTASK GENERATION

  • Cross-app tasks assign each subtask to one app and use history phrases matched against memory phrases when later subtasks depend on earlier information.Subtasks are generated by an MLLM and reviewed by humans.Source: §E.1 · p. 29

E.2 STAGE 1: TRAJECTORY SPLIT

  • Stage 1 segments screenshots according to the task’s ordered app list; an invalid segment sequence makes the task unsuccessful.Source: §E.2 · p. 29

E.3 STAGE 2: SEQUENTIAL SUBTASK SUCCESS DETECTION

  • Stage 2 evaluates subtasks sequentially, supplies stored memory when required, and marks the whole task successful only if every subtask passes.A failed predecessor prevents later subtasks from being checked.Source: §E.3 · p. 30

E.4 APPROACH EVALUATION AND RESULTS

Cross-app success detection is evaluated against human judgments and yields lower performance than single-app detection, in a setting dominated by true negatives. The appendix describes the prompts and configurations used for segmentation, memory, subtask generation, and agent evaluation.

E.4 APPROACH EVALUATION AND RESULTS

  • Cross-app success detection scores lower than single-app detection; with over 90% true negatives, a small number of errors substantially affects F1, while false positives and false negatives occur at similar rates.The authors nevertheless report that detection reflects each agent’s performance.Source: §E.4 · p. 30

E.5.1 SYSTEM PROMPT OF STAGE 1

  • Stage 1 prompt instructions split screenshots into app segments in the supplied app-list order and disregard irrelevant screens and distractors.The prompt requires at least two screenshots per app interaction, except that the final app may not require a quit action.Source: §E.5.1 · p. 30

E.5.2 USER PROMPT OF STAGE 1

  • The Stage 1 user prompt supplies the app list and requires the final output to preserve its order.Source: §E.5.2 · p. 31

E.5.3 SYSTEM PROMPT OF STAGE 2 MEMORY

  • The Stage 2 memory prompt asks the MLLM to summarize screenshot information relevant to a supplied description, excluding unrelated content.The user prompt provides that description as memory_text.Source: §E.5.3 · p. 31, §E.5.4 · p. 31

E.5.5 SUBTASK GENERATION

  • Subtask-generation prompts specify each subtask’s app, action, history dependence, and memory, with adjacent subtasks assigned to different apps.A later subtask can reuse the prior subtask’s memory when it depends on that information.Source: §E.5.5 · pp. 31–32

F.1 AGENT CONFIGURATION

  • Agent configurations upgraded seven agents’ off-the-shelf models to GPT-4o, used Auto-UI-Base for Auto-UI and DigiRL, and retained developers’ defaults for other settings.Agents were halted after twice the task’s golden-step count; some agents also received specific input or history features.Source: §F.1 · p. 32

F.2 EXPERIMENTAL RESULTS

Agent performance improves on easier tasks and declines as difficulty rises. The appendix identifies the task groups and notes that AutoDroid was evaluated only on single-app tasks.

F.2 EXPERIMENTAL RESULTS

  • Tables 11–13 report detailed results for single-app Chinese, cross-app English, and cross-app Chinese tasks, respectively.The table captions identify task performance; token costs for four agents are omitted because they use locally hosted open-source models.Source: §F.2 · p. 32, Table 12 · p. 33, Table 13 · p. 33

F.3 PERFORMANCE ACROSS TASK DIFFICULTY LEVELS

  • Agents perform better on easier tasks, consistent with the benchmark’s increasing difficulty levels and the overall success-rate trends.Lower-level tasks serve as subtasks for higher-level tasks.Source: §F.3 · p. 33

G EXPERIMENTS ON OPEN-ENDED SINGLE-APP ENGLISH TASKS

An initial ten-task experiment finds that the MLLM evaluator aligns with human judgments on open-ended English tasks, while agent success rates differ from those on closed-ended tasks. The paper presents this as preliminary evidence and calls for a larger task collection.

G EXPERIMENTS ON OPEN-ENDED SINGLE-APP ENGLISH TASKS

H CASE STUDY

  • The case studies cover an AppAgent UI-structure misinterpretation, MobileAgent repetitive or unrelated behavior, and combined actions by M3A.Source: H CASE STUDY · p. 35

H.1 APPAGENT ON CONTACT_2 TASK

The trajectory examples show UI-structure misinterpretation and wasted actions, while M3A’s combined input-and-enter action used one fewer step than the human trajectory.

H.1 APPAGENT ON CONTACT_2 TASK

  • AppAgent misread a contact pop-up’s XML as one element and issued invalid tap(2), limiting interaction with its operable components.The error occurred at the final step of contact_2.Source: §H.1 · p. 35, Figure 11 · p. 35

H.2 MOBILEAGENT ON EXPEDIA_3 TASK

  • MobileAgent’s expedia_3 trajectory included repetitive actions, a ChatGPT detour, and a changing screen during a waiting period, leaving too few opportunities to finish.The screen changed despite no valid action between the second and third screenshots.Source: §H.2 · p. 36, Figure 12 · p. 36, Figure 13 · p. 37

H.3 M3A VS HUMAN ON GOOGLE_TASKS_0 TASK

  • M3A’s combined text-input and enter action completed the Google Tasks step in one fewer step than the human trajectory.The comparison is shown in Figure 14a and Figure 14b.Source: §H.3 · p. 38, Figure 14 · p. 38
Loading 2410.15164v3…