PaperlayerView exact revision on arXiv

Source-linked AI summary

MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents

Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, Shoufa Chen

arXiv:2406.08184v1cs.AIcs.HC

TL;DR

Mobile-agent benchmarks are limited, and judging completion is difficult because GUI tasks can follow varied paths and reach unexpected states. MobileAgentBench offers real-device Android evaluation with built-in tasks and flexible success checks. It evaluates five agents across multiple metrics and reports baseline results for future development.

  • Problem

    Mobile-agent benchmarks are limited, and automatic task-success judgment is difficult because GUI navigation can lead agents into unknown states.

    Source: §2.2 · p. 3

  • Method

    MobileAgentBench provides 100 built-in tasks across ten open-source apps and checks task completion through final UI state, with Python-defined success criteria and Android Accessibility Services.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    The paper reports evaluations of five popular agents across multiple metrics, providing baseline results for mobile LLM agent development.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The reported agent evaluations provide baselines intended to advance future mobile LLM agent development.

    Source: §6 · p. 9

    Python task configurations remain difficult for people without technical backgrounds, and UIAutomator cannot dump view hierarchies with dynamic screen contents.

    Source: §5 · p. 8

Abstract

from arXiv · show

Large language model (LLM)-based mobile agents are increasingly popular due to their capability to interact directly with mobile phone Graphic User Interfaces (GUIs) and their potential to autonomously manage daily tasks. Despite their promising prospects in both academic and industrial sectors, little research has focused on benchmarking the performance of existing mobile agents, due to the inexhaustible states of apps and the vague definition of feasible action sequences. To address this challenge, we propose an efficient and user-friendly benchmark, MobileAgentBench, designed to alleviate the burden of extensive manual testing. We initially define 100 tasks across 10 open-source apps, categorized by multiple levels of difficulty. Subsequently, we evaluate several existing mobile agents, including AppAgent and MobileAgent, to thoroughly and systematically compare their performance. All materials are accessible on our project webpage: https://MobileAgentBench.github.io, contributing to the advancement of both academic and industrial fields.

1 Introduction

Mobile agents can interact with smartphone GUIs, but evaluating them is difficult because existing benchmarks have limitations in usability, success-path flexibility, and realism. MobileAgentBench introduces an accessible Android benchmark with built-in tasks and systematic agent comparisons.

1 Introduction

  • Mobile agents use LLMs to interpret user intent and operate apps through GUI actions such as clicking, typing, and swiping.The GUI input may be a view tree or a screenshot, and the approach requires no modifications to existing apps.Source: §1 · pp. 1–2
  • Existing mobile-agent benchmarks face usability and scalability burdens, may reject valid alternate completion paths, and sometimes rely on screenshots rather than real devices.These limitations complicate benchmark extension, task-success judgment, and handling of unexpected states.Source: §1 · p. 2
  • MobileAgentBench evaluates mobile LLM agents in the Android ecosystem and requires fewer than ten lines of additional code to integrate standard agents.It supports testing across Android versions and on actual devices.Source: §1 · p. 2
  • Its initial release provides 100 tasks across ten open-source apps, with task-success conditions configurable in a few lines of Python.The benchmark also introduces a task-terminating-state method intended to handle multiple possible success paths.Source: §1 · p. 2
  • The paper evaluates state-of-the-art mobile LLM agents and provides a systematic comparison and baseline data for the community.The benchmark framework is described as making evaluation fully autonomous and reliable.Source: §1 · p. 2

2 Related Work

Related work describes mobile LLM agents’ shared interaction pattern and techniques for improving their performance. It also reviews benchmark approaches to task judging, including their difficulty with alternate action paths and side effects.

2.1 Mobile LLM Agents

  • Mobile agents use a task prompt, current UI view, available actions, and history to select actions step by step, then send control signals to Android devices.Multimodal agents may also receive a screenshot.Source: §2.1 · p. 3
  • AndroidArena [24] compresses view hierarchies, MobileAgent [21] uses OCR and icon detectors, AppAgent [26] explores apps before RAG-based deployment, and CogAgent [11] and Octopus v2 [5] offer compact-model approaches.AppAgent [26] summarizes exploration into a document used during deployment.Source: §2.1 · p. 3

2.2 Benchmarks for Mobile LLM Agents

  • Benchmarking is limited, and dynamic GUI navigation makes automatic task-success judgment difficult because agents can enter unknown states.Manual verification is tedious and time-consuming.Source: §2.2 · p. 3
  • AITW [19] uses screenshot states and human-annotated actions, so it can fail on an unannotated action sequence even if the agent eventually completes the task.AndroidArena [24] instead compares action sequences using longest common subsequences, but its subsequence success rule can falsely accept side-effecting redundant actions.Source: §2.2 · pp. 3–4
  • LlamaTouch [28] examines final UI state, but UI-only checking can miss outcomes such as network requests, while coordinate-based click matching may be inaccurate.The paper notes that touchable areas can differ from visible view borders; Android Accessibility Service is used to capture app events faithfully.Source: §2.2 · p. 4

3 MobileAgentBench

MobileAgentBench automates mobile-agent evaluation on real devices or emulators, judging success from final UI state and app events. It provides 100 tasks across 10 apps, difficulty levels, and a lightweight integration API.

3.1 Method

  • The benchmark runs agents on real devices or emulators via ADB, retrieves app-state information, and automatically advances tasks without human supervision.The event listener forwards app events to the benchmark server for success assessment.Source: §3.1 · pp. 4–5
  • Success is judged in real time from final UI state rather than action sequences, with app events capturing transient actions such as saving a note.An Android Accessibility Service forwards events; task-specific Python criteria can combine view-hierarchy checks with event signals.Source: §3.1 · pp. 4–5

3.2 Task Description

  • The initial benchmark contains 100 daily-activity tasks across 10 SimpleMobileTools apps, classified by expert-checked minimum completion steps.Easy tasks take at most 2 steps, medium tasks 3–5, and hard tasks 6 or more.Source: §3.2 · p. 6

3.3 Usage

  • Integrating a standard agent requires fewer than 10 lines of additional code, using an orchestrator that collects action data and runs tasks in sequence.The agent entry function takes the task prompt, and success is judged on the fly.Source: §3.3 · p. 6

4 Experiments and Agent Evaluations

The experiments define six measures of agent performance, evaluate five mobile agents in Android environments, and compare their success and efficiency across agents, task difficulty, and backbone models.

4.1 Metrics

  • The evaluation defines six metrics covering task completion, step-wise efficiency, latency, token use, and false-negative and false-positive stopping errors.The token metric uses GPT-4V token conventions as a rough estimate of LLM cost.Source: §4.1 · p. 7

4.2 Environment Setup

  • Five agents were tested on a Pixel 3a emulator with Android 14, while AutoDroid used Android 9 because some dependencies did not support newer systems.CogAgent used 4-bit quantization and a vanilla screenshot-to-next-action setup without history; AppAgent’s self-exploration was enabled.Source: §4.2 · p. 7

4.3 Results

  • AppAgent has the highest success rate, while CogAgent has the lowest; AutoDroid’s success rate is similar to MobileAgent’s but its step-wise efficiency is significantly lower.AutoDroid and CogAgent have very low latency and high false-negative rates; AppAgent has a high false-positive rate despite its success-rate lead.Source: Table 2 · p. 7, §4.3 · pp. 7–8
  • Most agents achieved higher success rates on easier tasks, while AppAgent performed better on medium tasks under a step limit of twice the minimum steps.The authors attribute this pattern to the limited opportunity to correct earlier actions on easy tasks compared with medium and hard tasks.Source: Figure 4 · p. 8, §4.3 · p. 8
  • AutoDroid’s text-based GPT-3.5 agent outperformed some agents using GPT-4V, while the authors note screenshots may help with image recognition or unavailable view hierarchies.The authors interpret the comparison as showing that textual view hierarchies contain important information for GUI navigation.Source: §4.3 · p. 8

5 Limitations and Future Work

The authors identify two limitations: task configuration still requires technical skills, and UIAutomator cannot dump view hierarchies during persistent animations or videos.

5 Limitations and Future Work

  • Python snippets make task-success configuration accessible to AI/ML researchers but remain difficult for people without a technical background.The authors plan to explore automatic task configuration without code.Source: §5 · p. 8
  • UIAutomator cannot dump the view hierarchy when an app’s persistent animations or videos keep its main thread busy.The authors propose replacing UIAutomator with another framework for retrieving view information.Source: §5 · p. 8

6 Conclusion

MobileAgentBench evaluates mobile LLM agents on Android with built-in tasks and final-app-state checks. The authors report results for five agents across multiple metrics as baseline data for future development.

6 Conclusion

  • MobileAgentBench provides 100 built-in tasks for automatically evaluating agents on real Android devices.The benchmark is for mobile LLM agents on the Android platform.Source: §6 · p. 9
  • Researchers can extend the benchmark with customized tasks using basic Python coding skills.Source: §6 · p. 9
  • The benchmark checks task completion through Android Accessibility Services and the final app state, and reports five agents’ results across multiple metrics.The results are presented as baseline data to support future mobile LLM agent development.Source: §6 · p. 9

Related papers

Loading 2406.08184v1…