PaperlayerView exact revision on arXiv

Source-linked AI summary

AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents

Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, Yuxiao Dong

arXiv:2410.24024v2cs.AI

TL;DR

Existing Android-agent work lacked systematic research on training and evaluating both open- and closed-source models. ANDROIDLAB provides a shared operational environment and benchmark for text-only and multimodal agents, and fine-tuning with its dataset raises average success rates for both model types.

  • Problem

    Prior Android-agent benchmarks and studies lacked systematic exploration of training and evaluation, with recent benchmarks primarily testing prompt-based improvements.

    Source: §1 · p. 1

  • Method

    ANDROIDLAB combines a standard Android operation environment and reproducible benchmark, aligning actions across XML and screenshot-based SoM modes for LLMs and LMMs.

    Source: §1 · pp. 1–2

  • Results

    Fine-tuning raised average success rates from 4.59% to 21.50% for LLMs and from 1.93% to 13.28% for LMMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Fine-tuning and further optimization could narrow the performance gap between open-source models and closed-source solutions.

    Source: §6 · p. 10

Abstract

from arXiv · show

Autonomous agents have become increasingly important for interacting with the real world. Android agents, in particular, have been recently a frequently-mentioned interaction method. However, existing studies for training and evaluating Android agents lack systematic research on both open-source and closed-source models. In this work, we propose AndroidLab as a systematic Android agent framework. It includes an operation environment with different modalities, action space, and a reproducible benchmark. It supports both large language models (LLMs) and multimodal models (LMMs) in the same action space. AndroidLab benchmark includes predefined Android virtual devices and 138 tasks across nine apps built on these devices. By using the AndroidLab environment, we develop an Android Instruction dataset and train six open-source LLMs and LMMs, lifting the average success rates from 4.59% to 21.50% for LLMs and from 1.93% to 13.28% for LMMs. AndroidLab is open-sourced and publicly available at https://github.com/THUDM/Android-Lab.

1 Introduction

ANDROIDLAB addresses the lack of systematic Android-agent evaluation across open- and closed-source models and modalities with a shared operational environment, benchmark, and training data. Fine-tuning open-source models on Android Instruct improves their average success rates across both text-only and multimodal models.

1 Introduction

  • XML mode for text-only models and SoM mode for multimodal models use aligned actions, enabling comparisons in a shared action space.ReAct and SeeAct are also implemented in both modes.Source: §1 · p. 1, Figure 1 · p. 2
  • The reproducible benchmark contains 138 tasks across nine apps on Android virtual devices with preloaded histories and offline data, avoiding external network or time dependencies.Tasks are evaluated through required page-state sub-goals, with UI-tree matching used to verify traversal and assess completion and progress.Source: §1 · p. 2
  • ANDROIDLAB evaluates task completion by decomposing tasks into required sub-goals and considering a task complete only when all are correctly addressed.The benchmark also introduces reversed redundancy and reasonable-operation metrics to assess action efficiency.Source: §1 · p. 2
  • Across 17 evaluated models, GPT models exceeded 30% success while the best open-source models reached only around 5%; more complex reasoning frameworks brought marginal gains despite longer inference times.The authors therefore propose fine-tuning small open-source models as a possible way to narrow the performance gap and improve accessibility.Source: §1 · p. 2
  • Android Instruct contains 10.5k traces and 94.3k operation records, of which 6,208 steps from benchmark apps were used to fine-tune six text-only and multimodal models.The dataset includes tasks, phone-screen states, XML information, and operations.Source: §1 · pp. 2–3
  • Fine-tuning raised average success rates from 4.59% to 21.50% for LLMs and from 1.93% to 13.28% for LMMs.The training data includes over 726 traces and more than 6,208 fully aligned XML and SoM steps.Source: §1 · p. 2, Figure 2 · p. 3

2 Retated Work

Prior agent benchmarks span static and interactive settings, but mobile benchmarks vary in task diversity and support for training-data construction. Related mobile-agent work uses modular systems and multimodal models, while complex planning can constrain inference speed and practical deployability.

2 Retated Work

  • Desktop benchmarks include static single-step or sequence evaluation without virtual environments, alongside interactive browser and Unix-like-system environments.The passage distinguishes static evaluation from dynamic benchmarks that provide interactive environments.Source: §2 · p. 3
  • Android benchmarks progressed from static systems to dynamic evaluation and cross-app tasks, but AndroidWorld’s 116 tasks across 20 apps do not support instruction-tuning data construction.The cited work also describes limited task diversity in Android Arena and standardized virtual devices in B-MOCA.Source: §2 · p. 3
  • Mobile-agent approaches range from multiple execution modules and image-action mapping to LLM-based action prediction and zero-shot GPT-4V agents.The cited zero-shot agents face planning complexity that limits inference speed and practical deployability due to security restrictions.Source: §2 · pp. 3–4
  • ANDROIDLAB task examples and app/subcategory distributions accompany an evaluation rule requiring every sub-goal to be correctly addressed for task completion.Source: Figure 3 · p. 4

3 ANDROIDLAB

ANDROIDLAB provides a shared Android operation environment for text-only and multimodal agents, alongside a reproducible benchmark of tasks evaluated through device states and task completion. Its task definitions and evaluation designs support comparisons across operation and query tasks under controlled conditions.

3.1 The Operation Environment

  • XML mode lets text-only models select compressed-XML elements, while SoM mode lets multimodal models select numbered screen elements aligned with the same objects and action space.The shared action space includes phone operations and shortcut keys, with a Finish action for returning results or answers; ReAct and SeeAct add reasoning and action-grounding frameworks.Source: §3.1 · pp. 4–5

3.2 The Reproducible Benchmark

  • The benchmark includes 138 tasks from nine apps, comprising operation tasks and query tasks, and uses task completion as its primary metric.It also tracks valid actions and operation-sequence redundancy as auxiliary metrics.Source: §3.2 · p. 5

3.2.1 Task Formulation

  • Each task is formalized as Task(E, I, F, M), where E is the pre-packaged device environment, I the natural-language instruction, F the testing framework, and M the backbone model.Operation tasks produce a sequence of device states; query tasks produce that state sequence plus an answer based on exploration.Source: §3.2.1 · p. 5

3.2.2 Reproducible Designs

  • The benchmark defines 138 tasks: 93 operation tasks and 45 query tasks.Evaluation fixes device time and geolocation, keeps apps offline with preloaded records, and establishes predefined answers for query tasks.Source: §3.2.1 · p. 5, §3.2.2 · p. 5

3.2.3 Metrics

  • Operation-task success requires completing every sub-goal, checked against specific UI-tree elements; query answers are verified against standard answers by advanced LLMs.Sub-goal success is reported only for operation tasks; reversed redundancy is calculated on completed tasks, while the reasonable-operation ratio counts actions followed by screen changes. Figure 4 illustrates selected steps and stresses recording each sub-goal's completion status.Source: §3.2.3 · pp. 5–6, Figure 4 · p. 6

4 Android Instruction Data

Android Instruction data is built to support training open-source Android agents, using task generation, self-exploration, and manual annotation to create operation traces. The paper describes the annotation workflow, training subset, and fine-tuning setup.

4 Android Instruction Data

  • The data effort targets open-source, deployable Android agents, with training data intended to support mobile operations across diverse environments.The authors identify generating training data for mobile operations as a central challenge.Source: §4 · p. 6
  • The online annotation tool records phone interactions and captures screenshots and page XML before each action; ANDROIDLAB connects to devices through ADB for data generation.The tool supports annotators working on real devices rather than only emulators.Source: §4 · pp. 6–7

4.1 Data Construction

  • Task generation seeds instructions from academic datasets and manual writing, then uses language models to expand them into reviewed, executable tasks.The task-generation method is intended to create tasks controllably for specific apps.Source: §4 · p. 6, §4.1 · p. 7
  • LLMs and LMMs explore tasks automatically, and a reward model replaced manual trace selection after 500 traces had been gathered.The models mark completion by outputting “finish.”Source: §4.1 · p. 7
  • Manual annotation checks task feasibility, familiarizes annotators with the app, documents task execution, and has a second annotator cross-verify each trace.Source: §4.1 · p. 7
  • The process produced 10.5k traces and 94.3k steps; training uses 726 traces and 6,208 steps from apps in the ANDROIDLAB benchmark.Figure 5 presents statistics for the Android Instruction dataset across benchmark apps.Source: §4.1 · p. 7, Figure 5 · p. 7

4.2 Annotation Tool

  • ADB can time out while waiting for dynamic UI components to become idle, so the tool uses Android Accessibility Service to retrieve page XML at annotator-chosen times.Clicks and swipes are captured from screen events, while text input is entered in one ADB keyboard operation.Source: §4.2 · pp. 7–8

4.3 Training

  • Six text-only and multimodal model backbones were fine-tuned for five epochs with batch size 32, maximum sequence length 4096, and learning rate 1e-5.Base models were selected for fine-tuning except Qwen2-VL-7B-Instruct, for which no open-source base model was available.Source: §4.3 · p. 8

5 Experiments

The experiments compare open- and closed-source models across XML and SoM modes, then examine instruction tuning, agent frameworks, and device sizes. Fine-tuning substantially improves open-source models, while framework and screen-size effects vary.

5.1 Experiment Setup

  • Formal tests launch each task in its specified app, cap execution at 25 steps, allow a 3-second VM response interval, and use greedy generation.The baselines include open- and closed-source LLMs and LMMs, with GPT-4o and Gemini-1.5-Pro also tested using ReAct and SeeAct.Source: §5.1 · p. 8

5.2 Main Results

  • XML mode: GPT-4-1106-Preview leads with SR 31.16% and Sub-SR 38.21%; SoM: GPT-4o leads with SR 31.16% and Sub-SR 35.02%.Table 1 reports the main results across both modes.Source: §5.2 · p. 8, Table 1 · p. 9

5.3 Additional Findings

  • Instruction tuning lifts average success rates from 4.59% to 21.50% for LLMs and from 1.93% to 13.28% for LMMs.LLaMA3.1-8B SR rises from 2.17 to 23.91, while Qwen2-7B SR rises from 4.35 to 19.57.Source: §5.3 · p. 9, §5.2 · p. 9
  • SoM performance is optimal on commonly sized phones such as Pixel 7 Pro and Pixel 8 Pro, and drops on smaller Pixel 3a and larger Pixel Fold screens.The reported challenges are increased scrolling on smaller screens and landscape orientation on the Pixel Fold.Source: §5.3 · p. 10, Figure 6 · p. 10
  • ReAct significantly improves performance only in XML mode, while SeeAct is inconsistent; these frameworks also increase token usage.XML+ReAct averages 67.89 tokens, compared with 4.96 tokens for models after instruction tuning.Source: §5.3 · p. 10, Table 2 · p. 10, Table 3 · p. 10

6 Conclusion

The conclusion presents AndroidLab as a reproducible Android-agent environment and benchmark, alongside an instruction dataset for fine-tuning. Experiments report improvements in fine-tuned open-source models, while leading closed-source models retain advantages in success rates and efficiency.

6 Conclusion

  • AndroidLab integrates XML and SoM modes with a consistent action space and a reproducible benchmark of 138 tasks across nine apps.Its evaluation assesses task completion and progress.Source: §6 · p. 10
  • The Android Instruct dataset contains 10.5k traces and 94.3k steps, and fine-tuning with it improves open-source model performance.Fine-tuning raised success rates and operational efficiency, though leading closed-source models still lead in success rates and efficiency.Source: §6 · p. 10

A Details of Tasks

The appendix describes the evaluation apps and four interaction modes. XML and SoM use different observations and action targeting, while ReAct and SeeAct modify prompting.

A Details of Tasks

  • The evaluation uses nine apps selected to work independently without internet access or user-account login, supporting consistent replication.The apps cover finance, calendar, reading, clock, contacts, maps, music, settings, and video conferencing.Source: A Details of Tasks · p. 13

B.1 XML mode

  • XML mode provides task instructions, interaction history, and compressed XML; models return function-call actions on XML coordinates.The associated table lists Android evaluation apps, example tasks, sub-goals, and task counts.Source: §B.1 · p. 13, Table 4 · p. 14

B.2 SoM mode

  • SoM mode provides a marked screenshot with task instructions and interaction history, and actions target elements by marked index.Unlike XML mode, it uses multimodal screenshot input.Source: §B.2 · p. 13

B.3 ReAct mode

  • ReAct prompting adds step-by-step thinking before the model outputs its action, using the XML or SoM input setup.The paper follows the ReAct prompting approach.Source: §B.3 · p. 13

B.4 SeeAct mode

  • SeeAct adapts browser prompts for Android and uses two rounds: first describing an action and element, then producing the function-call format.The prompts use screenshots for multimodal testing and compressed XML for text-only testing.Source: §B.4 · p. 13

C Details of Android Instruction Dataset

The appendix details safeguards and procedures for collecting Android Instruct annotations. Annotators carry out phone tasks, record operation traces, and can review or exclude sensitive material.

C.1 Details of Human Annotation

  • Annotators were informed of training and release plans, could opt out, stored data locally, and could skip tasks; usernames and chat logs were masked, and sensitive data was not released.Annotators signed contracts and received compensation.Source: §C.1 · p. 15

C.2 Instructions Given To Annotators

  • Each labeling task gives a target instruction, and annotators complete it on their phones; a trace consists of multiple recorded single-step operations.The annotation instructions include tasks such as navigating to a destination using a map app.Source: §C.2 · pp. 15, 19
  • Collection records operation details, before-and-after screenshots, and current-page XML; annotators could skip screenshots or tasks and review whether to keep data.The instructions say information is not transmitted in real time during collection.Source: §C.2 · pp. 15, 19
  • The phone setup requires ADB connectivity and USB debugging; annotators verify the device connection and wait for each operation’s completion message.If the annotation state cannot be recovered after an invalid operation, the instructions require restarting the task.Source: §C.2 · pp. 19–20
  • Annotators install ADB Keyboard, enable it as the input method, and check that it is active before entering text.The setup instructions also recommend turning off Android animations to improve XML retrieval via ADB.Source: §C.2 · p. 19
  • The annotation plugin supports clicking or swiping, text entry, and Home, Back, or Enter key presses; annotators must wait for completion before the next action.The instructions warn that proceeding without the required wait can invalidate data.Source: §C.2 · p. 20
  • Annotators can delete saved data when errors or sensitive information occur, remove incorrect steps that did not affect phone state, and identify sensitive content for removal.Submissions must contain one instance of each non-skipped task.Source: §C.2 · p. 20

D.1 Detail results across different APPs

Across apps, GPT-4o and GPT-4-1106-Preview lead baseline task completion, while ReAct and SeeAct improve completed-task counts for several models, especially on complex tasks.

D.1 Detail results across different APPs

  • GPT-4o and GPT-4-1106-Preview complete 78 and 79 tasks, respectively, and outperform other models across XML and SoM modes.Table 5 reports task completions across apps; GPT-4-1106-Preview leads XML mode with 43 tasks, while GPT-4o excels in SoM mode.Source: §D.1 · pp. 20–21, Table 5 · p. 21
  • GPT-4o rises from 35 to 46 completed tasks in XML+ReAct mode and from 43 to 44 in SoM+ReAct mode.Gemini-1.5-Pro increases from 26 to 43 tasks; Table 6 summarizes improvements after applying ReAct and SeeAct.Source: §D.1 · p. 21, Table 6 · p. 21
  • ReAct and SeeAct improvements are especially notable for high-complexity, multi-step tasks and apps such as Bluecoins.GPT-4o leads across the frameworks.Source: §D.1 · p. 21

D.2 Detail results across different multi-modal training mode

The multimodal training-mode comparison holds the training data and base model settings constant, while describing the distinction between BBOX and SoM supervision and a limitation of AITW data.

D.2 Detail results across different multi-modal training mode

  • Table 7 compares multimodal training modes using the same training data and base model, with SoM adding a set-of-mask index.The table identifies CogVLM2 as the base model.Source: §D.2 · p. 21, Table 7 · p. 22
  • Table 8 reports significantly improved model performance in XML+ReAct mode.The caption describes the table as assessing the impact of ReAct and SeeAct.Source: Table 8 · p. 22
  • AITW provides click points rather than bounding boxes, and its points cannot be reconstructed into click boxes from XML.The passage says this makes AITW and similar datasets theoretically more challenging to learn from.Source: §D.2 · pp. 21–22

D.3 Detail results of SeeAct and ReAct methods

Figure 8 presents detailed results on the impact of SeeAct and ReAct, covering four metrics.

D.3 Detail results of SeeAct and ReAct methods

  • Figure 8 reports the impact of SeeAct and ReAct on model performance across all four metrics.The passage identifies the figure as a detailed results view but does not state the metrics or their outcomes.Source: §D.3 · p. 22
Loading 2410.24024v2…