PaperlayerView exact revision on arXiv

Source-linked AI summary

Android in the Wild: A Large-Scale Dataset for Android Device Control

Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, Timothy Lillicrap

arXiv:2307.10088v2cs.LGcs.CLcs.HC

TL;DR

Device-control systems need to interpret natural-language commands and execute tasks through a device interface, but research lacks large, diverse datasets for this setting. The paper introduces AITW, a large dataset of screenshot-action demonstrations organized for robustness evaluation, and shows its coverage across tasks, applications, and interface conditions.

  • Problem

    Device-control systems should execute natural-language instructions without manual interaction, supporting users who cannot physically operate devices.

  • Method

    AITW provides large-scale natural-language task instructions paired with screenshot observations and precise coordinate-based gesture actions, with dataset splits for novel tasks, Android versions, and applications.

  • Results

    AITW contains 715k episodes spanning 30k unique task instructions, 159 Android apps, and 198+ websites, with moderate-length tasks and varied instruction lengths.

  • Takeaways & Limitations

    AITW supplies experimental setups for evaluating device-control models under novel language, Android versions, applications, and websites.

  • Takeaways & Limitations

    The dataset may not fully represent continuous user-interface evolution within individual apps or websites.

Abstract

from arXiv · show

There is a growing interest in device-control systems that can interpret human natural language instructions and execute them on a digital device by directly controlling its user interface. We present a dataset for device-control research, Android in the Wild (AITW), which is orders of magnitude larger than current datasets. The dataset contains human demonstrations of device interactions, including the screens and actions, and corresponding natural language instructions. It consists of 715k episodes spanning 30k unique instructions, four versions of Android (v10-13),and eight device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It contains multi-step tasks that require semantic understanding of language and visual context. This dataset poses a new challenge: actions available through the user interface must be inferred from their visual appearance. And, instead of simple UI element-based actions, the action space consists of precise gestures (e.g., horizontal scrolls to operate carousel widgets). We organize our dataset to encourage robustness analysis of device-control systems, i.e., how well a system performs in the presence of new task descriptions, new applications, or new platform versions. We develop two agents and report performance across the dataset. The dataset is available at https://github.com/google-research/google-research/tree/master/android_in_the_wild.

1 Introduction

Device-control systems aim to execute natural-language goals through direct UI manipulation, but existing datasets are limited in scale, diversity, platform coverage, and reliance on UI metadata. AITW addresses these gaps with a large, varied Android dataset and baseline agents designed for generalization analysis.

  • Motivation: Natural-language device control could assist users unable to operate devices manually because of physical or situational impairments.The motivation includes visual or motor disabilities as well as contexts such as driving or cooking.
  • Research gap: Direct UI manipulation requires systems to understand screens and map high-level commands into executable action plans.This avoids dependence on application-specific APIs, which are not generally available across applications and functions.
  • Research gap: Existing datasets have limited demonstrations and instruction diversity, are platform-specific, and often assume accessible tree-based UI metadata.Some also use low-level, step-by-step commands tied to specific visible UI elements rather than high-level user goals.
  • Dataset contribution: 715k episodes and 30k unique task instructions make AITW orders of magnitude larger than existing device-control datasets.Episodes contain natural-language goals, screenshots, and observation-action sequences, with gestures represented at arbitrary screen coordinates.
  • Dataset contribution: AITW collects diverse task instructions and execution paths from humans, LLM-generated prompts, and technical documentation.The collection includes both multi-step demonstrations and single-step trajectories produced through hindsight language relabeling.
  • Evaluation: AITW supports generalization analysis across previously seen and novel task descriptions, Android versions, applications, and websites.The authors implement a behavioural-cloning agent and a pre-trained-LLM agent as baselines.

2 Related work

Prior device-control datasets either ground low-level commands in UI elements or represent high-level goals with screenshots plus platform-specific UI trees. AITW extends the high-level-goal paradigm with substantially greater scale and pixel-based screen features.

  • Existing datasets: Existing grounding datasets pair screens and low-level commands with corresponding UI elements, sometimes using synthetic commands or multi-step command sequences.Examples include referring-expression grounding in RicoSCA and MiniWoB++ sequences of commands such as clicking and submitting.
  • Existing datasets: High-level-goal datasets typically provide screenshots together with Android View Hierarchies or web DOM trees.PixelHelp contains 187 high-level task goals, while UGIF extends similar queries to multiple languages.
  • Scale comparison: 4.7k task demonstrations and 276 unique instructions make MoTIF the largest prior dataset described, whereas AITW contains 715,142 episodes and 30,378 prompts.AITW is reported as two orders of magnitude larger than MoTIF.
  • Representation: Pixel-based screen features such as OCR and icon labels provide an alternative to relying exclusively on platform-specific UI metadata.AITW observations use screenshots and can be augmented with these inferred screen features.

3 Android in the Wild (AITW)

AITW combines naturalistic Android task demonstrations with precise gesture recording, hindsight relabeling, and pixel-derived screen features. Its five dataset groups cover multi-step and single-step tasks across diverse apps, websites, instructions, and execution paths.

  • Dataset composition: AITW comprises four multi-step groups—GOOGLEAPPS, INSTALL, WEBSHOPPING, and GENERAL—and one single-step group, SINGLE.The groups cover Google applications, app installation and login, e-commerce, miscellaneous tasks, and hindsight-annotated single-step actions.
  • Collection pipeline: The two-stage collection first records end-to-end emulator tasks, then applies hindsight language relabeling to identify shorter single-step sequences.Raters annotate subsequences of roughly two to five frames and label at least K >= 3 subsequences per video.
  • Action representation: AITW records TOUCH and LIFT events at 10Hz and represents gestures with start and end screen coordinates.Taps are approximately zero-distance gestures, while arbitrary drags support scrolling and carousel interaction.
  • Action representation: Each action is encoded by type, touch_point, optional lift_point, and optional typed_text, including gestures, typing, navigation buttons, and task status.The action vocabulary includes dual-point gesture, type, go_back, go_home, enter, task_complete, and task_impossible.
  • Screen representation: Pixel-based screen processing detects UI bounding boxes with OCR text or one of 96 IconNet icon classes.The resulting representation is noisy and less comprehensive than UI metadata, but does not require a platform UI tree.

4 Experimental setup

AITW evaluates device-control generalization through standard and out-of-distribution splits that hold out Android versions, instruction templates, or application domains. The setup uses template-based partitioning to test unseen language patterns and domain-based partitioning to test unseen apps and websites.

  • Standard: AITW uses episode-wise 80/10/10% training, validation, and test splits for each dataset under the standard setup.Scores are averaged across the four multi-step datasets and SINGLE.
  • Unseen Android version: The unseen Android version split trains and validates on Android versions 10–12, then tests entirely on Android version 13 devices.The held-out version contains unseen graphical components and execution flows.
  • Unseen subject and unseen verb: Subject and verb OOD splits mask instruction phrases into templates so training and testing use disjoint language patterns.The procedure targets generalization to unseen instructions and sometimes entirely new tasks.
  • Unseen subject and unseen verb: AITW contains 6,111 subject templates and 22,122 verb templates, randomly assigned to 80/10/10% train, validation, and test splits.Each episode is assigned according to its instruction template.
  • Unseen domain: The unseen-domain split assigns entire web domains or app names to 80/10/10% splits, excluding SINGLE and GOOGLEAPPS.WEBSHOPPING and GENERAL use web domains, while INSTALL uses app names for tasks involving the installed app.

5 Experiments

The experiments compare behavioral-cloning and LLM-based agents using offline action matching, with human evaluation validating the automated metric on a subset. BC performs best across all splits, while LLM performance is constrained by its element-based action space.

  • Agents: BC outputs an action type and dual-point gesture, supporting taps and scrolls at arbitrary screen locations rather than only specific UI elements.BC-history uses two prior screen-action steps, while BC-single omits that history.
  • Agents: LLM-0 and LLM-hist-5-CoT predict supported AITW actions from HTML-converted screen descriptions, using zero-shot or five-shot Chain-of-Thought prompting.The screen representation is derived from Android View Hierarchy, OCR, and detected icons.
  • Evaluation: Human validation is more accurate but costlier, whereas offline matching is cheaper and reproducible and can undercount functionally equivalent actions.For example, navigation-bar Back and an app-specific Back button may both complete the task.
  • Evaluation: Offline action matching compares predicted and ground-truth action types and gesture locations, then computes partial and complete episode scores.Partial score is correct actions divided by episode length; complete score requires partial match of 1.0.
  • Evaluation: The four agents are evaluated on five AITW splits, with BC tested on complete test sets and LLM evaluated on 288 randomly sampled episodes per split.Table 4 reports average partial matching scores.
  • Results: BC performs best across all splits and remains reasonably effective on OOD tasks, especially subject and verb template splits.The LLM OOD experiments use only the few examples included in their prompts, limiting use of the training distribution.
  • Results: 33% of standard-test episodes contain non-element tap actions infeasible for the LLM action space; across feasible actions, LLM-hist-5-CoT reaches 58% partial match.The result identifies an action-space mismatch rather than only a prompting difference.
  • Results: On GOOGLEAPPS, performance is highest on the standard split but lower on subject and verb OOD splits because prompts are distinct yet densely demonstrated.The split-specific difficulty reflects the dataset’s prompt composition.

6 Discussion

AITW has several scope and representativeness limitations, including nonrepresentative participants and content, non-native interaction patterns, mobile-only coverage, incomplete UI drift coverage, and privacy and misuse concerns.

  • The raters and Internet content are not representative of the global population, and all dataset prompts are in English.The authors note that prompts could potentially be translated and evaluated with multilingual models.
  • Raters use a mouse and keyboard rather than a native touch interface, which may produce different user patterns.
  • The dataset is derived from mobile-phone interactions and could be augmented with additional form factors such as tablets.
  • AITW’s unseen-domain split may not fully represent the continuous evolution of an individual app or website’s UI.The unseen Android-version split captures some UI drift through changes in Google apps across Android versions.
  • Raters were instructed not to enter personally identifiable information, and the dataset contains no interactions from real users.
  • The dataset could be misused to override anti-fraud mechanisms or manipulate deployed models through prompts and screen representations.

7 Future Work

Future work targets multimodal device-control models and evaluation methods that better accommodate alternative task-completion paths and user preferences.

  • A natural next model type is a multimodal foundation model that consumes raw pixels and outputs gestures at arbitrary points.
  • Foundation models may benefit from fine-tuning on AITW training sets.
  • Because tasks often have multiple valid solutions, future evaluations could avoid penalizing correct actions that differ from human demonstrations.
  • Constraining agents to achieve goals optimally, however defined, may increase user satisfaction with trained models.

8 Conclusions

The paper presents AITW to address the shortage of large, comprehensive datasets for natural-language mobile device control. AITW combines large-scale, diverse demonstrations with evaluation setups for novel tasks, language, platforms, and applications.

  • Natural-language mobile device control requires translating high-level instructions into execution plans that operate interfaces as humans do.
  • AITW contains 715k episodes across more than 350 Android applications and websites, with varied task instructions and execution paths.
  • AITW provides evaluation setups for novel tasks and language, Android versions, and applications and websites.

Appendix A Dataset collection

The appendix describes participant compensation, collection-quality procedures, prompt-template extraction, and example episodes for AITW dataset construction.

  • Participants were paid contractors receiving standard contracted wages compliant with living-wage laws in their employment countries.The authors do not provide further compensation details because of global privacy concerns.
  • Raters received written and video instructions, performed test demonstrations, and had a subset of multi-step trajectories manually inspected for quality and diversity.
  • Manual reviews of some hindsight-relabelled trajectories encouraged descriptive, unambiguous task labels and discouraged vague or oversimplified technical language.
  • The authors used prompts to extract subject and verb templates from task commands.
  • Figures 4, 5, and 6 show examples of AITW episodes.

B.1 Behavioral Cloning

The Behavioral Cloning agent uses a Transformer to combine task instructions, the current screen, and action-observation history. Its dual-point gesture output supports precise interactions with widgets requiring scrolling.

  • The BC agent is a Transformer conditioned on the task instruction, current screen, and stacked history of observations and actions.Natural-language instructions are represented with BERT embeddings, while screen elements and gesture history receive learned representations.
  • Dual-point gesture outputs represent start and end coordinates, supporting precise scrolling of carousels, calendars, and sliders.Taps are special cases where the start and end positions are approximately equal.
  • The agent is trained with cross-entropy loss using a Transformer with four layers, dropout rate 0.1, AdamW, learning rate 0.0001, and batch size 128.Training uses a 2x2 slice of a V2 TPU.
  • Hyperparameter search uses the validation set, after which the best model is evaluated on the test set for final results.
  • The BC-history agent is the best-performing agent whose performance is broken down across AITW generalization splits and dataset portions.Table 5 reports partial match scores for these breakdowns.

B.2 LLM

The LLM agent receives a textual screen representation, task instruction, available actions, and optionally prior actions. Its prompt examples use step-by-step reasoning to select actions such as clicks, navigation, typing, or directional scrolling.

  • The LLM agent is prompted to predict an AITW action from a textual description of the mobile screen.The screen representation is converted to HTML using OCR and detected icons.
  • Available actions include clicking an element, typing text, navigating home or back, and scrolling in four directions.
  • Prompt examples map screen context and instructions to actions such as clicks, home navigation, or downward scrolling.Examples include opening apps, clearing a search bar, and accessing the app drawer.
  • Each dataset datapoint is stored as a GZIP-compressed TFRecord with metadata fields including Android API level, activity, device type, episode ID, length, and goal information.
Loading 2307.10088v2…