Source-linked AI summary
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, Oriana Riva
TL;DR
Autonomous-agent research needs realistic, reproducible benchmarks because existing evaluations can miss valid execution paths and mobile-specific challenges. AndroidWorld addresses this with dynamic, parameterized Android tasks and durable ground-truth rewards, while its best agent achieves a 30.6% success rate and performance changes substantially across task variations.
Problem
Realistic evaluation remains a challenge because agents may follow multiple valid paths, encounter nondeterministic environments, and learn from mistakes during execution.
Method
AndroidWorld benchmarks agents on dynamically parameterized tasks across 20 Android apps, using Android system state and dedicated task logic to provide ground-truth rewards.
Results
30.6% success rate was achieved by M3A on AndroidWorld, while experiments found significant performance variation under changes in intent parameters.
Takeaways & Limitations
AndroidWorld supports reproducible evaluation across diverse mobile tasks and enables robustness analysis under varied real-world conditions.
Takeaways & Limitations
AndroidWorld currently supports open-source apps and built-in system apps rather than trending applications, whose shortcuts and UI affordances may differ.
Abstract
from arXiv · showhide
Autonomous agents that execute human tasks by controlling computers can enhance human productivity and application accessibility. However, progress in this field will be driven by realistic and reproducible benchmarks. We present AndroidWorld, a fully functional Android environment that provides reward signals for 116 programmatic tasks across 20 real-world Android apps. Unlike existing interactive environments, which provide a static test set, AndroidWorld dynamically constructs tasks that are parameterized and expressed in natural language in unlimited ways, thus enabling testing on a much larger and more realistic suite of tasks. To ensure reproducibility, each task includes dedicated initialization, success-checking, and tear-down logic, which modifies and inspects the device's system state. We experiment with baseline agents to test AndroidWorld and provide initial results on the benchmark. Our best agent can complete 30.6% of AndroidWorld's tasks, leaving ample room for future work. Furthermore, we adapt a popular desktop web agent to work on Android, which we find to be less effective on mobile, suggesting future research is needed to achieve universal, cross-platform agents. Finally, we also conduct a robustness analysis, showing that task variations can significantly affect agent performance, demonstrating that without such testing, agent performance metrics may not fully reflect practical challenges. AndroidWorld and the experiments in this paper are available at github.com/google-research/android_world.
1 INTRODUCTION
ANDROIDWORLD addresses the lack of realistic, reproducible mobile benchmarks by providing durable ground-truth rewards across diverse, dynamically generated Android tasks. The paper establishes baseline agent performance and tests robustness to task variations.
- Realistic evaluation is difficult because agents can take multiple valid paths, encounter nondeterminism, and recover from mistakes during online task execution.
- ANDROIDWORLD provides ground-truth rewards for 116 programmatic tasks spanning 20 Android apps, with dynamically generated parameters producing millions of unique goals and conditions.It uses Android OS state-management mechanisms to keep reward checks durable across varying task parameters.
- ANDROIDWORLD extends its Android task suite with web tasks by integrating the MiniWoB++ benchmark.
- 30.6% success rate was achieved by M3A on ANDROIDWORLD, exceeding an Android-adapted web agent but remaining below the 80.0% human success rate.The study also reports that multimodal perception generally does not outperform text-only input.
- Performance varied significantly with changes in intent parameters, motivating evaluation across variable task conditions.
2 RELATED WORK
Prior benchmarks cover web, desktop, and mobile agent evaluation, but mobile environments remain limited in interaction diversity and application coverage. ANDROIDWORLD broadens this setting through dynamic task and start-state construction with non-invasive rewards.
- Existing evaluation environments include synthetic websites, simulated domains, enterprise software, and desktop operating-system interfaces, but many provide limited task variability or coverage.
- ANDROIDWORLD dynamically constructs start states and varies task parameters in unlimited ways, enabling evaluation under varying real-world conditions.
- Human evaluation and multimodal-model approaches address settings without automatic rewards, but multimodal evaluation still requires further research to match manually coded reward accuracy.
- AndroidEnv requires source-code modifications and task-specific logging, whereas ANDROIDWORLD uses a non-invasive reward system.
- Mobile datasets such as AitW, AndroidControl, PixelHelp, AndroidArena, LlamaTouch, UGIF, and MoTIF primarily provide demonstrations across Android apps and mobile websites.
3 ANDROIDWORLD
ANDROIDWORLD is a reproducible Android benchmark that exposes agents to realistic mobile interactions, dynamically parameterized tasks, and programmatic reward signals across diverse applications.
- Environment: ANDROIDWORLD provides agents with screenshots, accessibility-tree observations, and human-like actions including gestures, typing, and navigation buttons.
- Task suite: The benchmark contains 116 practical tasks across 20 diverse applications, including open-source and built-in Android apps.Tasks cover activities such as note-taking, scheduling, messaging, and system utilities.
- Reproducibility: ANDROIDWORLD controls the emulator, timestamp, application states, task setup, reward logic, and teardown procedures for reproducible execution.Its primary configuration uses a Pixel 6 emulator running Android 13 with a fixed task-start timestamp.
- Parameterized tasks: Randomly seeded task parameters generate practically infinite initial conditions and success criteria while preserving valid evaluation logic across instantiations.This dynamic construction supports larger-scale robustness, adaptability, and learning analyses than static task sets.
- Reward signals: ANDROIDWORLD derives rewards mainly from Android system state through adb, using UI validation when direct state inspection is impractical.System-state validation supports accurate, durable, reusable evaluation and composite tasks.
- Benchmark extension: MobileMiniWoB++ extends the framework with browser-style tasks while retaining AndroidWorld’s task interface and observation and action conventions.It replaces DOM access with Android accessibility-tree and screenshot observations and renders some web controls as native Android widgets.
4 ANDROIDWORLD AS A COMPUTER-CONTROL BENCHMARK
The benchmark evaluates Android-control agents across its task suites, showing useful mobile interaction abilities but substantial gaps from human performance and persistent mobile-specific weaknesses.
- Evaluation: The evaluation compares M3A variants and an adapted SeeAct baseline across ANDROIDWORLD and MobileMiniWoB++ using multiple foundation models.Experiments use fixed seed and temperature settings and impose task-specific step limits.
- Agents: M3A is a zero-shot Android agent that combines ReAct-style and Reflexion-style prompting to interpret instructions, act, and update decisions from outcomes.
- Prompting: 19.8% versus 30.6% success shows that M3A-SIMPLE performs substantially worse than full M3A on ANDROIDWORLD with GPT-4.On MobileMiniWoB++, M3A-SIMPLE reaches comparable performance of 67.7%, indicating differing sensitivity to prompting complexity across suites.
- Failure modes: Agents struggle with visual cues, mobile affordances, precise grounding, error recovery, and memory-intensive interactions.These difficulties include manipulating text or sliders, confirming system state, transcribing across apps, and adapting after reasoning mistakes.
- Baseline comparison: SeeAct performs less effectively than M3A on ANDROIDWORLD and similarly on MobileMiniWoB++, reflecting difficulties adapting web-oriented action selection to mobile interactions.Its weaknesses include long-presses, swipes, screen-element selection, and remembering outcomes rather than only cached actions.
- Efficiency: 3.9 minutes versus 2.5 minutes is the average completion time for M3A and its text-only version, respectively, making both slower than humans.
5 ROBUSTNESS ANALYSIS
ANDROIDWORLD’s robustness analysis shows that agent success depends substantially on task parameters and varies across seeds, while performance remains broadly capable across tested OS versions.
- Seed variation: 27.6%, 26.3%, and 33.2% success across three seeds produce a 29.0% mean for M3A with GPT-4 Turbo and accessibility-tree input.
- Robustness conditions: Different task parameters produce significantly more performance variation than repeated trials with identical parameters.The analysis uses representative tasks spanning interaction patterns and complexity levels, with 20 trials per task under the strongest configuration.
- Observed variability: Fixed-seed trials still fail on some add and edit tasks and rarely solve delete tasks, while model non-determinism causes additional variation.Differences are statistically significant for add expense and edit note tasks under different seeds.
- OS variation: 28.4% success on a Pixel 5 running Android 12 shows performance variation similar to random-seed changes while retaining agent capabilities across device layouts.The primary setup uses a Pixel 6 running Android 13.
- Implications: The results support evaluating agents across varied conditions and representing performance by the mean across seeds.The authors also identify non-zero rewards under some seeds as a possible opening for RL-like future improvements.
6 CONCLUSION
ANDROIDWORLD is a realistic, robust Android environment for evaluating autonomous agents across diverse tasks and apps. Its dynamically generated tasks provide reproducible evaluation and opportunities for online-learning research.
- ANDROIDWORLD provides a reproducible suite of 116 dynamically generated tasks across 20 Android apps, creating millions of unique goals.
- The benchmark establishes performance results with M3A while supporting development and evaluation across a wide range of Android tasks and apps.
- ANDROIDWORLD’s dynamic task generation opens research opportunities for online learning in computer-use agents.
APPENDIX A LIMITATIONS
ANDROIDWORLD’s current scope centers on open-source Android apps and built-in system apps, while autonomous agents also raise risks involving malicious use and broader societal effects.
- ANDROIDWORLD currently supports open-source Android apps with over 1 million downloads and built-in system apps.
- The selected open-source apps may present harder interaction challenges because their UIs are less optimized and offer fewer shortcuts and affordances.
- Malicious actors could engineer agents to bypass CAPTCHAs, spam users, or alter prompts and screen outputs for harmful objectives.
- Automation agents may transform societal norms, disrupt employment, and modify human behavior, with efficiency gains potentially exploitable by malevolent forces.
- The Android screen state includes pixels, an accessibility tree, and processed UI elements, while asynchronous actions require state capture after possible delays.
C.2 ACTION SPACE
ANDROIDWORLD combines direct UI controls, task-management actions, and optional high-level device APIs, while its task suite uses parameterized initialization and validation across Android and mobile web settings.
- C.2 ACTION SPACE: The action space includes clicking, scrolling, text input, navigation, app launching, long presses, and keyboard-enter actions.
- C.2 ACTION SPACE: Task-management actions report status, provide answers, wait through UI transitions, or handle internal errors.
- C.2 ACTION SPACE: High-level APIs supplement UI control with direct actions such as sending SMS, opening web pages, and managing contacts for hybrid interaction research.
- C.3 MOBILEMINIWOB++: MobileMiniWoB++ renders HTML5 input elements with native Android widgets and excludes tasks that are unsolvable or malfunctioning on the emulator.
- D.1 APP SELECTION: App selection considers use case, popularity, consistency, and reproducibility, using locally stored apps sourced from F-Droid with controlled versions.
- D.2 TASKS: Side-effect tasks use Python classes with parameter generation, initialization, evaluation, and teardown, while information-retrieval tasks use structured protobuf definitions.
- D.3 INFORMATION RETRIEVAL TASKS: LLM-generated task definitions require human review because they can contain missing or hallucinated fields, incompatible parameters, insufficient parameter use, or vague prompts.
D.4 HUMANS FOR TASK ANALYSIS
Human participants supported task analysis and solvability checks, while the benchmark and agent use screen pixels, accessibility-derived UI elements, screenshots, and structured task definitions.
- D.4 HUMANS FOR TASK ANALYSIS: Six programming-proficient volunteers analyzed task difficulty, duration, and category and identified over 30 bugs during annotation.
- D.4 HUMANS FOR TASK ANALYSIS: Two software engineers completed tasks once each on an Android emulator to measure human performance, with errors mainly involving misinterpretations or minor mistakes.
- D.4 HUMANS FOR TASK ANALYSIS: Participants were informed about data use and did not need to enter personal information in the tested tasks.
- D.4 HUMANS FOR TASK ANALYSIS: ANDROIDWORLD tasks vary in difficulty, duration, and category, and task templates expose attributes that can be parameterized in unlimited ways.
- E.1 M3A OBSERVATIONS: ANDROIDWORLD observations include raw screen pixels, SoM-annotated screenshots, and a list of UI elements on screen.
E.3 ERROR ANALYSIS
The analysis categorizes agent failures into perceptual, reasoning, missing-knowledge, and grounding errors, illustrated through Android task examples. These errors arise from misreading screen state, misunderstanding task requirements, lacking UI knowledge, or interacting incorrectly with positioned elements.
- Perceptual errors: Perceptual errors occur when the model fails to recognize crucial screen elements, such as an unchecked “All-day” checkbox.The example concerns creating a recurring event in Simple Calendar Pro.
- Reasoning errors: Reasoning errors occur when the model misinterprets task requirements or current state, leading to incorrect actions.In Markor, the model mistakenly assumes the note name is already entered and types the note content into the “Name” field.
- Missing knowledge errors: Missing-knowledge errors occur when the model lacks necessary understanding of an application’s UI, such as how to delete all files at once.The agent searches for a bulk-delete option instead of deleting files individually.
- Grounding errors: Grounding errors occur when the model interacts incorrectly with UI elements based on their spatial or contextual positioning.In Markor, clicking the text field places the cursor after existing content, so new text is appended rather than prepended.
E.4 AGENT ROBUSTNESS EXPERIMENTS
The robustness experiment evaluates the agent on five AndroidWorld tasks spanning note editing, expense entry, calendar deletion, file deletion, and activity counting.
- Evaluated tasks: The evaluation runs the agent on five tasks covering editing, expenses, calendar events, filesystem operations, and sports activity counting.The tasks are MarkorEditNote, ExpenseAddSingle, SimpleCalendarDeleteEventsOnRelativeDay, FilesDeleteFile, and SportsTrackerActivitiesCountForWeek.
- Evaluated tasks: The experiment reports task nicknames used in the figures and directs readers to Table 5 for additional task details.The listed nicknames are EditNote, AddExpense, DeleteEvent, DeleteFile, and CountActivities.
E.5 AGENT STRUGGLES DUE TO TASK PARAMETERIZATION
Task parameterization substantially changes agent difficulty: different seeds expose different UI requirements and task variants. The reported failures often arise when success requires discovering interactions such as horizontal scrolling or multi-step text replacement.
- ExpenseAddSingle: Seed-dependent expense categories determine whether the required option is visible or requires horizontal scrolling.The agent completes visible categories such as “Housing” and “Social” but consistently fails on categories such as “Food” and “Other” that require scrolling.
- MarkorEditNote: MarkorEditNote difficulty varies across header, footer, and replacement variants selected by the seed.The replacement variant requires long-pressing, selecting text, deleting it, and entering new text, whereas the header variant is simpler.
- Agent interaction: The Android agent prompt supports accessibility-tree elements, screenshots, scrolling, and one action at a time.The prompt asks the agent to inspect the current screen and action history before selecting the next target and operation.
- Agent interaction: The prompt represents screen elements as ordered multiple-choice options and requires standardized element, action, and value outputs.Supported actions include clicking, text input, long press, navigation, swiping, opening apps, waiting, and termination.
APPENDIX F ANDROIDWORLD TASK LIST
The AndroidWorld task list spans system utilities, browser interactions, expenses, notes, media, maps, recipes, and music. Tasks include both completion and information-retrieval objectives, with step limits calibrated from human performance.
- Task design: Task limits are based on human performance, allowing agents approximately twice the human step count while preventing infinite loops.The maximum number of steps per task is denoted S.
- Task categories: Information-retrieval objectives ask agents to answer questions about notes, recipes, folders, and application state.Examples include checking whether a Joplin note is a todo, counting meeting attendees, retrieving recipe quantities, and counting folder to-dos.
- Task categories: The suite also includes browser, camera, clock, contacts, maps, recipe, and music tasks with parameterized names, content, locations, and song lists.Examples include browser drawing and maze tasks, stopwatch and timer actions, OsmAnd markers and tracks, recipe management, and playlist creation or export.
- Cross-application tasks: Cross-application tasks require transferring or transcribing information between gallery, VLC, Markor, messaging, and recipe applications.Examples include transcribing receipts or videos into Markor and importing recipes from images or text files.