A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation

1 Introduction

• Static screenshot evaluation misses dynamic state changes and cascading errors, and can penalize valid alternative trajectories.Dynamic benchmarks based on open-source apps also exclude widely used closed-source applications.Source: §1 · p. 1
• A3 benchmarks 100 daily-life tasks from 20 popular dynamic online apps spanning 20 Google Play Store categories.The apps include news, navigation, travel, email, and shopping.Source: §1 · p. 2
• A3 uses MLLMs as reward models to progressively verify task success and essential-state achievements, including intermediate progress when a task is incomplete.Both large commercial models and fine-tuned lightweight A3RM models can assess progress and final objectives.Source: §1 · p. 2
• A3 provides tools for streamlined agent execution, trajectory collection from agents and humans, and customizable evaluation.Dynamic frameworks can face unstable environments, difficult resets, and reliance on manual evaluation.Source: §1 · p. 2

2.1 GUI Agents

• GUI-agent approaches include fine-tuning a general-purpose MLLM, such as Bai et al. (2025b), and framework-style systems such as Mobile-Use.Frameworks may combine GUI-specific perception or UI-tree modules with multi-agent planning or shortcut guidance.Source: §2.1 · pp. 2–3

2.2 GUI Agent Benchmarks

• Static benchmarks such as AITW, AMEX, and AndroidControl assess single-step action accuracy, not sequential interaction.This limitation motivated dynamic evaluation frameworks.Source: §2.2 · p. 3
• Dynamic benchmarks vary in task and app coverage; AndroidWorld and AndroidLab use open-source, offline apps and omit common online apps and stochastic events.AndroidArena focuses on system apps, while SPA-bench (Chen et al., 2024b) has instability and unreliable resets.Source: §2.2 · p. 3
• Rigid checks such as predefined-answer matching and exact-state matching fail to capture task-completion nuances in dynamic environments.The authors identify a need for ecologically valid tasks with flexible, semantically aware evaluation.Source: §2.2 · p. 3

3.1 Apps & Tasks

• A3 curates 100 tasks across 20 popular apps and categories, selecting apps that support reliable resets and limiting tasks to their core functions.Tasks are classified as operation or information-query tasks and by difficulty using the number of steps required by a human expert.Source: §3.1 · p. 4

3.2 Essential-State Evaluation

• Essential states are observable semantic milestones that decompose a task into verifiable outcomes, unlike rigid sub-goals tied to internal UI elements.The states are designed to remain invariant across changing app content and diverse execution paths.Source: §3.2 · pp. 4–5
• Human operators define essential states from diverse successful trajectories, then audit them for visual verifiability, criticality, and task coverage.The protocol includes collaborative state definition and an independent audit.Source: §3.2 · p. 5
• A3 defines task success as achieving all associated essential states and reports partial progress with the Essential-State Achieved Rate.The rate is the number of achieved essential states divided by the total number of essential states.Source: §3.2 · p. 5
• A sliding window passes trajectory screen frames to an MLLM judge, which checks for essential states in each segment until the trajectory is covered.The window advances through the trajectory at a predefined interval.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3 MLLM-Based Judgment

• MLLM judgment addresses the difficulty of verifying semantic milestones from inconsistent UI trees and the limits of whole-trajectory evaluation.A3 decomposes evaluation into discrete essential-state checks rather than judging a multi-step trajectory at once.Source: §3.3 · pp. 5–6
• A3 supports commercial MLLM judges while introducing A3RM, a fine-tuned Qwen3-VL-8B reward model for local deployment without API fees.The commercial API cost motivates optimizing sliding-window parameters to reduce calls without compromising evaluation fidelity.Source: §3.3 · p. 6

3.4 A3 Pipeline Suite

• The open-source A3 pipeline includes AITK for agent-device interaction, data collection, and task execution, plus an evaluator applicable to standard trajectory data.AITK uses a plug-in architecture for custom agents and tasks.Source: §3.4 · p. 6

4.1 A3 Reward Model (A3RM)

• A3RM uses Qwen3-VL-8B with a window size of 2 and interval size of 1 for data collection and training.The authors report that preliminary experiments favored the small window, likely because of the 8B model’s more limited context capacity.Source: §4.1 · p. 6
• A3RM training data combines 8,241 expert-trajectory step samples, including 981 positive and 7,260 negative samples, with 1,083 additional agent-trajectory negatives.Annotators labeled essential-state transitions as positive; other expert transitions were negative, and agent trajectories supplied mined negatives.Source: §4.1 · pp. 6–7
• A3RM reaches 95.7% Precision (+8.4 points) at the essential-state level and 96.0% Precision (+10.3 points) at the task level, maintaining parity in Recall with Gemini-2.5-pro.The held-out A3 test comparison indicates that the specialized model rejects ambiguous states that zero-shot models often hallucinate as successes.Source: Table 2 · p. 7, §4.1 · p. 7
• Training oversampled positive examples fourfold per epoch to address class imbalance and avoid a model that predicts only negative outcomes.The model was fine-tuned with dynamic sampling policy optimization (DAPO).Source: §4.1 · p. 7

4.2 A3 Benchmark

• The benchmark evaluation covers seven single-model agents and two frameworks, using their official public prompts and inference settings.The report identifies Table 3 for A3RM-evaluated results and Table 6 for Gemini-evaluated results.Source: §4.2 · p. 7
• 53.0% SR for T3A + Gemini-2.5-pro sets the benchmark upper bound, while InfiGUI-R1 leads open-source single-model agents at 27.0% SR.Most models’ success rates on Hard tasks frequently approach zero, and even the top-performing system fails nearly half the tasks.Source: §4.2 · p. 7, Table 3 · p. 8
• Across agents, ESAR is substantially higher than SR, indicating that agents often reach early task stages but fail before the final state.The authors associate this gap with insufficient long-horizon robustness and terminal errors; case studies also report failures in progress awareness.Source: §4.2 · p. 8

5 Conclusion

• A3 benchmarks dynamic tasks from popular online applications, bridging static evaluations and real-world utility.The benchmark’s online-app tasks address settings that static evaluation does not cover.Source: §5 · p. 8
• Essential-state evaluation and fine-tuned A3RM provide a visual-based measure of granular agent progress beyond binary success.Source: §5 · p. 8
• Experiments find that current agents struggle with long-horizon robustness, and the open-sourced pipeline provides infrastructure to accelerate development of mobile assistants.Source: §5 · p. 8

Limitations

• The benchmark is confined to Android; restrictive system policies prevent reproducible iOS virtualization and programmatic control.Source: Limitations · p. 9
• Instant messaging is excluded because authentication protocols and privacy concerns make automated account resets impractical.Source: Limitations · p. 9
• A3RM has high precision but remains probabilistic and can still hallucinate.Source: Limitations · p. 9

A.1 App List

• The app list covers 20 applications selected from Google Play Store top charts across 20 categories.Source: Table 4 · p. 11, §A.1 · p. 11

• Table 8 lists ten examples of A3 tasks and their corresponding essential states.Source: §A.2 · p. 11

• The study tests sliding-window sizes from 2 to 6, using interval sizes equal to half the window size.Source: §A.3 · p. 11
• Window size 4 with interval 2 is preferred because sizes 3 and 4 yield the highest accuracy, while size 4 has the lowest average computational cost.Source: §A.3 · p. 11
• Accuracy degrades at both window-size extremes: small non-overlapping windows can split state transitions, while large windows downscale frames and obscure UI details.Source: §A.3 · p. 11

A.4 Gemini Evaluation

• T3A + Gemini-2.5-pro leads at 58.0% SR; most open-source agents fall to single-digit success on Hard tasks.Source: §A.4 · pp. 11–12, Table 6 · p. 13
• Gemini scores agents higher than A3RM, though their relative rankings remain largely consistent across evaluators.Source: §A.4 · p. 12

A.5.1 General Agent Issues

• Agents repeat already-completed steps and often fail to stop after all essential states are met, exposing weak progress awareness on multi-step tasks.Source: §A.5.1 · p. 12
• Screen-comprehension failures include misidentifying click targets and mislocalizing their coordinates.Source: §A.5.1 · pp. 12–13

A.5.2 Dynamic Online Apps Specialty

• Online-app errors include similar search-result distractions, ads or sponsored content, and dense interfaces with nested navigation.Source: §A.5.2 · p. 13

A.6 Generalization Analysis

• On 25 unseen tasks, A3RM outperforms Gemini-2.5-pro across all metrics; continued training improves Precision and Accuracy while maintaining stable Recall.Source: §A.6 · p. 14

A.7 Risks & License

• Because some online apps require login, agents’ wrong actions may cause content malfunctioning.Source: §A.7 · p. 14