Source-linked AI summary
APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
Jintian Feng, Long Chen, Xiao Yu, Jiayi Dai, Chenglong Liu, Haoru Wang, Zizhen Xue, Yuxuan Shi, Ziyang Wang, Yichen Gong
TL;DR
Mobile GUI-agent evaluation must balance realistic interaction with reproducibility, since live apps vary and trace matching can reject valid workflows. AppSim-Bench addresses this with controllable simulated apps and deterministic outcome verification, then shows that autonomous mobile execution remains far from solved.
Problem
Live-app variability and path-based evaluation make it difficult to measure mobile GUI agents’ capabilities both realistically and reproducibly.
Method
AppSim-Bench uses task-relevant simulated apps, controllable backend data, deterministic outcome verification, and a coding-agent-assisted, human-verified construction workflow.
Results
50.27% is the leading model’s task-completion rate, while 28.55% of tasks remain unsolved by any evaluated agent.
Takeaways & Limitations
Reliable autonomous mobile execution remains unresolved, with performance declining on longer workflows and numerical reasoning tasks.
Takeaways & Limitations
The benchmark simplifies dynamic content, system-level factors, functional breadth, and data scale, so complementary real-world studies remain needed for deployment-time robustness.
Abstract
from arXiv · showhide
Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible. Existing benchmarks typically trade off these goals: simplified apps lack real-world mobile complexity, whereas live commercial apps introduce uncontrolled variation from recommendations, advertisements, accounts, and changing content. We propose AppSim-Bench, which addresses this trade-off through controllable simulated apps that preserve task-relevant interaction logic while supporting deterministic evaluation. Built through a coding-agent-assisted and human-verified workflow, it contains 557 tasks across 17 high-frequency Chinese and English apps. Its controllable backend data and outcome-based verification remove major sources of environmental stochasticity, enabling reproducible cross-model comparison. Evaluating 19 GUI agents, spanning general-purpose and GUI-specialized systems, we find that autonomous mobile execution remains far from solved. The best model completes only 50.27% of tasks, and 28.55% of tasks are not solved by any agent. Further analysis shows that failures concentrate in longer workflows, numerical reasoning tasks, and inefficient trajectories marked by high action overhead and budget exhaustion. Our project is available at https://github.com/Acrab-Agentic-Labs/AppSim.
1 Introduction
Mobile GUI agents can perform multi-step tasks, but evaluating their capabilities reproducibly remains difficult because trace-based methods reject valid alternative paths and live apps vary across runs. AppSim-Bench addresses this gap with controllable simulated apps, deterministic outcome verification, and a broad task suite.
- Trace-based evaluation can mark valid alternative workflows as failures because it compares execution paths with a human-annotated path.
- Live mobile environments vary through recommendations, advertisements, account states, and dynamic loading, undermining reproducible comparison.
- AppSim-Bench uses simulated apps with task-relevant UI flows, resettable initial states, and controllable backend data to reduce evaluation noise.
- 557 tasks across 17 simulated apps cover Chinese and English mobile ecosystems.
- A deterministic protocol separates benchmark specification, reference execution, and official scoring across state, answer, and hybrid task types.
2 Related Work
Mobile GUI-agent benchmarks have progressed from static traces to dynamic and physical-device evaluation, but realism often comes at the cost of reproducibility. Simulated environments improve control, while AppSim-Bench targets pixel-grounded GUI interaction rather than code generation or API composition.
- Static and trace-based evaluation: Static screenshot, demonstration, and trace benchmarks are easy to standardize but struggle when the same goal has multiple valid trajectories.
- Dynamic evaluation: Dynamic benchmarks execute interactive applications and evaluate completion through system-state changes, providing more realistic measurements.
- Real-world mobile evaluation: Physical-device evaluation increases realism but remains vulnerable to changing recommendations, advertisements, and online content.
- Simulated environments: AppWorld shares controllability and outcome-based verification with AppSim-Bench but evaluates code-generating agents through 457 APIs rather than pixel-grounded GUI agents.
3 AppSim-Bench
AppSim-Bench combines natural-language tasks, task-relevant simulated apps, controllable backend data, and deterministic outcome verification. Its 557-task suite spans diverse Chinese and English apps and includes long-horizon and numerical-reasoning workflows.
- Benchmark components: The benchmark has three components: natural-language tasks, simulated apps with controllable backend data, and automated outcome-based evaluation.
- Task and app coverage: 557 tasks span 17 applications from Chinese and English ecosystems, including social, commerce, travel, food-delivery, navigation, and media scenarios.
- Task construction: Fifteen volunteers supplied representative real-world interactions that were refined into natural, unambiguous, machine-interpretable instructions.
- Numerical reasoning: Numerical-reasoning tasks require extracting interface numbers and performing counting, arithmetic, comparison, or threshold filtering.
- Simulated app construction: Coding agents generate candidate apps from human specifications, after which developers inspect, execute, and correct them.
- Controlled environments: Deterministic initial states and controllable data limit variation from personalization, advertisements, recommendation updates, and dynamic content.
- Automated evaluation: Outcome evaluation checks final goals through state, answer, and hybrid verification rather than predefined action-path matching.
- Benchmark statistics: 282 tasks require at most 5 reference steps, 140 require 6–10, and 135 require at least 11; 183 tasks involve numerical reasoning.
4 Experiments
AppSim-Bench evaluates mobile GUI agents with controlled simulated apps and outcome-based metrics, revealing substantial performance gaps, shared failures, and degradation on longer and numerically demanding tasks. Efficiency analyses further associate lower accuracy with longer trajectories and budget exhaustion.
- Human Validation of Visual Realism: 46.3% of static-screenshot judgments were undecided, and simulations were chosen as real in 19.0% of decided cases.For dynamic sequences, the corresponding figures were 35.7% and 15.2%.
- Human Validation of Visual Realism: 4.28% is the pessimistic upper bound for success attributable to pruning extraneous real-app states.The simulator preserves task-relevant pages and transitions while removing states outside benchmark scope.
- Overall Performance: Claude-Opus-4.7 reaches 50.27% overall accuracy, while GUI-specialized agents remain below this ceiling.Mobile GUI execution remains unreliable across the evaluated agent set.
- Overall Performance: 28.55% of tasks are solved by no model, whereas only 0.36% are solved by all evaluated models.The distribution indicates shared blind spots beyond differences in aggregate accuracy.
- Task Length as a Long-horizon Bottleneck: Accuracy consistently decreases as reference-step count increases, with GPT-5 falling from 43.62% to 31.43% and then 14.07%.The strongest result on the longest tasks is only 33.33%, and the decline also holds for GUI-specialized agents.
- Task Type Analysis: Threshold filtering is especially difficult: Claude-Opus-4.7 reaches 42.86%, GPT-5.5 reaches 33.33%, and most other models remain below 15%.The task requires extracting values, manipulating them, comparing them, and maintaining constraints during interaction.
- Action Efficiency and Budget Exhaustion: Accuracy is negatively associated with average valid actions, Action Overhead, and budget exhaustion, with budget exhaustion showing correlations of -0.705 and -0.699.The association is weakest for Action Overhead because GUI-specialized agents produce short trajectories on the few tasks they solve.
- Action Efficiency and Budget Exhaustion: Budget-exhausted trajectories achieve only 0.44% accuracy, while 1,690 contain at least three consecutive repeated actions and 1,347 contain at least five.These repetitions indicate limited progress, although they are not exact loop counts because repeated actions can be valid when screen states change.
5 Conclusion
AppSim-Bench provides a realistic, controllable, and reproducible environment for evaluating mobile GUI agents. Across 19 agents, results show that reliable autonomous mobile execution remains unresolved.
- AppSim-Bench evaluates mobile GUI agents in realistic and controllable environments using controllable backend data and deterministic outcome verification.
- 50.27% of tasks are completed by the leading model, while 28.55% remain unsolved by every evaluated agent.
- Performance drops sharply on longer workflows and numerical reasoning tasks.
- High action overhead and frequent interaction-budget exhaustion indicate major execution inefficiency.
- The evaluation highlights needs for improved long-horizon state tracking, numerical constraint handling, and recovery from stalled interactions.
Limitations
AppSim-Bench prioritizes reproducibility through simulated environments, but this choice limits realism and scope. Its fidelity and quality evidence are also bounded by indirect measurements and limited auditing.
- AppSim-Bench omits live personalization, advertisements, network variation, account-specific content, and continuously changing services, so complementary real-world studies remain necessary.
- 557 tasks across 17 apps provide broad but incomplete coverage, with future expansion needed for cross-app, real-time, accessibility-oriented, and security-sensitive scenarios.
- The simulators simplify functional breadth, dynamic content, system-level factors, and data scale in exchange for deterministic verification.
- Fidelity and construction quality rely on indirect behavioral measures, human inspection, and a single-app error audit rather than standardized agreement or broad residual-bug evidence.
Ethics Statement
The benchmark is intended for non-commercial, reproducible research and uses synthetic or manually defined data rather than commercial services or private records. Its simulated apps are constructed and evaluated through specified schemas, task-relevant flows, human inspection, and deterministic state or answer checks.
- AppSim-Bench is intended for non-commercial research and is not connected to commercial services, user accounts, payment systems, or private user data.
- All backend data are manually defined or synthetic, and no real user records are used.
- Backend schemas retain task-required entities and attributes while omitting unrelated fields to keep app state sufficient for evaluation.
- Task-relevant page hierarchies and navigation paths constrain the simulated UI flows.
- Coding agents generate candidate implementations under human-provided specifications, after which developers inspect, execute, and correct them.
- Construction specifies task requirements, backend schemas, page hierarchies, and verification signals for each simulated app.
- Evaluators verify state tasks by comparing final state St with goal G, while answer tasks use schema validation, canonicalization, and deterministic matching.
- Hybrid tasks require both state-transition and answer-schema checks, and audited answer extraction produced no errors in 3,933 cases.
D Benchmark Construction Quality Control
Benchmark construction combines lead-developer implementation with independent quality-control inspection and iterative correction. Reviewers tested both successful and deliberately failing trajectories, while Uber Eats required repeated correction rounds before page acceptance.
- Each simulated app was implemented by a lead developer and independently inspected by several quality-control reviewers before acceptance.
- Figure 7 compares real-app and simulator page transitions, marking retained task-relevant pages against excluded real-app pages.
- Table 4 counts human inspection-and-correction iterations before each Uber Eats page was accepted.
- Reviewers tested correct-completion and deliberate-failure trajectories and found no evaluator misjudgments.
- No Uber Eats page passed on the first attempt; pages required about 2.3 correction rounds on average.
E Behavioral Fidelity of Task-aware Pruning
AppSim-Bench preserves task-relevant interaction paths while pruning irrelevant pages, then measures whether this pruning materially affects evaluation outcomes. Audits indicate that the residual effect is small and does not provide an exploitable shortcut.
- Behavioral fidelity: Multiple valid paths are implemented for each retained task, while irrelevant pages display a development placeholder to signal leaving task scope.The placeholder is the sole behavioral difference from a fully realistic app in pruned regions.
- Audit design: The audit covered six apps across four categories and three agents selected to span the observed accuracy range.Manual replay was required to determine whether trajectories reached a placeholder, so the audit sampled rather than covered all 19 agents.
- Residual effect: 7.34% is the maximum residual for any single agent, while the aggregate residual is 4.28% across 654 executions.The residual counts executions that entered a placeholder and still succeeded, using a deliberately pessimistic upper bound.
- Interpretation: Replacing placeholders with fully implemented pages would shift measured accuracies by at most a few points, and pruned regions do not constitute an exploitable shortcut.Residual effects were concentrated in a few app–agent combinations and were near zero elsewhere.
F Error Decomposition on Numerical-reasoning Tasks
The numerical-reasoning audit separates interface reading from downstream reasoning. Across audited failures, perception was nearly error-free, while reasoning accounted for almost all observed failures.
- Error decomposition: Numerical-reasoning tasks require first extracting interface values, then applying counting, arithmetic, comparison, or threshold filtering.Failures in these stages have different implications for what the benchmark measures.
- Audit design: 1,287 execution steps were audited manually because a judge model prone to misreading could not reliably assess another model’s extraction.The manual approach limited the audit scope.
- Findings: Only one of 1,287 steps contained a perception or extraction error, so almost all observed failures arose during reasoning.The audit targeted JD because its dense prices, discounts, quantities, and ratings place the heaviest load on perception.
- Findings: Gemini-2.5-Pro failed 25/27 audited tasks while showing near-zero extraction errors, further indicating reasoning rather than reading as the discriminating factor among audited state-of-the-art agents.The authors caution that weaker agents may exhibit more perception errors.
G Failure Modes of Unsolved Tasks
Among 159 tasks unsolved by every evaluated agent, failures cluster around long-horizon execution, incomplete information gathering, and numerical reasoning chains. The hardest tasks combine several of these capability gaps.
- Failure modes: 159 tasks were solved by no evaluated agent, with failures concentrating in three capability gaps.These gaps are long-horizon workflows, incomplete information gathering, and numerical reasoning chains.
- Long-horizon workflows: Long workflows require agents to track completed actions, interpret the current screen against the instruction, and prevent intermediate errors from accumulating.This failure mode concerns maintaining progress and task context across multiple steps.
- Information gathering: Incomplete information gathering occurs when agents stop exploring too early, such as inspecting only the visible portion of a list before deciding.The failure reflects decisions made from partial interface information.
- Numerical reasoning: Numerical reasoning chains fail when agents must enumerate relevant interface values before comparing or aggregating them, because a break at any link causes failure.The paper’s hardest examples stack this gap with incomplete information gathering.
- Combined difficulty: AMAZON#21 required finding the three most expensive cart items and summing them, but no agent produced the verifier’s expected total of 3118.98.Every reported total was lower, consistent with deciding before the list was fully enumerated.
K Example Agent Trajectory
A solved Amazon-cart task illustrates outcome-oriented GUI execution: the agent opens the app, reaches the cart, exhaustively scrolls through all items, identifies the cheapest electronics item, and returns its name and price.
- Task setup: The task asks for the cheapest electronics item in the cart and requires inspecting the full list before comparing prices.The trajectory uses open_app, click, scroll, and answer actions before reporting completion.
- Information gathering: The agent performs repeated downward scrolls because the cart initially exposes only its top portion and additional items appear incrementally.It stops after reaching the end-of-list marker.
- Reasoning and answer: After seeing all 12 items, the agent identifies Marshall Acton III as the cheapest electronics item at $199.99.The answer is returned in structured form with the item name and price.
- Verification: The verifier confirms the answer contains the gold keys Marshall Acton III and 199.99, marking the task solved.The agent then emits a complete status action.