Source-linked AI summary
Benchmarking General Mobile Assistants in Challenging Real-World Scenarios
Yiqi Zhu, Feiyu Gao, Jiaxing Fan, Jiahui Zeng, Minggang Wu, Chenliang Li, Haiyang Xu, Peng Li, Ming Yan, Yang Liu
TL;DR
Mobile-agent benchmarks have not fully captured the diversity and complexity of realistic mobile use. GMA addresses this gap with a reproducible benchmark spanning seven applications and 300 progressively difficult tasks, and evaluates frontier models and harness designs. Performance declines sharply on complex workflows, while harness choices can help demanding tasks but do not transfer uniformly across foundation models.
Problem
Existing mobile benchmarks provide strong foundations but have limited application coverage and task designs that do not fully capture realistic mobile-use diversity and complexity.
Method
GMA develops seven open-source-based applications and 300 tasks across four difficulty tiers, then evaluates eight frontier models and controlled context-retention and state-tracking harness choices.
Results
Performance consistently declines as task complexity increases; no evaluated model completes more than 20% of realistic tasks perfectly, while harness effects vary by foundation model.
Takeaways & Limitations
GMA provides a broader, more challenging testbed for evaluating mobile agents and studying how harness design shapes performance on complex workflows.
Takeaways & Limitations
Harness mechanisms are not universally transferable across foundation models, and broader experiments are needed to characterize their generalization.
Abstract
from arXiv · showhide
Graphical user interfaces have emerged as an important environment for evaluating autonomous AI agents on multimodal interactive tasks. Existing benchmarks such as AndroidWorld and MobileWorld provide strong foundations for mobile agent evaluation, but their application coverage and task design do not yet fully capture the diversity and complexity of realistic mobile use. We present GMA, a benchmark for evaluating general mobile assistants in challenging real-world scenarios. GMA introduces seven applications based on open-source projects, spanning domains such as lifestyle sharing and travel planning, and 300 tasks across four difficulty tiers, from atomic actions to complex multi-step workflows. We evaluate eight frontier models and find that performance declines substantially as task complexity increases, with current agents remaining far from reliably handling realistic user requirements. We further conduct controlled ablation studies of agent harness choices, including context retention and explicit state tracking, under a shared environment, model setting, and task taxonomy. Results show that appropriate harness design can meaningfully improve performance, particularly on demanding workflows, while the effectiveness of specific designs can vary across foundation models. Overall, GMA complements existing benchmarks by expanding application coverage and task complexity, providing a challenging testbed for evaluating mobile agents and studying how harness design supports reliable execution in complex mobile workflows.
1 INTRODUCTION
GMA addresses limits in mobile-agent evaluation by broadening application coverage and task complexity, then evaluates frontier models and harness choices in this setting.
- Realistic mobile GUI workflows require visual perception, intent grounding, long-horizon planning, information retrieval, constraint satisfaction, and error recovery.
- GMA expands mobile evaluation across broader functional domains, interface designs, interaction patterns, and workflows with cross-state dependencies.
- The benchmark contains seven custom applications and 300 tasks across Atomic, Compositional, Cross-Application, and Realistic tiers.
- Across evaluated frontier models, performance consistently declines with task complexity, and no model completes more than 20% of realistic tasks perfectly.The highest Average score on the realistic tier is 45.27.
- Controlled harness studies test context retention and explicit state tracking under a shared environment, model setting, and task taxonomy.
2 RELATED WORK
Related work has progressed from static GUI diagnostics to interactive, long-horizon agent evaluation, while recent studies increasingly treat the agent harness as a deliberate design component.
- Web, software-engineering, and GUI benchmarks evaluate agents through grounded actions, repository issue resolution, or interface interaction.
- Single-screenshot and recorded-trajectory benchmarks diagnose visual grounding and action prediction but do not fully capture closed-loop state changes.
- Mobile benchmarks extend evaluation through cross-application workflows, user interaction, MCP augmentation, online execution, personalization, and robustness.
- Agent harnesses organize interaction loops, manage context, connect models with actions, and coordinate perception, planning, memory, verification, and execution.
- GMA contributes seven custom applications spanning diverse interfaces and application-specific workflows for broader mobile-agent evaluation.
- Recent harnesses use specialized roles, unified execution loops, dynamic context handling, persistent memory, progress tracking, and action verification.
3 GMA ENVIRONMENT
GMA builds a reproducible, diverse Android ecosystem from open-source applications and unified state infrastructure, with tasks stratified by progressively demanding interaction requirements.
- Applications: Seven custom applications cover messaging, music, lifestyle sharing, local discovery, food delivery, e-commerce, and travel planning.
- Applications: Travel combines flight, hotel, and attraction planning with search, comparison, booking, payment, scheduling, and multi-field form completion.
- Reproducibility: The applications adapt functional open-source projects, self-host backends and databases, and remove production-service variability for deterministic evaluation.
- Reproducibility: GMA combines custom applications with Android utilities and retained MobileWorld applications to support sophisticated cross-application workflows.
- State infrastructure: A unified asset abstraction represents typed task-state elements such as contacts, messages, events, posts, orders, and bookings across applications.
- State infrastructure: State adapters and composable verification expose standardized operations and evaluate asset changes through strict success and fine-grained partial scores.
- Task design: The benchmark stratifies tasks by operation count, cross-application dependencies, information synthesis, and context ambiguity.
- Task design: Atomic tasks test single-step perception and control, while compositional tasks test multi-operation planning, retrieval, and state tracking.
4 EVALUATION ON GMA
GMA evaluates mobile agents with deterministic state-based metrics across diverse applications and difficulty tiers, finding sharp degradation on complex and realistic workflows.
- Experimental settings: Evaluation uses standard GUI operations plus call user and answer actions for ambiguous parameters and text-based deliverables.
- Experimental settings: The baseline agent uses zero-shot autoregressive reasoning from raw screenshots, without XML accessibility trees or structural UI hierarchies.
- Experimental settings: Each episode has a 150-step budget, and Success requires every subtask while Average measures the proportion completed.
- Results: Qwen3.8-Max leads overall with Success rate 67.33 and Average score 80.35, while GPT-5.6 Sol ranks second with 62.67 and 75.86.
- Results: On atomic tasks, the strongest model reaches 93.15 Success and 93.84 Average, but Qwen3.8-Max reaches only 52.56 Success on cross-application tasks.
- Results: No model exceeds 20.00 Success on realistic tasks; Claude Opus 4.7 leads Average at 45.27, while Qwen3.8-Max scores 38.22.
- Results: Realistic-task rankings differ from overall rankings, showing that simpler-task strength does not ensure reliable holistic workflows with long-horizon and cross-application dependencies.
- Results: Application performance varies substantially: Travel has Average score 20.17, whereas Meituan and HMDP achieve substantially stronger results.
5 HARNESS MATTERS FOR GMA TASKS
Harness choices substantially affect mobile-agent performance, especially on complex workflows, but their benefits depend on context size, task difficulty, and the underlying foundation model.
- 5.1 CONTEXT RETENTION: Eight retained screenshots raise Qwen3.7-Plus’s overall Success rate from 43.33 to 52.00 and Average score from 57.20 to 67.10 compared with one screenshot.On realistic tasks, Average score increases from 9.97 to 21.44, but retaining 16 screenshots provides no further overall improvement.
- 5.1 CONTEXT RETENTION: Removing previous reasoning traces lowers overall Success rate from 49.33 to 41.67 and Average score from 63.78 to 55.77 under the two-image setting.Average score also drops on compositional, cross-application, and realistic tasks, including from 11.00 to 6.89 on realistic tasks.
- 5.1 CONTEXT RETENTION: Captioning older screenshots raises realistic-task Average score from 11.00 to 34.60 under the two-image setting.Overall Success rate increases from 49.33 to 51.33 and Average score from 63.78 to 65.13.
- 5.2 EXPLICIT STATE TRACKING: With one retained screenshot, explicit state tracking increases Success rate from 43.33 to 53.67 and Average score from 57.20 to 67.85.Its benefit becomes smaller as more screenshots are retained, and with eight screenshots Average score decreases slightly from 67.10 to 66.95.
- 5.2 EXPLICIT STATE TRACKING: State tracking improves Average score across atomic and compositional tasks, remains mixed on cross-application tasks, and yields large gains on realistic tasks.With one screenshot, realistic-task Average score increases from 9.97 to 25.89; with sixteen, it rises from 23.81 to 37.72.
- 5.2 EXPLICIT STATE TRACKING: For the first three difficulty tiers, state-tracking gains generally shrink as more screenshots are retained, whereas realistic tasks retain a 13.91-point gain with sixteen screenshots.The sixteen-image realistic-task score increases from 23.81 to 37.72, only slightly below the 15.92-point gain with one screenshot.
- 5.3 MODEL-SPECIFIC HARNESS DESIGN: Structured state tracking improves Qwen3.7-Plus’s Success rate from 49.33 to 54.33 and Average score from 63.78 to 68.41.Its realistic-task Average score rises from 11.00 without state tracking to 24.92, while free-form state tracking provides limited overall improvement.
- 5.3 MODEL-SPECIFIC HARNESS DESIGN: For GPT-5.6 Sol, free-form state tracking reaches a 65.67 Success rate, while Structured reduces overall Average score from 75.86 to 73.96.Structured tracking also lowers Average score on cross-application tasks from 64.21 to 54.04 and realistic tasks from 43.64 to 42.56.
6 CONCLUSION
GMA broadens mobile-agent evaluation with diverse applications, progressive task complexity, and reproducible infrastructure. Results show increasing difficulty remains challenging, while harness choices can affect performance differently across models.
- GMA includes seven custom applications, 300 tasks, four progressive difficulty tiers, and reproducible evaluation infrastructure.
- Performance declines as task complexity increases, with cross-application and realistic workflows remaining particularly challenging.
- Context retention and explicit state tracking can meaningfully affect performance, but their benefits vary across foundation models.