Source-linked AI summary
MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
Xuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang, Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, Weiyun Wang, Xiangyu Zhao, Jixuan Chen, Haodong Duan, Tianbao Xie, Chenyu Yang, Shiqian Su, Yue Yu, Yuan Huang, Yiqian Liu, Xiao Zhang, Yanting Zhang, Xiangyu Yue, Weijie Su, Xizhou Zhu, Wei Shen, Jifeng Dai, Wenhai Wang
TL;DR
GUI-agent benchmarks often assess isolated capabilities, accuracy, or narrow scenarios, leaving comprehensive, efficiency-aware, multi-platform evaluation limited. MMBench-GUI addresses this with a hierarchical benchmark and EQA metric, finding that visual grounding is the primary performance determinant and that agents incur substantial redundant steps.
Problem
Existing GUI-agent benchmarks incompletely assess interconnected capabilities, operational efficiency, and diverse real-world platforms and applications.
Method
MMBench-GUI organizes over 8,000 multi-platform tasks into four increasingly complex levels and introduces EQA to jointly assess task success and execution efficiency.
Results
Visual grounding was the primary performance determinant: with the same planner, improving localization increased SR 2.8× (∆= 17.25), while stronger planning yielded marginal returns.
Takeaways & Limitations
Reliable GUI automation requires precise visual grounding, effective planning, cross-platform generalization, and early stopping to reduce redundant execution.
Takeaways & Limitations
A limited action space restricts agents’ ability to execute planned actions, especially in GUI task collaboration scenarios.
Abstract
from arXiv · showhide
We introduce MMBench-GUI, a hierarchical benchmark for evaluating GUI automation agents across Windows, macOS, Linux, iOS, Android, and Web platforms. It comprises four levels: GUI Content Understanding, Element Grounding, Task Automation, and Task Collaboration, covering essential skills for GUI agents. In addition, we propose a novel Efficiency-Quality Area (EQA) metric to assess GUI agent execution efficiency in online automation scenarios. Through MMBench-GUI, we identify accurate visual grounding as a critical determinant of overall task success, emphasizing the substantial benefits of modular frameworks that integrate specialized grounding modules. Furthermore, to achieve reliable GUI automation, an agent requires strong task planning and cross-platform generalization abilities, with long-context memory, a broad action space, and long-term reasoning playing a critical role. More important, task efficiency remains a critically underexplored dimension, and all models suffer from substantial inefficiencies, with excessive redundant steps even when tasks are ultimately completed. The integration of precise localization, effective planning, and early stopping strategies is indispensable to enable truly efficient and scalable GUI automation. Our benchmark code, evaluation data, and running environment will be publicly available at https://github.com/open-compass/MMBench-GUI.
1 INTRODUCTION
MMBench-GUI is a hierarchical, cross-platform benchmark addressing limitations in existing GUI-agent evaluations through four progressive capability levels and unified coverage across widely used operating systems. It also introduces EQA to assess online-task success together with execution efficiency, while identifying precise visual grounding as a core determinant of task success.
- Motivation: Existing GUI-agent benchmarks largely assess isolated capabilities and insufficiently cover real-world GUI systems and overall agent abilities.
- Benchmark design: MMBench-GUI evaluates GUI agents through four progressive levels: content understanding, element grounding, task automation, and task collaboration.The levels span basic interface understanding to complex cross-application task execution.
- Findings: Precise visual grounding is a core determinant of task success, despite general language models excelling at high-level planning and reasoning.
- Benchmark design: The benchmark provides hierarchical, cross-platform evaluation across Windows, Linux, macOS, Android, iOS, and the Web under a unified protocol.It uses progressive task organization, fine-grained difficulty stratification for static tasks, and cleaned splits and novel constructions for dynamic tasks.
- Efficiency evaluation: EQA jointly evaluates online-task success and efficiency by considering when tasks finish within the step budget and measuring action redundancy.The metric extends success-rate-focused evaluation toward agents that are capable and efficient.
2 RELATED WORKS
Prior GUI-agent research is organized around modular schemes that combine general-purpose vision-language planners with specialized grounding models. Existing GUI benchmarks address screenshot, webpage, and cross-platform video-based question answering, but general multimodal QA benchmarks largely overlook the intertwined visual-textual demands of GUIs.
- GUI Agents: Modular GUI-agent schemes typically pair general-purpose VLMs, such as GPT-4o, as planners with specially trained GUI grounding models.This paradigm is represented by multiple recent works and focuses on UI-element grounding.
- GUI Benchmarks: GUI understanding requires jointly interpreting intertwined visual and textual cues, a capability largely outside general-purpose multimodal QA benchmarks.The passage frames this gap as a limitation of existing broad multimodal evaluation resources.
- GUI Benchmarks: ScreenQA, WebSRC, and GUI-World provide GUI question-answering resources from Android screenshots, web pages, and cross-platform video data, respectively.These datasets extend GUI evaluation across distinct content formats and platforms.
3 MMBENCH-GUI
MMBench-GUI is a hierarchical benchmark for systematically evaluating GUI agents across platforms and progressively complex capabilities. It spans content understanding, element grounding, task automation, and task collaboration, while introducing efficiency-aware evaluation and broad, balanced task coverage.
- Framework: MMBench-GUI organizes GUI-agent evaluation into four ascending levels that address isolated benchmark limitations and progressively more demanding scenarios.The levels are L1-GUI Content Understanding, L2-GUI Element Grounding, L3-GUI Task Automation, and L4-GUI Task Collaboration.
- Scale and Coverage: The benchmark contains over 8,000 tasks spanning diverse platforms, with task counts shrinking from 3582 at L1 to 248 at L4.This taper reflects increasing annotation cost and difficulty while retaining about 250 samples for top-tier evaluation.
- L1–L2: L1 tests GUI content understanding as a fundamental prerequisite for subsequent tasks, while L2 measures accurate spatial localization of actionable GUI elements.L2 instructions can derive from direct user tasks or an agent’s internal reasoning and output target coordinates (x, y).
- L3 Task Automation: L3 evaluates multi-step task automation within a single application, and MMBench-GUI-macOS adds 70 curated tasks across 9 widely used macOS applications.The macOS set contains 35 L3 tasks and 35 L4 tasks.
- Efficiency Evaluation: EQA jointly measures task success and completion speed, rewarding agents that solve more tasks using fewer steps rather than relying only on Success Rate.The metric is inspired by the AP computation protocol in COCO and uses M = 101 uniformly spaced evaluation points.
- L4 Task Collaboration: L4 assesses reasoning, collaboration, and adaptive automation across applications, including global reasoning, inter-app dependencies, and complex workflows.The benchmark’s platform distribution avoids over-specialisation: no single platform exceeds 22% of the corpus.
4 BENCHMARKING GUI AGENT BASELINES
MMBench-GUI benchmarks contemporary proprietary and open-source GUI agents using screenshot-and-instruction inputs across four task levels and multiple platforms. Results show strong content-understanding and grounding differences, but low automation and collaboration success, with platform, instruction, and long-horizon execution constraints remaining substantial.
- Evaluation Setup: The benchmark evaluates proprietary and open-source models using screenshots and task descriptions without auxiliary A11y trees or Set-of-Marks data.Models are evaluated through a unified OpenAI API-compatible interface, while L3 and L4 are judged by final-state success criteria.
- L1-GUI Content Understanding: InternVL3-72B achieves the highest L1 content-understanding scores across every platform and difficulty tier, with performance decreasing as task difficulty increases.Qwen2.5-VL-72B and Qwen-Max-VL generally rank below InternVL3-72B, while Android and Web show greater variability for many models.
- L2-GUI Element Grounding: UI-TARS-72B-DPO and InternVL3-72B lead L2 grounding, achieving overall averages of 74.25% and 72.20%, respectively, with strong cross-platform consistency.GPT-4o and Claude-3.7 remain near zero, while grounding generally favors mobile and Web platforms and declines from Basic to Advanced instructions.
- L3-GUI Task Automation: GPT-4o + UI-TARS-1.5-7B achieves the best L3 average SR at 26.60%, while most other models remain below 20%.UI-TARS-72B-DPO performs best among GUI-specific models on overall SR and EQA, particularly on Linux and Android.
- L4-GUI Task Collaboration: L4 performance falls sharply: GPT-4o + UI-TARS-1.5-7B reaches only 8.78% average SR, and most models remain below 6%.The decline highlights the difficulty of cross-application, long-horizon, and multi-step execution.
- Execution Efficiency and Platforms: Increasing the maximum allowed steps from 15 to 50 improves SR and EQA, but overall completion remains low across models and settings.Top-performing agents generally achieve higher SR and EQA on Android and Web than on desktop platforms, especially macOS.
5 ANALYSIS AND DISCUSSION
The analysis identifies visual grounding as the primary bottleneck in GUI automation, while also highlighting major efficiency, action-space, generalization, and cross-application memory limitations. It advocates modular grounding, early stopping, richer actions, and stronger cross-context capabilities for reliable and scalable agents.
- Model capabilities: General-purpose language models excel at planning and self-reflection but struggle with fine-grained visual interactions.Their L2 scores are 2.87 for GPT-4o and 4.66 for Claude-3.7, versus 65.68% for UGround-V1-7B.
- Visual grounding: Improving localization with the same planner increased success rate by 2.8× (∆= 17.25), establishing visual grounding as the primary performance bottleneck.Localization gains are nearly linear, while improving reasoning yields diminishing returns once the agent sees sufficiently well.
- Efficiency: Efficiency remains underexplored because agents can complete tasks while taking redundant steps, motivating EQA-based evaluation and early stopping.GPT-4o paired with specialized grounders raises 50-step success rate by roughly 5.7% but still incurs EQ2 = 7-8.
- Action space: Restricted and platform-incompatible action spaces prevent planned operations, especially tab switching and cross-window information transfer in web and collaborative tasks.Prompt-based action definitions incompletely represent desktop, mobile, and web operations, and models may emit incompatible actions.
- Generalization: Agent effectiveness deteriorates sharply as local difficulty or global complexity rises from L1 to L4, exposing weak generalization to harder GUI situations.The bottlenecks include small or non-salient visual cues, longer credit-assignment chains, and exponentially expanding noisy action spaces.
- Cross-application collaboration: Multi-application failures primarily reflect limited cross-context memory and action space, with GPT-4o+UI-TARS-1.5-7B falling from 26.6% SR on L3 to 8.8% on L4.UI-TARS-72B-DPO similarly drops from 25.3% to 8.0%, while failures concentrate at window or tab boundaries.
6 CONCLUSION
MMBench-GUI is presented as a hierarchical, multi-platform framework for evaluating GUI automation agents, with accurate visual grounding, sophisticated planning, and robust cross-platform generalization identified as key performance factors. Future work will broaden model coverage, deepen analysis, and expand tasks with error attribution.
- Conclusion: MMBench-GUI comprehensively evaluates GUI automation agents across multiple operating systems and diverse tasks.The framework is described as hierarchical and multi-platform.
- Conclusion: Accurate visual grounding, sophisticated planning, and robust cross-platform generalization emerge as critical performance factors.These factors are highlighted as key bottlenecks in GUI automation agent performance.
- Future Work: Future work will broaden model coverage across open-source, proprietary, and latest RL-based systems and difficulty levels.The study plans to evaluate a wider spectrum of models across all difficulty levels.
- Future Work: Future work will conduct finer-grained analyses with a richer experimental pool to produce more robust and generalizable findings.The proposed deeper analysis is intended to strengthen the study’s findings.
- Future Work: Future work will expand online tasks to cover broader applications and support error attribution.The planned task expansion combines broader task coverage with analysis of error sources.