Source-linked AI summary
VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Tasks
Beitong Zhou, Zhexiao Huang, Yuan Guo, Zhangxuan Gu, Tianyu Xia, Zichen Luo, Fei Tang, Dehan Kong, Yanyi Shang, Suling Ou, Zhenlin Guo, Changhua Meng, Shuheng Shen
TL;DR
Existing GUI grounding benchmarks have limited scale and coverage, and often emphasize narrow tasks or specialized domain knowledge. VenusBench-GD combines cross-platform data with a hierarchical set of grounding tasks and a human-reviewed, model-assisted annotation pipeline. Its results show a split: general MLLMs match or surpass GUI-specialized models on basic tasks, while advanced tasks favor specialized models.
Problem
Existing GUI grounding benchmarks are limited in scale and diversity, often require specialized domain knowledge, and underrepresent complex, compositional interactions in everyday GUI workflows.
Source: §1 · pp. 1–2Method
VenusBench-GD combines a three-platform, bilingual benchmark with six basic and advanced grounding tasks and an annotation pipeline using expert review and model-assisted checks.
Source: §1 · p. 2Results
General MLLMs match or surpass GUI-specialized models on basic tasks, while advanced reasoning and functional grounding favor specialized models; Qwen3-VL-8B leads comparable-scale models on average basic-task accuracy at 76.96%.
Source: §1 · p. 3Takeaways & Limitations
The benchmark provides a multidimensional evaluation intended to differentiate GUI agents and support valid evaluation across tasks and languages.
Source: §5 · p. 8Models often fail to compose multiple visual attributes in visual grounding, selecting partially matching elements after focusing on only one salient feature.
Source: §10 · p. 15
Abstract
from arXiv · showhide
GUI grounding is a critical component in building capable GUI agents. However, existing grounding benchmarks suffer from significant limitations: they either provide insufficient data volume and narrow domain coverage, or focus excessively on a single platform and require highly specialized domain knowledge. In this work, we present VenusBench-GD, a comprehensive, bilingual benchmark for GUI grounding that spans multiple platforms, enabling hierarchical evaluation for real-word applications. VenusBench-GD contributes as follows: (i) we introduce a large-scale, cross-platform benchmark with extensive coverage of applications, diverse UI elements, and rich annotated data, (ii) we establish a high-quality data construction pipeline for grounding tasks, achieving higher annotation accuracy than existing benchmarks, and (iii) we extend the scope of element grounding by proposing a hierarchical task taxonomy that divides grounding into basic and advanced categories, encompassing six distinct subtasks designed to evaluate models from complementary perspectives. Our experimental findings reveal critical insights: general-purpose multimodal models now match or even surpass specialized GUI models on basic grounding tasks. In contrast, advanced tasks, still favor GUI-specialized models, though they exhibit significant overfitting and poor robustness. These results underscore the necessity of comprehensive, multi-tiered evaluation frameworks.
1. Introduction
Existing GUI grounding benchmarks are limited in scale, coverage, and task complexity. VenusBench-GD introduces a broad, hierarchical evaluation benchmark, with results showing that model comparisons differ between basic and advanced tasks.
1. Introduction
- Existing benchmarks can saturate on narrow tasks, demand specialized domain knowledge, and underrepresent the complex interactions involved in real-world GUI workflows.ScreenSpot-V2 approaches 95% accuracy with current methods, while many benchmarks emphasize locating icons or text.Source: §1 · pp. 1–2
- VenusBench-GD covers 97 applications across three platforms and 10 domains, with 6,166 annotated image-instruction pairs and 13 UI element types.It provides instructions and screenshots in English and Chinese.Source: §1 · p. 2
- Its annotation pipeline combines expert human review, model-assisted consistency checks, and repeated localization calibration, yielding higher label quality than existing benchmarks in blind quantitative analysis.The paper describes these steps as multiple rounds of verification and localization calibration.Source: §1 · p. 2
- The six-task hierarchy separates basic element, spatial, and visual grounding from advanced functional, reasoning, and refusal grounding.The tiers assess UI perception alongside higher-order reasoning and robustness to complex scenarios and ambiguity.Source: §1 · p. 2
- The benchmark spans desktop, mobile, and web, consolidating representations of the same software across platforms for its domain-distribution overview.The paper describes the benchmark as more generalizable, realistic, and challenging for next-generation multimodal grounding models.Source: Figure 3 · p. 3, §1 · p. 3
- General MLLMs match or surpass specialized models on basic tasks, while specialists lead on advanced reasoning and functional tasks, which score significantly lower overall.The reported top comparable-scale basic-task average is Qwen3-VL-8B at 76.96%.Source: Figure 1 · p. 1, §1 · p. 3
2. Related Work
The paper situates GUI grounding as the link between language-based planning and interface actions, then identifies gaps in benchmark scale, platform and domain coverage, and task granularity. VenusBench-GD addresses these gaps with a six-task taxonomy, a multi-stage data pipeline, broad benchmark coverage, and annotation-quality analysis.
2.1. GUI Grounding
- GUI grounding maps natural-language commands to interface elements, linking high-level planning to low-level actions.Earlier models used supervised fine-tuning, while more recent approaches use reinforcement learning with continuous or spatial reward signals.Source: §2.1 · p. 3
2.2. GUI Benchmarks
- Prior benchmarks vary in scale and scope, with some limited to one platform or specialized applications and MMBench-GUI offering only about 3,500 element-grounding samples.The paper also criticizes MMBench-GUI’s coarse basic/advanced taxonomy and lack of annotation-quality checks and error analysis.Source: §2.2 · p. 3
3.1. Tasks Design
- The taxonomy separates basic element, spatial, and visual grounding from advanced reasoning, functional, and refusal grounding.Refusal grounding tests whether a model refrains from predicting a box for a nonexistent element.Source: §3.1 · p. 4
3.2. Bottom-up Data Pipeline
- The four-stage pipeline combines screenshot collection, UI-element localization, hierarchical instruction generation, and filtering for high-quality, non-trivial samples.Element boxes receive at least two review rounds; advanced-task generation includes perturbed infeasible instructions, reasoning scenarios, and pre-/after-click screenshots for functional grounding.Source: §3.2 · pp. 4–6
3.3. Data Statistics
- VenusBench-GD spans major platforms and has 6,166 samples, 65 screen sizes, broad resolution and element-area variation, and the longest average instructions among six benchmarks.The authors present these statistics as indicators of robustness challenges and linguistic diversity.Source: §3.3 · p. 6
3.4. Data Quality Analysis
- VenusBench-GD has the lowest error rate across the three annotation-error types at 2.6%, reducing inaccurate localization from CAGUI’s 11.6% to 1.0%.Inaccurate localization is the predominant source of annotation errors across all evaluated benchmarks.Source: §3.4 · p. 6
4. Experiments
The experiments compare open- and closed-source VLMs and GUI agents across grounding tasks, then analyze reasoning-enabled and multilingual performance. General models are competitive on basic tasks, while specialized models retain advantages on advanced tasks but show weak refusal robustness.
4.1. Experiment Settings
- The evaluation covers open- and closed-source VLMs and GUI-specific models, with correctness based on point-in-box localization or prescribed refusal-format adherence.Accuracy is averaged over UI elements; refusal is correct only when the output follows the required format.Source: §4.1 · pp. 6–7
4.2. Benchmark Results
- General VLMs match or surpass GUI-specialized models on basic grounding, while specialized models lead on advanced reasoning and functional tasks but often fail refusal grounding.Most specialized models show almost no ability to abstain from invalid queries; UI-Venus-Ground-72B reaches 51.33% accuracy on Refusal Grounding.Source: §4.2 · p. 7
4.3. Analysis
- Qwen3-VL-8B-Thinking scores 46.6%/56.7% on ScreenSpot-Pro and OSWorld-G, below Qwen3-VL-8B-Instruct at 54.6%/58.2%.The analysis says additional reasoning improves performance only in more complex scenarios.Source: §4.3 · p. 7
- Qwen3-VL-4B improves from 72.54 to 81.32 in basic understanding under Chinese instruction, while advanced-task performance fluctuates, possibly due to instruction ambiguity.The analysis reports that the model typically scores higher on basic tasks in Chinese than English.Source: §4.3 · p. 7
5. Conclusion
VenusBench-GD combines broad application coverage with six hierarchical grounding tasks and a human-AI collaborative annotation pipeline. The authors present it as a multidimensional framework for evaluating GUI agents across languages and task complexity.
5. Conclusion
- The benchmark’s six hierarchical grounding tasks range from basic localization to complex reasoning and refusal scenarios.The conclusion describes this as a rigorous multidimensional evaluation framework.Source: §5 · p. 8
- VenusBench-GD spans 97 real-world applications across 10 domains.The conclusion also describes the benchmark as spanning web, mobile, and desktop platforms.Source: §5 · pp. 7–8
- A human-AI collaborative annotation pipeline is presented as ensuring minimal label noise and supporting data quality and cross-lingual relevance.The conclusion links these properties to meaningful differentiation and valid evaluation of state-of-the-art GUI agents.Source: §5 · p. 8
6. Data Privacy
The benchmark screenshots come from freshly created synthetic accounts, and user-facing content was fabricated. The authors also report collecting interface captures under platform terms of service for non-commercial research.
6. Data Privacy
- Screenshots use freshly created synthetic accounts, with fabricated names, messages, and profile details; the benchmark therefore contains no personally identifiable information.The authors report that it poses no privacy or compliance risks.Source: §6 · p. 11
- Application interfaces were accessed and captured under the respective platforms’ terms of service for non-commercial research use.The authors describe this controlled data-generation protocol as adhering to responsible-AI dataset publication ethics.Source: §6 · p. 11
7. Dataset Statistics
The dataset statistics describe application coverage across 10 domains and summarize variation in screen resolution, UI element size, and instruction length.
7. Dataset Statistics
- The application set spans 10 domains, including creative, deployment, e-commerce, entertainment, finance, knowledge, productivity, social, travel, and utility.The passage points to Table 7 for the application distribution.Source: §7 · p. 11
- The statistics describe distributions of image resolution, UI element size, and instruction length, with diversity in each dimension by design.Figure 6 is identified as showing these distributions.Source: §7 · p. 11
8. Benchmark Tasks
VenusBench-GD combines dense UI-element annotation with task-specific pipelines for basic and advanced grounding. Its construction includes iterative quality control, model-assisted instruction creation, and difficulty-based sampling.
8. Benchmark Tasks
- The dataset exhaustively labels approximately 20 UI elements per screenshot, with bounding boxes tightly enclosing each element’s interactive region.Semantically similar, commonly co-occurring types are merged, such as notification/dialog and checkbox/radiobox.Source: §8 · p. 11
- Validation accuracy rose from 84.54% to 95.47% to 98.25% across three refinement rounds, with independent reviewers checking a random subset of approximately 15% of annotations.Source: §8 · p. 11
- Basic grounding covers element, spatial, and visual tasks; instructions use target category, text, appearance, and spatial context, followed by human validation.Source: §8 · pp. 11–12
- Advanced tasks use domain-template reasoning annotations, attribute-perturbed ungroundable refusal instructions, and paired pre-click and after-click screenshots for functional instructions.Source: §8 · pp. 12–13
- A model-agreement filter removes instances all tested models solve and flags shared wrong answers for annotator review.Source: §8 · p. 13
- Instances are classified by model accuracy as difficult (≤20%), medium (40–60%), or easy (all others), then sampled in a 4:4:2 ratio.Source: §8 · p. 14
9. Experimental Analsysis
Experiments compare reasoning-enabled and instruction-tuned models, language-dependent performance, and human performance. Results show different patterns across basic and advanced grounding tasks.
9. Experimental Analsysis
- The thinking model improves reasoning grounding by 22.8% and functional grounding by 12.2% over the instruct model, while performing slightly worse on basic tasks.Source: §9 · p. 14, Figure 9 · p. 14
- The thinking model’s simpler-task performance may be affected by redundant or irrelevant outputs and sensitivity to bounding-box annotation quality.Source: §9 · p. 14
- Language-dependent results reverse the English-versus-Chinese ordering of OpenCUA-7B and UI-TARS-7B.Source: §9 · p. 14, Table 11 · p. 15
- The passage reports that, on advanced tasks, models exceeded SOTA by 41.6% in reasoning, 11.8% in functional, and 17.8% in refusal grounding, respectively.Source: §9 · p. 15
10. Error Analaysis
The error analysis identifies failures in matching semantic meaning, locating targets precisely, composing visual attributes, reasoning through multiple steps, and mapping abstract functions to UI affordances.
10. Error Analaysis
- Element grounding fails when identity depends on abstract concepts rather than literal text, and OCR struggles with small or stylized text.Source: §10 · p. 15
- In cluttered UI regions, models can identify a coarse area yet confuse the target with adjacent distractors, failing to combine what and where.Source: §10 · p. 15
- Visual grounding errors arise when models attend to one salient feature but ignore other instructed attributes such as color, shape, or relation.Source: §10 · p. 15
- Reasoning grounding exposes failures in multi-step comparison, filtering, or ordinal selection, such as choosing the highest-priced item instead of the second-highest.Source: §10 · pp. 15–16
- For abstract functional commands such as “Enter Fullscreen,” models may search for literal text instead of identifying a nonliteral UI affordance.Source: §10 · p. 16
F. Failure in Contradiction Detection and Task Re-
The analysis describes models’ over-compliance with impossible or contradictory instructions, while the annotation workflow combines human labeling, task-specific pipelines, and model assistance.
F. Failure in Contradiction Detection and Task Re-
- Refusal grounding fails when models follow factually impossible or visually contradictory instructions rather than rejecting them.The authors suggest contrastive examples or a preliminary check of the instruction’s premises against the screenshot.Source: F. Failure in Contradiction Detection… · p. 16
11. Human Annotation
- The three-month annotation workflow averaged 20 labeled elements per image and used task-specific pipelines, at least two annotation rounds per subtask, quality checks, and language models.The team included 17 primary annotators, two quality-assurance specialists, and one coordinator, with approximately two weeks of training.Source: §11 · p. 16