AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark

Hongxin Li; Xiping Wang; Jingran Su; Zheng Ju; Yuntao Chen; Qing Li; Zhaoxiang Zhang

Source-linked AI summary

AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark

Hongxin Li, Xiping Wang, Jingran Su, Zheng Ju, Yuntao Chen, Qing Li, Zhaoxiang Zhang

arXiv:2604.24441v1Apr 27, 2026cs.CV

TL;DR

Existing GUI benchmarks provide limited evidence about whether agents understand implicit functionality, transition logic, and resulting interface states. AutoGUI-v2 builds a multi-platform benchmark through recursive VLM-human annotation and finds a grounding–captioning capability split alongside broad difficulty with complex interaction logic.

Problem
Existing benchmarks largely bifurcate between black-box task completion and shallow grounding, failing to assess deep GUI functionality and transition understanding.
Method
AutoGUI-v2 recursively parses multi-platform screenshots into hierarchical functional regions, verifies and refines annotations, and generates region- and element-level functionality tasks.
Results
Open-source models outperform commercial models on functionality-oriented grounding, while commercial models dominate functionality captioning; complex interactions and plausible distractors remain difficult.
Takeaways & Limitations
AutoGUI-v2 offers a systematic way to measure GUI functionality understanding, dynamics, and state-transition comprehension beyond simple grounding.
Takeaways & Limitations
The benchmark still requires human bounding-box correction, covers only single-step or next-step outcomes, and does not analyze how functionality understanding relates to multi-step planning.

Abstract

from arXiv · show

Autonomous agents capable of navigating Graphical User Interfaces (GUIs) hold the potential to revolutionize digital productivity. However, achieving true digital autonomy extends beyond reactive element matching; it necessitates a predictive mental model of interface dynamics and the ability to foresee the "digital world state" resulting from interactions. Despite the perceptual capabilities of modern Vision-Language Models (VLMs), existing benchmarks remain bifurcated (focusing either on black-box task completion or static, shallow grounding), thereby failing to assess whether agents truly comprehend the implicit functionality and transition logic of GUIs. To bridge this gap, we introduce AutoGUI-v2, a comprehensive benchmark designed to evaluate deep GUI functionality understanding and interaction outcome prediction. We construct the benchmark using a novel VLM-human collaborative pipeline that recursively parses multi-platform screenshots into hierarchical functional regions to generate diverse evaluation tasks. Providing 2,753 tasks across six operating systems, AutoGUI-v2 rigorously tests agents on region and element-level semantics, grounding, and dynamic state prediction. Our evaluation reveals a striking dichotomy in VLMs: while open-source models fine-tuned on agent data (e.g., Qwen3-VL) excel at functional grounding, commercial models (e.g., Gemini-2.5-Pro-Thinking) dominate in functionality captioning. Crucially, all models struggle with complex interaction logic of uncommon actions, highlighting that deep functional understanding remains a significant hurdle. By systematically measuring these foundational capabilities, AutoGUI-v2 offers a new lens for advancing the next generation of GUI agents.

1. Introduction

Existing GUI benchmarks largely assess either black-box task completion or shallow grounding, leaving deeper functionality and transition reasoning under-evaluated. AutoGUI-v2 addresses this gap with hierarchical functional annotations and reveals distinct, persistent weaknesses across VLMs.

Existing benchmarks either measure task success without diagnosing functionality interpretation or evaluate simplistic element localization without transition logic and GUI context.
AutoGUI-v2 recursively decomposes multi-platform screenshots into hierarchical functional regions, verifies and manually refines them, and generates region- and element-level functionality tasks.The benchmark combines VLM-based region parsing, scoring, manual annotation, and element-level grounding assistance.
2,753 evaluation tasks across six operating systems test functional semantics, grounding, and interaction-outcome prediction.
Open-source models outperform commercial models on functionality-oriented grounding, whereas commercial models dominate functionality captioning.The comparison separates localizing where a function is from reasoning about what a region or element does.
Performance drops for irregular regions and complex interactions, while plausible visual distractors expose failures in context-aware functionality understanding.
The benchmark extends GUI evaluation beyond reactive element grounding toward comprehension of GUI dynamics and state transitions.

2. Related Works

GUI benchmarks evolved from grounding to offline and online agent evaluation, but commonly omit interaction-induced state changes. AutoGUI-v2 is positioned as a broader benchmark for functionality understanding across GUI regions and elements.

Grounding benchmarks evaluate localization from natural-language instructions, progressing from ScreenSpot to higher-resolution and multi-application datasets.
AutoGUI-v2 provides rich functional semantics for both GUI elements and regions compared with existing benchmarks.
Offline and online benchmarks assess agents on static snapshots or dynamic environments, respectively, across web, mobile, desktop, and cross-platform settings.
Existing benchmarks typically overlook whether agents understand interaction-induced state changes and essential GUI dynamics.
GUI agent architectures have shifted from API-dependent systems toward scalable end-to-end visual methods, later augmented by reinforcement learning.

3. AutoGUI-v2 Construction

AutoGUI-v2 constructs hierarchical functional-region annotations and uses them to generate region- and element-level tasks that test functionality beyond visual appearance. Its human-in-the-loop pipeline combines VLM proposals, verification, recursive division, manual correction, and similarity-based task construction across diverse GUI sources.

Definitions: AutoGUI-v2 defines functional regions as hierarchical groupings of UI elements that collectively serve a unified purpose.Parent regions are recursively decomposed into granular child units.
Data Collection: The benchmark aggregates screenshots from desktop, web, and mobile sources to provide broad platform and interface coverage.Sources include ScreenSpot-Pro, OSWorld-G, MMBenchGUI, AgentNet, and AMEX.
Hierarchical Region Annotation: Gemini-2.5-Pro-Thinking proposes functional regions with contextual functionality, layout descriptions, normalized boxes, and divisibility flags, then recursively divides regions marked divisible.The process begins from a region image and continues until regions are no longer dividable.
Hierarchical Region Annotation: A separate VLM checks cropped regions for completeness and boundedness, accepting proposals at average score >= 0.8 or regenerating them for up to 3 iterations.Human annotators subsequently correct bounding boxes and validate re-annotations against contextuality, high-level purpose, and non-hallucination criteria.
Task Generation: Similarity-based grouping generates functionality-oriented grounding tasks, while element-level construction targets visually similar components with distinct functions.Region groups use description embeddings; element groups use visual embeddings, textual filtering, and clustering, forcing contextual rather than appearance-only reasoning.
Task Generation: 3,710 hierarchical functional regions yield 2,753 evaluation tasks, including 889 region-level and 1,864 element-level tasks.The benchmark spans multiple platforms and resolutions and tests diverse functional semantics.

4. Experiments

AutoGUI-v2 evaluates region- and element-level GUI functionality understanding through grounding, captioning, and interaction-focused analyses. Results show a capability split between open-source models’ grounding strengths and commercial models’ captioning strengths, while complex, irregular, and distractor-rich cases remain difficult.

Region-Level Functionality Understanding: Functionality-based grounding consistently underperformed description- or intent-based grounding, indicating that abstract functional localization is harder than matching explicit visual attributes.UGround-V1-7B dropped from 88.5% to 64.7%, while OS-Atlas-Base-7B dropped from 72.4% to 56.3%.
Region-Level Functionality Understanding: Irregular regions generally reduced performance by at least 5% relative to the best-performing region types, while dense contextual surroundings improved grounding for some models and sparse inputs favored others.Commercial and GUI-specialized models generally preferred Dense settings, whereas some general-purpose open-source models performed best in Sparse settings.
Region-Level Functionality Understanding: Hard negatives produced higher error rates than Easy negatives across models, showing that plausible functional distractors expose gaps in context-aware discrimination.GPT-5’s Hard error rate was 9.1%, compared with 2.8% for Easy samples.
Element-Level Functionality Understanding: Element-level grounding also favored open-source and GUI-oriented models, but accuracy generally increased with surrounding-element density, suggesting that functional groups provide useful context.Qwen3-VL-32B-Instruct was the top element-grounding performer, although domain-specific fine-tuning remained insufficient for complete benchmark mastery.
Region-Level Functionality Understanding: Commercial VLMs dominated functionality captioning, contrasting with open-source models’ grounding advantage and revealing different strengths in localizing functions versus explaining what regions or elements do.The captioning results associate commercial models’ advantage with broad world knowledge, whereas GUI fine-tuning improves localization.
Element-Level Functionality Understanding: Captioning performance depended on action type: models handled overt state changes such as Type and Long-Press better than subtle Hover interactions.Hover offers weaker visual feedback, such as a tooltip, for learning the associated function.
Case Visualization: Qualitative analysis found strong performance on common elements but failures on abstract components and dynamic, stateful controls, including status bars, specialized icons, control bars, and collapsible-folder chevrons.These failures indicate limited robustness in understanding complex GUI dynamics.

5. Conclusion

AutoGUI-v2 evaluates context-aware GUI functionality understanding at region and element levels, revealing that complex interactions and GUI dynamics remain highly challenging for current models.

AutoGUI-v2 evaluates context-aware GUI functionality understanding at both region and element levels.
The benchmark reveals a grounding-reasoning divergence among leading VLMs.
Current models continue to struggle with understanding complex interactions and GUI dynamics.

6. Implementation Details of AutoGUI-v2 Datasets

AutoGUI-v2 datasets are built from diverse GUI sources and processed through a hierarchical, VLM-human collaborative annotation pipeline. Automated proposals are recursively verified and refined to produce structured drafts for subsequent correction.

6.1. GUI Data Sources: AMEX provides 104K high-resolution screenshots from 110 popular mobile applications for GUI data construction.
6.1. GUI Data Sources: The pipeline uses selected screenshots and detected similarity groups to support region- and element-level task generation.
6.2. Implementation Details: VLM outputs are processed because generated bounding boxes may be shifted, incomplete, or excessively large.
6.2. Implementation Details: A recursive propose-verify-refine loop hierarchically decomposes GUI screenshots into functional regions.
6.2. Implementation Details: Regions marked divisible and larger than 30 pixels in both width and height are recursively processed until termination conditions are met.
6.2. Implementation Details: Stage 1 produces a verified hierarchical draft that reduces the cost and cognitive load of later manual correction and re-annotation.

1. Completeness (scomp

The completeness measure evaluates whether each proposed child region is fully visible and functionally coherent within its parent region.

The checking model scores each cropped child region from 0–3 for full visibility and functional coherence in its parent context.

2. Boundedness (sbound

The boundedness stage accepts proposals using geometric quality thresholds, then applies human correction and semantic re-annotation to produce precise, provenance-preserving functional-region annotations.

2. Boundedness: A proposal is accepted when its average completeness score and proportion of tightly bounded children meet their respective thresholds.
2. Boundedness: Failed proposals trigger up to three regenerated attempts, after which the highest-scoring proposal is retained.
6.4. Functional Region Re-Annotation Details: Because automated bounding boxes lack pixel-perfect precision, Stage 2 uses a custom web application for human-in-the-loop correction.
6.4. Functional Region Re-Annotation Details: The correction interface provides full-image context, hierarchy navigation, zoomed previews, and server-backed saving of revised coordinates.
6.4. Functional Region Re-Annotation Details: Corrections are saved non-destructively with updated bounding-box fields and timestamped fix files, preserving the original metadata.
6.4.4. Functionality Re-Annotation: After geometry correction, Stage 3 prompts Gemini-2.5-Pro-Thinking to revise functionality and descriptions so semantics align with the corrected boxes.
6.4.4. Functionality Re-Annotation: Re-annotation outputs are stored as separate artifacts rather than overwriting existing files, preserving data provenance.
6.4.4. Functionality Re-Annotation: The final benchmark data combines human-verified geometry with VLM-revised semantics and standardized region taxonomy.

6.6. More Dataset Statistics

AutoGUI-v2 combines diverse functional-region statistics with visually similar, functionally distinct groups to construct challenging grounding and captioning tasks.

Cost Analysis: $1.072 is the estimated total cost of processing one 1920 × 1080 screenshot.The estimate combines functional-region division, regional refinement, and re-annotation costs.
Region Statistics: 54.3% of detected regions are Global Navigation & Structure types, while Content & Data Display accounts for 11.1%.The listed region categories also include Interaction & Input, Contextual & Temporary Regions, purely static content, individual elements, and system artifacts.
Task Statistics: Figures 7–10 characterize task attributes by action type, density class, and similarity-group size for grounding and captioning tasks.These statistics cover both functionality-based element grounding and functionality-based element captioning.
Similarity Groups: Visually similar but functionally distinct regions are grouped through semantic embeddings, VLM verification, and deterministic geometric refinement.The process excludes parent-child pairs, resolves overlaps, enforces group sizes of 2–5, merges duplicates, and applies a subtractive second VLM pass when needed.
Similarity Groups: Final validation retains only groups with no bounding-box overlaps and strictly 2–5 elements.Oversized or merged groups are processed again before the final Python-based validation.

7. Extra Evaluation Details

Additional inspections expose distinct failure modes across grounding and captioning: models trade semantic identification against localization precision and struggle with GUI hierarchy and dynamic outcomes.

Region Grounding: Gemini-2.5-Pro-Thinking often identifies the correct target but predicts loose or shifted boxes, whereas Qwen3-VL-32B-Instruct produces tighter boxes but often selects the wrong element.The comparison indicates a trade-off between semantic reasoning and localization precision.
Region Captioning: Both Gemini-2.5-Pro-Thinking and Qwen3-VL-32B-Instruct commonly confuse System vs. Application layers and functionalities within a single application.Qwen additionally shows sporadic System vs. System and Inter-App confusions.
Element Grounding: 90% of Gemini-2.5-Pro-Thinking’s inspected element-grounding failures arise from incorrect reasoning, compared with 10% from misunderstanding functionality.The inspection covered 30 cases in which the model could often identify the target in reasoning but failed to predict bounding-box coordinates.
Element Captioning: Gemini-2.5-Pro-Thinking incorrectly predicts that clicking a system-tray app icon returns to the previous screen instead of minimizing the entire app window.The model also confuses alignment buttons in Linux spreadsheet software, illustrating failures on dynamic and closely related controls.
Element Captioning: The inspected failure cases include misunderstanding functionality in an AutoCAD close-button question.The question asks where hovering reveals a button that turns red to confirm exiting the application.

8. Prompt Suites

The prompt suites operationalize hierarchical region annotation, quality assessment, re-annotation, and similarity-based task generation through structured outputs and explicit constraints.

8.1. AutoGUI-v2 Annotating Prompts: Each annotation output specifies an id, bounding box, divisibility, region type, visual description, and contextual functionality.The format allows up to 10 regions and defines dividable=true as permitting decomposition into smaller self-contained functional regions.
8.1. AutoGUI-v2 Annotating Prompts: The examples distinguish whole-GUI, application-window, system-bar, dock, and dialog functionality across Ubuntu, Visual Studio Code, Excel, and YouTube interfaces.The examples pair layout descriptions with operational functionality descriptions.
8.1. AutoGUI-v2 Annotating Prompts: Quality-assessment prompts score cropped regions for completeness and boundedness using a 0–3 completeness scale and a Yes/No boundedness label.The examples assess occlusion, cropping, visibility, and framing of GUI regions.
8.1. AutoGUI-v2 Annotating Prompts: Re-annotation prompts revise region functionality and layout descriptions when visual evidence conflicts with previous annotations.The VLM is told that a human corrected the bounding box and should ignore conflicting prior text.

9. Societal Impact

The paper discusses accessibility benefits alongside labor, surveillance, security, and representation risks associated with increasingly capable GUI agents.

Labor and Surveillance: GUI agents may reduce cognitive load but could also facilitate workplace surveillance or displace data-entry and QA roles.The authors advocate human-in-the-loop deployment, with agents serving as augmenting copilots rather than autonomous replacements.
Data Ethics and Bias: The dataset is biased toward English-language, Western-centric design patterns, limiting cross-cultural generalization and universal applicability without diverse testing.The authors state that the dataset uses open-source repositories and public interfaces scrubbed for offensive content.

10. Limitations and Future Work

AutoGUI-v2 remains partly dependent on human annotation and evaluates functionality mainly through single-step outcomes. Its relationship to longer-horizon planning in multi-platform interactions is also left for future study.

Annotation scalability: Human annotators must correct detected functional-region bounding boxes, limiting the scalability of the annotation pipeline.The authors propose reliable self-verification with model assistance as a possible replacement for human labor.
Task relevance: Functionality descriptions are tied to single-step instructions, so interaction outcomes cover only the next step rather than longer sequences.This constrains the benchmark’s task relevance for multi-step interaction.
Future work: The study does not analyze how functionality understanding correlates with ultimate planning ability in multi-platform interaction scenarios.The authors identify this relationship as an essential topic for future work.