PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark

Hongxin Li, Xiping Wang, Jingran Su, Zheng Ju, Yuntao Chen, Qing Li, Zhaoxiang Zhang

arXiv:2604.24441v1cs.CV

TL;DR

Existing GUI benchmarks largely assess black-box task completion or shallow element grounding, leaving deeper functionality and transition understanding underexamined. AutoGUI-v2 builds a multi-platform benchmark for region- and element-level functionality, revealing a grounding-reasoning divergence and persistent difficulty with complex GUI interactions.

  • Problem

    Existing GUI benchmarks largely assess black-box completion or simplistic element localization, leaving GUI functionality, context, and transition logic insufficiently evaluated.

    Source: §1 · p. 2

  • Method

    AutoGUI-v2 uses a VLM-human collaborative pipeline to annotate hierarchical functional regions and generate tasks for grounding and interaction-outcome prediction.

    Source: §1 · p. 2

  • Results

    Models show a grounding-reasoning divergence: open-source models excel at functionality-oriented grounding, while commercial models dominate functionality captioning, and complex interactions remain challenging.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    AutoGUI-v2 provides a benchmark for measuring context-aware GUI functionality understanding and highlights deep GUI dynamics as a continuing challenge for current models.

    Source: §5 · p. 9

    The benchmark’s functionality descriptions are tied to single-step instructions, so interaction outcomes cover only the next step rather than longer task sequences.

    Source: §10 · p. 47

Abstract

from arXiv · show

Autonomous agents capable of navigating Graphical User Interfaces (GUIs) hold the potential to revolutionize digital productivity. However, achieving true digital autonomy extends beyond reactive element matching; it necessitates a predictive mental model of interface dynamics and the ability to foresee the "digital world state" resulting from interactions. Despite the perceptual capabilities of modern Vision-Language Models (VLMs), existing benchmarks remain bifurcated (focusing either on black-box task completion or static, shallow grounding), thereby failing to assess whether agents truly comprehend the implicit functionality and transition logic of GUIs. To bridge this gap, we introduce AutoGUI-v2, a comprehensive benchmark designed to evaluate deep GUI functionality understanding and interaction outcome prediction. We construct the benchmark using a novel VLM-human collaborative pipeline that recursively parses multi-platform screenshots into hierarchical functional regions to generate diverse evaluation tasks. Providing 2,753 tasks across six operating systems, AutoGUI-v2 rigorously tests agents on region and element-level semantics, grounding, and dynamic state prediction. Our evaluation reveals a striking dichotomy in VLMs: while open-source models fine-tuned on agent data (e.g., Qwen3-VL) excel at functional grounding, commercial models (e.g., Gemini-2.5-Pro-Thinking) dominate in functionality captioning. Crucially, all models struggle with complex interaction logic of uncommon actions, highlighting that deep functional understanding remains a significant hurdle. By systematically measuring these foundational capabilities, AutoGUI-v2 offers a new lens for advancing the next generation of GUI agents.

1. Introduction

AutoGUI-v2 addresses the gap between black-box task completion and shallow grounding benchmarks by evaluating deep GUI functionality understanding and interaction-outcome prediction. Its 2,753 tasks across six operating systems reveal divergent VLM strengths and persistent failures on implicit, complex GUI logic.

Motivation

  • Effective GUI autonomy requires more than reactive element grounding: agents must model implicit region functionality, interface dynamics, and the digital-world state produced by actions.This predictive reasoning is presented as necessary for generalization across diverse scenarios.Source: §1 · p. 2
  • Existing benchmarks either assess task success as a black box or simplify grounding to element localization, leaving transition logic, GUI context, and interaction outcomes largely untested.AutoGUI [15] adds functionality understanding but remains limited in platform diversity and transition-dynamics coverage.Source: §1 · p. 2

Benchmark contribution

  • AutoGUI-v2 evaluates both region- and element-level GUI functionality, including target localization and interaction-outcome prediction, across 2,753 tasks and six operating systems.A VLM-human pipeline recursively parses screenshots into hierarchical functional regions, while OmniParser-v2 assists element-level grounding and captioning task generation.Source: §1 · p. 2

Evaluation findings

  • Open-source models such as Qwen3-VL outperform commercial models such as Gemini-2.5-Pro-Thinking on functionality-oriented grounding, whereas commercial models dominate functionality captioning.The former concerns locating where a function is, while the latter concerns reasoning about what a region or element does.Source: §1 · p. 2
  • Models perform poorly on irregular regions, complex interactions, and visually plausible functional distractors, indicating inadequate context-aware understanding of implicit functionality.These failures suggest reliance on overt visual cues rather than robust reasoning about GUI functionality and state transitions.Source: §1 · p. 2

2. Related Works

GUI benchmarks evolved from grounding evaluation to offline and online agent assessment, but generally overlooked interaction-induced state changes. Multimodal GUI agents likewise progressed toward scalable visual and reinforcement-learning systems while evaluations remained focused on task execution rather than GUI dynamics.

2.1. GUI Benchmarks

  • GUI benchmarks progressed from element grounding to offline snapshot evaluation and online dynamic-environment evaluation, culminating in AutoGUI-v2’s comprehensive functionality-understanding focus.Grounding began with ScreenSpot [6] and expanded through ScreenSpot Pro [17], UI-Vision [30], and OSWorld-G; later paradigms included Mind2Web [8], GUI Odyssey, GUICourse, OSWorld, and related platforms.Source: §2.1 · pp. 2–3
  • Existing benchmarks typically omit interaction-induced state changes, while AutoGUI-v2 evaluates GUI functionality with region-level granularity and sufficient localization context.This distinguishes AutoGUI-v2 from GUI Knowledge Bench, which considers interaction outcomes but lacks region-level granularity and sufficient context for element localization.Source: §2.1 · p. 3

2.2. Multi-Modal GUI Agents

  • Multimodal GUI agents shifted from API-dependent systems to scalable end-to-end visual methods, with reinforcement learning further improving generalization and reducing data reliance in complex environments.Examples span AppAgent [62], SheetCopilot [14], and OmniParser through CogAgent [12], ShowUI [20], UGround [10], SpiritSight [13], Aguvis, Jedi, OS-Atlas, GUI-Xplore [37], and UI-Tars, alongside reinforcement-learning paradigms [23] [28].Source: §2.2 · p. 3
  • Despite architectural advances, agent evaluations remain centered on task execution rates rather than foundational GUI dynamics and digital state prediction.AutoGUI-v2 addresses this evaluation gap by probing underlying digital state prediction useful for agent ability.Source: §2.2 · p. 3

3. AutoGUI-v2 Construction

AutoGUI-v2 constructs a hierarchical, human-validated benchmark from multi-source GUI screenshots, generating region- and element-level tasks that test functionality-aware grounding and interaction-outcome prediction. The pipeline combines recursive annotation, similarity-based grouping, VLM verification, and specialized task construction across diverse platforms and interfaces.

3.1. Definitions

  • Functional regions hierarchically group UI elements serving a unified purpose, while visually similar elements with distinct functions require contextual rather than appearance-only reasoning.Parent regions are recursively decomposed into granular child units, and element-level evaluation targets visually analogous but functionally distinct components.Source: §3.1 · pp. 3–4

3.2. Data Sources

  • Screenshots are aggregated from ScreenSpot-Pro [17], OSWorld-G, MMBenchGUI, AgentNet [49], and AMEX to cover desktop, web, and mobile domains.The multi-source collection is designed to provide robust coverage across platforms and interface types.Source: §3.2 · p. 4

3.3.1. Hierarchical Region Annotation Pipeline

  • A three-stage human-in-the-loop pipeline generates hierarchical region annotations through automated VLM proposals, manual bounding-box correction, and functionality re-annotation with human validation.Proposals recursively divide regions, accept average refinement scores of at least 0.8, and regenerate up to three times otherwise; corrected crops are re-annotated and checked for contextuality, high-level purpose, and non-hallucination.Source: §3.3 · p. 4, §3.3.1 · p. 4

3.3.2. Task Generation

  • Region-level tasks cluster description embeddings, verify and refine groups in screenshot context, then create functionality-oriented grounding and captioning questions with hard and easy negatives.Grounding asks agents to localize regions from functional requirements, while captioning asks them to predict interaction outcomes with randomized answer positions.Source: §3.3.2 · pp. 4–5

3.4. Element-Level Functionality Understanding

  • The element-level pipeline detects interactive components with OmniParser-v2, groups them using DINO-v3 visual embeddings, filters candidates by textual similarity, and constructs analogous functional tasks.Disjoint Set Union consolidates clusters, while element-specific interactions such as double-click and long-press distinguish visually similar hard negatives.Source: §3.4 · p. 5

3.5. Dataset Statistics

  • The pipeline yields 3,710 hierarchical functional regions and 2,753 evaluation tasks: 889 region-level and 1,864 element-level, spanning multiple platforms and resolutions.These tasks test diverse functional semantics through region- and element-level benchmark evaluations.Source: §3.5 · p. 5

4. Experiments

AutoGUI-v2 evaluates VLMs on region- and element-level GUI functionality understanding, grounding, captioning, and interaction-outcome prediction. Results show a split between open-source models’ grounding strength and commercial models’ captioning strength, while complex, implicit, and stateful interactions remain difficult.

4.1. Experimental Settings

  • The evaluation measures grounding with IoU and Grounding Center Accuracy, captioning with multi-choice accuracy, and visual interference with NID-based Sparse, Medium, and Dense groups.NID counts neighboring element centers inside an expanded target-centered analysis region, enabling robustness analysis under different clutter levels.Source: §4.1 · p. 5

4.2.1. Region-Level Functionality Understanding

  • Open-source VLMs lead functionality grounding, with GLM-4.5V at 84.6% and Qwen3-VL-32B-Instruct at 84.4%, while Gemini-2.5-Pro-Thinking reaches 77.6% and GPT-5 31.4%.Functionality grounding is consistently harder than description grounding, including UGround-V1-7B’s drop from 88.5% to 64.7% and OS-Atlas-Base-7B’s drop from 72.4% to 56.3%.Source: Table 3 · p. 6, §4.2.1 · p. 6

4.2.2. Element-Level Functionality Understanding

  • Commercial VLMs dominate functionality captioning, whereas open-source and GUI-specialized models lead element grounding, confirming a divergence between localization and abstract purpose description.Element functionality grounding is harder than appearance- or intent-based grounding, and Qwen3-VL-32B-Instruct is the top grounding performer among the evaluated models.Source: Table 4 · p. 6, Table 5 · p. 6, §4.2.2 · p. 8
  • Complex or implicit actions remain difficult: element grounding is strongest for overt actions such as Type and Long-Press but collapses for Right-Click and Drag, while Hover is a captioning failure point.Grounding generally improves with higher NID, suggesting that surrounding functional context can disambiguate elements, while hard negatives produce higher error rates than easy negatives.Source: Table 6 · p. 7, Table 7 · p. 7, §4.2.2 · p. 8

4.2.3. Case Visualization

  • Qualitative cases show Gemini-2.5-Pro-Thinking handles common GUI targets and navigational outcomes but fails on abstract, dynamic, stateful, and specialized controls.Examples include long status bars, tiny productivity-software icons, application control bars, and collapsible-folder chevrons.Source: Figure 5 · p. 8, §4.2.3 · p. 9

5. Conclusion

AutoGUI-v2 benchmarks context-aware GUI functionality understanding at region and element levels, revealing a grounding–reasoning divergence and persistent difficulty with complex interactions and GUI dynamics.

Conclusion

  • AutoGUI-v2 evaluates context-aware GUI functionality understanding at both region and element levels while benchmarking leading VLMs.The benchmark is intended to support analysis of GUI agents’ functional understanding.Source: §5 · p. 9
  • Results reveal a grounding-reasoning divergence and show that current models still struggle with complex interactions and GUI dynamics.These findings motivate future efforts to improve GUI agents.Source: §5 · p. 9

Benchmark · 6. Implementation Details of AutoGUI-v2 Datasets

AutoGUI-v2 builds its datasets from diverse GUI benchmarks and generates hierarchical functional-region annotations through a recursive, verified VLM pipeline. The process produces auditable drafts that support later human correction while controlling annotation cost and geometric imprecision.

6.1. GUI Data Sources

  • The benchmark draws screenshots and tasks from mobile, desktop, web, and cross-platform GUI sources, using filtering or sampling to select inputs for functional-region and task generation.Sources include AMEX, AndroidControl, ScreenSpot-Pro, AgentNet, MMBenchGUI, and OSWorld-G3, with source-specific screenshot selection procedures.Source: §6.1 · p. 13

6.2. Functional Region Division Details

  • The annotation pipeline addresses VLM geometric imprecision through a multi-stage, recursive propose-verify-refine process orchestrated by a FunctionalRegionAnnotator.The pipeline is designed to mitigate shifted, incomplete, or oversized bounding boxes rather than accepting raw VLM outputs directly.Source: §6.2 · p. 13

6.2.1. Algorithm: Recursive “Divide-and-Verify”

  • The divide-and-verify algorithm starts from the full screenshot and recursively proposes child regions with bounding boxes, functionality, descriptions, and divisibility flags.Nodes are processed using a depth-first traversal implemented by the annotate_image function.Source: §6.2.1 · p. 13
  • Each region proposal undergoes iterative refinement controlled by max_refine instead of being accepted after the VLM’s first output.This loop searches for an optimal division of the current region.Source: §6.2.1 · pp. 13–14

6.2.2. Automated Quality Verification

  • A separate high-speed checking model evaluates each proposal using the CHECK_REGION_COMPLETENESS_PROMPT scoring procedure.The verification stage assesses proposed child regions before a best proposal is selected.Source: §6.2.2 · p. 14, §6.3 · p. 14

6.2.3. Recursion, Termination, and Caching

  • Recursion continues for dividable regions wider and taller than 30 pixels, while branches terminate at non-dividable, undersized, or maximum-depth regions.Metadata, cropped images, and raw VLM responses are cached to create an auditable hierarchy for the later interactive correction interface.Source: §6.2.3 · p. 14
  • Stage 1 produces a high-quality, verified hierarchical draft that reduces the cost and cognitive load of manual correction and re-annotation.The cached tree and stack files provide the foundation for the VLM-human collaborative system’s subsequent stages.Source: §6.2.3 · p. 14

1. Completeness (scomp

Completeness is assessed by checking each cropped child region within its parent context for full visibility and functional coherence.

1. Completeness

  • The checking model evaluates the cropped child region I_i within its parent region context.The parent region is marked with a red rectangle for reference.Source: §1 · p. 14
  • The completeness score ranges from 0 to 3.This score quantifies the assessment of the child region.Source: §1 · p. 14
  • The score assesses whether the region is fully visible and functionally coherent.Source: §1 · p. 14

2. Boundedness (sbound · 6.4. Functional Region Re-Annotation Details · 6.5. Functional Region Classification Details

The pipeline combines thresholded automated region proposals with human-verified geometry, VLM-revised semantics, and hierarchical taxonomy classification. These stages produce provenance-preserving, high-fidelity functional-region annotations for stratified benchmark analysis.

2. Boundedness (sbound

  • Proposals pass only when completeness and bounded-child ratios meet thresholds; failed proposals are refined up to three times, otherwise the highest-scoring proposal is retained.Example thresholds are ≥2.5 for completeness and ≥0.8 for bounded children, with N = 3 refinement attempts.Source: §2 · p. 14

6.4. Functional Region Re-Annotation Details

  • Because VLM boxes lack pixel-perfect precision, a human-in-the-loop web application corrects bounding boxes while preserving the original annotations and correction provenance.The interface serves annotation data through FastAPI and saves corrected metadata as timestamped _fix files rather than overwriting source files.Source: §6.4 · p. 14, §6.4.1 · pp. 14–15

6.4.1. Annotation Correction Server

  • The correction server discovers cached trees, exposes image and node APIs, and saves human-verified coordinates through a dedicated correction endpoint.Its dashboard computes node and correction statistics, while node reads merge original VLM output with existing fixes.Source: §6.4.1 · p. 14

6.4.2. Interactive Correction Interface

  • Annotators inspect searchable GUIs, hierarchical D3.js trees, and full screenshots with SVG box overlays before selecting regions for correction.The interface uses a three-panel layout comprising image list, hierarchy panel, and full-image panel.Source: §6.4.2 · pp. 15–16

6.4.3. Convenient Bounding Box Correction Workflow

  • The correction workflow provides full-context viewing, sharpened zoom, edge snapping, and corner snapping to improve boundary precision.OpenCV.js supports sharpening, Canny and Hough-line edge snapping, and Good Features to Track corner snapping.Source: §6.4.3 · p. 16

6.4.4. Functionality Re-Annotation

  • After geometry correction, Gemini-2.5-Pro-Thinking re-annotates only corrected nodes using the full screenshot, corrected crop, and flawed original annotations.The model is instructed to disregard prior text when inconsistent with new visual evidence and revise functionality and descriptions.Source: §6.4.4 · p. 16

6.4.5. Non-Destructive Output

  • Revised functionality, descriptions, and rationales are saved as separate artifacts, yielding functional regions with human-verified geometry and VLM-revised semantics.This non-destructive output preserves provenance and creates high-fidelity benchmark data points.Source: §6.4.5 · pp. 16–17

6.5. Functional Region Classification Details

  • A hierarchical taxonomy of 10 categories and over 40 types classifies regions with context-aware prompts, few-shot examples, robust parsing, and inheritance fallbacks.The taxonomy covers containers, navigation, content, interaction, contextual elements, and leaf types such as buttons and links; outputs are modular JSON artifacts used for component-level breakdowns.Source: §6.5 · p. 17, §6.5.1 · p. 17, §6.5.2 · p. 17

6.6. More Dataset Statistics · 6.7. Similar Functional Region Grouping Details

Sections 6.6–6.7 quantify dataset-generation costs and task attributes, then detail a multi-stage procedure for grouping visually similar but functionally distinct regions. The grouping pipeline combines semantic embeddings, VLM verification, deterministic geometric processing, abnormal-group refinement, and final validation.

6.6.1. Cost Analysis

  • $1.072 dollars is the estimated cost to process one 1920 × 1080 screenshot, combining region detection, refinement, and re-annotation.The estimate comprises $0.4375 for detection, $0.453 for maximum-repeat refinement, and $0.1815 for re-annotation.Source: §6.6.1 · p. 17

6.6.2. Task Statistics

  • The generated tasks are characterized by action type, density class, and similarity-group size, with region-type proportions reported separately.These attributes are illustrated across Figures 7–10, while Table 8 reports detected functional-region proportions.Source: §6.6.2 · p. 18, Table 8 · p. 18, Figure 7 · p. 18

6.7. Similar Functional Region Grouping Details

  • Similar-region groups are designed to test functional discrimination rather than superficial visual matching through semantic embedding, VLM verification, and deterministic geometric refinement.The groups contain visually similar but functionally distinct regions and support the benchmark’s grounding-task construction.Source: §6.7 · p. 18

6.7.1. Initial Semantic Grouping.

  • Qwen3-Embedding forms initial groups from cosine similarity above 0.6, excluding parent-child pairs and retaining only manually corrected bounding boxes.The affinity matrix is computed over embeddings of functional-region visual descriptions.Source: §6.7.1 · p. 18

6.7.2. Visual Verification by VLM.

  • Gemini-2.5-Pro-Thinking verifies visual similarity, adds nonoverlapping missing candidates, and constrains groups to 2–5 elements.Verification checks shape, color, or icon-style similarity while ensuring grouped elements perform different functions.Source: §6.7.2 · pp. 18–19

6.7.3. Geometric and Topological Processing.

  • Python refinement removes overlaps, discards groups below two elements, flags groups above five, and merges groups sharing at least two elements while reapplying hierarchy filters.Overlap resolution typically removes the larger-area element when conflicts occur.Source: §6.7.3 · p. 19

6.7.4. Refinement of Abnormal Groups.

  • Oversized or merged groups receive a second VLM pass that selects a 2–5-element subset maximizing functional confusion, using subtraction only.The model may remove elements but cannot add new ones during this abnormal-group refinement.Source: §6.7.4 · p. 19

6.7.5. Final Validation.

  • Only groups with no bounding-box overlaps and a final size of 2–5 elements pass validation and enter task generation.The final constraints are rechecked in Python after all grouping and refinement stages.Source: §6.7.5 · p. 19

7. Extra Evaluation Details

Extra evaluations show that VLM failures arise from a trade-off between semantic identification, spatial precision, hierarchical interface understanding, and interaction-outcome prediction. These errors reveal persistent reliance on surface visual cues rather than robust GUI functionality reasoning.

7.1. Functionality-Based Region Grounding Failure Case Inspection

  • Gemini identifies semantically relevant regions but produces spatially imprecise boxes, whereas Qwen3-VL-32B-Instruct yields accurate boxes but often targets the wrong elements.Gemini’s boxes are loose or shifted despite covering the correct semantic region, while Qwen prioritizes visual saliency over functional-text alignment.Source: §7.1 · p. 19, Figure 11 · p. 20

7.2. Functionality-Based Region Captioning Failure Case Inspection

  • Both models confuse system and application layers or distinct functions within one application, while Qwen additionally exhibits system-system and inter-application confusions.These failures indicate weak hierarchical GUI understanding and dependence on surface-level visual features rather than complex interactive logic.Source: §7.2 · pp. 19, 21, Figure 12 · p. 21

7.3. Functionality-Based Element Grounding Failure Case Inspection

  • Among 30 inspected element-grounding cases, 90% stemmed from incorrect reasoning and 10% from misunderstanding functionality, despite Gemini often identifying the target element.Gemini’s reasoning sometimes computes box coordinates redundantly but incorrectly, preventing accurate localization.Source: §7.3 · p. 21, Figure 13 · p. 22

7.4. Functionality-Based Element Captioning Failure Case Inspection

  • Gemini failed to predict interaction outcomes and distinguish similar controls, such as minimizing a system-tray app instead of returning to the previous screen.It also confused Linux spreadsheet alignment buttons and failed to identify the hover target that turns red before closing AutoCAD.Source: §7.4 · p. 21, Figure 14 · p. 23

8. Prompt Suites

This section describes the structured prompt suites used to generate AutoGUI-v2 functionality questions and annotate GUI regions hierarchically. It covers annotation-quality assessment, visually similar region selection, and evaluation prompts for region- and element-level grounding and captioning across multiple task types.

8.1. AutoGUI-v2 Annotating Prompts

  • The annotation prompts require hierarchical region decomposition with IDs, bounding boxes, divisibility labels, types, visual descriptions, and functionality descriptions, while limiting outputs to ten regions.For individual elements, the prompt instructs the model to describe only the overall GUI; corrupted or unprocessable screenshots yield an empty array.Source: §8.1 · pp. 25, 27
  • Annotation examples cover complete desktops, modal dialogs, single elements, and invalid screenshots, illustrating how regions are marked dividable or non-dividable and described by contextual functionality.Examples include Ubuntu with Visual Studio Code, an Excel Move/Copy Sheet dialog, a YouTube search bar, and an empty output for corrupted input.Source: §8.1 · pp. 26–27
  • A dedicated quality prompt scores a functional-region crop’s completeness from 0–3 and checks boundedness, enforcing that a region marked “Boundedness: No” cannot receive completeness 3.The example assigns completeness 1 when right-side cropping removes system-tray content.Source: §8.1 · p. 31
  • A re-annotation prompt revises each region’s layout and functionality descriptions, exemplified by a detailed Ubuntu dock description covering application launching, task management, and access to installed programs.The revised output identifies dock placement, appearance, application icons, active-app indication, Trash, and the applications grid.Source: §8.1 · p. 33

8.2. AutoGUI-v2 Task Generation Prompts

  • The task-generation prompt suite also includes a separate functionality-oriented captioning prompt, listed alongside the grouping, subset-selection, and grounding prompts.Table 15 is identified as the prompt for constructing functionality-oriented captioning tasks.Source: §8.2 · p. 34, Table 15 · p. 44
  • Task-generation prompts verify candidate groups using strict visual similarity, meaningful functional differences, non-overlapping bounding boxes, exhaustive candidate inspection, and final group sizes of 2–5 regions.The verification example removes vertically oriented or light-theme regions and adds visually matching application status, search, and OS top bars to form a five-region group.Source: §8.2 · pp. 36–38
  • The subset-selection prompt chooses only listed region IDs, selects 2–5 non-overlapping regions, prioritizes visual similarity, and preserves functional diversity; its example contrasts OS- and application-level bars across horizontal and vertical layouts.The selected set contains the Ubuntu top bar, Ubuntu dock, VS Code activity bar, and VS Code status bar, yielding four visually similar but functionally distinct UI-chrome regions.Source: §8.2 · pp. 40–41
  • Functionality-oriented grounding prompts generate interaction-outcome questions with randomized region options, a correct answer, and an explanation linking the selected region to its resulting system controls.The example asks which region adjusts volume or manages network connections and explains that the correct region opens the system control menu.Source: §8.2 · p. 43
  • The task-generation prompt converts a screenshot, target region, and visually similar alternatives into a multiple-choice question about the expected interaction outcome.The example asks what goal clicking an icon achieves and maps each option to a region-specific functionality.Source: §8.2 · p. 45
  • The generated options cover system-level controls, application view switching, application launching or switching, and project-status information.The example assigns these functions to the top bar, VS Code Activity Bar, Ubuntu Dock, and VS Code Status Bar, respectively.Source: §8.2 · pp. 45–46
  • The prompt requires shuffled options with region identifiers, a correct answer, and an explanation contrasting each region’s scope and unique functionality.The explanation distinguishes system-wide behavior from application-level navigation and project-specific status information.Source: §8.2 · pp. 45–46
  • The task-generation workflow assembles the final question as JSON after checking wording, option mapping, the correct answer, and the explanatory rationale.The stated goal is clarity, concision, and educational value.Source: §8.2 · pp. 45–46

8.3. AutoGUI-v2 Evaluation Prompts

  • Evaluation prompts for AutoGUI-v2 functionality, description, and intent-based element grounding are provided in Tables 16, 17, and 18.The passages identify these tables as the sources of the corresponding VLM evaluation prompts.Source: §8.3 · p. 47
  • Functionality-based region grounding uses the prompt shown in Table 16, while functionality-based element captioning uses the prompt shown in Table 19.For element captioning, the target element or region is marked by a red rectangle in the screenshot.Source: §8.3 · p. 47

9. Societal Impact

VLM-based GUI agents may improve accessibility while creating risks of labor displacement, surveillance, and digital insecurity. Responsible deployment therefore requires human oversight and caution about the benchmark’s English- and Western-centric bias.

Labor and Surveillance

  • Robust GUI agents could enable workplace surveillance or displace data-entry and QA roles, motivating a human-in-the-loop copilot model rather than autonomous replacement.The paper frames automation as reducing cognitive load but argues that ethical deployment should keep humans in control.Source: §9 · p. 47

Data Ethics and Bias

  • Because the dataset favors English-language and Western-centric interfaces, benchmark-only evaluation may not generalize cross-culturally or support universal applicability.The dataset uses scrubbed open-source repositories and public interfaces, but the authors caution that broader diverse testing remains necessary.Source: §9 · p. 47

10. Limitations and Future Work

AutoGUI-v2 remains limited by human-dependent annotation, single-step task descriptions, and the unexamined relationship between functionality understanding and multi-step planning. Future work targets model-assisted verification and analysis of how functionality understanding influences GUI-agent reasoning and planning.

Annotation automation

  • Human correction of detected functional-region bounding boxes limits annotation scalability, motivating reliable self-verification by capable models.The current annotation system is semi-automated and still requires human annotators to correct bounding boxes.Source: §10 · p. 47

Task relevance

  • Functionality descriptions are tied to single-step instructions, so interaction outcomes capture only the next step rather than longer task sequences.This leaves task relevance beyond two-step interaction outcomes insufficiently represented.Source: §10 · p. 47

Planning relevance

  • The benchmark does not analyze how functionality understanding correlates with planning ability in multi-platform GUI interactions.Future work will examine its use in GUI agents’ chain-of-thought reasoning and its effect on planning performance.Source: §10 · p. 47

Related papers

Loading 2604.24441v1…