UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction

Shravan Nayak; Xiangru Jian; Kevin Qinghong Lin; Juan A. Rodriguez; Montek Kalsi; Rabiul Awal; Nicolas Chapados; M. Tamer Özsu; Aishwarya Agrawal; David Vazquez; Christopher Pal; Perouz Taslakian; Spandana Gella; Sai Rajeswar

Source-linked AI summary

UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction

Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A. Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M. Tamer Özsu, Aishwarya Agrawal, David Vazquez, Christopher Pal, Perouz Taslakian, Spandana Gella, Sai Rajeswar

arXiv:2503.15661v2Mar 19, 2025cs.CVcs.AIcs.CL

TL;DR

Desktop GUI agents lack comprehensive offline benchmarks because existing evaluations focus mainly on web and mobile environments, while desktop software lacks standardized representations and automation APIs. UI-Vision introduces a permissively licensed benchmark built from densely annotated demonstrations across 83 applications and organized into Element Grounding, Layout Grounding, and Action Prediction. Evaluation reveals significant gaps across all three tasks, including weak grounding, layout understanding, click prediction, and drag handling.

Problem
Existing GUI benchmarks primarily target web and mobile environments, leaving desktop agents under-evaluated despite desktop software's importance and lack of standardized representations.
Method
UI-Vision builds an offline benchmark from densely annotated human demonstrations across 83 desktop applications and evaluates agents on Element Grounding, Layout Grounding, and Action Prediction.
Results
State-of-the-art models show significant gaps across all three tasks, including 25.5% Element Grounding accuracy, 30.8 Layout Grounding IoU, and 19.7% click-action recall.
Takeaways & Limitations
UI-Vision provides a fine-grained evaluation framework for desktop GUI agents and exposes challenges in visual understanding, spatial reasoning, and complex interactions.
Takeaways & Limitations
The benchmark is offline, uses heuristic sampling, and relies on a single human demonstration for action tasks, limiting coverage of interaction variations.

Abstract

from arXiv · show

Autonomous agents that navigate Graphical User Interfaces (GUIs) to automate tasks like document editing and file management can greatly enhance computer workflows. While existing research focuses on online settings, desktop environments, critical for many professional and everyday tasks, remain underexplored due to data collection challenges and licensing issues. We introduce UI-Vision, the first comprehensive, license-permissive benchmark for offline, fine-grained evaluation of computer use agents in real-world desktop environments. Unlike online benchmarks, UI-Vision provides: (i) dense, high-quality annotations of human demonstrations, including bounding boxes, UI labels, and action trajectories (clicks, drags, and keyboard inputs) across 83 software applications, and (ii) three fine-to-coarse grained tasks-Element Grounding, Layout Grounding, and Action Prediction-with well-defined metrics to rigorously evaluate agents' performance in desktop environments. Our evaluation reveals critical limitations in state-of-the-art models like UI-TARS-72B, including issues with understanding professional software, spatial reasoning, and complex actions like drag-and-drop. These findings highlight the challenges in developing fully autonomous computer use agents. By releasing UI-Vision as open-source, we aim to advance the development of more capable agents for real-world desktop tasks.

1. Introduction

Desktop GUI agents remain difficult to evaluate because existing benchmarks emphasize web and mobile settings, while desktop interfaces lack standardized representations and automation APIs. UI-Vision addresses this gap with an offline benchmark spanning diverse applications, dense annotations, and three evaluation tasks, revealing substantial weaknesses in current models.

Motivation: Desktop GUI evaluation is underexplored because desktop interfaces lack standardized text representations and automation APIs available for web environments.Web benchmarks commonly rely on DOM structures and HTML metadata, whereas desktop interaction depends on precise mouse and keyboard control.
Benchmark: UI-Vision benchmarks desktop agents across 83 software applications and six domains in an offline setting.The benchmark targets real-world software variability, visual perception, and open-ended GUI interactions.
Benchmark: The benchmark combines dense UI annotations with three tasks: Layout Grounding, Element Grounding, and Action Prediction.These tasks assess functional-region recognition, precise component localization, and interaction prediction for clicking, dragging, and typing.
Findings: 25.5% accuracy is the best reported Element Grounding result, achieved by UI-TARS, while Gemini-1.5-Pro reaches 30.8 IoU on Layout Grounding.The results indicate persistent difficulty with fine-grained element understanding and high-level recognition of structured UI regions.
Findings: 19.7% recall on click actions is achieved by UI-TARS, and all evaluated models struggle with drag actions.The reported weaknesses span grounding, layout understanding, and motion-based interaction execution.

2. Related Work

Prior GUI benchmarks are fragmented across element grounding, action prediction, and platform types, with layout grounding and desktop environments receiving limited coverage. UI-Vision responds with a diverse, permissively licensed desktop benchmark containing rich annotations across many software environments.

Agent approaches: Pure language agents rely on HTML, accessibility trees, OCR, or other structured metadata, limiting applicability to raw visual interfaces.These approaches also rely on closed-source models and can struggle with generalization.
Agent approaches: Multimodal agents enable pixel-level grounding and context-aware navigation, but progress is constrained by limited large-scale visual data and standardized benchmarks.The related work motivates a benchmark centered on visual desktop interaction.
Existing benchmarks: Element grounding and action prediction have dedicated benchmarks, but layout grounding is often overlooked.Coordinate-based methods do not capture structural relationships between GUI elements.
Existing benchmarks: GUI benchmarks are disproportionately focused on web and mobile platforms, while desktop benchmarks are either online or small-scale.Desktop environments remain underexplored despite their importance in professional workflows.
UI-Vision: UI-Vision spans 83 software environments and provides rich human annotations with 8227 query-label pairs across three tasks.Its permissive licensing and broad application coverage target gaps identified in existing benchmark comparisons.

3. UI-Vision

UI-Vision is constructed from annotated human demonstrations across diverse open-source desktop applications and converts them into three perception-to-action benchmark tasks. Its dense keyframe annotations and long-horizon trajectories support fine-grained evaluation of GUI understanding and interaction.

Data Collection: UI-Vision curates 83 open-source desktop platforms across six domains, with 5–7 computer-use tasks designed around real-world workflows per platform.The domains include Productivity, Development, Creativity, Education, Browsers, and Social Media/Entertainment.
Data Collection: Expert annotators record task videos, action logs, and pre-action keyframes, then label UI elements with bounding boxes and descriptions.Separate annotators and authors perform multi-stage quality checks, producing 450 demonstrations across 83 applications.
Dataset Statistics: Each keyframe contains 5 to 200 labeled bounding boxes, averaging 71 annotations per keyframe.This dense annotation setup is reported as substantially richer than previous works.
Element Grounding: Element Grounding predicts UI-component bounding boxes from queries using basic, functional, and spatial subtasks.The subtasks test direct localization, function-based identification, and localization through spatial relationships.
Layout Grounding: Layout Grounding clusters UI elements into functional and semantic groups and predicts bounding boxes enclosing those groups.The task targets structural relationships and larger functional regions in GUI layouts.
Action Prediction: Action Prediction uses task goals, current screenshots, and previous action histories to predict structured interaction sequences.The resulting benchmark contains 3191 action-annotation pairs over 442 computer-use tasks.

4. Experiments

UI-Vision evaluates desktop GUI agents across grounding, layout understanding, and action prediction, exposing persistent weaknesses in fine-grained localization, spatial reasoning, cluttered interfaces, and motion-based interactions.

Evaluation setup: UI-Vision evaluates models across Element Grounding, Layout Grounding, and Action Prediction using task-specific metrics for localization, layout alignment, and interaction accuracy.The benchmark includes category-level results across desktop applications and compares closed-source models, open-source VLMs, and GUI agents.
Element Grounding: 25.5% accuracy is achieved by UI-TARS on Element Grounding, while spatial grounding reaches only 14.9% for the best model.GUI-specialized models outperform general VLMs, but overall fine-grained desktop grounding remains difficult.
Element Grounding: Larger UI elements are easier to ground, while accuracy drops sharply as screenshots contain more elements.Blender and GIMP average 112 bounding boxes and 418 square pixels per element, and show among the lowest performance.
Element Grounding: Element Grounding failures include fine-grained ambiguity, missing domain knowledge, small components, and incorrect transfer of layout assumptions across platforms.The analysis points to iterative zooming and external software documentation as possible responses to specific failure modes.
Layout Grounding: VLMs outperform GUI agents on Layout Grounding, while IoU increases with bounding-box area and decreases as element density rises.Complex, cluttered layouts remain the primary bottleneck for precise layout generation.
Action Prediction: Action Prediction remains weakest for click and drag actions, whereas typing and hotkeys perform better when prompts provide explicit or predictable information.Gemini-1.5-Pro leads closed-source models on click and drag, while GPT-4o performs better on keyboard-related actions.
Action Prediction: Separating textual action planning from coordinate grounding improves click recall 2x to 5x, bringing GPT-4o, Claude, and Gemini to 26.7%, 26.9%, and 25.0%, respectively.The results indicate that closed-source models can predict actions but struggle to ground them accurately on screen.
Action Prediction: Open-source GUI agents show stronger grounding but weaker instruction adherence, including omitted drag actions and unsolicited wait() or finish() actions.UI-TARS reaches 19.7% click recall, while ShowUI fails to generate drag actions entirely.

5. Conclusion

UI-Vision introduces a large-scale desktop GUI benchmark spanning 83 applications and three structured evaluation tasks. Its experiments expose substantial limitations in current models and motivate further research in desktop UI interaction modeling.

Benchmark contribution: UI-Vision covers 83 desktop applications and supports Element Grounding, Layout Grounding, and Action Prediction.The benchmark measures UI-element localization, functional grouping with bounding boxes, and action selection for task completion.
Findings and significance: Experiments reveal significant challenges in grounding UI elements and predicting actions accurately.The benchmark is intended to support further research by providing an open-source evaluation resource.

6. Limitation and Future works

UI-Vision is currently limited to offline evaluation and uses heuristic sampling, single demonstrations, and no combined mouse-keyboard actions. Future work targets broader online assessment, more representative sampling, multiple trajectories, videos, and complex combined actions.

Offline tasks do not yet assess agents in online environments, limiting evaluation of real-world interactions.
Heuristic sampling may not produce fully representative UI elements for evaluating agentic abilities.
A single human action demonstration may miss alternative ways to complete tasks and introduce evaluation bias.
The benchmark does not include human-recorded videos for assessing GUI action understanding.
Combined mouse-keyboard actions, such as holding Control while dragging, remain unevaluated.

7. Impact Statement

UI-Vision is presented as a desktop GUI benchmark intended to advance automation, productivity, and software development. The paper also identifies privacy, environmental, and user-dependence risks associated with GUI agents and their deployment.

UI-Vision aims to standardize desktop GUI evaluation and advance GUI model development.
GUI agents could automate repetitive and complex desktop tasks, helping users focus on creative and strategic activities.
The benchmark may support software development by helping developers complete unit tests and collect interface feedback.
Deployment on personal devices raises privacy concerns because GUI agents may expose sensitive information.
Vision-language-model inference creates environmental concerns through increased energy consumption.
Reliance on highly capable GUI agents may reduce manual proficiency and problem-solving skills over time.

B. UI-Vision Benchmark Tasks

UI-Vision constructs benchmark tasks from annotated desktop interfaces using automated generation, filtering, and human verification. The tasks cover functional and spatial element grounding, layout grouping, and standardized action prediction.

Benchmark construction: The workflow begins with bounding-box annotations and samples subsets to maintain platform coverage and ease evaluation.
Element Grounding: Synthetic model predictions, deduplication, and human verification create curated Element Grounding data.
Element Grounding: Functional Element Grounding uses screenshots and annotations to generate descriptions of how UI elements are used.
Element Grounding: Spatial queries identify the closest neighboring element in the up, bottom, left, or right direction, filtering invalid or distant cases.
Layout Grounding: Layout Grounding groups annotated elements by functionality and verifies clusters and enclosing bounding boxes through human review.
Layout Grounding: 311 human-verified query-label pairs span 77 platforms, with functional groups typically containing 5–10 UI elements.
Action Prediction: Action Prediction standardizes interactions into click, move to, drag, typing, and hotkey categories while removing redundant move actions.

C. Data Statistics and Examples

UI-Vision contains dense screenshots and varied GUI interaction examples. Its visualizations illustrate element localization challenges across basic, functional, and spatial grounding settings.

Screenshots contain a mean of 74.3 bounding boxes, while human recordings average 38.2 seconds.
Figure 7 shows Element Grounding examples requiring localization from straightforward labels such as “crop tool” or “skip all breakpoints.”
The figures present task examples across multiple software applications and different aspects of UI-Vision’s benchmark.

D.1. Element Grounding

Element Grounding tests whether agents can locate UI components from basic, functional, and spatial descriptions. Failure cases show persistent difficulty with fine-grained ambiguity, domain-specific knowledge, small elements, and cross-platform layouts.

D.1. Element Grounding: Models often identify the correct nearby region but select the wrong target among visually similar candidates.The authors connect this failure to a need for more advanced validation strategies.
D.1. Element Grounding: Domain-specific knowledge remains insufficient for interpreting software-specific controls and terminology.The paper suggests retrieval-augmented generation or external software documentation as possible remedies.
D.1. Element Grounding: Small UI elements in high-resolution, densely populated interfaces are especially difficult for models to locate.An iterative zoom-in strategy is proposed as one potential response.
D.1. Element Grounding: Models sometimes transfer layout assumptions across platforms, misplacing controls such as iOS minimize buttons according to Windows conventions.This demonstrates a cross-platform generalization failure in interface grounding.
D.1. Element Grounding: Element Grounding includes basic descriptions, functional requests, and spatial relations for locating UI components.Basic examples use labels such as “crop tool,” while functional queries describe operations rather than visual labels.

D.2. Layout Grounding

Layout Grounding evaluates whether agents can identify meaningful functional regions rather than merely individual elements. Models frequently produce overly broad boxes, weak functional groupings, or semantically superficial matches.

D.2. Layout Grounding: Closed-source models often return bounding boxes larger than the minimal ground-truth region.The ground-truth region is usually contained within the prediction, but the layout partition is not understood precisely.
D.2. Layout Grounding: Open-source agents struggle to group interface elements into coarser functional regions, even when queries name many constituent elements.The paper links this limitation to insufficient training data for higher-level interface perception.
D.2. Layout Grounding: Some agents rely on element-label positions rather than understanding functional groupings.This behavior indicates a weaker generalized representation of interface structure.
D.2. Layout Grounding: When uncertain, open-source agents may select items sharing query words without being semantically equivalent to the requested region.The authors identify this as a need for deeper semantic understanding beyond keyword matching.

D.3. Action Prediction

Action Prediction examines whether agents can convert UI understanding into correct clicks, drags, and typing actions. Errors arise from inaccurate grounding, limited platform knowledge, and the complexity of dense software environments.

D.3. Action Prediction: Action Prediction failures often occur when models infer the intended action but apply it to the wrong UI element.The paper identifies poor grounding as a major error source for both open- and closed-source models.
D.3. Action Prediction: Models frequently lack platform-specific knowledge, leading to hallucinated actions or nonsensical predictions.The authors attribute this pattern partly to insufficient exposure to diverse desktop environments during training.
D.3. Action Prediction: UI-TARS records its highest error rate, 85%, on creativity platforms with dense interfaces and extensive functionality.Small, tightly packed elements and complex workflows increase the difficulty of selecting appropriate actions.
D.3. Action Prediction: Grounding accuracy shows no clear strong correlation with screenshot resolution for UI-TARS-72B and UGround-v1-72B.Accuracy fluctuates across resolution levels, suggesting other factors may have greater influence.
D.3. Action Prediction: All evaluated models show significant accuracy drops on less common applications such as FreeCAD and QGIS compared with common applications.The comparison serves as a proxy for cross-software generalization because the exact training data are unknown.

E.4. Analysis of Planner vs. Grounding Model Contributions

The planner-versus-grounder analysis reports improved recall when language-model planners are combined with grounding models, while the benchmark’s visualizations and efficiency analyses expose varied interaction and perception challenges. The accompanying examples emphasize dense annotations, action-trajectory simplification, and recurring grounding errors across desktop software.

E.4. Analysis of Planner vs. Grounding Model Contributions: Combining language-model planners with grounding models significantly improves recall.The authors attribute the gain primarily to better action selection because the grounding model supplies coordinates.
E.4. Analysis of Planner vs. Grounding Model Contributions: Productivity tools show the largest error-rate reduction, 26%, despite entertainment tools having higher baseline grounding accuracy.This comparison is presented as evidence of the planner’s differing impact across platform categories.
E.4. Analysis of Planner vs. Grounding Model Contributions: UI-Vision evaluates desktop agents across common and rare software as a proxy for cross-software generalization.The benchmark compares subsets such as VSCode against FreeCAD, QGIS, and Qt Creator.
E.4. Analysis of Planner vs. Grounding Model Contributions: Efficiency reporting covers latency, output tokens, and GPU usage across Element Grounding, Layout Grounding, and Action Prediction.Latency uses default Hugging Face implementations, while token counts use GPT-4 tokenization.
E.4. Analysis of Planner vs. Grounding Model Contributions: Recorded action trajectories retain coordinates while grouping drag-related events and removing redundant mouse-move actions for evaluation.The trajectory examples illustrate how raw computer-use actions are simplified into descriptive evaluation inputs.
E.4. Analysis of Planner vs. Grounding Model Contributions: Failure examples include fine-grained ambiguity, domain-knowledge gaps, small-element errors, cross-platform transfer, and inaccurate layout grouping.These cases connect the benchmark’s visual tasks to concrete limitations in current GUI agents.