PaperlayerView exact revision on arXiv

Source-linked AI summary

GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents

Jian Mu, Chaoyun Zhang, Chiming Ni, Lu Wang, Bo Qiao, Kartik Mathur, Qianhui Wu, Yuhang Xie, Xiaojun Ma, Mengyu Zhou, Si Qin, Liqun Li, Yu Kang, Minghua Ma, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

arXiv:2511.04307v2cs.AI

TL;DR

Desktop CUAs face a shortage of realistic tasks, scalable trajectory collection, and unified evaluation across grounding, parsing, and action prediction. GUI-360° addresses these gaps with an LLM-augmented dataset-construction pipeline and a large, multimodally annotated corpus. Evaluations find out-of-the-box model limitations and training gains, while results remain far from human-level reliability.

  • Problem

    Desktop CUA research lacks realistic task collections, scalable data pipelines, and unified benchmarks for grounding, screen parsing, and action prediction.

    Source: §1 · p. 2

  • Method

    GUI-360° uses an LLM-augmented pipeline to source real-world queries, construct templates, instantiate and execute tasks, and filter quality, producing multimodal trajectories for three CUA tasks.

    Source: §1 · p. 2

  • Results

    State-of-the-art vision–language models show out-of-the-box limitations, while fine-tuning and reinforcement learning on GUI-360° improve performance; results remain far from human-level reliability.

    Source: §1 · p. 3, §5 · p. 16

  • Takeaways & Limitations

    GUI-360° provides a challenging training and evaluation resource for desktop CUAs and a foundation for future progress.

    Source: §5 · p. 16

Abstract

from arXiv · show

We introduce GUI-360$^\circ$, a large-scale, comprehensive dataset and benchmark suite designed to advance computer-using agents (CUAs). CUAs present unique challenges and is constrained by three persistent gaps: a scarcity of real-world CUA tasks, the lack of automated collection-and-annotation pipelines for multi-modal trajectories, and the absence of a unified benchmark that jointly evaluates GUI grounding, screen parsing, and action prediction. GUI-360$^\circ$ addresses these gaps with an LLM-augmented, largely automated pipeline for query sourcing, environment-template construction, task instantiation, batched execution, and LLM-driven quality filtering. The released corpus contains over 1.2M executed action steps across thousands of trajectories in popular Windows office applications, and includes full-resolution screenshots, accessibility metadata when available, instantiated goals, intermediate reasoning traces, and both successful and failed action trajectories. The dataset supports three canonical tasks, GUI grounding, screen parsing, and action prediction, and a hybrid GUI+API action space that reflects modern agent designs. Benchmarking state-of-the-art vision--language models on GUI-360$^\circ$ reveals substantial out-of-the-box shortcomings in grounding and action prediction; supervised fine-tuning and reinforcement learning yield significant gains but do not close the gap to human-level reliability. We release GUI-360$^\circ$ and accompanying code to facilitate reproducible research and accelerate progress on robust desktop CUAs. The full dataset has been made public on https://huggingface.co/datasets/vyokky/GUI-360.

1 INTRODUCTION

GUI-360° addresses persistent gaps in desktop CUA data and evaluation with a large-scale dataset and benchmark. Its benchmark results show that current vision–language models struggle out of the box, while training on the dataset improves performance.

1 INTRODUCTION

  • Desktop CUA research lacks authentic task collections, scalable collection-and-annotation pipelines, and a unified benchmark spanning grounding, parsing, and action prediction.Prior datasets often focus on a single aspect or narrow application subset.Source: §1 · p. 2
  • GUI-360° uses an LLM-augmented pipeline for template construction, task instantiation, batched execution, and quality filtering, with minimal human intervention.Task intents are harvested from authentic sources such as search logs, community forums, and in-app help content.Source: §1 · p. 2
  • The corpus contains over 1.2M executed action steps across thousands of trajectories, with screenshots, accessibility metadata, goals, reasoning traces, and successful and failed runs.These annotations support GUI grounding, screen parsing, and action prediction.Source: §1 · p. 2
  • GUI-360° combines direct GUI operations with higher-level API calls and covers grounding, parsing, and action prediction alongside accessibility information and reasoning supervision.The authors describe this combination as a first among GUI datasets.Source: §1 · p. 3
  • Off-the-shelf vision–language models struggle with grounding in heterogeneous layouts and stepwise action prediction; fine-tuning yields gains, but reliability remains below human level.Action-prediction failures can lead to cascading errors.Source: §1 · p. 3

2 RELATED WORK

Prior CUA work spans visual grounding, accessibility-based control, and API integration, while desktop datasets remain less explored than web and mobile resources. Existing desktop benchmarks can require costly annotation or cover narrow task subsets; GUI-360° positions its automated pipeline as a scalable resource across three core capabilities.

2 RELATED WORK

  • Prior CUA approaches use accessibility metadata, screen parsing, visual grounding, hybrid control, and APIs to interact with desktop applications.When accessibility information is missing, some agents rely on screen parsing and visual grounding.Source: §2 · p. 3
  • Desktop data resources remain comparatively underexplored, and existing benchmarks may depend on costly human annotation or cover narrow task subsets.UI-Vision provides human-annotated trajectories and GUI states; OfficeBench uses handcrafted cases and oracle evaluation.Source: §2 · pp. 3–4
  • GUI-360° presents an automated data-collection pipeline for a dataset and benchmark spanning GUI grounding, screen parsing, and action prediction.The authors characterize it as the most comprehensive and scalable dataset–benchmark suite to date for training and evaluating GUI agents.Source: §2 · pp. 3–4

3 THE COLLECTION OF GUI-360◦

GUI-360° is built through query acquisition, automated trajectory collection, and evaluation/post-processing to create a scalable dataset supporting multiple core GUI-agent capabilities. Its pipeline grounds real-world queries in executable environments, records multimodal action traces, and filters and structures the resulting data.

3 THE COLLECTION OF GUI-360◦

  • The collection pipeline combines real-world query acquisition, automatic trajectory collection, and evaluation and post-processing into a scalable resource for multiple GUI-agent capabilities.It is organized as three stages, from query acquisition through trajectory collection to evaluation and post-processing.Source: §3 · p. 4

3.1 QUERY ACQUISITION

  • The 66 environment templates—30 each for Word and Excel and 6 for PowerPoint—cover approximately 95% of prototypical queries, while filtering retains 59,553 of 79,075 candidates (75.3%).Queries come from in-app help, community websites, and search queries; LLM-based template matching and concretization ground them in executable environments.Source: §3.1 · pp. 5–6

3.2 AUTOMATIC TRAJECTORY COLLECTION

  • TrajAgent batches task execution by preparing each environment, recording stepwise GUI states and metadata, and resetting the environment afterward.The automated process is designed to eliminate human intervention while collecting consistent, high-fidelity trajectories.Source: §3.2 · p. 7

3.2.1 TRAJAGENT DESIGN

  • TrajAgent’s MasterAgent decomposes tasks and dispatches them to ExecutionAgents, which perceive the GUI and act through UI automation or APIs while a Recorder logs pre- and post-action states.Its two-stage execution uses GPT-4o first and GPT-4.1 for failures, reaching 26.09% overall completion.Source: §3.2.1 · p. 8

3.3 EVALUATION AND POST-PROCESSING

  • Post-processing validates trajectories with EvaAgent, removes incomplete or low-quality records, and normalizes the remaining data for three downstream tasks.On 100 sampled trajectories, EvaAgent’s judgments agreed with human annotators 86% of the time.Source: §3.3 · p. 9

3.4 GUI-360◦STATISTICS

  • 13,750 trajectories with over 105k steps form the training corpus, while GUI-360°-Bench adds 3,439 trajectories and 26k steps; the corpus also includes 62,170 failure trajectories with 1,093,525 steps.The dataset pairs 210k screenshots with 17.7M annotated UI elements; queries span Word (41.0%), Excel (31.6%), and PowerPoint (27.4%).Source: §3.4 · pp. 10–11, Table 6 · p. 10

4 EXPERIMENT

The experiments benchmark GUI grounding, screen parsing, and action prediction, comparing general-purpose models with specialized models and GUI-360°-based training. Results show substantial weaknesses in general-purpose models and gains from specialized parsing, fine-tuning, and accessibility-enhanced inputs.

4 EXPERIMENT

  • Experiments compare zero-/few-shot models with GUI-360° post-training on an 80/20 training/held-out benchmark split.Screen-parsing metrics are macro-averaged across images; greedy IoU matching retains pairs above 0.5.Source: §4 · p. 11, §4.2 · pp. 12–13

4.1 GUI GROUNDING

  • GUI-360-trained Qwen2.5 7B-SFT and UI-TARS-1.5 7B-SFT exceed 82% accuracy, while general-purpose GPT models remain below 12%.Grounding accuracy is evaluated by whether the predicted coordinate falls inside the target element’s bounding box.Source: §4.1 · pp. 11–12, Table 7 · p. 12

4.2 SCREEN PARSING

  • OmniParser variants exceed 0.40 overall F1 and 0.73 mean IoU, versus general-purpose VLMs at F1 0.019–0.128 and mean IoU 0.229–0.578.OmniParser-v2 improves modestly but consistently over OmniParser; Excel remains challenging for specialized parsers.Source: §4.2 · p. 13, Table 8 · p. 14

4.3 ACTION PREDICTION

  • GPT-4o action-prediction accuracy increased from 3.12% with screenshots alone to 36.71% with visual+a11y input; Qwen-2.5 7B rose from 17.52% to 50.08% after visual-only SFT.SFT benefits diminish when a11y information is introduced.Source: §4.3 · p. 15
  • Argument mismatch remains the dominant action-prediction failure; a11y reduces Coord. OOB errors, but argument prediction still trails function prediction.SFT brings large gains in function match and especially argument match, reducing errors by more than half across domains.Source: §4.3 · p. 16

5 CONCLUSION

GUI-360◦ combines a large, richly annotated desktop-agent dataset with a benchmark spanning key CUA tasks. Evaluation shows that fine-tuning and reinforcement learning improve results, but reliability remains far from human-level.

5 CONCLUSION

  • GUI-360◦ addresses gaps in realistic task collections, scalable data pipelines, and unified benchmarks with over 1.2M steps across thousands of annotated trajectories.The corpus includes screenshots, accessibility metadata, reasoning traces, and successful and failed executions.Source: §5 · p. 16
  • Fine-tuning and reinforcement learning deliver consistent improvements, yet results remain far from human-level reliability.The evaluation highlights both the difficulty of desktop CUAs and GUI-360◦’s utility as a training and evaluation resource.Source: §5 · p. 16
  • The authors release GUI-360◦, GUI-360◦-Bench, and the data-collection framework to the research community.They aim for these resources to support advances in screen understanding, multimodal reasoning, and action planning.Source: §5 · p. 16

A ACTION SET

  • The action set combines general GUI operations with application-specific APIs, balancing broad interface coverage with precise semantics for structured tasks.GUI actions apply across interfaces, while Word, Excel, and PowerPoint APIs expose document-specific operations.Source: A ACTION SET · p. 19

B GUI-360◦SCHEMA

GUI-360◦ standardizes execution steps with visual, structural, and cognitive context. The paper also summarizes general-purpose and GUI-specialized baselines evaluated across its three core tasks.

B GUI-360◦SCHEMA

  • The execution-step schema combines screenshots, accessibility metadata and UI trees, plus observations, reasoning, actions, and evaluations to support grounding, parsing, and action prediction.The standardized records support supervised training and fine-grained evaluation across applications.Source: B GUI-360◦SCHEMA · p. 19

C BASELINE DETAILS

  • The evaluated baselines include general-purpose vision–language models and GUI-specialized approaches for grounding, screen parsing, and action prediction.The paper introduces the model groups in the following baseline descriptions.Source: C BASELINE DETAILS · p. 19

C.1 GUI GROUNDING •

The GUI-grounding baselines span general-purpose vision–language models, GUI-focused models, and a task-adapted fine-tuning variant.

C.1 GUI GROUNDING

  • GPT-4o was used off-the-shelf for grounding, o3 was evaluated zero-shot for grounding, GPT-4.1 emphasizes instruction-following and tool use, and GPT-5 is a strong general reasoning baseline.Source: §C.1 · pp. 19–20
  • GUI-oriented baselines include Qwen2.5-VL-7B Bai et al. (2025), UGround-7B Gou et al. (2024), Aguvis-7B Xu et al. (2024), UI-TARS-1.5 Qin et al. (2025), and GUI-Actor Wu et al. (2025).The passages describe these as an open-source multimodal baseline and GUI-focused models for grounding, reasoning, or interactive tasks.Source: §C.1 · p. 20
  • A Qwen2.5-VL-7B variant fine-tuned on GUI-360◦ is included for task-adapted grounding.The action-set table groups supported actions across Word, Excel, and PowerPoint into shared GUI actions and application-specific APIs.Source: §C.1 · p. 20, Table 11 · p. 20

C.2 SCREEN PARSING •

Screen-parsing baselines include general-purpose vision–language models, an open-source multimodal model, and dedicated parsing tools. The execution-step schema also records trajectory context and status.

C.2 SCREEN PARSING

  • GPT-4o, GPT-4.1, o3, and GPT-5 are evaluated as general-purpose models for screen-element detection and localization.The passage gives their original bibliography markers as Hurst et al. (2024) and OpenAI (2025b;d;c).Source: §C.2 · p. 20
  • Qwen2.5-VL-7B Bai et al. (2025) is included as an open-source multimodal baseline.Source: §C.2 · p. 20
  • OmniParser and OmniParser-v2 Lu et al. (2024) produce element sets containing names and bounding boxes from screenshots.Source: §C.2 · p. 20
  • The execution-step schema records screenshots, accessibility data, reasoning traces, actions, template, trajectory position, observations, intermediate thoughts, and CONTINUE or FINISH status.Source: Table 12 · p. 21, §C.2 · p. 21

C.3 ACTION PREDICTION •

The action-prediction evaluation spans proprietary and open-source models, including a GUI-360° fine-tuned variant. Together, the baselines support evaluation across grounding, parsing, and action prediction.

C.3 ACTION PREDICTION •

  • GPT-4o, GPT-4.1, o3, and GPT-5 are evaluated in visual-only and visual+a11y settings.Source: §C.3 · p. 21
  • Qwen2.5-VL-7B serves as an open-source baseline for structured action generation.Source: §C.3 · p. 21
  • Qwen2.5-VL-7B-SFT is fine-tuned on GUI-360° for step-wise function, argument, and status prediction.Source: §C.3 · p. 21
  • The baseline suite spans general-purpose VLMs and GUI-focused agents for evaluating challenges across grounding, parsing, and action prediction.The authors use this diversity to measure how far current models remain from robust, human-level computer-using agents.Source: §C.3 · p. 21

D IMPLEMENTATION DETAILS

Implementation uses parallel Windows 11 virtual machines for trajectory collection and Azure OpenAI models for benchmark evaluation. Fine-tuning and reinforcement-learning runs use a four-GPU A100 setup.

D IMPLEMENTATION DETAILS

  • Fifteen Windows 11 virtual machines, each with four CPU cores, execute tasks in parallel; failed tasks are retried with GPT-4.1 after GPT-4o execution.Both collection models use temperature 0.0 for deterministic, reproducible outputs.Source: D IMPLEMENTATION DETAILS · p. 21
  • GPT-4o, GPT-4.1, o3, and GPT-5 are accessed through Azure OpenAI Service for benchmark evaluation.Evaluation decoding temperature is 0.0 to minimize variance and ensure consistency.Source: D IMPLEMENTATION DETAILS · pp. 21–22
  • SFT and RL runs use four NVIDIA A100 40GB GPUs with FP16 mixed-precision training.Learning rates are tuned over {1e-5, 5e-6, 1e-6}; checkpointing and gradient accumulation support stable long-trajectory training.Source: D IMPLEMENTATION DETAILS · p. 22

ETHICS STATEMENT

The authors describe privacy, security, and software-compliance measures for dataset construction and release. They limit released scenarios to non-sensitive office applications and research use.

ETHICS STATEMENT

  • No human subjects are directly involved, and queries and trajectories are generated in controlled sandboxes without PII or sensitive user data.Source: ETHICS STATEMENT · p. 22
  • The collection process follows software licenses and platform usage guidelines, with post-processing to validate, sanitize, and structure trajectories.These steps are intended to reduce incomplete, low-quality, or potentially misleading samples and risks of harmful insights or erroneous model behaviors.Source: ETHICS STATEMENT · p. 22
  • Dataset release is restricted to non-sensitive Word, Excel, and PowerPoint contexts, excluding scenarios that could pose privacy, security, or safety risks.The dataset is intended solely for academic research on GUI-agent robustness, generalization, and evaluation.Source: ETHICS STATEMENT · p. 22

REPRODUCIBILITY STATEMENT

The authors plan to release the dataset, benchmark split, collection code, execution framework, and query-instantiation templates to support reproducibility.

REPRODUCIBILITY STATEMENT

  • The planned release includes all data-collection code, the execution framework, and templates used to instantiate user queries.Source: REPRODUCIBILITY STATEMENT · p. 22
  • GUI-360° and the GUI-360°-Bench split will be publicly available, with task definitions, filtering statistics, and execution details described in the paper and appendix.Source: REPRODUCIBILITY STATEMENT · p. 22
  • Supplementary materials provide implementation details, hyperparameters, and evaluation protocols to support reproduction, verification, and extension of the work.Source: REPRODUCIBILITY STATEMENT · p. 22

Related papers

Loading 2511.04307v2…