PaperlayerView exact revision on arXiv

Source-linked AI summary

ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data

Zhaoyang Liu, Jingjing Xie, Zichen Ding, Zehao Li, Bowen Yang, Zhenyu Wu, Xuehui Wang, Qiushi Sun, Shi Liu, Weiyun Wang, Shenglong Ye, Qingyun Li, Xuan Dong, Yue Yu, Chenyu Lu, YunXiang Mo, Yao Yan, Zeyue Tian, Xiao Zhang, Yuan Huang, Yiqian Liu, Weijie Su, Gen Luo, Xiangyu Yue, Biqing Qi, Kai Chen, Bowen Zhou, Yu Qiao, Qifeng Chen, Wenhai Wang

arXiv:2509.15221v2cs.CV

TL;DR

Computer use agents lack large-scale, open-source data and models, despite needing rich software and operational knowledge. ScaleCUA combines cross-platform data collection with unified agent models, achieving state-of-the-art or competitive results across cross-platform benchmarks.

  • Problem

    Computer use agents lack large-scale, open-source data and models, while rich in-domain software and operational knowledge remains a substantial gap for foundation models.

    Source: §1 · p. 2

  • Method

    ScaleCUA curates cross-platform GUI data through an interactive pipeline combining automated agents and human experts, and develops models unifying perception, reasoning, and action.

    Source: §1 · p. 3

  • Results

    ScaleCUA achieves state-of-the-art or competitive performance across understanding, grounding, and end-to-end task-completion benchmarks spanning platforms.

    Source: §1 · p. 3, §6 · p. 15

  • Takeaways & Limitations

    The cross-platform corpus and standardized action space provide a unified foundation for evaluating and training computer use agents.

    Source: §6 · p. 15

    Agent-collected data quality still lags expert crowdsourcing, while expert-curated data remains costly to scale.

    Source: LIMITATIONS · p. 15

Abstract

from arXiv · show

Vision-Language Models (VLMs) have enabled computer use agents (CUAs) that operate GUIs autonomously, showing great potential, yet progress is limited by the lack of large-scale, open-source computer use data and foundation models. In this work, we introduce ScaleCUA, a step toward scaling open-source CUAs. It offers a large-scale dataset spanning 6 operating systems and 3 task domains, built via a closed-loop pipeline uniting automated agents with human experts. Trained on this scaled-up data, ScaleCUA can operate seamlessly across platforms. Specifically, it delivers strong gains over baselines (+26.6 on WebArena-Lite-v2, +10.7 on ScreenSpot-Pro) and sets new state-of-the-art results (94.4% on MMBench-GUI L1-Hard, 60.6% on OSWorld-G, 47.4% on WebArena-Lite-v2). These findings underscore the power of data-driven scaling for general-purpose computer use agents. We will release data, models, and code to advance future research: https://github.com/OpenGVLab/ScaleCUA.

1 INTRODUCTION

ScaleCUA addresses the data and generalizability bottleneck for computer-use agents by building cross-platform training data and scalable models. The work combines automated interaction with expert-annotated trajectories and evaluates agents across GUI benchmarks.

1 INTRODUCTION

  • Scarce, costly-to-annotate action trajectories and rapidly changing software constrain CUA data scale and model generalizability, motivating an open cross-platform corpus and general-purpose agent models.The introduction contrasts these constraints with the abundance of online image–text pairs.Source: §1 · p. 2
  • The dataset pipeline spans six platforms and three GUI task domains, collecting observations, metadata, and trajectories through automated-agent and expert-human loops.It includes 471K GUI-understanding examples, 17.1M grounding samples, over 15K weak-semantic trajectories, and 4K high-level goal-directed trajectories.Source: §1 · pp. 2–3
  • The evaluation spans GUI understanding, grounding, and end-to-end task completion on cross-platform benchmarks, with ScaleCUA achieving state-of-the-art or competitive performance.The study also compares agentic workflows with native models and examines how data sources, training tasks, and agent designs affect performance.Source: §1 · p. 2
  • ScaleCUA models unify perception, reasoning, and action, supporting grounding, direct-action, and reasoned-action inference through a standardized cross-platform action space.The models are trained on the collected corpus using Qwen2.5-VL; the paper reports benefits from data augmentation, weak-semantic trajectories, and general reasoning data for planning.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans VLMs, GUI-agent architectures, and interaction datasets, but existing GUI corpora remain limited in coverage and diversity, particularly for desktop environments. ScaleCUA aims to train cross-platform GUI agents and open-source data to advance the field.

Vision-Language Models (VLM)

  • General VLMs increasingly gain GUI knowledge during pretraining or SFT, yet can struggle with easy computer-use tasks, and their training GUI corpora remain largely proprietary.Source: §2 · p. 3

GUI Agents

  • Modular GUI agents separate planning and grounding, while native agents predict executable actions end-to-end; agentic workflows incur latency and token costs and depend on VLM capability.Native agents are described as offering tighter perception-action alignment; the paper aims to train cross-platform base models and open-source data.Source: §2 · p. 3

GUI Datasets

  • Open GUI datasets cover mobile, web, desktop, and tutorial-derived interactions, yet remain limited in coverage and diversity, particularly for desktop environments.The cited work also identifies data scarcity as a challenge for UI-element grounding and multi-step task planning.Source: §2 · p. 4

3 CROSS-PLATFORM INTERACTIVE DATA PIPELINE

ScaleCUA addresses costly, low-quality GUI trajectory collection with a cross-platform pipeline combining agent interaction and human expertise. It curates the resulting data into supervision for GUI understanding, grounding, and task planning.

3.1 OVERVIEW

  • The pipeline pairs agent interaction with human and autonomous trajectory acquisition, then annotates screenshots and structural metadata into training corpora.The two loops are the Agent-Environment Interaction Loop and the Agent-Human Hybrid Data Acquisition Loop.Source: Figure 2 · p. 4, §3.1 · p. 5
  • A unified interface spans Windows, Ubuntu, macOS, browsers, Android, and iOS, using platform-specific metadata extraction, OmniParser for inaccessible or incomplete layouts, and a shared action space.The action space combines universal operations with platform-specific actions, supporting consistent behavior modeling across desktop, browser, and mobile environments.Source: §3.1 · pp. 5–6

3.2 TASK-ORIENTED DATA CURATION

  • Curation yields GUI Understanding, GUI Grounding, and Annotated Trajectories, including perception and reasoning tasks, point/box/action grounding, and reasoning-prompt trajectory enrichments.Understanding covers element-level appearance, text, layout, function, and intent, plus interface and screen-transition captions; grounding is expanded through LLM-driven augmentation.Source: §3.2 · pp. 6–7
  • Rule-driven DFS exploration supplies weak-semantic flows, while expert operators provide goal-directed demonstrations across daily, entertainment, and productivity tasks.The explored sequences are segmented by screen similarity; expert demonstrations provide high-quality supervision for reasoning and planning.Source: §3.1 · p. 5, §3.2 · p. 6

3.3 DATA STATISTICS

  • 471K understanding examples, over 17.1M grounding annotations, and 19K task-planning trajectories averaging 9 steps make up the corpus.The trajectories include both human-curated and enhanced examples.Source: §3.3 · p. 7
  • Over 2M raw screenshots support a dataset spanning desktop, mobile, and web, with three task families in a unified format.Desktop tasks dominate, while mobile and web environments are also substantially represented.Source: §3.3 · p. 7

4 SCALING UP COMPUTER USE AGENTS

Section 4 defines screenshot-grounded GUI interaction, presents three inference modes, and specifies ScaleCUA’s model training configurations and general-data mixtures.

4.2 BASE AGENT MODELS

  • ScaleCUA uses Qwen2.5-VL in grounding, direct-action, and reasoned-action modes, trading localization, fast executable control, and more reliable, interpretable reasoning against added latency.Grounding focuses on locating UI elements; direct action supports a streamlined perception-action loop, while reasoned action adds explicit rationale that is useful for ambiguous or long-horizon tasks but increases computational cost and latency.Source: Figure 4 · p. 8, §4.2 · pp. 8–9

4.1 TASK DEFINITION

  • The agent acts in an environment from a task instruction, observations, and prior action-observation history, then executes parameterized actions; ScaleCUA uses screenshots to avoid noisy accessibility-tree and DOM inputs.Observations may include screen pixels, accessibility trees, or DOM data, but this work models tasks from screenshots to align with human behavior.Source: §4.1 · p. 8

4.3 TRAINING RECIPES

  • Training uses 128 A100 GPUs for the 3B and 7B models and 128 H200 GPUs for the 32B model; all use a 1.0 × 10−5 learning rate and 40960-token maximum length.The 3B setup uses mini-batch size 4 with accumulation 1; 7B and 32B use mini-batch size 2 with accumulation 2, yielding effective batch size 4 for 32B.Source: §4.3 · p. 9
  • General-purpose data makes up 25%, 50%, and 75% of GUI-specific training corpora for the 3B, 7B, and 32B models, respectively, with larger models receiving higher shares.The authors report strong GUI-agent benchmark performance for the 3B model with 25% general data and raise the share for larger models because they can memorize more samples.Source: §4.3 · p. 9

5 EXPERIMENTS

ScaleCUA is evaluated across GUI understanding, grounding, and end-to-end task completion, with analyses of data choices and inference trade-offs. Results show strong performance across platforms, while diagnostic analyses identify benchmark-specific effects of resolution, reasoning, and data composition.

5.1 EVALUATION SETUP

  • Evaluations use pure visual observation across GUI understanding, grounding, and task-completion benchmarks spanning desktop, mobile, and web environments.The setup includes offline AndroidControl and online OSWorld, WindowAgentArena, MacOSArena, AndroidWorld, and WebArena-Lite-v2; the latter upgrades WebArena-Lite to address tasks unsuitable for pure visual evaluation or requiring actions beyond the desktop environment.Source: §5.1 · pp. 9–11

5.2.1 GUI UNDERSTANDING

  • 94.4% on MMBench-GUI L1 hard-level tasks lets ScaleCUA-32B outperform all proprietary and open-sourced models.ScaleCUA-3B attains 83.6% on easy tasks, surpassing Qwen2.5-VL-72B by +16.6 points; ScaleCUA-32B scores 92.5% on medium tasks and matches GUI-Owl-32B on easy tasks at 92.5%.Source: Table 2 · p. 10, §5.2.1 · p. 11

5.2.2 GUI GROUNDING

  • 59.2% on ScreenSpot-Pro and 94.7% on ScreenSpot-v2 are ScaleCUA-32B’s reported grounding scores, with the model described as state of the art across benchmarks.ScaleCUA-3B reaches 89.2% on ScreenSpot-v2, outperforming Qwen2.5-VL-7B by +0.4%; ScreenSpot-Pro results are strong across creative, CAD, and office domains.Source: §5.2.2 · pp. 11–12, Table 4 · p. 12

5.2.3 TASK COMPLETION

  • 47.4% on WebArena-Lite-v2 (50 steps) and 44.2% (15 steps) make native ScaleCUA-32B strongest there, beating UI-TARS-72B-DPO by +26.0 and +20.8 points.On WindowsAgentArena (15 steps), ScaleCUA-32B reaches 24.2% versus UI-TARS-72B-DPO at 17.9%; GPT-4o with ScaleCUA-7B as grounder reaches 48.3% on AndroidWorld and 35.1% on WebArena-Lite-v2 (50 steps). Gains scale monotonically from 3B to 32B on Windows and Web, while macOS results remain low and AndroidWorld favors planner–grounder workflows.Source: §5.2.3 · p. 12, Table 5 · p. 13

5.3 ABLATION ON OUR DATASET

  • Data augmentation raises ScreenSpot-Pro performance from 37.8% to 41.3%, while raw coordinates outperform normalized coordinates, 42.3% versus 37.9%.At 2K, ScreenSpot-Pro reaches 45.5% versus 42.3%, while OSWorld-G is 52.5% versus 54.3%; higher resolution slightly reduces OSWorld and AndroidWorld success. Rule-based weak semantic trajectories provide low-cost interface-navigation supervision despite lacking explicit high-level goals.Source: §5.3 · p. 13, Table 6 · p. 14

5.4 DIAGNOSTIC ANALYSIS ON AGENTS

  • Reasoned Action Mode improves success rates by +1.4% to +8.2% over Direct Action Mode across benchmarks, at the cost of higher inference latency.Grounding accuracy rises from 720p to 2K with diminishing returns at 4K; ScreenSpot-Pro benefits most, while evaluation above 1080p does not improve ScreenSpot-v2 or OSWorld-G and may slightly degrade them. Increased training data benefits all benchmarks, with most gains saturating beyond 75%, except WebArena-Lite-v2, which improves nearly linearly; adding general multimodal data improves general benchmarks but gradually reduces GUI performance, peaking around a 75% general-data ratio for larger models.Source: §5.4 · pp. 13–14, Figure 5 · p. 14

6 CONCLUSION

ScaleCUA combines large-scale cross-platform data, a standardized action space, and a dual-loop pipeline to train and evaluate computer use agents. Its models set records across GUI understanding and grounding tasks, though long-horizon reasoning and cross-application generalization remain limitations.

6 CONCLUSION

  • ScaleCUA significantly outperforms open-source and proprietary baselines, setting records in GUI understanding and grounding and performing strongly in realistic interactive environments.ScaleCUA-32B achieves state-of-the-art GUI-understanding accuracy across platforms and difficulty levels; results include ScreenSpot-v2, ScreenSpot-Pro, OSWorld-G, L2-GUI grounding, and online benchmarks.Source: §6 · p. 15
  • ScaleCUA unifies training and evaluation across desktop, mobile, and web through large-scale cross-platform data and a standardized action space.Its dual-loop pipeline combines automated agent-environment interaction with human-in-the-loop annotation to build benchmarks for GUI understanding, element grounding, and task completion.Source: §6 · p. 15
  • Long-horizon reasoning and cross-application generalization remain limitations, while ScaleCUA’s released resources are intended to support future work on capable, trustworthy, deployable agents.The work frames scaling specialized training pipelines as promising for robust, general-purpose computer use systems.Source: §6 · p. 15

LIMITATIONS

ScaleCUA establishes a cross-platform GUI-agent foundation, but limitations remain in data collection, agentic capabilities, and memory. Further work is needed for closed-loop learning and high-level reasoning.

Limitations

  • Agent-collected trajectories remain weaker than expert-curated data, while expert crowdsourcing is costly to scale, leaving an effective self-improving collection-and-refinement loop unresolved.Rule-based exploration often produces semantically weak trajectories.Source: LIMITATIONS · p. 15
  • The study omits reflection, memory-based decisions, hierarchical planning, Policy Reward Models, and Reinforcement Learning, which may improve reasoning and adaptability but require strong base models.These approaches are left unexplored as the work focuses on robust cross-platform base models.Source: LIMITATIONS · p. 15
  • The current memory mechanism stores previous operations as flat history, limiting long-horizon reasoning and potentially causing suboptimal execution; future work should capture temporal dependencies.The paper identifies lightweight, effective memory systems as an important direction.Source: LIMITATIONS · p. 15
  • The framework provides a unified foundation for GUI-agent training, but closed-loop learning and high-level reasoning remain necessary for fully autonomous computer-use agents.The paper presents base agent models and a cross-platform trajectory-collection pipeline while acknowledging remaining limitations.Source: LIMITATIONS · p. 15

A APPENDIX · A.1 MORE RESULTS

The appendix reports additional ScaleCUA results across grounding, Android planning, scientific planning, and general VLM benchmarks. It also examines how adding general-purpose training data affects benchmark performance.

A.1 MORE RESULTS

  • On MMBench-GUI L2 (Wang et al., 2025c), ScaleCUA-32B scores 96.4 on Android, 93.9 on Web, 81.2 on Linux, and 88.1 on macOS in the basic setting.It scores 76.3 on Web and 81.7 on Android in the advanced setting; ScaleCUA-7B and ScaleCUA-3B average 78.2 and 73.7, respectively.Source: §A.1 · p. 22, Table 7 · p. 22
  • 60.6 overall on OSWorld-G (Xie et al., 2025) is ScaleCUA-32B’s Ubuntu-grounding result, with 70.0 layout understanding, 66.7 element recognition, and 51.0 fine-grained manipulation.All ScaleCUA models underperform on Refusal because Refusal-specific JEDI training data was excluded; the paper says including it could bias models toward refusal states and diminish active exploration, reducing task success.Source: §A.1 · pp. 22–23, Table 8 · p. 23
  • On AndroidControl (Li et al., 2024b), ScaleCUA-7B has the highest Low task completion rate, while ScaleCUA-32B has the most reliable grounding and the highest High success rate.ScaleCUA-32B also has the smallest degradation from Low to High; the smaller models sustain a favorable trade-off across both settings. Small type-prediction variance across sizes suggests residual failures stem more from long-horizon interaction and error accumulation than intent misclassification or localization.Source: §A.1 · p. 23, Table 9 · p. 23
  • On ScienceBoard (Sun et al., 2025), ScaleCUA-32B outperforms GPT-4o (1.6) but trails Qwen2.5-VL-72B (12.9) and Claude-3.7-Sonnet (10.5).The models perform better in domains requiring factual and visual-text reasoning than in those requiring specialized symbolic workflows.Source: §A.1 · p. 24, Table 10 · p. 24
  • For transfer evaluation, the authors augment training with InternVL3 post-training data (Zhu et al., 2025) spanning OCR, mathematics, coding, reasoning-QA, and multimodal understanding.The four general multimodal benchmarks assess mathematical and commonsense reasoning, text comprehension, and open-domain visual question answering; “ScaleCUA-3B (25%)” denotes general-purpose data at 25% of core GUI data.Source: §A.1 · p. 24, Table 11 · p. 24
  • With 50% general-purpose data, ScaleCUA-3B rises from 52.8 to 58.7 on MathVista and from 48.8 to 52.4 on MMMU, while maintaining stable RealWorldQA performance.Increasing the ratio to 75% or 100% gives only marginal or inconsistent benefits; scaling model capacity amplifies the gains, although the 7B and 32B models remain substantially below the general-benchmark baseline.Source: §A.1 · p. 24, Table 11 · p. 24

A.2 PUBLIC DATA USED IN TRAINING

The section summarizes the public datasets used to train ScaleCUA, with statistics referring to the portions used in the experiments rather than the original source-dataset sizes.

A.2 PUBLIC DATA USED IN TRAINING

  • Table 12 summarizes the public training datasets, and its statistics cover only the portions used in the experiments, not the original dataset sizes.Source: §A.2 · p. 25

A.3 ACTION SPACE

ScaleCUA defines a shared, platform-aware action space that supports semantically consistent GUI control across desktop, mobile, and web environments.

A.3 ACTION SPACE

  • The unified action space supports Desktop (Windows, macOS, Ubuntu), Mobile (Android, iOS), and Web platforms through a platform-aware but semantically consistent interface.Table 13 presents the action space.Source: §A.3 · p. 25
  • Its commands include shared operations such as click, write, wait, and terminate, alongside platform-specific controls such as mobile swipe and long press and desktop or web mouse and keyboard actions.Web also supports swipe for modern interactive elements.Source: §A.3 · p. 25
  • Explicit action arguments, including coordinates and keypresses, enable structured outputs across grounding, direct-action, and reasoned-action inference modes.The shared interface is intended to simplify training and inference and support modular training, policy transfer, and scalable data annotation.Source: §A.3 · p. 25

A.4 ERROR CASE ANALYSIS

ScaleCUA’s errors across desktop, Android, and web reveal weaknesses in tracking UI state, following task constraints, and recovering when actions fail silently. The paper proposes verification, recovery-aware reinforcement learning, and memory-augmented planning as potential remedies.

A.4 ERROR CASE ANALYSIS

  • On desktop, ScaleCUA skips procedural prerequisites such as selecting files or highlighting text, then can repeat ineffective actions when silent failures leave the interface unchanged.These failures point to incomplete understanding of interface states and sub-task dependencies, and a lack of robust error recovery.Source: §A.4 · p. 26, Figure 6 · p. 26, Figure 7 · p. 27
  • On Android, ScaleCUA ignores positional constraints when editing text and repeats actions when it fails to detect visual feedback or confirm a state transition.Examples include inserting text at the current cursor rather than the top of a note and repeating a camera shutter action.Source: §A.4 · pp. 27–28, Figure 8 · p. 27
  • On the web, ScaleCUA selects visually salient but instruction-inconsistent items, overlooking explicit constraints such as product category, post ownership, or topical relevance.The analysis identifies this as a semantic-disambiguation weakness and a bias toward superficial cues.Source: §A.4 · p. 28, Figure 9 · p. 28
  • The paper proposes screen-change detectors and visual precondition checks, recovery-aware reinforcement-learning rewards, and episodic memory to prevent ineffective loops and support task progress.These avenues target action-effect verification, satisfaction of prerequisites before proceeding, and recall of past interaction outcomes.Source: §A.4 · p. 28

A.5 THE DETAILS OF DATA CURATION · A.5.1 DATA SOURCES

ScaleCUA-Data draws on real-world GUI interfaces across desktop, mobile, tablet, and web environments. Its broad cross-platform coverage is described as supporting comprehensive domain knowledge and improved generalization.

A.5.1 DATA SOURCES

  • The collection spans seven listed platforms—Windows, Ubuntu, macOS, iOS, iPadOS, Android, and Web—with applications across productivity, communication, entertainment, browsing, and utilities.Source: §A.5.1 · p. 28
  • Desktop sources include Windows native and third-party applications, plus open-source and system software on Ubuntu and macOS.Examples include Microsoft Office Suite, Adobe Creative Cloud, and Visual Studio on Windows, and LibreOffice, GIMP, Terminal, Finder, and Safari on Ubuntu and macOS.Source: §A.5.1 · p. 28
  • Mobile data covers iOS system and third-party apps, while Android contributes the greatest diversity of sources across system and commercial software.Examples cited include Weibo, Notability, and Spotify on iOS, and WeChat, Taobao, TikTok, and Google Suite on Android.Source: §A.5.1 · p. 29
  • Tablet collection focuses on frequently used iPadOS system applications.Source: §A.5.1 · p. 29
  • Web data covers more than 200 frequently accessed sites across six domains, captured through static DOM snapshots and dynamic interaction traces.The domains include e-commerce, social media, education, government services, travel, and developer tools; examples include Amazon, YouTube, Reddit, Wikipedia, Coursera, and GitHub.Source: §A.5.1 · p. 29
  • The high-coverage corpus of real-world graphical interfaces is described as giving the model comprehensive domain knowledge and significantly improving generalization.Source: §A.5.1 · p. 29

A.5.2 GUI UNDERSTANDING

The GUI-understanding corpus covers element- and screenshot-level semantics through tasks for fine-grained perception, grounding, and reasoning about static and dynamic interfaces.

GUI UNDERSTANDING

  • Five element-level tasks cover appearance captioning, referring OCR, layout understanding, functionality captioning, and user-intention prediction.They target visual attributes, text grounding, spatial relations, component roles, and likely user goals from interface context and interactions.Source: §A.5.2 · p. 29
  • Two screenshot-level tasks address holistic interface comprehension through interface captioning and temporal change through screen-transition captioning.The latter describes differences between consecutive screenshots, including state updates, navigation events, or content refreshes.Source: §A.5.2 · p. 29
  • Vision-language models automatically generate annotations using visual context, structural metadata, and interaction histories for both task levels.The resulting corpus is intended to support fine-grained perception, robust grounding, and high-level reasoning in complex GUI environments.Source: §A.5.2 · p. 29

A.5.3 METADATA EXTRACTION … A.9 PROMPT ENGINEERING

This section describes the methods and resources behind ScaleCUA’s cross-platform data construction, including platform-specific UI extraction, exploration, annotation, and WebArena-Lite-v2 refinements. It also documents and releases prompts for agent operation and annotation, including a structured system prompt for three inference modes, to support reproducibility and transparency.

A.5.3 METADATA EXTRACTION

  • Windows, Ubuntu, and macOS pipelines traverse accessibility trees and filter or flatten elements into semantic, coordinate-aware metadata; macOS supplements limited accessibility data with screenshot-aligned OmniParser V2.The platform pipelines use UI Automation, XML parsing, and the macOS Accessibility API, respectively.Source: §A.5.3 · pp. 30–31
  • Android parses accessibility XML into actionable records, iOS uses OmniParser V2, and web extraction combines Selenium/CDP classification with geometric and visibility validation.The web pipeline also gathers descriptions from multiple element attributes and filters elements during random-walk exploration.Source: §A.5.3 · pp. 31–32

A.5.4 GUI GROUNDING

  • The GUI grounding corpus supervises point, bounding-box, and action targets, and uses GPT-4o-generated annotations to expand coverage across layouts and interaction patterns.Annotations reuse UI appearance, spatial, and functional descriptions, with action grounding pairing spatial targets and atomic operations.Source: §A.5.4 · pp. 32–33

A.5.5 WEAK-SEMANTIC TRAJECTORY

  • Rule-driven exploration is segmented at screenshot-similarity boundaries into weak-semantic trajectories that preserve partial continuity and expose agents to common GUI flows and state transitions.Heuristics encourage deeper, less-visited states, while similarity-based boundaries restart exploration at terminal or redundant states.Source: §A.5.5 · p. 33, §A.6.3 · p. 37

A.5.6 HUMAN-CURATED TRAJECTORY

  • Human-curated goal-oriented trajectories complement automated data, addressing stochastic exploration, missed context-specific operations, and fragmented or noisy action sequences.Experts record coherent tasks across platforms in isolated Docker environments, producing reference paths for aligning agents with human intent and behavior.Source: §A.5.6 · p. 33

A.5.7 ANNOTATION SCHEMES

  • The annotation scheme deduplicates screenshots, samples 25–40 elements, generates element and transition descriptions, and labels trajectory steps with separate XML operation and reasoning tags.It also augments instructions and step-level reasoning to address writing-style variation and model-labeling noise.Source: §A.5.7 · p. 34, §A.6.5 · p. 39

A.5.8 MORE DETAILS OF DATA DISTRIBUTION

  • 471.4K GUI Understanding examples cover 355.5K screenshots, while over 17.1M GUI Grounding examples span 1.6M images.The grounding examples include point, bounding-box, and action formats.Source: §A.5.8 · p. 34, Table 15 · p. 35
  • 19K unique Task Planning trajectories include 4K expert-curated demonstrations and 48.2K Enhanced examples, alongside Weak Semantics trajectories from rule-based exploration.The trajectory composition is organized across platforms and trajectory types.Source: §A.5.8 · p. 34, Figure 10 · p. 35

A.7.1 COMMON PROBLEMS

  • Common collection problems are addressed with Adaptive Bounding Box Tightening to align click targets with content and history-based filtering to steer random walks toward novel pages.ABT contracts boxes around detected uniform-color whitespace, improving alignment especially on simple solid-color backgrounds; exploration filters elements matching the previous screenshot by text and IoU to avoid revisiting persistent components.Source: §A.7.1 · pp. 40–41

A.7.2 WINDOWS

  • Windows data collection adapts APIs for Java applications, filters noisy trees and redundant or occluded captures, and uses DFS-inspired random walks to favor newly appeared UI elements.The specialized Java Access Bridge supplies complete trees where Win32 parsing fails; geometric refinement detects occluded elements, and exploration can briefly follow an external application before returning to the primary app.Source: §A.7.2 · pp. 41–43

A.7.3 UBUNTU

  • Ubuntu denoises accessibility trees, corrects occlusion and coordinate errors, and improves exploration by limiting actions to interactive elements and prioritizing newly appearing ones.Filtering removes structural roles and occluded containers; short random interactions can recalibrate offset coordinates. If an action reveals no new elements, exploration falls back to unvisited known elements in the current state.Source: §A.7.3 · pp. 44–46

A.7.4 MACOS

  • macOS extraction prunes and tightens interactive accessibility-tree nodes, then fuses vision-detected boxes with the tree to recover controls missing from private system-panel layers.The pipeline maps logical coordinates to device pixels, removes tiny or uninformative boxes, and merges overlapping text and interactive nodes; Omniparser detections are retained above an IoU threshold of 0.15 or when selected during exploration.Source: §A.7.4 · pp. 46–47

A.7.5 MOBILE

  • Mobile data collection reduces occlusion errors, restores missing clickable attributes, and resolves semantic and cross-platform functional ambiguity through hierarchy and click-priority heuristics.The pipeline uses uiautomator2.dump_hierarchy() to prune occluded nodes and improve hierarchy coverage, inherits clickable status from parents only when all descendants are non-clickable, and prioritizes leaf nodes and icons in icon-text pairs.Source: §A.7.5 · p. 48

A.7.6 WEB

  • Web collection simulates missing native browser UI and supplements automation-tool data with native-desktop trajectories, while addressing page heterogeneity, dynamic states, and weak element semantics.The authors advocate prioritizing recall of valid interactive elements and aggregating text, alt, title, and aria-label features to ground annotations. They recommend native desktop environments for future web-task research because missing browser UI can constrain exploration.Source: §A.7.6 · pp. 48–50

A.8.1 CLARIFICATIONS ON WEB ONLINE BENCHMARK

  • WebArena-Lite-v2 offers 154 refined local tasks designed for headed and headless desktop action, pure-vision evaluation, and flexible answer validation.The authors favor local environments for stability and accessibility, citing online-site changes, anti-automation barriers, and uneven access across regions. They also revised tasks and evaluation functions, while noting that some evaluations remain static.Source: §A.8.1 · pp. 50–51

A.9 PROMPT ENGINEERING

  • The prompt suite covers data filtering, annotation, and ScaleCUA agent tasks, including GUI understanding, grounding, trajectory annotation, reasoning traces, and alternative actions.Prompts are tailored to VLMs such as GPT-4o and Claude-3.7 to support downstream training; releasing them is intended to aid reproducibility and transparency.Source: §A.9 · p. 52

A.9.1 PROMPTS FOR OUR AGENT

  • A shared system-prompt template defines action semantics for Grounding, Direct-Action, and Reasoned-Action modes, with inputs and outputs structured around screenshots, task context, and executable actions.Direct-Action Mode also includes past interaction history and formats its response as an intended operation followed by executable commands.Source: §A.9.1 · pp. 52, 54, 55
  • The authors propose decoupling action-space definitions from core prompt logic to support plug-and-play extensions, including device-specific actions such as mobile swipe or desktop hotkey.They describe this modularity as a foundation for a unified prompting framework adaptable to diverse GUI control systems.Source: §A.9.1 · p. 52
  • The agent prompt takes a screenshot, task description, and interaction history as input, then requests the next move in tagged reasoning, operation, and executable-action fields.The prompts also instruct the agent to avoid invalid states and actions outside the defined action space.Source: §A.9.1 · p. 56

A.9.2 PROMPTS FOR ANNOTATIONS

  • Annotation prompts cover trajectory filtering, GUI understanding, grounding supervision, and chain-of-thought generation, with the authors reporting that their labels benefit agent training.They use VLMs such as GPT-4o and Claude-3.7, and the authors aim to standardize annotation practices and lower barriers to research.Source: §A.9.2 · p. 57
  • Element and interface prompts request distinctive visual and spatial descriptions, interaction affordances, and functional context while avoiding direct copying of A11Tree attributes or reference markers.Transition prompts infer state changes and user intent from consecutive screenshots and an operation; other templates derive high-level navigation goals, low-level instructions, and rationales from trajectories and task context.Source: §A.9.2 · pp. 57–58
  • An LLM-assisted filter judges whether a marked area is a valid, clickable focus element, treating obscured, noninteractive, empty, erroneous, or anomalous regions as non-clickable.The web-acquisition examples illustrate misclassified unclickable elements and hierarchy or visibility filtering failures, alongside successful hierarchy analysis.Source: §A.9.2 · pp. 59, 62, Figure 30 · p. 62

Related papers

Loading 2509.15221v2…