Source-linked AI summary
ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data
Zhaoyang Liu, Jingjing Xie, Zichen Ding, Zehao Li, Bowen Yang, Zhenyu Wu, Xuehui Wang, Qiushi Sun, Shi Liu, Weiyun Wang, Shenglong Ye, Qingyun Li, Xuan Dong, Yue Yu, Chenyu Lu, YunXiang Mo, Yao Yan, Zeyue Tian, Xiao Zhang, Yuan Huang, Yiqian Liu, Weijie Su, Gen Luo, Xiangyu Yue, Biqing Qi, Kai Chen, Bowen Zhou, Yu Qiao, Qifeng Chen, Wenhai Wang
TL;DR
Computer-use agents remain limited by scarce, costly, and inaccessible open data and foundation models. ScaleCUA builds cross-platform data with automated agents and human experts, trains unified agents with flexible inference modes, and reports strong cross-benchmark performance, including state-of-the-art results.
Problem
Computer-use research lacks large-scale, open, diverse trajectories and accessible foundation models, while existing data is costly to collect and annotate.
Method
ScaleCUA combines a dual-loop agent–environment and agent–human data pipeline with a six-platform, three-domain dataset and unified agent action space.
Results
ScaleCUA significantly outperforms open-source and proprietary baselines and achieves state-of-the-art accuracy across GUI understanding platforms and difficulty levels.
Takeaways & Limitations
Cross-platform data scaling and standardized actions provide a unified foundation for training and evaluating computer-use agents.
Takeaways & Limitations
Agent-collected data remains lower quality than expert crowdsourcing, while reflection, memory, hierarchical planning, PRMs, and reinforcement learning are not incorporated.
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) have enabled computer use agents (CUAs) that operate GUIs autonomously, showing great potential, yet progress is limited by the lack of large-scale, open-source computer use data and foundation models. In this work, we introduce ScaleCUA, a step toward scaling open-source CUAs. It offers a large-scale dataset spanning 6 operating systems and 3 task domains, built via a closed-loop pipeline uniting automated agents with human experts. Trained on this scaled-up data, ScaleCUA can operate seamlessly across platforms. Specifically, it delivers strong gains over baselines (+26.6 on WebArena-Lite-v2, +10.7 on ScreenSpot-Pro) and sets new state-of-the-art results (94.4% on MMBench-GUI L1-Hard, 60.6% on OSWorld-G, 47.4% on WebArena-Lite-v2). These findings underscore the power of data-driven scaling for general-purpose computer use agents. We will release data, models, and code to advance future research: https://github.com/OpenGVLab/ScaleCUA.
1 INTRODUCTION
ScaleCUA addresses the scarcity of open computer-use data by combining cross-platform collection with human expertise and training unified agents for GUI understanding, grounding, and action. The resulting models support flexible inference and achieve competitive or state-of-the-art performance across cross-platform evaluations.
- Motivation: The dataset addresses scarce, costly, and rapidly obsolescing computer-use trajectories that are less available than ordinary image–text data.
- Approach: The pipeline combines automated agent interaction with expert-annotated trajectories to improve coverage and quality across heterogeneous GUI environments.
- Contributions: ScaleCUA introduces a cross-platform dataset covering six platforms and three GUI-centric task domains for training universal computer-use agents.The domains are understanding, grounding, and task completion.
- Models: ScaleCUA unifies perception, reasoning, and action while supporting grounding, direct-action, and reasoned-action inference modes.
- Evaluation: The study evaluates understanding, grounding, and end-to-end task completion across cross-platform online benchmarks, reporting state-of-the-art or competitive performance.
2 RELATED WORK
Prior work spans VLM-based GUI agents and datasets for mobile, web, and desktop interaction. However, the surveyed resources generally emphasize particular platforms, task types, or collection strategies rather than a unified cross-platform scope.
- GUI Agents: VLM-based GUI agents commonly separate high-level planning from specialized grounding to operate graphical interfaces.
- GUI Datasets: Existing datasets provide substantial mobile, web, and desktop interaction resources, including screens, demonstrations, trajectories, and grounding examples.
- GUI Datasets: The related resources motivate broader datasets that combine heterogeneous platforms, task types, and interaction supervision.
3 CROSS-PLATFORM INTERACTIVE DATA PIPELINE
The Cross-Platform Interactive Data Pipeline combines standardized agent–environment interaction with agent–human acquisition to build diverse GUI training data. It spans multiple platforms and task types, producing large-scale supervision for understanding, grounding, and planning.
- Overview: The pipeline integrates automated exploration and expert collection through two synergistic loops, then annotates screenshots and structural metadata into training corpora.
- Overview: The unified interface spans desktop, browser, and mobile environments, while a unified action space combines universal operations with platform-specific controls.
- Data Acquisition: Human-curated trajectories provide realistic, goal-directed demonstrations for reasoning and planning, while weak-semantic trajectories offer low-cost supervision for interface navigation.
- Data Statistics: The corpus contains 471K understanding examples, over 17.1M grounding annotations, and 19K task-planning trajectories averaging 9 steps.
- Data Statistics: The dataset covers understanding, grounding, and task planning across desktop, mobile, and web, whereas prior resources typically focus on a single task domain.
- Data Statistics: The pipeline collects over 2M raw screenshots and uses human, VLM-driven, and rule-driven strategies to broaden platform and GUI-state coverage.
4 SCALING UP COMPUTER USE AGENTS
ScaleCUA models formulate GUI interaction as sequential agent–environment control over visual observations and histories. They provide grounding, direct-action, and reasoned-action modes, with design choices balancing speed, interpretability, and computational cost.
- Task Definition: The agent maps a task, current observation, and interaction history to an action executed by the environment.
- Inference Paradigms: ScaleCUA offers grounding, direct-action, and reasoned-action paradigms for locating UI elements, emitting executable controls, or reasoning before acting.
- Inference Paradigms: Direct Action Mode removes intermediate reasoning to support rapid perception–action loops and efficient execution in time-sensitive GUI tasks.
- Inference Paradigms: Reasoned Action Mode adds an explicit chain-of-thought before each action, improving behavior quality and interpretability while increasing computational cost and latency.
- Training: The models are trained with varying proportions of general-purpose multimodal data, with the 3B, 7B, and 32B configurations using 25%, 50%, and 75%, respectively.
5 EXPERIMENTS
ScaleCUA is evaluated across GUI understanding, grounding, and end-to-end task completion on diverse cross-platform benchmarks. The results show strong performance gains from cross-platform GUI-specific data, while ablations reveal trade-offs involving resolution, reasoning, data scale, and multimodal mixing.
- 5.1 EVALUATION SETUP: ScaleCUA is evaluated with pure visual observation across understanding, grounding, and task-completion benchmarks spanning multiple platforms.The evaluation includes MMBench-GUI L1, ScreenSpot-v2, ScreenSpot-Pro, OSWorld-G, AndroidControl, and online GUI interaction environments.
- 5.2.1 GUI UNDERSTANDING: 94.4% on MMBench-GUI L1 hard-level tasks makes ScaleCUA-32B outperform all proprietary and open-source models evaluated.ScaleCUA-3B reaches 83.6% on easy tasks, while ScaleCUA-32B reaches 92.5% on medium tasks.
- 5.2.2 GUI GROUNDING: 59.2% overall on ScreenSpot-Pro and 94.7% average on ScreenSpot-v2 establish ScaleCUA-32B as a leading GUI grounding model.ScaleCUA-3B reaches 89.2% on ScreenSpot-v2, outperforming Qwen2.5-VL-7B by +0.4%.
- 5.2.3 TASK COMPLETION: 47.4% on WebArena-Lite-v2 with a 50-step budget is ScaleCUA-32B’s strongest native result, exceeding UI-TARS-72B-DPO by +26.0 points.It also reaches 24.2% on WindowsAgentArena at 15 steps, while a GPT-4o planner with ScaleCUA-7B grounder reaches 48.3% on AndroidWorld.
- 5.3 ABLATION ON OUR DATASET: 41.3% on ScreenSpot-Pro with augmentation versus 37.8% without it, while raw coordinates reach 42.3% versus 37.9% for normalized coordinates.Higher resolution improves ScreenSpot-Pro but can slightly reduce OSWorld and AndroidWorld success rates.
- 5.4 DIAGNOSTIC ANALYSIS ON AGENTS: Reasoned Action Mode improves success rates over Direct Action Mode by +1.4% to +8.2%, but increases inference latency.Performance gains from additional training data generally saturate beyond 75%, whereas WebArena-Lite-v2 remains more data-sensitive.
6 CONCLUSION
ScaleCUA combines large-scale cross-platform GUI data with unified base agent models to support training and evaluating computer use agents. Experiments report strong performance across GUI understanding, grounding, and interactive benchmarks.
- ScaleCUA models consistently establish new performance records across GUI understanding, grounding, and online interaction benchmarks.The reported gains span multiple platforms and difficulty levels, including ScreenSpot-v2, ScreenSpot-Pro, OSWorld-G, L2-GUI grounding, and online benchmarks.
LIMITATIONS
The framework establishes a unified foundation for cross-platform GUI agents, but important limitations remain in data quality, agentic reasoning, memory, and closed-loop learning.
- Expert-curated data remains costly to scale, while rule-based agent exploration can produce semantically weak trajectories.The challenge is to couple automated collection with iterative model refinement in a self-improving loop.
- Advanced mechanisms such as reflection, memory-based decision-making, hierarchical planning, PRMs, and reinforcement learning remain unexplored.These techniques are identified as promising for reasoning, adaptability, and long-horizon decision making, but they require strong base models.
- The current flat-history memory limits long-horizon reasoning and may cause suboptimal executions.Future work should develop lightweight memory systems that capture temporal dependencies.
- Further work is needed to enable closed-loop learning and high-level reasoning for fully autonomous computer use agents.The authors identify these capabilities as critical for realizing more intelligent agents.
A.1 MORE RESULTS
Additional evaluations show that ScaleCUA performs strongly across grounding, planning, and general multimodal benchmarks, while revealing trade-offs across model sizes and data mixtures.
- MMBench-GUI L2: ScaleCUA-32B achieves leading basic-difficulty scores on MMBench-GUI L2 for Android, Web, Linux, and macOS, while remaining competitive on advanced settings.The reported basic scores are 96.4, 93.9, 81.2, and 88.1, respectively.
- OSWorld-G: 60.6 overall performance on OSWorld-G shows strong Ubuntu grounding, including 70.0 layout understanding, 66.7 element recognition, and 51.0 fine-grained manipulation.ScaleCUA-32B underperforms on Refusal because refusal-specific training data was excluded.
- AndroidControl: ScaleCUA variants perform consistently strongly on AndroidControl, with larger models favoring grounding reliability and smaller models favoring execution efficiency.ScaleCUA-32B has the highest high-complexity success rate and smallest degradation from Low to High.
- ScienceBoard: On ScienceBoard, ScaleCUA-32B surpasses GPT-4o at 1.6 but remains below Qwen2.5-VL-72B at 12.9 and Claude-3.7-Sonnet at 10.5.The models perform better on factual and visual-text reasoning than on specialized symbolic workflows.
- General VLM benchmarks: Increasing general-purpose data to 25–50% improves MathVista and MMMUvalid, but gains saturate at 75–100% and remain below the general-benchmark baseline for 7B and 32B models.For ScaleCUA-3B, MathVista rises from 52.8 to 58.7 and MMMU from 48.8 to 52.4 at 50% general-purpose data.
A.2 PUBLIC DATA USED IN TRAINING
The public-data summary reports the datasets used to train ScaleCUA and clarifies that statistics refer to the portions used in the experiments.
- Public data used in training: Table 12 summarizes the public datasets used for training ScaleCUA, reporting the portions actually used rather than the original source-dataset sizes.The distinction matters because the experimental pipeline processed and filtered source data.
A.3 ACTION SPACE
ScaleCUA uses a unified, semantic action space to support consistent control across desktop, mobile, and web platforms while retaining platform-specific operations.
- Action-space design: The unified action space abstracts low-level GUI operations into semantic commands shared across Windows, macOS, Ubuntu, Android, iOS, and Web.Universal actions include click, write, wait, and terminate, while platform-specific commands include swipe, long press, dragTo, and hotkey.
- Action-space design: Explicit action arguments such as coordinates and keypresses support precise structured outputs for grounding, direct-action, and reasoned-action inference.The design aims to simplify training and inference while supporting both generalization and specialization.
A.4 ERROR CASE ANALYSIS
Error analysis identifies failures in procedural prerequisite reasoning, visual state tracking, positional semantics, and semantic disambiguation across desktop, Android, and web tasks.
- Cross-platform failure patterns: ScaleCUA frequently repeats ineffective actions when UI feedback is silent, exposing weak state tracking and the absence of robust error recovery.Desktop examples include repeating compression or formatting actions after missing selection prerequisites; Android examples include repeated photo capture.
- Desktop: Desktop failures often violate procedural prerequisites, such as compressing files without selecting them or changing font styles without highlighting text.These errors reflect incomplete understanding of interface states and sub-task dependencies.
- Android: Android errors include ignoring positional instructions and repeating actions when the interface shows no obvious state change.The agent inserts text at the current cursor rather than the top of a note and repeats a camera action after an initial shutter press.
- Potential remedies: The analysis points to visual reasoning, memory-based state modeling, task-guided grounding, state verification, and recovery signals as promising remedies.Suggested mechanisms include precondition checking, screen-change detection, and rewards for predicate satisfaction before proceeding.
- Web: Web errors show semantic disambiguation problems, with the agent favoring visually salient but instruction-inconsistent items over explicit category, ownership, or topical constraints.Complex initial environments with many posts further challenge tasks requiring multiple filtering conditions.
A.5.1 DATA SOURCES
ScaleCUA-Data spans desktop, mobile, tablet, and web environments with broad application coverage. This high-coverage corpus is reported to improve generalization.
- ScaleCUA-Data spans seven operating systems, including Windows, Ubuntu, macOS, iOS, iPadOS, Android, and Web.
- The dataset covers productivity, communication, entertainment, browsing, and utility applications across platforms.
- Web data includes more than 200 frequently accessed sites, captured through static DOM snapshots and dynamic interaction traces.
- The resulting real-world GUI corpus provides broad platform coverage and is reported to improve model generalization.
A.5.2 GUI UNDERSTANDING
The GUI-understanding corpus combines element-level and screenshot-level tasks to support fine-grained perception, grounding, and reasoning over static and dynamic interfaces.
- Element-level understanding: The corpus defines five element-level tasks covering appearance, OCR, layout, functionality, and user-intention prediction.
- Screenshot-level understanding: Two screenshot-level tasks target holistic interface comprehension and temporal changes between consecutive screenshots.
- The combined tasks support fine-grained perception, robust grounding, and high-level reasoning in complex GUI environments.
- The corpus uses visual context, structural metadata, and interaction histories to automatically generate annotations with vision-language models.
A.5.3 METADATA EXTRACTION
ScaleCUA combines platform-specific metadata extraction, automated exploration, and human-curated trajectories to build a broad training corpus. The pipeline addresses automation gaps while retaining scalable weak-semantic data and high-quality supervision.
- Platform metadata extraction: Platform-specific extractors convert accessibility trees and interface attributes into structured metadata for automated GUI tasks.Windows, Ubuntu, macOS, and mobile platforms use distinct accessibility or hierarchy extraction procedures.
- Weak-semantic trajectories: Weak-semantic trajectories preserve partial continuity and structural coherence despite lacking alignment with manually defined tasks.
- Data-acquisition limitations: Automation remains limited by stochastic exploration, incomplete goal-directed coverage, and fragmented or noisy behaviors.
- Human-curated trajectories: Human-curated trajectories provide high-quality supervision and reference paths that align training with human planning and behavior.
- Corpus composition: The training corpus contains 471.4K GUI-understanding examples, over 17.1M grounding examples, and 19K unique task-planning trajectories.
- Web limitations: Web collection faces omitted native browser UI, heterogeneous websites, and temporal mismatches between parsing and screenshot capture.