PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenCUA: Open Foundations for Computer-Use Agents

Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, Jiarui Hu, Yuyan Wang, Jixuan Chen, Yuxiao Ye, Danyang Zhang, Dikang Du, Hao Hu, Huarong Chen, Zaida Zhou, Haotian Yao, Ziwei Chen, Qizheng Gu, Yipu Wang, Heng Wang, Diyi Yang, Victor Zhong, Flood Sung, Y. Charles, Zhilin Yang, Tao Yu

arXiv:2508.09123v3cs.AIcs.CV

TL;DR

Open CUA systems lack transparent, scalable infrastructure and data for studying their capabilities, limitations, and risks. OpenCUA provides annotation tools, broad datasets, processing pipelines, reflective reasoning, and training recipes; OpenCUA-72B reaches 45.0% on OSWorld-Verified, while the models generalize across domains and benefit from more test-time computation.

  • Problem

    Critical training data, architectures, and development processes for capable CUA systems remain closed, limiting transparent study of their capabilities, limitations, and risks.

  • Method

    OpenCUA combines cross-OS human-demonstration annotation, AgentNet datasets, state-action processing, reflective long CoT synthesis, and mixed-data training recipes.

  • Results

    45.0% success rate on OSWorld-Verified was achieved by OpenCUA-72B, establishing a new state-of-the-art among open-source models, with strong cross-domain generalization and test-time compute scalability.

  • Takeaways & Limitations

    OpenCUA releases its annotation tool, datasets, code, and models as open foundations for further CUA research.

  • Takeaways & Limitations

    OSWorld performance is sensitive to small initial environmental variations, even at temperature=0, producing divergent outcomes from nearly identical states.

Abstract

from arXiv · show

Vision-language models have demonstrated impressive capabilities as computer-use agents (CUAs) capable of automating diverse computer tasks. As their commercial potential grows, critical details of the most capable CUA systems remain closed. As these agents will increasingly mediate digital interactions and execute consequential decisions on our behalf, the research community needs access to open CUA frameworks to study their capabilities, limitations, and risks. To bridge this gap, we propose OpenCUA, a comprehensive open-source framework for scaling CUA data and foundation models. Our framework consists of: (1) an annotation infrastructure that seamlessly captures human computer-use demonstrations; (2) AgentNet, the first large-scale computer-use task dataset spanning 3 operating systems and 200+ applications and websites; (3) a scalable pipeline that transforms demonstrations into state-action pairs with reflective long Chain-of-Thought reasoning that sustain robust performance gains as data scales. Our end-to-end agent models demonstrate strong performance across CUA benchmarks. In particular, OpenCUA-72B achieves an average success rate of 45.0% on OSWorld-Verified, establishing a new state-of-the-art (SOTA) among open-source models. Further analysis confirms that our approach generalizes well across domains and benefits significantly from increased test-time computation. We release our annotation tool, datasets, code, and models to build open foundations for further CUA research.

1 Introduction

OpenCUA addresses the lack of transparent, scalable CUA infrastructure, data, and development processes with an open framework spanning annotation, datasets, processing, training, and evaluation. Its models show strong benchmark performance, including a 45.0% OSWorld-Verified success rate for OpenCUA-72B.

  • OpenCUA provides an open-source framework for studying CUA capabilities, limitations, and risks amid proprietary state-of-the-art systems.
  • Existing open-source CUA efforts lack scalable infrastructure for collecting diverse demonstrations and remain limited in GUI dataset scope and scale.
  • The framework combines cross-OS annotation, AgentNet data, state-action processing, reflective long CoT synthesis, and training recipes.
  • 45.0% success rate on OSWorld-Verified was achieved by OpenCUA-72B, establishing a new state-of-the-art among open-source models.

2 AGENTNET Collection

AGENTNET is built from natural human computer-use demonstrations collected across operating systems, applications, and websites, then converted into compact state-action trajectories. The resulting dataset contains 22,625 complex, multimodal tasks spanning Windows, macOS, and Ubuntu.

  • AGENTNET Collection: AGENTNET prioritizes natural user behavior across diverse computer environments and user scenarios through the AGENTNET TOOL annotation application.
  • AGENTNET TOOL: The annotation tool records screen videos, mouse and keyboard signals, and accessibility trees on annotators’ personal computers.
  • Trajectory Construction: Raw high-frequency interactions are compressed and paired with representative pre-action states to form compact state-action trajectories.
  • AGENTNET Statistics: 22,625 human-annotated tasks span 12K Windows, 5K macOS, and 5K Ubuntu trajectories, with resolutions from 720p to 4K.
  • AGENTNET Statistics: The dataset covers over 140 applications and 190 websites, with trajectories averaging 18.6 steps and often involving multi-app workflows.

3 Training Computer-Use Agent Model

OpenCUA trains computer-use models by augmenting compact state-action data with structured, reflective reasoning and carefully designed data mixtures. The recipe improves performance across model architectures and supports scaling with data and test-time computation.

  • Motivation: Directly fine-tuning VLA models on AGENTNET state-action trajectories yields only 4.4% success on OSWorld, motivating the proposed training recipe.
  • Reasoning Augmentation: The structured L3→L2→L1 CoT framework moves from visual observation to reflective planning and finally an executable action.
  • Reflective CoT Synthesis: The reflector checks demonstrations for incorrectness and redundancy, while the generator and summarizer synthesize reasoning for usable steps.
  • Context and Inference: L2 CoT is used at inference because its richer reasoning supports reflection, planning, and reasoning, with higher Pass@n scalability than Pass@1.
  • Training Data Mixtures: Training mixes grounding, planning, computer-use, and general vision-language or text data, with staged strategies varying by resources and model target.

4 Experiments

Experiments evaluate OpenCUA across online, offline, and GUI-grounding benchmarks, finding strong performance scaling with data and model size. OPENCUA-72B reaches 45.0% on OSWorld-Verified, the best result among open-source models.

  • Online agent evaluation: 45.0% average success rate on OSWorld-Verified makes OPENCUA-72B the best-performing open-source model.The result is reported at the 100-step budget and closes part of the gap to proprietary agents.
  • Test-time computation: OPENCUA-32B gains 4.4% from 15 to 50 steps but only 0.7% from 50 to 100 steps.The paper attributes limited larger-budget gains to unused extra steps and difficulty recognizing mistakes, recovering from errors, and stopping.
  • Test-time computation: 53.2% Pass@3 success for OPENCUA-72B rises from 45.0% Pass@1 on OSWorld-Verified.The increase suggests headroom for post-training, reranking, or multi-agent methods.
  • Offline agent evaluation: On AGENTNETBENCH, OpenAI CUA outperforms all open-source zero-shot models and approaches fine-tuned agents.It particularly excels at terminate-state detection and content-based actions; offline rankings generally match online rankings.
  • GUI grounding evaluation: 60.8% on ScreenSpot-Pro and 37.3% on UI-Vision demonstrate strong GUI grounding for OPENCUA-72B.OPENCUA-32B also reaches 55.3% on ScreenSpot-Pro, while OPENCU-7B reaches 50.0%.
  • Data scaling: 72% average improvement follows scaling Ubuntu data from 3K to 10K trajectories, while Win/Mac scaling from 3K to 14K yields 125%.Performance scales consistently across benchmarks with both in-domain and out-of-domain data.

5 Analysis

The analysis examines test-time computation, robustness, cross-platform generalization, reasoning formats, history representations, and data ablations. Results show gains from additional computation, cross-platform training, reflective reasoning, multi-image history, and mixed CoT data, alongside sensitivity to environment variation.

  • Test-time computation: Pass@16 raises OSWorld success from 16.9% to 34.6% at 15 steps and from 18.4% to 39.2% at 50 steps.These correspond to gains of +104% and +113%, respectively.
  • Robustness: Pass@16 reaches 38.60% at 50 steps versus 20.10% for Pass@1 under temperature 0, despite nearly identical initial states.The analysis identifies environment dynamics, minor omissions, and different solution paths as sources of outcome variance.
  • Cross-platform generalization: Ubuntu-trained models perform better on OSWorld, while Windows/macOS-trained models perform better on WindowsAgentArena and AGENTNETBENCH.The cross-domain gap narrows on WindowsAgentArena, suggesting partial transfer of application-level knowledge across operating systems.
  • Reasoning formats: L2 reasoning achieves the best inference performance, while training only on L2 data reduces performance to 13.1 compared with mixed L1, L2, and L3 training.The authors attribute L2’s inference advantage to higher-quality planning and reflection, while L3 can include irrelevant screenshot elements.
  • History representation: Multiple screenshots substantially improve performance over single-image inputs, whereas L2 textual history offers no benefit and may introduce distracting hallucinations.The adopted default is L1 CoT with three images, balancing performance and efficiency.
  • Data mixtures: General-domain text data slightly improves agentic performance.The ablation compares training with and without the 35% general-text mixture.
  • Reasoning formats: Reflective long CoT improves OSWorld performance from 11.5 to 15.3.The analysis associates this gain with reflection focused on error correction.

6 Related Work

Related work covers execution-level CUA benchmarks and three broad approaches to building agents: text-based, vision-centric, and modular agent frameworks.

  • CUA benchmarks and datasets: Execution-level benchmarks evaluate autonomous agents in genuine software environments across desktop and web tasks.Covered workflows include office productivity, code editing, file management, and operating-system configuration.
  • CUA frameworks and models: Text-based agents operate on structured GUI metadata such as DOM trees or accessibility labels and issue symbolic commands.These systems avoid relying on raw screen pixels.
  • CUA frameworks and models: Vision-centric agents integrate screen imagery, while modular frameworks add components such as vision encoders, planners, memory, and tool APIs.These approaches target long-horizon tasks requiring perception, reasoning, and control.

7 Conclusion

OPENCUA addresses critical gaps in computer-use agent development through an open framework whose released components support broader study of agent capabilities, limitations, and risks.

  • 7 Conclusion: OPENCUA provides annotation infrastructure, data-processing pipelines, diverse datasets, training recipes, and evaluation benchmarks.The framework is presented as a foundation for CUA research.
  • 7 Conclusion: The models demonstrate strong benchmark performance, data scaling laws, and cross-domain generalization capabilities.These conclusions are stated across the framework’s reported evaluations and analyses.
  • 7 Conclusion: Releasing tools, datasets, code, and models enables systematic investigation of CUA capabilities, limitations, and risks.The stated scope concerns agents mediating digital interactions and executing consequential decisions.

8 Aknowledgement

The authors acknowledge contributors, reviewers, infrastructure providers, and prior open-source projects supporting OPENCUA’s development.

  • 8 Aknowledgement: The authors thank named colleagues and anonymous reviewers for discussions, feedback, infrastructure support, and guidance.The acknowledgement names Yu Su, Caiming Xiong, Jin Zhang, Hao Yang, Zhengtao Wang, and Yanxu Chen.
  • 8 Aknowledgement: Moonshot AI provided training infrastructure and annotated data for the project.
  • 8 Aknowledgement: The annotation tool builds on the open-source projects DuckTrack and OpenAdapt.The authors express gratitude to their contributors.

A Limitations

AGENTNET’s scalability is constrained by human annotation capacity and by selection bias from requiring informed consent for data collection.

  • A Limitations: AGENTNET expansion beyond its current size requires additional human annotation resources despite AGENTNET TOOL streamlining collection.The paper suggests alternative data sources or semi-automated annotation as possible ways to address this constraint.
  • A Limitations: Informed-consent requirements exclude users who decline participation after considering potential risks, introducing selection bias.The authors accept this boundary to uphold responsible data collection.

B AGENTNETBENCH

AGENTNETBENCH evaluates computer-use agents through offline step-level action matching across diverse domains and operating systems, while relating offline performance to online task success.

  • B AGENTNETBENCH: AGENTNETBENCH covers 100 tasks across Work, Daily, Professional, and System & Web Setup domains on Windows and macOS.The benchmark contains 38 Work, 29 Daily, 24 Professional, and 9 System & Web Setup tasks, with 61 Windows and 39 macOS tasks.
  • B AGENTNETBENCH: AGENTNETBENCH annotates multiple plausible actions per step instead of requiring one ground-truth action, accommodating valid interaction variability.This design contrasts with earlier offline benchmarks that define a single action at each step.
  • B AGENTNETBENCH: Step Success Rate evaluates individual actions using type-specific matching criteria, including bounding boxes, edit distance, exact key matches, and correct termination timing.Scroll actions additionally require the correct direction and coordinates.
  • B AGENTNETBENCH: Offline AGENTNETBENCH performance shows a clear positive power-law correlation with online task success rates under low step budgets.The offline benchmark primarily measures first-choice step accuracy, whereas online agents can recover from earlier errors through self-reflection.

C Dataset Statistics, Annotation Details, and AgentNetTool Details

The paper characterizes AgentNet’s task and action diversity, describes its human annotation pipeline and tooling, and documents automated privacy analysis with human review.

  • Dataset Statistics: AgentNet tasks span over 140 applications and 190 mainstream websites, with nearly half involving web applications.Applications are tracked through process names, while websites are recorded through a browser plugin.
  • Dataset Statistics: 30.6% of tasks use multiple applications or websites, while 12.9% require professional knowledge and 12.9% use uncommon features.Tasks average 18.6 steps, and complexity is assessed across five dimensions.
  • Action Distribution: Click is the dominant action at more than 60% on all three operating systems, while secondary action patterns differ by platform.macOS has more scrolling and hotkeys, Windows has more right- and middle-clicks, and Ubuntu has the greatest share of text input.
  • AgentNetTool Details: The annotation tool captures screen videos, mouse and keyboard signals, accessibility trees, and trajectory metadata for state-action conversion and verification.Action reduction maps atomic inputs to semantic actions such as click, type, drag, move, and scroll; a11y trees and HTML support textual observation and localization.
  • Privacy Analysis: GPT-based privacy classification uses four sensitivity levels, but keyword reliance can overclassify benign cases such as viewing an unfilled password field.Human review is used to identify whether genuinely confidential information was entered.
  • Human Verification: Human verification can improve precision in false-alarm cases, but it increases labor and time and may vary with guideline enforcement.Reviewers typically label a displayed but blank password field as None.

D OSWorld-Verified Results Detail

OSWorld-Verified evaluations are analyzed alongside failure categories showing that domain knowledge, pixel precision, error recovery, termination, and long-horizon coherence remain difficult.

  • D OSWorld-Verified Results Detail: OPENCUA-32B and OPENCUA-7B were evaluated on OSWorld-Verified with three turns, and Pass@3 success rates were also calculated.Table 11 reports evaluation results for OPENCUA models under this protocol.
  • Failure Analysis: Domain-specific GUI knowledge and procedural skills remain failure sources, including spreadsheet functions and bulk-fill operations.The examples include failures involving VLOOKUP and filling cells from the row above.
  • Failure Analysis: Pixel-accurate tasks frequently fail because agents may select extra characters during fine-grained edits.The paper notes that letter-level grounding data are also difficult to synthesize.
  • Failure Analysis: Agents may repeat ineffective incorrect actions, loop indefinitely, continue after completion, or terminate prematurely.These failures reflect unreliable error recovery and termination decisions.
  • Failure Analysis: OSWorld tasks requiring more than 30–50 gold actions challenge the agent’s ability to maintain coherent context over long sequences.The paper illustrates this with a multi-category desktop file-organization task requiring content-based classification.
  • Failure Analysis: Although the agent can detect some mistakes and reflect, it may misjudge high-precision edits and lacks a reliable undo-and-retry strategy.The paper gives the example of inserting text one character off while judging the action correct.

F OSWorld Case Example

The case example shows an agent installing a local Chrome extension by navigating to extension management, enabling developer mode, and selecting the extension folder. The process includes an initially incorrect Web Store path before successful recovery and installation.

  • Task setup: The task is to install a manually developed Chrome extension from the Desktop directory into Google Chrome.The agent begins from the Chrome homepage and considers the browser menu as the direct route to extension management.
  • Navigation: The agent first navigates through Chrome’s Extensions menu to the extensions management page.The menu exposes both Manage Extensions and Chrome Web Store options.
  • Recovery: The Web Store route is an incorrect detour because local extension installation does not require sign-in or store access.The agent returns from Google sign-in and searches for a local-file installation path.
  • Installation: The agent enables Developer mode, revealing the Load unpacked control for installing an extension from local files.This converts the extensions page into the appropriate workflow for the Desktop-resident extension.
  • Installation: The agent opens the Desktop directory, selects the helloExtension folder, and confirms it with Select.The selected folder is the manually developed extension targeted for installation.
  • Outcome: The installation completes successfully, and Chrome displays the Hello Extensions extension with version 1.0 and its description.The workflow returns to the Extensions management page after the file-browser confirmation.

G.1 L1 Data Example

The L1 data example pairs task instructions and screenshots with action-oriented outputs for GUI interaction. It emphasizes concise next-action prediction, while the broader training formats add reasoning, observation, and action context.

  • L1 format: L1 data includes CoT-L1(Action) in the inner monologue, while the assistant’s final output is included in the loss calculation.The format concentrates supervision on action-oriented reasoning and the final response.
  • Action sequence: The L1 action sequence provides concrete GUI commands such as opening the document, selecting the target word, copying it, and clicking the search tab.These steps specify targets by their visible names rather than relying only on coordinates.
  • Task example: The example task asks the agent to find “Travelers” in a document, look up its meaning on Wikipedia, and add the description as a footnote.The task sequence includes opening the document, selecting and copying the word, and switching to Google search.
  • Action prediction: The prompt requires generating the next move from the screenshot, task instruction, and previous steps when available.This frames each response as a state-dependent action prediction.
  • Format comparison: The L2 format adds Thought and Action, whereas the L3 format adds Observation, Thought, and Action.The prompts require progress assessment, error recovery, candidate-action evaluation, and anticipated consequences.
  • Input handling: For text input, the instructions require tracking cursor position, consolidating repeated keypresses, and specifying the expected final text.The example anticipates replacing “gpt” with “Travelers Wikipedia” in the search box.
Loading 2508.09123v3…