PaperlayerView exact revision on arXiv

Source-linked AI summary

UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action

Yuhao Yang, Zhen Yang, Zi-Yi Dou, Anh Nguyen, Keen You, Omar Attia, Andrew Szot, Michael Feng, Ram Ramrakhya, Alexander Toshev, Chao Huang, Yinfei Yang, Zhe Gan

arXiv:2510.17790v3cs.CVcs.CL

TL;DR

Computer-use agents are limited by brittle GUI-only execution, whereas tool-based interfaces offer more efficient structured operations. UltraCUA combines GUI primitives with high-level tools through scalable tool acquisition, verifiable synthetic tasks, hybrid trajectories, and two-stage training. The resulting models improve benchmark performance and generalize across platforms, while the paper identifies an inference-speed limitation for OpenCUA-series comparisons.

  • Problem

    GUI-only computer-use agents rely on primitive actions that create cascading failures and inefficient execution compared with tool-based interfaces.

  • Method

    UltraCUA combines GUI primitives and high-level tools using automated tool acquisition, 17,000+ verifiable tasks, hybrid trajectories, and supervised fine-tuning followed by online reinforcement learning.

  • Results

    UltraCUA achieves state-of-the-art performance on real-world benchmarks, including a 22% relative improvement over base models across 7B and 32B scales.

  • Takeaways & Limitations

    The hybrid action space is identified as the essential driver of stronger, more robust, and more efficient computer-control agents.

  • Takeaways & Limitations

    OpenCUA-series comparisons use fewer than 4 average runs because of sub-optimal inference speed and infrastructure.

Abstract

from arXiv · show

Computer-use agents face a fundamental limitation. They rely exclusively on primitive GUI actions (click, type, scroll), creating brittle execution chains prone to cascading failures. While API-driven agents harness rich capabilities through structured interfaces and tools, computer-use agents remain constrained to low-level visual interactions. We present UltraCUA, a foundation model that transcends this limitation through hybrid action-seamlessly unifying primitive GUI operations with high-level tool execution. Our innovation rests on four critical advances. First, an automated pipeline extracts and scales tool capabilities from software documentation and code repositories. Second, a synthetic data engine produces 17,000+ verifiable tasks capturing real-world computer-use complexity. Third, comprehensive hybrid action trajectory collection incorporates both GUI primitives and strategic tool calls. Fourth, a two-stage training methodology combines supervised fine-tuning with online reinforcement learning, enabling intelligent action selection between GUI and API. Evaluation with our 7B and 32B UltraCUA models reveals transformative performance gains. On OSWorld, UltraCUA achieves 22% relative improvement while executing 11% faster than existing approaches, averagely. Cross-domain validation on WindowsAgentArena demonstrates robust generalization with 21.7% success rate, surpassing Windows-trained baselines. The hybrid action paradigm proves essential, reducing error propagation while improving execution efficiency. This work establishes a scalable paradigm bridging primitive GUI interactions and high-level tool intelligence, enabling more resilient and adaptable computer use agents for diverse environments and complex real-world tasks.

1 Introduction

UltraCUA addresses the brittleness and inefficiency of GUI-only computer-use agents by combining primitive visual actions with high-level tool execution. It contributes scalable tool acquisition, verifiable synthetic tasks, hybrid trajectories, and two-stage training, yielding gains across benchmarks and platforms.

  • Motivation: GUI-only agents accumulate cascading errors and require many fragile interactions for operations that native tools can perform directly.The paper contrasts spreadsheet extraction through lengthy GUI sequences with single-call spreadsheet APIs.
  • Approach: Hybrid action combines GUI universality with tool efficiency, selecting tool calls when advantageous and GUI interactions when tools are unavailable or insufficient.This orchestration aims to maximize task success while minimizing execution overhead.
  • Contributions: The automated tool pipeline extracts capabilities from documentation, integrates open-source implementations, and generates tools on demand.It produces hundreds of tools for diverse environments.
  • Contributions: The synthetic data engine produces 17,000+ verifiable tasks through instruction-first and evaluator-first generation pipelines.The two pipelines provide contextual task diversity and reliable completion detection.
  • Contributions: UltraCUA uses hybrid trajectories and two-stage training to teach strategic selection between GUI actions and tools.The model is trained with supervised fine-tuning followed by online reinforcement learning.
  • Results: UltraCUA-7B achieves 21.7% success on WindowsAgentArena without Windows-specific training, surpassing models trained on Windows data.The result supports cross-platform generalization across operating systems.

2 Methodology

The methodology builds a broad hybrid action ecosystem by automatically collecting tools and generating verifiable, diverse computer-use tasks. Evaluator-first generation supports reliable reinforcement-learning signals, while instruction-first generation expands realistic task coverage.

  • 2.1 Automated Tool Collection: UltraCUA defines tools as high-level interfaces encapsulating action sequences while excluding interactions requiring visual grounding, such as clicks.Tools are exposed through Python signatures with descriptive docstrings.
  • 2.1 Automated Tool Collection: The tool ecosystem combines documentation extraction, open-source implementations, and coding-agent-generated tools to replace inefficient GUI sequences with programmatic execution.The pipeline collects hundreds of tools across applications and usage scenarios.
  • 2.2 Synthetic Data Generation: 17,000+ verifiable tasks are generated by combining evaluator-first and instruction-first synthetic data pipelines.Evaluator-first tasks prioritize precise verification; instruction-first tasks arise from explored interface states.
  • 2.2.1 Evaluator-First Generation: Evaluator-first generation composes environment state-checking functions into complex verification conditions, ensuring programmatic completion detection.Tasks are then generated to satisfy the configured evaluators.
  • 2.2.2 Instruction-First Generation: Instruction-first generation explores diverse UI states and creates contextually appropriate tasks, producing 12,000+ tasks verified by an evaluator agent.This strategy allows flexible execution paths and complements systematic evaluator-based coverage.
  • 2.2 Synthetic Data Generation: The two generation strategies trade off verification precision and task diversity, jointly supplying reliable RL signals and broad real-world coverage.Evaluator-first tasks tend to be harder, whereas instruction-first tasks are naturally easier and more diverse.

2.3 Training a Foundation Agent with Hybrid Action

UltraCUA trains hybrid-action agents in two stages: supervised fine-tuning establishes competent behavior, and online reinforcement learning optimizes action selection through exploration and outcome-based rewards.

  • Training Strategy: Supervised fine-tuning followed by online reinforcement learning first establishes hybrid-action competence and then optimizes GUI-versus-tool selection.The curriculum uses self-play on synthetic tasks during the second stage.
  • Hybrid Execution: An integrated working memory records objectives, progress, and persistent intermediate state across switches between programmatic tools and GUI actions.The mechanism supports coherent multi-step execution when information must persist across modalities.
  • Stage 1: Supervised Fine-Tuning: 26.8K successful trajectories are converted into per-turn training samples so each action decision receives equal loss weight.This prevents overfitting to early trajectory steps.
  • Stage 2: Online Reinforcement Learning: Online reinforcement learning explores alternative hybrid solution paths and samples tasks with difficulty scores in [0.4, 0.8] to focus learning in the model’s productive range.The policy uses a GRPO variant with clip-higher and no KL regularization to encourage diverse action sequences.
  • Stage 2: Online Reinforcement Learning: The reward combines sparse environment success with a 0.3 tool-use bonus on successful trajectories containing tool calls.This reward design encourages efficient hybrid action while avoiding format penalties that can dominate early learning.

3 Experiments

UltraCUA is evaluated on in-domain OSWorld, cross-platform WindowsAgentArena, and component ablations, showing gains from hybrid action, working memory, and online reinforcement learning.

  • 3.2 Main Results: 41.0% success rate is achieved by UltraCUA-32B on OSWorld at 15 steps, outperforming Claude 3.7 Sonnet at 27.1% and OpenAI CUA at 26.0%.UltraCUA-7B reaches 28.9%, surpassing UI-TARS-1.5-7B at 23.4% with a 23.5% relative improvement.
  • 3.2 Main Results: 21.7% success rate is achieved by UltraCUA-7B on WindowsAgentArena without Windows-specific fine-tuning, exceeding Qwen2-VL-7B at 13.5% and UI-TARS-1.5-7B at 18.1%.The reported improvement over UI-TARS-1.5-7B is 20% relative, testing transfer from Ubuntu-based training to Windows 11.
  • 3.3.1 The Impact of Hybrid Action: Hybrid action improves OSWorld success from 23.4% to 27.0%, a 15.4% relative increase, while maintaining similar step counts.The ablation attributes the effectiveness gain in complex automation tasks to adding programmatic tools.
  • 3.3.1 The Impact of Hybrid Action: Hybrid action improves the GTA1-7B + o3 framework from 44.0% to 48.2% success and reduces average steps by 14.9%.The comparison evaluates the same multi-agent framework with and without programmatic tools.
  • 3.3.3 The Impact of Online RL: Online reinforcement learning raises UltraCUA-7B from 27.0% to 28.9%, while tool-related failures fall from 122 to 66.The reported behavior change includes more selective tool use after RL, alongside increasing outcome and format rewards.
  • 3.3.2 The Impact of Working Memory: Working memory increases success from 25.4% to 27.0% and slightly reduces average steps, especially for persistent-state and cross-application workflows.The reported efficiency gain reflects avoiding redundant navigation and information extraction.
  • 3.3.4 Out-of-Distribution Tool Generalization: Unseen tools produce a modest +1.9% relative success-rate gain at inference time, but adaptation increases the number of steps.The evaluation tests tools unavailable during training because of the context-length limit.

4 Conclusion

UltraCUA bridges GUI agents and API-based agents through a hybrid action space that combines low-level GUI operations with high-level tool use. Its scalable tool acquisition, verifiable task generation, and two-stage training support strong benchmark performance and robust computer control.

  • UltraCUA bridges general-purpose GUI agents and specialized API-based agents through a novel hybrid action space.The action space integrates low-level GUI actions with high-level tool use.
  • The paper’s core contributions are automated tool acquisition, verifiable hybrid-task generation, and a two-stage training curriculum for strategic agent behavior.
  • UltraCUA achieves state-of-the-art performance on real-world benchmarks, with ablations identifying hybrid action as the essential driver of success.The conclusion characterizes the paradigm as more effective for building strong, robust, and efficient general computer-control agents.

A.1 Related Work

Related work spans multimodal GUI agents and tool-augmented language models. UltraCUA addresses the shared limitation of primitive GUI action spaces by adding high-level tools while retaining visual generality.

  • GUI automation research includes web, desktop, and mobile agents operating from structured interfaces or pixels, with multi-agent and unified-model designs.
  • Most powerful computer-use agents rely on primitive clicks, typing, and scrolling, which creates brittleness and long-horizon planning challenges.
  • UltraCUA introduces a hybrid action space that retains GUI generality while adding the efficiency and robustness of high-level tools.
  • Tool-augmented language-model research developed API selection, large-scale tool use, tool-making, and online reinforcement-learning approaches.

A.2 The Use of Large Language Models

The paper reports using language models as writing and presentation aids, while keeping research ideas, experiments, implementation, and core contributions author-developed.

  • LLMs assisted with language polishing, grammar checking, formatting suggestions, and recommendations for data visualization.
  • The authors state that LLMs did not assist with the research ideas, experimental design, implementation, or core scientific contributions.

A.3 Details for Programmatic Tools

The OSWorld tool collection covers 10 application domains with 881 programmatic tools. These tools support both basic navigation and complex application-specific desktop operations.

  • 881 programmatic tools are available across 10 OSWorld application domains.Individual domains contain between 4 tools in System and 135 tools in VS Code.
  • The collection ranges from basic navigation tools such as jump_to_next_tab to complex operations such as batch_spreadsheet_numeric_formatter.
  • The tool coverage is intended to support automation across diverse desktop workflows and software environments.

A.4 Details for Synthetic Tasks

UltraCUA’s synthetic dataset contains 17,864 verified tasks spanning 10 application domains, combining evaluator-first and instruction-first generation to balance complexity and coverage.

  • Dataset scope: 17,864 tasks span 10 application domains, covering productivity software, specialized tools, development environments, web interactions, and cross-application workflows.Chrome contributes 2,826 tasks, while multi-apps contributes 2,113 tasks for cross-application testing.
  • Generation approaches: 13,477 tasks come from instruction-first generation, while evaluator-first generation contributes 4,387 high-quality tasks with complex multi-step instructions.The two approaches respectively prioritize broad functionality coverage and complex, verifiable task construction.
  • Domain distribution: Task coverage ranges from LibreOffice’s 5,885 combined tasks and VS Code’s 1,990 tasks to GIMP’s 1,121 tasks and Chrome’s 2,826 tasks.The dataset includes both common productivity workflows and specialized application tasks.

A.5 Qualitative Examples

The qualitative examples show UltraCUA alternating between GUI primitives and programmatic tools according to each task’s needs. This hybrid orchestration supports precise context-setting, direct navigation, nuanced interaction, and scripted automation.

  • Representative cases: Figures 5–7 present email, Chrome-history, and image-processing workflows as representative cases of selecting between GUI actions and programmatic tools.Together, the examples illustrate hybrid execution across productivity, web, and desktop automation tasks.
  • Email processing: UltraCUA combines a precise GUI selection of the “Bills” folder with select_all and add_or_remove_star tool calls for a robust bulk operation.The example illustrates a handoff from context-setting through GUI interaction to general-purpose programmatic execution.
  • Browser history: The agent opens the history page programmatically, then uses typing and clicks to filter and delete specific YouTube entries.This division avoids brittle menu navigation while retaining GUI control for the nuanced filtering and deletion steps.
  • Image processing: For batch image processing, UltraCUA opens a terminal, installs ImageMagick, writes a multi-line shell script, and executes it.The example demonstrates programmatic code generation for a workflow described as intractable for purely GUI-based agents.
Loading 2510.17790v3…