PaperlayerView exact revision on arXiv

Source-linked AI summary

CUA-Master: Scaling Computer-use Agents through Open-World Recursive Improvements

Xinlei Yu, Gen Li, Ji Zhao, Zhangquan Chen, Congcong Wang, Shuai Dong, Zelin Peng, Yuqi Xu, Hongrui Wu, Kaiwen Tuo, Cheng Yang, Kaituo Feng, Yifan Wu, Jiayi Zhang, Dongming Wu, Wenqi Shao, Jiaqi Wang, Xiangyu Yue

arXiv:2611.03126v1cs.AI

TL;DR

CUA development faces data limitations in diversity, controllability, and reliability, while conventional and self-generated recursive pipelines do not fully address evolving capability gaps. CUA-Master combines open-world exploration, data construction, and model optimization in nested recursive flywheels. CUA-Master-27B achieves competitive performance across GUI-only and GUI-CLI hybrid benchmarks, alongside progressive gains across iterations.

  • Problem

    CUA data remains limited in diversity, controllability, and reliability, while conventional pipelines lack iterative feedback and self-generated recursive approaches remain closed-world.

    Source: §1 · p. 2

  • Method

    CUA-Master couples an outer open-world data-model flywheel with an inner flywheel that mines and reconstructs video workflows, propagates reusable primitives, and uses snapshot-rollback retries.

    Source: §1 · p. 2

  • Results

    CUA-Master-27B scored 80.2 on OSWorld Verified and achieved competitive performance across GUI-only and GUI-CLI hybrid benchmarks.

    Source: §1 · p. 2, §4 · p. 12

  • Takeaways & Limitations

    The framework improves data-synthesis diversity, coverage, reliability, and controllability, with progressive gains across recursive iterations.

    Source: §1 · p. 2

    The pipeline incurs substantial computational and infrastructure costs, and synthesized-data quality depends on capable annotator models.

    Source: E Limitations · p. 19

Abstract

from arXiv · show

Despite recent advances in CUAs, their development remains constrained by the limited diversity, controllability, and reliability of interaction data. Moreover, conventional optimization pipelines lack iterative feedback, while existing recursive improvement approaches often rely on self-generated experiences, limiting ability expansion and yielding diminishing returns. To address these challenges, we introduce CUA-Master, a nested double-loop recursive improvement framework. At the system level, an outer world-data-model flywheel continuously incorporates open-world corpus and leverages evolving model weaknesses to guide subsequent data construction and optimization iterations. At the data level, an inner flywheel couples open-world seeding and autocatalytic propagation. The former mines authentic workflows as seed tasks from instructional videos, and employs visual-semantic dual-path trajectory reconstruction; The latter transforms these seeds into reusable primitives for controllable synthesis of diverse, complex, and verifiable tasks. To enable efficient and reliable trajectory collection, we further introduce a feedback-guided snapshot-rollback retry mechanism in various real environments. Extensive experiments demonstrate that CUA-Master-27B achieves competitive performance across GUI-only and GUI-CLI hybrid benchmarks, scoring 80.2 on OSWorld Verified, 36.9 on OSWorld 2.0, 43.9 on WeaveBench, and 25.0 on Agents' Last Exam. Our framework also substantially improves the quality and efficiency of data synthesis, and exhibits continuous gains across recursive iterations.

1 Introduction

CUA development is constrained by limited data diversity, controllability, and reliability, alongside pipelines that do not use evolving model weaknesses to guide data construction. CUA-Master combines open-world exploration with iterative data and model improvement.

1 Introduction

  • Conventional pipelines generate data before training, while self-generated recursive approaches remain closed-world and bounded by model capabilities.The authors argue these patterns can leave persistent capability gaps unaddressed, reinforce distributional biases, and yield diminishing marginal returns.Source: §1 · p. 2
  • Synthetic-data approaches face intertwined gaps in diversity, controllability, and reliability, limiting coverage of capabilities needed by general-purpose CUAs.Restricted sources can produce homogeneous tasks; rewriting and templates offer limited control over abilities, complexity, and dependencies; complex rollouts can fail more often.Source: §1 · p. 2
  • CUA-Master’s outer world-data-model flywheel uses evolving capability gaps to guide exploration and data generation while incorporating external knowledge.This closes the loop among open-world exploration, data construction, and model optimization.Source: §1 · p. 2
  • Its inner data flywheel couples video-derived open-world seed tasks with reusable primitives composed into verifiable tasks with controlled complexity and application scope.Seed trajectories are reconstructed through a visual-semantic dual-path mechanism.Source: §1 · p. 2
  • A feedback-guided snapshot-rollback retry mechanism preserves valid execution prefixes and recovers from localized failures without restarting entire rollouts.The framework uses this mechanism to improve rollout reliability and efficiency.Source: §1 · p. 2
  • 80.2 on OSWorld Verified, 36.9 on OSWorld 2.0, 43.9 on WeaveBench, and 25.0 on Agents’ Last Exam; CUA-Master-27B shows competitive performance and progressive gains across iterations.The framework also substantially improves data-distribution diversity and coverage, and synthesis reliability and controllability.Source: §1 · p. 2

2 Methodology

CUA-Master links open-world workflow discovery, controllable task synthesis, and model training in recursive flywheels. Its methodology combines video-derived task supervision, reusable operational primitives, and rollback-based trajectory collection.

2.1.1 Task Definition / 2.1.2 Hybrid Action Space

  • CUA tasks pair an instruction with an interactive environment and task-specific verifier, while the hybrid action space supports interleaved GUI and structured CLI operations.The verifier assesses whether execution achieves the instruction’s objective; GUI observations use screenshots and CLI observations use shell outputs.Source: §2.1.1 · p. 3, §2.1.2 · p. 3

2.1.3 Environment Infrastructure

  • A backend-agnostic, profile-driven infrastructure orchestrates reproducible CUA tasks across Linux, Windows, and MacOS using versioned VM images.The rollout system supports over 100 environments and more than 2,000 concurrent VMs.Source: §2.1.3 · pp. 3–4

2.2.1 Data Flywheel

  • Open-world seeding supplies real workflows, autocatalytic propagation expands them into executable tasks, and model gaps guide subsequent exploration and data scaling.The outer world-data-model loop feeds evolving capabilities and limitations back into data acquisition.Source: §2.2.1 · pp. 4–5, §2.2.2 · p. 5

2.3 Open-world Seeding / 2.3.1 Corpus Harvesting

  • Open-world seeding targets underrepresented workflows by harvesting tutorial videos from a taxonomy-organized application search space and filtering unsuitable candidates.The corpus search spans eight categories, 36 subdomains, and over 650 applications, using metadata and noise filters.Source: §2.3 · p. 5, §2.3.1 · p. 5

2.3.2 Seed Task Mining / 2.3.3 Dual-path Trajectory Reconstruction

  • Seed tasks mined from instructional videos are reconstructed through semantic action cues and visual interface changes, followed by annotation checks and trajectory verification.Task mining segments independent, state-continuous episodes and excludes non-operational content; reconstruction aligns transcript and visual triggers before refining actions.Source: §2.3.2 · pp. 5–6, §2.3.3 · p. 6

2.4.1 Primitive-Based Composable Task Synthesis

  • Reusable application-level primitives encode abilities, preconditions, environment requirements, and verifier specifications, enabling tasks to be composed under complexity and application constraints.Composition checks compatibility; one selected application yields a single-application task, while multiple applications yield a cross-application task.Source: §2.4.1 · p. 7

2.4.2 Resilient Trajectory Collection

  • Snapshot-rollback retries preserve valid interaction prefixes, resume from before a detected divergence using diagnostic feedback, and retain only successful, efficient trajectories.Retries continue until verification passes or the retry budget is exhausted.Source: §2.4.2 · p. 7

2.5.1 Recipe / 2.5.2 Supervised Fine-Tuning (SFT) / 2.5.3 Reinforcement Learning (RL)

  • Training iterates SFT using assessed model gaps to scale data, then applies RL to high-quality synthetic tasks with verifier-derived outcome rewards.The final RL stage follows gradual SFT saturation and uses execution results without additional manual reward annotation.Source: §2.5.1 · p. 8, §2.5.3 · p. 8

3 Experiments

The experiments compare CUA-Master with generalist and specialist agents under consistent evaluation protocols, then assess benchmark performance, efficiency, generalization, hybrid actions, and recursive data and model flywheels.

3.1.1 Baselines

  • The evaluation compares generalist foundation models and specialist CUA agents, re-evaluating baselines with consistent environments, action interfaces, and criteria unless specified otherwise.The benchmark set includes GUI-only and GUI+CLI hybrid tasks, and reports steps, tokens, runtime, and score.Source: §3.1.1 · p. 8, §3.1.2 · pp. 8–9, Table 1 · p. 9

3.1.3 Implementations

  • Training is initialized from Qwen-3.5-27B, with SFT iterations on cumulatively expanded data totaling about 115K tasks at the last iteration and an RL stage using about 12K high-quality tasks.Source: §3.1.3 · p. 9

3.2.1 CUA Ability Evaluation

  • 80.2 on OSWorld Verified, 36.9 on OSWorld 2.0, 43.9 on WeaveBench, and 25.0 on Agents’ Last Exam; CUA-Master-27B improves over UI-Mate-27B on all four.It performs comparably to Qwen-3.8-27B on GUI-only tasks, gains on hybrid scenarios, and outperforms Qwen-CUA-397B-A17B on hybrid benchmarks.Source: Table 1 · p. 9, §3.2.1 · p. 10

3.2.2 Performance-efficiency Comparison

  • On OSWorld 2.0, CUA-Master-27B scores 36.9 versus Qwen-3.8-27B’s 36.5 while using 24% less runtime; on WeaveBench it matches GPT-5.6-Luna with 28% less runtime.On Agents’ Last Exam, it matches Kimi-K3 and Qwen-3.8-Max while requiring 34–36% less time; Figure 3 encodes runtime, score, and token consumption.Source: §3.2.2 · p. 10, Figure 3 · p. 10

3.2.3 Generalization

  • CUA-Master scores 56.2 on WindowsAgentArena, 43.0 on MacOSWorld, and 59.9 on MacAgentBench, demonstrating cross-platform generalization.Table 2 reports evaluation across Windows and macOS environments.Source: §3.2.3 · p. 10, Table 2 · p. 10

3.3.1 Effectiveness of Hybrid Action Space

  • Hybrid execution on WeaveBench scores 43.9, exceeding GUI-only by 37.3 points and CLI-only by 32.4, while using 23.8% fewer steps than GUI-only.On Agents’ Last Exam, hybrid execution raises the score from 5.9 to 25.0 over GUI-only with 26.7% fewer steps; GUI and CLI actions serve complementary operations.Source: §3.3.1 · p. 11, Figure 6 · p. 11

3.3.2 Analysis of Data Flywheel

  • The data flywheel spans existing task-embedding distributions and sparsely covered regions, while rollout success approaches 50% at K=40 primitives across N=3 applications versus 14.6% for the baseline.At K/N=50/8, success is 18.7% versus 0.0%; Figure 7 visualizes task embeddings and Table 3 compares rollout success across difficulty levels.Source: Table 3 · p. 11, Figure 7 · p. 11, §3.3.2 · p. 12

3.3.3 Analysis of World-data-model Flywheel

  • Successive training iterations improve all four benchmark scores, with OSWorld Verified and WeaveBench rising by over 25 points; final RL adds approximately 15 points on OSWorld 2.0 and Agents’ Last Exam.Stronger annotator models yield greater gains throughout the process, while self-policy annotation brings relatively limited improvements.Source: §3.3.3 · p. 12, Figure 8 · p. 12, Figure 9 · p. 12

4 Conclusion

CUA-Master combines an outer world-data-model flywheel with an inner data flywheel to support open-world, iterative improvement. The authors report competitive performance across GUI-only and GUI-CLI hybrid benchmarks, with strong execution efficiency and consistent gains across iterations.

4 Conclusion

  • The outer flywheel uses open-world knowledge and evolving model weaknesses to guide further iterations.The inner flywheel couples open-world seeding with autocatalytic propagation for data synthesis.Source: §4 · p. 12
  • Visual-semantic trajectory reconstruction, primitive-based task composition, and snapshot-rollback retries support diverse, controllable, and reliable data synthesis.Source: §4 · p. 12
  • CUA-Master-27B achieves competitive performance across GUI-only and GUI-CLI hybrid benchmarks, with strong execution efficiency and consistent recursive gains.Source: §4 · p. 12

A.1 CUA Specialists

CUA research has progressed from visual grounding toward autonomous interaction and long-horizon execution, while expanding into hybrid action spaces. Recent work also emphasizes data synthesis and specialized training for complex environments.

A.1 CUA Specialists

  • Early work established foundational GUI perception and grounding, followed by agents integrating visual understanding, reasoning, and action planning.The passage names OS-Atlas, Tong-UI [71], and CogAgent [15] among the foundational works, then Aguvis [59], UI-TARS-2 [34], Agent S [1], and MAI-UI [74].Source: §A.1 · p. 17
  • Later systems advanced reliable data synthesis and specialized training paradigms for complex environments.Examples include OpenCUA [51], CUA-Gym [49], EvoCUA-1.5 [17], ScaleCUA [28], UI-Mate [9], and UI-Venus-2 [48].Source: §A.1 · p. 17
  • Recent research extends beyond GUI-only interaction by combining GUI actions with terminal, file-system, structured-tool, or command-line operations.The passage cites UI-TARS-2 [50], ToolCUA [16], and HybridCUA [8] as examples.Source: §A.1 · p. 17

A.2 Scalable Data Synthesis

Scalable data synthesis draws on both naturally occurring interaction resources and environment-based task generation. CUA-Master unifies open-world workflow acquisition with primitive-based compositional synthesis to expand tasks toward greater diversity, verifiability, and complexity.

A.2 Scalable Data Synthesis

  • One synthesis line mines interaction traces, instructional videos, or web tutorials, while another generates tasks through environment interaction.Examples include OpenCUA [51], WildGUI [57], VideoCUA [20], AgentTrek [60], TongUI [71], OS-Genesis [44], and AutoPlay [39].Source: §A.2 · p. 17
  • Jointly achieving real-world diversity, controllable task complexity, reliable verification, and efficient execution remains challenging.Source: §A.2 · p. 17
  • CUA-Master unifies open-world workflow acquisition with primitive-based compositional synthesis to expand tasks into diverse, verifiable, and increasingly complex computer-use tasks.Source: §A.2 · p. 17

A.3 Recursive Improvements

Recursive-improvement approaches use environmental feedback, iterative refinement, or self-generated challenges to improve agents. The passage identifies risks in recursive training and contrasts them with a cycle grounded in refreshed real-world knowledge.

A.3 Recursive Improvements

  • Recursive-improvement approaches include environmental feedback and trajectory refinement, self-evolving curricula, self-generated verifiable challenges, and evolving world models.Examples named are Re-ReST [11], Agent-R [70], WebRL [33], Self-Challenging Agents [75], and WebEvolver.Source: §A.3 · p. 17
  • Recursive training may face distributional narrowing, accumulated synthetic errors, and diminishing task novelty.Source: §A.3 · p. 18
  • The world-data-model flywheel anchors recursive improvement in refreshed real-world knowledge, with open-world seeding, compositional propagation, and model feedback forming an adaptive cycle.Source: §A.3 · p. 18

B Details of Training Data

CUA-Master’s training data span a broad application taxonomy, and the data distribution is reallocated across recursive iterations to reflect changing capability gaps.

B Details of Training Data

  • The dataset contains 127K samples—115K for SFT and 12K for RL—across 8 domains, 36 subdomains, and over 650 applications.The largest domain shares are Business & Administration (18.4%) and Creative Production (16.5%).Source: B Details of Training Data · p. 18
  • Across iterations, data allocation shifts toward Software Development, Science & Research, and Productivity, with the final RL stage more balanced across domains.This reallocates data in line with evolving capability gaps while maintaining broad application coverage.Source: B Details of Training Data · p. 18

C Details of Methodology

The methodology combines GUI and CLI interaction with extensible tools, uses role-specific annotation across its data-construction pipeline, and analyzes performance by domain.

C Details of Methodology

  • CUA-Master uses a hybrid action space combining GUI actions, CLI actions, and extensible registered tools.Table 7 presents the hybrid atomic action space.Source: C Details of Methodology · p. 18, Table 7 · p. 19
  • A general-purpose annotator performs eight specialized data-construction roles using prompts tailored to each role’s inputs, objectives, and structured outputs.The prompts cover operations including task mining, action annotation, verification, and synthesis.Source: C Details of Methodology · p. 18
  • The paper reports domain-level results for OSWorld Verified, OSWorld 2.0, and WeaveBench.The corresponding tables present domain results for OSWorld Verified and OSWorld 2.0, and level-average scores on WeaveBench.Source: Table 4 · p. 19, Table 5 · p. 19, C Details of Methodology · p. 19

D Details of Experiments

Domain-level results show broad coverage and general performance improvements, while identifying remaining gaps in specialized manipulation and complex application workflows.

D Details of Experiments

  • Domain-level results across three benchmarks show broad coverage and general performance improvements, but gaps remain in specialized manipulation and complex application workflows.The results are reported in Tables 4, 5, and 6.Source: D Details of Experiments · p. 19

E Limitations

The framework’s recursive data and optimization pipeline carries substantial compute and infrastructure costs, and synthesized-data quality depends on capable annotator models.

E Limitations

  • Large-scale environment interactions, trajectory collection, and iterative optimization impose substantial computational and infrastructure costs.Snapshot-rollback improves rollout efficiency, but VM deployment still requires substantial infrastructure support.Source: E Limitations · p. 19
  • Synthesized-data quality depends on annotator capability; stronger annotators yield greater performance gains but add inference costs and may limit accessibility.Source: E Limitations · p. 19

Related papers

Loading 2611.03126v1…