PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experience

Taofeng Xue, Chong Peng, Mianqiu Huang, Linsen Guo, Tiancheng Han, Haozhe Wang, Jianing Wang, Xiaocheng Zhang, Xin Yang, Dengchang Zhao, Jinrui Ding, Xiandi Ma, Yuchen Xie, Peng Pei, Xunliang Cai, Xipeng Qiu

arXiv:2601.15876v2cs.AI

TL;DR

Native computer-use agents are limited by static imitation, which misses causal feedback in long-horizon computer tasks. EvoCUA combines verifiable synthesis, scalable asynchronous rollouts, and iterative experience-based optimization. It reaches 56.7% on OSWorld, establishes open-weights state of the art, and shows consistent gains across foundation-model scales.

  • Problem

    Static imitation of fixed datasets does not capture the causal feedback inherent in real-world computer use, limiting evidence for long-horizon interaction learning.

  • Method

    EvoCUA couples executable-validator task synthesis, tens of thousands of asynchronous sandbox rollouts, and iterative optimization from successful and failed trajectories.

  • Results

    56.7% success rate on OSWorld establishes EvoCUA as a new open-weights state of the art, with consistent gains across foundation models of varying sizes.

  • Takeaways & Limitations

    Learning from evolving interactive experience provides a generalizable path for improving native computer-use agents across model scales.

  • Takeaways & Limitations

    A performance gap remains between current open models and leading closed-weights systems or human-level reliability, motivating further online interaction.

Abstract

from arXiv · show

The development of native computer-use agents (CUA) represents a significant leap in multimodal AI. However, their potential is currently bottlenecked by the constraints of static data scaling. Existing paradigms relying primarily on passive imitation of static datasets struggle to capture the intricate causal dynamics inherent in long-horizon computer tasks. In this work, we introduce EvoCUA, a native computer use agentic model. Unlike static imitation, EvoCUA integrates data generation and policy optimization into a self-sustaining evolutionary cycle. To mitigate data scarcity, we develop a verifiable synthesis engine that autonomously generates diverse tasks coupled with executable validators. To enable large-scale experience acquisition, we design a scalable infrastructure orchestrating tens of thousands of asynchronous sandbox rollouts. Building on these massive trajectories, we propose an iterative evolving learning strategy to efficiently internalize this experience. This mechanism dynamically regulates policy updates by identifying capability boundaries -- reinforcing successful routines while transforming failure trajectories into rich supervision through error analysis and self-correction. Empirical evaluations on the OSWorld benchmark demonstrate that EvoCUA achieves a success rate of 56.7%, establishing a new open-source state-of-the-art. Notably, EvoCUA significantly outperforms the previous best open-source model, OpenCUA-72B (45.0%), and surpasses leading closed-weights models such as UI-TARS-2 (53.1%). Crucially, our results underscore the generalizability of this approach: the evolving paradigm driven by learning from experience yields consistent performance gains across foundation models of varying scales, establishing a robust and scalable path for advancing native agent capabilities.

1 Introduction

EvoCUA addresses the limits of static imitation for long-horizon GUI tasks by combining verifiable synthesis, scalable interaction, and iterative learning from experience. It achieves 56.7% on OSWorld, outperforming open- and closed-source comparison models while showing gains across foundation-model scales.

  • Static imitation fails to capture the causal feedback inherent in real-world computer use, motivating experience scaling through massive interactive rollouts.
  • EvoCUA unifies verifiable task synthesis, high-throughput interaction infrastructure, and evolutionary optimization into a self-sustaining learning cycle.
  • The Verifiable Synthesis Engine generates diverse tasks with executable validators, providing precise and deterministic supervision instead of ambiguous natural-language rewards.
  • Asynchronous infrastructure orchestrates tens of thousands of concurrent interactive sessions, supplying real-time feedback and state transitions for on-policy optimization.
  • Iterative learning consolidates successful routines and analyzes failed trajectories to rectify errors, converting accumulated experience into a precise execution policy.
  • 56.7% success rate on OSWorld surpasses OpenCUA-72B (45.0%) and UI-TARS-2 (53.1%), with consistent gains across foundation models of varying sizes.

2 Preliminaries

The paper formulates computer use as a partially observable decision process with explicit reasoning, verifiable rewards, and a policy refined through an adaptive synthesis-and-experience cycle.

  • Computer use is modeled as a POMDP with explicit reasoning and a co-evolutionary cycle of verifiable task synthesis and policy refinement.
  • The interaction tuple includes state, action, thought, observation, transition, and synthesized reward components.
  • The agent observes rendered screenshots rather than the underlying computer state, and recent visual and compressed textual history conditions decisions.
  • The unified action space combines coordinate-based mouse events, keyboard inputs, and special control primitives.
  • The policy generates a reasoning trace before selecting an executable action, making action execution conditional on explicit reasoning.
  • The objective maximizes expected verification success over a task distribution whose complexity and diversity adapt to the current policy.
  • An experience pool aggregates fresh trajectories from large-scale interaction, closing the loop between synthesis, execution, and on-policy optimization.

3 Verifiable Synthesis Engine

The Verifiable Synthesis Engine constructs diverse, executable, and decontaminated training tasks by jointly generating instructions and validators, testing them in sandboxes, and filtering failures.

  • The engine uses generation-as-validation to co-generate each instruction with a deterministic executable validator, grounding rewards in final-state verification.
  • Structured task-space construction organizes domains, resources, and atomic capabilities so primitive skills can be recombined into diverse scenarios.
  • The synthesis pipeline progresses through task-space construction, dual-stream generation, and quality assurance.
  • A hierarchical taxonomy spans applications and user personas, while hybrid resources combine parametrically generated documents with public internet data.
  • A VLM task architect generates an achievable instruction and corresponding ground-truth files and evaluator code in parallel.
  • Sandbox execution feeds successful outputs and failure errors back into the workflow, improving validator and ground-truth quality before standardized formatting.
  • Quality assurance filters false positives, false negatives, and leakage through rollout consistency checks and tri-fold decontamination.
  • Tens of thousands of verifiable training instances are produced, reducing reliance on manual data curation.

4 Scalable Interaction Infrastructure

EvoCUA’s interaction infrastructure provides reproducible, isolated, and elastic sandbox execution for continuous large-scale experience generation, from environment abstractions to virtualization and OS calibration.

  • The platform provides a high-throughput gymnasium for continuous, diverse, interactive feedback under heterogeneity, high concurrency, and strict session isolation.
  • Tools encapsulate immutable, version-controlled environment definitions and APIs, supporting hundreds of environment types with reproducibility.
  • Clusters instantiate tools with configurable resource quotas and scale from debugging sessions to tens of thousands of concurrent training nodes.
  • The microservices architecture targets I/O bottlenecks and rapid environment scaling for massive exploration.
  • The architecture routes requests through an asynchronous gateway and distributed scheduler to parallel sandbox clusters.
  • The gateway separates control-plane lifecycle management from data-plane interaction, while the scheduler supports burst bootstrapping of tens of thousands of sandboxes within one minute.
  • Hybrid QEMU-KVM virtualization inside Docker provides kernel-level isolation while maintaining near-native GUI rendering and I/O performance.
  • A calibrated Ubuntu 22.04 image addresses input determinism, rendering consistency, and runtime stability through system-level patches, fonts, and dependencies.

5 Evolving Paradigm via Learning from Experience

EvoCUA evolves computer-use policies through staged learning from experience, progressing from behavioral priors and successful trajectories to failure correction and boundary-focused exploration. Its cycle combines structured reasoning, adaptive computation, denoised supervision, and scalable experience accumulation to improve execution robustness and generalization.

  • Evolving paradigm: The evolving paradigm progresses from supervised cold-start training to rejection sampling fine-tuning and reinforcement learning for failure correction and complex interaction dynamics.These stages establish behavioral priors, consolidate successful experiences, and rectify failures through interaction.
  • Action and thought spaces: The agent uses unified mouse, keyboard, and control actions alongside a structured reasoning space linking visual observations to execution.The reasoning schema includes goal clarification, observation consistency, self-verification, and reflection-based correction.
  • Evolving paradigm: Failed rollouts are converted into supervision by restoring pre-error states, filtering for state consistency, and sampling successful remedial paths.This reflection-and-correction mechanism targets critical errors while accounting for sandbox non-determinism.
  • Rejection Sampling Fine-Tuning: Dynamic compute budgeting allocates rollout resources according to task-specific proficiency, pruning solved tasks and concentrating computation on high-variance boundary queries.The system selects a budget from hierarchical levels paired with descending success-rate thresholds.
  • Rejection Sampling Fine-Tuning: Step-level denoising removes redundant actions and preserves reasoning with failure termination for infeasible tasks, producing higher-quality supervision.The resulting pipeline scales the experience pool to tens of thousands of trajectories while interleaving general-purpose multimodal data to prevent catastrophic forgetting.
  • Reinforcement Learning: Iterative policy updates convert transient interaction experience into robust parameters, combining rejection fine-tuning with reinforcement learning to improve reliability across boundary conditions.The authors describe this cycle as stabilizing standard-task performance while improving robustness and generalization.

6 Evaluation

EvoCUA’s evaluation shows strong online computer-use performance and generally preserved capabilities across model scales, while exposing distribution-mismatch limitations and scaling constraints. Its evolutionary components contribute cumulative gains through structured initialization, successful-trajectory learning, failure correction, and repeated experience scaling.

  • Online Agentic Capability: 56.7% success rate on OSWorld makes EvoCUA-32B the top evaluated open-weights model, exceeding OpenCUA-72B at 45.0%.It also surpasses UI-TARS-2-2509 at 53.1% and narrows the gap with Claude-4.5-Sonnet at 58.1% to 1.4% under equivalent step constraints.
  • Offline Grounding and General Capabilities: EvoCUA’s post-training preserves or slightly improves OpenCUA-72B performance across grounding and general benchmarks, including improved OSWorld-G results.The analysis reports parity or slight gains while preserving MMMU scores when the data distribution is aligned.
  • Offline Grounding and General Capabilities: EvoCUA-32B declines on ScreenSpot-Pro and MMMU relative to Qwen3-VL-32B-Thinking under a mismatched non-thinking general-data distribution.Its output length also falls from 3,620 to 2,514 tokens, alongside a shift in output style.
  • Component Analysis on EvoCUA-32B: +4.84% from the unified action space and +2.62% from cold-start training provide foundational gains before iterative experience learning.The results characterize structured action definitions and coherent reasoning patterns as prerequisites for effective large-scale experience learning.
  • Component Analysis on EvoCUA-32B: +3.13% from Rejection Fine-Tuning, +3.21% from DPO, and +1.90% from another evolutionary iteration show cumulative gains from successful-experience consolidation and failure correction.The additional iteration supports iterative refinement of the model’s capability boundary.
  • Generalizability on OpenCUA-72B: +8.12% from three rounds of pure RFT on OpenCUA-72B indicates that experience learning generalizes to a stronger backbone without explicit cold start.The same analysis reports DPO at +3.02% and RFT at +3.69%.
  • Scaling Analysis: +4.93% is the peak 32B gain at Pass@16, while increasing inference capacity from 15 to 50 steps yields a +16.25% improvement over baseline.Beyond 50 steps, improvement moderates because the current training distribution contains few trajectories exceeding 50 steps.
  • Scaling Analysis: Iterative experience scaling reaches +6.79 pp after Round 2 on 226k samples and +8.12 pp after Round 3 using 1M aggregated samples.The analysis identifies a trade-off among data scale, off-policy distribution, and signal-to-noise ratio, requiring optimized data quality and on-policy alignment.

7 Future Work on Online Agentic RL

The section identifies training-inference discrepancy and high training cost as challenges for GUI reinforcement learning, then presents STEPO as a step-level optimization approach. Experiments report that STEPO outperforms GRPO trained with final trajectories.

  • The section notes that online agentic RL remains future work because sufficient model training and comprehensive benchmark evaluations were not completed.
  • Trajectory-level training can cause training-inference discrepancy in GUI tasks because intermediate-step supervision is not retained.
  • STEPO addresses this discrepancy by performing policy optimization over step-level samples rather than only final trajectories.
  • STEPO samples trajectory groups, computes within-group advantages, and allocates those advantages to all steps and tokens in each trajectory.
  • Uniformly allocating trajectory advantages across steps encourages high-advantage trajectories to shorten execution and low-advantage trajectories to explore more steps.
  • STEPO training performance is significantly superior to GRPO trained with final trajectories in online RL experiments on OpenCUA-32B.

8 Related Work

Related work spans foundation VLMs, generalist GUI agents, visual grounding, and the shift from imitation toward experience-based reinforcement learning. EvoCUA distinguishes itself by generating verifiable synthetic experiences at scale.

  • Generalist GUI Agents and Benchmarks: OSWorld is presented as a primary testbed for evaluating online agent performance, alongside OpenCUA, UI-TARS-2, and Step-GUI.
  • Generalist GUI Agents and Benchmarks: EvoCUA differs from demonstration-heavy approaches by using autonomously synthesized, verifiable experiences.
  • Visual Grounding and Action Execution: Visual grounding research has developed specialized architectures for precise GUI grounding and high-resolution layouts, informing EvoCUA’s execution primitives.
  • From Imitation to Learning from Experience: Training paradigms are shifting from Behavior Cloning toward Reinforcement Learning for multi-turn GUI interaction and reasoning capabilities.
  • From Imitation to Learning from Experience: EvoCUA addresses data scarcity with a verifiable synthesis engine that produces scalable, ground-truth-verified synthetic data for iterative learning from experience.

9 Conclusion

EvoCUA combines verifiable synthesis and scalable interaction infrastructure to convert synthetic compute into training signals. On OSWorld, it achieves a 56.7% success rate and establishes a new open-weights state of the art, while a gap with closed-weights systems remains.

  • 56.7% success rate on OSWorld establishes EvoCUA as a new state of the art among open-weights models.
  • EvoCUA integrates verifiable synthesis with scalable interaction infrastructure to convert synthetic compute into high-quality training signals.
  • A performance gap persists between current open models and leading closed-weights systems or human-level reliability.
  • Preliminary online reinforcement learning results show a consistent upward trend in reward accumulation, motivating future expansion of online evolutionary training.

A Unified Action Space

EvoCUA uses a unified native action space in which the agent interacts with the environment through the computer_use function and corresponding action arguments.

  • The agent invokes the computer_use function with a specific action and its corresponding arguments.
  • The unified native action space defines the actions available to EvoCUA for computer interaction.

B Cold Start: Hindsight Reasoning Generation

EvoCUA’s cold-start process retrospectively augments physical interaction traces with reasoning aligned to execution, including planning, state tracking, reflection, correction, and verified termination. Step-level DPO then converts failure and success trajectories into preference data around critical error points.

  • Hindsight Reasoning Generation: Hindsight Reasoning Generation retrospectively creates reasoning traces from known execution paths, establishing alignment between cognition and observed actions.The process uses context-aware prompts organized by execution phase.
  • Initialization: At initialization, prompts resolve task ambiguity by stating the environment state, goal, high-level plan, and rationale for the next action.The model receives the user instruction, initial screenshot, and first executable code block.
  • Intermediate Steps: Intermediate-step prompts identify environmental changes and explain why the next action advances the workflow, while describing UI targets semantically instead of by pixel coordinates.This semantic abstraction is intended to remain robust to layout variations.
  • Reflection and Correction: For error-recovery trajectories, reflection prompts expose the prior failure’s root cause and require a corrected plan, embedding self-correction in the training trace.Resume traces begin with a dedicated “Reflection:” header before transitioning to a different approach.
  • Termination Verification: Termination reasoning compares the final screenshot with the initial instruction and requires visual evidence of completion or failure before issuing the terminate signal.This conditions stopping on task verification rather than memorized trajectory length.
  • Step-Level DPO: Step-level DPO identifies critical forking points by aligning rejected failure actions with nearby successful actions, then constructs correction and reflection preference pairs.The procedure uses target failure trajectories, reference success trajectories, and a VLM for alignment and synthesis.

D Trajectory Analysis and Visualization

The EvoCUA Trajectory Inspector validates synthetic experience by displaying frame-level relationships among observations, reasoning, and executable actions. A spreadsheet task demonstrates checks for goal grounding and stateful GUI execution, while Figure 8 also covers termination logic.

  • Trajectory Inspector: The Trajectory Inspector examines frame-by-frame alignment among visual observations, reasoning traces, and executable code actions.It is designed for granular behavioral diagnosis and validation of synthetic experience quality.
  • Representative Task: A spreadsheet task requiring row-wise maxima in Column G serves as a long-horizon testbed for checking synthesis consistency.Figure 8 visualizes representative timestamps from this task.
  • Goal Clarification: The inspector validates goal grounding by showing the agent’s explicit paraphrase of the spreadsheet instruction at the initial step.The displayed reasoning restates finding the greatest value and placing it in Column G.
  • Stateful Interaction: The inspector validates stateful interaction by confirming the Shift-Select sequence key_down: shift → click → key_up: shift.This example tests the Unified Action Space on a stateful operation.
  • Validation Checks: Figure 8 also presents validation of complex stateful primitives and termination logic to ensure task completeness.These checks extend beyond action alignment to completion verification.
Loading 2601.15876v2…