PaperlayerView exact revision on arXiv

Source-linked AI summary

CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents

Bowen Wang, Dunjie Lu, Junli Wang, Tianyi Bai, Shixuan Liu, Zhipeng Zhang, Haiquan Wang, Hao Hu, Tianbao Xie, Shuai Bai, Dayiheng Liu, Que Shen, Junyang Lin, Tao Yu

arXiv:2605.25624v2cs.AIcs.LG

TL;DR

Scalable computer-use agent training lacks data that combines executable tasks with deterministic, verifiable rewards across applications. CUA-Gym co-generates and filters training tuples and adds mock web environments; its trained models reach 62.1% on OSWorld-Verified and also improve on held-out WebArena.

  • Problem

    Scalable CUA RLVR data remains limited because task instructions, executable environment states, and deterministic rewards must be jointly engineered across applications.

    Source: §1 · p. 2

  • Method

    CUA-Gym uses coordinated agents to co-generate task instructions, environment states, and rewards, filters tuples with LLM voting and rollouts, and adds 94 controllable mock web applications.

    Source: §1 · p. 2, §2.1 · p. 4, §2.2 · p. 4

  • Results

    CUA-Gym-A3B reaches 62.1% on OSWorld-Verified, and trained checkpoints also improve on held-out WebArena while OSWorld performance scales with data volume without saturation in the examined regime.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The results support scaling CUA RLVR through data volume and environment diversity, with transfer to a held-out browser benchmark.

    Source: §3.1 · p. 6, §7 · p. 10

    Rewards verify terminal states rather than the path taken, so they cannot distinguish a clean edit from a destructive sequence that recreates the same final state.

    Source: §6 · p. 10

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has driven breakthroughs in domains such as math, tool-use, and software engineering, yet its extension to computer-use agents (CUAs) has been bottlenecked by the scarcity of scalable training data with deterministic rewards. Constructing such data for CUAs requires consistent task instruction, executable environment, and verifiable reward. However, hand-curated benchmarks achieve high reward fidelity but cover few applications and LLM-as-judge-based datasets scale broadly but lack reliable verification. We present CUA-Gym, a scalable pipeline that co-generates task instructions, environment states, and reward functions. Concretely, a Generator agent constructs the initial and golden environment states, and a separate Discriminator agent writes the reward function from the task specification. An orchestrator agent drives the two through iterative rounds upon execution. Generated tuples then pass a final filter combining LLM majority voting and agent rollouts, ensuring quality beyond the per-task adversarial loop. To address the scarcity of training environments, we further synthesize CUA-Gym-Hub, a broad suite of high-fidelity mock web applications grounded in real-world software-use distributions, expanding the scale of CUA RLVR data by magnitude. Using this pipeline, we construct CUA-Gym, a dataset of 32,112 verified RLVR training tuples grounded in 110 environments. Trained with GSPO on CUA-Gym, our CUA-Gym-A3B and CUA-Gym-A17B achieve 62.1% and 72.6% on OSWorld-Verified, outperforming prior open-source CUAs at comparable scales, with performance scaling smoothly in both data volume and environment diversity. The same checkpoints also improve on the held-out WebArena benchmark, indicating transfer beyond the training environments. We will open-source the full synthesis pipeline, dataset, CUA-Gym-Hub environments, and models.

1 Introduction

CUA-GYM addresses the shortage of scalable, verifiable computer-use training data by co-generating tasks, environments, and rewards, and by expanding the pool of reusable environments. Its trained models achieve strong OSWorld-Verified results, while experiments identify environment diversity as a complementary scaling axis.

1 Introduction

  • Existing CUA approaches lack simultaneous deterministic rewards, broad coverage, and scalable task diversity: imitation [24] [31], judge scoring [33], and code-native pipelines [5] [37] each fall short.The authors identify the bottleneck as structural: each task requires an instruction, executable environment state, and reward function, whose hand-authoring takes hours.Source: §1 · pp. 1–2
  • A Generator creates initial and golden states, a Discriminator independently writes rewards, and an Orchestrator iterates them; final filtering combines LLM voting and agent rollouts.The pipeline isolates reward construction from the Generator’s scripts to mitigate reward hacking and verifies the tuple through execution.Source: §1 · p. 2
  • CUA-GYM-A3B and A17B reach 62.1% and 72.6% on OSWorld-Verified, setting open-source CUA state-of-the-art results at their respective scales.The smaller A3B matches the untrained A17B base with roughly 10× fewer total parameters.Source: §1 · p. 2
  • CUA-GYM contains 32,112 verified training tuples across 110 environments, including 16 desktop applications and 94 synthesized mock web applications.The trained checkpoints also improve their base models on held-out WebArena, showing transfer to a real browser benchmark.Source: §1 · p. 2
  • Expanding the environment pool from 10 to 80 yields gains that trajectory volume alone cannot recover, identifying environment diversity as a complementary scaling axis.The environments are selected using occupational taxonomies and software-use distributions, then implemented as reusable mock web applications that support thousands of tasks each.Source: §1 · p. 2
  • RL training induces multi-action tool calls that compress trajectories by 33–45% at matched task performance.The paper describes this as an emergent efficiency behavior.Source: §1 · p. 2

2 Methods

CUA-Gym co-generates task instructions, reproducible environment states, and verifiable rewards, then filters tuples for end-to-end utility. CUA-Gym-Hub expands environment coverage with reusable, programmatically controlled mock applications.

2.1 Data Synthesis

  • The pipeline generates each task as an instruction-context pair, then uses isolated Generator and Discriminator agents to build initial and golden states and a progressive reward.The context draws on real-world usage research, software documentation, and prepared assets; tasks are annotated by difficulty, domain, and applications. An Orchestrator iterates the agents until agreement conditions are met.Source: §2.1 · p. 3
  • Accepted tuples pass an LLM majority-vote assessment and teacher-model rollout to detect ambiguity, infeasibility, and rewards that fail to track task success.The vote evaluates consistency, executability, hack-risk, clarity, and difficulty calibration; rollouts verify solvability and reward alignment.Source: §2.1 · p. 4

2.2 Environment Scaling

  • CUA-GYM-HUB provides 94 self-contained mock applications with a unified HTTP API for state inspection, injection, and reset, enabling reusable task-generation environments.The applications span widely used products and long-tail targets; structured state transitions provide an unambiguous substrate for reward computation.Source: §2.2 · p. 4
  • A Plan Agent specifies each mock from web research and reference screenshots; Dev and Web Agents iteratively implement and test its interactive elements against a UI-layout oracle.The resulting mock includes a SKILL.md documenting its API and verification templates for the data-synthesis pipeline.Source: §2.2 · p. 4
  • State injection lets one mock support many task-specific worlds, while session isolation scopes mutations and resets so parallel RL workers can share backends without seeing one another’s changes.The application code and interface remain unchanged as injected state alters the available entities and feasible tasks.Source: §2.2 · p. 4, Figure 3 · p. 5

3 Experiments

Experiments evaluate agents trained on CUA-GYM data; RLVR improves both model scales on OSWorld-Verified and transfers to held-out WebArena.

3.1 Experimental Setup

  • The RLVR training set contains 10,858 verified tuples across 80+ environments, plus 3,578 successful Claude-Sonnet-4-6 rollout trajectories for SFT warm-up.The environment pool combines O*NET and Anthropic Economic Index [3]-grounded mock web applications with representative desktop applications.Source: §3.1 · p. 5, §3 · p. 5
  • Trajectory slicing retains recent multimodal observations and actions while replacing stale screenshots with placeholders, preserving late-turn supervision and reusable KV-cache prefixes.It generates multiple training samples per rollout under a fixed context budget rather than discarding older state with a sliding window.Source: Figure 4 · p. 5, §3.1 · pp. 5–6
  • Training uses GSPO [38] with verifiable rewards in [0, 1], group-normalized advantages, sequence-level importance ratios, clipping, and a KL penalty against the reference policy.Source: §3.1 · p. 6
  • 62.1% OSWorld-Verified for CUA-GYM-A3B vs. 54.5% for Qwen3.5-35B-A3B, and 72.6% for CUA-GYM-A17B vs. 62.2% for Qwen3.5-397B-A17B.These are gains of +7.6 pp and +10.4 pp, respectively.Source: Figure 5 · p. 6, §3.1 · p. 6
  • 44.5% WebArena for CUA-GYM-A3B vs. 40.8% for Qwen3.5-35B-A3B, and 56.0% for CUA-GYM-A17B vs. 54.0% for Qwen3.5-397B-A17B.WebArena site clones are disjoint from the 94 mock web applications used during training.Source: Figure 5 · p. 6, §3.1 · p. 6

4 Analysis

Across CUA-GYM analyses, increasing verified-data scale and environment diversity improves training outcomes, while RL also produces stable, more efficient multi-action behavior. The dataset and CUA-GYM-HUB provide broad, reusable coverage for these experiments.

4 Analysis

  • CUA-GYM contains 32,112 verified training tuples across 110 environments, with the web portion released as the reusable CUA-GYM-HUB suite.Its pool includes 94 web application categories based on O*NET occupational taxonomies and Anthropic Economic Index software-use distributions, plus a complementary desktop suite. No task category exceeds 21%; 45% of tasks are hard and 38% involve cross-application workflows. The authors describe it as, to their knowledge, the largest open-source corpus combining programmatic verification with joint desktop and web coverage.Source: §4 · p. 7

4.1 Data Scaling

  • The 12K-tuple run reaches the highest peak and sustains higher training curves than the 3K and 1.4K runs, with no observed saturation or instability.The runs preserve their ordering throughout training; all improve monotonically from the SFT baseline. The 12K curve departs from that baseline earlier, while the smaller runs plateau closer to it.Source: §4.1 · pp. 7–8, Figure 7 · p. 8

4.2 Environment Scaling

  • The broad 80-environment setting outperforms the narrow 10-environment setting despite using 4× fewer trajectories per environment.At fixed total data, expanding coverage from 10 to 80 environments yields a modest improvement; doubling trajectories on the 80-environment pool produces a substantially larger improvement. Distillation comparisons isolate these effects while holding student initialization and SFT hyperparameters constant.Source: Figure 8 · p. 8, §4.2 · pp. 8–9

4.3 Emergent Multi-Action Tool Calls

  • RL raises average tool calls from approximately one per step to a stable 1.4–1.9 band, shortening trajectories by 33–45% at matched task performance.The authors attribute batching to step-budget pressure under group-normalized advantage estimation: rollouts pack deterministic action sequences into turns but omit actions needing visual verification or dependent on nondeterministic state. This reduces rollout cost and inference latency without architectural or reward-shaping changes.Source: Figure 9 · p. 9, §4.3 · p. 9

5 Related Work

Prior digital-agent research shows that executable task synthesis with programmatic verification can scale RLVR training data, while GUI-agent work has not combined deterministic rewards, broad OS-level coverage, and scalable task diversity. Existing GUI approaches trade application generality against reliable verification, motivating CUA-Gym’s focus on that gap.

Data Synthesis and Post-Training for Digital Agents

  • Executable task synthesis with programmatic checks is the dominant recipe for scalable RLVR data, and prior software and terminal pipelines report continued scaling with task volume.SWE-smith and SWE-Gym report log-linear performance scaling unsaturated at tens of thousands of tasks; Endless Terminals observes the same trend for terminal operation.Source: §5 · p. 9

Task and Environment Synthesis for GUI Agents

  • GUI reward estimation generalizes across applications but can destabilize training, while code-native reward oracles support controllable web environments confined to browser interactions.ZeroGUI [33] reports false-positive rewards destabilizing training; GUI-Genesis [5], InfiniteWeb [37], and AutoWebWorld [26] use code-native oracles.Source: §5 · pp. 9–10
  • No existing GUI-agent work combines deterministic verifiable rewards, broad OS-level application coverage, and scalable task diversity for RLVR training.Desktop-focused efforts either use checklist-based VLM evaluation [1] or cover a narrow set of applications [19] [32] [30] [28]; OSWorld [27] is a complementary real-desktop evaluation benchmark.Source: §5 · p. 10

6 Limitations

CUA-Gym scales verifiable rewards by checking terminal states, but this misses how agents reach them and does not eliminate reward hacking. Its mock environments omit some real-world behaviors, and single-seed large runs limit conclusions about the pipeline’s ceiling and deployment robustness.

6 Limitations

  • Terminal-state rewards cannot distinguish a clean edit from a destructive sequence that recreates the same final state.The information barrier and forbidden-pattern scan reduce, but do not formally eliminate, reward hacking through loose semantic checks or state proxies correlated with task success.Source: §6 · p. 10
  • CUA-Gym-Hub mock environments approximate real applications and omit authentication flows, third-party integrations, network latency, rate limits, and rare server-side failures.Source: §6 · p. 10
  • The largest RL runs use single seeds, so results indicate the value of verified task and environment scale in the studied regime, not the pipeline’s ceiling or deployment robustness.The authors cite compute cost as the reason for single-seed runs.Source: §6 · p. 10

7 Conclusion · Appendix Contents

CUA-GYM combines adversarial task-and-reward generation with rollout-based filtering, while CUA-GYM-HUB broadens the available environments. The conclusion reports continued data scaling on OSWorld-Verified, transfer to WebArena, and release of the pipeline, environments, and data.

7 Conclusion

  • On OSWorld-Verified, models trained on CUA-GYM data scaled with data volume without saturating in the examined regime, and the same checkpoints improved on held-out WebArena.The authors interpret these results as evidence that task and reward diversity were a substantive bottleneck for current GUI agents, not as evidence of the pipeline’s ultimate ceiling.Source: §7 · p. 10
  • CUA-GYM co-generates task instructions, environment states, and reward functions for reinforcement learning with verifiable rewards training of computer-use agents.Source: §7 · p. 10
  • A Generator and Discriminator operate under a process-level information barrier, with an iterative loop enforcing tuple consistency at both endpoints.Source: §7 · p. 10
  • A dataset-level filter catches failures that become visible only under rollout.Source: §7 · p. 10
  • CUA-GYM-HUB comprises 94 mock web applications grounded in O*NET occupational taxonomies and the Anthropic Economic Index.Source: §7 · p. 10
  • The authors release the full synthesis pipeline, CUA-GYM-HUB environments, and generated data for studying task and environment scaling.Source: §7 · p. 10

A Data Synthesis Pipeline Details · A.1 Pipeline Pseudocode

The pipeline co-generates task states and a reward function through isolated virtual machines and role-separated agents, then accepts only tuples that converge and pass dataset-level filtering.

A.1 Pipeline Pseudocode

  • The Orchestrator provisions isolated initial and golden virtual machines and dispatches the Generator and Discriminator under an information barrier.The Generator creates setup and golden-patch scripts; the Discriminator reads the task specification, context, and domain skill to write the reward function.Source: §A.1 · p. 14
  • The loop evaluates the reward on both states, repeats with structured feedback until all five agreement conditions hold, and returns failure if the round budget is exhausted.Before evaluation, the reward code is statically scanned against a forbidden-pattern list.Source: §A.1 · p. 14
  • A converged tuple is submitted to the dataset-level filter and returned as verified only if that filter accepts it.The output is the task instruction, initial state, golden state, and reward function—or failure.Source: §A.1 · p. 14

A.2 Task Generation … A.6 Output Bundle and OSWorld Compatibility

This section describes task sampling for coverage and diversity, tuple checking and filtering, and domain-specific SKILL.md guides for synthesis. It highlights the guides’ versioned “bitter lessons,” which record recurring debugging failures to avoid rediscovering them in later adversarial rounds.

A.2.3 Three-Pass Generation: Breadth, Gap-fill, Edge Cases

  • Tasks are sampled across feature taxonomies and a five-dimensional scenario matrix, then generated in breadth, gap-fill, and edge-case passes.Taxonomy leaves describe atomic UI capabilities; the worked Calc example has 147 leaves across nine top-level subtrees, with at least 25 tasks targeted per leaf. Later passes emphasize under-covered cells, cross-application and hard tasks, and boundary cases.Source: §A.2.1 · p. 14, §A.2.2 · p. 14, §A.2.3 · p. 14

A.2.4 Anti-Repetition Rules

  • Near-duplicate instructions are rejected using embedding similarity, 4-gram overlap, and per-application template limits, while hard and cross-application tasks must comprise at least 40% and 35% of the corpus.No domain may exceed 21%, each application must receive at least 1,000 verified tuples and remain below 15%, and sampling ends once cell minima and the project budget are exceeded.Source: §A.2.4 · p. 15, §A.2.5 · p. 15

A.3.1 Five Agreement Conditions (Formal Definition)

  • Loop acceptance requires executable initial and golden states, rewards of 0.0 and 1.0 at those endpoints, and a reward function without forbidden patterns.The Orchestrator checks all five conditions; Generator and Discriminator processes have isolated working directories and communicate only through the Orchestrator.Source: §A.3.1 · p. 16, §A.3.2 · p. 16

A.3.3 Forbidden Pattern List for Anti-Hacking Static Scan

  • A static scan rejects reward code with forbidden verification patterns, and the feedback loop is capped at five rounds before an unresolved tuple is rejected.Failed rounds return a structured review with failing conditions and observed rewards; the Generator uses it as its sole feedback signal.Source: §A.3.3 · pp. 16–17, §A.3.4 · p. 17

A.4.1 LLM Majority Voting: Voters, Rubric, and Aggregation

  • Independent foundation-model critics admit tuples only when a strict majority votes keep or modify-query; ties reject, and the released instruction follows the majority.When modify-query wins, the highest-confidence revised query is selected; severity is reported as the maximum assigned by any voter.Source: §A.4.1 · pp. 17–18

A.4.2 Teacher-Model Rollout Verification with VLM-as-a-Judge

  • Teacher rollouts score each tuple with both its programmatic reward and a VLM judge, accepting concordant partial-success cases and routing disagreements for reward refinement.Tuples with both scores at zero are removed, while those scoring one on the first trial are down-sampled; unresolved disagreements after two refinement passes are dropped. The rollout policy is Claude-Sonnet-4-6.Source: §A.4.2 · p. 18

A.4.3 Per-Stage Filter Yield

  • About 3,100 tuples are rejected by majority voting and another 1,278 by teacher rollouts, leaving 32,112 verified tuples.The stages catch different failure modes: static specification defects and, respectively, infeasibility or triviality under realistic agent behavior.Source: §A.4.3 · p. 18

A.5.1 SKILL.md Standard Structure

  • Each application has a six-section SKILL.md guide, drafted from documentation research and refined using rejection logs to add recurring pitfalls.The Calc example covers setup, editing and verification APIs, progressive reward templates, and brittle behaviors such as formula values requiring cached results; guides are versioned with the pipeline.Source: §A.5.1 · p. 18, §A.5.2 · pp. 18–19, §A.5.3 · pp. 19–20

A.5.3 Example: Excerpt of a Domain SKILL.md

  • Versioned “bitter lessons” record recurring debugging pitfalls from early development, helping Generator and Discriminator agents avoid wasting adversarial rounds rediscovering them.New entries are added when recurring failure modes appear in rejection logs.Source: §A.5.3 · p. 20

A.6.1 File Layout per Verified Task … B.4.1 Plan Agent: Web Research and DESIGN.md

CUA-Gym’s appendices specify a drop-in verified-task bundle and a reproducible pipeline for building mock web environments, from occupationally grounded application selection to deterministic state verification and multi-agent design. The mocks retain reference-inspired interfaces while removing network-dependent behavior, with compliance review for sensitive cases.

A.6.1 File Layout per Verified Task; A.6.2 config.json Schema and Evaluator Contract

  • Each verified task bundles setup, golden-state, reward, metadata, and instruction files, while its unextended config.json follows the OSWorld evaluator contract for drop-in compatibility.The same bundle is consumed by OSWorld, the SFT teacher-rollout pipeline, and the RL trainer.Source: §A.6.1 · p. 21, §A.6.2 · p. 21

B.1.1 Mapping O*NET SOC Major Groups to Web Application Categories; B.1.2 Anthropic Economic Index Weighting; B.1.3 Coverage Cutoff Criteria

  • Mock categories map many-to-many from O*NET occupational groups, with Anthropic Economic Index usage weights prioritizing synthesis and a cutoff excluding categories below 1% of weighted usage.The approach favors high-fidelity coverage of the head of the usage distribution over lower-fidelity coverage of the full long tail.Source: §B.1.1 · p. 21, §B.1.2 · p. 22, §B.1.3 · p. 22

B.2.1 Application-by-Application Inventory; B.2.4 Per-Mock LOC and Endpoint Statistics

  • The released suite contains 94 synthesized mocks; 91% of measured mocks fall in the 2.5K–10K LOC band.The inventory groups applications by category; structural measurements cover 99 directories, including five scaffolds, and examine LOC, route count, and entity count. Figure 10 shows 32 representative mock landing pages.Source: §B.2.1 · p. 22, Table 3 · p. 22, Figure 10 · p. 23

B.2.2 Real-World Reference Mapping and Fidelity Notes; B.2.3 Trademark, Trade Dress, and Asset Compliance

  • Mocks preserve reference-inspired layouts, navigation, and features but replace authentication, external data, and network calls; release compliance includes generic renaming, original assets, and additional review for TOS-sensitive applications.TOS-sensitive financial, healthcare, or government mocks may be withheld and replaced with category-level substitutes when functional reproduction is forbidden.Source: §B.2.2 · pp. 22, 24, §B.2.3 · p. 24

B.3.1 Endpoint Contracts; B.3.2 Session Isolation via sid

  • A same-origin four-endpoint API supports state injection, inspection, reset, and uploads, with UUID session identifiers isolating concurrent rollouts.The /post endpoint controls initial and current snapshots, /go returns state differences, /state exposes raw current state, and /upload supports task attachments.Source: §B.3.1 · p. 25, §B.3.2 · p. 25

B.3.3 State Diff Computation Rules; B.3.4 Integration with reward.py

  • The /go state diff uses flat key paths, ordered-array comparisons, and masking of volatile fields; reward.py turns its changes into partial-credit assertions scored from 0 to 1.The canonical reward pattern fetches the diff once and accumulates credit across assertions.Source: §B.3.3 · p. 26, §B.3.4 · p. 26

B.4.1 Plan Agent: Web Research and DESIGN.md

  • Before coding, the Plan Agent researches each target application and produces DESIGN.md, workflow and layout notes, an in-memory data model, and a prioritized Dev Agent work queue.Its contract excludes authentication, real network communication, and persistence beyond localStorage, defining the mock as an interactive sandbox rather than a functional reference-service clone.Source: §B.4.1 · p. 26

B.4.2 Dev Agent: Implementation Protocol … B.5.3 Salesforce Mock: Form-Heavy Workflow

The mock-building protocol combines file-based implementation, automated UI and code audits, and bounded repair cycles. Case studies show distinct verifiable state models for communication, project-management, and CRM workflows.

B.4.2 Dev Agent: Implementation Protocol

  • The Dev Agent builds each mock as a Vite + React single-page app with a fixed project layout, file-based coordination, and a priority order that addresses correctness issues before feature work.It consumes Plan Agent artifacts; it does not message the Plan or Web Agents directly.Source: §B.4.2 · p. 26

B.4.3 Web Agent: Playwright Verification against UI Tree

  • The Web Agent uses headless Playwright to test every catalogued interactive element, compare the UI with design artifacts, and report functional, visual, and code-level issues in TEST.md and AUDIT.md.Reports use P0/P1/P2 severity levels aligned with the Dev Agent’s fix priorities.Source: §B.4.3 · p. 27

B.4.4 Iterative Convergence Loop

  • Plan, Dev, and Web Agents repeat test, fix, and re-test rounds until P0 and P1 issues are cleared or the round budget expires; unresolved mocks are flagged for review and excluded.The Plan Agent re-engages only for fundamental specification gaps.Source: §B.4.4 · p. 27

B.5.1 Slack Mock: State Schema and Feature Inventory

  • The Slack mock models channels with ordered, metadata-rich messages and a top-level current user, supporting user-switching tasks and verifiable channel, message, thread, reaction, and direct-message actions.It omits presence, push notifications, and workspace-level admin policies; tasks requiring them are deferred to other categories.Source: §B.5.1 · p. 27

B.5.2 Jira Mock: Cross-Entity State Transitions

  • The Jira mock enforces project, sprint, issue, workflow, and board invariants, and verifies multi-step tasks by inspecting diffs over the issues collection.Sprint membership must match the sprint’s project, status transitions follow the workflow graph, and board columns derive from issue status.Source: §B.5.2 · p. 27

B.5.3 Salesforce Mock: Form-Heavy Workflow

  • The Salesforce mock’s lead-to-opportunity conversion spans forms with ∼25 fields, and rewards verify populated fields, types, and value constraints rather than the UI path used.Agents may use any UI route; visually unfamiliar agents can solve these tasks once they identify the fields.Source: §B.5.3 · p. 27

C Training Details … C.2.2 SFT Hyperparameters

The training details define the CUA agent’s interaction format and rollout scaffold, then describe trajectory slicing and the SFT warm-up used to prepare training data and models.

C.1.1 Observation and Action Space

  • The agent observes screenshots and emits XML-wrapped tool calls; its action primitives cover pointer, keyboard, navigation, and control operations, with consecutive actions permitted in one call.The action set includes terminate for episode success or failure; call_user is reserved for evaluation, not RL.Source: §C.1.1 · pp. 27–28

C.1.2 Trajectory Slicing Scheme

  • Trajectory slicing lets rollouts that can reach roughly 200K tokens fit a 144K context cap by emitting samples every 10 turn-pairs, collapsing earlier screenshots into placeholders.The first slice preserves the raw trajectory; later slices retain older turns as text placeholders while keeping subsequent multimodal observations intact.Source: §C.1.2 · p. 28
  • Every slice inherits the parent episode’s full reward, while overflow slices become all-masked dummy samples; fewer than 1% of slices overflow in practice.Only assistant responses in the response portion contribute loss, excluding stale-policy messages and turns with more than 10 tool calls.Source: §C.1.2 · pp. 28–29

C.1.3 Inference-Time Configuration

  • Rollouts stop at 100 model steps or a terminate action, use at most 2,048 output tokens per turn, and retry sampler-side failures up to 5 times before failed rollouts are discarded.Training and validation share an sglang decoding stack; validation uses temperature 0.6 and top-p 0.95, while training uses the rollout-config temperature, defaulting to 1.0.Source: §C.1.3 · p. 29

C.1.4 System Prompt and Tool Definitions

  • A constant system prompt defines tool-call grammar, response shape, and termination semantics, while each task instruction arrives in the first user message.The prompt provides no task-specific information.Source: §C.1.4 · p. 29

C.2.1 Teacher Rollout Curation

  • SFT initialization retains only successful teacher-policy rollouts, sampling 4 per task at temperature 1.0 to produce 3,578 trajectories and discard about 50% of rollouts.The corpus spans the full task distribution, with no additional SFT-stage reweighting or domain rebalancing.Source: §C.2.1 · p. 30

C.2.2 SFT Hyperparameters

  • SFT uses assistant-token cross-entropy for 1 epoch with AdamW, peak learning rate 7×10^-6, batch size 512 prompts, and 256K-token sequences across 512 GPUs.It uses cosine decay to 7×10^-7, weight decay 0.01, bf16 mixed precision with FP8 matmul weights, and the same trajectory slicing as RL.Source: §C.2.2 · p. 30

C.3 GSPO Training … C.4.3 Wall-Clock Time and Cost per Run

The training setup combines GSPO-specific choices with large, disaggregated training and rollout clusters. The reported runs take about five days each, with cost dominated by rollout while a 2,000-VM pool keeps rollout latency from becoming a bottleneck.

C.3.1 Hyperparameters

  • The GSPO runs add a Chinese-token penalty with coefficient 0.5, disable format, language, and endless penalties and group filtering, and keep the entropy bonus at zero.Other hyperparameter values follow the GSPO formulation in §3.1 except where noted; Table 6 reports the values used for all RL runs.Source: §C.3.1 · p. 30, Table 6 · p. 31

C.3.2 Reference Policy Update Schedule

  • Setting β = 0 avoids loading or evaluating a reference policy, freeing GPU memory for rollout batches and enabling 24 rollout instances per node without contention.Early tests at β ∈{10−3, 10−2} showed neither training-stability gains nor task-success improvements at the reported data scales.Source: §C.3.2 · p. 30

C.3.3 Advantage Normalization Details

  • Advantages use group-wise mean-centered rewards across G = 16 rollouts, without standard-deviation normalization.Because rewards are already scaled to [0, 1], dividing by the group standard deviation could amplify noise on near-uniform groups; zero-variance groups yield zero advantage and no gradient.Source: §C.3.3 · p. 31

C.4.1 Hardware and Cluster Setup

  • The runs use scale-matched turbo and plus clusters of 192 and 512 NVIDIA H200 GPUs, respectively.Each cluster splits evenly into training and rollout subclusters. Both use TP = 2, EP = 8, and CP = 4; pipeline depth is PP = 2 for turbo and PP = 8 for plus. Rollouts use TP = 8, data-parallel replication factor 4, speculative decoding, and sglang.Source: §C.4.1 · p. 31

C.4.2 VM Provisioning for Rollouts

  • A pool of 2,000 OSWorld VMs achieves over 75% average utilization per training step, and rollout latency is not the training bottleneck.The router restores the task-configured snapshot for each rollout and streams screenshots and rewards; after termination, it resets the VM to a clean snapshot before release.Source: §C.4.2 · pp. 31–32

C.4.3 Wall-Clock Time and Cost per Run

  • Both 1000-step runs take approximately 5 days: Qwen3.5-35B-A3B uses ∼23,040 GPU-hours, while Qwen3.5-397B-A17B uses ∼61,440 GPU-hours.Cost is dominated by rollout, with the trainer comparatively idle while waiting for episode completions. The 1.4K and 3K data ablations finish proportionally faster but use the same per-step compute because trajectory volume, not step count, differs.Source: §C.4.3 · p. 32

C.5 Reproducibility Checklist

The appendix reproduces the NeurIPS Paper Checklist, including Yes/No/NA responses and justifications cross-referenced to relevant sections.

C.5 Reproducibility Checklist

  • The checklist is reproduced in full, with responses and justifications cross-referenced to the synthesis pipeline, hyperparameters, infrastructure, and limitations.The references point to §§B.4, C.3.1, C.4.1, and 6, respectively.Source: §C.5 · p. 32

D Extended Experiments … E.1.3 Cross-App – Slack + Impress + PDF: Prioritization Deck

This section presents extended experiments and released-task examples, characterizing learned multi-action behavior in OSWorld-Verified rollouts and illustrating realistic setups with decomposed, verifiable rewards. Examples include VS Code file tasks and a cross-app workflow that turns Slack poll results into an Impress prioritization deck and PDF, with rewards checking the resulting artifacts.

D.1 Emergent Multi-Action Behavior

  • Across 1,105 rollouts, the checkpoint averaged 1.41 tool calls per model step; 69.6% of steps had one call and 26.7% had two.Three-, four-, and five-or-more-call steps accounted for 2.4%, 0.8%, and 0.5%, respectively.Source: §D.1 · p. 32

D.1.1 Top-K Batched Action Patterns

  • Type→key and mouse_move→scroll dominate batching, accounting for ∼75% of batched steps; action-and-observe pairs and longer macros appear less often.Table 7 reports the most frequent length-≥2 sequences, including type→key (3,942 steps) and mouse_move→scroll (1,052 steps).Source: §D.1.1 · p. 32, Table 7 · p. 32

D.1.2 Avoided Non-Deterministic Action Categories

  • Pointer-click variants appeared alone 94–98% of the time and left_click 90.6%, while scroll, key_down, key_up, and left_click_drag were never solo.The reported pattern batches mechanical gesture components and deterministic chains, but defers across actions whose outcomes depend on UI state.Source: §D.1.2 · p. 33, Table 8 · p. 33

E.1 End-to-End Task Walk-throughs

  • The walkthroughs present four released dataset tuples across desktop, web, and cross-application domains, quoting task instructions and summarizing reward decompositions with distinctive reward-code components.Full setup and reward scripts are released with the data.Source: §E.1 · p. 33

E.1.1 Desktop – LibreOffice Calc: Padded ID Formula

  • The Calc task asks for TEXT formulas to pad employee IDs to five digits, with reward split across the first formula, propagation to B3:B6, and correct A-column references.The components are weighted 0.40, 0.30, and 0.30, respectively.Source: §E.1.1 · p. 33

E.1.2 Desktop – VS Code: Custom File Associations

  • The VS Code task maps *.config files to JSON and *.tmpl files to HTML; reward parses settings.json, while verification is file-based rather than GUI- or screenshot-based.The three reward components check for a non-empty files.associations object, the *.config mapping, and the *.tmpl mapping, weighted 0.35, 0.35, and 0.30.Source: §E.1.2 · pp. 37–38
  • The VS Code reward assigns 0.35 each to the associations key and *.config → json mapping, and 0.30 to *.tmpl → html.It checks that the key is non-empty and that both extension mappings match.Source: §E.1.2 · pp. 41–42

E.1.3 Cross-App – Slack + Impress + PDF: Prioritization Deck

  • The cross-app task reuses Slack state, creates a 5-slide PPTX deck, exports it to PDF, and checks the deck structure and PDF page count.The reward also verifies the PDF artifact exists.Source: §E.1.3 · p. 42
  • The PPTX reward checks exactly 5 slides, title and quarter text, voting-methodology counts, all 10 ranked features with vote counts, and top-5 effort estimates.These are components C1–C5 of the seven-part reward.Source: §E.1.3 · p. 42
  • 0.15 reward components separately require exactly 3 Q2 roadmap commitments from the top-5 and a PDF artifact with 5 pages.The roadmap and PDF checks are components C6 and C7.Source: §E.1.3 · p. 43
  • The Slack mock supplies a poll-results post with 10 candidate features and customer, internal, and total vote counts for transfer into the deck.The task requests reading the #product-feedback results posted by Frank Zhang.Source: §E.1.3 · pp. 42–43
  • The setup reuses the same Slack session state for initial and golden VMs so cross-VM reward inspections agree; the final reward is capped at 1.0.The scoring script totals criterion-level scores before applying the cap.Source: §E.1.3 · pp. 44, 49
  • The task builds a five-slide Q2 2026 prioritization deck from Slack data, with methodology, ranked results, top-five deep dives, and roadmap commitments, then exports a five-page PDF.The specified deck includes methodology based on 28 customers and 12 internal stakeholders, a table covering 10 features, and a Q2 commitment slide with three committed items and seven deferred to Q3.Source: §E.1.3 · pp. 44–45
  • The reward rubric checks slide count, title and methodology text, ranked-table feature names and vote totals, top-five effort estimates, roadmap commitments, and PDF page count.The ranked-results check is weighted 0.25, while the other criteria carry weights from 0.10 to 0.15.Source: §E.1.3 · p. 45

E.1.4 Cross-App – PDF →Calc: Multi-Currency Expense Analysis … E.3 Reward Hacking Cases Caught by the Information Barrier

This section presents task and reward-verification examples spanning PDF-to-Calc spreadsheet work, Shopify vendor consolidation, and product verification. It also describes information barriers and static scans used to address reward-hacking patterns in generated tuples.

E.1.4 Cross-App – PDF →Calc: Multi-Currency Expense Analysis

  • The agent extracts a 50-row, five-page PDF of multi-currency expenses and creates a Calc workbook with verification formulas, discrepancy flags, a SUMIFS summary sheet, and a pie chart.The source is synthesized from scratch; the report covers 50 expenses across six currencies.Source: §E.1.4 · pp. 49, 51
  • Six independently scored components cover file structure, two formula columns, yellow highlighting, a SUMIFS summary, and a pie chart, with weights totaling 1.0.The components assign 0.20 to file structure and yellow flags, and 0.15 to each formula column, the summary sheet, and the chart.Source: §E.1.4 · p. 49
  • Partial completion earns 0.85 when either the chart or discrepancy column is omitted, while only the full workflow earns 1.0.This makes completion of every workflow component necessary for full credit.Source: §E.1.4 · p. 49
  • The generated data includes six rows with discrepancies, testing whether the agent flags differences between stated USD amounts and amounts computed from local values and exchange rates.The task instruction specifies yellow fill for rows where discrepancy exceeds $0.01.Source: §E.1.4 · pp. 49–51
  • The spreadsheet reward verifier checks formula and formatting requirements, a currency summary with SUMIFS, and a pie chart, then caps the reward at 1.0.Its listed criteria include exactly six yellow-filled rows and currency-summary checks.Source: §E.1.4 · pp. 55–56

E.2 Full Rollout Trajectory: Shopify Vendor Consolidation

  • The Shopify task changed products from BasicWear and HomeGoods to UnifiedBrands, appended a fixed marketing line, and left other vendors’ products untouched.The rollout reports both targeted products updated and the LeatherCo and SportStep products unchanged.Source: Step 15 · p. 65
  • The rollout demonstrates chained actions, per-product completion checks, and recovery by returning to the product list after a navigation slip.The recovery pattern follows a failed click when attempting to reopen Ceramic Mug.Source: Step 17 · p. 67

Step 18.

  • At Step 18, the trajectory identifies the mug as the item whose description is being confirmed.Source: Step 18 · p. 67

Step 19.

  • The task ends with reward 1.0 after both products are verified, while the Leather Wallet and Running Shoes remain untouched.The Ceramic Mug is verified with vendor UnifiedBrands and a description including “Now part of the UnifiedBrands family.”Source: Step 19 · p. 68

E.3 Reward Hacking Cases Caught by the Information Barrier

  • The information barrier and forbidden-pattern scan are designed to prevent reward hacking; offending tuples are rejected at write-time, and feedback prompts a reward that performs actual verification.Source: §E.3 · pp. 68–69
  • Two described exploits are direct Boolean assignment without computation and awarding credit for file existence without inspecting contents.The static scan flags a constant verification flag and a file-existence check as forbidden patterns.Source: §E.3 · pp. 68–69

E.4 Adversarial Loop Iteration Trace … F.1 Distributional Breakdowns

The trace shows a targeted Generator–Discriminator correction turning a failed workbook tuple into an accepted one. The reward-function gallery demonstrates verification beyond cell and formula checks through chart, image, and filesystem structure.

E.4 Adversarial Loop Iteration Trace

  • In Round 1, a copy-paste regression gave Optimistic the Pessimistic values, so the golden-state reward scored 0.65 and triggered targeted feedback.The task required two named scenarios with distinct 5-tuples, and the reward parsed the XLSX XML to verify them.Source: §E.4 · p. 69
  • Round 2 accepted the corrected tuple: reward(golden) == 1.0 and reward(initial) == 0.0, with both scenarios’ names and distinct value sets passing.The reward checks the workbook’s underlying XLSX XML; the golden file’s scoring breakdown assigns points to scenario names and each scenario’s values.Source: §E.4 · p. 70
  • A tightly scoped diagnosis followed by a targeted fix in Round 2 accounts for the majority of multi-round convergences in the released dataset.After receiving the feedback, the Generator wrote distinct inputCells and the Discriminator re-scored both endpoint states.Source: §E.4 · p. 70

E.5 Reward Function Gallery

  • Chart rewards inspect chart presence and type, data range, title, and axis labels through spreadsheet chart abstractions.The example checks a column chart and whether its series references the 12-month range.Source: §E.5 · pp. 70–71
  • Image-property rewards check page orientation, margins, and watermark-image anchors, while treating page size and order as preconditions rather than scored components.The example checks that all four margins are set to 1.5 cm and that an image is anchored on the target sheet.Source: §E.5 · p. 71
  • Filesystem rewards verify each file is in its category folder and absent from the Desktop root, preventing copy-without-delete actions from earning credit.These structural, image-property, and filesystem checks complement cell-formula, JSON-key, and document-content checks across the released dataset’s task distribution.Source: §E.5 · p. 71

F.1.1 Tasks per Environment … G Prompts Catalog

This section characterizes CUA-Gym’s task distribution, difficulty, action vocabulary, cross-application workflows, instruction lengths, and tuple-generation costs. It documents the prompts and agent roles for task generation and verification, filtering, orchestration, and mock-application development and testing.

F.1.1 Tasks per Environment

  • Knowledge-work scenarios account for 84% of the corpus, while no single scenario exceeds 21% across its 110 environments.Tasks also span multiple constituent applications within scenarios.Source: §F.1.1 · p. 72, Table 9 · p. 72

F.1.2 Domain × Difficulty Matrix

  • 44.6% of tasks are hard and 37.7% medium; multi-application domains are especially hard, including 95.8% hard in Desktop × Web.Cross-Desktop is 80.1% hard and Cross-Web 87.6% hard; Spreadsheet tasks are 17.8% easy, 47.2% medium, and 35.0% hard.Source: §F.1.2 · p. 72, Table 10 · p. 72

F.1.3 Action Verb Frequency

  • Surface-edit imperatives such as open, create, set, add, insert, change, and remove dominate task instructions, alongside composition, extraction, and verification verbs.The long tail includes verbs appearing in fewer than 50 tasks, such as merge, flatten, normalize, reconcile, and annotate.Source: §F.1.3 · p. 73, Table 11 · p. 73

F.1.4 Cross-App Pair Co-occurrence

  • 12,311 tasks (38.3% of the corpus) involve multiple apps, with document-production triplets accounting for most cross-app traffic.Developer pairs include terminal+vscode and files+vscode; web-mock pairs cluster around salesforce, gmail, slack, and notion. Pair counts are per pair, so one multi-app instruction may contribute to several pairs.Source: §F.1.4 · p. 73, Table 12 · p. 73

F.1.5 Instruction Length Distribution

  • Task instructions average 54.1 words (median 41), with 90% under 108 words and a long tail reaching 390 words.Short instructions typically describe single-step actions, while 100-plus-word instructions encode multi-stage workflows with conditional branches and acceptance criteria.Source: §F.1.5 · p. 73

F.2.1 Per-Tuple Token Consumption; F.2.2 VM-Hours per Verified Tuple

  • Each verified tuple costs approximately $0.11 in LLM use and $0.32 in VM compute, with dual-VM verification taking about 45 minutes.For 32,112 tuples, estimated LLM spend is $3,300–$3,500 and cumulative VM compute is approximately $10K; VM-hours dominate marginal tuple cost by roughly 3×.Source: §F.2.1 · p. 73, §F.2.2 · p. 74

G.1 Prompt Engineering Notes; G.2 Orchestrator System Prompt; Orchestrator system prompt

  • The Orchestrator coordinates the Generator-Discriminator loop, monitors five agreement conditions, and emits the final accepted tuple to disk without writing scripts; reward-gen gets only the sandbox directory and cannot see setup-gen scripts or data files.Reward-gen must explore the VM through the state-only API; this isolation is the pipeline’s load-bearing security guarantee.Source: §G.2 · p. 74, Orchestrator system prompt · p. 75

G.3 Generator Subagent System Prompt

  • Generator prompts distinguish Task-Gen instruction design from Setup-Gen creation of initial and golden states, requiring grounded task context and concrete expected outcomes.Task-Gen uses breadth, gap-fill, and edge-case passes with deduplication and diversity rules; Setup-Gen produces scripts in an adversarial review loop.Source: §G.3 · pp. 76–77
  • The Generator derives initial_setup.py and golden_patch.py from task context, enforces negative initial-state constraints, and makes targeted REVIEW.md fixes across rounds before VM execution.It must prepare the GUI start state and preserve an untouched initial reference copy for in-place multimedia tasks.Source: §G.3 · p. 78

G.4 Discriminator Subagent System Prompt

  • The Discriminator writes rewards from task semantics without seeing setup scripts, verifies golden reward 1.0 and initial reward 0.0, and scores only task-introduced changes.Rewards should use progressive partial credit and treat data-integrity checks as precondition gates rather than scoring components.Source: §G.4 · pp. 78–79

G.5 Five Agreement Condition Verifier Prompts

  • The five agreement conditions are checked programmatically rather than by an LLM prompt, with the Discriminator reporting its verdict, per-condition evidence, failures, and setup feedback in REVIEW.md.The schema covers whether both environments execute, whether golden and initial rewards equal 1.0 and 0.0, and whether forbidden patterns appear.Source: §G.5 · p. 80

G.6 Filter Critic Prompts

  • The dataset critic conservatively keeps, revises only the query, or rejects a task; setup, golden setup, and reward remain immutable, and high-risk or irreparable defects favor rejection.It rejects such problems as rewards tied to implementation traces or fragile state, and subjective tasks with rigid single-answer rewards.Source: §G.6 · pp. 80–81

G.7 Mock Synthesis Agent Prompts

  • Plan, Dev, and Web agents coordinate through file artifacts: Plan specifies mock workflows, Dev implements them, and Web tests interactions, visual fidelity, and state APIs.A round completes after TODO-listed interactions are exercised, visual differences fall below threshold, and state endpoints respond under session isolation.Source: §G.7 · pp. 81–82

Related papers

Loading 2605.25624v2…