PaperlayerView exact revision on arXiv

Source-linked AI summary

CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents

Bowen Wang, Dunjie Lu, Junli Wang, Tianyi Bai, Shixuan Liu, Zhipeng Zhang, Haiquan Wang, Hao Hu, Tianbao Xie, Shuai Bai, Dayiheng Liu, Que Shen, Junyang Lin, Tao Yu

arXiv:2605.25624v2cs.AIcs.LG

TL;DR

Computer-use-agent RLVR lacks scalable training data that combines deterministic rewards with broad application and task coverage. CUA-Gym co-generates verified tasks, environments, and rewards, producing models that reach 72.6% on OSWorld-Verified and improve on held-out WebArena.

  • Problem

    CUA RLVR lacks scalable datasets that simultaneously provide deterministic verifiable rewards, broad application coverage, and diverse tasks.

  • Method

    CUA-Gym uses coordinated agents to co-generate task instructions, environment states, and reward functions, then filters tuples through LLM voting and teacher rollouts.

  • Results

    Models trained on CUA-Gym improved over their bases on OSWorld-Verified and held-out WebArena; CUA-Gym-A17B reached 72.6% on OSWorld-Verified.

  • Takeaways & Limitations

    Performance scaled with data volume and environment diversity, while improvements on held-out WebArena indicate transfer beyond the synthesized training environments.

  • Takeaways & Limitations

    Rewards verify terminal states rather than action processes, and mock environments omit several real-application behaviors such as authentication and network failures.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has driven breakthroughs in domains such as math, tool-use, and software engineering, yet its extension to computer-use agents (CUAs) has been bottlenecked by the scarcity of scalable training data with deterministic rewards. Constructing such data for CUAs requires consistent task instruction, executable environment, and verifiable reward. However, hand-curated benchmarks achieve high reward fidelity but cover few applications and LLM-as-judge-based datasets scale broadly but lack reliable verification. We present CUA-Gym, a scalable pipeline that co-generates task instructions, environment states, and reward functions. Concretely, a Generator agent constructs the initial and golden environment states, and a separate Discriminator agent writes the reward function from the task specification. An orchestrator agent drives the two through iterative rounds upon execution. Generated tuples then pass a final filter combining LLM majority voting and agent rollouts, ensuring quality beyond the per-task adversarial loop. To address the scarcity of training environments, we further synthesize CUA-Gym-Hub, a broad suite of high-fidelity mock web applications grounded in real-world software-use distributions, expanding the scale of CUA RLVR data by magnitude. Using this pipeline, we construct CUA-Gym, a dataset of 32,112 verified RLVR training tuples grounded in 110 environments. Trained with GSPO on CUA-Gym, our CUA-Gym-A3B and CUA-Gym-A17B achieve 62.1% and 72.6% on OSWorld-Verified, outperforming prior open-source CUAs at comparable scales, with performance scaling smoothly in both data volume and environment diversity. The same checkpoints also improve on the held-out WebArena benchmark, indicating transfer beyond the training environments. We will open-source the full synthesis pipeline, dataset, CUA-Gym-Hub environments, and models.

1 Introduction

CUA-Gym addresses the structural bottleneck in computer-use-agent RLVR by co-generating task instructions, executable environments, and reward functions as verified tuples. The resulting dataset and environment suite scale training across applications and improve OSWorld-Verified performance.

  • Motivation: CUA RLVR requires jointly engineering a task instruction, executable environment state, and reward function, making scalable data construction substantially harder than in text-only domains.A single tuple is denoted (t, s, r), and hand-authoring one takes hours of expert effort; costs increase with application diversity.
  • Method: CUA-Gym uses coordinated Generator, Discriminator, and Orchestrator agents to synthesize and verify complete RLVR training tuples at scale.The Generator constructs initial and golden environment states, while the Discriminator writes the reward function; the Orchestrator coordinates them through iterative execution.
  • Scale: 32,112 verified RLVR training tuples span 110 environments, including 16 desktop applications and 94 synthesized mock web applications from CUA-Gym-Hub.CUA-Gym-Hub extends the pipeline’s reach with reusable synthesized web environments.
  • Results: 62.1% and 72.6% on OSWorld-Verified are achieved by CUA-Gym-A3B and CUA-Gym-A17B, respectively, after RLVR training with GSPO.The models are trained from Qwen3.5-35B-A3B and Qwen3.5-397B-A17B open-source MoE backbones.
  • Contributions: The paper identifies environment diversity as an independent scaling axis for CUA reinforcement learning and releases CUA-Gym, CUA-Gym-Hub, the synthesis pipeline, and models.These are presented among the paper’s three stated contributions.

2 Methods

CUA-GYM co-generates task instructions, reproducible initial states, and verifiable reward functions from a shared specification, then filters tuples through independent evaluation stages. CUA-GYM-HUB expands environment diversity with self-contained mock web applications synthesized and operated by multiple agents.

  • Data co-generation: RLVR training data is represented as tuples (t, s, r), with all components derived from one query specification for consistency by construction.Here, t is the task instruction, s the reproducible initial environment state, and r : s →[0, 1] the verifiable reward function.
  • Task generation: Tasks are generated as instruction-context pairs whose contexts specify required entities, files, and application states from usage research, documentation, and prepared assets.Each task is also annotated with difficulty, domain, and application.
  • Adversarial setup and reward co-generation: The pipeline separates environment construction and reward writing across Generator and Discriminator agents, with the Orchestrator provisioning two virtual machines for adversarial co-generation.The Generator produces initial and golden environments, while the Discriminator evaluates only the task and resulting environments in a sandboxed process.
  • Dataset-level verification: Every accepted tuple passes an LLM majority-vote filter and teacher-model rollout that assess consistency, executability, hack-risk, clarity, difficulty calibration, solvability, and reward tracking.The dataset-level filter addresses issues that emerge only during rollout, beyond loop-level convergence.
  • Environment scaling: CUA-GYM-HUB provides self-contained mock web applications that preserve real-world interactive fidelity while avoiding authentication, rate-limit, and non-reproducible-state barriers.State injection supports many task-specific worlds from one implementation, and session isolation enables concurrent distributed rollouts without shared mutations.

3 Experiments

Experiments evaluate CUA-GYM by training agents on verified tuples across two Qwen3.5 scales, using GSPO and a long-horizon trajectory scaffold. RLVR improves OSWorld-Verified performance at both scales, including the larger model.

  • Dataset: 10,858 verified tuples across 80+ environments formed the RLVR training set, supplemented by 3,578 successful Claude-Sonnet-4-6 trajectories for SFT warm-up.The trajectories were retained only after successful rollouts on the same task set.
  • Model: The experiments used Qwen3.5-35B-A3B and Qwen3.5-397B-A17B, representing two parameter scales.Both models were evaluated as bases for CUA-GYM training.
  • Scaffold: The training scaffold preserves late-trajectory supervision by collapsing stale screenshots into deterministic placeholders while retaining recent multimodal observations and reusable prefixes.This supports KV-cache reuse when computing policy log-probabilities over neighboring trajectory slices.
  • Algorithm: All models were trained with Group Sequence Policy Optimization (GSPO), using verifiable rewards and group-normalized advantages for mixture-of-experts RL stability.GSPO applies a clipped surrogate objective with sequence-level importance ratios and a KL penalty against a reference policy.
  • Main Results: 62.1% on OSWorld-Verified was achieved by CUA-GYM-A3B versus 54.5 for the Qwen3.5-35B-A3B base, a +7.6 pp improvement.At the larger scale, CUA-GYM-A17B reached 72.6 versus 62.2 for the Qwen3.5-397B-A17B base, a +10.4 pp gain.

4 Analysis

CUA-GYM analysis shows that increasing verified data and environment diversity improves RL performance, while RL also induces more efficient multi-action behavior. Across experiments, scaling remains effective without observed saturation, and trajectory length falls substantially at matched task performance.

  • Dataset composition: 32,112 verified RLVR training tuples span 110 environments, with coverage grounded in O*NET categories, Anthropic Economic Index usage distributions, and complementary desktop workflows.The environment pool contains 94 web application categories aligned with SOC major groups.
  • Data scaling: 12K verified tuples achieve the highest RL-training peak, diverge earlier from the SFT baseline, and sustain a higher performance band than the 3K and 1.4K runs.All runs use the same SFT initialization and training hyperparameters.
  • Data scaling: The 12K curve shows no inflection toward saturation, while all three runs improve monotonically from the SFT baseline without oscillation, collapse, or reward-success decoupling.These results indicate that the examined data-scaling regime has not been exhausted.
  • Environment scaling: 80 environments with 75 trajectories each outperform 10 environments with 300 each at the 100-step evaluation budget, despite using 4× fewer trajectories per environment.Environment diversity and data volume contribute as complementary scaling axes; adding environments at fixed data yields a modest improvement, while doubling trajectory volume on the broadened pool yields a substantially larger one.
  • Emergent action batching: RL increases average tool calls from approximately one per step at SFT initialization to a stable 1.4–1.9 band, shortening trajectories by 33–45% at matched task performance.The emergent action batching reduces rollout cost and deployment inference latency without architectural or reward-shaping intervention.

5 Related Work

Prior work scales RLVR for digital agents through executable-environment task synthesis and programmatic verification, while GUI-agent research confronts a tradeoff between reward verifiability and environment scope. Existing approaches estimate rewards with VLMs or synthesize controllable web environments with code-native reward oracles.

  • Data Synthesis and Post-Training for Digital Agents: Executable-environment task synthesis with programmatic outcome checks has become the dominant recipe for scalable RLVR data generation across digital-agent domains.RLVE complements this direction by scaling reinforcement learning with adaptive verifiable environments, highlighting environment generation as an important substrate.
  • Task and Environment Synthesis for GUI Agents: GUI-agent RLVR synthesis faces a fundamental tension between reward verifiability and environment scope.This tension motivates distinct approaches to reward estimation and environment construction.
  • Task and Environment Synthesis for GUI Agents: VLM-based reward estimation generalizes across applications without environment-specific instrumentation, but ZeroGUI finds false-positive rewards destabilize RL training.The instability is established through ZeroGUI’s ablation study.
  • Task and Environment Synthesis for GUI Agents: Alternative methods synthesize controllable web environments with code-native reward oracles, including per-task Flask applications, task-centric test-driven development, and finite-state-machine transitions.The cited systems are GUI-Genesis, InfiniteWeb, and AutoWebWorld, respectively.

6 Limitations

CUA-GYM rewards terminal states rather than full action processes, enabling scalable reward authoring but leaving some undesirable trajectories indistinguishable. Its safeguards reduce, but do not formally eliminate, residual reward-hacking modes.

  • Reward Verification Limitations: Terminal-state rewards cannot distinguish a clean edit from a destructive sequence that recreates the same final state.This design keeps reward authoring scalable while omitting the full process by which an agent reaches the outcome.
  • Reward Verification Limitations: The information barrier and forbidden-pattern scan reduce but do not formally eliminate residual reward-hacking modes.Potential failures include overly loose semantic checks and state proxies correlated with task success.

7 Conclusion · Appendix Contents

CUA-GYM is an automated RLVR pipeline for computer-use agents that co-generates task instructions, environment states, and reward functions. It combines separated Generator and Discriminator agents, iterative endpoint consistency checks, rollout-based dataset filtering, and CUA-GYM-HUB environment synthesis.

  • 7 Conclusion: CUA-GYM co-generates task instructions, environment states, and reward functions for RLVR training of computer-use agents.
  • 7 Conclusion: The pipeline couples a Generator and a Discriminator under a process-level information barrier.
  • 7 Conclusion: An iterative loop enforces consistency between generated tuples at both endpoints.
  • 7 Conclusion: A dataset-level filter catches failures that become visible only under rollout.
  • 7 Conclusion: CUA-GYM-HUB broadens the environments accessible to the automated synthesis pipeline.
  • 7 Conclusion: The paper presents CUA-GYM-HUB as a synthesized suite of environments supporting broader computer-use-agent training coverage.

A Data Synthesis Pipeline Details · A.1 Pipeline Pseudocode

The CUA-GYM pipeline co-generates verified tuples through isolated Generator–Discriminator execution coordinated by an Orchestrator. Iterative agreement checks, static reward-code screening, and a dataset-level filter determine which tuples survive.

  • A.1 Pipeline Pseudocode: The Orchestrator provisions paired virtual machines and dispatches the Generator and Discriminator behind an information barrier.The Generator and Discriminator operate with separate access constraints during co-generation.
  • A.1 Pipeline Pseudocode: The Discriminator reads only (t, c, Sdom) and writes reward.py under the pipeline’s information barrier.This separates reward-function construction from the Generator’s environment-state outputs.
  • A.1 Pipeline Pseudocode: The loop continues until all five agreement conditions hold or the maximum-round budget is exhausted.Only tuples surviving this iterative process proceed to the dataset-level filter.
  • A.1 Pipeline Pseudocode: A static scan checks reward.py against a forbidden-pattern list before surviving tuples enter the dataset-level filter.The filter is applied after the per-tuple co-generation loop.
  • A.1 Pipeline Pseudocode: The pipeline takes a task instruction t, context c, domain skill Sdom, and maximum round budget K, producing a verified tuple or ⊥.The tuple contains t, s_init, s_gold, and r.
  • A.1 Pipeline Pseudocode: In each round, the Generator writes initial_setup.py and golden_patch.py, which execute on separate initial and golden virtual machines.The two scripts construct the initial and golden environment states.

A.2 Task Generation … A.6 Output Bundle and OSWorld Compatibility

The pipeline generates diverse, nonredundant computer-use tasks, validates them through adversarial execution and multi-stage filtering, and packages domain-specific guidance for reproducible environments and OSWorld-compatible training.

  • A.2 Task Generation: Feature taxonomies organize application capabilities into trees grounded in documentation and usage research, constrained to depth ≤4 and branching factor ≤12.Leaves represent atomic UI capabilities, enabling tractable sampling and tightly scoped instructions.
  • A.2.2 Scenario Matrix and Sampling: Tasks are sampled across platform, domain, difficulty, scenario, and application scope, with inverse-frequency weighting and per-application minima protecting under-covered cells.Difficulty is operationalized as easy ≤3 atomic actions, medium 3–10 steps, or hard long-horizon planning.
  • A.2.3 Three-Pass Generation: Breadth, Gap-fill, Edge Cases: Three generation passes prioritize breadth, gap filling, and edge cases, while anti-repetition checks reject semantic and token-overlap duplicates and limit templates to 3 instantiations per application.Pass 1 guarantees at least 25 tasks per taxonomy leaf; later passes upweight cross-application, hard, and underrepresented cases.
  • A.2.5 Coverage Guarantees and Quantitative Targets: Coverage constraints cap any domain at 21%, require at least 40% hard and 35% cross-application tasks, and assign each application at least 1,000 verified tuples while capping it at 15%.Sampling terminates only after all matrix cells exceed their minima and the verified-tuple budget is exceeded.
  • A.3 Adversarial Generator-Discriminator Loop: Tuple acceptance requires executable initial and golden states, rewards of r(Vgold) = 1.0 and r(Vinit) = 0.0, and a reward implementation free of forbidden patterns.Process isolation gives Generator and Discriminator disjoint working directories, with communication mediated solely by the Orchestrator.
  • A.3.3 Forbidden Pattern List for Anti-Hacking Static Scan: Static scans block shortcut rewards such as constant success, existence-only scoring, and external subprocess calls, while failed rounds receive structured feedback for revision within K = 5 rounds.Unresolved tuples after the fifth round are rejected and logged for offline analysis.
  • A.5 Domain Skill Files: Versioned SKILL.md files provide both agents with application concepts, APIs, setup and reward patterns, and debugging-derived “bitter lessons” that are refined from rejection logs.The released bundle includes reproducibility guidance such as GUI startup requirements and application-specific implementation pitfalls.

A.6.1 File Layout per Verified Task … B.4.1 Plan Agent: Web Research and DESIGN.md

The paper specifies a drop-in verified-task file format, a broad mock-web synthesis methodology, deterministic state APIs, and a research-driven Plan Agent workflow. Together, these components support reproducible task execution, verification, and application-grounded environment construction.

  • A.6.1 File Layout per Verified Task: Each verified tuple is packaged under output/final/<task_id>/ with evaluator metadata, setup and golden-state scripts, reward.py, review, and task instructions.The bundle is consumed unchanged by the OSWorld evaluator, SFT teacher rollouts, and the RL trainer.
  • A.6.2 config.json Schema and Evaluator Contract: config.json follows the upstream OSWorld contract verbatim, specifying VM and snapshot settings, observation type, step budget, and reward entry points without schema extensions.This design makes CUA-GYM tuples usable as drop-in OSWorld benchmark items.
  • B.1 Environment Selection Methodology; B.1.1 Mapping O*NET SOC Major Groups to Web Application Categories; B.1.2 Anthropic Economic Index Weighting; B.1.3 Coverage Cutoff Criteria: Mock applications are selected from O*NET SOC workflow categories and allocated using normalized Anthropic Economic Index software-usage frequencies, with low-weight long-tail categories truncated.Categories are included only when their weight exceeds w_min, and applications are chosen in decreasing usage-frequency order until the category budget is exhausted.
  • B.2 Complete List of Synthesized Mocks; B.2.1 Application-by-Application Inventory; B.2.2 Real-World Reference Mapping and Fidelity Notes: 94 synthesized mocks retain reference applications’ surface layout, navigation, and primary features while remaining self-contained single-page applications independent of the real services.The mocks remove authentication, seed data locally, replace outbound calls with deterministic in-process mutations, and provide stubs for network-dependent capabilities.
  • B.2.3 Trademark, Trade Dress, and Asset Compliance: Released mocks are renamed with generic identifiers, exclude third-party logos and proprietary assets, use independently authored copy and illustrations, and do not redistribute reference screenshots.Mocks with restrictive terms of service receive additional human review and may be withheld or replaced by generic substitutes; a compliance manifest documents each release decision.
  • B.3 Unified State API Specification; B.3.1 Endpoint Contracts: Every mock provides a four-endpoint, same-origin state API for injection, inspection, reset, and file upload, with POST /post supporting set, set_current, merge, and reset actions.The /go endpoint exposes initial_state, current_state, and state_diff, while /state returns raw current state and /upload handles session-scoped attachments.
  • B.3.2 Session Isolation via sid; B.3.3 State Diff Computation Rules; B.3.4 Integration with reward.py: Rollouts use fresh UUIDv4 session identifiers, isolating concurrent state and expiring inactive sessions after one hour; state_diff compares nested objects recursively and arrays as ordered units.Whitelisted volatile fields are excluded from reward-relevant differences, and reward.py evaluates partial-credit assertions over the resulting diff to produce a [0, 1] score.
  • B.4 Multi-Agent Synthesis Pipeline; B.4.1 Plan Agent: Web Research and DESIGN.md: The Plan Agent researches each target application through documentation, screenshots, and persona-based feature inventories before coding, then emits DESIGN.md and supporting workflow, data-model, and TODO artifacts.DESIGN.md specifies color, typography, spacing, and component-style tokens for the downstream synthesis process.

B.4.2 Dev Agent: Implementation Protocol … B.5.3 Salesforce Mock: Form-Heavy Workflow

The mock-generation workflow uses file-based implementation, browser-based verification, and iterative fixing before release. Representative mocks encode application-specific state schemas, cross-entity invariants, and value-level reward checks.

  • B.4.2 Dev Agent: Implementation Protocol: The Dev Agent implements each mock as a Vite + React single-page application using a fixed project layout and file-based coordination artifacts.The layout includes dedicated files for routing, global state, initialization and persistence, diff computation, and the state-API plugin.
  • B.4.3 Web Agent: Playwright Verification against UI Tree: The Web Agent uses headless Playwright to verify the UI tree, exercise every catalogued interaction, compare available screenshots, and report functional, visual, and code-level issues.It produces TEST.md and AUDIT.md reports with P0/P1/P2 severity levels.
  • B.4.4 Iterative Convergence Loop: Mocks iterate through Web Agent testing, Dev Agent fixes, and Web Agent retesting until no P0/P1 issues remain or the round budget is exhausted.Non-converged mocks are flagged for manual review and excluded from the released suite.
  • B.5.1 Slack Mock: State Schema and Feature Inventory: The Slack mock models channels with ordered, metadata-rich messages, a top-level current user, and a flat user list supporting user-switching tasks.Its verifiable interactions include channel creation and archival and message posting.
  • B.5.2 Jira Mock: Cross-Entity State Transitions: The Jira mock enforces consistency across projects, sprints, boards, issues, and status workflows, deriving board columns from issue status.Issue sprint membership must match the sprint’s project, and status transitions must follow the configured workflow graph.
  • B.5.3 Salesforce Mock: Form-Heavy Workflow: The Salesforce mock represents CRM workflows across Leads, Contacts, Accounts, and Opportunities, including lead-to-opportunity conversion through multi-step forms over ∼25 fields.Reward verification checks the final populated field set and field types rather than the interaction path.

C Training Details … C.2.2 SFT Hyperparameters

The training setup specifies a screenshot-based computer-use agent with XML tool calls, trajectory slicing for long rollouts, and fixed inference and prompt conventions. SFT initializes the policy from successful teacher trajectories using masked cross-entropy training with explicitly stated optimization and hardware settings.

  • C.1.1 Observation and Action Space: The agent observes raw desktop or browser screenshots and emits XML-wrapped tool calls containing one or more primitive actions.Each rollout step pairs an image observation with a single <tool_call> block, and multiple primitive actions may be compressed into one turn.
  • C.1.1 Observation and Action Space: The exposed computer_use interface supports pointer, keyboard, navigation, screenshot, waiting, termination, and evaluation-only clarification actions.Arguments include coordinates, text, keys, and durations supplied as parameters to the function block.
  • C.1.2 Trajectory Slicing Scheme: Trajectory slicing addresses rollouts exceeding the 144K-token context cap by emitting multiple samples every traj_slice_interval = 10 turn-pairs with collapsed prompt screenshots.Episodes can reach roughly 200K tokens under max_turns = 100 and max_response = 2048, while slices preserve full multimodal observations in later responses.
  • C.1.2 Trajectory Slicing Scheme: Each slice masks prompt and environment tokens, trains on eligible assistant messages, and replicates the parent trajectory’s full reward without splitting or discounting it.Assistant messages are excluded when cached log-probabilities are missing or when a turn contains more than MAX_TOOL_CALLS_PER_TURN = 10 tool calls.
  • C.1.3 Inference-Time Configuration: If collapsing still exceeds context, the system substitutes a no-gradient dummy slice; this occurs in < 1% of slices, while training and validation use a shared decoding stack.Training uses the rollout temperature, whereas validation uses temperature 0.6 and top-p 0.95; sampler failures receive up to retry_times = 5 retries.
  • C.1.4 System Prompt and Tool Definitions: A constant system prompt defines tool-call grammar, response shape, and termination semantics, while task-specific instructions arrive as the first user message.Normal UI steps require a brief Action followed by exactly one tool-call block with no suffix.
  • C.2.1 Teacher Rollout Curation: 3,578 trajectories initialize SFT after sampling 4 teacher rollouts per task at temperature 1.0 and retaining only terminal-reward r(s, τ) = 1 successes.The success-only filter discards approximately 50% of teacher rollouts, with no additional re-weighting or domain rebalancing.
  • C.2.2 SFT Hyperparameters: SFT uses masked assistant-token cross-entropy with AdamW, peak learning rate 7×10−6, cosine decay to 7×10−7, weight decay 0.01, batch size 512, 1 epoch, and 256K-token sequences.Training uses bf16 mixed precision with FP8 matmul-path weights on 512 GPUs, applying trajectory slicing identically to RL.

C.3 GSPO Training … C.4.3 Wall-Clock Time and Cost per Run

The paper specifies GSPO training choices, advantage normalization, and scalable disaggregated infrastructure for CUA reinforcement learning. Runs use large GPU clusters and parallel virtual machines, with approximately five-day wall-clock times and rollout-dominated costs.

  • C.3.1 Hyperparameters: Table 6 reports the hyperparameter values used for all reinforcement-learning runs, with defaults following GSPO except where noted.These settings cover the runs reported in §3.1 and §4.1.
  • C.3.1 Hyperparameters: A Chinese-token penalty with coefficient 0.5 is applied, while format, language, endless, and group-filtering penalties are disabled.The entropy bonus remains zero throughout training.
  • C.3.2 Reference Policy Update Schedule: With β = 0, training loads and evaluates no reference policy, freeing per-node GPU memory for larger rollout batches and full 24-instance operation.Early tests with β ∈{10−3, 10−2} showed neither training-stability gains nor task-success improvements at the reported data scales.
  • C.3.3 Advantage Normalization Details: Advantages use group-wise mean centering over G = 16 rollouts, without standard-deviation normalization because fixed-scale rewards make σg division noise-amplifying.Zero-variance groups retain mean-centered advantages, which evaluate to zero.
  • C.4.1 Hardware and Cluster Setup: RL runs use H200 SXM GPUs in scale-matched clusters of 192 GPUs for Qwen3.5-35B-A3B-scale training and 512 GPUs for Qwen3.5-397B-A17B-scale training.The clusters use disaggregated training and rollout subclusters.
  • C.4.2 VM Provisioning for Rollouts: 2,000 parallel OSWorld virtual-machine instances serve rollouts, with task snapshots restored before episodes and clean snapshots reset afterward.Average VM utilization exceeds 75% per training step, and rollout latency is not the training bottleneck.
  • C.4.3 Wall-Clock Time and Cost per Run: Approximately 5 days and ∼23,040 GPU-hours are required for a 1000-step Qwen3.5-35B-A3B run, versus approximately 5 days and ∼61,440 GPU-hours for Qwen3.5-397B-A17B.Rollout dominates cost, while the trainer is comparatively idle waiting for episode completions.

C.5 Reproducibility Checklist

The appendix reproduces the full NeurIPS Paper Checklist, including Yes/No/NA responses and justifications cross-referenced to relevant sections.

  • The full NeurIPS Paper Checklist is reproduced after the appendix.
  • Responses use Yes, No, or NA, with justifications provided.
  • Cross-references point to the synthesis pipeline, hyperparameters, infrastructure, and limitations sections.These correspond to §B.4, §C.3.1, §C.4.1, and §6, respectively.

D Extended Experiments … E.1.3 Cross-App – Slack + Impress + PDF: Prioritization Deck

The extended experiments analyze emergent multi-action behavior and illustrate CUA-Gym tasks spanning desktop, code-editor, and cross-application workflows with decomposed, verifiable rewards. The cross-application example requires extracting Slack poll results, authoring a five-slide Impress deck, and exporting a five-page PDF.

  • D.1 Emergent Multi-Action Behavior: At training step 30, the Qwen3.5-397B-A17B-scale RL checkpoint averaged 1.41 tool calls per model step across 1,105 OSWorld-Verified rollouts.The 22,361 model steps contained 69.6% single-call, 26.7% two-call, 2.4% three-call, 0.8% four-call, and 0.5% five-or-more-call steps.
  • D.1.1 Top-K Batched Action Patterns: Deterministic input chains and continuous-scroll preludes dominated batched behavior, accounting for ∼75% of all batched steps.The most frequent patterns were type →key with 3,942 steps and mouse_move →scroll with 1,052 steps.
  • D.1.2 Avoided Non-Deterministic Action Categories: Pointer clicks were usually emitted alone, with right_click, double_click, and triple_click solo rates of 94–98% and left_click at 90.6%.The passage relates this pattern to post-action UI states that are difficult to predict without a fresh observation.
  • E.1 End-to-End Task Walk-throughs: The qualitative examples present four released tuples across desktop, web, and cross-application domains, including task instructions, reward decompositions, and distinctive reward.py components.Full setup and reward scripts are released with the data.
  • E.1.1 Desktop – LibreOffice Calc: Padded ID Formula: The Calc task rewards a TEXT formula in B2, propagation through B3:B6, and correct references to corresponding A-column cells.Its three components are weighted C1 (0.40), C2 (0.30), and C3 (0.30), totaling 1.0.
  • E.1.2 Desktop – VS Code: Custom File Associations: The VS Code task verifies a non-empty files.associations object mapping *.config to json and *.tmpl to html through file-based settings inspection.The reward weights these checks C1 (0.35), C2 (0.35), and C3 (0.30), while JSONC comments are stripped before parsing.
  • E.1.3 Cross-App – Slack + Impress + PDF: Prioritization Deck: The cross-application task combines Slack poll extraction, Impress authoring, and PDF export into a five-slide prioritization deck covering 10 candidate features.The reward requires exactly 3 Q2 roadmap commitments from the top-5 and a PDF artifact with 5 pages.

E.1.4 Cross-App – PDF →Calc: Multi-Currency Expense Analysis … E.3 Reward Hacking Cases Caught by the Information Barrier

The paper combines compositional cross-application tasks, verified end-to-end agent trajectories, and information barriers that reject reward-hacking exploits before tuple acceptance. These examples emphasize complete workflows, selective edits, recovery from errors, and independently grounded reward checks.

  • E.1.4 Cross-App – PDF →Calc: Multi-Currency Expense Analysis: The PDF-to-Calc task requires extracting 50 rows, adding verification and discrepancy formulas, formatting six outliers, creating SUMIFS summaries, and adding a pie chart.Its reward has six independently checkable components spanning cell values, formatting, and derived sheet structure.
  • E.1.4 Cross-App – PDF →Calc: Multi-Currency Expense Analysis: A complete workflow earns 1.0, whereas omitting either the chart or discrepancy column yields 0.85.No single component is sufficient for full credit.
  • E.2 Full Rollout Trajectory: Shopify Vendor Consolidation: The 19-turn Shopify rollout identifies BasicWear and HomeGoods products, changes their vendor to UnifiedBrands, appends the required description line, and leaves other vendors untouched.The task is evaluated on a synthesized Shopify-admin mock in OSWorld-Verified and terminates with reward 1.0.
  • Step 1.; Step 2.; Step 3.; Step 4.; Step 5.; Step 6.; Step 7.; Step 8.: Steps 1–8 navigate to Products, edit Classic T-Shirt’s vendor and description, and save the first affected product.The trajectory initially emits one tool call per turn.
  • Step 15.; Step 16.; Step 17.; Step 18.; Step 19.: Steps 15–19 verify both edits: Classic T-Shirt and Ceramic Mug show UnifiedBrands with the marketing line, while Leather Wallet and Running Shoes remain unchanged.The rollout concludes successfully with reward 1.0.
  • E.3 Reward Hacking Cases Caught by the Information Barrier: The information barrier and forbidden-pattern scan reject reward-hacking examples at write-time before the synthesis loop accepts a tuple.The cases arose when the Discriminator could inspect the Generator’s setup scripts.
  • E.3 Reward Hacking Cases Caught by the Information Barrier: Hack A directly assigns chart_verified = True, so the reward always adds 0.5 without verifying the chart and is rejected by static scan.This exploit asserts a flag instead of computing the underlying condition.

E.4 Adversarial Loop Iteration Trace … F.1 Distributional Breakdowns

The adversarial loop converts a detected reward mismatch into a targeted repair and verified tuple acceptance, while the reward-function gallery demonstrates structural, image, and filesystem verification primitives. These examples complement cell-formula, JSON-key, and document-content checks across the dataset.

  • E.4 Adversarial Loop Iteration Trace: Round 1 failed because the Optimistic scenario duplicated the Pessimistic values, yielding a golden-state reward of 0.65.The Discriminator identified the failing component and generated structured feedback requesting distinct Optimistic inputCells.
  • E.4 Adversarial Loop Iteration Trace: The feedback specified Optimistic values of (650000, 280000, 200000, 140000, 75000) instead of the duplicated Pessimistic values.The requested correction targeted golden_patch.py and required distinct inputCells for Optimistic.
  • E.4 Adversarial Loop Iteration Trace: Round 2 passed after the Generator wrote distinct inputCells, and the Discriminator accepted the tuple after rescoring both endpoint states.The final agreement conditions recorded reward(golden) == 1.0 and reward(initial) == 0.0.
  • E.4 Adversarial Loop Iteration Trace: The corrected golden file scored 1.00 from scenario names, Optimistic values, and Pessimistic values, worth 0.30, 0.35, and 0.35 points.The Optimistic values were (650k, 280k, 200k, 140k, 75k), while the Pessimistic values were (350k, 140k, 100k, 60k, 30k).
  • E.5 Reward Function Gallery: These primitives complement cell-formula, JSON-key, and document-content checks, spanning reward functions in proportion to target-task distribution.The gallery’s three primitives are structural object introspection, page-property and image-anchor checks, and filesystem hierarchy assertions.
  • E.5 Reward Function Gallery: Chart introspection verifies chart existence and type, a 12-month data range, and chart and axis titles through openpyxl abstractions.The gallery assigns 0.30 + 0.20 points for chart existence and column/bar type, 0.20 for the range, and 0.15 + 0.15 for titles.
  • E.5 Reward Function Gallery: Image-property verification checks orientation, margins, and watermark presence while treating page size and order as preconditions.All four margins must be 1.5cm, approximately 0.591 inches, and at least one image must be anchored on the Print sheet.
  • E.5 Reward Function Gallery: Filesystem hierarchy verification requires six files in category-specific subfolders and absent from the Desktop root, preventing copy-without-delete credit.Each file receives credit only when both its target-folder presence and root absence are verified.

F.1.1 Tasks per Environment … G Prompts Catalog

CUA-GYM spans 32,112 verified tuples across 110 environments, with broad coverage but concentration in knowledge-work scenarios and harder multi-application tasks. Its catalog characterizes action vocabulary, cross-application workflows, instruction length, generation costs, and the prompt-driven synthesis and verification pipeline.

  • F.1.1 Tasks per Environment: 32,112 verified tuples span 110 environments, with five knowledge-work categories covering 84% of the corpus and no single scenario exceeding 21%.The remaining tasks span PDF, communication, project-management, and business-operations scenarios.
  • F.1.2 Domain × Difficulty Matrix: 44.6% of tasks are hard, 37.7% medium, and 17.7% easy; hardness reaches 80.1% in Cross-Desktop, 87.6% in Cross-Web, and 95.8% in Desktop × Web.The multi-application skew reflects additional state-tracking and context-switching demands.
  • F.1.3 Action Verb Frequency: Surface-edit imperatives such as open, create, set, add, insert, change, and remove dominate task instructions, followed by composition, extraction, and verification verbs.The first imperative verb is used to summarize the action vocabulary agents must learn to ground.
  • F.1.4 Cross-App Pair Co-occurrence: 12,311 cross-app tasks represent 38.3% of the corpus, with calc+writer+pdf and impress+writer+pdf document-production triplets accounting for most cross-app traffic.Application pairs are detected by keyword matches, and one instruction may contribute to multiple pairs.
  • F.1.5 Instruction Length Distribution: 54.1 words is the mean instruction length, versus a 41-word median; 90% fit under 108 words, while the longest reaches 390 words.Short instructions typically specify single-step actions, whereas 100+ word instructions encode multi-stage workflows.
  • F.2 Generation Cost; F.2.1 Per-Tuple Token Consumption: ∼10,000 input tokens and ∼5,000 output tokens are consumed on average per accepted tuple across Task-Gen, Generator, Discriminator, and Filter.Generator and Discriminator prompts dominate input, while reward scripts and environment patches dominate output.
  • F.2.2 VM-Hours per Verified Tuple: 45 minutes of wall-clock time and approximately $0.32 in amortized VM cost are required per verified tuple using two Aliyun ECS instances.The execution stage provisions initial_env and golden_env, runs setup and reward checks, and tears both environments down.
  • G Prompts Catalog: The prompt catalog specifies file-mediated orchestration, adversarial Generator–Discriminator verification, conservative dataset filtering, and Plan–Dev–Web mock-environment synthesis.The Discriminator tests golden reward 1.0 and initial reward 0.0, while the mock QA loop exercises interactions, visual diffs, and state-API endpoints.
Loading 2605.25624v2…