Source-linked AI summary
ProCUA-SFT Technical Report
Jaehun Jung, Ximing Lu, Brandon Cui, Muhammad Khalifa, Shaokun Zhang, Hao Zhang, Jin Xu, Amala Sanjay Deshmukh, Karan Sapra, Andrew Tao, Yejin Choi, Jan Kautz, Mingjie Liu, Yi Dong
TL;DR
Training computer-use agents is limited by the cost and scale of collecting diverse trajectories in fully instantiated desktops. ProCUA-SFT addresses this with automated synthetic data generation and achieves 45.0% on OSWorld after one epoch, outperforming the base model and AgentNet-trained models.
Problem
Computer-use agent progress is bottlenecked by the expense of collecting diverse trajectories in fully instantiated desktops, while existing open-source systems reach only 25.0–26.3% on OSWorld.
Method
ProCUA-SFT automatically synthesizes grounded tasks and verifies feasibility with binary preconditions, using one VLM for goal generation, verification, and trajectory execution.
Results
45.0% on OSWorld after one training epoch is +18.7 percentage points over the base model and +35 percentage points over AgentNet-trained models.
Takeaways & Limitations
ProCUA-SFT provides an open dataset of 3.1M samples from 93K synthetic trajectories that outperforms human demonstrations for computer-use SFT on OSWorld.
Takeaways & Limitations
The paper’s planned iteration depends on stronger open-weight VLMs, additional operating systems, and external reward models becoming available.
Abstract
from arXiv · showhide
Training computer-use agents (CUAs) -- models that interact with graphical desktops through screenshots and keyboard/mouse actions -- requires large-scale, diverse trajectory data collected in full desktop environments. The largest public resource, AgentNet (22.5K human trajectories), leads to negative transfer when used for supervised fine-tuning (SFT): continuing training UI-TARS 7B on AgentNet causes OSWorld success rate to fall from 26.3% to 8-10%. We present ProCUA-SFT, a dataset of 3.1M step-level SFT samples distilled from 93K synthetic trajectories across 2,484 application combinations. The dataset is produced by a fully automated pipeline that (i) synthesizes grounded tasks on live desktops seeded with real-world content -- 912 spreadsheets from SpreadsheetBench, approximately 10K permissively-licensed presentations from Zenodo10K, and multi-application OSWorld configs -- and (ii) verifies each task's feasibility through binary precondition checking before rollout. A single VLM (Kimi-K2.5) serves as goal generator, precondition judge, and trajectory executor, eliminating planner-actor capability gaps. Each trajectory is expanded into step-prefix samples that exactly reproduce the context layout seen at inference time. Fine-tuning UI-TARS 7B on ProCUA-SFT for one epoch yields 45.0% on OSWorld -- an 18.7 percentage-point improvement over the base model and over 35% above AgentNet-trained counterparts. A subset of ProCUA was incorporated into the training data for the Nemotron 3 Nano Omni model, contributing to its computer-use capabilities.
1. Introduction
Progress in computer-use agents is constrained by training data: AgentNet fine-tuning produces severe negative transfer, while ProCUA-SFT uses verified synthetic trajectories and real-world desktop content to improve OSWorld performance. Its pipeline relies on a single VLM and yields substantial gains after one epoch of SFT.
- Introduction: 26.3% and 25.0% are the OSWorld success rates of UI-TARS-7B and OpenCUA-7B, respectively, underscoring the field’s early progress.OSWorld benchmarks realistic desktop tasks, and CUA trajectories require fully instantiated desktops.
- Introduction: 22.5K human-annotated AgentNet trajectories cause UI-TARS 7B’s OSWorld success rate to fall from 26.3% to 8–10% after one epoch.The reported negative transfer is attributed to limited task diversity and missing complex cross-application workflows.
- Introduction: 3.1M step-level SFT samples are distilled from 93K synthetic trajectories spanning 2,484 application combinations in ProCUA-SFT.The dataset is designed to address the training-data bottleneck through broad synthetic coverage.
- Introduction: Kimi-K2.5 serves as ProCUA-SFT’s goal generator, precondition judge, and trajectory executor, with only goals passing all binary preconditions proceeding to rollout.Tasks are grounded by independently checking preconditions against desktop screenshots and OS-level configuration, with failed verdicts retried.
- Introduction: 45.0% on OSWorld is achieved by UI-TARS 7B after one SFT epoch on ProCUA-SFT, an 18.7 pp gain over the base model and more than 35 pp above AgentNet-trained models.The desktops are seeded with 912 SpreadsheetBench spreadsheets, approximately 10K Zenodo10K presentations, and multi-application OSWorld configurations.
2. Related Work
Prior GUI-agent research spans screenshot-based perception, diverse desktop and web benchmarks, and human- or synthetically generated training trajectories. ProCUA-SFT distinguishes itself through in-loop feasibility verification, realistic seeded desktop content, and scale.
- Vision-based GUI agents: Screenshot-based GUI agents replace structured DOM or accessibility inputs with visual perception, using approaches such as Pix2Act and CogAgent.Pix2Act uses demonstrations and tree-search self-play, while CogAgent employs dual-resolution visual encoding at 1120×1120.
- Benchmarks: 369 OSWorld tasks in real Linux, Windows, and macOS virtual machines form the primary evaluation target, alongside web, Windows, Android, and MiniWoB++ benchmarks.WebArena and VisualWebArena contain 812 and 910 tasks, respectively; Windows Agent Arena has 150+ tasks, and AndroidWorld has 116 tasks across 20 apps.
- Training data for GUI agents: Human-demonstration resources include AITW, Mind2Web, and AgentNet, with AgentNet providing 22.5K cross-platform trajectories and causing negative transfer during SFT.AgentNet is described as the largest desktop-targeted resource to date, but its SFT use causes negative transfer.
- Synthetic trajectory generation: Automatic synthesis methods reduce annotation costs through tutorial replay, exploration, website annotation, subtask composition, LLM alternatives, online RL, and failed-trajectory refinement.Examples include AgentTrek, Explorer, InSTA, AgentSynth, PC Agent, PC Agent-E, DigiRL, and AgentRefine.
- ProCUA-SFT: 3.1M samples from 93K trajectories across 2,484 application combinations make ProCUA-SFT the largest open-source desktop CUA training dataset.It verifies task feasibility with in-loop binary precondition checking and seeds desktops with externally sourced real spreadsheets and presentations rather than default or empty states.
3. ProCUA-SFT
ProCUA-SFT builds difficult, diverse computer-use trajectories by combining grounded task synthesis, real-world desktop content, and a scalable collection pipeline. A single VLM generates, verifies, and executes tasks, while context-matched step-prefix conversion preserves inference-time supervision.
- Unified policy: Kimi-K2.5 performs goal generation, judging, and execution, aligning proposed task achievability with the actor’s actual capabilities.Using one VLM reduces the planner–actor capability gap and lets the rollout self-signal success or failure through a reserved terminate call.
- Grounded task synthesis: Binary precondition checks verify that synthesized goals are achievable on the current desktop before rollout.The Judge independently evaluates each requirement and feeds failed verdicts into subsequent goal-generation attempts.
- Grounded task synthesis: Setup configuration is supplied to both generator and Judge, covering resources that may exist but are not visible in the initial screenshot.This closes the grounding loop over files, applications, browser tabs, and other state established by the configuration.
- Content seeding: Trajectories begin from OSWorld, SpreadsheetBench, or Zenodo10K configurations, injecting real spreadsheets, presentations, and multi-application desktop states.These external contents support challenging workflows such as cross-application reasoning and multi-file document tasks.
- Trajectory conversion: The SFT conversion reproduces rollout context windowing, retaining three recent screenshots while summarizing older steps textually.Matching the training layout to inference preserves the context structure seen by the supervised model.
- Trajectory conversion: Step-prefix expansion converts each long trajectory into multiple supervised examples instead of wasting supervision on a single sample.The pipeline thereby turns intermediate rollout prefixes into additional training instances.
4. Empirical Validation
ProCUA substantially improves OSWorld performance over AgentNet-trained UI-Tars, supported by longer trajectories, OSWorld-aligned application coverage, and a more keyboard-oriented action mix. An ablation further identifies application-combination coverage as the most effective diversity axis for CUA SFT.
- OSWorld performance: 44.99% OSWorld success follows one epoch of ProCUA training, improving original UI-Tars by 18.68%, while AgentNet training plateaus at ~8-10%.At a similar number of training steps, ProCUA recovers the original UI-Tars performance.
- Dataset differences: ProCUA trajectories average ~30 steps versus ~17 for AgentNet, with a substantially heavier tail and longer horizons.Figure 3 reports mean lengths of approximately 29.7 versus 18.6 steps.
- Dataset differences: Nearly half of labeled ProCUA trajectories target LibreOffice applications, while another 20% involve multi-application workflows aligned with OSWorld’s evaluation distribution.LibreOffice coverage comprises Impress 24%, Calc 17%, and Writer 6%; the comparison uses AgentNet’s 5k labeled Ubuntu trajectories.
- Dataset differences: ~63% of AgentNet action calls are clicks versus ~41% for ProCUA, which shifts more actions toward keyboard primitives and text entry.The paper characterizes keyboard shortcuts and direct text input as less brittle than pixel-accurate clicks for fine-grained navigation and form-filling.
- Diversification ablation: 30.9% OSWorld success from app-combination round-robin selection exceeds the base-model baseline of 26.3% and the non-diversified variant’s 27.3%.Action-set selection reaches 24.9%, while nested combined selection reaches 25.5%; application-combination coverage therefore dominates action-type balancing.
5. Analysis
The analysis models GUI-agent trajectories as screen-transition graphs and shows that ProCUA contains mostly short, linear, single-application workflows. Rare-app synthesis shifts training data toward more structurally complex, multi-application coordination patterns.
- Trajectory Complexity: Trajectories are represented as directed screen-transition graphs, with nodes denoting distinct application-visual states and edges denoting action-triggered transitions.Eight metrics capture screens, transitions, cycles, actions, applications, app switches, and linearity.
- Dataset Statistics: 34,854 trajectories span 61 application combinations; the median visits 5 screens, crosses 5 transitions, and executes 19 actions.The median linearity score is 0.86, and 42.3% of trajectories are perfectly linear.
- Dataset Statistics: 73.9% of trajectories use one application, while 18.5% use two, 6.6% use three, and 0.9% use four or more.Approximately 35.8% are acyclic, 56.7% contain 1–5 cycles, and 7.5% contain more than 5 cycles.
- Representative Patterns: Trajectory complexity depends on inter-application coordination rather than application count alone: four-application tasks can remain linear or become highly cyclic.Repeated cross-referencing between applications produces distinct graph topologies.
- Rare-App Synthesis: Rare-app synthesis targets 2,065 underrepresented trajectories and produces a 353-trajectory subset with higher structural complexity than full ProCUA.Mean applications rise from 2.0 to 3.3, app switches from 2.0 to 4.8, screens from 6.4 to 8.6, and actions from 26.6 to 35.2.
- Rare-App Synthesis: The rare-app subset reduces dominant single-app categories and adds multi-app combinations that each represent 0.6–2.0% of the subset but less than 0.1% of the full set.Examples include Chrome + LO Calc, Files + LO Writer, Desktop + VS Code, and Chrome + Files + Terminal.
6. Conclusion
ProCUA-SFT is an open dataset of 3.1M SFT samples from 93K fully synthetic computer-use trajectories. One epoch of UI-TARS 7B fine-tuned on it reaches 45.0% on OSWorld, outperforming human-demonstration SFT through grounded synthesis and precondition verification.
- Contribution: ProCUA-SFT contains 3.1M SFT samples derived from 93K fully synthetic computer-use trajectories.The dataset is open.
- Results: 45.0% OSWorld success is reached by UI-TARS 7B after one epoch, exceeding the base model by +18.7 pp and AgentNet by +35 pp.The AgentNet comparison uses 22.5K human trajectories.
- Design choices: Grounded task synthesis with in-loop precondition verification ensures generated goals are feasible in the current desktop state.The paper identifies this as one of three critical design choices.
- Future work: The authors plan to iterate on ProCUA-SFT as stronger open-weight VLMs, additional OS platforms, and external reward models become available.These are identified directions for future iteration.