SoftwareGym: Scaling Computer-Use Agent Training with Diverse Synthesized Software Environments

1 INTRODUCTION

• CUA training needs diverse, realistic, reliable trajectories, but existing approaches do not jointly meet diversity, functional-reliability, and scalability requirements.Native application integration limits coverage and risks application-specific overfitting; browser-hosted environments limit interaction depth and workflow complexity.Source: §1 · pp. 1–2
• SoftwareGym automates design, implementation, and evaluation, combining varied interface designs with source-level verification and checkpoint-based GUI testing and repair.Only applications completing both source-level and GUI verification enter the library.Source: §1 · p. 2
• OSWorld Pass@3 rises from 41.39% to 56.97% for a model trained on SoftwareGym trajectories without OSWorld applications, tasks, or trajectories.The model also achieves 40.0% Pass@3 on SoftwareGym-Bench.Source: §1 · p. 2
• SoftwareGym synthesizes 182 runnable applications across twelve categories with a mean GUI task pass rate of 91.5%, and provides a 120-task SoftwareGym-Bench.The benchmark spans the same twelve categories.Source: §1 · p. 2

Data Synthesis in Existing Software

• Existing native and mobile applications preserve authentic states and complex workflows, but coverage is constrained by application availability and integration cost, with repeated training potentially encouraging application-specific overfitting.SoftwareGym instead synthesizes new executable applications with controllable functions, interfaces, and interaction logic.Source: §2 · p. 3

Synthesis of Browser-Hosted Software Environments

• Browser-hosted environments simplify deployment and state control, but their shared browser, HTML, and DOM substrate limits native application structure and persistent workflows.SoftwareGym instead synthesizes standalone desktop applications.Source: §2 · p. 3

3 METHOD

• SoftwareGym's three stages specify application functions and interfaces, realize and repair applications using GUI failures, then convert verified functions into executable tasks.Applications that complete the verification loop enter the software library.Source: §3 · p. 3, Figure 2 · p. 4

3.1 SOFTWARE DESIGN

• Function design turns an application type and resources into a specification linking user actions to expected state updates, visible GUI responses, and acceptance checkpoints.It connects domain-specific functions into complete workflows, including cross-module state changes, failure handling, and recovery.Source: §3.1 · p. 3
• Style variants share functionality but differ in product position, information architecture, interaction patterns, and visual identity, with structural checks used to retain distinct executable blueprints.The pipeline regenerates the least distinct blueprint when minimum pairwise tree-edit distance falls below a predefined threshold.Source: §3.1 · pp. 3–4

3.2 SOFTWARE IMPLEMENTATION AND EVALUATION

• GUI evaluation independently tests running applications against fixed cases with explicit initial states and visible checkpoints, returning traces and failures to guide iterative repair.Evaluation stops when all cases pass or the version budget is exhausted; design-fidelity checks also guard against repair deviating from the intended interface.Source: §3.2 · pp. 4–5

3.3 TASK SYNTHESIS

• Task synthesis admits only capabilities backed by passed GUI evidence, composes them into difficulty- and state-controlled workflows, and validates each natural-language request against its executable specification.Easy, normal, and hard tasks use increasing functional composition and dependencies; tasks can start from a file or the default blank state.Source: §3.3 · pp. 5–6, Figure 3 · p. 5

4.1 EXPERIMENTAL SETUP

• Applications are synthesized with Codex CLI agents, then the Qwen3.5-9B model is trained on SoftwareGym-only trajectories and evaluated on OSWorld and SoftwareGym-Bench.The setup evaluates 244 OSWorld tasks and uses three independent rollouts per task.Source: §4.1 · pp. 6–7

4.2 MAIN RESULT ANALYSIS

• Pass@3 on OSWorld rises from 41.39% to 56.97% with SoftwareGym-only training and no OSWorld training data.The gain spans all eight evaluated domains.Source: §4.2 · pp. 7–8, Table 1 · p. 7
• Pass@3 on SoftwareGym-Bench rises by 14.2 points, from 25.8% to 40.0%, for the same 9B base model.The resulting model exceeds Qwen3.7-Plus and several substantially larger specialized CUAs.Source: Table 2 · p. 7, §4.2 · p. 8
• 182 runnable applications span twelve categories; GUI evaluations cover all 49 major modules and 78.8% of concrete functions across 121 applications in eight aligned categories.The figure summarizes application breadth and GUI-evidence coverage.Source: §4.2 · p. 8, Figure 4 · p. 8
• Mean GUI task pass rate increases from 54.9% at v1 to 81.1% at v2 and 91.5% at v3, with gains consistent across categories.The figure summarizes reliability gains across the synthesized library.Source: §4.2 · p. 8, Figure 6 · p. 8
• Across 20 sampled applications, synthesis averages 237.3 minutes, 14.8 million tokens, 1,718 model calls, and an estimated $70.76 per application.The figure decomposes resource use across design, implementation, and evaluation.Source: §4.2 · p. 8, Figure 5 · p. 8

4.3 ABLATION STUDIES

• Full Design raises macro-average normalized tree-edit-distance diversity from 4.9 to 83.6 and improves diversity across all categories.The comparison holds functional requirements constant.Source: §4.3 · p. 9, Figure 7 · p. 9
• GUI-guided repair reaches an 83.1% pass rate versus 48.8% for source-only repair; mixed-source training reaches 47.5% on SoftwareGym-Bench versus 40.0% for SoftwareGym-only training.The repair comparison starts both methods at 50.4%; the training-source comparison keeps the model and evaluation protocol fixed.Source: §4.3 · p. 9, Table 3 · p. 9

• 182 applications across twelve categories achieve a mean GUI task pass rate of 91.5%, and SoftwareGym-only training improves OSWorld Pass@3 from 41.39% to 56.97%.The model uses no OSWorld application data for training.Source: §5 · p. 9
• The trained model reaches 40.0% on SoftwareGym-Bench and outperforms several larger specialized agents.Source: §5 · p. 10
• SoftwareGym broadens the interfaces, interaction patterns, and workflows available for CUA training as a scalable complement to native software.Source: §5 · p. 10

A.1 BENCHMARK CONSTRUCTION

• Each task is built backward from a target outcome and key semantic steps, producing a reachable solution path and a well-defined terminal state.Distractors and protected objects require state-grounded decisions about what to modify, ignore, or preserve.Source: §A.1 · p. 14
• A reproducible initializer restores each task's initial state, and a programmatic evaluator checks the resulting state and artifacts against the target.This makes success deterministically verifiable and results comparable across models and runs.Source: §A.1 · p. 14

B DEFINITIONS AND COMPUTATION OF REPORTED MEASURES

• Layout-archetype distance is the Hamming distance across navigation, spatial organization, object browsing, and action placement; a distance of at least 2 requires differences on two dimensions.The constraint separates style variants before detailed blueprint generation and reduces convergence on the same interface organization.Source: B DEFINITIONS AND COMPUTATION OF… · p. 17
• Structural blueprint diversity uses normalized layout-tree edit distances, removing text and visual styling so the measure reflects component hierarchies.The pipeline uses a 0.60 diversity gate, regenerating the blueprint with the smallest average distance when the minimum pairwise distance is below the threshold.Source: B DEFINITIONS AND COMPUTATION OF… · p. 17
• Visual fidelity scores implementation of blueprint design tokens using palette, primary-color, corner-radius, font, and visual-reference components.The gate requires Fvis ≥ 0.80 and retention of the primary color; failures yield concrete token mismatches for repair.Source: B DEFINITIONS AND COMPUTATION OF… · p. 17
• Functional coverage counts taxonomy functions with passed, directly relevant GUI evaluation evidence, and reports an unweighted category macro-average as its primary measure.Level-1 modules count as covered when at least one child function has such evidence; related but insufficient evidence is excluded.Source: B DEFINITIONS AND COMPUTATION OF… · pp. 17–18

C ABLATION PROTOCOLS AND COMPLETE RESULTS

• The ablation section separately examines style design, GUI-feedback repair, and training-source choices, with selections and outputs frozen in versioned manifests.Appendix B defines the reported metrics, and Appendix E details trajectory filtering for the training-source ablation.Source: C ABLATION PROTOCOLS AND COMPLETE… · p. 19

C.1 STYLE-DESIGN ABLATION

• The comparison covers twelve software categories, with ten blueprints per category and method and identical functional requirements and icon inventories.This yields 120 blueprints per method, 240 total.Source: §C.1 · p. 19
• Full Design varies information architecture, layout, visual system, interaction model, and product identity, rejecting structurally redundant designs; Simple Design only asks for ten different layouts.Both methods use the same artifact schema and final validators.Source: §C.1 · p. 19
• 83.6 versus 4.9: Full Design raises the macro-average structural-diversity score over Simple Design.Full Design scores higher in all twelve categories; none of the Simple Design category sets passes the production diversity gate.Source: Table 4 · p. 19, §C.1 · p. 19

C.2 EVALUATION-GUIDED IMPLEMENTATION-REPAIR ABLATION

• The paired comparison contains three applications from each of twelve categories, for 36 applications with complete artifacts through both repair rounds.Scores average GUI task pass rates across applications, weighting the twelve categories equally.Source: §C.2 · p. 19
• Both branches start from the same v1 source and use two repair rounds with GUI evaluation after each; only the GUI-guided branch passes each report to the Coder.The source-only branch records but withholds the report, relying on source-code evidence for its next repair.Source: §C.2 · p. 19
• 83.1% versus 48.8% at v3: GUI-guided repair leads source-only repair by 34.3 points across 36 applications.From the shared 50.4% starting score, v2 scores are 70.5% versus 50.5%; GUI-guided repair wins all 36 applications (95% CI: 28.2–41.0).Source: §C.2 · p. 19, Table 5 · p. 20

C.3 TRAINING-SOFTWARE-SOURCE ABLATION

• Before length filtering, SoftwareGym-only and OSWorld-only datasets are each matched to 3,715 trajectories and 55,263 serialized examples, with identical split sizes and outcome and process thresholds.Post-threshold selection is source-specific.Source: §C.3 · p. 20
• The combined model concatenates the complete frozen datasets, giving each source equal contribution while preserving source-prefixed identifiers and original splits.All three Qwen3.5-9B models use one epoch of full-parameter language-model SFT, with the visual encoder and merger frozen.Source: §C.3 · p. 20
• 56.97% Pass@3 on OSWorld without OSWorld training, versus 61.48% for OSWorld-only; SoftwareGym-Bench results are 40.0% SoftwareGym-only, 33.3% OSWorld-only, and 47.5% combined.The combined model retains 58.61% on OSWorld.Source: Table 6 · p. 20, §C.3 · p. 20, Table 7 · p. 21

D COMPLETE PER-RUN RESULTS FOR THE MAIN EVALUATION

• Per-run averages describe individual-run evaluator performance and are distinct from task-level Pass@3.The appendix reports three runs, their arithmetic mean (Avg@3), and task-level Pass@3 for each domain.Source: D COMPLETE PER-RUN RESULTS FOR THE… · p. 22

D.1 OSWORLD

• 244 tasks across eight application domains are covered in each OSWorld run.Table 8 reports Run 1–3 scores, Avg@3, and Pass@3 on OSWorld.Source: §D.1 · p. 22, Table 8 · p. 22

D.2 SOFTWAREGYM-BENCH

• Each SoftwareGym-Bench run covers 120 tasks, with ten tasks from each of twelve categories.Table 9 reports Run 1–3 scores, Avg@3, and Pass@3 on SoftwareGym-Bench.Source: §D.2 · p. 23, Table 9 · p. 23

E TRAJECTORY SCORING AND FILTERING

• Trajectory selection evaluates both whether an agent reaches the requested final state and whether its interaction process is reliable.After a structural audit, two independent judges score candidates before qualified trajectories are converted, selected, and group-split.Source: E TRAJECTORY SCORING AND FILTERING · p. 26

E.1 TWO-STAGE TRAJECTORY SCORING

• Outcome scoring uses up to five final assistant messages, process scoring uses all messages and up to ten screenshots, and both scores must reach 0.8.Outcome judgments treat paired screenshots as primary evidence and check final evidence against each material request requirement.Source: §E.1 · p. 26

E.2 MESSAGE CONVERSION, SELECTION, AND SPLITTING

• OSWorld retains the highest-scoring qualified attempt per task, while all score-qualified SoftwareGym attempts proceed to conversion.After strict format and consistency validation, SoftwareGym caps frozen semantic groups at two trajectories; OSWorld balances domains and difficulty, with approximately 90/5/5 group-constrained splits.Source: §E.2 · p. 27