PaperlayerView exact revision on arXiv

Source-linked AI summary

Gym-Anything: Turn any Software into an Agent Environment

Pranjal Aggarwal, Graham Neubig, Sean Welleck

arXiv:2604.06126v1cs.LGcs.AI

TL;DR

Gym-Anything addresses the costly, limited construction of realistic environments for computer-use agents by using a multi-agent creation-and-auditing pipeline. Applied to GDP-grounded software selection, it produces CUA-World, whose training supports smaller-agent distillation and whose test-time auditing improves long-horizon performance.

  • Problem

    Existing computer-use research focuses on short-horizon tasks over few applications, while realistic environments require costly expert setup and verification.

  • Method

    Gym-Anything converts software into interactive environments through multi-agent generation, evidence-based auditing, correction, and GDP-grounded selection of 200 applications.

  • Results

    CUA-World contains over 10,000 tasks across 200 applications, while test-time auditing improves Gemini-3-Flash on CUA-World-Long from 11.5% to 14.0%.

  • Takeaways & Limitations

    CUA-World provides broad realistic supervision, and CUA-World-Long remains challenging even for frontier models, indicating realistic computer-use is far from solved.

  • Takeaways & Limitations

    GDP-grounded selection provides a ranking rather than precise dollar-level attribution, and many professionally used software applications remain excluded.

Abstract

from arXiv · show

Computer-use agents hold the promise of assisting in a wide range of digital economic activities. However, current research has largely focused on short-horizon tasks over a limited set of software with limited economic value, such as basic e-commerce and OS-configuration tasks. A key reason is that creating environments for complex software requires significant time and human effort, and therefore does not scale. To address this, we introduce Gym-Anything, a framework for converting any software into an interactive computer-use environment. We frame environment creation itself as a multi-agent task: a coding agent writes setup scripts, downloads real-world data, and configures the software, while producing evidence of correct setup. An independent audit agent then verifies evidence for the environment setup against a quality checklist. Using a taxonomy of economically valuable occupations grounded in U.S. GDP data, we apply this pipeline to 200 software applications with broad occupational coverage. The result is CUA-World, a collection of over 10K long-horizon tasks spanning domains from medical science and astronomy to engineering and enterprise systems, each configured with realistic data along with train and test splits. CUA-World also includes CUA-World-Long, a challenging long-horizon benchmark with tasks often requiring over 500 steps, far exceeding existing benchmarks. Distilling successful trajectories from the training split into a 2B vision-language model outperforms models 2$\times$ its size. We also apply the same auditing principle at test time: a separate VLM reviews completed trajectories and provides feedback on what remains, improving Gemini-3-Flash on CUA-World-Long from 11.5% to 14.0%. We release all code, infrastructure, and benchmark data to facilitate future research in realistic computer-use agents.

1 Introduction

Gym-Anything addresses the difficulty of building realistic, large-scale computer-use environments by treating environment creation as an audited multi-agent task. It produces CUA-World and CUA-World-Long, enabling broad training and challenging evaluation across economically relevant software.

  • Motivation: Current computer-use benchmarks mainly cover short-horizon tasks in few applications, whereas professional workflows require hundreds of steps across heterogeneous software and domain-specific data.This limits how faithfully existing evaluations reflect real economic work and how much diverse long-horizon training signal they provide.
  • Framework: Environment creation is itself framed as a coding and computer-use task, with creation agents configuring software and independent auditors checking evidence of correct setup.The loop uses screenshots, logs, and data outputs because environment state is more reliable than an agent’s claims.
  • Benchmark: Over 10,000 tasks across 200 software applications form CUA-World, spanning medical science, astronomy, engineering, finance, enterprise systems, and education with train and test splits.The software selection is grounded in economically important occupations and GDP data.
  • Long-horizon evaluation: CUA-World-Long contains realistic tasks often requiring hundreds of steps, and even the strongest frontier model achieves only a 27.5% pass rate.Test-Time Auditing improves Gemini-3-Flash from 11.5% to 14.0%, but the benchmark remains largely unsolved.
  • Contributions: The paper contributes Gym-Anything, CUA-World, training and test-time scaling results, and releases its code, infrastructure, and benchmark data.The framework supports distillation into a 2B vision-language model that outperforms models 2× its size.

2 Methodology

The methodology grounds software selection in U.S. economic data and represents tasks as interactive environments with explicit initial states, instructions, and verification. A unified library then supports realistic applications across operating systems and compute backends.

  • Problem setup: A task consists of an environment with initial state s0, a natural-language instruction p, and a verification function V that scores the agent trajectory.Episodes reset to s0, expose observations, accept mouse and keyboard actions, and run for up to T steps.
  • GDP-grounded selection: Software selection estimates GDP per occupation, discovers occupation-specific software, attributes GDP to software, filters sandboxable candidates, and selects 200 applications.The selection prioritizes software driving more economic activity while balancing economic importance and coverage.
  • GDP-grounded selection: GDP attribution uses computer-use, category-share, and software-share factors to decompose occupational GDP into software-level estimates.The category and software shares are estimated by an LLM with web-search access.
  • Sandboxability: Sandboxable software must satisfy constraints including self-hostability, free use without license restrictions, and a graphical user interface.These constraints exclude some economically important applications that require paid licenses, organizational credentials, or specialized hardware.
  • Environment representation: The Gym-Anything library replaces manual snapshots with a unified, scriptable specification that can express complex software configured with realistic data across operating systems.The library manages execution across Linux, Windows, Android, and multiple compute backends.

3 Scaling Computer-Use Agent Software Applications

Gym-Anything scales software environment construction through iterative creation, independent auditing, correction, and shared memory. Applying the pipeline yields 200 realistic environments across three operating systems and diverse application types.

  • Motivation: Naive environment creation often stops early, uses placeholder data, leaves the wrong starting screen, or claims completion without verification.These failures motivate a multi-agent framework that iteratively creates, audits, and improves environments.
  • Creation agent: A creation agent researches configuration, downloads real-world data, writes setup scripts, and verifies the software by running and interacting with it.The fixed library interface turns environment creation into a coding and computer-use task.
  • Audit agent: An independent audit agent checks screenshots, logs, configuration files, and scripts against quality checklists, running the environment when necessary.Separating creation from auditing reduces self-confirmation bias and makes quality judgments more interpretable.
  • Shared memory: Shared memory stores software-specific and general learnings about failed attempts and fixes so later environment construction can improve.The memory is initialized with workflow guidance and updated after each environment.
  • Output: The pipeline constructs environments for 200 software applications across Linux, Windows, and Android, including desktop applications and multi-service enterprise systems with realistic data.The pipeline is automated and extensible to additional software within the stated compute budget.

4 Scaling Tasks

Task scaling separates expensive seed-task design from cheaper amplification, then filters generated tasks by checking their starting states. Checklist-based VLM verification uses privileged information to award granular partial credit while enforcing workflow integrity.

  • Task-generation challenge: Task generation is a separate scaling challenge because setup scripts, data formats, instructions, and starting states can be inconsistent or infeasible.Using agentic models to author and validate every task would be prohibitively expensive.
  • Propose-and-amplify: A proposer agent creates a small set of realistic, difficult seed tasks, and a non-agentic language model amplifies them into additional tasks using in-context examples.Sequential generation with prior instructions as context helps enforce diversity.
  • Task validation: An automated VLM filter launches each generated task and removes tasks whose observed starting state does not match the instruction.This catches errors without requiring interactive execution during non-agentic generation.
  • Checklist verification: The verifier extracts privileged ground truth from setup scripts and uses it with the instruction to create weighted checklist items for each task.The resulting verification function can award partial credit on complex multi-step trajectories without manual annotation.
  • Integrity verification: An integrity checklist assigns zero score if the agent bypasses the intended software workflow, directly edits files, or exploits environment artifacts.This ensures that task success reflects interaction through the designated application interface.

5 CUA-World

CUA-World applies Gym-Anything to create a large, diverse benchmark with verified tasks, train/test splits, and a dedicated long-horizon subset. Its construction combines automated task generation, contamination filtering, and trajectory-guided difficulty increases.

  • 12,103 tasks and environments span 200 software applications, with checklist-based verification and separate Train and Test splits.
  • CUA-World simultaneously supports 200+ software varieties, 10K+ tasks, long-horizon evaluation, all 22 major occupation groups, automated environment creation, and a training split.
  • Contamination filtering flags task pairs with similarity scores of 4 or higher and removes connected contaminated groups between splits.
  • CUA-World-Long contains 200 tasks, one per software application, generated through trajectory-guided analysis to make them harder than existing tasks.
  • All 200 CUA-World-Long tasks were manually verified for correct setup and meaningfulness against eight quality criteria, with failed tasks iteratively refined.

6 Experimental Setup

CUA-World serves both as training data for distilling smaller models and as an evaluation benchmark for computer-use agents. Experiments use successful teacher trajectories, ablations of distillation choices, and checklist-based evaluation across standard and long-horizon test sets.

  • CUA-World is used for two roles: distilling smaller models from training trajectories and evaluating computer-use agents.
  • For each training task, Kimi-K 2.5 generates k = 4 trajectories until one is correct, yielding roughly 2000 successful rollouts across 10,000 tasks.
  • Distillation ablations examine teacher selection and the number of steps and samples per trajectory before training on all trajectories.
  • Current agents often stop after a few dozen steps on extremely long tasks, making mistakes or prematurely claiming completion.
  • An audit agent reviews complete screenshot trajectories after termination signals and provides feedback when the task remains incomplete.
  • Evaluation reports Average Score from 0-100 for partial credit and Pass Rate for fully completed tasks on CUA-World-Test and CUA-World-Long.

7 Results and Analysis

Results show that distillation improves small-model performance, while realistic long-horizon tasks remain difficult and expose failures related to compute budgets, software generalization, visual complexity, and workflow integrity.

  • Main results: 4.4% pass rate after distillation exceeds the 3.9% pass rate of Qwen3-VL-4B, showing that CUA-World-Train improves the 2B model.Qwen3-VL-2B improves from 1.6% to 4.4% pass rate.
  • Long-horizon performance: 27.5% pass rate is achieved by GPT-5.4 on CUA-World-Long after removing the cost cap and increasing the step limit to 2,000.Under the original constraints, frontier-model performance is much lower and incurs fewer steps.
  • Scaling: Performance improves with more software and tasks, following a roughly log-linear trend of about 3.5 points per doubling of training data.The trend holds when scaling either the number of applications or the fraction of tasks across all 200 applications.
  • Generalization: Distillation improves both seen and unseen-software performance, but OOD gains recover only 22-27% of all-software improvement versus 65-87% on IID software.At 25% of software, IID scores rise from 16.7 to 24.2, while OOD scores rise from 12.3 to 14.1.
  • Long-horizon performance: 14.0% pass rate with test-time auditing exceeds Gemini-3-Flash’s 11.5% under the same 2,000-step cap.Performance rises sharply only after budgets exceed roughly 100 steps, reaching 11.5% at about 1,300 average steps before auditing.
  • Software categories: High visual complexity remains a bottleneck for small models: Qwen3-VL-2B falls from 3.2% on low-complexity software to 0.0% on high-complexity software.Distillation improves absolute performance at both levels but does not remove the low-to-high complexity decline.
  • Verification: 93.3% task-level agreement with humans is achieved by the checklist-based VLM verifier, while integrity checks flag about 1.5% of high-scoring runs.The verifier comparison covers 60 sampled trajectories, and the integrity analysis covers about 3,000 trajectories.
  • Task generation: 88.9% setup success with seed examples exceeds 55.2% without seeds, supporting the propose-and-amplify strategy for task generation.The comparison spans generated tasks across 10 software applications.

8 Related Work

Prior computer-use benchmarks are either static and action-matching or interactive but narrow in software coverage. Gym-Anything extends automated environment and task generation to real GUI software requiring installation, configuration, and realistic data, while using checklist-based evaluation to address limitations of existing verifiers.

  • Computer-use benchmarks: Static benchmarks evaluate action matching, whereas interactive benchmarks execute tasks but cover only narrow slices of the software landscape.Existing interactive coverage includes limited web, desktop, and application-specific environments.
  • Environment generation: Previous automated generation methods create tasks or trajectories within existing environments but do not create new real GUI environments.Earlier environment-generation work targets text planning, embodied AI, APIs, code editing, and simulations rather than installed software with realistic data.
  • Evaluation: Programmatic verifiers are reliable but labor-intensive and binary, while prior VLM evaluators lack ground-truth answers and cannot detect workflow shortcuts.Gym-Anything uses checklist-based VLM verification to evaluate subtasks and integrity.
  • Scope: The paper provides additional related-work coverage on training methods, economic grounding, and per-benchmark comparisons.These comparisons are provided in the appendix.

9 Conclusion

Gym-Anything converts arbitrary software into interactive environments through a multi-agent generation, auditing, and correction loop, enabling CUA-World’s broad task collection. Distillation helps smaller agents and test-time auditing improves long-horizon performance, but realistic computer-use remains unsolved.

  • Framework: Gym-Anything reduces environment creation to setup scripts and configuration files within a multi-agent generation, auditing, and correction loop.The framework targets realistic environments at scale.
  • Benchmark: CUA-World contains over 10K tasks across 200 GDP-grounded software applications, with diverse domains, operating systems, verification, and long-horizon splits.The collection includes train, test, and CUA-World-Long splits.
  • Implications: CUA-World provides useful supervision for smaller agents through distillation, and test-time auditing improves performance on especially long-horizon tasks.These results establish training- and test-time uses for the benchmark.
  • Open challenge: CUA-World-Long remains challenging even for frontier models, indicating that realistic computer-use remains far from solved.The conclusion frames stronger models, stronger verifiers, and broader software coverage as directions for future work.

11 Limitations

The benchmark’s software selection is an economic ranking rather than precise dollar attribution, excludes much professionally used software, and does not guarantee every task is solvable.

  • GDP-grounded software selection ranks economic importance but does not provide precise dollar-level attribution.
  • Many professionally used applications remain excluded, and performance on free alternatives may not predict performance on their commercial counterparts.
  • Manual checks confirmed environment launches and task starting states, but the authors did not solve every task end-to-end and cannot guarantee universal solvability.
  • VLM checklist verifiers support evaluation, but they remain imperfect despite high agreement with human annotations.

12 Ethics Statement

The paper frames its contribution as infrastructure and benchmarking for computer-use agents, while acknowledging deployment risks and limiting released capabilities.

  • The work introduces environment-creation and test-time-auditing methods rather than a model exceeding existing frontier capabilities.
  • All software is freely available, and datasets come from public sources or synthetic generation.
  • The paper acknowledges that autonomous deployment of computer-use agents may pose risks.
  • The GDP-based selection pipeline estimates occupational software importance through wage bills, labor compensation, and national GDP scaling.
  • Software categories and products are generated, deduplicated, validated, and checked for real-world existence using language models and search grounding.
  • The resulting product rankings are estimates from the pipeline and should not be treated as authoritative source rankings.

A.6 Phase 6: Practical Access-Barrier Evaluation

The access-barrier phase filters software for sandboxability, selects economically important and diverse applications, and supports modular execution across platforms and backends.

  • Access-barrier evaluation: Free GUI access is insufficient when software also requires accounts, organizational credentials, or specialized hardware.
  • Access-barrier evaluation: 8,013 products (48%) are trainable, while 8,591 (52%) are not because of account, organizational, or hardware requirements.
  • Access-barrier evaluation: Selectable products must run on supported operating systems, avoid paid-only or CLI-only access, be sandbox-ready or self-hostable, and pass access-barrier checks.
  • Tiered selection: Non-selectable products are replaced by the closest selectable alternative in the same category, while retaining the original economic slot.
  • Tiered selection: The pipeline selects approximately 500 products across all 22 SOC major groups and builds environments for 200, including 53 retained selections affected by a software-selection bug.
  • Execution framework: The framework uses a shared environment specification, modular runners, staged checkpoints, and decoupled programmatic, image-match, or hybrid verifiers.
  • Execution framework: Execution artifacts support both evaluation and training-data collection for trajectory distillation.

C.4 Integrity Check Analysis

Integrity checks identify workflow shortcuts alongside task completion, but the analysis reveals both genuine violations and false positives caused by incomplete execution or tool substitution.

  • Integrity outcomes: Across approximately 3,000 trajectories, high-scoring runs produced 21 integrity flags: 15 true positives and 6 false positives.
  • Integrity outcomes: Most flags did not change pass rates because task-completion scoring had already failed those trajectories; three perfect-score flags were false positives.
  • Violation examples: Integrity analysis examined fabricated data, outputs not derived from tool results, and hardcoded exclusions despite task instructions requiring genuine workflows.
  • False positives: Incomplete execution is not itself a workflow bypass, as illustrated by an Oracle Database run incorrectly flagged for lacking a reconciliation report.
  • Tool-use failures: Agents unable to operate specialized GUIs may switch to terminal commands and produce correct output through the wrong tool.
  • Environment-audit examples: Environment audits found authentic infrastructure and generally feasible tasks, but critical screenshot-labeling errors undermined start-state evidence for one Odoo task.
  • Environment-audit examples: Other Odoo evidence showed correctly completed workflows, including a lead form and an opportunity marked WON with probability 100%.

E.3 Example Audit: Wireshark Environment (Mixed Results)

The Wireshark audit passed with moderate issues: authentic task data and mostly acceptable checks coexisted with leaked answers, over-prescriptive instructions, misleading evidence, and low task difficulty.

  • Overall assessment: Overall verdict: PASS with moderate issues, despite mostly GOOD or ACCEPTABLE ratings for several tasks.The audit found issues in task descriptions, verifiers, start-state evidence, data quality, and evidence documentation.
  • Task descriptions: The DNS task leaked its ground-truth answer “19,” allowing agents to pass without analysis.The example value in the task description matched the actual number of DNS queries.
  • Task descriptions: The protocol-hierarchy task prescribed the exact menu path, button, and copy method, reducing it to a step-by-step walkthrough.The audit rated the task over-detailed because little remained for the agent to determine independently.
  • Evidence verification: The Task 3 completion screenshot showed the Ethernet tab rather than the requested IPv4 tab, so the claimed answer was not visibly verified.The start state was correct, but the evidence for task completion was misleading.
  • Data quality: The five tasks used genuine official Wireshark sample captures, but the captures contained only 35–92 packets and offered low analytical challenge.The audit characterized the data as authentic but educational samples rather than realistic enterprise traffic.
  • Audit impact: The audit process identified issues spanning leaked ground truth, unverifiable evidence, and insufficiently challenging authentic data, motivating fixes before benchmark inclusion.The authors describe these audits as catching both evidence failures and task-design problems.

F.3 Trajectory Analysis Example: 3D Slicer

The 3D Slicer analysis found pervasive failures driven by indecision while searching anatomical slices, whereas preloaded relevant views enabled faster core measurements. These observed weaknesses informed a harder long-horizon task design.

  • Trajectory Analysis: All 34 analyzed Slicer3D trajectories failed, with only two tasks receiving any points.The best score was 31/100, while the second scored 5/100; the remaining 32 tasks scored zero.
  • Failure Pattern: The Scrolling Loop: The dominant failure mode was a scrolling loop in which agents searched image slices indefinitely without placing measurements.Agents could identify the task and activate the measurement tool, but continued scrolling until exhausting their steps.
  • Success Pattern: Decisive Action: The highest-scoring task used preloaded relevant structures, allowing the agent to place a measurement immediately in six steps.The agent later ran out of steps before completing the report, but completed the core measurement.
  • Identified Agent Weaknesses: The analysis identified missing anatomical knowledge and difficulty committing under uncertainty as key agent weaknesses.These weaknesses included locating specific anatomical levels and completing multi-step medical protocols.
  • Task Design: These failure patterns directly informed a CUA-World-Long task requiring long-horizon planning and commitment across multiple anatomical structures.The task was designed to be genuinely difficult rather than merely repetitive.

L Propose-and-Amplify Ablation: Qualitative Analysis

Seed examples made amplified tasks more realistic, longer, and more richly initialized across Firefox, AstroImageJ, and Moodle. Without seeds, generated tasks more often reduced software use to simpler feature operations.

  • Realism: Seeded tasks reflected professional workflows using real websites, astronomical objects and techniques, and institutional Moodle operations.Examples included investigative research, photon-transfer analysis, light-curve detrending, grade auditing, and custom role creation.
  • Difficulty and horizon: Seeded tasks typically required 50–80 steps and chained multiple operations, compared with 30–50-step single-feature tasks without seeds.Seeded workflows included researching, downloading, organizing, synthesizing, processing, measuring, reporting, configuring, and verifying.
  • Setup script quality: Seeded setup scripts performed substantial data preparation, whereas unseeded scripts more often launched applications or opened URLs with minimal preparation.Seeded examples included Hubble downloads, physically plausible FITS data, and multi-table SQL inserts.
  • Summary: Seed tasks taught the amplifier both what realistic professional work looks like and how to prepare rich initial states with real or realistic data.Without these examples, the amplifier more often produced simpler, less realistic, and less reproducible tasks.

M Trajectory Behavioral Analysis

The behavioral analysis discovered recurring, environment-agnostic patterns by summarizing and matching agent trajectories without initially exposing pass or fail labels. Passed and failed trajectories differed notably in access blockers, tool pivoting, and save/export behavior.

  • Dataset and pipeline: The analysis covered 2,981 trajectories, including 701 passed and 2,280 failed runs.The pipeline analyzed evaluation trajectories across environments before comparing behavioral patterns.
  • Stage 1: Per-trajectory behavioral summary: Per-trajectory summaries were generated without pass/fail labels or predefined categories to reduce bias toward expected failure modes.The summaries described each trajectory’s phases and behavior in the model’s own words.
  • Stage 2: Pattern discovery: Pattern discovery consolidated recurring environment-agnostic behaviors into 15 canonical patterns.Randomly mixed trajectory summaries were processed in batches, then overlapping patterns were merged.
  • Stage 3: Pattern matching: Pattern matching measured both step fraction and presence rate, allowing multiple patterns to apply to the same trajectory step.The two metrics capture how extensively a pattern appears and how often it appears across trajectories.
  • Results: Access blockers occupied 23% of failed-trajectory steps versus 4% of passed-trajectory steps, while save/export appeared in 52% versus 33% of trajectories.Tool pivoting was also present in 38% of failed trajectories, and save/export behavior was more common among passed trajectories.
Loading 2604.06126v1…