PaperlayerView exact revision on arXiv

Source-linked AI summary

Gym-Anything: Turn any Software into an Agent Environment

Pranjal Aggarwal, Graham Neubig, Sean Welleck

arXiv:2604.06126v1cs.LGcs.AI

TL;DR

Computer-use research lacks scalable environments for realistic, long-horizon work across diverse professional software. Gym-Anything addresses this with creation-audit agents, producing CUA-World and showing that test-time auditing raises Gemini-3-Flash performance on CUA-World-Long from 11.5% to 14.0%.

  • Problem

    Existing benchmarks cover limited software and short-horizon tasks, leaving agents’ ability to handle realistic professional workflows an open question.

    Source: §1 · p. 2

  • Method

    Gym-Anything uses a multi-agent creation-audit-correction loop to convert software into environments, selecting applications through GDP-grounded occupational coverage.

    Source: §1 · p. 2, §2.2 · p. 4, §3 · p. 6

  • Results

    Test-time auditing improves Gemini-3-Flash on CUA-World-Long from 11.5% to 14.0%, while CUA-World spans over 10,000 tasks across 200 applications.

    Source: §1 · p. 3

  • Takeaways & Limitations

    CUA-World provides broad training and evaluation coverage, while realistic long-horizon computer use remains far from solved.

    Source: §9 · pp. 17–18

    A large fraction of professionally used software remains excluded, and performance on free alternatives may not predict performance on commercial counterparts.

    Source: §11 · p. 18

Abstract

from arXiv · show

Computer-use agents hold the promise of assisting in a wide range of digital economic activities. However, current research has largely focused on short-horizon tasks over a limited set of software with limited economic value, such as basic e-commerce and OS-configuration tasks. A key reason is that creating environments for complex software requires significant time and human effort, and therefore does not scale. To address this, we introduce Gym-Anything, a framework for converting any software into an interactive computer-use environment. We frame environment creation itself as a multi-agent task: a coding agent writes setup scripts, downloads real-world data, and configures the software, while producing evidence of correct setup. An independent audit agent then verifies evidence for the environment setup against a quality checklist. Using a taxonomy of economically valuable occupations grounded in U.S. GDP data, we apply this pipeline to 200 software applications with broad occupational coverage. The result is CUA-World, a collection of over 10K long-horizon tasks spanning domains from medical science and astronomy to engineering and enterprise systems, each configured with realistic data along with train and test splits. CUA-World also includes CUA-World-Long, a challenging long-horizon benchmark with tasks often requiring over 500 steps, far exceeding existing benchmarks. Distilling successful trajectories from the training split into a 2B vision-language model outperforms models 2$\times$ its size. We also apply the same auditing principle at test time: a separate VLM reviews completed trajectories and provides feedback on what remains, improving Gemini-3-Flash on CUA-World-Long from 11.5% to 14.0%. We release all code, infrastructure, and benchmark data to facilitate future research in realistic computer-use agents.

1 Introduction

Gym-Anything addresses the mismatch between short-horizon benchmarks and realistic professional computer use by turning economically important software into audited interactive environments. It produces CUA-World, a broad task collection with long-horizon evaluation, training, and test-time auditing results.

Motivation

  • Existing benchmarks focus on short-horizon tasks in a narrow set of consumer applications, whereas professional workflows span hundreds of steps across heterogeneous software configured with domain-specific data.This limits both faithful evaluation of economically relevant work and the diversity of training trajectories; realistic environment creation can require weeks of expert effort per application.Source: §1 · p. 2

Environment construction

  • Gym-Anything converts software into standardized interactive environments through a creation-audit loop in which agents write setup scripts, provide evidence, and independently verify quality.The library abstracts low-level infrastructure, while shared memory lets creation agents improve across attempts.Source: §1 · p. 2, Figure 2 · p. 3

Task generation and evaluation

  • Tasks are generated by amplifying a small set of agent-produced seed tasks with a cheaper language model and evaluated using checklist-based VLM verification with privileged setup information.Weighted subtasks enable partial credit, while embedded setup data supplies verifier checks unavailable to task-solving agents.Source: §1 · p. 2, Figure 2 · p. 3

Benchmark and training

  • CUA-World contains over 10,000 tasks across 200 software applications, three operating systems, diverse domains, and train/test splits; distilling training trajectories into a 2B model outperforms models 2× larger.Performance scales with the number of software and environments in training, and the distilled model generalizes to unseen software.Source: §1 · p. 3, Figure 2 · p. 3

Long-horizon evaluation

  • CUA-World-Long remains difficult: the strongest frontier model reaches only 27.5% pass rate, while Test-Time Auditing improves Gemini-3-Flash from 11.5% to 14.0%.The benchmark contains one realistic, failure-mode-targeting task per software, often requiring hundreds of steps; agents commonly stop before completion.Source: §1 · p. 3

2 Methodology

Gym-Anything formalizes computer-use tasks as scored interactions with initialized software environments and selects software by estimated economic importance under sandboxability constraints. Its library represents environments with staged setup scripts and declarative configuration, enabling reusable, scalable construction across software and infrastructure.

2.1 Problem Setup

  • A task pairs an initialized interactive software environment with a natural-language instruction and verification function, while agents act from observations through keyboard and mouse inputs for up to T steps.The final score is determined by the verification function applied to the agent’s trajectory.Source: §2.1 · p. 4

2.2 GDP-Grounded Software Selection

  • The GDP-grounded selection pipeline estimates occupation GDP, attributes it to software, filters for sandboxability, and selects 200 applications across tiers balancing economic importance and occupational diversity.Sandbox-ready software must be self-hostable, free-tier, GUI-based, and usable without unsimulatable hardware; approximately 3,400 of 16,600 applications meet these constraints.Source: §2.2 · pp. 4–5
  • Occupation GDP is estimated from employment and wages, then decomposed across software using computer-use, category, and within-category share factors informed by occupational data and web-grounded LLM estimates.The software catalog contains approximately 16,600 applications across approximately 1,400 categories and is deduplicated, validated, and checked against hallucinated entries.Source: §2.2 · p. 5

2.3 The Gym-Anything Library

  • The Gym-Anything Library defines each environment with install, configure, and task-setup scripts plus declarative configuration, allowing tasks to share reusable stages through a standard gymnasium-style interface.The library manages container orchestration, display forwarding, input translation, checkpointing, caching, and execution across Linux, Windows, and Android with multiple compute backends.Source: §2.3 · pp. 5–6

3 Scaling Computer-Use Agent Software Applications

Gym-Anything turns environment setup into an iterative multi-agent coding and computer-use task, addressing the difficulty of configuring and verifying realistic software environments at scale. Its creation-audit-memory pipeline produces 200 environments across Linux, Windows, and Android with realistic data.

Motivation

  • Environment creation is scalable because the task is constrained to writing setup scripts and configuration files, while correctness is verified by launching and interacting with the software.Without this automation, installation, domain-data configuration, and verification can require weeks of expert effort per application.Source: §3 · p. 6

Multi-agent framework

  • The framework iteratively creates, audits, and improves environments because naive agents often stop early, use placeholder data, leave incorrect screens, or skip verification.A shared memory accumulates learnings across attempts, while re-prompting agents to reread guidelines and checklists recovers omissions caused by context fatigue.Source: §3 · pp. 6–7

Creation agent

  • Creation agents research software configuration, obtain real-world data, implement setup scripts, launch applications, visually verify states, debug failures, and produce screenshots or logs as evidence.Each agent receives a software-agnostic workflow prompt plus the target software and studies previously created environments before implementation.Source: §3 · p. 7

Audit agent

  • Audit agents adversarially evaluate screenshots, logs, configuration, and scripts against quality checklists, running the environment when necessary and reporting critical issues for correction.Separating auditing from creation reduces self-confirmation bias, improves audit detail and accuracy, and makes quality independently interpretable to human authors.Source: §3 · p. 7

Shared memory

  • Shared memory stores software-specific and general lessons, enables parallel agents to reuse discoveries, and periodically summarizes accumulated notes to limit context growth.A summarization agent runs every L environments, adding theoretically only ∼1/L overhead compared with repeatedly reading the full software-specific memory.Source: §3 · p. 7

Output

  • 200 software applications across Linux, Windows, and Android were converted into environments configured with realistic public email, medical imaging, financial, and government data.The pipeline is fully automated and extensible to additional software, with the 200-application scope set by the current compute budget.Source: §3 · pp. 7–8

4 Scaling Tasks

Gym-Anything scales task creation by proposing a small set of realistic, difficult, diverse seed tasks per software, amplifying them with a language model, and filtering invalid or duplicate tasks. It verifies long-horizon trajectories with a privileged-information, checklist-based VLM that supports partial credit and enforces workflow integrity.

4 Scaling Tasks

  • Task generation is a separate scaling bottleneck because naive prompting produces invalid, trivial, impossible, or repetitive tasks, while fully agentic authoring and validation is prohibitively expensive.Source: §4 · p. 8
  • An agentic proposer creates a small set of high-quality seed tasks per software, targeting realism, long-horizon difficulty, and functional diversity through interactive execution and verification.This expensive process occurs once per software and covers core functionality across relevant occupations.Source: §4 · p. 8
  • Gemini 3 Pro amplifies seed tasks sequentially with prior instructions as context, then semantic-similarity filtering removes duplicates to improve diversity.Source: §4 · p. 8
  • An automated VLM filter launches each generated task, observes its starting state, and discards tasks whose state does not match the described expectation.Source: §4 · p. 8

4.1 Task Verification

  • The verifier combines task instructions with privileged ground truth extracted from setup scripts to generate granular checklists for evaluating long-horizon trajectories.Privileged information includes deterministically configured facts such as tumor locations or expected account balances and assists verification without changing the evaluated agent’s task.Source: §4.1 · p. 8
  • Checklist judgments award partial credit for completed subtasks without manual annotation, while a separate integrity checklist assigns zero if the agent bypasses the intended software workflow.Integrity checks require using the intended interface and prohibit direct file editing or environment-artifact shortcuts; the method was more reliable than end-state-only VLM and programmatic verification in human-agreement comparisons (§7.3).Source: §4.1 · p. 9

5 CUA-World

CUA-World provides 12,103 verified tasks and environments spanning 200 software applications, all 22 major occupation groups, long-horizon evaluation, automated creation, and train/test splits. CUA-World-Long adds 200 trajectory-guided tasks that often exceed 200 human steps and challenge models beyond 500 steps.

CUA-World

  • 12,103 tasks and environments span 200 software applications with checklist-based verification, covering all 22 major occupation groups and providing train/test splits.The collection combines interactive environments at scale, long-horizon evaluation, automated environment creation, and a training split.Source: §5 · p. 9, Table 2 · p. 10, Figure 5 · p. 10

Contamination filtering

  • Contamination filtering assigns connected components of similar tasks wholly to Train or Test, minimizing cross-split leakage while accepting some false positives.Task pairs scoring 4 or higher on an 8-point similarity scale are flagged, and manual verification found very few false negatives.Source: §5 · p. 10

CUA-World-Long

  • CUA-World-Long contains 200 trajectory-guided tasks, one per software application, designed by analyzing strong-agent failures and common failure modes.Task generation also uses eight quality guidelines covering real-world relevance, objective evaluability, realistic data, and related criteria.Source: §5 · p. 11
  • Often requiring more than 200 human steps, CUA-World-Long tasks frequently defeat current models even after 500 steps.All 200 tasks were manually verified for correct setup and meaningfulness, with failed tasks iteratively refined against the quality criteria.Source: §5 · p. 11

6 Experimental Setup

The experiments use CUA-World both to distill smaller computer-use models from teacher trajectories and to evaluate agents, including long-horizon behavior with test-time auditing. Evaluation uses weighted checklist scores and pass rates on full and long-horizon test splits under fixed episode budgets.

6 Experimental Setup

  • CUA-World serves as both training data for distilling smaller models and an evaluation benchmark for computer-use agents.The training procedure uses successful teacher trajectories, while evaluation measures agent performance on held-out tasks.Source: §6 · p. 11

6.1 Training

  • The training pipeline distills roughly 2000 successful trajectories from Kimi-K 2.5 into Qwen3-VL-2B-Thinking across 10,000 tasks.Four teacher trajectories are generated per task until at least one is correct; the best configuration is selected after ablating teacher choice and trajectory sampling designs.Source: §6.1 · p. 11

6.2 Test-Time Auditing (TTA) Agent

  • Test-time auditing addresses premature stopping on extremely long tasks by checking completed trajectories and prompting the main agent to continue when subtasks remain.The audit agent receives all screenshots but not the main model’s chain-of-thought, and explains missing work when the task is incomplete.Source: §6.2 · p. 11

6.3 Evaluation

  • Evaluation reports Average Score for partial checklist credit and Pass Rate for fully completed tasks on CUA-World-Test and 200-task CUA-World-Long.Episodes allow 200 steps on CUA-World-Test and 500 steps or $5 on CUA-World-Long; Gemini 3 Flash is the default verifier.Source: §6.3 · p. 11

7 Results and Analysis

Results show that distillation improves small-model performance, while CUA-World-Long remains difficult because success requires substantial computation. Analyses further identify scaling, verification, behavioral, software-complexity, and environment-generation factors that shape performance.

7.1 Main Results

  • 4.4% pass rate for distilled Qwen3-VL-2B versus 1.6% before distillation, surpassing the 3.9% of Qwen3-VL-4B despite being half its size.On CUA-World-Test, Gemini-3-Flash leads frontier models with a 50.1 average score and 22.6% pass rate.Source: §7.1 · p. 12, Table 3 · p. 12
  • 27.5% pass rate for GPT-5.4 after removing the cost cap and allowing 2,000 steps, versus 3% under the capped setting.Gemini-3-Flash reaches 11.5% at approximately 1,300 steps, but test-time costs rise to about $16 for Gemini and $18 for GPT-5.4 per trajectory.Source: §7.1 · p. 12, Table 4 · p. 12
  • ~3.5-point score gains accompany each doubling of training data across software count or task fraction, following a roughly log-linear scaling trend.The result supports further scaling of Gym-Anything’s environment and task pipeline.Source: §7.1 · p. 12, Figure 6 · p. 13
  • OOD training recovers only 22-27% of the all-software improvement, compared with 65-87% on IID software, showing that unseen-software generalization remains limited.Distillation nevertheless improves average scores from 16.7 to 24.2 on IID software and from 12.3 to 14.1 on OOD software at 25% software coverage.Source: §7.1 · pp. 12–13, Figure 7 · p. 13

7.2 Scaling Test-Time Compute

  • 14.0% pass rate under the same 2,000-step cap with Test-Time Auditing, up from 11.5% without auditing; pass rate rises sharply only beyond roughly 100 steps.The auditor provides feedback on missed subtasks, especially when the agent stops prematurely.Source: §7.2 · pp. 13–14, Figure 6 · p. 13

7.3 Benchmark Analysis

  • 78% of failed-trajectory steps are retry loops versus 39% for passed trajectories, while verification checks occur in 91% of passed versus 70% of failed trajectories.UI exploration remains high for both outcomes, at 76% versus 67%, indicating substantial navigation overhead.Source: §7.3 · p. 14, Figure 8 · p. 14
  • 93.3% task-level human agreement for checklist-based VLM verification exceeds 81.7% for direct VLM verification and 43.3% for programmatic verification.Integrity checks flagged approximately 1.5% of high-scoring runs, with 15 true positives among 21 flags.Source: §7.3 · pp. 15–16

7.4 Gym-Anything Pipeline Ablations

  • Seed examples raise setup success from 55.2% to 88.9%, while cross-model audits identify 2.1 additional issues per environment on average across 10 software applications.CUA-World-Long has an overall mean of 425 steps, and smaller models show steeper declines from general to specialized software than larger models.Source: §7.3 · pp. 14–15, §7.4 · p. 16

8 Related Work

Prior computer-use benchmarks are either static or small-scale interactive, while existing environment-generation and evaluation methods do not support realistic, scalable GUI software tasks. Gym-Anything addresses these gaps through automated creation, task amplification, and checklist-based auditing across broad software and occupational coverage.

Benchmarks for computer-use agents

  • CUA-World provides 10K+ interactive tasks across 200+ software applications on four platforms, with train/test splits, long-horizon evaluation, and coverage of all 22 SOC occupation groups.Prior interactive benchmarks cover narrow software slices and rely on manual environment creation, while static datasets evaluate action matching rather than execution.Source: §8 · p. 17

Automated environment and task generation

  • Gym-Anything uniquely combines creation-audit, propose-and-amplify, and shared-memory mechanisms to generate executable tasks in real GUI software across environments.Earlier methods generate tasks within existing environments or target text, APIs, replicas, or single ecosystems without installing and configuring realistic multi-OS software.Source: §8 · p. 17

Evaluation of computer-use agents

  • The checklist-based VLM verifier uses setup-script ground truth and integrity checks to detect workflow shortcuts, avoiding per-task code while addressing limitations of binary programmatic verification.Existing programmatic verifiers are reliable but labor-intensive, whereas prior VLM evaluators lack ground-truth answers and cannot detect shortcuts such as fabricated outputs or tool misuse.Source: §8 · p. 17

9 Conclusion

Gym-Anything makes it scalable to convert arbitrary software into realistic computer-use environments, yielding CUA-World, a broad benchmark for economically relevant and long-horizon tasks. The conclusion emphasizes that progress requires both stronger agents and scalable environment construction, while CUA-World-Long remains challenging even for frontier models.

Framework

  • Gym-Anything scales environment creation by combining setup scripts and configuration files with a multi-agent loop of generation, auditing, and correction.The framework addresses the difficulty of constructing realistic environments at scale.Source: §9 · p. 17

Benchmark

  • CUA-World contains over 10K GDP-grounded tasks across 200 software applications, diverse occupations and domains, multiple operating systems, checklist-based VLM verification, and train/test/long-horizon splits.The released benchmark is paired with code and infrastructure for future research.Source: §9 · p. 17

Results

  • CUA-World provides supervision for training smaller agents through distillation, while test-time auditing improves performance on especially long-horizon tasks.These findings show that the benchmark supports both training and evaluation-time improvements.Source: §9 · p. 18
  • CUA-World-Long remains challenging even for frontier models, indicating that realistic computer-use is far from solved.The benchmark targets long-horizon computer-use settings that continue to expose substantial capability gaps.Source: §9 · p. 18

Implications

  • Progress in computer-use agents requires stronger models alongside scalable construction of the environments and tasks used for training and evaluation.The authors position Gym-Anything, CUA-World, and the released infrastructure as a foundation for more capable agents, stronger verifiers, and broader software coverage.Source: §9 · p. 18

11 Limitations

Gym-Anything’s benchmark construction and evaluation involve important limitations in software coverage, solvability guarantees, estimation precision, and verifier reliability.

11 Limitations

  • The GDP-grounded selection provides a relative ranking rather than precise dollar-level attribution, and alternative share-factor estimation methods may be more accurate.The framework uses the strongest available LLM with web-search access to estimate Equation 1’s share factors.Source: §11 · p. 18
  • Many professionally used applications remain excluded, and it is unknown whether performance on free sandboxable alternatives predicts performance on commercial counterparts.The authors select the closest sandboxable alternative when software cannot be freely sandboxed, such as because of licensing.Source: §11 · p. 18
  • Although environments and task starting states were manually checked, the authors did not solve every task end-to-end and therefore cannot guarantee universal solvability.A fully human-verified benchmark is identified as future work.Source: §11 · p. 18
  • VLM checklist verifiers show high agreement with human annotations but remain imperfect and potentially vulnerable to adversarial exploitation.The authors propose manually vetted programmatic verifiers as a complementary future direction.Source: §11 · p. 18

12 Ethics Statement … A.5 Phase 5: GDP Attribution

The paper acknowledges deployment risks while limiting its release to environment-creation and auditing methods, using public or synthetic data and freely available software. Its GDP-grounded software-selection pipeline estimates occupational and product economic importance through staged calculation, discovery, cleanup, enrichment, and attribution.

12 Ethics Statement

  • The work acknowledges autonomous computer-use risks but releases no model exceeding existing frontier capabilities, using freely available software and public or synthetic datasets.Source: §12 · p. 18

A GDP-Grounded Software Selection: Full Pipeline

A.1 Phase 1: Occupation GDP Calculation

  • GDP values for 894 O*NET occupations are computed by scaling employment-weighted wages to labor compensation and then national GDP.The procedure uses BLS OEWS May 2024 wages and BEA compensation and GDP ratios.Source: §A.1 · p. 26

A.2 Phase 2: Software Discovery

  • GPT-5 discovers occupation-to-software categories, while product enumeration and fuzzy deduplication produce a catalog of approximately 16,000 products across approximately 1,400 categories.The discovery stage yields 5,584 occupation-to-category pairs across 894 occupations.Source: §A.2 · pp. 26–27

A.3 Phase 3: Catalog Cleanup

  • Three cleaning passes validate product-category membership, verify real current availability through Google Search-grounded LLM checks, and remove mismatches or hallucinated products.LLM calls are cached in JSONL format for reproducibility.Source: §A.3 · p. 27

A.4 Phase 4: Catalog Enrichment

  • Products are enriched in parallel with pricing and open-source status, interface type, and trainability categories based on installation or access requirements.Source: §A.4 · p. 27

A.5 Phase 5: GDP Attribution

  • Product GDP estimates aggregate occupation-level allocations constrained by O*NET computer-use scores and normalized category and product shares.The highest-estimated products are Microsoft Excel, Microsoft Word, Google Chrome, Microsoft Outlook, and Visual Studio Code; these rankings are pipeline estimates and should not be treated as sourced rankings.Source: §A.5 · p. 27

A.6 Phase 6: Practical Access-Barrier Evaluation … C Prompts

This section describes how the pipeline evaluates practical access barriers and implements software environments with modular infrastructure, realistic data, checkpointing, verification, and provenance auditing. It also details validation of privileged information and two-stage VLM evaluation of task completion and integrity violations.

A.6 Phase 6: Practical Access-Barrier Evaluation

  • Results show that 8,013 products (48%) are trainable, while 8,591 (52%) are not: 4,651 require organizational accounts, 3,279 free accounts, and approximately 661 hardware.The assessment asks whether products require external accounts, organizational or institutional credentials, specialized physical hardware, and whether they are trainable in a sandbox.Source: §A.6 · pp. 27–28

A.7 Phase 7: Tiered Selection

  • The selection pipeline requires supported platforms, non-paid access, non-CLI interfaces, sandbox readiness, and passage of the access-barrier evaluation.Non-selectable products are replaced by the closest same-category selectable alternative, which inherits the original economic slot while the original is marked covered.Source: §A.7 · p. 28
  • Approximately 500 selected products cover all 22 SOC major groups, with environments built for 200 under the available compute budget.Fifty-three environments retained after a selection-code bug would also have appeared in the planned full 500-product release.Source: §A.7 · p. 28

B Gym-Anything Framework: Technical Details

  • Gym-Anything uses modular environment specifications and a common runner interface so observation modalities, actions, operating systems, compute backends, and verification strategies can be extended independently.The schema specifies runtime resources, interfaces, accounts, mounts, and security; automatic execution selects Docker, Apptainer, or a local runner.Source: B Gym-Anything Framework: Technical… · p. 28, §B.1 · p. 29, §B.2 · p. 29

B.3 Progressive Checkpointing

  • Progressive checkpoints skip completed installation, configuration, and task setup stages, while full-state QEMU restoration takes approximately 3 seconds and copy-on-write supports 400+ parallel environments.Checkpoints are shared across tasks when possible and task-specific after initialization; disk-state restoration instead requires rebooting and takes 2–5 minutes.Source: §B.3 · pp. 29–30

B.5 Verification System

  • Decoupled verifiers support programmatic checks, image matching, and cascaded verification, while each episode records logs, screenshots, video, metadata, rewards, and setup outputs.The resulting artifacts support both evaluation and trajectory distillation, and remote workers plus a dashboard enable distributed execution and replay.Source: §B.5 · p. 30, §B.6 · p. 30, §B.7 · p. 30

C.1 Creation Agent Prompt (Overview)

  • The Creation Agent follows seven phases covering framework research, application installation, implementation, interactive testing, and documentation, with all task data required to be real and publicly sourced.The prompt directs screenshot-based testing to confirm application state, real-data loading, and task completion, then requires evidence documents and a clean uncached test.Source: §C.1 · pp. 31–32

C.2 Privileged Information Audit Prompts

  • The Privileged Information Audit analyzes task files, verifies provenance and metadata claims through grounded web search, and synthesizes only externally validated information into the final report.It checks datasets, source URLs, case identifiers, synthetic-data status, hardcoded versus computed values, and each claim’s verified value and confidence.Source: §C.2 · pp. 32–33
  • Privileged-information audits include only verified facts, mark uncertain values as unverified, and record metadata, sources, verification status, and confidence in validated_pi.json.The summary explicitly prioritizes omitting unsupported information over including fabricated privileged information.Source: §C.2 · pp. 33–34

C.3 VLM Checklist Verifier Prompts

  • The verifier first generates a structured checklist from the task description and validated privileged information, then scores agent trajectories from sampled screenshots.Task-completion items total exactly 100 points and require visually verifiable evidence, while integrity items target cheating or shortcuts.Source: §C.3 · p. 34
  • Task-completion checklist items receive pass, partial, or fail judgments, whereas integrity checks receive binary pass/fail judgments based on clear visual evidence.Partial credit represents meaningful but incomplete progress, and integrity failure requires evidence of cheating rather than ordinary task failure.Source: §C.3 · p. 35
  • Integrity failures are reserved for hardcoding, copying expected values, fabricating results without the software, or bypassing the required workflow.An agent that honestly attempts a task but fails is explicitly treated as an integrity pass.Source: §C.3 · p. 35

C.4 Integrity Check Analysis … 1 High

This section examines integrity and contamination safeguards for constructing reliable software environments, including evidence requirements and adversarial auditing. It also reports audit findings from concrete environments, highlighting documentation and start-state evidence issues while noting that other tasks passed without verifier, infrastructure, or feasibility problems.

C.4 Integrity Check Analysis

  • Among 21 integrity flags in high-scoring Gemini-3-Flash runs, 15 were true positives and 6 false positives, but none changed the overall pass rate.Eighteen flagged cases already failed the task checklist, while the three perfect-score cases were all false positives; the checks remain useful for harder tasks where task scoring may miss shortcuts.Source: §C.4 · p. 36
  • Integrity checks detected agents fabricating report values, independently computing results, hardcoding exclusions, or using a terminal instead of the required specialized GUI.These examples show correct-looking outputs or high task-completion scores can still violate the intended workflow; terminal fallback occurred when the agent could not operate PsychoPy’s Coder interface.Source: §C.4 · pp. 36–37

C.5 Contamination Filtering Prompt

  • Contamination filtering compares every within-environment task pair and assigns connected components containing similarity scores ≥4 to the same train/test split.Categories 4–8 represent truly interchangeable, duplicated, subset, or superset tasks; common substeps alone receive category 3 and do not trigger contamination grouping.Source: §C.5 · p. 37

D.1 Evidence Requirements

  • Environment evidence comprises timestamped screenshots, per-task structured verification JSON, and export-script output proving valid parseable JSON.The artifacts verify application and task start states, absence of blocking errors, filesystem and service conditions, and successful result export without launching the VM.Source: §D.1 · p. 37

D.3 Worked Example: Android Studio — Offline Caching Feature

  • The Android Studio worked example documents nine intentional bugs and a 12-item evidence checklist, with every listed check marked PASS despite screenshots showing a remaining unresolved ksp build error.The evidence records project files, permissions, cleanup, timestamps, bug presence, Gradle operations, export validity, network access, and GUI accessibility.Source: §D.3 · pp. 38–40

E Audit Quality Checklist and Example Audits

  • Gym-Anything pairs each Creation Agent with an independent Audit Agent instructed to distrust comments, verbal claims, and unverified screenshots while checking environment quality.The audit checklist covers task-description clarity, start-state correctness, data authenticity, adversarial treatment of comments, and evidence verification.Source: E Audit Quality Checklist and Example… · p. 40, §E.1 · pp. 40–41

E.2 Example Audit: Odoo CRM Environment (Critical Issues Detected)

  • An Odoo CRM audit passed the environment with moderate issues, identifying mislabeled start-state and final-state screenshots that undermined evaluation evidence.The environment used real Odoo demo data and authentic Docker infrastructure, but the key screenshots showed an empty new-lead form rather than the CRM kanban; additional issues included an incomplete dialog, a field-label mismatch, and data inconsistencies.Source: §E.2 · pp. 41–42
  • The audit found the lead task’s start-state screenshot showed an empty form instead of the CRM kanban pipeline, while its completion evidence passed with realistic seeded data.Additional issues included a mislabeled description field, an incorrect final-pipeline screenshot, an incomplete activity-dialog screenshot, phone-number inconsistency, and an inaccurate README record count.Source: §E.2 · p. 42, §1 · p. 42
  • Tasks 2–4 passed their start-state and completion screenshot checks, with minor concerns about customer linking and a dialog captured mid-process.Task 2 had unclear customer linking, while Task 3’s dialog screenshot was incomplete.Source: §E.2 · p. 42
  • Task 5 passed with authentic evidence showing the opportunity moved from Proposition at 60% probability to Won at 100%.The completion screenshot showed a green WON ribbon and Odoo’s celebration animation.Source: §E.2 · p. 42

1 High

  • The high-severity issue summary listed incorrect screenshots as the main problems, while verifiers, infrastructure setup, and overall task feasibility had no issues.The summary also recorded medium- and low-severity documentation, data-consistency, and record-count discrepancies.Source: §1 · p. 43

E.3 Example Audit: Wireshark Environment (Mixed Results) … F.2 Task Generation Pipeline

This section presents the quality principles and automated pipeline used to generate and verify realistic, occupationally relevant, long-horizon tasks across 200 software environments. It also describes how auditing and failure-driven feedback identify and correct task-design, evidence, and data-quality issues.

E.3 Example Audit: Wireshark Environment (Mixed Results)

  • The Wireshark audit passed infrastructure and start-state setup but found leaked ground truth, over-prescriptive instructions, misleading evidence, and authentic yet trivially small captures, yielding PASS with moderate issues.The five captures contained 35–92 packets, and the evidence screenshot showed the Ethernet rather than IPv4 tab for the top-sender task.Source: §E.3 · pp. 43–45

E.4 Cross-Round Audit Examples: How the Creation-Audit Loop Corrects Issues

  • Repeated creation-audit cycles corrected distinct failures: PEBL fixed impossible response-key instructions, SeisComP added missing GUI and data evidence, and Graphite replaced click-by-click directions with goal-oriented instructions.Round 2 audits confirmed PASS after each correction, and over-prescriptive descriptions were reported across multiple environments.Source: §E.4.1 · p. 46, §E.4.2 · p. 47, §E.4.3 · pp. 47–48

E.5.1 Visallo (Investigative Analytics Platform)

  • In Visallo, cross-model auditing additionally caught a file-path mismatch, an Elasticsearch client-version mismatch, and a third-party ontology dependency that self-auditing missed.Both audits nevertheless identified missing data loading and the dashboard’s “NO DATA” counter-evidence.Source: §E.5.1 · p. 48

E.5.2 Mattermost (Team Messaging Platform)

  • In Mattermost, cross-model auditing additionally identified missing task-specific screenshots, unextracted evidence files, and limited seeded conversations lacking threads, reactions, and attachments.Both audits agreed that the environment used real GitHub Releases data and had a browser mismatch.Source: §E.5.2 · p. 48

E.5.3 OpenELIS (Laboratory Information Management System)

  • In OpenELIS, cross-model auditing revealed absent laboratory data, hardcoded patient records contradicting sourcing claims, and a missing Creatinine test that could make a task infeasible.Both audits identified the infeasible validation task and unreliable login-page start state, but self-auditing missed the data-source contradiction and missing test.Source: §E.5.3 · p. 48

E.5.4 Summary

  • Cross-model audits found 2.1 additional issues per environment on average across 10 applications, while both audit configurations detected critical and high-severity problems.Cross-model audits were more adversarial toward AgentC’s claims, especially data quality, file paths, and evidence documentation.Source: §E.5.4 · p. 49

F CUA-World-Long: Quality Guidelines and Generation Pipeline

  • CUA-World-Long contains 200 tasks requiring 200+ interaction steps each, generated automatically by analyzing trajectory failures and targeting systematic weaknesses rather than relying on manual authoring.Every generated task must satisfy eight quality guidelines and pass multi-stage verification before benchmark inclusion.Source: F CUA-World-Long: Quality Guidelines… · p. 49

F.1 Quality Guidelines

  • Each CUA-World-Long task is included only when it clearly satisfies all eight quality principles.The principles define the quality screen applied to the long-horizon benchmark tasks.Source: §F.1 · p. 49
  • Tasks must be realistic, economically relevant, objectively evaluatable, software-relevant, feasible to set up, genuinely difficult, and long-horizon without artificial subtask inflation.Difficulty should arise from domain complexity, multi-step reasoning, and planning rather than repetitive chaining or unnecessary task length.Source: §F.1 · p. 49

F.2 Task Generation Pipeline

  • The pipeline begins by using VLM checklist scores from existing trajectories to identify failures, design candidate tasks, and evaluate them against all eight quality guidelines.The verifier processes runs in parallel with up to 16 workers, and its failure signals guide task design; candidates that are artificially hard or insufficiently long-horizon are rejected.Source: §F.2 · p. 50
  • The implementation creates task metadata, setup and export scripts, and a verifier while enforcing answer non-leakage, determinism, and established file patterns.The system studies at least 3–4 existing tasks before implementing the required files.Source: §F.2 · p. 50
  • Live testing verifies 12 environment and task checkpoints through commands or screenshots, fixing issues immediately rather than relying on assumptions.Checks cover software state, data validity, permissions, timestamps, task-data consistency, executable operations, exports, JSON validity, network access, and GUI accessibility.Source: §F.2 · p. 50
  • Evidence collection and PI auditing produce structured artifacts and validated ground truth by independently checking metadata against files, live computations, and authoritative web sources.Claims are labeled verified, contradicted, or unverified, and the resulting privileged-information summary supports later VLM trajectory scoring; the full pipeline runs autonomously across 200 environments with stage timeouts and environment-specific caching.Source: §F.2 · pp. 50–51

F.3 Trajectory Analysis Example: 3D Slicer … K.2 Evaluated Models

This section analyzes agent failures and task diversity in CUA-World, covering occupationally grounded workflows across a broad range of software and domains. It also documents contamination control, software and occupational categorization, and the models and evaluation harnesses used in the Gym-Anything pipeline.

F.3 Trajectory Analysis Example: 3D Slicer

  • Across 34 3D Slicer trajectories, all tasks failed, with only tumor_ventricle_proximity scoring 31/100 and ivc_diameter_assessment scoring 5/100.The dominant failure was a scrolling loop: agents searched image slices indefinitely instead of committing to measurements, whereas preloaded visible structures enabled decisive action in six steps for the highest-scoring task.Source: §F.3 · p. 51
  • The Slicer failure analysis directly shaped a CUA-World-Long task requiring commitment under uncertainty and long-horizon planning across multiple anatomical structures.The resulting challenge targets domain-knowledge gaps, uncertainty management, and complex multi-step medical protocols rather than mere repetition.Source: §F.3 · p. 51

G Task Examples

  • CUA-World examples span software tasks in Blender, Google Earth Pro, GIMP, Eclipse, Ardour, GeoGebra, Autopsy, DBeaver, QGIS, OpenLCA, BlenderBIM, and RStudio.The examples cover animation, geospatial analysis, image editing, software testing, audio production, digital forensics, databases, life-cycle assessment, BIM coordination, and statistical modeling.Source: G Task Examples · p. 52

H.1 Pairwise Similarity Scoring

  • The contamination pipeline scores within-environment task pairs on an eight-point similarity scale, flags scores ≥4, builds a graph, and assigns each connected component wholly to one split.Components are greedily assigned toward target train/test ratios, followed by automated verification that no contaminating edge crosses the split boundary.Source: §H.1 · p. 56, Table 12 · p. 56, §H.2 · p. 56

H.3 Aggregate Statistics

  • 98.2% of task pairs scored 3 and only 0.65% were flagged as contaminating; 94.3% of connected components were singletons.Table 13 summarizes the filtering statistics, while flagged pairs were grouped into components to prevent cross-split contamination.Source: §H.3 · p. 56, Table 13 · p. 57

H.4 Manual Verification

  • A web dashboard lets annotators inspect flagged pairs, review judge reasoning, and override borderline score-4 decisions as a final train/test integrity safeguard.This supplements the automated graph-based filtering process with human review.Source: §H.4 · p. 57

I Software Categorization

  • The 200+ software products are categorized by visual complexity: 83 low, 91 medium, and 34 high, based on interface visual density and spatial complexity.Source: I Software Categorization · p. 57

J Occupational Coverage of CUA-World

  • CUA-World’s 13K+ tasks span all 22 SOC major occupation groups, with Gemini 3 Flash identifying each task’s professional occupation and mapping it to a SOC category.Representative examples include sales operations, school administration, IT support, intelligence analysis, and logistics.Source: J Occupational Coverage of CUA-World · p. 58
  • Occupational coverage spans technical, scientific, and engineering roles through tasks in Wireshark, Stellarium, TopoCal, OpenVSP, QGroundControl, OpenBCI GUI, and Slicer3D.Examples include incident-response packet isolation, spacecraft line-of-sight simulation, survey computation, aerodynamic design, drone configuration, neuroscience analysis, and MRI signal-to-noise calculation.Source: J Occupational Coverage of CUA-World · pp. 58–59
  • Service, administrative, agricultural, construction, and industrial workflows include scheduling, sales, food service, warranty tracking, farm records, solar modeling, equipment repair, SCADA logic, and operations dashboards.The examples span roles such as career counselor, food-service manager, administrative assistant, farmer, solar installer, medical-equipment repairer, SCADA technician, and NOC analyst.Source: J Occupational Coverage of CUA-World · pp. 59, 61, 62, 63
  • Educational, creative, and media-oriented examples require course-progress configuration, exam administration, visual analysis, image editing, interior design, and browser setup for specialized work.These tasks use Moodle, Safe Exam Browser, Active Inspire, GIMP, Sweet Home 3D, and Chrome to represent instructor, coordinator, artist, designer, and prepress roles.Source: J Occupational Coverage of CUA-World · pp. 60, 64
  • Healthcare and legal tasks cover prescribing, immunization and laboratory records, medical imaging, probation reviews, contract editing, and document discovery across specialized applications.The workflows assign roles including physician, medical assistant, radiologist, radiologic technologist, probation officer, paralegal, and legal assistant.Source: J Occupational Coverage of CUA-World · pp. 60–61
  • CUA-World spans occupations including quality control, brewing, transportation, commercial diving, and ship operations through tasks in diverse software.Examples include ERPNext, LibreOffice Calc, Sygic GPS, Subsurface, and Chrome.Source: J Occupational Coverage of CUA-World · p. 64

K.1 Models Used Across the Pipeline

  • Table 14 lists the models and harnesses used at each stage of the Gym-Anything pipeline.Empty table cells indicate that no harness was used.Source: §K.1 · p. 64, Table 14 · p. 65

K.2 Evaluated Models

  • Four frontier models—Gemini 3 Flash, Kimi-K 2.5, Claude Sonnet 4.6, and GPT-5.4—are evaluated on CUA-World-Long, excluding Claude Opus 4.6 because of cost constraints.The authors note that a $5-per-task budget across 200 tasks would limit Opus 4.6 to roughly 100 steps, and its near-parity with Sonnet 4.6 on OSWorld suggests limited additional signal.Source: §K.2 · p. 64
  • GPT-5.4 and Claude Sonnet 4.6 use official agent harnesses, while Gemini 3 Flash and Kimi-K 2.5 use an adapted Qwen3-VL harness from OSWorld.The Qwen3-VL student model uses the OSWorld harness directly.Source: §K.2 · p. 64

L Propose-and-Amplify Ablation: Qualitative Analysis

Seed examples make amplified tasks more realistic, longer-horizon, and better grounded in rich initial states across Firefox, AstroImageJ, and Moodle. Without seeds, the amplifier generally defaults to simpler feature demonstrations with minimal preparation and weaker reproducibility.

Realism

  • Seeded tasks mirror professional workflows with role-specific research, real astronomical objects and techniques, or institutional operations, whereas unseeded tasks mostly demonstrate isolated software features.Moodle is a partial exception, likely because its structure is well represented in LLM training data.Source: L Propose-and-Amplify Ablation:… · p. 65

Difficulty and horizon

  • Seeded tasks require 50–80 steps across multi-stage workflows, compared with 30–50-step unseeded tasks that typically perform single-feature operations.The seeded workflows chain research, data handling, processing, configuration, verification, and reporting across the three applications.Source: L Propose-and-Amplify Ablation:… · p. 66

Setup script quality

  • Seeded setup scripts span 50–164 lines and prepare substantial real or realistic data, while unseeded scripts span 40–120 lines and often only launch applications or open URLs.Seeded preparation includes archived astronomical data, physically plausible FITS noise models, and multi-table database inserts; unseeded setups are less reproducible, though occasional high-quality exceptions occur.Source: L Propose-and-Amplify Ablation:… · p. 66

Summary

  • Seed tasks teach the amplifier professional work patterns and rich initial-state preparation that prompts alone do not provide, preventing generic, simpler, and less reproducible outputs.The comparison covers Firefox, AstroImageJ, and Moodle, illustrating why seed examples improve amplified task quality.Source: L Propose-and-Amplify Ablation:… · pp. 65–66

M Trajectory Behavioral Analysis · N Extended Related Work

The appendix analyzes 2,981 CUA-World trajectories to identify behavior patterns distinguishing passed from failed runs, and situates Gym-Anything and CUA-World against prior benchmarks, environment-generation, training, evaluation, and occupation-grounding work.

M Trajectory Behavioral Analysis

  • The behavioral pipeline summarizes 2,981 trajectories, discovers 15 environment-agnostic patterns, and matches those patterns separately across passed and failed trajectories.It uses LLM-generated per-trajectory summaries, shuffled mixed-environment batches, consolidation, and two metrics: step fraction and presence rate.Source: M Trajectory Behavioral Analysis · p. 66
  • 23% of steps in failed trajectories involve access blockers versus 4% in passed ones, while tool pivoting appears in 38% versus 19% and save/export in 52% versus 33%.Figures 11 and 12 report step-fraction and presence-rate comparisons across all 15 discovered patterns, including retry loops, UI exploration, and verification checks.Source: M Trajectory Behavioral Analysis · p. 66, Figure 11 · p. 67, Figure 12 · p. 67

N Extended Related Work

  • Prior benchmarks trade off scale, execution-based verification, application breadth, or task horizon, whereas CUA-World provides 10K+ tasks across 200+ applications with train/test splits and GDP-grounded coverage.The comparison spans static datasets, browser benchmarks, desktop and mobile environments, and single-application datasets; CUA-World also includes a long-horizon benchmark requiring 200+ steps.Source: N Extended Related Work · p. 67
  • Gym-Anything extends environment and task generation by creating real GUI software environments through an audited coding-agent loop and generating executable tasks through propose-and-amplify.Shared agent memory accumulates learnings so newer environments can be created faster, unlike approaches bounded by pre-existing environments or narrow domains.Source: N Extended Related Work · p. 68
  • Gym-Anything’s checklist-based VLM verifier uses setup-script information to check outputs against ground truth without per-task evaluator code and detects workflow bypasses.This extends prior script-based, LLM-judge, and VLM-based evaluation approaches while addressing integrity failures such as fabricated report data or unintended terminal use.Source: N Extended Related Work · p. 68
  • Where prior occupation research measures AI exposure and existing benchmarks are programming-centric, Gym-Anything selects software through GDP attribution and covers all 22 SOC major occupation groups.This produces interactive, execution-verified tasks rather than one-shot occupation evaluations.Source: N Extended Related Work · pp. 68–69

Related papers

Loading 2604.06126v1…