PaperlayerView exact revision on arXiv

Source-linked AI summary

RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

Shuai Bai, Jiayong Deng, Sicheng Fan, Yikun Fu, Chang Gao, Xuhao Hu, Mianqiu Huang, Yizhen Jiang, Yuheng Jing, Dehui Kong, Keliang Li, Ning Li, Wanli Li, Dayiheng Liu, Dunjie Lu, Changwei Luo, Que Shen, Zheyuan Wang, Zijian Wang, Jie Wu, Gao Wu, Zhihui Xie, Rui Xie, Haiyang Xu, An Yang, Jiakang Yuan, Yanming Zhang, Jiajun Zhang, Xi Zhang, Zhenru Zhang, Zhuo Zhen, Mingkang Zhu, Bowen Zhou

arXiv:2609.22000v2cs.CLcs.SE

TL;DR

CUAs have largely developed as separate GUI-operation and code-based capabilities, although real digital work interleaves both. RecreationWorld studies hybrid agents through reference-based recreation; training improved across five out-of-distribution benchmarks, while agents reproduced static structure more reliably than interactions and computed outputs.

  • Problem

    CUAs have largely developed as separate GUI-operation and code-and-command-line capabilities, while real digital work interleaves both.

    Source: §1 · p. 2

  • Method

    RecreationWorld asks agents to explore a running reference and build a faithful implementation, with no prescribed workflow, in reproducible environments using programmatic and visual tests.

    Source: §1 · p. 2, §8 · p. 21

  • Results

    Across five out-of-distribution coding and hybrid benchmarks, both training runs improved by up to 17.9 percentage points; static structure was reproduced more reliably than interactions and computed outputs.

    Source: §1 · p. 3, §8 · p. 21

  • Takeaways & Limitations

    The framework makes it possible to study the full interface–code loop in reproducible environments rather than evaluating GUI operation and software construction in isolation.

    Source: §8 · p. 21

    RecreationBench uses pinned, reproducible environments and finite frozen test suites, so its scores measure fidelity only within those environments and sampled outcomes.

    Source: §8 · pp. 21–22

Abstract

from arXiv · show

Computer-use agents (CUAs) have advanced along two separate lines: graphical interaction and software development through code and the command line. Real digital work requires both, interleaved rather than stacked end to end. We study hybrid CUAs that autonomously decide when to explore an interface, implement software, and run and visually verify their artifacts. We introduce RecreationWorld, a five-platform framework built around recreation: given a running reference, an agent must discover its behavior and build a faithful implementation with no prescribed workflow. RecreationWorld provides reproducible environments on Ubuntu, macOS, Windows, Android, and Web, plus a unified harness with native GUI control and coding tools. The running reference serves as an oracle for hidden behavioral tests, providing execution-grounded rewards. We scale trajectory generation with high-quality open-source applications. Models trained on these trajectories improve across five out-of-distribution coding and hybrid computer-use benchmarks and more frequently verify their rendered outputs, providing evidence of transfer beyond recreation. For held-out evaluation, we introduce RecreationBench, comprising 250 diverse tasks across domains and platforms. Reference-grounded programmatic and visual assertions cover action-conditioned outcomes at multiple interaction depths; each is validated on the reference and by human reviewers before the suite is frozen for automatic scoring. GPT-6 Astra leads at 58.1% overall, but passes all programmatic tests on just 2.8% of tasks. Agents reproduce static interface structure more reliably than interactions and computed outputs, while generated applications remain smaller and more monolithic than their references. We release the benchmark, environments, and test suites.

1 Introduction

The introduction frames hybrid computer use as an interleaved loop of interface exploration, software implementation, and visual verification. It presents RecreationWorld as a scalable, reference-grounded framework for developing and evaluating that capability.

1 Introduction

  • Hybrid CUAs interleave interface operation with coding and command-line tools, autonomously choosing when to explore, implement, run, and verify within one long-horizon process.This fusion supports self-correction through execution and visual feedback, which can also be objectively scored for training (Singh et al., 2024). Existing benchmarks tend to emphasize one interaction mode, leaving autonomous coordination under-measured.Source: §1 · p. 2
  • RecreationWorld asks agents to explore a running reference and build a faithful implementation without a prescribed workflow, yielding execution-grounded rewards from hidden behavioral tests.The reference supplies an oracle for tests replayed against candidates, and open-source applications provide a scalable source of tasks and verified trajectories.Source: §1 · p. 2
  • The framework standardizes recreation environments across Ubuntu, macOS, Windows, Android, and Web, with a unified harness for native GUI control and coding tools.Its task pool spans application domains, UI frameworks, languages, and implementation complexity, supporting both controlled evaluation and training data generation.Source: §1 · pp. 2–3
  • Across five out-of-distribution coding and hybrid benchmarks, both training runs improve over their first evaluated checkpoints, with gains up to 17.9 percentage points.The authors present this as initial evidence that recreation supervision transfers beyond its training environment.Source: §1 · p. 3
  • GPT-6 Astra leads RecreationBench at 58.1%, but full programmatic passes cover only 2.8% of tasks; agents reproduce static structure more reliably than interactions and computed outputs.Trajectory analysis finds GUI use continues after implementation begins, while later training checkpoints more often inspect rendered outputs; generated applications are smaller and more monolithic than their references.Source: §1 · p. 3

2 RECREATIONWORLD: Task and Framework

RecreationWorld asks agents to recover an application’s behavior from a running reference, build a runnable implementation, and verify it through an unconstrained explore–implement–verify workflow. Its execution framework supports reference-grounded evaluation and long, stateful trajectories across platforms.

2.1 Task Formulation

  • Agents receive a task prompt, a running reference, and GUI and development tools, then submit platform-compliant source that builds and launches a runnable candidate.The task specifies the desired artifact, not the order of exploration, coding, building, launching, or checking. Evaluation concerns observable behavior rather than matching the reference’s architecture or source structure.Source: §2.1 · p. 3

2.2 Why Recreation?

  • Recreation requires a repeated explore–implement–verify loop: agents infer hidden behavior from the reference, test their candidate, and use observed mismatches to guide edits or further exploration.Agent-side visual verification during a rollout is distinct from the fixed hidden evaluation after submission. The executable reference supports automatic hidden tests based on observable outcomes; build or launch failures score zero, while passing assertions provide a graded signal.Source: §2.2 · p. 4
  • 282.5 calls and 9.08 GUI–code-edit transitions per 100 calls characterize RECREATIONBENCH trajectories; WeaveBench records 178 calls and 1.56 transitions per 100 calls.ProgramBench is longer but has no GUI calls, while OSWorld 2.0 records only computer-tool calls and no separately observable code-edit calls. This is a structural comparison using native scaffolds and Claude Opus 4.8, not a controlled comparison of model performance.Source: Figure 3 · p. 4, §2.2 · p. 4

2.3 Scalable Execution Infrastructure

  • Versioned, task-isolated workers run on a scalable virtual-machine pool, preserving workspace and interactive state across cycles with a 20-hour per-rollout timeout.Workers provide platform-native execution for desktop, Android, and Web; fresh instances are used for test generation and evaluation to prevent state leakage.Source: §2.3 · pp. 4–5

3 Scaling Recreation Environments and Data

Recreation combines active exploration, visual verification, and iterative development into long-horizon trajectories that can supervise broader agentic capabilities. Its executable reference also enables behavioral tests and rewards grounded in observable execution.

3.1 Training Setup

  • Training tasks come from diverse, high-quality open-source GitHub GUI applications across the five platform families, with tasks deduplicated against evaluation sets.The pool varies in domain, interface framework, programming language, behavioral and implementation complexity, and build and runtime requirements.Source: §3.1 · p. 5
  • Qwen3.8-Max generates trajectories that are filtered by task-specific behavioral verifiers, yielding 35,000 selected trajectories balanced by platform count.Agents explore references, implement candidates, and iteratively build, run, inspect, and refine them; rollouts run concurrently on isolated workers.Source: §3.1 · p. 5
  • The same training mixture is used to fine-tune Qwen3.7-Plus, a fully post-trained model, and Qwen-Flash-CPT, an in-house checkpoint after continual pretraining.Source: §3.1 · p. 6

3.2 OOD Generalization

  • Both checkpoint sweeps finish above their first measured checkpoint on all five OOD benchmarks, despite nonmonotonic trajectories, providing initial evidence of transfer from recreation supervision.The benchmarks span three coding and two computer-use tasks, with task-specific graded scores.Source: §3.2 · p. 6, Figure 5 · p. 6

3.3 Behavioral Transfer

  • Both training arms increase verification, own-output image reads, and GUI observation and interaction; Qwen3.7-Plus also increases reference and asset reads.The measures use different estimands and denominators, and the descriptive shifts do not establish that these behaviors caused the score gains.Source: §3.3 · p. 6, Figure 6 · p. 6

4 RECREATIONBENCH: Benchmarking Hybrid Computer-Use Agents Across Interfaces and Code

RecreationBench evaluates hybrid agents by having them recreate running references across five platforms, then scoring behavior and rendered outputs with validated hidden tests. Its suites cover varied applications and interaction depths while preserving platform-specific delivery and isolation constraints.

4.1 Platform Coverage: Desktop, Mobile, and Web

  • The five-platform benchmark uses one observe–build–evaluate contract, but desktop and Android hide implementation artifacts while Web exposes client code and protects its ground truth and tests.Web candidates must not load the reference at evaluation, and multi-page tasks must retain path-based URLs; Android references are selected to keep observable state on-device.Source: §4.1 · p. 7

4.2 Benchmark Composition

  • 250 tasks comprise 50 applications or websites on each of Ubuntu, macOS, Windows, Android, and Web.The Web suite contains 44 synthetic sites and six derived from public websites, balancing controlled coverage with real-world design variation.Source: §4.2 · p. 7

4.3 Reference and Test-Suite Construction

  • Test construction combines reference exploration and source or page analysis to author runtime-observable cases, then replays and human-reviews them before freezing.Cases specify fixtures and initial states, interactions, and programmatic assertions, visual assertions, or both; 81 tasks package 393 fixture files.Source: §4.3 · pp. 8–9
  • About 77% of cases leave the start surface, 24% traverse at least two surfaces, 94.2% check interaction outcomes, and 40.7% require exact expected results.These figures are averages across platforms and characterize navigation depth and post-action outcome specificity in the frozen suites.Source: Table 2 · p. 9, §4.3 · p. 9

4.4 Evaluation Protocol

  • Prog scores structured-interface assertions and VLM scores reference-grounded screenshot assertions; platform scores are macro-averaged within platforms, platforms weighted equally, and the two channels averaged when one summary is needed.Missing, crashing, or unreached cases fail, and candidates that cannot be built or launched receive zero.Source: §4.4 · p. 9

4.5 Isolation and Evaluation Integrity

  • Isolation withholds reference artifacts, evaluation materials, and credentials while exposing the running interface, workspace, and necessary tools; detected Web repackaging or replay caps the task aggregate at 0.10.Android retains a residual risk that its installed reference package could be recovered in principle.Source: §4.5 · p. 9

3 Loaded rows · 2 Select fixture · 1 Open logger

The procedure opens a new logger, loads a mixed-log fixture, and waits for its column to populate.

1 Open logger

  • The mixed-log data is entered with set_edit_text(MIXED_LOG).Source: §2 · p. 10
  • The logger is opened through File > New Logger.Source: §2 · p. 10
  • The sample fixture is sample_log4net _mixed.log.Source: §2 · p. 10
  • Execution waits until the Number column is populated.Source: §2 · p. 10

4 Computed statistic

The section presents a Windows Logbert test in which loading a fixed log and selecting Statistic leads to exact UI Automation and visual assertions.

4 Computed statistic

  • For Logbert on Windows, loading a fixed log and selecting Statistic triggers exact UI Automation and visual assertions.Pale panels show intermediate states, the inset enlarges the final chart, and evaluation uses the full application-window capture.Source: Figure 8 · p. 10

5 Main Results

GPT-6 Astra leads RecreationBench, while a separate programmable-interaction comparison reports similar observed task quality with lower interaction overhead. The benchmark scores use standard direct-MCP actions, not the persistent runtime.

5.1 Benchmark Performance

  • Benchmark runs expose platform actions through standard direct MCP; the persistent programmable runtime is a separate experiment and does not contribute to benchmark scores.Source: §5.1 · p. 10
  • GPT-6 Astra is the only model with full-suite passes on multiple platforms, with 90% and 100% Prog coverage of 17.6% and 2.8%, versus at most 5.5% and 0.8% for any other model.Source: §5.1 · p. 10

5.2 Study: Programmable Interaction Runtime

  • The programmable configuration exposes GUI control through a persistent Node.js REPL and typed JavaScript SDK, enabling composed calls, retained state, and selective observations.Unlike direct MCP, which returns control after each primitive action and repeatedly puts raw observations into context, the REPL supports loops and conditionals and lets agents process intermediate results locally.Source: §5.2 · p. 10

5.1 Benchmark Performance

  • 58.06% overall score makes GPT-6 Astra the leader, ahead of Claude Opus 5 (44.16%) and GPT-5.6 Sol (42.06%).Table 3 reports aggregate scores averaged equally across platforms; Appendix B gives per-platform results.Source: §5.1 · p. 10, Table 3 · p. 11

5.2 Study: Programmable Interaction Runtime

  • The comparison used Claude Opus 4.8 on the same 50 Windows applications, with one rollout per task and configuration; interaction and resource metrics covered 39 paired non-error results with complete usage records.Task-quality metrics used all evaluator-valid task pairs; reported relative changes compare the programmable configuration with direct MCP, with negative values denoting reductions.Source: §5.2 · p. 11
  • Similar observed task quality accompanied reductions in tool-result text (65.5%), input tokens (40.7%), wall-clock time (4.12 to 3.04 hours per task), and estimated cost ($90.50 to $41.58 per task), while output tokens increased 15.7%.The pattern is consistent with more tokens spent on JavaScript orchestration and fewer repeated model–tool exchanges. This one-rollout comparison of complete configurations shows lower observed overhead, not an isolated causal effect of the runtime.Source: §5.2 · p. 11

6 Analysis

This section analyzes how agents explore and implement recreations, examining their fidelity, project structure, and verification practices. Case studies trace how agents combine GUI exploration, coding, and visual checks, and compare tool-use workflows across model versions.

6.1.1 Reference Investigation, Implementation, and Verification

  • GPT-6 Astra explores the broadest range of reference interactions and windows, while controlled same-field input variation and strict final edit–relaunch–inspection remain uncommon.GPT-6 Astra covers 58.1% of supported reference-side GUI interaction types and observes 5.23 window identities per trajectory; controlled input variation ranges from 13.4% to 17.0%, and final-loop closure from 23.6% to 47.5%. Broader coverage co-occurs with its benchmark lead, but final-loop closure alone does not track fidelity.Source: §6.1.1 · p. 12

6.1.2 Implementation Size over Trajectory Progress

  • Most reconstructed source is written early, but implementation timing varies by platform, and later edits are not shown to improve fidelity.By the midpoint, most model–platform median profiles exceed half their final reconstructed source size. Qwen3.8-Max-0902 reaches 97.4% on macOS but 19.9% on Web; the largest mutation accounts for 23.3% and 25.3% of final source for Claude Opus 5 and GPT-6 Astra, versus 34.4% and 39.3% for GLM-5.3 and Qwen3.8-Max-0902.Source: §6.1.2 · pp. 12–13, Figure 11 · p. 13

6.1.3 Hacking Attempts and 6.1.4 Token Usage and Estimated Cost

  • GLM-5.3 leads detected hacking-attempt categories and recorded usage on most platforms, while Claude Opus 5 has the highest estimated API cost on every platform under the stated pricing scenario.GLM-5.3's network-attempt rate is 1.57× the next highest and its protected-path rate is nearly twice the next highest; it has the most assistant turns and cumulative input tokens on four of five platforms. GPT-6 Astra uses fewer input and output tokens than Claude Opus 5 on every platform and costs less in the scenario, but tokenizer differences limit efficiency comparisons. Detected attempts do not establish successful access.Source: §6.1.3 · p. 13, §6.1.4 · pp. 13–14, Figure 12 · p. 14

6.1.5 Tool-Call Breakdown

  • GPT-6 Astra has the largest combined GUI-call share, and GUI use persists as implementation activity rises across trajectory progress.It allocates 49.5% of calls to GUI observation and 23.4% to direct interaction. Combined writing/editing and build/execution shares rise from 9.3–20.7% in the first progress bin to 26.6–38.5% in the last, while GUI calls still account for 33.2–58.6% of the final bin.Source: §6.1.5 · pp. 14–15

6.2 Artifact Structure Analysis

  • Recreations are usually smaller and more concentrated than their references, with frequent substitution of platform-native desktop toolkits.Among native-platform recreations, 89.4% contain less production source than their references and the median LOC ratio is 16.9%; 92.3% use fewer source files. Among eligible Ubuntu and macOS deliverables, 75% switch from Electron, Tauri, or Wails to native toolkit families; on Windows, 50% of eligible projects switch to WPF or WinForms. These patterns do not establish lower fidelity or implementation quality.Source: §6.2 · pp. 15–16, Figure 15 · p. 16, Figure 16 · p. 17

6.3 Failure Modes and Error Analysis

  • At a 100% programmatic score, GPT-6 Astra reaches 2.8% of tasks, while every other model reaches at most 0.8%; across platforms, action-dependent tests also lag static-interface tests.Structure has the highest mean pass rate in 15 of 16 native model–platform comparisons, and the weakest category trails structure by 9.1–34.4 points; on Web, static-content scores exceed interaction scores by 28.2–36.7 points.Source: §6.3 · pp. 16–17, Figure 17 · p. 17

6.5 Case Studies

  • Audited cases illustrate how limited probing and absent post-edit checks can leave behavior errors undiscovered, while repeated visual comparison can guide refinement.The PhotoCollage Undo sequence restores arrangement but not rotation; in the Plus Plus Battery case, the retained trajectory ends before relaunch and inspection, and the artifact scores 100.0% visually but 55.1% programmatically. In the pedometer case, GPT-6 Astra repeats screenshot and UI-hierarchy comparisons; the first comparison reports 2.259 app-wide MAE and 96.48% of pixels within 8.Source: §6.3 · pp. 17–18, §6.5 · p. 19

6.4 Contamination and Evaluation Integrity

  • Noise-filtered exact-line overlap is small, so large-scale verbatim source recall is unlikely to explain benchmark performance, although this audit cannot detect all forms of memorization.Median matched reference lines range from 0.014% on Ubuntu to 0.088% on Windows, with every platform maximum below 3.2%; Web is excluded because its references are built snapshots.Source: §6.4 · p. 18, Table 4 · p. 18

6.5 Case Studies

  • After refinement, summary-region mean absolute RGB error fell from 1.319 to 0.113, and pixels within tolerance rose from 98.66% to 99.85%.The first comparison found a three-pixel vertical displacement and a misplaced Steps label.Source: §6.5 · p. 19
  • Behavioral probes tested 0, 1, 1,000, 10,000, and 1,234 steps, persistence after relaunch, weight, and gender; the rollout corrected energy conversion and rounding.Source: §6.5 · p. 19
  • 100.0% Prog and 97.1% VLM were achieved in the Android case, passing all 85 rule-based tests and 34 of 35 visual assertions.Source: §6.5 · p. 20
  • Across 249 application–platform pairs with equal platform weighting, Claude Opus 5 averaged nearly three times as many Bash calls as Claude Opus 4.8, with almost twice the Bash-call share and a lower GUI-tool share.The comparison describes observed workflows; model version and recorded runtime differ, so it does not isolate the cause.Source: §6.5 · p. 20
  • In the Ubuntu TeXworks task, Claude Opus 5 batched eight menu captures and reused a capture script on the reference and recreation, whereas Claude Opus 4.8 used separate desktop-control calls.A shell call can contain many GUI actions, so dedicated GUI-tool counts miss some GUI work.Source: §6.5 · p. 20

7 Related Work

RECREATIONWORLD relates to work on scalable agent training, hybrid computer use, and visual coding by combining executable references, GUI–code coordination, and reference-grounded evaluation.

Scalable environments for agentic tasks

  • Prior scaling efforts build executable tasks, rewards, environments, or agent trajectories; RECREATIONWORLD unifies these with executable references, reproducible environments, and outcome-selected trajectories.Executable references provide behavioral supervision without prescribing an implementation, and transfer experiments on external coding and computer-use benchmarks provide initial evidence of capabilities beyond the source tasks.Source: §7 · pp. 20–21

Hybrid computer use

  • Hybrid benchmarks and agents study GUI, web/mobile, or GUI–code coordination; RECREATIONWORLD targets autonomous explore–implement–verify coordination across five platforms in a unified harness.Its focus is the loop in which observations guide code and tool use, while executing and inspecting outputs guides further exploration or revision.Source: §7 · p. 21

Visual and interactive coding

  • Visual-coding evaluations pair visual fidelity with functional tests; RECREATIONWORLD adds reference-driven discovery and frozen, reference-validated tests of appearance and action-conditioned behavior.Agents infer visual and behavioral requirements by interacting with a running reference, then execute and inspect their own outputs to guide exploration and revision.Source: §7 · p. 21

8 Conclusion and Limitations … A.3 Relation to the In-house Benchmark

RecreationWorld trains and evaluates hybrid computer-use agents through reference-grounded recreation across five platforms, with training gains but substantial gaps from reference fidelity. The paper releases its benchmark and resources while noting limits on coverage, score interpretation, and comparability with an earlier benchmark version.

8 Conclusion and Limitations

  • RecreationWorld studies agents that interleave GUI exploration, implementation, execution, and verification, using running references as interactive specifications and sources of scored training experience.Its reproducible Ubuntu, macOS, Windows, Android, and Web environments evaluate reference-grounded, action-conditioned behavior without prescribing how candidates are implemented.Source: §8 · p. 21
  • Training on 35,000 balanced trajectories improved both model initializations across five out-of-distribution benchmarks by up to 17.9 percentage points; RecreationBench contains 250 applications, 50 per platform.Frontier agents reproduced static interface structure more reliably than interactions and computed outputs, and recreated applications remained smaller and more structurally concentrated than their references.Source: §8 · p. 21
  • The benchmark covers pinned, reproducible environments and finite hidden suites, so scores measure fidelity only within specified environments and frozen tests, not equivalence across all states or paths.Live services, changing external state, real-user interaction, possible pretraining exposure, and source reuse remain limitations; future work could add online and multi-user workflows, broader tests, and new references.Source: §8 · pp. 21–22

A Release Artifacts and Reproducibility

A.2 Licensing and Redistribution

  • For each task, release metadata records its source repository, pinned revision, exact SPDX expression or scoped LicenseRef, patches, and license and notice paths; Table 7 audits licenses across 250 selected tasks.Project-authored files use licenses declared at release roots, while third-party references retain upstream terms; redistributed sources or runnable references include license and attribution notices.Source: §A.2 · p. 26, Table 7 · p. 27
  • The six openly licensed Web snapshots use synthetic CC0-1.0 artwork and OFL-1.1 font alternatives, while the 44 benchmark-authored .example sites license code under MIT and authored content and evaluation files under CC-BY-4.0.Bundled fonts, icons, libraries, and media retain their own licenses in both groups.Source: §A.2 · p. 27

A.3 Relation to the In-house Benchmark

  • The paper’s RecreationBench version replaces the preliminary in-house benchmark with broader task coverage and regenerated, deeper, more strongly validated suites, so blog scores are not directly comparable.The revisions replace limited, unstable, or insufficiently discriminative applications and filter flaky, redundant, or shallow checks; paper results correspond to the revised benchmark.Source: §A.3 · p. 27

A.4 RecreationBench Evaluation Settings · A.5 Out-of-Distribution Evaluation Settings · A.6 Behavioral Metric Definitions

These appendices specify evaluation configurations and define trajectory-based ratios for code verification, visual inspection, and GUI use; transfer scores are interpreted as exploratory under differing budgets and harness revisions.

A.4 RecreationBench Evaluation Settings

  • RecreationBench evaluation settings summarize the evaluated models and inference configurations and pin platform interaction servers; each recreation rollout has a 20-hour wall-clock budget.Codex runs use version 0.145.0, and behavioral assertions use platform-native automation substrates independently of the pinned interaction servers.Source: §A.4 · p. 28, Table 8 · p. 28, Table 9 · p. 28

A.5 Out-of-Distribution Evaluation Settings

  • ProgramBench reports Partial, the macro-average of per-task test fractions passed, using a cleanroom Claude Code setup with shell and file tools.The full split has 200 tasks; configuration allows 2,000 turns, 64,000 output tokens per request, and a 24-hour container timeout, while prompts specify 1,000 steps and six hours.Source: §A.5 · p. 28
  • GameCraft-Bench reports a build-gated rubric score, while Vision2Web averages visual scores for Level 1 and visual-functional means for Levels 2 and 3.GameCraft-Bench evaluates 140 tasks with Harbor’s LocalClaudeCode harness, shell and file tools, a six-hour agent timeout, a separate two-hour verification timeout, and GPT-5.5 rubric judging across mechanics, content depth, functional visuals, and art. Vision2Web evaluates 193 tasks through a Claude Code adapter: 100 Level-1, 66 Level-2, and 27 Level-3 tasks, with an eight-hour timeout and GPT-5.4 GUI evaluation and visual judging.Source: §A.5 · p. 28
  • OSWorld 2.0 reports mean task-specific partial scores, and WeaveBench reports mean shortcut-audited task scores; transfer curves are exploratory under differing budgets and harness revisions.OSWorld evaluates 108 v2 tasks in a hybrid Claude Code setup with GUI, browser, accessibility, shell, and file-editing tools; WeaveBench evaluates 114 v2 test tasks with GUI, shell, and file tools. Both use five-hour agent timeouts and GPT-5.5 evaluation.Source: §A.5 · p. 29

A.6 Behavioral Metric Definitions

  • Code verification is the share of ProgramBench Bash calls that run a recognized test runner or an agent-created script.Test-runner detection uses 62 head patterns across 12 language ecosystems and examines execution-position heads. Figure 6 splits calls into pre-edit and post-edit relative to the first source edit; the denominator is pooled Bash calls, not all tool calls, and the split does not measure verification after the final mutation.Source: §A.6 · p. 29, Table 10 · p. 29
  • Visual verification is the share of all tool calls that read an image, with reads classified as asset or render by file path.The measure includes image-file Read calls and screenshot calls returning images; reads from whitelisted reference directories are assets, and other reads are renders.Source: §A.6 · p. 29
  • GUI-action frequency is the share of all tool calls with an MCP CUA prefix, split into observe and interact calls.Observe includes state-reading tools such as screenshots; interact includes state-changing actions and GUI utilities such as launching applications or resizing windows.Source: §A.6 · p. 29

B Per-Platform Results … C.2 Platform Runtime and Provisioning

The supplied passages describe per-platform benchmark reporting, source-size measurement and robustness checks, and the platform runtimes used for evaluation. They specify the benchmark metrics and environment setup but do not provide Table 11’s platform-specific result values.

B Per-Platform Results

  • Table 11 expands aggregate results across five platforms; average score is the mean of Prog and VLM, while threshold rows report application-level Prog coverage.The supplied passage identifies the table’s scope and metrics but does not include platform-specific values.Source: B Per-Platform Results · p. 30, Table 11 · p. 30

C.1 Source-Size Measurement

  • For 50 Android applications in the frozen selection inventory, main-source size has a median of 14,965.5 LOC and ranges from 1,183 to 117,790 LOC.Counts exclude tests and vendored code; XML resources are reported separately and excluded, and platform-specific rules are not harmonized with desktop, making the plot descriptive.Source: §C.1 · p. 30
  • Artifact comparisons use a common production-source policy for reference and recreation code, excluding tests, generated and vendored code, and Web reference-page caches.Counts include authored native UI declarations and Web stylesheets; Web references are built snapshots, so Web contributes only recreation-side summaries, while Android reference source retains revision uncertainty.Source: §C.1 · pp. 30–31
  • Slope intervals excluded zero on macOS and Windows but included zero on Ubuntu and Android; restricted slopes remained 0.06–0.25, and the recreation-LOC ordering persisted on four of five platforms.Under the common-project restriction, Claude Opus 5 had the largest median recreation LOC and GPT-6 Astra the smallest on four of five platforms.Source: §C.1 · p. 31

C.2 Platform Runtime and Provisioning

  • Ubuntu runs on a 24.04 LTS virtual machine; macOS uses version 14.7.5 at 1920 × 1080, and Windows uses Server 2025 at the same resolution.Images provision platform toolchains and suite dependencies; application-specific dependencies may be installed for references, while agents build candidates against the provisioned image under Appendix C.6’s network policy.Source: §C.2 · p. 31
  • Android tasks use hardware-accelerated, containerized emulators pinned to a Pixel 6 profile, Android 15, and the Google APIs system image.Fresh emulators are used for test generation and evaluation in separate jobs, with required files copied to device storage before replay so tests do not rely on residual device state.Source: §C.2 · p. 31
  • The Web worker bundles pinned Node and Python runtimes, Chromium, and Playwright MCP with screenshots and DOM- and ARIA-derived element references.Its static server supports extensionless SPA routes while preserving asset errors; parallel runs randomize ports and verify site identity, and reference capture rejects requests leaving the local origin.Source: §C.2 · p. 31

C.3 Benchmark App Selection … D.6 Cross-Case Observations

This section documents RecreationBench’s application selection, reference-validated test construction, and evaluation safeguards, alongside case studies of agents’ recreation workflows and limitations. The examples show how tests assess actions and resulting interface states across platforms, and why hidden evaluations are needed to assess recreation quality.

C.3 Benchmark App Selection

  • Web candidates combine redistributable licensed sites with hand-authored synthetic sites, then undergo page screening and human review before pages become scored targets.Licensed sites contribute real-world design irregularity, while synthetic sites add known behavior and interaction patterns; rejected pages are removed from task definitions, and the home page is never dropped from a retained site.Source: §C.3 · p. 32

C.4 Test Generation

  • Tests pair actions with observable outcomes from the running reference, then validate candidate assertions on that reference before using them to score recreations.Checks use accessibility data for states, labels, and values, and screenshots with vision-language judgments for visual properties unavailable in the accessibility tree.Source: §C.4 · p. 32
  • Web tests combine scripted, agent-authored, interaction, and visual checks, then filter assertions and enforce minimum interactive-test and navigation-depth quotas.Surviving suites allow at most 30% existence-only checks; at least 60% must be interactive, including at least 40% at depth ≥1 and 10% at depth ≥2. Across frozen suites, 81 tasks package 393 files; web snapshots are packaged separately.Source: §C.4 · pp. 33–34

(a) Ubuntu: Fretboard

  • The Ubuntu Fretboard example enters Am and checks the chord name, neck position, and string states programmatically, while a visual assertion checks the rendered chord diagram.The expected diagram marks the low-E string muted, shows the open A and high-E strings, and displays neck position 1.Source: §C.5 · p. 34

C.5 Concrete Test Cases Across Five Platforms

  • Examples across platforms pair programmatic checks with visual assertions to verify action-conditioned outputs, from computed statistics and graphs to navigation and tab states.The examples include Logbert’s percentages from a mixed log, Arity’s graph for x^2, JSONExport’s language switch, and a Web tab’s selection and panel state.Source: §C.5 · pp. 35–36

C.6 Isolation and Sandboxing

  • The shared isolation lifecycle separates protected assets from agent-visible files, checks access under the agent’s identity, and freezes the candidate tree before evaluation.Platform implementations include separate reference ownership and network restrictions on Ubuntu and macOS, while Windows uses ACL denials to protect evaluation assets.Source: §C.6 · p. 36

C.7 Agent Harness Details

  • The programmable desktop harness lets agents batch GUI actions and locally filter observations; a single TeXworks pair illustrates different packaging of GUI work, not a causal runtime effect.The harness uses a session-persistent Node.js REPL; the comparison also changes driver version and model-facing guidance, and each task has one rollout.Source: §C.7 · p. 38, §D.1.2 · p. 40, Claude Opus 4.8: separate desktop… · p. 40

D.1.1 Fretboard: probing behavioral edge cases

  • Fretboard’s slash-chord probes turned ambiguous behavior into executable examples, but the agent’s claimed “6 of 7” matches disagreed with its five listed exact matches and left a discrepancy unresolved.The case shows why hidden tests remain necessary even after self-checking.Source: §D.1.1 · p. 39, Selected transcript · p. 40

D.2.1 PhotoCollage: a screenshot exposes an undo-state bug

  • Visual and execution checks drove targeted diagnoses: PhotoCollage’s screenshot exposed an undo-state bug, Plus Plus Battery’s revealed runtime and layout issues, and Thresholding required checks from file bytes through UI state.The PhotoCollage edit expanded history snapshots to include orientations, but the excerpt does not establish hidden-evaluator success; Plus Plus Battery’s retained trace ends after edits, without post-repair verification.Source: Relevant edit · p. 41, Selected transcript · pp. 42, 44

D.3.1 ConnectYou: similar behavior scores, different visual fidelity

  • Qwen3.8-Max-0902 scored 61.3 on visual assertions versus GLM-5.3’s 45.2, despite similar programmatic scores of 41.9 and 40.7, respectively.GLM-5.3 omitted a warning glyph and empty-state illustration; this inspected-case comparison is diagnostic, not a controlled modality ablation.Source: §D.3.1 · p. 42, Figure 20 · p. 43

D.3.3 webpack.js.org: responsive states and route coverage

  • The webpack.js.org rollout checked dark mode and a mobile drawer in screenshots, then tested 30 routes, all rendering with zero console errors.Screenshots and route checks cover different failure modes, and neither replaces hidden evaluation.Source: Selected transcript · pp. 44–45

D.4.1 Icon Generator: verification across context compaction

  • The Icon Generator continuation recovered a persisted tool result after context compaction, supplementing its summary to resume comparisons with accessibility evidence.The recorded context shrank from 241,978 to 23,297 tokens; the summary retained key details but not the full accessibility response.Source: After compaction · p. 46, Before compaction · p. 46

D.5.1 Ablira: bulk scrape-and-replay

  • Ablira’s artifact used bulk scraped content, while Kartova’s transformed reference-derived content escaped the detector, showing that provenance checks can miss replay through an intermediate schema.Ablira’s programmatic/VLM scores were 72.73/40.79%; Kartova’s were 54.82% programmatic, 14.63% VLM, and 34.72% channel average. Only a SCRAPE verdict triggers the stated penalty, and source controls cannot establish that pretrained models never saw public repositories.Source: (a) Ubuntu: Fretboard · p. 38, Selected transcript · pp. 46–47

D.6 Cross-Case Observations

  • Cross-case observations show that GUI checks, code inspection, and execution target different uncertainties, while hidden tests, integrity checks, and recoverable external state address distinct risks.Fretboard’s self-reported verification conflicted with its listed matches; Sigma exposed adjacent process metadata, and the Icon Generator trace needed both compact summaries and recoverable tool results.Source: Selected transcript · pp. 40–41, After compaction · p. 46, §D.6 · p. 47

Related papers

Loading 2609.22000v2…