PaperlayerView exact revision on arXiv

Source-linked AI summary

CUA-Sandbox: Efficient Environments for Computer-Use Agent Reinforcement Learning

Xin Yan, Zhengbo Jiao, Jiaqi Liu, Zhenglin Wan, SiYuan Ma, Xuliang Yu, Tianyi Jiang, Chubin Zhang, Pengfei Zhou, Wangbo Zhao, Xingrui Yu, Bo An, Yang You, Ivor Tsang

arXiv:2609.32750v1cs.AI

TL;DR

Conventional computer-use rollouts replicate initialized runtimes, adding memory and initialization costs as parallel environments grow. CUA-Sandbox separates private state capsules from shared runtimes and coordinates resets and branches transactionally. It reports comparable or improved task success relative to Docker, including up to 6.20x higher rollout throughput.

  • Problem

    Conventional deployments replicate an initialized runtime for each independent rollout, incurring repeated memory and initialization costs as parallel environments grow.

    Source: §1 · p. 2

  • Method

    CUA-Sandbox gives each environment a private state capsule and bindings while reusing initialized application runtimes through state-scoped execution and transactional lifecycle operations.

    Source: §1 · p. 2

  • Results

    Across web and desktop evaluations, task success is comparable or improved relative to Docker, with up to a 6.20x increase in rollout throughput.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support runtime reuse as a practical systems component for agent training, with measured resource and lifecycle savings and maintained task performance.

    Source: §5 · p. 10

    Isolation assumes complete resource bindings, disjoint writable state, and a consistent generation throughout each operation.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Reinforcement learning enables computer-use agents to improve through interaction with real software environments, including websites and desktop applications. However, conventional deployments replicate an initialized runtime for each independent rollout, even when trajectories use the same software, incurring repeated memory and initialization costs as the number of parallel environments grows. Does an independent computer-use environment require an independent execution runtime? Our key observation is that trajectories require independent mutable state, while initialized application runtimes can be reused across concurrently evolving environments, making state the natural unit of environment independence. Guided by this observation, we introduce CUA-Sandbox, which separates private state capsules from shared runtimes through state-scoped execution and transactional lifecycle operations, including resets and branches, while retaining the original software interfaces and task evaluators. Experiments show comparable or improved task success relative to Docker, while substantially reducing rollout and resource costs. CUA-Sandbox achieves up to a 6.20x increase in rollout throughput, a 9.2x reduction in per-environment memory, and a 504x reduction in incremental storage.

1 Introduction

CUA training needs independent mutable state for parallel trajectories, but conventional per-environment runtimes repeatedly initialize the same software and raise memory and lifecycle costs. CUA-Sandbox reuses initialized runtimes with private state capsules, and experiments report comparable or improved task success alongside lower rollout and resource costs.

1 Introduction

  • Parallel trajectories need independent mutable environments, while conventional deployments instantiate a complete container, virtual machine, or application stack for each environment.Repeated initialization makes memory and lifecycle costs grow with environment count, limiting experience under fixed resources.Source: §1 · p. 2
  • The evaluation combines paired inference, matched post-training, rollout and operation measurements, and component ablations to examine task fidelity and resource costs.The studies compare task success and training benefits while distinguishing savings and lifecycle effects.Source: §1 · p. 2
  • CUA-Sandbox separates environment-specific state and bindings from reusable application runtimes, using state capsules, state-scoped execution, and transactional lifecycles across web and desktop tasks.It retains the original interaction and evaluation interfaces.Source: §1 · p. 2
  • Up to 6.20x increase in rollout throughput, 9.2x reduction in per-environment memory, and 504x reduction in incremental storage accompany comparable or improved task success relative to Docker.The evaluations cover web and desktop environments.Source: §1 · p. 2

2 Related Work

Prior work has expanded computer-use tasks, environments, and execution infrastructure, while tackling authoring, simulation, or provisioning costs. The remaining challenge is concurrent execution of independently diverging mutable states without a separate runtime for each environment.

2 Related Work

  • Computer-use benchmarks and agents span web, visual web, desktop, mobile, and long-horizon workflows, increasing the infrastructure burden of many complete environment instances.The related work traces this expansion from web interaction to broader platforms and workflows.Source: §2 · p. 3
  • Task-generation and learned-world approaches reduce environment authoring or rollout costs, but simulation and execution-free feedback omit application dynamics, hidden state, or real side effects.The passage frames high-fidelity agent learning as still dependent on costly stateful software execution at scale.Source: §2 · p. 3
  • Agent-execution systems reduce provisioning, reuse snapshots or sandboxes, and improve resource control, but still allocate a separate runtime to each concurrent environment.Prior runtime sharing generally assumes stateless or sequential execution, leaving concurrent mutable-state divergence open.Source: §2 · p. 4

3 CUA-Sandbox

CUA-Sandbox makes mutable state and its bindings environment-specific while reusing initialized application processes. Capsules hold trajectory state, bindings route execution to it, and lifecycle operations replace or branch it without rebuilding the runtime.

3.1 Overview

  • The overview replaces per-environment initialized processes with a shared runtime and environment-specific capsules and bindings, while keeping task setup, interfaces, and evaluation unchanged.Environment actions and reset or branching operations are directed to the corresponding capsule.Source: §3.1 · p. 4

3.2 Environment State Capsules

  • A state capsule gathers mutable resources that determine future observations and rewards into an independently managed environment state.Task resource contracts identify required databases, files, profiles, service state, and interface resources; state is organized as authoritative, derived, and ephemeral components.Source: §3.2 · pp. 4–5

3.3 State-Scoped Execution

  • Each action carries an environment identity and binding table so shared processes access that environment’s private branches, namespaces, and endpoints.Reads use composed views while writes terminate in private branches; isolation depends on complete resource bindings, disjoint writable state, and consistent generations.Source: §3.3 · p. 5

3.4 Lifecycle and Runtime Reuse

  • Reset, snapshot, clone, and fork act on capsule backends while preserving initialized processes; transitions freeze routes, drain work, stage successors, and atomically publish ready generations.Failed transitions retain the previously published generation; physical execution slots can be rebound among logical environments.Source: §3.4 · pp. 5–6

4 Experiments

The experiments compare CUA-Sandbox with Docker under matched conditions, assessing task performance, rollout efficiency, resource and lifecycle costs, and component ablations.

4.1 Experimental Setup

  • The setup matches Docker comparisons across benchmarks and controls model, prompts, evaluators, and training budgets; rollout measurements fix concurrency at eight environments.Task success and its unweighted mean are reported separately by benchmark.Source: §4.1 · p. 7

4.2 Agent Performance

  • Across twelve model–benchmark pairs, CUA-Sandbox scores above Docker; post-training finishes above Docker on WebArena and OSWorld and matches it on VisualWebArena.For OSWorld, the RL-stage gains are 3.88 points with CUA-Sandbox and 3.87 points with Docker.Source: §4.2 · p. 8

4.3 Rollout Efficiency

  • 6.20×: VisualWebArena throughput rises from 0.179 to 1.109 environments/s as batch time falls from 44.762 to 7.215 s.Preparation falls from 10.404 to 0.504 s; Figure 3 reports the eight-environment rollout measurements.Source: Figure 3 · p. 7, §4.3 · p. 8

4.4 Environment Resource and Lifecycle Costs

  • On WebArena, memory falls from 2,102.8 to 463.9 MiB and incremental storage from 6,757.1 to 13.1 MiB across eight concurrent environments.Memory and storage savings peak on different workloads; VisualWebArena Reddit has 22.0× memory and 9,535× storage reductions.Source: §4.3 · p. 8, §4.4 · p. 8
  • 15.17× and 17.84×: VisualWebArena Shopping has these create and fork speedups, while WebArena Shopping improves by 1.15× and 1.25×.WebArena Shopping reset speedup is 1.03×; capsule preparation and readiness checks still take time.Source: §4.4 · p. 9

4.5 Ablation Study

  • 5.476 s and 4.772 s: removing copy-on-write raises create and fork latency from 0.071 s and 0.054 s on the same 3.6 GB capsule.Storage delta rises from 0.11 MiB to 3.67 GiB, identifying state materialization as the principal avoided cost.Source: §4.5 · p. 9, Figure 4 · p. 9
  • 0.844 s and 1.086 s: non-database state management adds this much preparation and reset time on read-only WebArena task 21.All four HTTP readiness probes return status 200; this comparison measures adapter overhead, not isolation under state-changing actions.Source: §4.5 · p. 9
  • Across 100 trials, the full publication protocol has no mixed-generation events or publication during in-flight requests; direct publication has both events in every trial.The full protocol also rejects all 100 requests issued while the route is frozen.Source: §4.5 · p. 9, Figure 4 · p. 9

5 Conclusion

The conclusion presents CUA-Sandbox as a way to separate mutable state from initialized runtimes, retaining interfaces and evaluators while reducing measured costs. It also identifies sharing boundaries and further workloads needed to characterize them.

5 Conclusion

  • Runtime reuse is supported as a practical systems component for agent training, with measured resource and lifecycle savings translating into higher rollout throughput and maintained task performance.Safe sharing depends on complete resource contracts and binding coverage; resources that cannot be safely shared use private backends.Source: §5 · p. 10

AI Use Statement

A Supplementary Ablations and Execution Configurations

The supplementary section introduces controls and interpretation for component ablations, then points to additional execution configurations.

A. Supplementary Ablations and Execution Configurations

  • The ablations remove one mechanism from complete CUA-Sandbox, while the separate Docker comparison evaluates the execution backend as a whole.Appendix A.1 provides controls and measurement details; more experiments are provided in Appendix F.Source: A Supplementary Ablations and… · p. 14

A.1 Component Ablation Details

The appendix details three component ablations and describes additional configurations with distinct measurement conventions.

A.1 Component Ablation Details

  • Without copy-on-write, create, checkpoint, and fork latency rise from 0.071, 0.082, and 0.054 s to 5.476, 4.695, and 4.772 s, while storage grows from 0.11 MiB to 3.67 GiB.Source: §A.1 · p. 14
  • On read-only WebArena task 21, non-database state management adds 0.844 s to preparation and 1.086 s to reset; all four readiness probes return status 200.The comparison measures adapter overhead rather than isolation during state-changing interaction.Source: §A.1 · p. 14
  • Across 100 trials, the full freeze/drain and atomic-publication protocol has no mixed-generation events or publication during in-flight requests; direct publication has both in every trial.The full protocol rejects all 100 requests issued while the route is frozen; that comparison is inapplicable to direct publication.Source: §A.1 · p. 14

A.2 Additional Execution Configurations

  • Table 4 reports absolute launch, storage, and memory costs for WebServ Incus, fresh Docker, and DB/reflink configurations; DB/reflink marginal memory growth is not directly comparable to working-set memory.Launch latency is averaged over eight subsequent runs; storage is per environment.Source: §A.2 · p. 14, Table 4 · p. 14

B.1 Task Success by Site and Category

This appendix reports paired site and category task-success scores, absolute rollout measurements, model settings, and additional post-training details.

B.1 Task Success by Site and Category

  • Table 5 reports paired Docker and CUA-Sandbox success rates by site and category, with benchmark means reported in Table 1.Each pair uses the same model and evaluation protocol; the supplied passages do not provide the table's cell values.Source: Table 5 · p. 15, §B.1 · p. 15

B.2 Absolute Rollout Measurements

  • Table 6 gives the eight-environment rollout measurements underlying Figure 3, with resource costs per environment and throughput and batch time for the concurrent workload.Preparation and batch wall-clock times are in seconds.Source: Table 6 · p. 15, §B.2 · p. 15

C.1 Model Settings

  • Inference settings specify model-specific sampling parameters and output-token limits, including different limits for OSWorld and web tasks.Qwen3.5-9B and Gemma-4-31B-IT use enable_thinking=true; GPT-5.6-Terra uses reasoning_effort=low.Source: §C.1 · p. 16

C.2 More Details

  • Post-training compares independently collected successful teacher trajectories and corresponding SFT and on-policy RL runs under the two backends.The setup uses 5k trajectories, 2 SFT epochs, and 1 RL epoch; the execution environment is the only difference between the runs.Source: §C.2 · p. 16

C.3 Benchmarks and Evaluation Protocols

The evaluation covers state-changing web tasks and desktop workflows, using benchmark-specific observations and official evaluators. Scores average site or category success rates equally, and paired backend comparisons hold key settings constant.

Benchmarks

  • WebArena evaluates functional correctness on 812 tasks across self-hosted applications; this study uses Shopping, CMS, and GitLab state-changing tasks.Its evaluator checks the resulting website state rather than exact action sequences.Source: §C.3 · p. 16
  • VisualWebArena covers 910 visually grounded tasks in Classifieds, Shopping, and Reddit using the standard multimodal interaction protocol.Tasks require image and webpage-text reasoning with multi-step browser actions.Source: §C.3 · p. 16
  • OSWorld evaluates 369 open-ended tasks involving desktop applications, operating-system files, and cross-application workflows across five reported categories.Each task specifies an initial state and an evaluator that checks the resulting system state.Source: §C.3 · p. 16

Evaluation protocol

  • Agents receive benchmark-specific observations and actions, while both backends in each inference pair use the same model, prompts, decoding settings, horizon, interface, and official evaluator.The concurrency limit is also held constant across implementations.Source: §C.3 · pp. 16–17

Scoring and measurements

  • The benchmark score is the unweighted mean of site or category success rates, so groups contribute equally regardless of task count.Unresolved environment, browser, or evaluator errors on evaluated tasks count as failures; measurements are repeated five times unless specified otherwise.Source: §C.3 · p. 17

D Implementation Details

CUA-Sandbox defines resource contracts and capsule generations to bind each environment to private task state while sharing only resources whose accesses can be scoped. Generation-aware execution and lifecycle handling preserve that isolation boundary.

Resource contracts

  • A resource contract maps logical resource keys to state classes, backend adapters, and lifecycle policies, assigning unbindable mutable resources to private backends or coarser private contexts.The contract can cover databases, files, profiles, queues, displays, and other task-relevant resources.Source: D Implementation Details · p. 17

Capsule metadata

  • Capsule metadata coordinates environment, branch, and generation identity with authoritative deltas, private namespaces, ephemeral-resource reconstruction, and component versions.The manifest publishes these elements as one logical generation.Source: D Implementation Details · pp. 17–18

Bindings and isolation

  • State affecting future observations, evaluator-visible results, or external side effects must be represented in the capsule, scoped to the active context, or kept within a private component.Adapters select benchmark-specific resources while retaining the same isolation rule across web and desktop settings.Source: D Implementation Details · p. 18
  • Execution requires valid bindings for task-relevant resources, and writable state reachable by different environments must remain disjoint.Task-relevant mutable process state must be context-keyed or kept in a private execution component; otherwise that component is excluded from sharing.Source: D Implementation Details · p. 18

Asynchronous and background execution

  • Asynchronous requests and tracked background work inherit the environment context and generation lease; lifecycle transitions quiesce new work and drain in-flight work before publishing a successor generation.Unattributable or unquiesceable background activity uses a private or coarser-grained backend, which can reduce runtime reuse without relaxing isolation.Source: D Implementation Details · p. 18

E Scaling Analysis

The scaling analysis compares resource and throughput accounting with measured results across concurrency levels. CUA-Sandbox retains efficiency advantages over Docker in the tested workloads, while the accounting identifies costs that remain.

Resource model

  • The memory accounting separates shared initialized-runtime memory from private slot and capsule costs, and savings depend on workload and sharing boundary.Runtime sharing can still reduce memory when all environments are active; inactive trajectories add further savings.Source: E Scaling Analysis · p. 19

Storage and lifecycle cost

  • Storage savings arise from initialized writable state rather than duplicated immutable images; lifecycle operations still incur quiescence, state preparation, rebinding, and readiness checks.The accounting model alone does not establish throughput or environment density as concurrency increases.Source: E Scaling Analysis · p. 19

Concurrency scaling

  • At concurrency 16, CUA-Sandbox has 2.45×, 3.52×, and 5.14× higher throughput on WebArena, VisualWebArena, and OSWorld, respectively.Per-environment memory is reduced by 4.78×, 22.38×, and 3.16× on those workloads, respectively; the table covers concurrency 4 and 16.Source: E Scaling Analysis · p. 19, Table 7 · p. 19

Ablation

  • Table 8 measures an eight-environment ablation of copy-on-write and runtime sharing, distinguishing runtime RAM, agent/browser PSS, and writable storage.Lower values are better; the supplied passage does not report the ablation outcomes.Source: Table 8 · p. 20

F Runtime-sharing ablation.

Runtime sharing cuts memory beyond copy-on-write alone, while the case studies illustrate private state and coordinated resets for web and desktop tasks.

F Runtime-sharing ablation.

  • 3,261 MiB runtime RAM with sharing is 4.26× below the CoW configuration with independent runtimes and 4.77× below the official Docker baseline.Agent/Browser PSS stays approximately unchanged across configurations, while writable storage changes only slightly from independent to shared runtimes.Source: F Runtime-sharing ablation · p. 20

G Case Studies: State-Scoped Execution

  • The web case shares an initialized server while keeping database branches, sessions, cache namespaces, and browser profiles private; request context must also cover asynchronous work.An adapter unable to respect that context requires a private backend; the workflow is a design illustration, not a measured trajectory.Source: G Case Studies: State-Scoped Execution · p. 21, Figure 5 · p. 21
  • The desktop case keeps documents, profiles, configuration, displays, and message-bus endpoints private, with observations and evaluation following the same environment bindings.Components whose in-memory state or open handles cannot be safely rebound remain in a private session backend.Source: G Case Studies: State-Scoped Execution · p. 22, Figure 6 · p. 22
Loading 2609.32750v1…