PaperlayerView exact revision on arXiv

Source-linked AI summary

Replicate What Varies: Resource Sharing for Domain-Randomized Embodied Simulation

Zixu Hao, Weijun Wang, Xiangyu Li, Liang Mi, Hao Wu, Yunxin Liu, Ting Cao

arXiv:2610.33582v1cs.DCcs.AIcs.RO

TL;DR

Independently constructed domain-randomized environments can duplicate unchanged resources and limit how many simulation sandboxes fit on a machine. Patchwork represents environments as shared templates and explicit patches, selecting reusable templates and replaying mutations while sharing common resources. Its results support template-based sharing for deploying diverse environments within a fixed memory budget, with savings depending on template reuse and asset overlap.

  • Problem

    Independently materialized domain-randomized environments can duplicate unchanged resources, contributing to the CPU and GPU memory footprint that limits concurrent simulation sandboxes.

    Source: §1 · p. 1

  • Method

    Patchwork represents environment differences as declarative mutations, selects reusable templates, and materializes template–patch pairs with shared resources and isolated instance state.

    Source: §1 · p. 2, §5.1 · p. 5

  • Results

    For compatible task variants, model storage is shared within template groups and texture storage is reused across groups, with savings determined by template reuse and asset overlap.

    Source: §1 · p. 2, §12 · p. 15

  • Takeaways & Limitations

    Template-based resource sharing supports deploying diverse simulation environments within a fixed memory budget.

    Source: §12 · p. 15

    The current backend restricts supported mutations; broader serving policies would add storage, isolation, and validation requirements.

    Source: §10 · p. 14

Abstract

from arXiv · show

Simulation provides a low-cost sandbox where embodied agents can attempt tasks, observe outcomes, and learn from experience. Running many diverse simulation environments concurrently can consume substantial CPU and GPU memory because independently constructed scenes duplicate resources that remain unchanged across instances. We present Patchwork, a resource-sharing system for domain-randomized simulation, built around the representation $E=T+\Delta$: a shared template contains common resources, and a patch records the mutations needed to construct a target environment. Patchwork separates randomization sampling from replay and selects reusable templates together with patches that preserve each target environment's intended configuration. Its layered runtime shares compiled model structure and immutable assets while keeping mutable parameters and simulation state private to each instance. We implement Patchwork on robosuite and MuJoCo and evaluate it on LIBERO-plus and RoboCasa365 workloads. At 32 replicas, shared-template execution reduces CPU RSS by 88-92% relative to process-per-environment execution. Controlled experiments show how sharing shifts memory growth toward template diversity and unique asset content rather than instance count, within the supported sharing boundaries.

1 Introduction

Embodied simulation can provide scalable experience for agents, but diverse concurrent environments consume substantial memory when independently constructed scenes duplicate unchanged resources. Patchwork represents each environment as a shared template plus explicit mutations, then replays those mutations while sharing reusable resources and isolating instance-specific state.

1 Introduction

  • Diverse concurrent environments consume CPU and GPU memory, while independently constructed scenes can duplicate common resources that remain unchanged across variants.Domain randomization varies conditions such as lighting, textures, and object poses to expose policies to different conditions and support transfer to the real world.Source: §1 · p. 1
  • Patchwork represents an environment as E = T + ∆: a shared template holds invariant content, and a patch records mutations that produce an instance.This decomposition allows unchanged resources to be shared while varying parts remain private to each instance.Source: §1 · p. 2
  • Patchwork maps mutations to resource-specific sharing boundaries, sharing compiled structure and immutable assets while keeping mutable model parameters and runtime state private.Templates remain read-only, and supported mutations are directed to instance-owned resources.Source: §1 · p. 2
  • Patchwork separates randomization sampling from application and records sampled choices as declarative mutations that can be replayed without resampling.The replay is intended to reproduce the target instance while preserving its randomization choices.Source: §1 · p. 2
  • At 32 RoboCasa365 environments, Patchwork reduces CPU RSS and GPU memory by 92% and 93%, respectively, versus process-per-environment execution.The evaluation also finds that, for compatible variants, model storage is shared within template groups and texture storage is reused across groups.Source: §1 · p. 2

2 Background

The background describes how embodied simulation stacks compose task, environment, physics, and rendering components, and how MuJoCo separates compiled models from instance state. Randomization choices enter through several interfaces and at different points in environment construction.

2.1 Simulation Software Stack

  • The simulation stack combines task definitions, an environment framework, and physics and rendering backends; LIBERO and RoboCasa use robosuite with MuJoCo.MuJoCo provides physics and rendering, while robosuite supplies environment construction and controllers; LIBERO defines manipulation tasks and RoboCasa procedurally constructs household scenes and tasks.Source: §2.1 · pp. 2–3

2.2 MuJoCo Models and Simulation State

  • MuJoCo compiles an XML scene into an mjModel, while each mjData stores one instance’s evolving simulation state.During stepping, the model is read-only and each instance updates its own mjData, so an unchanged model can serve independently advancing instances.Source: §2.2 · p. 3
  • A structural model change requires recompilation, while editing a shared parameter such as lighting affects every instance using that model.Separate models avoid shared writes but duplicate unchanged model content.Source: §2.2 · p. 3

2.3 Domain Randomization Interfaces

  • Randomization can enter through task descriptions, construction-time choices, or direct edits to an existing model, with different timing across interfaces.LIBERO-plus variants are stored as separate BDDL files before scene compilation, RoboCasa samples layouts, styles, objects, and placements during construction, and robosuite modders edit model fields directly.Source: §2.3 · pp. 3–4

3 Characterizing Domain-Randomized Workloads

The workloads contain sparse, structured variation, but memory is concentrated in large model and asset resources and construction can couple randomization to reconstruction. The section derives requirements for representing changes explicitly, sharing resources at suitable boundaries, preserving isolation, and faithfully replaying sampled choices.

3.1 Variation Is Sparse and Structured

  • Across 16,170 LIBERO-plus task records, 5,855 distinct compiled models appear, while 75% of 16,000 non-base variants change just one semantic target.No measured variant changes more than 17 targets; the instruction-only variants do not require a different model.Source: §3.1 · p. 4

3.2 Memory Footprints Are Layered and Skewed

  • Textures account for 20–69% of LIBERO model footprints and, for CloseDrawer, 81% in RoboCasa v0.2 and 70% in RoboCasa365.By contrast, each instance’s mjData main buffer is less than 1% of model and asset footprint, reaching at most 0.77 MiB for measured LIBERO tasks and 2.07 MiB for RoboCasa365.Source: §3.2 · p. 4, Figure 4 · p. 5

3.3 Coupling Randomization with Construction Undermines Sharing and Replayability

  • Rebuilding each variant duplicates unchanged resources, while editing shared model fields affects every instance; RoboCasa hard resets can also reconstruct the scene and resample choices.The RoboCasa365 reconstruction path mixed restored episode metadata with freshly sampled task-specific choices, potentially producing an environment that no longer matches the recorded episode.Source: §3.3 · p. 4

3.4 Workload Properties and Requirements

  • The workload characterization motivates a template-and-patch decomposition and four requirements: explicit variation, resource-specific sharing, isolation with semantic equivalence, and faithful replay.Faithful replay preserves sampled choices without introducing additional randomization.Source: §3.4 · p. 4

4 Design Overview

Patchwork’s design connects environment specifications and randomization choices to template–patch pairs, then materializes them through resource-specific sharing. Its overview divides the system into mutation representation, sharing mechanisms, and template serving.

4 Design Overview

  • Patchwork takes environment specifications and randomization choices and constructs a common mutation representation.The representation makes differences between environments explicit and replayable.Source: §4 · p. 5
  • A serving policy selects reusable templates and represents each target environment as a patch over one template.Template serving discovers reusable templates and derives the patches that construct each target.Source: §4 · p. 5
  • Layered materialization shares compiled structure and immutable assets while isolating per-environment parameters and runtime state.The design overview identifies resource-specific mechanisms as the means to preserve instance isolation and simulation semantics.Source: §4 · p. 5

5 Mutation Representation and Materialization Contract

Patchwork represents environment changes as ordered, replayable mutations over a selected shared template. A backend resolver directs each mutation to its materialization destination while preserving the target environment’s intended configuration.

5.1 Mutation Specifications

  • A MutationSpec records a stable logical target, operation, and value; targets avoid exposing engine-specific storage locations.The generic representation admits set, add, and remove, while the current MuJoCo backend primarily supports isolated set operations for compile-free serving.Source: §5.1 · p. 5

5.2 Sampling and Replay

  • Sampling returns an ordered mutation sequence, and applying it consumes no random numbers, allowing the same choices to be stored and replayed.The sampler may use randomness for checks or retries, but those choices remain within sampling rather than application.Source: §5.2 · p. 6

5.3 Template and Patch Semantics

  • Each target environment is represented by a selected template and an ordered patch, valid only when every mutation has an interpretation relative to that template.Materializing the patch should produce the same model and execution behavior as independently constructing the target environment.Source: §5.3 · p. 6

5.4 Logical Targets and Resource Mapping

  • A backend-specific resolver maps logical targets to writes owned by the target instance, including model fields, dynamic state, or task metadata.Structural or asset-membership changes require a new template; model-field changes use instance-owned DR buffers, while poses and other dynamic values go to mjData.Source: §5.4 · pp. 6–7

6 Layered Model Resource Sharing

Patchwork separates sharing by resource type while keeping templates read-only and directing writes to instance-owned resources. Compiled structure and immutable assets are shared; mutable parameters and simulation state remain private.

6.1 Structure Layer: Model Forking

  • Instances share unchanged compiled model structure through a model fork, while each fork owns writable metadata and its own mjData.Robosuite construction metadata and a deterministic initial state provide a baseline for rebuilding private Python objects and replaying construction decisions.Source: §6.1 · p. 7, §6 · p. 7

6.2 Parameter Layer: Fork-Local DR Buffers

  • Fork-local deep-copied DR buffers isolate mutable model parameters whose writes and derived updates remain within instance-owned storage.Fields whose updates would affect shared storage are excluded from the supported DR vocabulary and require another template or materialization path.Source: §6.2 · p. 7

6.3 Asset Layer: Content-Addressed Deduplication

  • Content-addressed CPU and GPU pools share immutable meshes and textures by content identity, with explicit references held by models or contexts.Texture identity also includes texture metadata, and the CPU and GPU pools remain separate because they own different resource types.Source: §6.3 · p. 7

7 Template Selection and Serving

The policy layer assigns each environment specification to a template and patch, using incremental cache lookup or batch planning. It groups compatible profiles and records assignments for materialization.

7 Template Selection and Serving

  • Each policy output assigns every environment to one template and one patch.The policy can resolve specifications against cached templates or plan a complete batch before building new templates.Source: §7 · pp. 7–8

7.1 Incremental Selection and Batch Planning

  • Both planning interfaces use model, parameter, and asset profiles to check compatibility and derive patches.The frontend supplies profiles without exposing its construction path to the selection policy.Source: §7.1 · p. 8
  • Incremental selection uses registered templates when possible and registers a new source on a miss, avoiding forced matches to incompatible templates.It is the generic fallback because it does not require advance knowledge of the workload.Source: §7.1 · p. 8
  • Batch planning returns a build record for each template and an assignment for each workload member, allowing construction once per selected source.Planning completes before the cache applies the plan; materialization can run in parallel when the caller provides a parallel path.Source: §7.1 · p. 8
  • Batch planning is optional; the default policy combines incremental selection with grouping, while workload-specific policies can preserve the same plan interface.The LIBERO-plus offline index preselects representative sources and patches, with unresolved requests falling back to incremental selection.Source: §7.1 · p. 8

7.2 The Operational Serving Preorder

  • Under the supported mutation vocabulary, A is served by template B when an ordered patch materializes A from B with matching state and behavior.With composable patches, the serving relation is a preorder; its mutual-serving kernel partitions environments into classes.Source: §7.2 · p. 8
  • The current policy selects one representative per mutual-serving class; union templates remain outside the implementation without executable support for additional mutations.A structural–asset cover test can filter candidates, but passing it still depends on backend support for asset-binding selection and writes.Source: §7.2 · pp. 8–9

7.3 Equivalence-Based Serving; 7.4 Offline Index Construction and Validation

  • Equivalence-based serving groups profiles by unchanged structure and identical asset identities, then derives field-level patches from a selected representative.Grouping is independent of input order; an offline index records assignments for covered specifications, validates profile and execution behavior, and leaves uncovered requests to incremental selection.Source: §7.3 · p. 9, §7.4 · pp. 9–10

8 Implementation

Patchwork is implemented on the MuJoCo/robosuite stack with engine changes for shared model storage and asset pooling, plus a threaded runtime for concurrent environments.

8 Implementation

  • The MuJoCo/robosuite implementation comprises approximately 18K LoC.Source: §8 · p. 10

8.1 MuJoCo Engine Extensions

  • A shallow-copy fork and reference counting share an mjModel body while deep-copying each model's private domain-randomization buffer.DR groups redirect engine field pointers into instance-owned storage; the engine extensions derive buffer sizing, field copying, and pointer redirection.Source: §8.1 · p. 10

8.1 MuJoCo Engine Extensions; 8.2 Threaded robosuite Runtime

  • Texture pools deduplicate pixel data on CPU and share GPU texture objects across GL contexts using content and metadata.Environments run on fixed worker threads with EGL contexts in one share group; hot-path physics and controller computations move to C++ and release the GIL.Source: §8.1 · p. 10, §8.2 · p. 10

9 Evaluation

The evaluation compares Patchwork with independent-model execution across robosuite, LIBERO-plus, and RoboCasa365 workloads, measuring memory, throughput, template reuse, and correctness. It examines replica scaling, workload diversity, template serving, and component costs.

9.2 Resource Use and Throughput under Replica Scaling

  • At 32 replicas, shared-template threads reduce CPU RSS by 88% on LIBERO-plus and 92% on RoboCasa365 versus process-per-environment execution.On RoboCasa365, summed RSS falls from 123.8 to 10.5 GiB; with one 224×224 RGB camera, shared-template threads use 10.7 versus 27.7 GiB RSS and 3.9 versus 7.1 GiB GPU memory versus independent threads.Source: §9.2 · pp. 10–11, Figure 9 · p. 11, Table 3 · p. 11

9.3 Memory under Workload Diversity

  • At N=32, LIBERO-plus RSS rises from 2.98 to 7.14 GiB as template classes increase from K=1 to K=32, but varies by at most 55 MiB across 1–32 variants per class.When K=N, no instances reuse a template; the K=32 case approaches the independent-model baseline.Source: Figure 10 · p. 12, §9.3 · p. 12

9.4 Template Serving and Correctness

  • The LIBERO-plus offline index serves 5,855 distinct models with 3,042 templates, a 48% reduction, applying patches to the remaining 2,813 models.The index’s 16,170 records describe those models; patches have a median of six field writes and a maximum of 22.Source: Table 4 · p. 12, §9.4 · p. 12

9.3 Memory under Workload Diversity

  • With one RoboCasa365 template, RSS grows approximately linearly with unique texture bytes (slope 0.96, R2=0.966); at K=8, disjoint texture sets use 1.35 GiB more than shared sets.Native-style texture workloads at K=8 and N=32 reduce summed RSS by 31–33% versus independent models.Source: §9.3 · p. 12, Figure 11 · p. 13

9.4 Template Serving and Correctness

  • Offline and incremental selection choose the same number of templates, while median reuse rises from 6% at 16 requests to 50% at 256 requests.The offline index reduces online assignment cost from 0.79–0.88 s to 0.51–0.53 ms per request; seeded rollout checks also match states and images bitwise and agree on success flags.Source: §9.4 · pp. 12–13, Figure 12 · p. 13

9.5 Component Costs and Ablation

  • At K=8 and N=32, constructing the first template adds 1.24 GiB for LIBERO-plus and 1.79 GiB for RoboCasa365, while 32 instances add 1.24 and 1.99 GiB, respectively.Template capture takes about 1 ms, versus 0.3–0.8 s to construct and reset the source environment; creating an instance takes about 0.1 s.Source: Figure 13 · p. 13, §9.5 · p. 13
  • At N=32, enabling model and GPU texture sharing together reduces summed RSS to 10.5 GiB and GPU memory to 3.2 GiB, reductions of 7.0× and 13.7× versus both disabled.Model sharing alone lowers RSS from 74.0 to 31.4 GiB; GPU texture sharing alone lowers GPU memory from 44.1 to 5.9 GiB.Source: §9.5 · p. 13, Figure 14 · p. 14

10 Discussion

The discussion scopes Patchwork to CPU-based sandboxes with independently progressing environments and identifies the current mutation vocabulary as a boundary on sharing.

10 Discussion

  • Trade-offs against GPU batching for asynchronous agent requests remain future work; broader sharing could use asset rebinding and union templates, with added storage, isolation, and validation requirements.The current backend restricts supported mutations to keep their effects and isolation requirements explicit.Source: §10 · p. 14

11 Related Work

Related work spans heterogeneous simulation, sandbox memory reuse, and structure-aware sharing systems. Patchwork applies resource-specific sharing boundaries to simulation environments and their supported mutations.

Sharing in Heterogeneous Simulation

  • GPU simulators support heterogeneous variation through shared or per-world fields and USD instancing, but some texture changes require disabling physics replication and removing instancing.MuJoCo Warp uses a base model with candidate meshes and per-world selection fields; Isaac Lab maps asset prototypes to selected environments.Source: §11 · p. 14

Resource Sharing in Sandboxes and Serverless Systems

  • VM page deduplication, shared mappings, and serverless state reuse operate at page, process, or sandbox boundaries, whereas Patchwork derives sharing boundaries from simulation resources and mutations.Patchwork can reuse common assets across different model templates while applying parameter changes to private storage.Source: §11 · p. 14

Structure-Aware Resource Sharing

  • Structure-aware systems expose reuse through workload representations and coordinate reuse discovery with execution; Patchwork uses explicit mutations and resource profiles to select templates and materialize sharing.Examples include shared model tensors, block-managed KV caches, common-prefix discovery, and dataflow representations.Source: §11 · pp. 14–15

12 Conclusion

Patchwork represents domain-randomized environments as shared templates and explicit patches, connecting environment differences to resource-specific sharing and private state. Its robosuite/MuJoCo prototype reduces memory in replica-scaling and controlled heterogeneous workloads, with savings depending on template reuse and asset overlap.

12 Conclusion

  • The robosuite/MuJoCo prototype reduces memory in replica-scaling and controlled heterogeneous workloads, with savings determined by template reuse and asset overlap.The paper concludes that template-based sharing can deploy diverse simulation environments within a fixed memory budget.Source: §12 · p. 15

Related papers

Loading 2610.33582v1…