PaperlayerView exact revision on arXiv

Source-linked AI summary

EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management

Liang Mi, Weijun Wang, Bowen Gao, Tianze Yu, Zixu Hao, Han Xiao, Xin Ding, Mingzhe Huang, Xin He, Lu Shi, Hao Wu, Haipeng Dai, Guihai Chen, Yunxin Liu, Ting Cao

arXiv:2609.27547v1cs.LGcs.DC

TL;DR

Embodied RL has heterogeneous CPU and GPU demands, and existing systems still leave resources idle despite overlapping rollout and training. EBRL combines asynchronous pipelining with fine-grained resource management; across evaluated configurations, it improves end-to-end trajectory throughput over SOTA systems.

  • Problem

    Heterogeneous stage demands, exclusive GPU allocation, and rollout synchronization leave hardware underused in embodied RL.

    Source: §1 · pp. 1–2

  • Method

    EBRL overlaps rollout and training and uses profiled, runtime-adjusted CPU-core and GPU-SM allocations with stage quotas and batch sizes.

    Source: §1 · p. 2

  • Results

    EBRL achieves 1.30×–3.47× the end-to-end trajectory throughput of SOTA RL systems in each evaluated configuration.

    Source: §8 · p. 12

  • Takeaways & Limitations

    EBRL’s evaluation across diverse models, simulators, and hardware shows higher end-to-end trajectory throughput than SOTA systems in each configuration.

    Source: §8 · p. 12

    Colocated execution is unavailable on A100 GPUs for BEHAVIOR rendering because A100s lack the required RT cores.

    Source: §6.2 · p. 9

Abstract

from arXiv · show

Embodied reinforcement learning (RL) improves model capabilities with a pipeline of environment simulation, action generation, and model updates. These stages show heterogeneous CPU and GPU demands, making efficient resource utilization difficult. Recent systems overlap rollout (simulation and generation) with training for efficiency, but exclusive GPU allocation and synchronized barrier in rollout still leave substantial hardware resource waste. In this paper, we present EBRL, an asynchronous embodied RL training system with two core techniques. The asynchronous pipelined scheduler overlaps rollout and training, pipelines simulation and generation across environment groups, and carries out each environment independently, eliminating synchronization stalls. The fine-grained resource manager pools CPU cores and GPU streaming multiprocessors, and uses stage profiles and runtime feedback to adjust resource quotas and batch sizes to meet the shifting demands among stages. We implement EBRL on RLinf and evaluate it with four embodied policies and four simulation benchmarks across heterogeneous GPU testbeds. Experiments show that EBRL achieves 1.30-3.47 times the end-to-end rollout throughput and 2.5 times of training convergency compared to the SOTA embodied RL systems.

1 Introduction

Embodied RL combines simulation, action generation, and training, but heterogeneous stage demands and rollout synchronization leave hardware underused. EBRL addresses these constraints with asynchronous pipelining and fine-grained resource management.

1 Introduction

  • Embodied RL combines simulator interaction and model generation to collect trajectories, then trains on them to update weights for later rollouts.The pipeline repeats iteratively; RL can increase embodied-model success rates by more than 40%.Source: §1 · p. 1
  • Simulation needs CPU–GPU co-processing, while generation and training rely heavily on GPUs, making resource management harder at large environment scale.Rollout commonly simulates hundreds of environments concurrently and generates actions in batches.Source: §1 · p. 1
  • Existing systems overlap rollout with training, but exclusive GPU allocation leaves OpenPI on RoboCasa below 50% GPU utilization for over 90% of RL time.A synchronized rollout barrier can also let one slow environment delay the whole batch.Source: §1 · p. 1
  • EBRL’s asynchronous scheduler overlaps rollout with training and pipelines simulation with generation, while independent environment progress reduces waiting from variability.Generation and training share GPUs under separate logical SM quotas.Source: §1 · p. 2
  • EBRL’s resource manager pools CPU cores and GPU SMs, selects stage quotas and batch sizes from offline profiles, then adjusts allocations using runtime throughput.The allocator responds when it detects sustained stage imbalance.Source: §1 · p. 2
  • The paper characterizes underutilization as stemming from coarse-grained, stage-exclusive allocation and synchronization barriers across rollout iterations.EBRL combines an asynchronous pipelined scheduler and fine-grained resource management in a prototype system.Source: §1 · p. 2
  • EBRL achieves up to 1.98–2.98× throughput improvement over SOTA systems, evaluated across four embodied models, four simulators, and three heterogeneous testbeds.The evaluation uses models from both VLA and WAM families.Source: §1 · p. 2

2 Background

Embodied models map visual, robot-state, and language inputs to action chunks, while RL alternates trajectory collection and policy updates. Embodied RL systems use several execution modes, each with trade-offs.

2.1 Embodied Models

  • VLA models use a separate diffusion action head, whereas WAM models jointly denoise future-frame latents and action chunks with a DiT backbone.The paper says WAM’s anticipated visual outcomes anchor action generation and can benefit long-horizon tasks.Source: §2.1 · p. 2

2.2 RL for Embodied Model

  • Embodied RL iterations comprise simulation, generation, and training; simulator-stack choices determine hardware needs, including CPU-only MuJoCo and CPU–GPU PhysX workloads.Training updates the policy from rollout trajectories, and the updated policy is used for subsequent generation.Source: §2.2 · p. 3

2.3 Embodied Reinforcement Learning Systems

  • Colocated, disaggregated, and hybrid execution trade GPU switching overhead against waiting; asynchronous systems overlap rollout and training but tolerate moderate policy staleness.No single mode dominates: colocated tends to suit small models, while disaggregated tends to suit large ones.Source: §2.3 · p. 3

3 Motivation and Challenges

Existing embodied RL systems leave compute idle through coarse resource allocation and rollout synchronization. EBRL’s design targets both stage overlap and allocation across changing demands.

3.1 Limitations of SOTA Systems

  • During generation, more than 75% of time has GPU SM occupancy below 20%; during rollout, only 22 of 56 CPU cores sustain utilization above 50%, reflecting CPU underutilization.With the SOTA asynchronous pipeline [11], GPUs still have 47% idle time during embodied RL.Source: §3.1 · p. 4

3.2 Challenges

  • Synchronized simulation makes environments wait for stragglers: average step time is 40 ms, P99 reaches 60 ms, and the average trajectory length falls 31% by convergence.The latter shift changes the compute balance between rollout and training, complicating static allocation.Source: §3.2 · pp. 4–5
  • Efficient sharing requires sub-device CPU and GPU allocation, but the best split depends on coupled hardware, policy, task, and RL settings and changes during training.Simulation–generation dependencies and barriers also leave GPUs idle even when rollout overlaps with training.Source: §3.2 · p. 4

4 EBRL Design

EBRL coordinates shared CPU and GPU resources with overlapping execution across iterations, environment groups, and individual simulator steps. Its scheduler exposes independent work while preserving environment action order and stage dependencies.

4.1 Overview

  • EBRL combines a GPU–CPU resource pool, elastic allocator, and asynchronous scheduler to coordinate simulation, generation, and training.The pool exposes CPU-core sets, GPU SM quotas, and memory budgets; throughput feedback updates stage budgets over time.Source: §4.1 · p. 5

4.2 Asynchronous Pipelined Scheduler

  • The scheduler removes barriers at three levels: across rollout and training iterations, between simulation and generation groups, and between environment steps.These mechanisms operate within stage resource budgets chosen by the allocator.Source: §4.2 · p. 5

4.2.1 Inter-Iteration Rollout–Training Overlap

  • Rollout collects new trajectories while training processes an earlier batch, with generation and training sharing GPUs under separate logical SM quotas.Updated weights are published to generation workers after the preceding training update completes, without a global rollout–training barrier.Source: §4.2.1 · p. 5

4.2.2 Intra-Rollout Simulation–Generation Pipeline

  • Environment groups pipeline simulation and generation: one group can simulate while another receives newly generated actions.A bounded producer–consumer queue buffers observations; insufficient ready groups or mismatched stage capacities can still leave pipeline bubbles.Source: §4.2.2 · pp. 5–6, Figure 9 · p. 6

4.2.3 Step-Level Asynchronous Orchestration

  • Within each action chunk, environments advance independently; freed CPU cores are rebound across groups to ready workers, prioritizing environments with longer estimated step latency.This removes within-chunk step barriers while preserving each environment’s action order.Source: §4.2.3 · p. 6, Figure 10 · p. 6

4.3 Fine-Grained Resource Management

EBRL exposes CPU and GPU sub-device capacity, then uses profile-guided allocation to balance stage throughput under a fixed hardware budget. Runtime measurements can trigger reallocation as stage costs change.

4.3 Fine-Grained Resource Management

  • EBRL combines sub-device resource controls with elastic allocation to match stage capacities as training progresses.The allocator selects resource quotas and batch sizes to maximize end-to-end throughput under a fixed hardware budget.Source: §4.3 · pp. 6–7, §4.3.2 · p. 7

4.3.1 GPU–CPU Resource Pool

  • The GPU–CPU pool exposes GPU SM quotas, memory budgets, and CPU core sets through a common allocation interface.The resource types retain separate capacity constraints.Source: §4.3.1 · p. 7

4.3.2 Elastic Resource Allocation

  • The allocator maximizes predicted end-to-end throughput, constrained by the slower stage rates under steady-state overlap.It expresses stage rates in equivalent rollout chunk-steps per second and jointly chooses CPU configurations, GPU quotas, and batch sizes.Source: §4.3.2 · p. 7
  • Offline stage profiles estimate simulator, generation, and training throughput across resource and batch-size configurations.The initial allocation reserves simulator capacity near its throughput knee, then shifts remaining GPU capacity toward the slower of rollout and training while respecting memory limits.Source: §4.3.2 · pp. 7–8
  • When a sustained throughput imbalance is detected, runtime measurements update the lookup tables and trigger a new allocation.The trigger is a gap exceeding 15% of the faster stage for 20 consecutive steps, using a 20-step moving average.Source: §4.3.2 · p. 8

4.4 Runtime and Programming Support

EBRL provides a Python interface for configuring resources and training orchestration. The runtime profiles stages, applies an initial placement, and feeds execution statistics back into elastic balancing without imposing a rollout–training barrier.

4.4 Runtime and Programming Support

  • The programming model lets users configure the policy, simulator, and resource pool, then express a training loop that feeds execution statistics to the allocator.The runtime implements the loop through profiling, elastic placement, and asynchronous worker scheduling.Source: §4.4 · p. 8
  • Initialization profiles the selected policy and simulator, computes an initial allocation, and applies CPU bindings and GPU quotas at worker launch.The runtime exposes available CPU and GPU capacity through a ResourcePool.Source: §4.4 · p. 8
  • During training, throughput statistics can trigger elastic reallocation, while asynchronous pipelines exchange observations, actions, trajectories, and weights without a rollout–training barrier.The step-level scheduler dispatches ready simulator work within the current CPU allocation.Source: §4.4 · p. 8

5 Implementation

EBRL is implemented as a resource-management and runtime-orchestration layer on RLinf. It binds per-worker resource configurations and combines asynchronous pipelines with step-level CPU-core reassignment.

5 Implementation

  • EBRL adds a fine-grained resource and runtime layer on RLinf, reusing its worker dispatch, Ray launch, data channels, and training path.The implementation is primarily Python and totals approximately 6.0K lines of code.Source: §5 · p. 8
  • The placement backend maps configuration to per-rank resource bindings, enforcing GPU quotas through MPS or pre-created MIG partitions and CPU pinning through Linux affinity.MPS/MIG setup is performed once; EBRL does not manage the MPS daemon or mutate MIG instances at runtime.Source: §5 · p. 8
  • Stage-level execution uses bounded producer–consumer queues and shared-GPU generation and training, while step-level scheduling rebinds freed cores to the slowest ready environment.Runtime core rebinding via os.sched_setaffinity has less than 1 ms overhead.Source: §5 · p. 9

6 Evaluation

EBRL is evaluated across embodied policies, simulators, and heterogeneous GPU testbeds, with higher throughput than the compared systems and component studies examining how its mechanisms contribute.

6.1 Experimental Setup

  • The evaluation covers four policies, five simulator-policy configurations, PPO comparisons, GRPO tests, and heterogeneous CPU- and GPU-based simulation testbeds.The main testbeds are H100, A100, and H20; BEHAVIOR additionally uses RTX 4090 GPUs, and scaling extends to 16 GPUs.Source: §6.1 · p. 9

6.2 End-to-End Performance

  • Across nine VLA workload–testbed combinations, EBRL achieves 1.31×–3.47× the throughput of the strongest baseline, with further gains on LIBERO/DreamZero and BEHAVIOR.The DreamZero comparison covers A100; BEHAVIOR uses a mixed RTX 4090–A100 testbed.Source: §6.2 · p. 9, Figure 11 · p. 10, Figure 12 · p. 10
  • On LIBERO-Object/GR00T with PPO, EBRL completes 60 epochs approximately 2.5× faster than Sync, with both reaching approximately 98% final success.The comparison measures evaluation success rate against wall-clock training time.Source: §6.2 · p. 10, Figure 14 · p. 10
  • Under GRPO, EBRL exceeds Colocated, Hybrid, and Async throughput on both RoboCasa/OpenPI and ManiSkill/OpenVLA-OFT.The tested speedups are 2.20×, 3.45×, and 1.71× on RoboCasa and 1.90×, 2.39×, and 1.48× on ManiSkill, respectively.Source: §6.2 · p. 10, Figure 13 · p. 10
  • Throughput rises from 0.19 to 3.19 as EBRL scales from 1 to 16 GPUs.The intermediate plotted throughputs are 0.44, 0.84, and 1.67 at 2, 4, and 8 GPUs.Source: §6.2 · p. 10, Figure 15 · p. 11

6.3 Comprehensive Component-wise Analysis

  • Disabling resource allocation, intra-rollout simulation–generation pipelining, or inter-iteration rollout–training overlap reduces throughput by approximately 10%, 31%, or 26%, respectively.The component experiments use RoboCasa/OpenPI on H100 unless specified otherwise.Source: §6.3 · p. 10, Figure 16 · p. 11
  • As stage loads change, the allocator shifts GPU shares between training and generation using measured training times and runtime feedback.It reallocates toward training when policy fluctuations raise training costs.Source: §6.3 · p. 11, Figure 17 · p. 11
  • Core rebinding achieves 1.61× speedups in average and P50 action-chunk latency and 1.53× at P99 versus the batch-step baseline.P99 latency remains 6.3% above P50.Source: §6.3 · p. 11, Figure 18 · p. 11

7 Related Work

Related work addresses embodied policies and simulators, RL training systems, and resource management; EBRL combines these lines through fine-grained asynchronous execution.

7 Related Work

  • Embodied-model and simulator research improves robot capabilities and interaction-data collection, while policy optimization and adaptive rollout methods target the effectiveness of collected interactions.The discussed areas include VLA models, WAM policies, diffusion transformers, RL post-training, and GPU-parallelized simulation.Source: §7 · p. 11
  • Prior embodied RL systems explore stage placement, asynchronous execution, pipelining, and workload rebalancing to improve training efficiency.RLinf supports colocated, disaggregated, and hybrid execution modes.Source: §7 · p. 11
  • Resource-management approaches allocate CPU resources, multiplex GPUs, or schedule kernels; EBRL jointly selects CPU configurations, GPU quotas, and batch sizes from stage profiles, then adjusts allocations as stage costs change.The passage cites Synergy, GoldMiner, MPS, MIG, GMI-DRL, and Orion as examples of related approaches.Source: §7 · pp. 11–12

8 Conclusion

EBRL targets resource underutilization in embodied RL with asynchronous pipelining and fine-grained resource management, and reports higher end-to-end trajectory throughput across diverse configurations.

8 Conclusion

  • EBRL combines an asynchronous pipelined scheduler with a fine-grained resource manager to reclaim idle capacity left by whole-device GPU allocation and synchronization barriers.The system also provides programming support for users.Source: §8 · p. 12
  • Across diverse models, simulators, and hardware, EBRL achieves 1.30×–3.47× the end-to-end trajectory throughput of SOTA RL systems in each configuration.Source: §8 · p. 12

A Greedy Core Rebinding

The algorithm dispatches one eligible environment to a free core, prioritizing the environment with the largest recorded latency.

A Greedy Core Rebinding

  • When a core becomes free, the scheduler collects eligible environments with unfinished actions, no running step, and an available next action.Source: A Greedy Core Rebinding · p. 15
  • It atomically selects the ready environment with the largest H[e], rebinds the worker to the free core, and dispatches one simulator step.Source: A Greedy Core Rebinding · p. 15
  • After the step, it advances the environment's action index, updates its latency, releases the core, and triggers another dispatch.Source: A Greedy Core Rebinding · p. 15

B Simulation Performance

Explicit CPU affinity improves simulator throughput and reduces latency across three benchmarks.

B Simulation Performance

  • 1.38×, 1.97×, and 1.60× throughput gains over default OS scheduling on LIBERO, RoboCasa, and BEHAVIOR, respectively.Table 2 reports simulator throughput with optimal per-environment CPU bindings versus default OS scheduling.Source: B Simulation Performance · p. 15, Table 2 · p. 15
  • CPU bindings improve simulator performance by reducing scheduling preemption between environments.The passage also reports reduced latency, without giving benchmark-specific latency values.Source: B Simulation Performance · p. 15
Loading 2609.27547v1…