PaperlayerView exact revision on arXiv

Source-linked AI summary

A GPU-Parallel Framework for Heterogeneous Multi-Task Reinforcement Learning

Rui Zhang, Qiwei Wu, Zhengyu Zhang, Tao Li, Hongyu Zhou, Xiang Li, Yunrong Guo, Junjie Lai, Renjing Xu, Weihua Zhang

arXiv:2606.03335v3cs.RO

TL;DR

Existing benchmarks rarely combine GPU-parallel scale, heterogeneous manipulation tasks, and standardized multi-task evaluation. Hebero provides that benchmark and DGPO’s shared demonstration-guided PPO framework, with IW-ABC coordinating teacher guidance and task weighting. IW-ABC improves success over the strongest state-input baseline, and a jointly trained policy also performs four tasks on a physical robot.

  • Problem

    Existing benchmarks rarely combine GPU-parallel scale with heterogeneous manipulation tasks and standardized multi-task RL evaluation.

    Source: §1 · pp. 1–2

  • Method

    Hebero benchmarks one policy across 40 tasks, while DGPO reuses demonstrations for dense tracking rewards and asymmetric value learning in a shared PPO stack.

    Source: §1 · p. 2

  • Results

    90.1% state-input Mean SR: IW-ABC exceeds FAMO-ABC’s 82.3% by 7.8 percentage points; the visual policy reaches 93.5% mean success.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    A jointly trained state-based policy achieves 82.5% success on four physical Piper tasks without weight updates.

    Source: §1 · p. 2, §6 · p. 9

    IW-ABC’s time-indexed action targets can misalign under trajectory drift despite the tracking reward.

    Source: §6 · p. 9

Abstract

from arXiv · show

GPU-parallel simulation provides abundant robot interaction, but existing benchmarks rarely combine this scale with heterogeneous manipulation tasks and standardized multi-task RL evaluation. We introduce Hebero (Heterogeneous Benchmark for Robot Learning), a GPU-parallel Isaac Lab benchmark that enables efficient joint training and evaluation of a single policy across all 40 heterogeneous tasks. Scaling experiments show that increasing parallel replicas per task improves success under a fixed wall-clock budget. To support learning with sparse rewards and limited demonstrations, we propose Demonstration-Guided Policy Optimization (DGPO), which reuses demonstrations for dense tracking rewards and asymmetric value learning. Its shared stack supports controlled comparisons of learner-specific demonstration interfaces within PPO. Within DGPO framework, we introduce IW-ABC, which uses a lightweight per-task learning progress signal to coordinate adaptive behavior cloning (ABC), relaxing demonstration guidance with task progress, and importance weighting (IW), emphasizing lagging tasks in PPO updates. With 50 demonstrations per task, IW-ABC achieves 90.1% state-input mean success, outperforming the strongest baseline FAMO-ABC by 7.8 percentage points. Its visual counterpart reaches 93.5% mean success. Real-world experiments further demonstrate that a single multi-task policy trained in simulation can successfully perform four tasks on a physical Piper robot. The project page is available at https://hebero-rl.github.io/.

1 INTRODUCTION

Hebero combines heterogeneous GPU-parallel manipulation with joint training of one policy across 40 tasks, while DGPO provides a shared demonstration-guided PPO stack. IW-ABC improves state-input success over the strongest baseline, and the paper also reports scaling gains and physical-robot transfer.

1 INTRODUCTION

  • Hebero brings all 40 LIBERO tasks into one Isaac Lab GPU-parallel process with state and visual inputs, demonstrations, and standardized evaluation.The benchmark supports joint training and evaluation of multiple multi-task RL baselines.Source: §1 · pp. 1–2
  • DGPO reuses demonstrations for dense tracking rewards and asymmetric value learning, with a shared stack for controlled comparisons of demonstration interfaces under PPO.Reference learners include BC initialization, DAPG-style regularization, and an RFCL-inspired reset curriculum.Source: §1 · p. 2
  • IW-ABC uses per-task learning progress to relax adaptive behavior cloning guidance and emphasize lagging tasks in PPO updates.The same lightweight signal coordinates both imitation guidance and online task weighting.Source: §1 · p. 2
  • Across 40 tasks and 1,600 environments, Hebero delivers approximately 7.5k simulation steps per second on one L20 GPU, 3.4 times the throughput of a 96-core MuJoCo baseline.Increasing replicas per task improves success within a common wall-clock budget.Source: §1 · p. 2
  • 90.1% state-input Mean SR and 70.0% Long SR: IW-ABC improves over FAMO-ABC’s 82.3% Mean SR and 53.3% Long SR.The visual policy reaches 93.5% mean success; a jointly trained four-task policy achieves 82.5% success on a physical Piper robot without weight updates.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work supplies task diversity, GPU acceleration, demonstration use, and multi-task optimization methods separately. Hebero and DGPO bring these strands together, while IW-ABC combines task prioritization with dynamic teacher guidance.

2.1 GPU-PARALLEL MULTI-TASK ROBOT LEARNING

  • Hebero brings 40 LIBERO tasks into one heterogeneous GPU-parallel process for demonstration-guided joint RL, with state and visual policies under common training and evaluation interfaces.Earlier benchmarks offered task diversity, demonstrations, or GPU simulation in different combinations, but not this unified process.Source: §2.1 · p. 3

2.2 DEMONSTRATION-GUIDED AND MULTI-TASK REINFORCEMENT LEARNING

  • DGPO organizes varied demonstration uses through a shared trainability stack and learner-specific interfaces.Prior approaches use demonstrations for policy initialization or regularization, offline data in online RL, reset curricula, dense rewards, and other training signals.Source: §2.2 · p. 3
  • IW reweights PPO losses using lightweight online task-success statistics under fixed rollout allocation, and IW-ABC combines this prioritization with dynamic teacher guidance.The signal is independent of reward scale and separate from demonstration interfaces; it adapts teacher supervision to task and overall training progress.Source: §2.2 · p. 3

3 THE HEBERO BENCHMARK

Hebero defines a single task-conditioned policy evaluated across 40 heterogeneous tasks, implemented through a GPU-parallel benchmark and standardized learning interfaces. Its evaluation protocol resamples object placements to assess robustness beyond replaying demonstration starts.

3 THE HEBERO BENCHMARK

  • Hebero separates its benchmark definition from DGPO, which supplies a demonstration-guided approach using the benchmark’s accompanying trajectories.The benchmark specifies a multi-task RL problem, GPU-parallel environment construction, and standardized interaction and evaluation.Source: §3 · pp. 3–4

3.1 PROBLEM SETTING

  • The problem is to learn one task-conditioned policy over tasks that may differ in scenes, dynamics, rewards, and success predicates but share action space, horizon, and robot embodiment.Hebero uses a uniform distribution over 40 tasks and evaluates on those same tasks rather than a held-out set.Source: §3.1 · p. 4

3.2 BENCHMARK CONSTRUCTION

  • Hebero routes task-specific operations through one Isaac Lab simulator, renderer, rollout buffer, and policy update, enabling joint training across all 40 tasks.Offline task descriptors specify scene assets, reset rules, and success predicates.Source: §3.2 · p. 4

3.3 STANDARDIZED LEARNING INTERFACE

  • The standardized action interface uses six dimensions for relative end-effector translation and rotation and one for the binary gripper; actor observations combine proprioception, previous action, and task encoding.State actors receive an object–target pose representation, while visual actors instead use pooled frozen-ViT features from third-person and wrist RGB observations.Source: §3.3 · p. 4

3.4 EVALUATION PROTOCOL

  • Evaluation independently resamples object x and y positions within bounds from 50 demonstration starts, with collision rejection, to test placement robustness rather than trajectory memorization.Reported measures include Mean SR, suite-level SR, Long SR, task coverage at 80% success, and SR-AUC over PPO iterations.Source: §3.4 · p. 5

4 DGPO: A DEMONSTRATION-GUIDED MULTI-TASK TRAINING FRAMEWORK

DGPO makes joint learning across Hebero’s 40 tasks practical under sparse goal rewards by combining demonstration guidance with task-conditioned PPO. Its shared components support controlled comparisons of learner-specific demonstration interfaces, including IW-ABC’s adaptive guidance and task balancing.

4 DGPO: A DEMONSTRATION-GUIDED MULTI-TASK TRAINING FRAMEWORK

  • DGPO couples task-conditioned PPO with demonstration-based reward, reset, and critic guidance to address sparse-reward learning across Hebero’s 40 tasks.The framework separates shared training components from learner-specific demonstration interfaces.Source: §4 · p. 5

4.1 SHARED TRAINING COMPONENTS

  • The shared stack uses a synchronized demonstration command stream and tracking reward for both robot motion and task-relevant world states, alongside privileged critic inputs.The command stream maintains each environment’s task, demonstration, cursor, and trajectory references for reuse across components.Source: §4.1 · p. 5

4.2 REFERENCE LEARNERS AND DEMONSTRATION INTERFACES

  • Reference learners hold DGPO’s shared components fixed while varying the demonstration interface; the RFCL entry adds a reset curriculum, while PPO adds no demonstration mechanism.Interfaces include offline actor initialization, an offline likelihood regularizer, live time-aligned action targets, and demonstration-state resets.Source: §4.2 · p. 6, §4.3 · p. 6

4.3 IW-ABC REFERENCE RECIPE

  • IW-ABC uses one lightweight learning-progress signal to relax imitation pressure and prioritize lagging tasks, coordinating adaptive behavior cloning with importance weighting.The progress signal is computed from training rollouts using per-task success-rate moving averages, without additional evaluation.Source: §4.3 · p. 6
  • ABC relaxes action guidance as task success and training time increase, allowing early-solved tasks to relinquish imitation before the global schedule ends.Time annealing also prevents persistently difficult tasks from remaining indefinitely at maximum guidance.Source: §4.3 · p. 6
  • Tasks below the multi-task mean receive larger importance weights, and minibatch-normalized task weights scale all three PPO terms.The three terms are value-regression loss, policy loss, and policy entropy.Source: §4.3 · p. 7

5 EXPERIMENTS

The experiments assess benchmark scaling, compare PPO-based learners within DGPO, trace IW-ABC’s task-wise learning dynamics, and test joint-policy deployment on a physical robot.

5.1 SCALING EFFICIENCY OF THE HEBERO RL BENCHMARK

  • Hebero has higher simulation throughput than the evaluated MuJoCo configuration with practical GPU memory, and more replicas per task improve success and reach matched reward sooner on one 48 GB L20 GPU.Figure 3 evaluates mean success across all 40 tasks at a common training-budget endpoint and plots mean training reward against wall-clock time.Source: §5.1 · p. 7, Figure 3 · p. 7

5.2 CONTROLLED LEARNER COMPARISON WITHIN DGPO

  • With comparable complexity and throughput, IW-ABC outperforms FAMO-ABC; adding IW to ABC improves both Mean SR and Long SR, unlike IW paired with PPO.ABC, IW-ABC, and Vis IW-ABC cover 4, 7, and 9 of the ten Long tasks, respectively.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 LEARNING DYNAMICS OF IW-ABC

  • IW assigns greater PPO weight to slower-learning Goal and Long tasks until their success approaches the multi-task mean, while ABC coefficients reach a residual floor and relax demonstration-action constraints.The dynamics are traced using training success-rate EMAs and reconstructed importance-weighting and behavior-cloning coefficients.Source: §5.3 · pp. 8–9, Figure 5 · p. 9

5.4 MULTI-TASK DEPLOYMENT ON A PHYSICAL ROBOT

  • A jointly trained state-input policy achieved 82.5% overall success across 20 physical trials per task on four representative Piper tasks, with its weights fixed during deployment.The deployment used an external model-based perception pipeline; Figure 6 shows task rows and episode progress columns.Source: §5.4 · p. 9, Figure 6 · p. 9

6 CONCLUSION AND LIMITATIONS

Hebero supports joint GPU-parallel RL across 40 LIBERO tasks, and the conclusion identifies both a limitation of IW-ABC’s action targets and a restricted evaluation scope.

6 CONCLUSION AND LIMITATIONS

  • IW-ABC’s time-indexed action targets can misalign under trajectory drift despite the tracking reward, and training and evaluation are limited to LIBERO and RoboTwin.The authors identify extending the benchmark and DGPO to more challenging tasks and larger task sets as future work.Source: §6 · p. 9

A HEBERO BENCHMARK SPECIFICATION AND IMPLEMENTATION

This appendix specifies the benchmark independently of its learners, covering task compilation, heterogeneous simulation, action control, observations, rewards, success, and evaluation interfaces.

A.1 TASK COMPILATION AND HETEROGENEOUS SIMULATION

  • LIBERO BDDL tasks are compiled into cached JSON descriptors containing instructions, scenes, assets, spawn regions, goal predicates, and thresholds that govern task interfaces and logging.The original BDDL path is retained for provenance.Source: §A.1 · p. 14
  • MuJoCo MJCF assets are converted offline to cached USD files, while runtime descriptors resolve shared assets and retain task-specific placement and randomization.Environments instantiate task-assigned assets within one shared simulator and renderer, and the environment-to-task map routes resets, demonstrations, goal predicates, and statistics.Source: §A.1 · p. 14
  • The canonical benchmark contains ten tasks in each of the Goal, Object, Spatial, and Long suites; focused encoding can reduce input dimensions, while canonical encoding supports checkpoint loading across subsets.Task success is the conjunction of live-state relational, articulation, and activation predicates, supplying the common sparse-reward event and evaluation outcome.Source: §A.1 · p. 14

A.2 ACTION AND CONTROLLER INTERFACE

  • The normalized seven-dimensional action commands relative end-effector motion through an operational-space controller, with its final component opening or closing the gripper.Translation and axis-angle rotation are scaled by separate factors.Source: §A.2 · p. 15

A.3 ACTOR OBSERVATIONS AND TASK ENCODING

The appendix specifies actor observations and task encoding, joint-limit penalties, asymmetric critic inputs and visual features, time-indexed action targets, and minibatch weighting schedules.

A.3 ACTOR OBSERVATIONS AND TASK ENCODING

  • The default task encoding is a 36-dimensional multi-hot vector combining six shared subtask indicators with 30 dedicated task-ID bits, and the actor also receives a pose buffer with fixed entity slots.State and visual actors share task context and proprioception; their full-configuration inputs contribute 300 and 450 features, respectively.Source: §A.3 · p. 15, Table 3 · p. 15, Figure 7 · p. 16

A.4 JOINT-LIMIT PENALTIES

  • The joint-limit penalties use robot joint position bounds and velocity limits, with positive-part penalties summed across all joints.The reward coefficients are listed in table 4.Source: §A.4 · p. 16

B.1 ASYMMETRIC CRITIC INPUTS AND VISUAL FEATURES

  • The asymmetric critic receives a normalized demonstration cursor, while visual features come from frozen Tiny ViT patch tokens pooled by separately trainable actor and critic attention queries.The actor and critic apply the same pooling operation to the frozen tokens, with separate attention pools and MLPs trained by PPO.Source: §B.1 · pp. 16–17

B.2 TIME-INDEXED ACTION TARGETS

  • ABC uses direct time indexing for demonstration action targets, avoiding online state matching, phase estimation, or optimal-transport alignment.The cursor advances one control step at a time; demonstration-initial-state resets and a tracking reward support alignment, while imitation pressure decreases with task success and training time.Source: §B.2 · p. 17

B.3 IW-ABC MINIBATCH WEIGHTING AND SCHEDULES

  • Minibatch task weights are normalized to unit mean, preserving average loss scale while changing relative task contributions; weighted losses use one shared backward pass.The appendix figure visualizes the behavior-cloning and importance-weighting schedules.Source: §B.3 · p. 17, Figure 8 · p. 17

B.4 FAMO-ABC IMPLEMENTATION

FAMO-ABC adapts FAMO’s critic weighting to the shared PPO stack while retaining ABC. The reference-learner specifications describe offline behavior cloning, DAPG regularization, and an RFCL reset curriculum.

B.4 FAMO-ABC IMPLEMENTATION

  • FAMO-ABC retains ABC but replaces IW-weighted value loss with FAMO’s critic objective; its PPO policy surrogate and entropy retain uniform sample weights.FAMO task weights are updated from value-loss progress, with an additional critic forward pass and task-logit update after each PPO minibatch.Source: §B.4 · pp. 17–18

B.5 REFERENCE LEARNER SPECIFICATIONS

  • The reference learners use a shared preprocessed demonstration dataset, retaining trajectory and timestep indices for methods that use demonstration states or observation–action pairs.BC pretraining and DAPG use observation–action pairs, while RFCL uses stored states for curriculum resets.Source: §B.5 · p. 18
  • Offline BC trains an architecture-matched actor on the full demonstration dataset with mean-squared action error; BC→PPO transfers actor weights and starts online PPO with a fresh critic and optimizer.Offline pretraining is additional to BC→PPO’s online budget.Source: §B.5 · p. 18
  • DAPG adds a demonstration log-likelihood term to policy gradients, with an initial coefficient that decays per PPO update.The demonstration term uses observation–action pairs from the shared preprocessed dataset.Source: §B.5 · p. 18
  • RFCL uses an independent reverse-reset curriculum per task, then samples uniformly from that task’s demonstration initial states.During the reverse phase, environments pair with demonstration trajectories and sample reset cursors; after frontier mastery, cursors move backward, and resets switch to all demonstration starts once active cursors reach zero.Source: §B.5 · pp. 18–19

C REPRODUCIBILITY CONFIGURATION

The reproducibility configuration groups benchmark, architecture, optimization, reward, and learner-schedule settings by component. Supplementary analyses examine where task importance weighting is applied and how IW and ABC weights evolve across tasks.

C REPRODUCIBILITY CONFIGURATION

  • Table 4 groups benchmark settings, reference-policy architectures, optimization parameters, and learner-specific schedules by component.The tracking weights instantiate the component weights in the DGPO reward.Source: Table 4 · p. 19, C REPRODUCIBILITY CONFIGURATION · p. 19
  • The shared state/visual IW-ABC configuration specifies task-importance and ABC schedules, including a 200,000-control-step ABC horizon and a residual minimum BC weight of 0.1.The state actor uses hidden widths [512, 256, 128]; visual reference policies use a frozen Tiny ViT with third-person and wrist RGB views.Source: C REPRODUCIBILITY CONFIGURATION · pp. 20–21
  • The configuration also records the shared PPO horizon and optimizer settings, offline BC and DAPG schedules, RFCL reset-curriculum parameters, and visual-backbone specifications.The shared PPO horizon is 30,000 online iterations; offline BC uses 30,000 optimizer steps.Source: C REPRODUCIBILITY CONFIGURATION · pp. 20–21

D SUPPLEMENTARY IW-ABC ANALYSES

D.1 TASK WEIGHTING IN THE ACTOR AND CRITIC

The ablation compares policy-side task weighting with the default that also weights value regression, while keeping the ABC demonstration term unchanged. Weighting both actor and critic yields higher reported success and task coverage.

D.1 TASK WEIGHTING IN THE ACTOR AND CRITIC

  • The two PPO variants differ in whether task IW weights value regression; the ABC demonstration term is unchanged and omitted from the displayed objectives.The ablation retains the same learner family and other shared settings.Source: §D.1 · p. 21
  • Entropy follows the actor weight in both variants, and the authors adopt weighting of value regression as the default.Source: §D.1 · pp. 21–22
  • 11.9 percentage points higher Mean SR and 20.0 points higher Long SR, with task coverage rising from 31 to 35 of 40, when IW weights actor and critic rather than actor alone.The table compares policy-side weighting against the default that also weights value regression.Source: §D.1 · pp. 21–22, Table 5 · p. 22

D.2 TASK-LEVEL IW AND ABC DYNAMICS

Across the illustrated Vis IW-ABC tasks, adaptive behavior cloning relaxes demonstration guidance with task progress and training time, while importance weighting maintains priority for lagging tasks. Deployment details describe the physical Piper observation and perception pipeline.

D.2 TASK-LEVEL IW AND ABC DYNAMICS

  • G7 learns early and reduces both BC coefficient and relative PPO weight; L2 improves during annealing, while G5 and L5 retain high IW when lagging.G5 success rises around 17k iterations before IW falls as it approaches the multi-task mean; L5 remains unsolved within the observed budget and retains high IW. Figure 9 plots success EMA, raw IW, and BC coefficient for the four tasks, with ABC annealing ending at 200,000 control steps.Source: §D.2 · p. 22, Figure 9 · p. 22

E REAL-WORLD DEPLOYMENT DETAILS

  • FoundationPose estimates object poses from RGB-D observations, while the target pad uses color segmentation and depth back-projection; a frozen actor outputs joint-position targets and gripper commands in closed-loop control.Camera calibration and object-frame alignment express pose estimates in the robot base frame.Source: E REAL-WORLD DEPLOYMENT DETAILS · p. 22
  • The Piper actor input is 69-dimensional, combining a 42-dimensional object–target buffer with task identity, previous action, joint positions, and joint velocities.The object–target buffer has six slots, each storing a 3D position and a four-component quaternion; inactive task slots are zero.Source: E REAL-WORLD DEPLOYMENT DETAILS · p. 22
Loading 2606.03335v3…