PaperlayerView exact revision on arXiv

Source-linked AI summary

MASkillBlender: Decentralized Whole-Body Coordination for Multi-Humanoid Loco-Manipulation via Skill Blending

Yifan Hu, Luhang Hong, Mingkang Long, Danning Wang, Chengfeng Jia, Rong Su, Junjie Fu, Guanghui Wen

arXiv:2610.01102v1cs.ROcs.LGcs.MA

TL;DR

Multi-humanoid whole-body coordination remains difficult because existing approaches may require motion references, task-specific design, or substantial reward engineering. MASkillBlender composes reusable pre-trained skills with a shared decentralized policy and uses permutation-based augmentation for homogeneous teams. Simulations report coordinated behavior and strong task performance across tasks and humanoid embodiments.

  • Problem

    General multi-humanoid coordination remains challenging because single-humanoid skills do not encode inter-agent coordination, while existing methods can require motion references or task-specific design.

    Source: §1 · p. 2

  • Method

    MASkillBlender trains a shared decentralized high-level policy to compose fixed pre-trained skills using local observations and task-level rewards, with permutation-based augmentation for homogeneous teams.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    MASkillBlender achieves the highest success rate on all three Unitree H1 tasks, and simulations report versatile coordinated behaviors across tasks and embodiments.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    The framework demonstrates decentralized multi-humanoid coordination across diverse simulated tasks and humanoid embodiments using reusable skills and task-level rewards.

    Source: §1 · p. 2, §6 · p. 9

    The formulation assumes homogeneous teams and shared local policies, and policies are trained and evaluated primarily in simulation with state-based observations.

    Source: §6 · p. 9

Abstract

from arXiv · show

Coordinated multi-humanoid loco-manipulation is promising yet challenging due to high-dimensional whole-body control, decentralized decision making, and scalability. While recent reinforcement learning methods have improved single-humanoid whole-body control, extending them to the multi-humanoid setting remains nontrivial and often requires substantial reward engineering or task-specific design. We propose MASkillBlender, a general multi-agent reinforcement learning framework to achieve decentralized multi-humanoid whole-body coordination. By learning a shared decentralized high-level policy over reusable pre-trained single-humanoid skills, MASkillBlender enables coordinated behaviors using only task-level rewards, without requiring task-specific motion references. To improve learning efficiency, we further introduce a permutation-based data augmentation strategy for homogeneous multi-humanoid systems, and theoretically show that the permuted samples preserve the policy-gradient direction of the original samples under the homogeneous Markov game formulation. We evaluate MASkillBlender on multiple multi-humanoid coordination tasks across two humanoid embodiments. Simulation results demonstrate that the proposed framework consistently achieves strong task performance and enables coordinated behaviors across different tasks and humanoid embodiments.

1 Introduction

Multi-humanoid coordination is challenging because agents have complex interdependencies, while existing approaches often rely on motion references or teammate observations that add requirements or complicate scalability. MASkillBlender learns decentralized coordination by composing reusable single-humanoid skills with task-level rewards.

1 Introduction

  • Multi-humanoid control adds complex inter-agent dependencies to whole-body control, making general coordination methods an important step toward scalable humanoid teams.Humanoid loco-manipulation has advanced, but many real-world tasks require multiple humanoids to coordinate.Source: §1 · p. 2
  • Existing multi-humanoid methods often use motion references that require additional data and supervision, while motion objectives may compete with task rewards.Including teammate observations can improve coordination but may hinder scaling to larger teams and complicate robust deployment.Source: §1 · p. 2
  • Pre-trained single-humanoid skills reduce downstream low-level learning, but do not encode the inter-agent coordination required by multi-humanoid tasks.Composing these skills across agents under coupled task dynamics remains challenging.Source: §1 · p. 2
  • MASkillBlender learns a shared decentralized high-level policy over fixed pre-trained skills using local observations and task-level rewards, without task-specific motion references.The paper also introduces permutation-based augmentation, theoretically showing that permuted samples preserve the policy-gradient direction, and evaluates three tasks across two embodiments.Source: §1 · p. 2

2 Related Works

Prior humanoid-control work includes reference-conditioned and task-driven single-agent methods, alongside multi-humanoid methods that often depend on motion supervision or task-specific designs. The paper frames general, reference-free coordination with lightweight reward engineering as an open challenge.

2 Related Works

  • Single-humanoid RL methods either track motion references from MoCap or teleoperation, or use task-driven decomposition and reusable low-level skills.Multi-humanoid extensions of reference-conditioned methods require coordinated motion references that are harder to acquire.Source: §2 · pp. 2–3
  • Cooperative multi-humanoid approaches include retargeted multi-person motion, single-human motion priors, and learning object transport without motion supervision.CooHOI transfers single-agent interaction skills; TeamHOI uses masked motion priors and teammate-aware policies for carrying with varying team sizes.Source: §2 · p. 3
  • Existing cooperative methods are often task-specific or rely on motion retargeting, engineered rewards, and explicit teammate observations, leaving general reference-free coordination challenging.CDA-PPO is designed for two humanoids and uses extensively engineered locomotion and cooperation rewards.Source: §2 · p. 3

3 Preliminaries

The paper formulates multi-humanoid control using homogeneous Markov games and specifies local observations, whole-body actions, and a shared team reward. This formulation provides the setting for its decentralized coordination problem.

3.1 Homogeneous Markov Games

  • Homogeneous Markov games model agents with identical local state and action spaces, permutation-preserving joint rewards, and permutation-invariant transitions.The formulation also assumes shared observation spaces and permutation-preserving joint observations.Source: §3.1 · p. 3

3.2 Problem Formulation

  • Figure 2 depicts the framework’s skill selection and blending under a shared decentralized high-level policy, trained with MAPPO and permutation-based augmentation.The overview spans coordination across different humanoid embodiments without task-specific motion references or extensive reward engineering.Source: Figure 2 · p. 4
  • Each humanoid observes its proprioceptive state and local task goal, and acts through target whole-body joint positions mapped to motor torques by PD control.All humanoids share the same team reward, and the objective is to maximize expected discounted return from the initial-state distribution.Source: §3.2 · p. 4

4 Methodology

MASkillBlender combines fixed pre-trained primitive skills with a shared decentralized high-level policy, then augments training batches with agent permutations under homogeneous Markov-game assumptions. MAPPO trains the policy using centralized training and decentralized execution.

4.1 Decentralized Skill-Blended Control Policy

  • A shared local policy selects task-relevant skills, then uses each humanoid’s local observation to produce skill-specific goals and weights that blend primitive actions.The reusable library includes Walking, Reaching, and Squatting; for box pushing, the example selects Walking and Reaching.Source: §4.1 · p. 5
  • Fixed pre-trained primitive skills shift downstream learning toward high-level skill composition and inter-agent coordination rather than acquiring low-level whole-body motor skills from scratch.The framework uses local observations and task-level rewards; primitive skills remain fixed during high-level policy training.Source: §4 · p. 4, §4.1 · p. 5

4.2 Permutation-Based Data Augmentation

  • Permutation-based augmentation addresses the increased simulation cost of multi-humanoid RL by adding permuted versions of rollout samples to each training minibatch.The method exploits the permutation properties of homogeneous Markov games; its stated goal is improved data efficiency.Source: §4.2 · p. 5, §4.3 · p. 6
  • Under homogeneous Markov-game assumptions, a collected sample and its permuted counterpart yield the same policy gradient.The paper uses this result to justify permutation-based augmentation for on-policy MARL and high-dimensional multi-humanoid coordination.Source: §4.2 · p. 6

4.3 MASkillBlender

  • MAPPO trains the shared decentralized high-level policy with a centralized critic, while each humanoid executes the shared policy from its local observation.The critic receives the global state during training only.Source: §4.3 · p. 6

5 Experiments

Experiments evaluate MASkillBlender across Carry, Push, and Move, with ablations, deployment robustness tests, simulator transfer, and additional object and embodiment settings. MASkillBlender achieves the highest success rates across the three Unitree H1 tasks and shows effective transfer and generalization in the reported evaluations.

5.1 Experimental Setup

  • The H1 evaluation covers contact-rich two-humanoid Carry and Push and collision-avoiding three-humanoid Move, comparing MASkillBlender with centralized adapted SkillBlender and reference-based TeamHOI.Source: §5.1 · p. 7
  • Evaluation uses task-reward Return and task-specific errors and success rates, with success thresholds of Errorbox ≤0.1 for Carry, ≤0.2 for Push, or Errorpos ≤0.03 without Move collisions.Source: §5.1 · pp. 7–8

5.2 Main Results

  • MASkillBlender achieves the highest success rate on all three Unitree H1 tasks and highest final returns; TeamHOI fails to learn Move.The quantitative comparison is reported in Table 1.Source: Table 1 · p. 8, §5.2 · p. 8
  • On Move, TeamHOI's lower Errorhead reflects target orientation without reaching the target; on Carry, MASkillBlender transports coordinately while TeamHOI moves the box before coordinated locomotion.Figure 4 presents representative key frames for the Carry comparison.Source: Figure 4 · p. 8, §5.2 · p. 8

5.3 Ablation Studies

  • Permutation augmentation generally speeds learning and improves final returns, while removing Squatting from Carry or Walking from Push degrades learning.The reported augmentation drops occur across all three tasks; both skill-removal variants show degraded learning.Source: §5.3 · pp. 8–9

5.4 Policy Robustness under Deployment Variations

  • Without additional training or fine-tuning, policies maintain coordination under localization errors and control or communication delays; task decomposition also extends Carry and Push to asymmetric starts without policy training.Reduced reliance on inter-robot communication helps mitigate communication-delay effects.Source: §5.4 · p. 9

5.5 Sim2Sim Transfer

  • After transfer from Isaac Gym to MuJoCo, MASkillBlender preserves strong performance on Carry and Move; Carry uses brief broader-domain-randomization training, while Move transfers directly.Source: §5.5 · p. 9

5.6 Additional Multi-Humanoid Coordination Experiments

  • MASkillBlender supports smaller-object coordination in a box-exchange task and generalizes across Carry, Push, and Move to Unitree G1 without substantial modifications.Source: §5.6 · p. 9

6 Conclusion and Discussion

MASkillBlender coordinates homogeneous humanoid teams through a shared decentralized policy over reusable skills, with simulated results spanning tasks and embodiments. The discussion also identifies scope boundaries, while the appendix illustrates the homogeneous Markov game formulation and its notation.

6 Conclusion and Discussion

  • Simulations demonstrate versatility across coordination tasks and embodiments, plus robustness under deployment variations and cross-simulator transfer.The framework uses local observations and task-level rewards to coordinate reusable pre-trained skills.Source: §6 · p. 9
  • The current formulation assumes homogeneous teams and a shared local policy, and primarily evaluates state-based policies in simulation.The authors identify onboard perception, state estimation, contact-rich sim-to-real transfer, hardware safety, and primitive-skill coverage as remaining concerns.Source: §6 · p. 9

A Notations

  • The notation defines an N-element permutation and applies it by reordering the entries of a list.It also introduces a pointwise Softmax mapping, though the supplied passage does not include its complete formula.Source: A Notations · p. 13

B An Example of H-MGs

  • In the box-pushing example, the team reward and physics-based transitions are permutation invariant, while the joint observation is permutation-preserving.Each humanoid observes proprioception, box state, and goal information; the shared reward includes box-to-target and wrist-to-assigned-contact-point distances.Source: B An Example of H-MGs · p. 13

C Primitive Skill Specifications

The experiments use walking, reaching, and squatting as reusable primitive skills, with goals tailored to locomotion, wrist targets, and body height.

C Primitive Skill Specifications

  • Walking tracks commanded base motions for locomotion, with goals specified by planar linear velocity and yaw angular velocity.The H1 skills come from Humanoid-SkillBlender; G1 skills are trained in-house.Source: C Primitive Skill Specifications · p. 13
  • Reaching drives both wrists toward designated 3D targets while keeping the body grounded, using each wrist’s relative offset to its target as its goal.The detailed specifications are referenced in Appendix C of Kuang et al. [7].Source: C Primitive Skill Specifications · p. 13
  • Squatting lowers or raises the body to a desired base height, with the goal given by the relative height error between current and target root heights.Source: C Primitive Skill Specifications · p. 13

D Proofs

The proof uses permutation symmetry in the homogeneous Markov game to relate policies and value functions, while the training algorithm augments rollout minibatches before MAPPO updates.

D Proofs

  • Algorithm 1 applies permutation-based augmentation to sampled minibatches before updating policy and value parameters with MAPPO.Rollouts are collected from parallel environments using the decentralized policy, and advantage estimates are computed from the trajectories.Source: D Proofs · p. 14
  • Permutation-preserving rewards and observations support a permutation-invariant joint policy, value function, and action-value function.The proof passage states the team reward is permutation invariant; the subsequent argument establishes invariance of the joint policy, V, and Q.Source: D Proofs · pp. 13–14
  • The proof proceeds through coordinatewise parameter arguments and value-function invariance before concluding the stated result.The supplied proof excerpts introduce standard basis vectors, refer to a coordinatewise relation, and invoke invariance of V and Q.Source: D Proofs · p. 14

E Training Procedure of MASkillBlender

The training procedure is summarized in Algorithm 1, and the observation/action specification describes local inputs, policy output dimensions, and conversion of joint-position targets to torques.

E Training Procedure of MASkillBlender

F.1 Observations and Actions

  • Each local observation combines proprioceptive state and a task goal, yielding dimension 3d + 6 + dtask.The proprioceptive state includes joint angles, joint velocities, previous actions, base angular velocity, and base Euler angles.Source: §F.1 · p. 15
  • The decentralized high-level policy outputs skill goals and whole-body target joint positions, which a PD controller converts into torques.The stated output dimensionality is the sum across primitive skills of d + dc.Source: §F.1 · p. 15

F.2 Task Specifications

The task specifications vary the number of humanoids, interaction goals, and primitive skills while using a small set of task-completion reward terms. The experiments use H1 and G1 embodiments with fixed pre-trained skills and task-specific implementation choices.

F.2 Task Specifications

  • Carry and Push coordinate two humanoids around a shared box using Walking and Reaching, with Squatting also used for Carry; Move assigns three humanoids targets and uses Walking.Each task uses only a few reward terms directly related to task completion, avoiding extensive reward engineering.Source: §F.2 · p. 15

F.3 Humanoid Model Details

  • The experiments use a 19-DoF Unitree H1 and a modified 21-DoF Unitree G1, excluding hand links; G1 also has virtual wrist-query joints that are not actuated.The G1 model is derived from the official 23-DoF configuration.Source: §F.3 · p. 16

F.4 Implementation Details

  • MASkillBlender keeps pre-trained primitive policies fixed and trains actor and critic MLPs; training uses all permutations and 4096 parallel environments for Carry/Push and 2048 for Move.The listed H1 primitive policies have three hidden layers of sizes [512, 256, 128].Source: §F.4 · p. 16
  • The adapted TeamHOI baseline retargets human motion references to H1 and uses AMP alongside task rewards, while matching MASkillBlender’s task-level observation information.Carry and Move use Full AMP; Push uses Masked AMP.Source: §F.4 · pp. 16–17
  • A MASkillBlender H1 training trial takes approximately 50 hours for Carry, 76 for Push, and 35 for Move on a single GPU.The corresponding adapted TeamHOI times are approximately 60, 84, and 55 hours.Source: §F.4 · p. 17

F.5 Ablation Studies

The ablations examine permutation-based data augmentation and primitive-skill selection. Removing selected skills substantially degrades learning on the affected tasks.

Permutation-Based Data Augmentation

  • The quantitative ablation results for permutation-based data augmentation are reported in Table 4 for Unitree H1 tasks.The supplied passage identifies the table and its scope but does not state the cell values.Source: §F.5 · p. 17

Primitive Skill Selection

  • Figure 5 presents learning curves for the primitive-skill selection ablation on Unitree H1.The cited figure caption identifies the ablated skills as Squatting in Carry and Walking in Push.Source: §F.5 · p. 17, Figure 5 · p. 18
  • Removing Squatting from Carry or Walking from Push substantially degrades learning performance.Figure 5 presents the primitive-skill ablation on Unitree H1.Source: Figure 5 · p. 18

F.6 Policy Robustness under Deployment Variations

MASkillBlender is evaluated under deployment variations, across simulators, on a two-humanoid box exchange, and on the G1 embodiment. The reported results include effective coordination under noise and delays, successful transfer, staged box exchange, and cross-embodiment learning.

F.6 Policy Robustness under Deployment Variations

  • For asymmetric Carry and Push starts, walking brings each humanoid to a predefined pose, an early arrival waits, and the learned task policy then takes over without additional policy training.The squatting skill maintains a stable waiting posture; Figure 6 illustrates the deployment sequence.Source: Figure 6 · p. 19, §F.6 · p. 19
  • Under introduced deployment noise and delays, MASkillBlender maintained effective coordination across all three tasks, with a more noticeable success-rate drop on Push.The same checkpoint was evaluated for 50 rollouts without additional training or fine-tuning; broader domain randomization could potentially mitigate the Push drop.Source: §F.6 · p. 19

F.7 Sim2Sim Transfer

  • 92% of MuJoCo Carry episodes had final box-position error ≤0.1 m, and Move achieved a 99% success rate.Both tasks were evaluated over 100 episodes; Move used direct deployment without additional training, while Carry received brief continued training under broader domain randomization.Source: Table 6 · p. 20, §F.7 · p. 20

F.6 Policy Robustness under Deployment Variations

  • During cooperative Carry or Push, each humanoid executes the decentralized policy without explicit teammate-state communication, reducing dependence on continuous communication and sensitivity to latency.The task decomposition uses lightweight communication to determine when both humanoids are ready to enter the cooperative stage.Source: §F.6 · pp. 19–20

F.8 Two-Humanoid Box Exchange

  • In the two-humanoid box exchange, the robots move assigned boxes to an exchange region, switch responsibilities, and deliver the exchanged boxes to their final targets.The task demonstrates coordinated manipulation of smaller objects across two stages.Source: §F.8 · pp. 20–21, Figure 8 · p. 21

F.9 Additional Experiments on Unitree G1

  • On Unitree G1, MASkillBlender learned coordinated behavior across Carry, Push, and Move; Carry and Push box-position errors were comparable to H1, while Move had larger position and heading errors.The overall framework required no substantial modifications, though reward coefficients and training iterations received minor empirical adjustments.Source: §F.9 · p. 21, Table 7 · p. 22

Related papers

Loading 2610.01102v1…