PaperlayerView exact revision on arXiv

Source-linked AI summary

CoHuB: A Simulation Benchmark for Multi-Humanoid Collaboration

Hyunjin Park, Jebeom Chae, Minwoo Park, Sunghyun Park, Hanjun Yoo, Seoyeon Choi, Soochul Yoo, Joohwan Seo, Sarmad Idrees, Jae-Sang Hyun, Jongmin Lee, Roberto Horowitz, Youngwoon Lee, Jongeun Choi

arXiv:2609.34782v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Existing humanoid benchmarks largely focus on single-humanoid skills and do not evaluate multi-humanoid collaboration under egocentric visual observations. COHUB introduces a benchmark, synchronized multi-operator demonstrations, and policy evaluations; results show persistent challenges, including lower success in collaborative phases.

  • Problem

    Existing humanoid benchmarks largely focus on single-robot settings, while multi-agent visual benchmarks have primarily studied non-humanoid tabletop manipulation.

    Source: §1 · p. 2

  • Method

    COHUB provides a simulation benchmark with diverse multi-humanoid tasks, synchronized egocentric VR demonstrations, and evaluations of representative visuomotor policies.

    Source: §1 · p. 2

  • Results

    Across the task suite, policies show lower success in collaborative than non-collaborative phases, and current policies remain limited in multi-humanoid settings.

    Source: §4.3 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    COHUB provides a reproducible platform for developing and evaluating policies across diverse multi-humanoid collaboration patterns.

    Source: §5 · p. 10

Abstract

from arXiv · show

Many physical tasks in human environments require collaboration, from assisting a partner to jointly manipulating an object. Yet, existing humanoid benchmarks largely focus on single-humanoid skills and lack evaluation of multi-humanoid collaboration under egocentric visual observations. We introduce CoHuB (Collaborative Multi-Humanoid Benchmark), a simulation benchmark for multi-humanoid collaboration under egocentric visual observations. CoHuB provides 10 tasks, eight with two humanoids and two with three humanoids, spanning diverse collaboration patterns. We also provide synchronized demonstrations collected through a multi-operator VR teleoperation pipeline, in which each operator controls one humanoid from its egocentric view. Experiments with representative visuomotor policies reveal substantial challenges across different forms of coordinated perception and control. CoHuB provides a foundation for developing and evaluating multi-humanoid collaboration policies.

1 INTRODUCTION

Humanoid research has advanced, but collaboration remains important in human environments and largely outside existing single-robot benchmarks. COHUB addresses this gap with a multi-humanoid simulation benchmark, synchronized demonstrations, and evaluations of representative policies.

1 INTRODUCTION

  • Many human-environment tasks require collaboration, where shared-object forces can disturb balance and success can depend on coordinated roles and timing.Examples include holding a long or heavy object together and opening a door for someone whose hands are full.Source: §1 · p. 2
  • Existing humanoid simulation benchmarks focus on single robots, while multi-agent visual benchmarks have primarily addressed non-humanoid tabletop manipulation.Real-world demonstrations and shared environments for multi-humanoid collaboration are costly and difficult to reproduce, motivating simulation as a scalable alternative.Source: §1 · p. 2
  • COHUB introduces a simulation benchmark with 10 collaborative tasks: eight use two humanoids and two use three, spanning diverse collaboration patterns.The benchmark uses Unitree G1 humanoids with Dex3-1 hands and decomposes tasks into sequential phases for phase-wise evaluation.Source: §1 · p. 2
  • COHUB provides synchronized human demonstrations collected through multi-operator VR teleoperation, with each operator controlling one humanoid from an egocentric view.The paper evaluates representative imitation-learning, VLA, and WAM policies and examines task-phase performance and multi-agent design choices.Source: §1 · p. 2

2 RELATED WORK

Prior benchmarks and methods cover robot learning, humanoids, and multi-agent collaboration, but these strands provide limited support for task-oriented physical collaboration among humanoids under visual observations.

2 RELATED WORK

  • Humanoid simulation benchmarks increasingly cover whole-body interaction but remain primarily single-agent, while visual multi-agent benchmarks mainly use non-humanoid embodiments.Table 1 compares representative benchmarks by whole-body tasks, collaboration, and human teleoperation demonstrations.Source: §2 · p. 2, Table 1 · p. 3
  • Humanoid and multi-agent benchmarks have largely evolved separately, leaving limited support for diverse collaborative humanoid interaction.RoboFactory addresses visual multi-agent benchmarking but focuses on multi-arm embodiments and a narrow range of collaborative interactions.Source: §2 · p. 3
  • Prior multi-humanoid work studies state-based object transportation or paired-motion tracking, rather than task-oriented collaborative loco-manipulation.Multi-robot collaboration methods and pretrained VLA/WAM policies have primarily been demonstrated or evaluated in single-agent or manipulator settings.Source: §2 · p. 3

3 COHUB: A SIMULATION BENCHMARK FOR MULTI-HUMANOID COLLABORATION

COHUB is designed to support demonstration collection, policy learning, and evaluation for multi-humanoid collaboration. Its task taxonomy, multi-operator teleoperation pipeline, and full-task and phase-level protocols structure that support.

3.1 BENCHMARK ENVIRONMENT

  • COHUB uses a unified Isaac Lab environment with shared embodiment, action and observation spaces, and recording pipeline across tasks.Tasks vary in scene assets and success criteria; physics runs at 200 Hz, control at 50 Hz, and camera observations use RTX rendering.Source: §3.1 · p. 4

3.2 TASK DESIGN PRINCIPLES AND TAXONOMY

  • The benchmark classifies tasks by base movement and physical coupling, yielding four categories with two tasks each; the categories do not imply relative difficulty.Base movement distinguishes local adjustments from travel between areas; physical coupling distinguishes brief or absent shared-object contact from sustained simultaneous interaction.Source: §3.2 · p. 4, Figure 2 · p. 5

3.3 TASK SUITE

  • The suite contains eight two-humanoid tasks across the four categories and two additional three-humanoid tasks.The two-humanoid tasks include handover, pouring, joint lifting or carrying, and object transport; the three-humanoid tasks involve moving a table through a doorway or carrying a large table together.Source: §3.3 · pp. 5–6

3.4 MULTI-OPERATOR VR TELEOPERATION AND DATASET

  • COHUB extends VR teleoperation so multiple operators control separate humanoids from egocentric views in one synchronized simulation.The pipeline records states and actions online, then replays trajectories to render synchronized observations; the dataset has 600 demonstrations and 1,320 per-humanoid trajectories.Source: §3.4 · p. 6, Figure 3 · p. 6

3.5 EVALUATION PROTOCOL

  • Evaluation uses full-task rollouts for the main benchmark and a separate phase-level protocol for analyzing collaboration, with shared success criteria.Tasks have three sequential phases and task success requires completing the final phase; phase-level rollouts begin at the evaluated phase using training demonstrations to reduce effects of earlier failures.Source: §3.5 · pp. 6–7
  • The main benchmark evaluates final task success from task beginnings, using one randomly seeded checkpoint and 100 rollouts per task.Table 2 reports task success rates across the suite and distinguishes per-task from multi-task training; LatentToM is evaluated only on two-robot tasks.Source: §3.5 · p. 7, Table 2 · p. 7

4 EXPERIMENTS

The experiments benchmark representative visuomotor policies, analyze performance across collaboration phases, and test how observation scope and policy sharing affect results. Performance varies across policy families and design choices, with collaborative phases consistently more challenging in the evaluated task suite.

4.1 EXPERIMENTAL SETUP AND BASELINES

  • Eight visuomotor policies span standard IL, multi-agent IL, VLA, and WAM families, with training and execution configurations differing across families.VLA and WAM models are fine-tuned in multi-task settings, while standard IL policies are trained separately for each task and humanoid.Source: §4.1 · pp. 7–8

4.2 OVERALL BENCHMARK RESULTS

  • ACT is strongest among IL baselines, Fast-WAM has the highest average success among pretrained models, and VLAs achieve relatively low success rates.LatentToM and GauDP, despite explicitly modeling multi-agent interactions, achieve lower task success than ACT.Source: §4.2 · p. 8
  • Rollouts reveal distinct failures: π0.5 collides during Handover transfers, while Fast-WAM often fails to release the bottle; during transport, VLAs drop objects while Fast-WAM agents move inconsistently.The authors conclude that COHUB requires both capable single-agent behavior and reliable multi-agent collaboration.Source: §4.2 · p. 8
  • GPT-6 Astra reaches 70% on CoPouring and 10% on Pouring, with no successes on the remaining tasks; its settings limit direct comparison with trained baselines.The evaluation uses separate observation, control, and time-budget settings.Source: §4.2 · p. 8

4.3 HOW DOES PERFORMANCE DIFFER BETWEEN NON-COLLABORATIVE AND COLLABORATIVE PHASES?

  • Collaborative-phase success is consistently lower than non-collaborative success across policy families; Fast-WAM reaches 62.7% versus π0.5’s 52.4% on collaborative phases.The phase-level evaluation starts each phase from its beginning state in the same 50 demonstrations.Source: §4.3 · pp. 8–9, Figure 4 · p. 9

4.4 HOW DO MULTI-AGENT DESIGN CHOICES AFFECT PERFORMANCE?

  • Table 3’s configuration effects vary by model: Local Observation + Shared Policy is lowest for ACT and DP, while GR00T N1.7 achieves higher average success with Separate Policies and Local Observation.The authors hypothesize GR00T’s results may reflect distribution shift from its pretraining setting.Source: Table 3 · p. 9, §4.4 · pp. 9–10

5 CONCLUSION

COHUB is presented as a reproducible platform for evaluating multi-humanoid collaboration, with phase-wise analysis and synchronized demonstrations. The supplied technical passages also describe its per-robot controls, teleoperation labels, and egocentric policy inputs.

5 CONCLUSION

  • The authors conclude that current policies remain limited in multi-humanoid settings, especially during collaborative phases, and identify communication and long-horizon planning as future task-suite extensions.They state that successful collaboration requires policies that better respond to other humanoids during interaction.Source: §5 · p. 10

A SIMULATION ENVIRONMENT AND ROBOT SETUP

A.1 ACTION SPACE AND CONTROL

  • Task policies predict 31 absolute joint targets and a base command; a pretrained locomotion controller generates the 12 leg-joint motions from that command.Environment actions are applied at 50 Hz, with four physics steps per control step.Source: §A.1 · p. 16
  • Teleoperation records 32-D wrist, hand, and base inputs that provide 35-D action labels, while policy execution directly predicts joint targets without inverse kinematics.Default policy inputs are egocentric RGB, 43 joint positions, and role-specific language instructions when applicable; additional synchronized signals are recorded but excluded.Source: §A.1 · p. 16, §A.2 · p. 16

B TASK AND DATASET SPECIFICATIONS

The task and dataset specifications cover a ten-task suite spanning two- and three-humanoid settings.

B TASK AND DATASET SPECIFICATIONS

B.1 TASK SCENES AND RANDOMIZATION

This section describes fixed, adapted scenes and task randomization, then specifies phase-based success criteria, role-specific instructions, demonstration splits, and policy inputs.

B.1 TASK SCENES AND RANDOMIZATION

  • Ten tasks use fixed scenes adapted from SceneSmith, with task objects and furniture adjusted; evaluation samples initial object and target offsets from task-specific ranges.The suite includes two-humanoid tasks with low or high base movement and physical coupling, plus three-humanoid tasks such as MoveHouse and BigTable.Source: §B.1 · p. 16, Figure 5 · p. 17, Figure 6 · p. 18

B.2 PHASE DEFINITIONS AND SUCCESS CRITERIA

  • Each task has three sequential scored phases, and later phases count only after their predecessors; final placements require position, tilt, speed, and release-duration thresholds.FrameHang uses more permissive tilt and motion thresholds than the other listed final placements; CartService is collaborative overall despite having only non-collaborative phases.Source: §B.2 · p. 20, Table 6 · pp. 20–21

B.3 LANGUAGE INSTRUCTIONS

  • Tasks provide a shared instruction and one role-specific instruction per robot, with language-conditioned baselines generally using the role-specific version.For example, Handover assigns the transfer and placement roles to different robots, while CartService assigns trolley delivery and bottle placement.Source: §B.3 · pp. 21–22

B.4 DEMONSTRATION SPLIT AND USAGE

  • Each task has 60 demonstrations: the first 50 train the policies and the remaining 10 are used only to monitor validation loss.The same split applies across baselines, and validation demonstrations are excluded from gradient updates.Source: §B.4 · p. 23

C.1 POLICY INPUTS

  • ACT and DP train separately for each task and robot, whereas VLA and WAM policies share parameters across roles and tasks; policies use different observation inputs.Most listed policies use egocentric RGB and 43 joint positions; LatentToM additionally uses a shared third-person view, and GauDP processes agents’ egocentric views jointly.Source: §C.1 · pp. 23–24

C.2 TRAINING CONFIGURATIONS

The training configurations specify checkpoint selection and distinct recipes for standard imitation-learning, multi-agent imitation-learning, and pretrained VLA/WAM policies.

C.2 TRAINING CONFIGURATIONS

  • Evaluation uses each policy’s last training checkpoint, and epoch-based budgets are reported as epochs because optimizer-step counts depend on task trajectory lengths.Batch sizes are global and include gradient accumulation where used.Source: §C.2 · p. 24
  • ACT uses an ImageNet-pretrained ResNet-18 and predicts 50-step action chunks; DP uses a from-scratch ResNet-18 and a 100-step DDPM, executing 20 of 40 predicted actions before replanning.The two standard IL methods also differ in learning-rate schedule and temporal aggregation settings.Source: §C.2 · p. 24
  • LatentToM couples its two policies with a sheaf-consistency loss, while GauDP fine-tunes and freezes a shared pretrained encoder before training a separate diffusion policy for each task.Both predict 40 actions and execute 20 at each inference step.Source: §C.2 · p. 24
  • The VLA models share a LoRA fine-tuning recipe, while Fast-WAM fine-tunes its video expert and fully trains its action expert.VLA action chunks range from 30 to 50 steps; Fast-WAM predicts 32 actions.Source: §C.2 · pp. 24–25

C.3 ROLLOUT EVALUATION

The rollout protocol evaluates policies on matched environment seeds, samples task objects and targets from specified ranges, and records full-task and phase outcomes.

C.3 ROLLOUT EVALUATION

  • Each policy is evaluated in 100 rollouts per task using matched environment seeds 1–100.Initial object and target poses are sampled from the ranges specified in Appendix B.1.Source: §C.3 · p. 25
  • At 50 Hz, most tasks have 1,000-step episodes, with longer limits for TableAlign, MoveHouse, and CartService.Their limits are 1,100, 1,500, and 2,000 control steps, respectively.Source: §C.3 · p. 25
  • Rollouts record overall task success and cumulative outcomes for the task’s defined phases.Source: §C.3 · p. 25

C.4 AGENTIC ROBOT POLICY EVALUATION

The agentic evaluation tests GPT-6 Astra zero-shot with egocentric observations and iterative control feedback. It succeeds on some CoPouring and Pouring episodes but none on the other two-humanoid tasks.

C.4 AGENTIC ROBOT POLICY EVALUATION

  • A fresh Codex session evaluates each episode without COHUB training data or demonstration replay, using a task-specific role and control instruction.Astra repeatedly inspects both robots’ observations, issues commands, and uses execution feedback to revise its actions.Source: §C.4 · p. 25
  • Each episode allows 6,000 steps (120 s), including a 300-step initialization hold, and a 30-minute policy wall-clock budget; environment steps are not computed during policy reasoning.Actions execute in chunks of at most 25 control steps at 50 Hz, excluding policy reasoning time. Each task is evaluated over 10 episodes with environment seeds 0–9.Source: §C.4 · p. 25
  • Astra achieves 70% success on CoPouring and 10% on Pouring, with no successful episodes on the remaining tasks.The reported phase-wise tables cover the eight two-humanoid tasks.Source: §C.4 · p. 25, Table 9 · p. 26, Table 10 · p. 26

D PHASE-WISE RESULTS

The phase-wise results report cumulative success across three task-defined phases, with tables covering two- and three-humanoid settings and different base-movement conditions.

D PHASE-WISE RESULTS

E COLLABORATION FAILURE CASES

Representative rollouts show failures across all ten tasks, including mistimed handoffs, unsynchronized joint manipulation, and transport errors.

E COLLABORATION FAILURE CASES

  • In Handover and Pouring, coordination fails when the receiver is not ready or the receiving glass is misplaced.The bottle drops before the receiver closes its hand; the marble misses because the partner’s glass is not beneath the cup.Source: Figure 8 · p. 28, E COLLABORATION FAILURE CASES · p. 28
  • FrameHang and CoPouring fail through mismatched lifting or unsynchronized grasps, while CoCarry and BigTable show grip loss during transport.These failures lead to a missed peg, spilled apples, a fallen basket load, or a tilted table that does not reach the goal.Source: Figure 8 · p. 28, E COLLABORATION FAILURE CASES · p. 28
  • Other rollouts miss placement goals: bottles miss the bin or goal table, and tables deviate from target regions or get stuck at a doorway.The caption describes TrashCollection, CartService, TableAlign, and MoveHouse failures.Source: Figure 8 · p. 28, E COLLABORATION FAILURE CASES · p. 28

Related papers

Loading 2609.34782v1…