PaperlayerView exact revision on arXiv

Source-linked AI summary

MATE: Multi-Agent Virtual Teleoperation Platform for Humanoid Collaboration Data Collection

Yichuan Yu, Youzhuo Wang, Yiming Ren, Di Feng, Yexuan Yang, Bingxi Yang, Shengxiao Gong, Yujing Sun, Yuexin Ma

arXiv:2609.26520v1cs.RO

TL;DR

Humanoid collaboration data are difficult to scale because physical multi-robot collection requires costly hardware, dedicated spaces, and repeated resets. MATE enables distributed operators to collect coupled demonstrations in a shared virtual environment and introduces EAIS to prioritize interaction-critical segments. Evaluations report effective policy learning and zero-shot transfer to a physical humanoid without real-world fine-tuning.

  • Problem

    Physical multi-robot collaboration data are difficult to scale, while existing humanoid pipelines primarily focus on individual agents despite dependencies between agents’ actions, observations, and feasible behaviors.

    Source: Introduction · p. 2, Long-Horizon Robot Learning · p. 3

  • Method

    MATE collects synchronized, physically coupled humanoid demonstrations in a shared physics-based environment, while EAIS prioritizes task-progressing and interaction-significant segments within an execution-aligned prefix.

    Source: Introduction · p. 2, Execution-Aligned Interaction Sampling · pp. 4–5

  • Results

    MATE evaluations report effective policy learning and zero-shot transfer from virtual demonstrations to a physical humanoid without real-world fine-tuning.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    MATE provides a scalable, efficient, and cost-effective paradigm for acquiring interaction-rich humanoid collaboration experiences.

    Source: Conclusion · p. 7

    Transfer failures mainly arise from residual sim-to-real gaps in perception, object geometry, and contact dynamics.

    Source: Zero-Shot Transfer of MATE… · p. 7

Abstract

from arXiv · show

Humanoid robots require diverse embodied experiences to acquire complex loco-manipulation and collaborative skills. However, existing humanoid data pipelines primarily focus on individual agents, while physical multi-robot collaboration remains difficult to scale due to costly hardware, dedicated spaces, and repeated resets. In this work, we introduce MATE, a Multi-Agent virtual TEleoperation platform for humanoid collaboration data collection that enables multiple geographically distributed operators to simultaneously control whole-body humanoids in a shared physics-based environment. MATE removes the need for multiple physical robots and co-located operation while preserving physically coupled interactions among humanoids, objects, and environments. Using MATE, we construct a multi-humanoid collaboration dataset comprising 24.1 hours of coordinated behavior across 2,500 joint episodes and five long-horizon tasks, including object handover, relay delivery, environment interaction, and cooperative transport. To improve learning from these interaction-rich demonstrations, we introduce EAIS, an Execution-Aligned Interaction Sampling strategy that computes sampling signals within an execution-aligned prefix and prioritizes task-progressing and interaction-critical behaviors. We evaluate MATE with representative imitation learning and vision-language-action policies across diverse collaboration tasks. Experiments demonstrate efficient data collection, effective policy learning, and zero-shot transfer from virtual demonstrations to a physical humanoid without real-world fine-tuning. Project page: https://yerik-yu.github.io/MATE/

Introduction

MATE enables multiple operators to collect physically coupled humanoid collaboration demonstrations in a shared virtual environment, addressing the challenge of scaling long-horizon multi-agent data collection. The paper also introduces EAIS for prioritizing interaction-critical behavior and reports policy evaluation and zero-shot sim-to-real transfer.

Platform

  • MATE synchronizes multiple operators’ control streams in a shared physics-based environment, recording physically coupled joint episodes without multiple physical robots or co-located operation.One humanoid’s actions can dynamically influence the other agents’ observations, object states, and feasible behaviors.Source: Introduction · p. 2

Learning strategy

  • EAIS scores demonstration windows within an execution-aligned prefix, prioritizing sustained task progress and interaction significance.It targets sparse interaction-critical transitions amid frequent locomotion, approach, and waiting behaviors.Source: Introduction · p. 2

Evaluation

  • MATE evaluations show a scalable pathway for humanoid collaboration learning, including zero-shot transfer to a physical humanoid without real-world fine-tuning.The evaluations benchmark imitation-learning and vision-language-action policies on long-horizon loco-manipulation tasks.Source: Introduction · pp. 2–3

Dataset

  • The work constructs a dataset of long-horizon joint episodes covering diverse coordination and loco-manipulation behaviors.The contribution list identifies the dataset as a multi-humanoid collaboration dataset.Source: Introduction · p. 2

Related Work

Prior work advances humanoid data collection, multi-agent collaboration, and long-horizon policy learning, but scalable collection of synchronized, physically coupled interaction data and interaction-aware temporal selection remain challenges.

Humanoid Data Collection

  • Real-world teleoperation systems improve accessibility through low-cost and immersive interfaces (Zhao et al. 2023; Cheng et al. 2024), while simulation reduces hardware dependence; most collection paradigms still focus on individual robot behaviors.Whole-body teleoperation and simulation-based humanoid learning are among the approaches surveyed.Source: Humanoid Data Collection · p. 3

Multi-Agent Collaboration

  • Prior work explores coordination learning and cooperative manipulation (Lowe et al. 2017; Gao et al. 2024), but synchronized, physically coupled multi-agent data remain difficult to scale.MATE complements these approaches with synchronized multi-humanoid demonstrations collected through multi-user virtual teleoperation.Source: Multi-Agent Collaboration · p. 3

Long-Horizon Robot Learning

  • Long-horizon imitation methods include action chunking (Zhao et al. 2023) and diffusion policies (Chi et al. 2023), but existing temporal selection methods do not explicitly account for interaction-dependent behaviors.Navigation and waiting can dominate trajectories even though contact, handover, and coordination transitions occur sparsely.Source: Long-Horizon Robot Learning · p. 3

Methodology

MATE collects synchronized, physically coupled humanoid collaboration episodes through independent VR teleoperation in a shared simulation, then applies EAIS to prioritize execution-relevant training windows. EAIS favors task progress and interaction events over redundant portions of long-horizon demonstrations.

Methodology

  • MATE’s pipeline first collects synchronized joint episodes in shared simulation and then applies EAIS to prioritize task-progressing and interaction-event windows.The platform preserves interactions among humanoids, objects, and environments.Source: Methodology · p. 3

MATE Platform

  • MATE synchronizes independent operator commands in a shared physics environment, preserving how one humanoid’s actions change others’ observations and feasible actions.Each operator controls a designated humanoid through an independent VR-based interface.Source: MATE Platform · p. 4

Execution-Aligned Interaction Sampling

  • EAIS computes sampling signals within an execution-aligned prefix of each candidate window, using task progress, motion-mode transitions, action changes, object-state changes, and interaction events.The signals are drawn from the synchronized interaction trajectory within the prefix.Source: Execution-Aligned Interaction Sampling · pp. 4–5
  • EAIS downweights or excludes windows dominated by waiting, idle behavior, or jitter without progress, while prioritizing progress, coordination-relevant waiting, grasping, release, and contact transitions.The resulting sampling distribution preserves long-range context without requiring manually annotated task phases.Source: Execution-Aligned Interaction Sampling · p. 5

Dataset

MATE’s dataset centers on physically coupled multi-humanoid joint episodes rather than isolated robot trajectories. It contains 2,500 episodes across five long-horizon tasks spanning object transfer, environment interaction, and cooperative transport.

Dataset

  • MATE targets physically coupled multi-agent collaboration, contrasting with representative humanoid datasets that primarily cover single-agent trajectories.Its episodes preserve the coupled evolution of multiple humanoids, objects, and environments.Source: Dataset · p. 5

Collaborative Humanoid Episodes

  • 2,500 joint episodes—500 for each of five tasks—contain about 4.35 million time steps, or 24.1 hours, of coordinated behavior collected at 50 Hz.The episodes yield 5,000 robot-centric trajectories and preserve interaction histories that support studying waiting, handover, cooperative manipulation, and role transitions.Source: Collaborative Humanoid Episodes · p. 5

Long-Horizon Collaboration Tasks

  • The five tasks cover refrigerator placement, bottle handoff, bottle relay, bottle loading into a cart, and pushing a hospital bed through a constrained doorway.Together they involve locomotion, manipulation, temporal coordination, and physical interaction; the hospital-bed task requires sustained contact, locomotion coordination, forceful interaction, and long-horizon planning.Source: Long-Horizon Collaboration Tasks · pp. 5–6

Experiments

The experiments assess policy learning from MATE demonstrations, collection scalability, EAIS, and zero-shot sim-to-real transfer. They report transfer to a physical humanoid without real-world fine-tuning, alongside gains in collection efficiency and long-horizon learning.

Experiments

  • The experiments examine policy architectures, virtual-versus-physical collection scalability, EAIS versus uniform sampling, and Bottle relay sim-to-real transfer.The evaluation includes long-horizon rollouts and stage-conditioned trials for grasping, transport, placement, and pushing.Source: Experiments · p. 6, Experimental Setup · p. 6

Experimental Setup

  • Policies were evaluated across five collaboration tasks with human-demonstration partner trajectories, using robot-centric visual observations and proprioceptive states at test time.The five baselines were ACT, Diffusion Policy, ψ0, GR00T N1.7, and π0.5; each method used identical demonstrations, environments, partner procedures, and evaluation budgets.Source: Experimental Setup · p. 6

Learning from MATE Demonstrations

  • ψ0 achieved consistent performance across multiple tasks, while other policies learned individual skills but often struggled with full tasks from initialization.The evaluation table reports full-horizon stage completion and stage-conditioned skill completion over 10 trials; the full-horizon gap reflects accumulated temporal and interaction errors.Source: Learning from MATE Demonstrations · p. 6, Table 2 · p. 7

Scalability of MATE Data Collection

  • 2.17× improvement: MATE reduced amortized collection time per successful Bottle relay episode from 89 seconds with physical teleoperation to 41 seconds.The supplementary material provides further timing protocols and qualitative comparisons of reset, recovery, and hardware requirements.Source: Scalability of MATE Data Collection · p. 6

Analysis of Execution-Aligned Interaction Sampling

  • EAIS improved full-task completion and reduced successful-rollout execution time for ψ0 and GR00T compared with uniform sampling under matched training conditions.The study attributes the gains to emphasizing task-progressing and interaction-critical behaviors while preserving essential interaction context.Source: Analysis of Execution-Aligned… · p. 7, Table 3 · p. 7

Zero-Shot Transfer of MATE Demonstrations

  • 90% simulated success and 70% real-world success across 10 trials each; the Bottle relay policy used no real-world data or fine-tuning.The transferred behavior includes coordinating with partner bottle placement, grasping, transport, and placement.Source: Zero-Shot Transfer of MATE… · p. 7

Conclusion

MATE provides a virtual teleoperation platform for scalable humanoid collaboration data collection, and the paper reports learning and transfer results from its demonstrations. The authors position it as a foundation for scalable collection and learning of complex multi-agent embodied behaviors.

Conclusion

  • MATE provides a multi-agent virtual teleoperation platform for scalable humanoid collaboration data collection.Multiple operators control humanoids in a shared physics-based environment.Source: Conclusion · p. 7
  • The authors identify scalable humanoid collaboration data collection and learning complex multi-agent embodied behaviors in more realistic and diverse environments as opportunities.Source: Conclusion · p. 8

Supplementary Material

The supplementary material documents the MATE platform, dataset and task construction, EAIS implementation, and policy evaluation details. It also points readers to videos showing the collection workflow, collaborative tasks, policy rollouts, sampling comparison, and physical deployment.

Supplementary Material

  • The supplement outlines platform and dataset details, EAIS implementation, policy training and evaluation, failure analysis, scalability study, and sim-to-real deployment procedures.Source: Supplementary Material · p. 10

Supplementary Video

  • The project videos cover the multi-user collection workflow, five collaborative tasks, policy rollouts, EAIS comparison, and zero-shot physical deployment.Source: Supplementary Video · p. 10

MATE Platform Details

  • MATE synchronizes operator command streams before advancing a shared MuJoCo physics state, then records aligned robot, object, command, event, and camera data.The collection and downstream control frequency is 50 Hz; synchronized trajectories are later rerendered in Isaac Sim with changed camera configuration and visual appearance while preserving recorded trajectories and temporal alignment.Source: MATE Platform Details · p. 10

Dataset and Task Details

  • The dataset contains 2,500 joint episodes across five tasks, totaling 24.1 hours; each task has 450 training, 25 validation, and 25 test episodes.Training episodes combine 903 human-operated episodes with 1,597 validated variants derived from them.Source: Dataset and Task Details · pp. 10–11
  • The five tasks span direct transfer, sequential relay, partner-conditioned placement, articulated-environment interaction, and sustained contact with a large object.Augmentation perturbs object or interaction locations, or adds waiting time for one task; variants are retained after task-specific consistency, smoothness, contact, and replay checks.Source: Dataset and Task Details · p. 11

Execution-Aligned Interaction Sampling

EAIS scores candidate demonstration windows using signals from an execution-aligned prefix, then prioritizes task progress and interaction-critical behaviors. Its sampling plan also preserves contact-event coverage and a small amount of meaningful waiting context.

Execution-Aligned Interaction Sampling

  • EAIS evaluates candidate windows with H = 40 over the first 20 steps at 50 Hz, independently of a downstream policy’s action chunk or replanning schedule.These values define the sampling manifest used in the experiments.Source: Execution-Aligned Interaction Sampling · p. 11
  • Its label-free signals cover robot and scene state changes, motion and activity transitions, object changes, and contact events, without manually annotated task phases.The score uses robot i’s state, manipulated-object and articulated-environment states, and automatically detected contacts; activity and displacement evidence supports event detection.Source: Execution-Aligned Interaction Sampling · p. 11
  • Progress scoring compares active state groups with a selected target over execution-prefix checkpoints, using the earliest eligible future event or a capped fallback target.The execution prefix is evaluated at offsets 0, 5, 10, 15, and 20; if no event is at least five steps ahead, the target uses t + 80, capped at t+100 and episode end.Source: Execution-Aligned Interaction Sampling · p. 11
  • EAIS assigns priorities to qualified progress and transition windows, with higher base priority for contact acquisition than contact release, pre-contact leads, other transitions, or continuous progress.A pre-contact lead is certified progress whose next contact acquisition falls beyond the aligned-transition range but within the 40-step candidate window; resume-context windows receive base priority 0.06 when not already classified as progress or transition.Source: Execution-Aligned Interaction Sampling · p. 12
  • Allocation preserves an effective-sample-size fraction of at least 0.20 and contact-event coverage when a valid aligned window exists, while targeting 0.75% resume-context starts capped at 1%.This retains coordination-relevant waiting without letting prolonged idle behavior or jitter lacking progress, scene change, or interaction events dominate training.Source: Execution-Aligned Interaction Sampling · p. 12

Policy Training and Evaluation Details

The experiments train several policy families using MATE demonstrations and replayed human-operated partner trajectories. Evaluation uses closed-loop trials, while the uniform-sampling ablation changes only the sampling probabilities.

Policy Training and Evaluation Details

  • ACT and Diffusion Policy are trained from scratch, while ψ0, GR00T N1.7, and π0.5 are fine-tuned from official pretrained checkpoints.Each configuration is trained once and evaluated using its final designated checkpoint.Source: Policy Training and Evaluation Details · p. 12, Table 6 · p. 13
  • Across simulation rollouts, the partner follows varied replayed human-operated trajectories while the policy receives only its robot-centric visual observation and proprioceptive state.The variation includes partner motion and interaction timing, object placement, and articulated-environment state while preserving the task procedure.Source: Policy Training and Evaluation Details · p. 12
  • Full-horizon and applicable stage-conditioned results use 10 trials each under the formal 50-Hz closed-loop evaluation pipeline, with stage trials initialized from standardized entry states.A rollout fails on timeout, robot fall, dropped object, loss of required sustained contact, or failure to meet the completion criterion.Source: Policy Training and Evaluation Details · pp. 12–13
  • The uniform-sampling ablation preserves the demonstrations, data split, eligible starts, model, optimization budget, evaluation, and partner procedures, changing only probabilities from EAIS priorities to equal weighting.Ineligible or zero-weight terminal regions are excluded in both settings.Source: Policy Training and Evaluation Details · p. 13

Failure Case Analysis

EAIS improvements in full-task success and successful-rollout duration are consistent with fewer delayed responses and more reliable transitions. The analysis identifies timing, approach, contact, and cross-stage execution as sources of long-horizon difficulty.

Failure Case Analysis

  • Stage-conditioned grasping results do not capture the additional full-horizon demands of approaching, responding to partner progress, preserving contact, and transitioning without supplied stage boundaries.ACT and Diffusion Policy can acquire local grasping behaviors when the object and interaction state are already nearby.Source: Failure Case Analysis · p. 13
  • Delayed responses or inefficient waiting can alter the state for later approach, grasp, and placement behaviors, compounding timing errors across subsequent stages.The focal humanoid may need to approach, wait for a partner-conditioned change, or act promptly when interaction becomes available.Source: Failure Case Analysis · p. 13
  • Errors in base orientation, approach distance, or target localization can leave hands outside the reliable grasp region, while near-object stage initialization reduces this variation.This helps explain why local skill completion can be substantially higher than full-task completion.Source: Failure Case Analysis · p. 13
  • Marginal bottle grasps may destabilize during lifting or transport, and loss of sustained two-hand contact or asymmetric pushing can redirect a hospital bed.A successful grasp alone does not ensure transport or placement; the policy must preserve the object while changing whole-body motion, navigating, and releasing.Source: Failure Case Analysis · p. 13
  • EAIS’s reported improvements in full-task success and successful-rollout duration are consistent with fewer delayed responses and more reliable transitions across long-horizon coordination.EAIS increases representation of progress, contact transitions, and prompt responses after partner-conditioned state changes while retaining some meaningful waiting context.Source: Failure Case Analysis · pp. 13–14

Scalability of MATE Data Collection

On Bottle Relay, MATE reduces measured successful-episode collection time relative to physical teleoperation. Its virtual setup also changes the infrastructure needed to scale collaborative collection.

Scalability of MATE Data Collection

  • MATE requires 41 seconds per successful Bottle Relay episode versus 89 seconds for physical collection, a 2.17× speedup.MATE time is amortized over successful episodes and includes virtual resets; failed retries are excluded from both methods.Source: Scalability of MATE Data Collection · p. 14
  • Physical collaborative collection depends on complete humanoid systems, a sufficiently large and safe workspace, synchronized on-site operation, and physical initialization and recovery.These requirements couple the number of simultaneously controlled agents to available hardware and space.Source: Scalability of MATE Data Collection · p. 14
  • MATE permits geographically distributed operators to control multiple humanoids in a shared virtual environment, and new tasks can reuse the system with changed scenes, objects, and role specifications.Operators enter the same episode through independent interfaces.Source: Scalability of MATE Data Collection · p. 14

Sim-to-Real Deployment Details

MATE Bottle Relay demonstrations are adapted to a deployment-aligned virtual scene before training ψ0, then evaluated on a physical humanoid without real-world demonstrations or fine-tuning. In 10 trials per setting, the policy succeeds in both simulation and physical execution.

Sim-to-Real Deployment Details

  • Bottle Relay trajectories are replayed in a deployment-aligned virtual scene, adapting layout, cameras, and appearance while preserving trajectories, task structure, and interaction timing.The resulting robot-centric observations train the ψ0 policy.Source: Sim-to-Real Deployment Details · p. 14
  • The policy receives robot-centric images and proprioceptive state through the official ψ0–SONIC pathway, with predicted 78-dimensional actions executed by the SONIC controller.The evaluation uses no real-world demonstrations or fine-tuning.Source: Sim-to-Real Deployment Details · p. 14
  • After a human partner places the bottle at an intermediate location, the humanoid autonomously approaches, grasps, transports, and places it using visual observations and proprioceptive feedback.Partner placement position and timing vary across trials.Source: Sim-to-Real Deployment Details · p. 14
  • 90% success in simulation and 70% on the physical humanoid across 10 trials in each setting.The physical evaluation uses a human partner who varies bottle placement position and timing.Source: Sim-to-Real Deployment Details · p. 14
Loading 2609.26520v1…