PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Whole-Body Human-Humanoid Interaction from Human-Human Demonstrations

Wei-Jin Huang, Yue-Yi Zhang, Yi-Lin Wei, Zhi-Wei Xia, Juantao Tan, Yuan-Ming Li, Zhilin Zhao, Wei-Shi Zheng

arXiv:2601.09518v1cs.ROcs.AIcs.CV

TL;DR

Humanoid interaction research is limited by scarce high-quality HHoI data and policies that imitate trajectories without interactive understanding. The paper introduces PAIR to retarget HHI demonstrations into physically consistent HHoI data and D-STAR to decouple temporal and spatial action reasoning. D-STAR achieves 75.4% average success, exceeding Transformer and TCN baselines, while PAIR attains 0.841 contact F1.

  • Problem

    High-quality HHoI data is scarce, while conventional retargeting can break essential contacts and imitation policies can reduce interaction to trajectory mimicry.

  • Method

    PAIR uses contact-centric two-stage retargeting, and D-STAR separates when to act from where to act for synchronized whole-body collaboration.

  • Results

    75.4% average success for D-STAR exceeded Transformer (64.3%) and TCN (49.2%), while PAIR achieved 0.841 contact F1 (0.35 m, micro-averaged).

  • Takeaways & Limitations

    The framework provides a complete pipeline for learning whole-body HHoI from HHI demonstrations while preserving contacts and supporting responsive collaboration.

Abstract

from arXiv · show

Enabling humanoid robots to physically interact with humans is a critical frontier, but progress is hindered by the scarcity of high-quality Human-Humanoid Interaction (HHoI) data. While leveraging abundant Human-Human Interaction (HHI) data presents a scalable alternative, we first demonstrate that standard retargeting fails by breaking the essential contacts. We address this with PAIR (Physics-Aware Interaction Retargeting), a contact-centric, two-stage pipeline that preserves contact semantics across morphology differences to generate physically consistent HHoI data. This high-quality data, however, exposes a second failure: conventional imitation learning policies merely mimic trajectories and lack interactive understanding. We therefore introduce D-STAR (Decoupled Spatio-Temporal Action Reasoner), a hierarchical policy that disentangles when to act from where to act. In D-STAR, Phase Attention (when) and a Multi-Scale Spatial module (where) are fused by the diffusion head to produce synchronized whole-body behaviors beyond mimicry. By decoupling these reasoning streams, our model learns robust temporal phases without being distracted by spatial noise, leading to responsive, synchronized collaboration. We validate our framework through extensive and rigorous simulations, demonstrating significant performance gains over baseline approaches and a complete, effective pipeline for learning complex whole-body interactions from HHI data.

1. Introduction

The paper targets two HHoI bottlenecks: scarce, low-diversity interaction data and policies that mimic motion without understanding intent. It introduces PAIR for physically consistent HHoI data and D-STAR for synchronized whole-body collaboration.

  • HHoI requires advances beyond locomotion, manipulation, and single-agent learning toward dynamic collaboration in shared human spaces.
  • Real-world teleoperation data is high-fidelity but costly, slow, potentially unsafe, and insufficiently diverse, motivating scalable alternatives.
  • PAIR uses contact-centric, two-stage retargeting to preserve interaction semantics and physical consistency when converting human demonstrations into HHoI data.
  • Conventional imitation learning reduces high-fidelity demonstrations to trajectory mimicry, lacking relational geometry, timing, and interactive understanding.
  • D-STAR separates when to act from where to act through Phase Attention and a Multi-Scale Spatial module, then fuses them for synchronized whole-body behavior.
  • The framework is evaluated in simulation and on a humanoid robot, with real-robot trials testing executability under asynchronous sensing and standard whole-body control.

2. Related Work

Prior HHoI and retargeting methods address parts of the problem but remain limited by supervision requirements, open-loop behavior, task scope, or broken contacts under morphology mismatch.

  • Human–Humanoid Interaction: Existing HHoI approaches include teleoperation-based learning, open-loop social forecasting, upper-body interaction, compliance control, and robot-side multitask policies.
  • Human–Humanoid Interaction: These methods differ in supervision pathway and scope, often relying on robot-side supervision or broadening beyond social HHI.
  • Motion Retargeting for Interaction: Classical and learning-based retargeting mainly optimize kinematic similarity or style, often failing to preserve contact semantics under human–humanoid morphology mismatch.
  • Motion Retargeting for Interaction: The paper instead learns whole-body, closed-loop HHoI from abundant HHI demonstrations while decoupling when to act from where to contact.

3. PAIR: Physics-Aware Interaction Retargeting

PAIR identifies kinematic-only retargeting as a central obstacle because morphology differences can destroy the physical contacts that define an interaction.

  • Conventional retargeting preserves source kinematics by minimizing joint-position or joint-orientation differences, but this objective alone can break essential interaction contacts.
  • The failure arises directly from morphological disparities between humans and humanoids, making kinematic similarity insufficient for physical interaction.
  • Figure 2 visualizes the failure of source-kinematics matching and motivates the interaction-aware PAIR framework.

Physics-Aware Interaction Retargeting

The supplied passage only labels a coarse-to-fine distinction and does not provide enough content for a substantive section summary.

  • The passage labels a coarse-to-fine progression without explaining its stages, objective, or role in the retargeting pipeline.

STAGE 2: CONTACT-CENTRIC

PAIR formulates interaction retargeting as a contact-centric optimization problem and solves it with a two-stage strategy to preserve physical semantics across morphology differences. Its objective balances contact, kinematic, human-motion, and physical-plausibility terms while avoiding poor near-miss solutions.

  • PAIR formulation: PAIR optimizes robot and minimally adjusted partner motions to preserve interaction semantics across morphology differences.The formulation searches over the source interaction for an optimal robot motion and a minimally adjusted partner motion.
  • Objective: The composite objective combines contact, kinematic, human-motion, and regularization losses.The objective is Lretarget = wconLcon + wkinLkin + whumLhum + wregLreg.
  • Kinematic similarity: Kinematic similarity matches robot joint positions to a morphology-reshaped human skeleton but can itself cause contact breaking.The reshaped skeleton uses pelvis alignment, proportional bone-length scaling, and fixed joint correspondences.
  • Contact preservation: Contact preservation compares full pairwise-distance matrices between original human-human and optimized human-robot keypoint sets.The matrix-level constraint maintains contacts and appropriate social distances beyond brittle pointwise penalties.
  • Fidelity and plausibility: Human-motion fidelity permits small partner adaptations, while regularizers penalize jitter and uncomfortable joint angles.Selective upper-body adaptation supports task-consistent changes, and regularization includes temporal coherence and pose terms.
  • Two-stage optimization: A coarse-to-fine two-stage optimization strategy addresses the non-convex objective’s tendency toward physically implausible near-miss interactions.Stage 1 performs global kinematic initialization with moderate contact weighting to reach a desirable basin of attraction.

4. D-STAR: Decoupled Spatio-Temporal Action Reasoner

D-STAR addresses interactive imitation by separating temporal reasoning about when to act from spatial reasoning about where to act. Phase Attention and Multi-Scale Spatial features condition a diffusion planner, whose outputs are executed by a whole-body controller.

  • Motivation and overview: D-STAR separates timing and targeting, addressing conventional imitation learning’s conflation of when to act and where to act.The policy is hierarchical and uses decoupled spatio-temporal reasoning for interactive control.
  • Temporal encoding: A long-short temporal encoder processes full-history context and recent observations to capture interaction phases and short-term spatial cues.The encoder uses parallel long-term and short-term Transformer branches whose outputs are concatenated.
  • Planning and execution: Phase and spatial features jointly condition a diffusion planning head, while a Whole-Body Controller converts its reference action into executable joint targets.The policy operates at a lower update rate than the controller to preserve stability.
  • Phase Attention: Phase Attention predicts interaction phases and weights specialized temporal attention blocks to determine when an event unfolds.The paper uses Preparation, Act, and Follow-up phases with a transition-consistency loss.
  • Multi-Scale Spatial module: The Multi-Scale Spatial module encodes coarse-to-fine human-robot geometric neighborhoods to determine where relevant contact should occur.It aggregates near-, mid-, and far-range cues into a spatially aware representation.
  • Ablation: Removing either Phase Attention or the Multi-Scale Spatial module degrades tasks requiring precise temporal or spatial coordination.The ablation supports the necessity of both decoupled reasoning branches.

5. Experiments

Experiments show that PAIR improves physical consistency and smoothness in HHoI retargeting, while D-STAR substantially improves interactive-policy success and robustness over baselines. Ablations support the importance of contact-aware retargeting, two-stage optimization, and decoupled temporal-spatial reasoning.

  • Retargeting results: PAIR achieves 0.841 contact F1 at 0.35 m, improving 67.5% over ImitationNet and 22.2% over Simple MSE.The evaluation uses micro-averaged contact metrics across frames and hands.
  • Retargeting results: PAIR obtains the best kinematic similarity with JPE 0.174 m and reduces jerk by 69% versus Simple MSE.The gains do not trade off kinematic similarity or motion smoothness.
  • Retargeting ablations: Removing Human Adaptation or contact loss lowers contact F1 to 0.823 and 0.821, while collapsing two-stage optimization lowers it to 0.788.These ablations identify partner adaptation, explicit contact optimization, and coarse-to-fine optimization as contributors to physical precision.
  • Policy results: D-STAR reaches 75.4% average success, exceeding Transformer at 64.3% and TCN at 49.2%; Handshake success is 61.3% versus 32.3%.Naive Mimicry achieves 0% mean success, showing that replaying trajectories is inadequate for physical interaction.
  • Policy ablations: Removing PA harms temporally sensitive interactions, while removing MSS harms spatially precise interactions, confirming complementary temporal and spatial roles.The ablation evidence supports the design of decoupled reasoning streams.
  • Robustness: Across 0.8×–1.2× partner-scale and speed variations, performance degrades gracefully and retains 62.7% success at the most extreme setting.The policy is more resilient with smaller partners than larger ones, consistent with the G1 humanoid’s embodiment.

6. Conclusion

The paper addresses whole-body HHoI by correcting failures at both the data and policy levels, combining physically consistent retargeting with decoupled spatio-temporal reasoning. This pipeline aims to support responsive collaboration rather than simple trajectory replay.

  • PAIR generates physically consistent HHoI data from human demonstrations while preserving crucial contacts where naive retargeting fails.
  • D-STAR uses decoupled spatio-temporal reasoning to decode interactive intent for whole-body interaction.
  • The framework enables responsive collaboration rather than simple trajectory replay.

Supplementary Material

The supplementary material defines PAIR and D-STAR, then details their optimization, temporal-spatial reasoning modules, conditioning, diffusion planning, and controller execution. It also specifies implementation settings for producing and executing whole-body interaction actions.

  • PAIR: PAIR is a physics-aware retargeting pipeline designed to generate physically consistent HHoI data.Its objective includes kinematic similarity, interaction-contact consistency, human motion fidelity, and physical plausibility regularization.
  • PAIR: The retargeting process uses a coarse-to-fine, two-stage optimization schedule to reduce poor local minima and refine contacts.Stage 1 obtains a kinematically feasible trajectory, while Stage 2 increases contact refinement.
  • D-STAR: D-STAR decouples temporal phase inference from geometric action selection through Phase Attention and Multi-Scale Spatial modules.The temporal encoder combines long-term context with short-term reactivity, while spatial processing uses near, mid, and far interaction zones.
  • D-STAR: A conditional diffusion policy integrates the decoupled features and language command embedding to generate a high-level target.A whole-body controller then executes the target while maintaining balance and respecting joint limits.

A.3.1. High-Level Policy Training (Supervised Learning)

The high-level policy is trained end-to-end with supervised learning on the curated HHoI dataset using action, phase, and geometric objectives. Its outputs are passed to a distilled whole-body controller trained to operate under noisy observations.

  • High-Level Policy Training: The high-level policy is trained end-to-end on the curated HHoI dataset with a composite objective.The loss combines action prediction, phase supervision, and auxiliary geometric regularization.
  • High-Level Policy Training: Phase supervision combines phase classification with a KL-based loss for enforcing logical phase transitions.The auxiliary objectives also regularize facing orientation, keypoint positions, and spatial consistency.
  • High-Level Policy Training: The primary action loss predicts changes in root translation, root orientation, and joint angles using a weighted MSE.The stated weights are wtrans = 10.0, wrot = 10.0, and wdof = 1.0.
  • WBC-Sim Distillation: The controller is trained through privileged-to-noisy distillation, with an oracle using clean observations and a student using noisy observations with 10-frame history.All policy evaluations use the distilled student, and the simulation interface actuates 29-D joint-angle targets at 50 Hz.
  • Training Data: The curated dataset includes quality-controlled interaction sequences and manually annotated phases that supervise temporal reasoning.The phase annotations provide supervision for learning when to act, which conventional imitation learning lacks according to the passage.

B.4. Final Dataset Statistics and Properties

The final HHoI dataset is the output of principled task selection, quality control, and phase annotation. It spans substantial interaction data with temporal and spatial richness intended to support training and evaluation of the multi-scale policy.

  • Dataset Output: The final HHoI dataset is produced by a comprehensive pipeline whose output is used throughout the experiments.Its statistics are summarized in Table S4.
  • Dataset Statistics: The dataset spans approximately 6.9 hours of interaction data sampled at 50 Hz.Average sample durations range from 16.1 seconds for compact gestures to 25.9 seconds for complex contact-rich tasks such as hugs.
  • Dataset Properties: The dataset combines temporal depth, tracked key body joints, and balanced task distribution for multi-scale policy training and evaluation.
  • Evaluation: The dataset construction is assessed with metrics spanning physical consistency, contact preservation, plausibility, and smoothness.The evaluation suite contains 18 metrics.

C.2.1. Evaluation Metrics

The evaluation measures retargeting fidelity, contact preservation, physical plausibility, motion smoothness, and task-specific interaction success. Contact and gesture criteria use distance, duration, geometry, and motion-pattern checks.

  • Physical Consistency Metrics: Physical consistency is assessed with Joint Position Error and Average Workspace Distance for kinematic and pairwise spatial fidelity.The latter compares interaction-joint distance matrices between original human-human and retargeted human-robot interactions.
  • Contact Detection Metrics: Contact preservation is evaluated at 0.2 m, 0.35 m, and 0.5 m using precision, recall, F1-score, and accuracy.Hand-hand contact is classified per frame with optimal left/right matching and micro-averaged over frames and hands.
  • Physical Plausibility Metrics: Physical plausibility uses Large Angle Ratio and Angle Standard Deviation to flag extreme joint angles and characterize pose-distribution consistency.Large Angle Ratio counts frames whose axis-angle magnitudes exceed 0.5 rad.
  • Motion Smoothness Metrics: Motion smoothness is measured with mean and standard deviation of third-order joint-position jerk at the 50 Hz dataset frame rate.Both metrics penalize high-frequency jitter or indicate temporal stability through jerk variability.
  • Task Success Criteria: Six task-specific detectors define success for hug, handshake, high-five, wave, bend, and fly-kiss interactions.The detectors respectively use sustained contact, distance constraints, height validation, motion amplitude and direction changes, inclination angle, or a two-phase gesture sequence.
  • Evaluation Protocol: Detector hyperparameters were tuned on 50 validation sequences against the consensus of three human evaluators.The evaluation protocol also standardizes preparation, transitions, and retargeted lower-body motion before interaction execution.
Loading 2601.09518v1…