PaperlayerView exact revision on arXiv

Source-linked AI summary

Mimetic Embodiment Transfer of Human-Humanoid Interaction from In-the-Wild Videos

Nghia Le, Khang Nguyen, Ba Long Tran, Thi Quynh Nhu Pham, Binh Nguyen, Tuong Do, Erman Tjiputra, Hoan Nguyen, Tung D. Ta, Baoru Huang, Hai-Nguyen, Nguyen, Anh Nguyen

arXiv:2610.36615v1cs.RO

TL;DR

HumiTransfer addresses the challenge of transferring human interaction to humanoids while preserving partner relationships across differing embodiments. It reconstructs and retargets in-the-wild interactions, and the authors report best MPJPE, IRD, and CF1 results for HumiTransfer+ across both tested robots.

  • Problem

    Retargeting methods primarily optimize single-agent motion and do not preserve the inter-body relationships or identify the semantic cues needed for interaction across embodiments.

    Source: §I · p. 2

  • Method

    HumiTransfer reconstructs human interaction poses from in-the-wild videos and transfers one participant to a humanoid while preserving interaction semantics and distances under physical and kinematic constraints.

    Source: §I · p. 2, 2) Extraction of Human-Human… · p. 4

  • Results

    HumiTransfer+ achieves the best MPJPE, IRD, and CF1 results across both the Unitree R1 Edu and HighTorque Mini-hi.

    Source: 2) Mimetic Embodiment Transfer · p. 5

  • Takeaways & Limitations

    The authors report that HumiMimic serves as a benchmark for human-humanoid interaction work and that transferred motions were physically deployed on the Unitree R1 Edu.

    Source: §V · p. 8

    Substantial human-humanoid morphological differences can make some motions kinematically infeasible, and current real-robot deployment primarily uses predefined sequences rather than explicitly accounting for human motion.

    Source: D. Limitations & Future Work · p. 7

Abstract

from arXiv · show

Enabling a humanoid to interact naturally with human partners requires preserving not only individual motions but also the spatial, semantic, and contact relationships underlying the interaction (i.e., hugging, dancing, hand-shaking, etc.). Furthermore, a practical solution should generalize across various humanoid embodiments with distinct morphologies and kinematic constraints. To tackle this challenge, we propose HumiTransfer, a mimetic embodiment transfer framework that converts in-the-wild human-human interaction videos into physically plausible human-humanoid interactions. HumiTransfer first reconstructs 3D human motion with consistent foot-ground contact, then transfers one participant to a target humanoid embodiment via interaction modeling, using weighted joint-relative directions and anatomical offsets to prioritize interaction-critical inter-body relationships, while post-retargeting optimization enforces balance, contact, and collision avoidance. Subsequently, we apply HumiTransfer to 1,200 in-the-wild videos, producing approximately 500,000 frames of diverse interactions for the 26-DoF Unitree R1 Edu and the 23-DoF HighTorque Mini-hi humanoid robot, resulting in our HumiMimic dataset. Across both robots, HumiTransfer achieves superior motion fidelity, preserves inter-body relationships, and maintains contact accuracy compared with existing baselines. Moreover, we demonstrate that HumiMimic supports motion generation methods for synthesizing interactive humanoid motions following a human partner. Finally, our real robot deployments confirm the physical executability of the transferred motions in real-world settings.

I. INTRODUCTION

Meaningful human interaction depends on evolving relationships between partners, but existing retargeting and datasets do not jointly preserve interaction semantics across humanoid embodiments. HumiTransfer converts in-the-wild human interactions into human-humanoid motions, and HumiMimic combines diverse interaction data with support for two humanoid embodiments.

I. INTRODUCTION

  • Interaction requires coordinating spacing, orientation, rhythm, movement phase, contact, and adaptation to a partner rather than imitating isolated motions alone.The paper describes this as mimesis: representing and communicating experience through interaction in semantic and dynamic ways.Source: §I · p. 1
  • HumiTransfer translates one human participant from in-the-wild interactions into a humanoid while preserving interaction semantics and distances under embodiment-specific kinematic and physical constraints.The framework transfers poses to the 23-DoF HighTorque Mini-hi and 26-DoF Unitree R1 Edu.Source: §I · p. 2
  • HumiMimic contains 1,200 human-humanoid interaction sequences and approximately 500,000 frames extracted from in-the-wild human-human interaction videos.The dataset is presented as combining in-the-wild multi-person interactions with support for diverse humanoid embodiments, attributes not jointly provided by existing datasets.Source: §I · p. 2, §II · p. 2

II. RELATED WORK

  • Existing retargeting methods primarily optimize single-agent motion, while interaction datasets and humanoid datasets generally lack a joint provision of large-scale in-the-wild interactions and humanoid-compatible motion.HumiTransfer extends optimization-based retargeting to embodiment-specific interaction with a human participant.Source: §II · p. 2

III. HUMITRANSFER: MIMETIC EMBODIMENT TRANSFER FOR HUMAN-HUMANOID INTERACTION

HumiTransfer is presented as a three-stage framework that turns reconstructed 3D interactions into human-humanoid motions and HumiMimic. It combines interaction-preserving retargeting with post-retargeting optimization.

III. HUMITRANSFER: MIMETIC EMBODIMENT TRANSFER FOR HUMAN-HUMANOID INTERACTION

  • The framework starts with reconstructed 3D interaction motions, applies interaction-preserving human-to-humanoid retargeting, and then performs post-retargeting optimization.These are the three stages identified in the overview.Source: §III · p. 3
  • The output of the framework is HumiMimic, a human-humanoid interaction dataset.Source: §III · p. 3
  • The retargeting stage is framed as mimetic embodiment transfer that preserves interactions while translating human motion to a humanoid.Source: §III · p. 3

1) Data Collection & Statistics:

The dataset is built from in-the-wild two-person videos, then filtered and reconstructed into human interaction motions with additional fitting for foot-ground consistency. The resulting sequences serve as input for humanoid retargeting.

1) Data Collection & Statistics:

  • 1,200 two-person videos yield about 500,000 processed frames, spanning approximately 4.6 video hours and interactions from 100 to 1,000 frames per sequence.The dataset includes co-movement (29%), social exchanges (21%), dual sports (17%), and sparring (13%); Fig. 3 describes interaction-length and diversity distributions.Source: 1) Data Collection & Statistics · p. 3, Figure 3 · p. 3

2) Extraction of Human-Human Interaction Motion:

  • An automatic filtering step retains approximately 73% of frames before tracking participants, detecting keypoints, and reconstructing SMPL human poses.The extracted poses are refined through whole-body shape fitting and foot-ground contact consistency fine-tuning.Source: 2) Extraction of Human-Human… · p. 3
  • Foot-ground refinement minimizes foot-contact error and uses a Zero Moment Point regularizer to improve reconstructed-pose plausibility.The contact error uses SMPL foot joints, a ground-contact indicator, joint positions, a reference foot position, and an estimated planar-ground normal.Source: 2) Extraction of Human-Human… · pp. 3–4
  • Applying the fitting and contact-refinement objectives to detected humans produces SMPL frame sequences for retargeting humanoid motion with a human partner.Source: 2) Extraction of Human-Human… · p. 4

B. Interaction Modeling & Mimetic Embodiment Transfer

The section initializes humanoid poses by inverse-kinematics retargeting, then models human-humanoid interactions to preserve their semantics and geometric relationships. The model combines joint-relative directions, anatomical offsets, and weights that prioritize interaction-relevant joints.

B. Interaction Modeling & Mimetic Embodiment Transfer

1) Humanoid Poses Warm-Starting:

  • Humanoid poses are warm-started by scaling human poses to the target embodiment and optimizing its joints under kinematic constraints and joint limits.Inverse kinematics provides the initialization at each timestep.Source: 1) Humanoid Poses Warm-Starting · p. 4

2) Modeling of Human-Humanoid Interaction:

  • HumiTransfer models interactions with joint-relative direction and anatomical-offset matrices between the interacting subjects’ joints.The direction matrix encodes joint-based directional offset vectors; the anatomical-offset matrix encodes link-based offsets relative to the partner.Source: 2) Modeling of Human-Humanoid… · p. 4
  • Joint-importance weighting prioritizes body parts more involved in the interaction during retargeting.Distance weighting assigns larger weights to closer joint pairs and smaller weights to more distant pairs.Source: 2) Modeling of Human-Humanoid… · p. 4

3) Mimetic Embodiment Transfer Objective:

The transfer objective aligns the human-humanoid interaction with the original human-human interaction across the interaction horizon. It is combined with inverse-kinematics and smoothness terms to solve humanoid poses at each timestep; qualitative comparisons show HumiTransfer+ retaining partner contact and embodiment-specific poses.

3) Mimetic Embodiment Transfer Objective:

  • The mimetic transfer term aligns the human-humanoid interaction with the corresponding human-human interaction over the interaction horizon.The objective is expressed as Einteract(Rt, Pt, HSMPL,t).Source: 3) Mimetic Embodiment Transfer Objective · p. 4
  • At each timestep, the mink solver combines the interaction term with inverse-kinematics and Tikhonov smoothness terms to solve the humanoid pose.The passage denotes the solved pose as bRt.Source: 3) Mimetic Embodiment Transfer Objective · p. 4
  • HumiTransfer+ preserves embodiment-specific poses and maintains contact with the human partner, unlike the compared retargeting methods.The comparisons include naive IK [25], GMR [23], HumiTransfer−, and PAIR [37].Source: Figure 4 · p. 5

C. Post-Retargeting Interaction Optimization

Post-retargeting optimization addresses self-balancing and inter-body collision challenges while poses are rolled out. The optimization maintains a minimum safe distance between bodies throughout the interaction horizon, and the resulting sequences form HumiMimic.

C. Post-Retargeting Interaction Optimization

  • Post-retargeting optimization addresses self-balancing and inter-body collision challenges during pose rollout.The approach follows a full-body compliant human-humanoid interaction mechanism for self-balancing and maintains a minimum safe inter-body distance.Source: C. Post-Retargeting Interaction… · p. 5
  • The post-retargeting optimization is applied throughout the human-humanoid interaction horizon for all sequences, which are used to produce HumiMimic.The framework is applied to all human-human interaction sequences to yield the final dataset.Source: C. Post-Retargeting Interaction… · p. 5

IV. EXPERIMENT

The experiments evaluate HumiTransfer, test HumiMimic for generative human-humanoid interaction, and assess real-world deployment through execution on a humanoid robot.

IV. EXPERIMENT

  • Experiments assess HumiTransfer, HumiMimic’s utility for interaction generation, and real-robot execution of a trained policy.The policy is trained on HumiMimic and executed on a real humanoid robot.Source: §IV · p. 5

A. Assessment of Extraction & Retargeting in HumiTransfer

The authors assess whether reconstructed human-human interactions are physically plausible, comparing foot-ground consistency fine-tuning with standard SMPL-based fitting. Fine-tuning produces lower error across the reported physical-plausibility metrics.

1) Physical Plausibility of Extracted Human-Human Interaction:

2) Mimetic Embodiment Transfer:

HumiTransfer’s semantic and distance-based weighting performs best across three retargeting metrics on both humanoids. HumiMimic also shows higher motion diversity than AMASS, with qualitative samples illustrating varied interactions.

2) Mimetic Embodiment Transfer:

  • The retargeting evaluation compares MPJPE, pairwise-distance error, and contact-preservation F1 against naive IK and optimization-based fitting methods.CF1 measures contact preservation at a distance threshold of 20 cm.Source: 2) Mimetic Embodiment Transfer · p. 5, Table III · p. 5
  • HumiTransfer+ achieves the best MPJPE, IRD, and CF1 results on both Unitree R1 Edu and HighTorque Mini-hi.The comparison includes naive IK, GMR, PAIR, and HumiTransfer without weighting; the table evaluates both embodiments.Source: Table III · p. 5, 2) Mimetic Embodiment Transfer · p. 5

3) Diversity of Retargeted Human-Humanoid Motion:

  • Qualitative HumiMimic samples show numerous human-humanoid interaction types drawn from in-the-wild human-human videos.The paper identifies eight representative interaction motions.Source: Figure 5 · p. 6, 3) Diversity of Retargeted… · p. 6
  • HumiMimic diversity scores are 6.2427 for Unitree R1 Edu and 6.2285 for HighTorque Mini-hi, versus AMASS scores of 5.4834 and 5.4859, respectively.The diversity comparison covers human-humanoid interactions retargeted to the two humanoids.Source: Table IV · p. 6, 3) Diversity of Retargeted… · p. 6

B. Generative Human-Humanoid Interaction Benchmarking

The generative benchmark evaluates whether humanoids can respond to a human partner while coordinating motion and contact. Duolando performs best overall, while simulations report close CoM heights and preserved interaction relationships for both robots; real Unitree demonstrations include hugging and hand-shaking.

B. Generative Human-Humanoid Interaction Benchmarking

  • Given a human motion sequence, the benchmark asks models to synthesize responsive humanoid motion that synchronizes with the partner while maintaining physical and semantic coordination.The HumiMimic dataset is split 8:2 for training and testing; evaluation measures kinematic quality and interaction quality.Source: B. Generative Human-Humanoid… · p. 6
  • Duolando records FIDk values of 24.96 and 27.24, FIDcd values of 9.07 and 11.53, and fcontact values of 51.75% and 46.77%, achieving the best overall performance.The benchmark compares five motion-generation architectures on the HumiMimic test set.Source: Table V · p. 6, B. Generative Human-Humanoid… · p. 6

1) Self-Balancing Evaluation:

  • In simulation, Unitree R1 Edu and HighTorque Mini-hi maintain CoM-height differences of 2.9573 cm and 3.2021 cm, respectively, while preserving interaction-critical relationships.Both reproduce crouching motions while maintaining retargeted human CoM trajectories and their respective morphologies.Source: 1) Self-Balancing Evaluation · pp. 6–7, Figure 7 · p. 7

C. Self-Balancing Evaluation & Real-Robot Deployment

2) Real-Robot Deployment:

The authors deploy retargeted HumiMimic motions on Unitree R1 Edu using a motion-tracking controller and report real-world interaction demonstrations. They also identify morphology-related infeasibility and the current reliance on predefined motions without explicit human-motion response.

2) Real-Robot Deployment:

  • The Unitree R1 Edu deployment maps HumiMimic sequences to robot joints, checks motion constraints, and executes a trained motion-tracking policy through the C++ mimic controller.The policy is exported to ONNX and run at 50 Hz using robot motor states and IMU observations.Source: 2) Real-Robot Deployment · p. 7

D. Limitations & Future Work

  • Substantial human-humanoid morphological differences can make some motions infeasible; the Mini-hi’s broad shoulders and short arms prevent the hug achieved by Unitree R1 Edu.Current real-robot deployment primarily uses predefined sequences and does not explicitly account for human motion.Source: D. Limitations & Future Work · p. 7

V. CONCLUSION

  • The conclusion presents HumiMimic as a benchmark for future human-humanoid interaction work, supported by motion-generation benchmarking and Unitree R1 Edu deployment.The dataset follows a pipeline that extracts physically plausible human-human poses from in-the-wild videos and retargets one person to different humanoids.Source: §V · p. 8

Related papers

Loading 2610.36615v1…