PaperlayerView exact revision on arXiv

Source-linked AI summary

HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos

Yinhuai Wang, Qihan Zhao, Yuen Fui Lau, Runyi Yu, Hok Wai Tsui, Qifeng Chen, Jingbo Wang, Jiangmiao Pang, Ping Tan

arXiv:2602.02473v1cs.ROcs.LG

TL;DR

Humanoid interaction learning is limited by scarce realistic data and reliance on task-specific reward engineering. HumanX addresses this gap by combining XGen’s physically plausible video-to-data synthesis with XMimic’s unified imitation learning. Across 10 skills in five domains, it achieves over 8× higher generalization success than prior approaches and transfers skills to a physical Unitree G1.

  • Problem

    Existing humanoid interaction methods are limited by scarce realistic interaction data and meticulously engineered task-specific rewards.

  • Method

    HumanX combines XGen, which synthesizes and augments physically plausible interaction data from human video, with XMimic, which learns unified imitation policies.

  • Results

    HumanX achieves over 8× higher generalization success than prior approaches across 10 skills in five domains and transfers them to a Unitree G1 humanoid.

  • Takeaways & Limitations

    The learned skills include perception-free agile maneuvers and sustained closed-loop human-robot interactions, with each skill learned from a single video demonstration.

Abstract

from arXiv · show

Enabling humanoid robots to perform agile and adaptive interactive tasks has long been a core challenge in robotics. Current approaches are bottlenecked by either the scarcity of realistic interaction data or the need for meticulous, task-specific reward engineering, which limits their scalability. To narrow this gap, we present HumanX, a full-stack framework that compiles human video into generalizable, real-world interaction skills for humanoids, without task-specific rewards. HumanX integrates two co-designed components: XGen, a data generation pipeline that synthesizes diverse and physically plausible robot interaction data from video while supporting scalable data augmentation; and XMimic, a unified imitation learning framework that learns generalizable interaction skills. Evaluated across five distinct domains--basketball, football, badminton, cargo pickup, and reactive fighting--HumanX successfully acquires 10 different skills and transfers them zero-shot to a physical Unitree G1 humanoid. The learned capabilities include complex maneuvers such as pump-fake turnaround fadeaway jumpshots without any external perception, as well as interactive tasks like sustained human-robot passing sequences over 10 consecutive cycles--learned from a single video demonstration. Our experiments show that HumanX achieves over 8 times higher generalization success than prior methods, demonstrating a scalable and task-agnostic pathway for learning versatile, real-world robot interactive skills.

I. INTRODUCTION

HumanX addresses the limited availability of realistic interaction data and the task-specific reward engineering required by existing approaches. It combines XGen for physically plausible, augmentable interaction data with XMimic for unified imitation learning, achieving broad real-world skill generalization.

  • I. INTRODUCTION: HumanX compiles human video into generalizable humanoid interaction skills without task-specific reward design.Its two components are XGen, which synthesizes and augments interaction data, and XMimic, which learns by imitating that data.
  • I. INTRODUCTION: XGen prioritizes physically plausible interaction trajectories over photometrically faithful reconstruction to address occlusion and depth ambiguity.It combines motion estimates under physical priors rather than independently reconstructing human and object motion.
  • I. INTRODUCTION: XMimic targets accurate, natural, and generalizable human-object interaction through unified rewards, flexible perception, and additional architectural innovations.The supplied passage identifies unified reward and flexible perception as two of four innovations.
  • I. INTRODUCTION: HumanX evaluates 10 skills across basketball, football, badminton, cargo handling, and robot-human fighting on a Unitree G1 humanoid.Skills include perception-free basketball maneuvers and closed-loop interactions using MoCap-based object sensing.
  • I. INTRODUCTION: HumanX achieves over 8× higher generalization success than prior approaches while learning each skill from a single video demonstration.The evaluation includes over 80% average success for perception-free basketball skills and sustained passing interactions.

A. Extracting Humanoid Motion from Human Video

XGen extracts humanoid motion from monocular human video by estimating and retargeting human poses, then constructing interaction trajectories around an anchor-object relationship. Contact phases are refined with physical constraints to preserve plausible manipulation across morphologies.

  • A. Extracting Humanoid Motion from Human Video: A monocular video is processed into 3D human pose estimates before retargeting the motion to the target humanoid.GVHMR provides the initial pose sequence, while GMR performs keypoint alignment, skeleton scaling, and IK-based optimization.
  • A. Extracting Humanoid Motion from Human Video: The retargeted robot motion represents root pose and joint rotations for the humanoid embodiment.The robot root has a 6D pose, while the robot joints use one-dimensional rotations.
  • A. Extracting Humanoid Motion from Human Video: XGen segments each interaction into contact and non-contact phases around the beginning and end of object contact.For box carrying, the phases are before contact, during contact, and after contact.
  • A. Extracting Humanoid Motion from Human Video: The anchor-object relative motion transfers interaction semantics across human and humanoid morphologies.An anchor such as the midpoint between the palms defines the relationship used to synthesize the object trajectory.
  • A. Extracting Humanoid Motion from Human Video: Force-closure optimization refines robot poses during contact, and interpolation smooths motion at phase transitions.The optimization yields refined robot and object poses, while linear interpolation is applied around transitions.

2) The Non-Contact Phase:

XGen synthesizes non-contact object motion with physics simulation and augments interaction data across object and trajectory variations. These operations expand training coverage while preserving physically plausible interactions.

  • 2) The Non-Contact Phase:: Physics simulation generates non-contact object trajectories before and after contact.Post-contact trajectories are simulated forward from the ending pose, while pre-contact trajectories can be simulated backward from the starting pose.
  • 2) The Non-Contact Phase:: XGen augments object geometry by scaling meshes or replacing them with different geometries while preserving physical plausibility.A single demonstration can support similar actions on different objects.
  • 2) The Non-Contact Phase:: Contact-phase trajectories are varied through geometric transformations such as translation and scaling.For example, one box-lifting demonstration can generate lifts from different heights.
  • 2) The Non-Contact Phase:: Non-contact trajectory diversity increases by randomizing initial object velocity in simulation.One shuttlecock-hitting demonstration can produce different parabolic trajectories, and one basketball video can produce shots from various distances.
  • 2) The Non-Contact Phase:: XMimic uses the synthesized interaction data to learn a broad repertoire of humanoid interaction skills.The framework is designed to support accurate imitation, strong generalization, and flexible deployment.

A. Teacher-Student Training Architecture

XMimic trains interaction policies through a teacher-student pipeline: privileged-state teachers learn individual skills, then a student policy is distilled for realistic deployment. The student combines reinforcement learning with behavior cloning while using limited observations.

  • A. Teacher-Student Training Architecture: The policy outputs a Gaussian action distribution whose sampled joint actions are converted into torques by a PD controller.An MLP predicts the distribution mean, and the covariance matrix is learnable.
  • A. Teacher-Student Training Architecture: XMimic first trains individual teacher policies on XGen datasets using privileged state observations and PPO.Each teacher receives proprioception, privileged body information, and object state, then maximizes cumulative interaction-imitation reward.
  • A. Teacher-Student Training Architecture: The two-stage pipeline produces a student policy that can operate under realistic perceptual constraints and deploy directly in real-world settings.The teacher uses privileged information, whereas the distilled student is designed for deployment.
  • A. Teacher-Student Training Architecture: The student is trained on the union of skill datasets while excluding privileged state information from its observations.Its observations retain proprioception and optional object observations.
  • A. Teacher-Student Training Architecture: Student training combines the PPO policy-gradient objective with behavior cloning from pretrained teachers.The behavior-cloning term distills teacher knowledge into the unified student policy.

1) Perceiving External Force from Proprioception:

XMimic supports two deployment modes that trade external object sensing for proprioceptive simplicity, while its imitation objective tracks body, object, relative-motion, contact, and smoothness signals.

  • XMimic provides No External Perception and MoCap-based deployment modes for interaction skills.
  • NEP mode removes object observations so the robot relies solely on proprioception, enabling dynamic skills without external sensors but not non-contact catching.
  • MoCap mode trains with simulated frame loss, enabling zero-shot adaptation to intermittent real-world object-tracking streams.
  • Simulation results show accurate and natural basketball interactions across novel passing trajectories and target positions, alongside multiple football-kicking and badminton-hitting patterns.
  • XMimic’s composite reward combines body imitation, object tracking, relative body–object motion, contact timing, and motion regularization.The body term includes position, rotation, joint-position, velocity, and adversarial motion-prior components.

D. Simulation Settings

HumanX evaluates generalization by varying object states and interaction conditions, using disturbed initialization, interaction-aware termination, domain randomization, and diverse XGen data across simulation and real-robot experiments.

  • Simulation evaluation reports SR, Eo, Eh, and GSR, with GSR measured on cases sampled from the augmented distribution.
  • Domain randomization varies object and robot physics, perception noise, and external forces to support robust deployment.
  • Generalization means executing consistent interactions while adapting to variations in the interacting object’s state.
  • HumanX combines diverse XGen data, disturbed initialization, and interaction-aware termination to expand state coverage and prioritize interaction success.
  • From one video demonstration, HumanX learns sustained human–robot basketball passing for over ten consecutive cycles.
  • Experiments include three simulation skills and five real-world domains covering ten interaction skills on a Unitree G1 humanoid.

B. Simulation Experiments

Simulation and deployment experiments test HumanX across representative interaction tasks, perception settings, and skill patterns. The method achieves strong generalization from single demonstrations and supports diverse real-robot behaviors.

  • Main Evaluation and Ablation Study: Simulation evaluates basketball catch-shot, badminton hitting, and cargo pickup, comparing XMimic ablations with SkillMimic, OmniRetarget, and HDMI.
  • Main Evaluation and Ablation Study: Each task begins with one demonstration clip, while Data Augmentation and Teacher–Student settings expand training to 50 interaction clips.
  • Main Evaluation and Ablation Study: HumanX exceeds 80% average GSR, approximately 8× higher than HDMI, while reduced overfitting can slightly lower SR on some tasks.
  • Evaluation on Multi-Pattern Interaction Skills: The method learns multiple football-kicking and badminton-hitting patterns, with the teacher–student framework especially beneficial for multi-pattern learning.
  • Real-World Deployment: Real-robot evaluation covers NEP and MoCap schemes, with each skill trained from a single video demonstration.
  • NEP Mode: In NEP mode, five basketball skills—including pump-fake turnaround fadeaway—are evaluated over 10 trials per skill using proprioceptive control.

2) MoCap Mode:

MoCap-mode evaluation tests HumanX on sustained human-robot interactions and recovery under disturbances. The learned skills support prolonged closed-loop behavior, adaptive recovery, and deployment stability shaped by training perturbations.

  • MoCap Mode:: HumanX evaluates Cargo Pickup, sustained Basketball Catch-and-Pass, Football Kicking, and Reactive Fighting over 50 trials per skill.
  • MoCap Mode:: Over 10 consecutive basketball catch-and-pass cycles and over 14 football return kicks demonstrate prolonged interaction despite variable human behavior.The basketball policy can resume after a dropped ball, while football performance tolerates variability in human passes.
  • MoCap Mode:: Sustained random-force and MoCap-signal-loss augmentation are important for preventing balance loss and collapse during deployment.
  • MoCap Mode:: The robot maintains stable cargo grasping under pushes, retrieves relocated objects, and distinguishes fighting feints from genuine attacks.These behaviors indicate adaptive closed-loop execution rather than simple trajectory reproduction.
  • MoCap Mode:: HumanX acquires agile interaction skills across 10 skills and five domains, including perception-free maneuvers and sustained human-robot passes.

1) Body Motion Imitation Reward:

The body motion imitation reward decomposes imitation into multiple tracking terms, while the broader reward design also tracks object state and body-object relationships. These components support accurate and natural imitation of interaction behaviors.

  • 1) Body Motion Imitation Reward:: The body reward tracks body position, body rotation, joint position, body linear velocity, body angular velocity, and joint velocity.An adversarial motion prior further enhances body-motion smoothness and naturalness.
  • 1) Body Motion Imitation Reward:: Each non-AMP imitation sub-reward follows a shared formulation based on modality-specific imitation error, weight, and sensitivity hyperparameters.
  • 1) Body Motion Imitation Reward:: The object reward tracks object position and rotation, while the relative-motion reward preserves body-object positioning and orientation.The relative-motion terms use the same formulation for their respective errors.
  • 1) Body Motion Imitation Reward:: Relative position compares reference and simulated vectors defined from body keypoints to the object.A keypoint can be the left middle fingertip, and the object position supplies the corresponding object reference.
  • 1) Body Motion Imitation Reward:: Relative rotation uses a distance metric on SO(3) to preserve the object’s orientation relative to body keypoints.

4) Contact Graph Imitation Reward:

The contact-graph imitation reward teaches the policy to reproduce when and where the robot contacts an object. XGen supplies frame-level contact labels, which are compared with simulated contact states.

  • 4) Contact Graph Imitation Reward:: XGen annotates each frame with contact labels indicating whether the object and key bodies are touching.
  • 4) Contact Graph Imitation Reward:: The contact state is represented as a binary vector over contact bodies, with 1 indicating active contact.
  • 4) Contact Graph Imitation Reward:: The reward compares simulated and reference contact states through an element-wise absolute contact-error vector.
  • 4) Contact Graph Imitation Reward:: An exponential of the weighted contact error penalizes mismatches and encourages precise contact timing and location.

IX. PERCEIVING EXTERNAL FORCE FROM PROPRIOCEPTION

Humanoid proprioception can support implicit perception of external interaction forces. The analysis derives how force-related quantities arise from the robot’s dynamics and informs the policy’s observation design.

  • IX. PERCEIVING EXTERNAL FORCE FROM PROPRIOCEPTION: Humans can perform stable interactive behaviors without visual input by relying on tactile and force feedback.The paper uses this capability to motivate analogous force perception in humanoid robots.
  • IX. PERCEIVING EXTERNAL FORCE FROM PROPRIOCEPTION: The floating-base humanoid dynamics relate torque to inertia, Coriolis and centrifugal effects, gravity, friction, and external contact forces.
  • IX. PERCEIVING EXTERNAL FORCE FROM PROPRIOCEPTION: External forces on each joint can be estimated when the corresponding dynamic parameters and observations are available.
  • IX. PERCEIVING EXTERNAL FORCE FROM PROPRIOCEPTION: For the Unitree G1, joint positions and velocities are measurable, while commanded PD-controller torque approximates actual torque.
  • IX. PERCEIVING EXTERNAL FORCE FROM PROPRIOCEPTION: The final policy observations account explicitly or implicitly for all variables appearing in the external-force equation.
Loading 2602.02473v1…