PaperlayerView exact revision on arXiv

Source-linked AI summary

HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos

Yinhuai Wang, Qihan Zhao, Yuen Fui Lau, Runyi Yu, Hok Wai Tsui, Qifeng Chen, Jingbo Wang, Jiangmiao Pang, Ping Tan

arXiv:2602.02473v1cs.ROcs.LG

TL;DR

Humanoid interaction learning is limited by scarce realistic data and reliance on task-specific reward engineering. HumanX addresses this gap by combining XGen’s physically plausible video-to-data synthesis with XMimic’s unified imitation learning. Across 10 skills in five domains, it achieves over 8× higher generalization success than prior approaches and transfers skills to a physical Unitree G1.

Abstract

from arXiv · show

Enabling humanoid robots to perform agile and adaptive interactive tasks has long been a core challenge in robotics. Current approaches are bottlenecked by either the scarcity of realistic interaction data or the need for meticulous, task-specific reward engineering, which limits their scalability. To narrow this gap, we present HumanX, a full-stack framework that compiles human video into generalizable, real-world interaction skills for humanoids, without task-specific rewards. HumanX integrates two co-designed components: XGen, a data generation pipeline that synthesizes diverse and physically plausible robot interaction data from video while supporting scalable data augmentation; and XMimic, a unified imitation learning framework that learns generalizable interaction skills. Evaluated across five distinct domains--basketball, football, badminton, cargo pickup, and reactive fighting--HumanX successfully acquires 10 different skills and transfers them zero-shot to a physical Unitree G1 humanoid. The learned capabilities include complex maneuvers such as pump-fake turnaround fadeaway jumpshots without any external perception, as well as interactive tasks like sustained human-robot passing sequences over 10 consecutive cycles--learned from a single video demonstration. Our experiments show that HumanX achieves over 8 times higher generalization success than prior methods, demonstrating a scalable and task-agnostic pathway for learning versatile, real-world robot interactive skills.

I. INTRODUCTION

HumanX addresses the limited availability of realistic interaction data and the task-specific reward engineering required by existing approaches. It combines XGen for physically plausible, augmentable interaction data with XMimic for unified imitation learning, achieving broad real-world skill generalization.

I. INTRODUCTION

  • HumanX compiles human video into generalizable humanoid interaction skills without task-specific reward design.Its two components are XGen, which synthesizes and augments interaction data, and XMimic, which learns by imitating that data.Source: §I · p. 1
  • XGen prioritizes physically plausible interaction trajectories over photometrically faithful reconstruction to address occlusion and depth ambiguity.It combines motion estimates under physical priors rather than independently reconstructing human and object motion.Source: §I · pp. 1–2
  • XMimic targets accurate, natural, and generalizable human-object interaction through unified rewards, flexible perception, and additional architectural innovations.The supplied passage identifies unified reward and flexible perception as two of four innovations.Source: §I · p. 2
  • HumanX evaluates 10 skills across basketball, football, badminton, cargo handling, and robot-human fighting on a Unitree G1 humanoid.Skills include perception-free basketball maneuvers and closed-loop interactions using MoCap-based object sensing.Source: §I · p. 2
  • HumanX achieves over 8× higher generalization success than prior approaches while learning each skill from a single video demonstration.The evaluation includes over 80% average success for perception-free basketball skills and sustained passing interactions.Source: §I · p. 2, 2) Reinforcement Learning for Humanoid… · p. 3

A. Extracting Humanoid Motion from Human Video

XGen extracts humanoid motion from monocular human video by estimating and retargeting human poses, then constructing interaction trajectories around an anchor-object relationship. Contact phases are refined with physical constraints to preserve plausible manipulation across morphologies.

A. Extracting Humanoid Motion from Human Video

  • A monocular video is processed into 3D human pose estimates before retargeting the motion to the target humanoid.GVHMR provides the initial pose sequence, while GMR performs keypoint alignment, skeleton scaling, and IK-based optimization.Source: A. Extracting Humanoid Motion from… · p. 3
  • The retargeted robot motion represents root pose and joint rotations for the humanoid embodiment.The robot root has a 6D pose, while the robot joints use one-dimensional rotations.Source: A. Extracting Humanoid Motion from… · p. 3
  • XGen segments each interaction into contact and non-contact phases around the beginning and end of object contact.For box carrying, the phases are before contact, during contact, and after contact.Source: B. Synthesizing Humanoid-Object… · pp. 3–4
  • The anchor-object relative motion transfers interaction semantics across human and humanoid morphologies.An anchor such as the midpoint between the palms defines the relationship used to synthesize the object trajectory.Source: 1) The Contact Phase · p. 4
  • Force-closure optimization refines robot poses during contact, and interpolation smooths motion at phase transitions.The optimization yields refined robot and object poses, while linear interpolation is applied around transitions.Source: 1) The Contact Phase · p. 4

2) The Non-Contact Phase:

XGen synthesizes non-contact object motion with physics simulation and augments interaction data across object and trajectory variations. These operations expand training coverage while preserving physically plausible interactions.

2) The Non-Contact Phase:

  • Physics simulation generates non-contact object trajectories before and after contact.Post-contact trajectories are simulated forward from the ending pose, while pre-contact trajectories can be simulated backward from the starting pose.Source: 2) The Non-Contact Phase · p. 4
  • XGen augments object geometry by scaling meshes or replacing them with different geometries while preserving physical plausibility.A single demonstration can support similar actions on different objects.Source: C. Interaction Augmentation · p. 4, 1) Scaling Object Geometry · p. 4
  • Contact-phase trajectories are varied through geometric transformations such as translation and scaling.For example, one box-lifting demonstration can generate lifts from different heights.Source: 2) Enriching Object Trajectories in… · p. 4
  • Non-contact trajectory diversity increases by randomizing initial object velocity in simulation.One shuttlecock-hitting demonstration can produce different parabolic trajectories, and one basketball video can produce shots from various distances.Source: 3) Enriching Object Trajectories in… · p. 4
  • XMimic uses the synthesized interaction data to learn a broad repertoire of humanoid interaction skills.The framework is designed to support accurate imitation, strong generalization, and flexible deployment.Source: §IV · p. 4

A. Teacher-Student Training Architecture

XMimic trains interaction policies through a teacher-student pipeline: privileged-state teachers learn individual skills, then a student policy is distilled for realistic deployment. The student combines reinforcement learning with behavior cloning while using limited observations.

A. Teacher-Student Training Architecture

  • The policy outputs a Gaussian action distribution whose sampled joint actions are converted into torques by a PD controller.An MLP predicts the distribution mean, and the covariance matrix is learnable.Source: 1) Policy Formulation · p. 4
  • XMimic first trains individual teacher policies on XGen datasets using privileged state observations and PPO.Each teacher receives proprioception, privileged body information, and object state, then maximizes cumulative interaction-imitation reward.Source: 2) Training Privileged Teacher Policies · p. 4
  • The two-stage pipeline produces a student policy that can operate under realistic perceptual constraints and deploy directly in real-world settings.The teacher uses privileged information, whereas the distilled student is designed for deployment.Source: A. Teacher-Student Training Architecture · p. 4, Figure 5 · p. 5
  • The student is trained on the union of skill datasets while excluding privileged state information from its observations.Its observations retain proprioception and optional object observations.Source: 3) Distilling Teachers into a… · p. 5
  • Student training combines the PPO policy-gradient objective with behavior cloning from pretrained teachers.The behavior-cloning term distills teacher knowledge into the unified student policy.Source: 3) Distilling Teachers into a… · p. 5, Figure 5 · p. 5

1) Perceiving External Force from Proprioception:

XMimic supports two deployment modes that trade external object sensing for proprioceptive simplicity, while its imitation objective tracks body, object, relative-motion, contact, and smoothness signals.

  • XMimic provides No External Perception and MoCap-based deployment modes for interaction skills.Source: 2) Flexible Object Perception for… · p. 5
  • NEP mode removes object observations so the robot relies solely on proprioception, enabling dynamic skills without external sensors but not non-contact catching.Source: 2) Flexible Object Perception for… · p. 5
  • MoCap mode trains with simulated frame loss, enabling zero-shot adaptation to intermittent real-world object-tracking streams.Source: 2) Flexible Object Perception for… · p. 5
  • Simulation results show accurate and natural basketball interactions across novel passing trajectories and target positions, alongside multiple football-kicking and badminton-hitting patterns.Source: Figure 6 · p. 5, Figure 7 · p. 5
  • XMimic’s composite reward combines body imitation, object tracking, relative body–object motion, contact timing, and motion regularization.The body term includes position, rotation, joint-position, velocity, and adversarial motion-prior components.Source: C. Unified Interaction Imitation Reward · p. 5

D. Simulation Settings

HumanX evaluates generalization by varying object states and interaction conditions, using disturbed initialization, interaction-aware termination, domain randomization, and diverse XGen data across simulation and real-robot experiments.

  • Simulation evaluation reports SR, Eo, Eh, and GSR, with GSR measured on cases sampled from the augmented distribution.Source: Table I · p. 6
  • Domain randomization varies object and robot physics, perception noise, and external forces to support robust deployment.Source: 3) Domain Randomization · p. 6
  • Generalization means executing consistent interactions while adapting to variations in the interacting object’s state.Source: §V · p. 6
  • HumanX combines diverse XGen data, disturbed initialization, and interaction-aware termination to expand state coverage and prioritize interaction success.Source: §V · p. 6, 2) Interaction Termination · p. 6
  • From one video demonstration, HumanX learns sustained human–robot basketball passing for over ten consecutive cycles.Source: §V · p. 6
  • Experiments include three simulation skills and five real-world domains covering ten interaction skills on a Unitree G1 humanoid.Source: §VI · p. 6, A. Experimental Settings · p. 6

B. Simulation Experiments

Simulation and deployment experiments test HumanX across representative interaction tasks, perception settings, and skill patterns. The method achieves strong generalization from single demonstrations and supports diverse real-robot behaviors.

Main Evaluation and Ablation Study

Evaluation on Multi-Pattern Interaction Skills

Real-World Deployment

NEP Mode

  • In NEP mode, five basketball skills—including pump-fake turnaround fadeaway—are evaluated over 10 trials per skill using proprioceptive control.Source: 1) NEP Mode · p. 8

2) MoCap Mode:

MoCap-mode evaluation tests HumanX on sustained human-robot interactions and recovery under disturbances. The learned skills support prolonged closed-loop behavior, adaptive recovery, and deployment stability shaped by training perturbations.

MoCap Mode:

  • HumanX evaluates Cargo Pickup, sustained Basketball Catch-and-Pass, Football Kicking, and Reactive Fighting over 50 trials per skill.Source: 2) MoCap Mode · p. 8
  • Over 10 consecutive basketball catch-and-pass cycles and over 14 football return kicks demonstrate prolonged interaction despite variable human behavior.The basketball policy can resume after a dropped ball, while football performance tolerates variability in human passes.Source: 2) MoCap Mode · p. 8
  • Sustained random-force and MoCap-signal-loss augmentation are important for preventing balance loss and collapse during deployment.Source: Figure 12 · p. 8
  • The robot maintains stable cargo grasping under pushes, retrieves relocated objects, and distinguishes fighting feints from genuine attacks.These behaviors indicate adaptive closed-loop execution rather than simple trajectory reproduction.Source: 2) MoCap Mode · p. 8
  • HumanX acquires agile interaction skills across 10 skills and five domains, including perception-free maneuvers and sustained human-robot passes.Source: §VII · p. 8

1) Body Motion Imitation Reward:

The body motion imitation reward decomposes imitation into multiple tracking terms, while the broader reward design also tracks object state and body-object relationships. These components support accurate and natural imitation of interaction behaviors.

1) Body Motion Imitation Reward:

4) Contact Graph Imitation Reward:

The contact-graph imitation reward teaches the policy to reproduce when and where the robot contacts an object. XGen supplies frame-level contact labels, which are compared with simulated contact states.

4) Contact Graph Imitation Reward:

IX. PERCEIVING EXTERNAL FORCE FROM PROPRIOCEPTION

Humanoid proprioception can support implicit perception of external interaction forces. The analysis derives how force-related quantities arise from the robot’s dynamics and informs the policy’s observation design.

IX. PERCEIVING EXTERNAL FORCE FROM PROPRIOCEPTION

  • Humans can perform stable interactive behaviors without visual input by relying on tactile and force feedback.The paper uses this capability to motivate analogous force perception in humanoid robots.Source: §IX · pp. 12–13
  • The floating-base humanoid dynamics relate torque to inertia, Coriolis and centrifugal effects, gravity, friction, and external contact forces.Source: §IX · p. 13
  • External forces on each joint can be estimated when the corresponding dynamic parameters and observations are available.Source: §IX · p. 13
  • For the Unitree G1, joint positions and velocities are measurable, while commanded PD-controller torque approximates actual torque.Source: §IX · p. 13
  • The final policy observations account explicitly or implicitly for all variables appearing in the external-force equation.Source: §IX · p. 13

Related papers

Loading 2602.02473v1…