PaperlayerView exact revision on arXiv

Source-linked AI summary

HumDex: Humanoid Dexterous Manipulation Made Easy

Liang Heng, Yihe Tang, Jiajun Xu, Henghui Bao, Di Huang, Yue Wang

arXiv:2603.12260v2cs.RO

TL;DR

Humanoid dexterous manipulation depends on high-quality demonstrations, but collecting robot data is costly and existing teleoperation systems face portability, occlusion, or precision trade-offs. HumDex combines IMU-based whole-body tracking, learned hand retargeting, and human-data pretraining followed by robot-data fine-tuning. It reports faster collection, higher teleoperation and downstream policy success, and improved generalization to unseen settings.

  • Problem

    High-quality demonstrations for humanoid dexterous manipulation are costly to collect, while existing teleoperation systems face portability, occlusion, or precision limitations.

    Source: §I · pp. 1–2

  • Method

    HumDex combines IMU-based whole-body teleoperation and learned hand retargeting with human-data pretraining followed by robot-data fine-tuning.

    Source: §I · p. 2, A. Whole-Body Dexterous Teleoperation… · p. 3

  • Results

    For 60 episodes, collection time falls from 59.8 minutes (Baseline) to 44.3 minutes (Ours), while average teleoperation success is 91.7% for HumDex versus 74.6% for the baseline.

    Source: 3) Results · p. 7

  • Takeaways & Limitations

    The two-stage framework uses human motion data to improve generalization to unseen object positions, categories, and backgrounds while requiring only a small amount of robot-specific data.

    Source: §V · p. 9

    Training data could not be scaled further because of computational and time constraints.

    Source: §VI · p. 9

Abstract

from arXiv · show

This paper investigates humanoid whole-body dexterous manipulation, where the efficient collection of high-quality demonstration data remains a central bottleneck. Existing teleoperation systems often suffer from limited portability, occlusion, or insufficient precision, which hinders their applicability to complex whole-body tasks. To address these challenges, we introduce HumDex, a portable teleoperation system designed for humanoid whole-body dexterous manipulation. Our system leverages IMU-based motion tracking to address the portability-precision trade-off, enabling accurate full-body tracking while remaining easy to deploy. For dexterous hand control, we further introduce a learning-based retargeting method that generates smooth and natural hand motions without manual parameter tuning. Beyond teleoperation, HumDex enables efficient collection of human motion data. Building on this capability, we propose a two-stage imitation learning framework that first pre-trains on diverse human motion data to learn generalizable priors, and then fine-tunes on robot data to bridge the embodiment gap for precise execution. We demonstrate that this approach significantly improves generalization to new configurations, objects, and backgrounds with minimal data acquisition costs. The entire system is fully reproducible and open-sourced at https://github.com/physical-superintelligence-lab/humdex.

I. INTRODUCTION

Humanoid dexterous manipulation needs high-quality demonstrations, but collecting them is costly and current teleoperation systems trade portability against accuracy. HumDex combines portable IMU tracking and learned hand retargeting with a two-stage human-to-robot imitation-learning framework.

I. INTRODUCTION

  • Humanoid demonstration collection remains costly because complex robot morphology makes high-quality task data difficult to acquire.Existing humanoid manipulation methods rely heavily on robot teleoperation data.Source: §I · p. 1

A. Humanoid Whole-Body Dexterous Teleoperation

  • Mocap and exoskeleton systems offer accuracy but require fixed infrastructure, while VR systems are portable but face occlusion and reduced accuracy.Occlusion constrains feasible demonstrated motions; optimization-based hand retargeting also has limited accuracy and generalization.Source: §I · pp. 1–2

I. INTRODUCTION

  • HumDex uses portable IMU-based tracking and learned hand retargeting to collect whole-body dexterous demonstrations with smooth motions and no manual parameter tuning.The introduction reports improved real-world hand-retargeting performance relative to optimization-based alternatives.Source: §I · p. 2

A. Humanoid Whole-Body Dexterous Teleoperation

  • Prior humanoid teleoperation often uses simplified end effectors, whereas HumDex supports fine-grained control of a 20-DoF hand during whole-body tasks.Its example task includes grasping a barcode scanner and pulling its trigger.Source: A. Humanoid Whole-Body Dexterous… · p. 2

B. Dexterous Hand Retargeting

  • Dexterous hand retargeting maps human hand features to robot-hand configurations, and learning-based approaches predict configurations directly from observations.The paper frames this direction as reducing reliance on hand-crafted objectives and enabling constant-time inference.Source: B. Dexterous Hand Retargeting · p. 2

C. Imitation Policy Learning with Human Data

  • HumDex first trains on diverse human demonstrations, then fine-tunes on robot data to improve generalization without explicit human–robot alignment.The paper contrasts this with humanoid manipulation work relying on teleoperation data and notes the larger human–robot embodiment gap.Source: C. Imitation Policy Learning with… · p. 3

III. METHOD

HumDex maps an operator’s IMU-captured motion to a humanoid through a hierarchical control framework. A low-level controller handles body balance and locomotion, while learned hand retargeting supplies dexterous hand targets.

A. Whole-Body Dexterous Teleoperation System

  • HumDex maps wearable-IMU human motion to a humanoid for diverse loco-manipulation tasks using separate high- and low-level control components.The architecture follows TWIST2’s hierarchical design.Source: A. Whole-Body Dexterous Teleoperation… · p. 3

1) Low-Level Hierarchical Control Interface:

  • The low-level motion-tracking policy handles body stability, while the hand targets are directly executed through joint-position controllers.Whole-body joint targets are divided into body targets for balancing and locomotion and hand targets for precise dexterous motion.Source: 1) Low-Level Hierarchical Control… · p. 3
  • The framework also supports low-level controllers such as SONIC, with body commands streamed through ZMQ for low-latency synchronization.This complements the TWIST2-based control interface.Source: 1) Low-Level Hierarchical Control… · pp. 3–4
  • A lightweight MLP maps five glove-tracked fingertip positions to the robot’s 20-DoF hand joint angles.The resulting hand targets are concatenated with body targets to form the unified reference command.Source: 1) Low-Level Hierarchical Control… · p. 4

2) High-Level Teleoperation via IMU-based Retargeting:

IMU-derived motion is retargeted to the robot using pelvis-relative tracking to reduce reliance on drift-prone absolute positions. The system pairs this whole-body mapping with inertial hand tracking and a learned hand retargeter.

2) High-Level Teleoperation via IMU-based Retargeting:

  • The tether-free IMU interface is designed for unobstructed operation and reports continuous use beyond 10 hours and connectivity over 50+ meters.The system is described as lightweight and unobtrusive compared with VR-based tracking or bulky optical motion capture.Source: 2) High-Level Teleoperation via… · p. 4, 3) Hardware Setup · p. 4
  • Pelvis-relative end-effector positions and link orientations guide robot configuration, avoiding drift-prone absolute-position measurements during whole-body tracking.The formulation uses the standard GMR solver; L_R and L_P denote the link sets used for orientation and position tracking.Source: 2) High-Level Teleoperation via… · p. 4

3) Hardware Setup:

  • The body interface uses a commercial 15-node tracker and also demonstrates a custom SlimeVR-based wearable system costing less than $200.The custom configuration ranges from 6 to 16 nodes; each node weighs under 20 g and supports over 20 hours of operation.Source: 3) Hardware Setup · p. 4

B. Learning-Based Dexterous Hand Retargeting.

  • Inertial gloves capture five fingertip positions in the wrist frame, which the learned retargeter maps to executable 20-DoF robot-hand joint targets.A lightweight MLP is trained on paired samples generated by offline optimization-based retargeting and produces smooth, continuous trajectories without per-frame optimization.Source: B. Learning-Based Dexterous Hand… · pp. 4–5

C. Two-Stage Imitation Learning with Human Data

The imitation-learning pipeline uses human demonstrations to learn motion priors, then robot data to refine execution for the robot embodiment. The section’s evaluations include generalization to unseen positions, objects, and backgrounds, with robot training data restricted to the seen setting.

C. Two-Stage Imitation Learning with Human Data

  • The policy uses an RGB image and body-and-hand joint positions as observations, and predicts motion-tracker targets rather than direct joint positions.The setup uses ACT with a ResNet-18 visual encoder.Source: C. Two-Stage Imitation Learning with… · p. 5
  • Because human data lacks robot proprioceptive states, the method approximates them with the previous timestep’s action, supported by action–state correspondence analysis.The analysis reports that robot action a_t corresponds to state s_{t+1} with minimal latency.Source: C. Two-Stage Imitation Learning with… · p. 5
  • Training proceeds sequentially: human demonstrations first teach generalizable motion priors, and robot data then refines behavior for the robot embodiment.This avoids explicit human–robot data alignment and enables human data collection without alignment.Source: C. Two-Stage Imitation Learning with… · p. 5

IV. EXPERIMENTS

  • The evaluation asks whether human data improves generalization to new positions, objects, and backgrounds.In the Task 5 generalization test, robot training data contains only the seen position, object, and background setting.Source: §IV · p. 5

1) Tasks:

  • The task suite covers bimanual towel threading, door opening, and whole-body basket placement, alongside scanner-and-toy handling and bread packing.The tasks respectively involve disturbance-prone hanger manipulation, coordinated locomotion, and whole-body movement; the scanner task also requires a spare finger to pull its trigger.Source: 1) Tasks · p. 6

2) Baseline and Comparison:

HumDex is compared with a vision-based teleoperation baseline using shared-task collection efficiency, teleoperation success, and downstream policy success. HumDex also completes Scan&Pack despite baseline failure from occlusion.

2) Baseline and Comparison:

  • The baseline uses PICO humanoid teleoperation with added vision-based hand tracking; comparisons use 60 attempts per task and downstream policies trained without human data.Policy success rates measure demonstration quality.Source: 2) Baseline and Comparison · p. 6

3) Results:

  • HumDex completes Scan&Pack with 90% teleoperation success, while severe occlusion prevents the vision-based baseline from performing the task.A barcode scanner occludes the operator’s hand, disrupting vision tracking and precise index-finger control.Source: 3) Results · pp. 6–7
  • Collecting 60 episodes takes 44.3 minutes with HumDex versus 59.8 minutes with the baseline, a 26% improvement on the four shared tasks.The comparison excludes Scan&Pack because the baseline could not perform it.Source: 3) Results · pp. 6–7
  • Policies trained on HumDex demonstrations achieve 80.0% average success, versus 57.5% for baseline demonstrations; teleoperation success is 91.7% versus 74.6%.On Hang Towel, policy success is 19/30 versus 11/30.Source: 3) Results · p. 7

B. Dexterous Hand Retargeting

The section evaluates hand retargeting and human-data pretraining followed by robot-data fine-tuning. The two-stage policy improves performance across distribution shifts, while retargeting is assessed on representative poses and dexterity-critical teleoperation tasks.

B. Dexterous Hand Retargeting

  • The retargeter maps five fingertip positions to 20-DoF Wuji hand joint targets and is compared with per-frame constrained optimization.Both methods use the same human observations and output joint targets at the same control rate.Source: B. Dexterous Hand Retargeting · p. 7
  • On canonical finger-touch and rock-sign poses, the learned retargeter provides more reliable pose realization than per-frame optimization.The open-loop replay evaluates pose correctness and unwanted bending or collapse of non-target fingers.Source: B. Dexterous Hand Retargeting · pp. 7–8, Figure 4 · p. 7
  • Closed-loop hand evaluation tests scanner-trigger actuation, hanger stabilization through five wrist cycles, and grasping a doll three times larger than the original.Scanner success requires five trigger actuations held for three seconds each; the hanger must remain grasped throughout.Source: B. Dexterous Hand Retargeting · p. 8
  • The lightweight MLP requires a short paired-sample calibration dataset, typically about 20k frames recorded in under 20 minutes.Source: B. Dexterous Hand Retargeting · p. 8

C. Human Data for Policy Generalization

  • Human data spans 100 position-varied episodes and 300 each for object and background variation, alongside 50 robot episodes at a fixed position.The object set includes an apple, banana, and leaf; background variation changes table textures and distractors.Source: C. Human Data for Policy Generalization · p. 8
  • RobotOnly performs perfectly in Seen but drops to 30% in UnseenBg under distribution shifts, while Ours improves success rates by nearly 2× across all generalization settings.Ours pre-trains on diverse human data and transfers to the robot embodiment during fine-tuning.Source: C. Human Data for Policy Generalization · p. 8, Table III · p. 8
  • Naively mixing human and robot data achieves 0% success, whereas sequential training separates human feature learning from robot-specific action adaptation.The authors attribute Mix failure to conflicting action spaces and optimization signals across the embodiment gap.Source: C. Human Data for Policy Generalization · p. 8

V. CONCLUSION

HumDex combines portable whole-body teleoperation with human-motion data and two-stage imitation learning, improving collection efficiency, teleoperation success, and downstream policy performance. The study leaves larger-scale training data, broader hand interactions, and stronger hardware for future work.

V. CONCLUSION

  • HumDex significantly improves data collection efficiency, teleoperation success rate, and downstream policy performance across diverse whole-body manipulation tasks.The two-stage framework uses human motion data to improve generalization while reducing the need for teleoperation data.Source: §V · p. 9
  • Human motion data in a two-stage imitation learning framework improves generalization to unseen object positions, categories, and visual backgrounds with little robot-specific adaptation data.The authors describe this as providing motion and perception priors.Source: §V · p. 9

VI. LIMITATION.

  • Scaling training data remains constrained by computational and time limits, while broader hand postures, contact modes, and force-sensitive interactions remain open challenges.Hardware payload and actuation limits also prevented exploration of potentially more capable manipulation behaviors.Source: §VI · p. 9

APPENDIX

The appendix describes HumDex’s robot, modular tracking interfaces, and learning-based hand-retargeting network. Its hand network uses separate subnetworks for each finger and concatenates their outputs.

A. Hardware Setup

  • The Unitree G1 Edu+ has 29 active DoF and a built-in RealSense D435i camera, while each custom WUJI hand has 20 actuated DoF.Each hand has four independent DoF per finger.Source: A. Hardware Setup · p. 11
  • The modular interface separates body and hand tracking, supports commercial Vdhand and Manus gloves, and allows tracking modules to be mixed without retargeting changes.Body tracking options include a Vdmocap suit and a custom SlimeVR system priced below $200; inertial gloves capture grasping under self-occlusion.Source: A. Hardware Setup · p. 11

B. Hand Retargeting Details

  • The hand-retargeting network is an MLP with a finger-wise modular architecture: five subnetworks each output four values, concatenated into a 20-value output.Each subnetwork takes three inputs and uses two 128-unit fully connected layers before its four-value output.Source: B. Hand Retargeting Details · p. 11

C. Data Collection Process

HumDex records human and robot demonstrations using synchronized vision and proprioception data. Human recordings capture retargeted whole-body and hand targets, while robot recordings capture executed state and commands.

C. Data Collection Process

1. Human Data Collection

  • Human collection records egocentric RGB images alongside GMR-retargeted 35-DoF whole-body targets and 40-DoF bimanual hand targets.The operator wears the VIRDYN suit and gloves without active robot execution; the retargeting loop runs at 100 Hz and recording at 30 Hz.Source: §1 · pp. 11–12

2. Robot Data Collection

  • Robot collection records egocentric RGB, the full robot state, and applied whole-body and bimanual hand target commands at 30 Hz.The operator controls the physical G1 through the same real-time GMR solver, with commands synchronized to the ZMQ video stream.Source: §2 · p. 12

D. Policy Learning Details

The policy-learning appendix specifies the observation and action spaces and training schedules, then validates teleoperation tracking against a commercial system and reduced tracker configurations. Fourteen-node SlimeVR performance is close to the commercial baseline, while fewer nodes reduce reliability on complex tasks.

D. Policy Learning Details

  • The policy observes egocentric RGB and robot joint positions, and predicts target joint positions using an Action Chunking Transformer backbone.The observation includes 31 whole-body dimensions and 20 dimensions per hand; actions include 35 body DoF and 20 DoF per hand.Source: D. Policy Learning Details · p. 12
  • The two-stage schedule trains on human data for 1,000 epochs, then robot data for 4,000 epochs; single-stage robot training uses 5,000–8,000 epochs across tasks.Both stages use batch size 16.Source: D. Policy Learning Details · p. 12

E. Teleoperation System Validation

  • 48/60 vs. 50/60 on HangTowel: 14-node SlimeVR achieves success comparable to the commercial baseline, while reduced tracker counts lower reliability on complex tasks.The 7-node setup achieves 42/60 on HangTowel; the table reports success trials out of 60 attempts.Source: Table A1 · p. 12, E. Teleoperation System Validation · p. 12

F. Failure Case Analysis

The failure analysis identifies task-specific breakdowns in dexterous manipulation, bimanual coordination, locomotion, whole-body stability, and spatial generalization. Failures range from an ineffective initial grasp to grasping at a position offset from the target.

F. Failure Case Analysis

  • An initial grasp that is too low or rotated can restrict thumb reach and prevent the robot from pressing the scanner trigger.The robot may hold the scanner but still fail to activate it.Source: F. Failure Case Analysis · p. 12
  • Spatial misalignment between the hands can cause the towel to hit the hanger rim instead of threading through its bar.The left hand holds the hanger while the right holds the towel.Source: F. Failure Case Analysis · p. 12
  • The robot may unlatch the door but fail to walk forward continuously, leaving it short of the required 45° opening angle.The figure caption also describes doors left unopen beyond 45° because of insufficient forward locomotion.Source: F. Failure Case Analysis · pp. 12–13, Figure A2 · p. 13
  • Instability during squatting, standing, or lifting can propagate from jittery feet or lost balance and cause the basket to fall from the shelf.The basket may be placed on the shelf edge before falling.Source: F. Failure Case Analysis · p. 13
  • At unseen positions, the robot can grasp away from the bread and miss it, revealing a boundary in spatial generalization.The figure caption also identifies estimation errors at unseen generalized positions as a cause of empty-air grasps.Source: F. Failure Case Analysis · p. 13, Figure A2 · p. 13

G. Generalization Setting Details

The evaluation compares the robot fine-tuning distribution with position, object, and background variations reflecting diversity in human data. These settings test generalization across spatial, semantic, and visual changes.

G. Generalization Setting Details

Related papers

Loading 2603.12260v2…