PaperlayerView exact revision on arXiv

Source-linked AI summary

Human2Dex: Dexterous Robot Learning from Wrist-Centric Human Demonstrations

Jiachen Zhang, Yuanyuan Li, Shaqi Luo

arXiv:2610.13119v1cs.RO

TL;DR

Collecting dexterous manipulation demonstrations for each robot hand is costly, motivating a reusable human-data interface across embodiments. Human2Dex shares wrist-centric demonstrations and observation and motion semantics, then constructs hand-specific supervision; across six real-world tasks, it reports high success rates on both Linker O6 and Wuji compared with Human-Raw.

  • Problem

    Collecting dexterous manipulation demonstrations separately for each target hand is costly, and the paper asks whether one human source can provide reusable training data across hands.

    Source: §I · pp. 1–2

  • Method

    Human2Dex reuses wrist-centric human demonstrations, aligns observations with local appearance adaptation and a shared 21-point representation, and retargets motion into hand-specific supervision with GPIR interaction cues.

    Source: §I · p. 2, A. Shared Human Source and Conversion… · p. 3

  • Results

    Human2Dex achieves 109/120 successes on Linker O6 and 97/120 on Wuji; pooled success is 85.8% for Human2Dex versus 26.3% for Human-Raw.

    Source: C. Main System-Level Results · p. 6, §VI · p. 8

  • Takeaways & Limitations

    One human demonstration corpus supports independent policy learning on two calibrated dexterous hands while retaining hand-specific execution and training.

    Source: §V · p. 7

    Whole-object grasp-relative grounding is less direct for localized contacts such as a lid edge or drawer handle, motivating part-level or affordance-level representations for those tasks.

    Source: §V · p. 7

Abstract

from arXiv · show

Collecting dexterous manipulation demonstrations for each robot hand requires repeated task-specific data acquisition. We ask whether one shared corpus of natural human demonstrations can instead serve as a reusable data interface for different dexterous hands, without collecting task demonstrations on each target robot. We present Human2Dex, a wrist-centric human demonstration interface that separates shared human observation and motion semantics from embodiment-specific execution. It aligns observations through local hand-appearance adaptation and a shared 21-point hand representation with consistent structural semantics across human and robot observations. It constructs executable supervision by retargeting fused human hand motion to each target hand, and introduces the Grasp-Pocket Interaction Representation (GPIR) to express tracked objects relative to a hand-centered Functional Grasp Pocket. The corpus is converted into separate training sets for Linker O6 and Wuji, while action labels, normalization, training, and diffusion-policy checkpoints remain hand-specific. Across six real-world tasks, Human2Dex achieves 109/120 (90.8%) successes on Linker O6 and 97/120 (80.8%) on Wuji. Across both hands, Human-Raw achieves 63/240 (26.3%) successes, whereas the complete Human2Dex configuration achieves 206/240 (85.8%). These results support the use of wrist-centric human demonstrations as a reusable interface across calibrated dexterous hands while retaining embodiment-specific policy learning. The project website is publicly available at https://human2dex.pages.dev/.

I. INTRODUCTION

Human2Dex asks whether one wrist-centric human demonstration corpus can provide reusable training data for different dexterous hands without collecting task demonstrations on each robot. It separates shared human observation and motion semantics from target-specific execution and evaluates the interface on two hands across six tasks.

I. INTRODUCTION

  • The motivating gap is that collecting dexterous demonstrations separately for each target hand is costly and difficult to scale, while existing interfaces often rely on acquisition setups or mappings tied to robot compatibility.The question is whether the same human manipulation episodes can be converted into executable training data for different hands without target-robot task demonstrations.Source: §I · p. 1
  • Wrist-centric views resemble the eye-in-hand observations used for close-range dexterous control, but matching viewpoints does not resolve differences in appearance, structure, kinematics, commands, or grasp geometry.The introduction identifies observation alignment, target-specific executable supervision, and object–hand relations as linked correspondence challenges.Source: §I · pp. 1–2
  • Human2Dex reuses wrist-centric human demonstrations while separately adapting observations and motion semantics and constructing executable supervision for each target hand.It uses local hand-appearance adaptation, a shared 21-point representation, and retargeting to each calibrated hand.Source: §I · p. 2
  • Human2Dex adds GPIR, which expresses tracked objects relative to a hand-centered Functional Grasp Pocket to provide an explicit object–hand relation.The pocket is defined by the thumb–index grasp center.Source: §I · p. 2
  • The paper evaluates Linker O6 and Wuji on six real-world tasks, with a matched wrist-versus-head viewpoint comparison and additional reference-frame and task-wise analyses.The viewpoint comparison uses Linker O6–Croissant; the main evaluation covers both target hands.Source: §I · p. 2

A. Learning Robot Policies from Human Demonstrations

Prior interfaces establish human-to-robot compatibility in different ways, while Human2Dex shares human data semantics and performs embodiment adaptation after collection. Its conversion interface preserves shared observations while producing hand-specific training data and using a grasp-relative interaction cue.

A. Learning Robot Policies from Human Demonstrations

  • Human2Dex differs from robot-like, wearable, or sensing-equipped interfaces by retaining wrist-centric bare-hand demonstrations and adapting them after collection for each hand's kinematics and action space.Related approaches establish compatibility through collection hardware, shared end-effectors, or other sensing and mapping arrangements.Source: A. Learning Robot Policies from Human… · pp. 2–3

B. Visual and Kinematic Cross-Embodiment Alignment

  • A shared 21-point representation aligns observation semantics, while embodiment-specific retargeting, action labels, and policy checkpoints preserve each robot's own action space.The shared representation defines data semantics rather than a common robot action space.Source: B. Visual and Kinematic… · p. 3

C. Object-Centric and Functional Grasp Representations

  • GPIR expresses tracked objects relative to a hand-centered Functional Grasp Pocket and retains object scale and tracking reliability as a low-dimensional cue for sequential wrist-view policy control.It is not presented as general object reconstruction, a universal grasp representation, or a complete contact model.Source: C. Object-Centric and Functional Grasp… · p. 3

A. Shared Human Source and Conversion Interface

  • One wrist-centric human source is converted offline into separate training sets for target hands, with retargeting, action labels, normalization, and policy training remaining hand–task specific.Adding a target hand requires its kinematic model, retargeter, native action conversion, normalization, and policy training, but not recollecting task demonstrations.Source: A. Shared Human Source and Conversion… · p. 3, Figure 2 · p. 3

B. Hand-Structure-Aligned Observations

The observation branch adapts human-hand appearance and adds a shared 21-point structural description. Human and robot observations use consistent landmark identities and image-space conventions.

B. Hand-Structure-Aligned Observations

  • Local appearance randomization changes the human-hand image while leaving the object and background unchanged.The appearance-randomization operator is applied locally to the hand.Source: B. Hand-Structure-Aligned Observations · p. 3
  • The hand structure is rendered into the policy image with a fixed topology and color assignment that preserve landmark and finger identities across observations.The mapping supports a 224 × 224 policy image.Source: B. Hand-Structure-Aligned Observations · p. 3
  • At deployment, robot landmarks are generated by forward kinematics and camera projection, and human and robot images share the same landmark topology, colors, and image-space convention.Images, masks, and landmarks undergo the same crop, resize, and image-space perturbation.Source: B. Hand-Structure-Aligned Observations · pp. 3–4

C. Target-Specific Action Supervision

Human hand motion is fused and retargeted separately for each target hand to construct executable supervision. The resulting states are converted into the target hand's native command representation.

C. Target-Specific Action Supervision

  • Confidence-weighted PICO and wrist-RGB hand structures are fused in a shared wrist-centered MANO frame, then filtered to reduce jitter and bridge short tracking dropouts.The output is a fused 21-point structure used for retargeting.Source: C. Target-Specific Action Supervision · p. 4
  • Retargeting optimizes target-hand configurations using geometric relations between fused human landmarks and robot landmarks, rather than directly mapping human joint angles.The optimization enforces target-hand joint and coupling constraints.Source: C. Target-Specific Action Supervision · p. 4
  • Linker O6 and Wuji use different geometric objectives and target-hand constraints; Wuji's weighting shifts from whole-finger shape to fingertip position and direction near pinching.O6 optimizes six active joints with remaining joints mechanically coupled; Wuji solves a 20-DoF configuration.Source: C. Target-Specific Action Supervision · p. 4
  • The human wrist trajectory is mapped through a calibrated human-to-robot arm transform, and the optimized hand state is converted through a fixed embodiment-specific mapping to native target commands.The O6 mapping converts six optimized active-joint targets to native actuator commands.Source: C. Target-Specific Action Supervision · p. 4
  • During deployment, measured robot state is converted to the same target-hand state representation and normalized with the corresponding training-set statistics.The command dimensions are six for O6 and 20 for Wuji.Source: C. Target-Specific Action Supervision · p. 4

D. Grasp-Centered Interaction Representation

The Grasp-Pocket Interaction Representation (GPIR) describes tracked objects relative to a hand-centered grasp location. It combines relative position and scale with tracking reliability for policy conditioning.

D. Grasp-Centered Interaction Representation

  • For thumb–index grasps, fingertip tracks are smoothed with a causal three-frame median filter.The interaction branch uses the resulting grasp location to describe the object compactly.Source: D. Grasp-Centered Interaction… · p. 4
  • The Functional Grasp Pocket is the image-space midpoint between the thumb and index fingertips.For robot hands, fingertips are projected individually before computing the midpoint.Source: D. Grasp-Centered Interaction… · p. 4
  • Tracked object states include image centroid, mask area, confidence, and whether the state is observed or retained from memory.Short tracking failures use the last valid measurement with confidence decay and a memory-valid flag.Source: D. Grasp-Centered Interaction… · p. 4
  • GPIR represents whole-object alignment, scale, and tracking reliability relative to the hand-centered Functional Grasp Pocket.Multiple object states are concatenated for policy conditioning.Source: D. Grasp-Centered Interaction… · p. 4

E. Policy Learning

Each hand–task pair is trained independently using converted human observations, target-hand state, and optional interaction state. A diffusion policy predicts native action sequences, with an auxiliary object-state objective available during training.

E. Policy Learning

  • Each hand–task pair is trained independently, and the policy input combines converted observations, target-hand state, and interaction state when enabled.The target-hand state is always included; interaction state is optional.Source: E. Policy Learning · p. 4
  • A DINOv2 encoder combines image features with hand and optional interaction state to condition a 1-D U-Net that predicts a 16-step action sequence.The policy uses the Diffusion Policy objective.Source: E. Policy Learning · p. 4
  • The human-data interface is shared, but action construction, normalization, policy training, and checkpoints remain hand-specific.The policy predicts native commands for one hand–task pair.Source: Figure 3 · p. 5
  • Configurations with auxiliary supervision regress Pocket-relative object position and log-area using a masked Smooth-L1 loss.The auxiliary head is omitted at deployment, and its loss weight is zero when supervision is disabled.Source: E. Policy Learning · p. 5
  • At deployment, the policy runs at 16 Hz and executes the first eight actions of each predicted chunk before replanning.Training uses AdamW, a learning rate of 3 × 10−4, batch size 32, and 50 diffusion timesteps; inference uses 16 DDIM steps.Source: E. Policy Learning · p. 5

A. Research Questions

The research questions examine whether one human corpus supports policy learning across two dexterous hands, how observation alignment and fused hand supervision affect performance, and when grasp-centered grounding helps.

A. Research Questions

  • Can one human demonstration corpus support independent policy learning for two dexterous hands?The questions also examine the effects of observation alignment, fused human-hand supervision, and wrist- versus head-centric views.Source: A. Research Questions · p. 5
  • How do observation alignment and fused human-hand supervision affect performance, and how does wrist-centric observation compare with head-centric observation?These comparisons address the observation and motion-supervision components.Source: A. Research Questions · p. 5
  • When is grasp-centered interaction grounding useful, and how do reference coordinates and task geometry affect its performance?Source: A. Research Questions · p. 5

B. Experimental Setup

The evaluation uses a shared human corpus to train separate policies for Linker O6 and Wuji across six real-world tasks. Human2Dex achieves higher pooled success than the listed comparison configurations, while the pooled counts describe this benchmark rather than unseen-task populations.

B. Experimental Setup

  • The evaluation covers Linker O6 and Wuji, two anthropomorphic hands with different kinematic structures and calibrated wrist-mounted cameras.O6 has eleven DoFs, including six active DoFs; Wuji has twenty active DoFs.Source: B. Experimental Setup · p. 5
  • Both hands are evaluated on six real-world tasks, using 1,899 quality-controlled base episodes and 556,624 frames, with three appearance-rendered variants per episode.Configurations share base episodes and evaluation budgets, while O6 and Wuji use distinct actions, normalization, and checkpoints.Source: B. Experimental Setup · p. 5
  • Each method–task pair receives 20 physical trials, and success requires meeting its terminal criterion for two seconds.Wilson 95% intervals describe trial uncertainty for these fixed configurations; pooled counts are not estimates for unseen-task populations.Source: B. Experimental Setup · p. 5

C. Main System-Level Results

  • Human2Dex adds object tracking, Pocket-relative object input, and an auxiliary objective to the fused-supervision configuration.The system comparisons are separately trained configurations and do not isolate any component's causal effect.Source: B. Experimental Setup · p. 5, C. Main System-Level Results · p. 6
  • 85.8% pooled success for Human2Dex exceeds Human-Structured (67.9%), Human-Visual (60.0%), and Human-Raw (26.3%) across both hands.Human2Dex achieves 109/120 successes on O6 and 97/120 on Wuji.Source: C. Main System-Level Results · p. 6

D. Observation and Human-Motion Supervision Analyses

Wrist-view demonstrations outperform head-view demonstrations in the matched O6–Croissant diagnostic, while pooled success also rises across Raw, Visual, and Structured configurations. Pocket-relative object conditioning shows task-dependent gains, with the largest improvements on grasp-centric tasks.

D. Observation and Human-Motion Supervision Analyses

  • 26.3% pooled success for Human-Raw rises to 60.0% for Human-Visual and 67.9% for Human-Structured.All configurations use the same embodiment-specific retargeter; Structured replaces PICO-only hand structures with fused PICO/RGB supervision.Source: D. Observation and Human-Motion… · p. 6
  • In the matched head- versus wrist-view comparison on O6–Croissant (20 trials per condition), wrist-view demonstrations yielded 6/20 final successes (30%) and 4/20 first-attempt grasps (20%), versus 0/20 for both head-view measures.Source: Figure 5 · p. 6, D. Observation and Human-Motion… · p. 6

E. Interaction-Grounding Analysis

  • The Structured-to-Human2Dex transition adds object tracking, Pocket-relative object input, and an auxiliary prediction objective together.Because these components are added jointly, this comparison evaluates the integrated grounding configuration.Source: E. Interaction-Grounding Analysis · p. 6
  • Table I reports 20 trials per hand–task pair, with independently retargeted supervision and a separate policy; its final row uses task-specific object–pocket observations on all six tasks.Source: Table I · p. 7
  • 18/20 Human2Dex trials succeed on Croissant versus 17/20 for Human-Absolute; on Cup, the counts are 16/20 versus 9/20.The configurations differ only in referencing object position to the Functional Grasp Pocket rather than the image center.Source: E. Interaction-Grounding Analysis · p. 7
  • The complete interface improves most on Sponge, Croissant, Cup, and Bottle, with smaller or absent gains on Egg Carton and Open Drawer.The authors associate this task dependence with object-level grounding: whole-object grasping fits the pocket relation better than localized lid-edge or drawer-handle contact.Source: E. Interaction-Grounding Analysis · p. 7

V. DISCUSSION AND LIMITATIONS

Human2Dex reuses one human demonstration source across calibrated hands while keeping executable supervision and policy learning hand-specific. Its grasp-centered reference is most useful when object–hand alignment drives success, while whole-object grounding and the evaluation scope leave clear limits.

V. DISCUSSION AND LIMITATIONS

  • Human observations and motion representations are shared, while retargeting, executable action labels, normalization, and policy training remain specific to each hand–task pair.This separation avoids recollecting task demonstrations for each target hand while preserving each embodiment’s control structure.Source: §V · p. 7
  • Image-centered conditioning is competitive on Croissant, whereas Pocket-relative conditioning performs substantially better on Cup.The comparison suggests that grasp-centered referencing matters more when the object–hand relation directly determines task success.Source: §V · p. 7, Figure 6 · p. 7
  • Whole-object grounding improves performance most on Sponge, Croissant, Cup, and Bottle, but gains are smaller or absent on Egg Carton and Open Drawer.Localized lid-edge or handle interactions provide less direct cues for a whole-object representation, motivating part-level or affordance-level representations.Source: §V · p. 7
  • Evaluation covers two calibrated anthropomorphic hands and six tasks with separate policies; transfer to arbitrary morphologies and shared action spaces remains open.GPIR also assumes a specified task object and reliable image-space tracking, and does not explicitly identify localized contact regions.Source: §V · pp. 7–8
  • Without a robot-demonstration baseline, direct sample-efficiency comparisons between human and robot data remain future work.Source: §V · pp. 7–8

VI. CONCLUSION

  • Human2Dex achieves 109/120 successes on Linker O6 and 97/120 on Wuji across six real-world tasks.The result supports independent policy learning on both calibrated hands from one human demonstration source.Source: §VI · p. 8

Related papers

Loading 2610.13119v1…