Source-linked AI summary
Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration
Xinghao Zhu, Zixi Liu, Shalin Jain, Chenran Li, Milad Noori, Michael Andres Lin, Huihua Zhao, John Welsh, Mrinal Verghese, Wei Liu, Tingwu Wang, Xingye Da, Zhengyi Luo, Vishal Kulkarni, Naema Bhatti, Yuke Zhu, Linxi Fan, Bowen Wen, Danfei Xu, Soha Pouya, Yan Chang
TL;DR
Human demonstrations offer dexterous robots useful skills, but transferring them is difficult because existing approaches rely on brittle transfer assumptions or task- and object-specific aligned data. CHORD guides reinforcement learning by comparing human and robot contacts in object-centric wrench space; it reports strong benchmark and whole-body manipulation results and real-world transfer.
Problem
Demonstration-based optimization can rely on brittle transfer assumptions, while representation-learning approaches typically require aligned human-robot data for each task and object, limiting scalability beyond curated settings.
Source: §1 · p. 2Method
CHORD trains policies to match demonstrated object-state evolution using object-centric contact-wrench-space rewards that compare contacts by the motions they can induce.
Source: §1 · p. 2Results
82.12% average success across 1,831 benchmark tasks; CHORD also achieves 90.77% success in whole-body manipulation tasks and transfers to real-world open- and closed-loop settings.
Source: §1 · p. 2, §5 · p. 11Takeaways & Limitations
The formulation also adapts to whole-body manipulation from hand-only or whole-body demonstrations, with policies transferring to a real bimanual dexterous robot.
Source: §1 · p. 2, §5 · p. 11Real-world deployment currently relies on state-based observations.
Source: §6 · p. 11
Abstract
from arXiv · showhide
Dexterous robot manipulation can benefit from the abundance of human demonstrations, but transferring such demonstrations to robot policies remains challenging. We present Contact Wrench Guidance from Human Demonstration in Robotic Dexterous Manipulation (CHORD), a framework for long-horizon manipulation of rigid and articulated objects with reinforcement learning. The key idea is object-centric contact wrench space guidance: we represent human and robot motions by the forces and torques they can induce on the object, enabling similarity to be measured by the induced instantaneous motions. This guidance makes reinforcement learning more scalable for contact-rich dexterous manipulation. We further introduce a large-scale simulation benchmark with 4,739 bimanual dexterous manipulation tasks, constructed from motion-capture datasets and reconstructed in-house videos. Evaluated on 1,831 benchmark tasks, CHORD achieves an average success rate of 82.12%, demonstrating strong scalability. CHORD also generalizes to whole-body manipulation from hand-only and third-person demonstrations, achieving a 90.77% success rate, and the learned policies transfer to the real world in both open-loop and closed-loop settings.
1. Introduction
CHORD addresses the challenge of transferring human demonstrations to dexterous robots by comparing contacts through their object-level mechanical effects. It combines contact-wrench guidance with reinforcement learning and reports results across a large task benchmark and whole-body manipulation.
1. Introduction
- Prior demonstration-based optimization often depends on brittle transfer assumptions, while representation-learning methods typically need aligned human-robot data for each task and object.These constraints motivate seeking transferable and scalable task knowledge from human demonstrations.Source: §1 · p. 2
- Human hand motions cannot simply be replayed by robots because differences in morphology, kinematics, and hand geometry require different motions and contacts for the same object effect.Object-centric wrench space compares contacts by the motions they can induce, despite differences in contact locations, counts, and hand morphology.Source: §1 · p. 2
- CHORD uses contact-wrench-space rewards to guide reinforcement learning toward contacts that produce object-state evolution aligned with a human demonstration.The framework targets long-horizon, contact-rich tasks without task-specific robot demonstrations.Source: §1 · p. 2
- 82.12% average success across 1,831 tasks, with 90.77% success in whole-body manipulation tasks, and successful real-world deployment in open- and closed-loop settings.The benchmark comprises 4,739 tasks spanning rigid, articulated, and multi-object interactions.Source: §1 · p. 2
2. Related Work
Prior dexterous manipulation methods use task-level imitation, contact guidance, or auxiliary control to address difficult exploration. CHORD instead compares contacts through their induced object motion, including transient manipulation phases and differences in embodiment.
2. Related Work
- Prior simulation methods use object- and hand-reference tracking, while ManipTrans [2] rewards forces near demonstrated interactions and DexMachina and SPIDER [3] use contact locations with curricula such as virtual object control.Virtual object control applies an auxiliary wrench along the reference trajectory, providing a denser signal before successful contact.Source: §2 · p. 2
- Virtual object control can ease exploration but contact-location rewards can lead to local optima because location alone does not determine the resulting object motion.The same region can produce different motions depending on surface normal and force direction.Source: §2 · pp. 2–3
- CHORD compares human and robot contacts by their induced object-level mechanical effects, accommodating different contact locations, counts, and hand morphologies.Unlike prior wrench rewards aimed at static stability, this guidance supports transient non-force-closure phases such as pushing, levering, and sliding.Source: §2 · p. 3
3. Method
CHORD uses reinforcement learning to learn long-horizon manipulation from human references, combining object-state tracking with contact-wrench guidance. The method adds robustness mechanisms, extends to whole-body references, and introduces a 4,739-task bimanual simulation benchmark.
3. Method
- CHORD trains an RL policy to match demonstrated object-state evolution across rigid, articulated, and multi-object manipulation tasks, using retargeted human motion as a prior.The policy predicts actions from robot observations; retargeted motion is used in a residual action space.Source: §3 · p. 3, §3.2 · p. 5
3.1. RL with Wrench Space Contact Guidance
- The wrench reward compares human and robot force–torque support geometry per object part within relative tolerance bounds, alongside task-tracking and motion-imitation rewards.The support comparison guides contacts by their motion-inducing capability rather than matching wrench-matrix columns directly.Source: §3.1 · pp. 4–5
3.2. Efficient, Robust, and Generalizable Policy Learning
- Training uses reference-state resets with temporary VOC stabilization and demonstrated-wrench disturbances; noisy reconstructed contacts instead use a reduced force-closure objective.For whole-body references with inaccurate finger reconstruction, CHORD uses the reduced objective; hand-only references use an inpainting module to predict full-body motion.Source: §3.2 · p. 5
3.3. Dexterous Manipulation Benchmark with Human References
- The benchmark converts 4,739 open-source tasks and in-house video reconstructions into simulatable environments; its library includes more tasks, longer horizons, and denser contact events than prior works.It spans bimanual manipulation of single and multiple rigid objects, as well as articulated objects.Source: §3.3 · pp. 5–6
4. Experiments
Experiments evaluate CHORD at scale, examine contact-wrench guidance and its relation to success, and test long-horizon, whole-body, real-world, and teleoperation settings.
4.1. Quantitative Analysis
- Across 1,831 tasks, CHORD shows consistently strong performance across object types and horizons; stable training requires wrench-reward convergence before VOC decay.A rollout succeeds without object-centric termination, and a task is counted as successful when its completion ratio exceeds 0.7.Source: §4.1 · p. 6, Figure 5 · p. 7
- CHORD outperforms prior methods across all reported metrics and covers rigid-object, articulated-object, and multi-object categories.Table 1 compares prior methods across 60 tasks with 3 seeds.Source: Table 1 · p. 7, §4.1 · p. 7
4.2. Examining Contact Wrench Guidance
- Performance improves with richer contact guidance: CHORD performs best, position-only guidance degrades, and no-contact rewards perform weakest overall.The variants share non-contact rewards and differ only in contact supervision.Source: Table 2 · p. 8, §4.2 · p. 8
4.3. Correlation between Contact Wrench Reward and Task Success
- Pearson r≈0.80 overall and r=0.76–0.89 within every dataset; the relationship is monotonic, saturating, and supports CWS reward as a success correlate.Source: Figure 6 · p. 8, §4.3 · pp. 8–9
4.4. Diverse Capabilities Enable Long-Horizon Manipulation
- ADD-AUC ≈0.85−0.98 on 40−48-second sequences, while baseline methods degrade as interaction horizons increase.The figure reports per-sequence object-tracking performance across sequence sets and methods.Source: Figure 7 · p. 9, §4.4 · p. 9
4.5. CHORD Extends to Whole-Body Manipulation
- CHORD transfers hand-only references to G1/Dex3 under whole-body constraints; for TPV tasks, reconstruction noise makes reference wrench unreliable, so force-closure is used.The force-closure objective achieves consistent success across TPV tasks; an ablation finds contact-position rewards suffer in the five-finger-human to three-finger-G1 transfer, unlike wrench alignment.Source: §4.5 · pp. 9–10, Figure 8 · p. 10
4.6. Evaluation on Real World
- On real hardware, CHORD succeeds across rigid- and articulated-object tasks, though open-loop failures arise from initial placement and contact-transition sim-to-real divergence.For box pick-place, open-loop achieves 6/7 and closed-loop 7/7; state feedback tracks a fixed reference without reactive replanning.Source: §4.6 · p. 10, Figure 9 · p. 11
4.7. Comparing CHORD to Teleoperation
- Teleoperation often produced fragile mixer and waffle-iron grasps despite approximate object trajectories; visual-only contact inference, inverse-kinematics limits, and no joint-torque control constrained operators.Source: §4.7 · p. 11
5. Conclusion and Discussion
CHORD uses contact wrench-space guidance to learn dexterous, contact-rich manipulation from human demonstrations. The paper reports benchmark and whole-body results and real-world transfer.
5. Conclusion and Discussion
- CHORD achieves an 82.12% success rate on 1,831 evaluated tasks from a 4,739-task dexterous manipulation benchmark.The benchmark spans long-horizon, bimanual, rigid-object, and articulated-object manipulation.Source: §5 · p. 11
- CHORD achieves a 90.77% success rate in whole-body manipulation using either hand-only or whole-body human demonstrations.Source: §5 · p. 11
- Policies trained with CHORD transfer to the real world in both open-loop and closed-loop settings.Source: §5 · p. 11
6. Limitations
CHORD's limitations concern deployment observations, demonstration quality, and the suitability of object-pose error as an evaluation metric. The appendix supplies additional implementation, benchmark, and experiment details.
6. Limitations
- Real-world deployment currently relies on state-based observations, and effective contact guidance requires relatively clean human demonstrations.Noisy contacts can degrade reward quality and require a force-closure assumption.Source: §6 · p. 11
- Object-pose error can misrepresent task success: exact placement may be unnecessary, while small pose error can accompany functional failure.Source: §6 · p. 11
- The authors identify vision-based deployment, robustness to noisier demonstrations, and more task-aware success metrics as future work.Source: §6 · p. 11
Appendix
- The appendix provides additional details on noise ablations, reinforcement-learning implementation, video reconstruction, benchmark statistics, whole-body control, hardware experiments, and baselines.Source: Appendix · p. 16
A.1. Robustness on Human Demonstration Noise
The ablation examines how demonstration noise affects CHORD, comparing hand-pose noise, noisy contact guidance, and force-closure guidance. Moderate hand-pose noise is tolerated better than corrupted contact guidance.
A.1. Robustness on Human Demonstration Noise
- The ablation injects noise into demonstration data to test sensitivity to imperfections common in reconstructed demonstrations.The study defines settings with hand-pose noise alone, hand-pose plus contact noise, and noisy hand/contact data with force-closure reward.Source: §A.1 · p. 16
- The three settings distinguish noise in hand pose from noise in contact positions and surface normals, and compare contact-wrench with force-closure supervision.Source: §A.1 · p. 16
- Moderate hand-pose noise causes gradual performance degradation, whereas adding contact-guidance noise decreases performance more noticeably.Figure 10 compares task performance as injected noise increases.Source: §A.1 · p. 17, Figure 10 · p. 17
- At very high noise levels, force-closure guidance can outperform noisy wrench-space guidance, but it reduces dexterity by prioritizing stable grasps.Source: §A.1 · p. 17
B. Implementation Details of Reinforcement Learning
The implementation combines proprioceptive, object, command, action-history, and contact observations with residual hand actions and internal object-control terms. It also uses relative-pose rewards for two-object tasks and specifies episode resets and training details.
B. Implementation Details of Reinforcement Learning
- Each policy observation concatenates proprioceptive state, object state, task commands, action history, and contact information.Contact positions and force directions are transformed into the wrist frame, with inactive contacts zeroed.Source: B. Implementation Details of… · p. 17
- Observation corruption is disabled in large-scale experiments but enabled when training real-world closed-loop policies.Source: B. Implementation Details of… · p. 17
- Each hand uses residual wrist-position, wrist-orientation, and finger-joint actions, smoothed by an exponential moving average before controller execution.Position residuals are scaled to 0.05 m; wrist and finger residuals are scaled to 0.15 rad, with smoothing factor 0.3.Source: B. Implementation Details of… · p. 17
- Object-control terms have no policy action dimensions; they apply curriculum-scaled PD wrenches or efforts internally.Source: B. Implementation Details of… · p. 18
- Episodes end at the motion horizon or when wrist or object deviations exceed their allowed thresholds.Resets sample a random motion frame, restore object and wrist states, initialize fingers partially open, and apply a 20-step warm-up.Source: B. Implementation Details of… · p. 18
- Two-object tasks receive a relative object-pose reward that tracks demonstrated translation and rotation between objects, gated to demonstrated interaction phases.The gate uses demonstrated inter-object distance rather than current policy distance; the multiplicative reward requires both relative translation and orientation to match.Source: B. Implementation Details of… · p. 18
- CHORD trains per-sequence policies with FlashSAC using 2,048 environments and one L40S GPU for approximately two hours per task.The authors characterize this as recipe scalability across heterogeneous tasks, not cross-task policy generalization; no per-task hyperparameter tuning is used.Source: B. Implementation Details of… · p. 18
C. CHORD Validation from Human Video
The section validates CHORD using a brush demonstration reconstructed from monocular video and transferred to robot simulation. The full pipeline yields a reported grounding-quality score of 0.647 AUC.
C. CHORD Validation from Human Video
- The brush demonstration is reconstructed from a single monocular RGB video and an object-naming text prompt, and its trajectory is passed to CHORD for retargeting and simulation grounding.The reconstruction provides per-frame MANO hand poses and a textured object mesh with 6-DoF poses.Source: C. CHORD Validation from Human Video · p. 18
- The reconstruction pipeline uses monocular depth to align hand and object streams, then jointly refines and temporally smooths their outputs with Gaussian splats.The object mesh and 6-DoF pose are recovered alongside per-frame MANO hand fits, with both streams placed in a shared, roughly metric space.Source: C. CHORD Validation from Human Video · pp. 18–19
- CHORD transfers the reconstructed brush interaction to simulation despite substantial human–robot embodiment differences, producing physically plausible manipulation behavior.Figure 12 pairs representative human demonstration frames with the corresponding grounded robot execution.Source: Figure 12 · p. 19, C. CHORD Validation from Human Video · p. 20
- 0.647 AUC is reported for grounding quality, validating the pipeline from noisy human-video observations to executable robot trajectories.The authors describe this as validating reconstruction, contact extraction, contact-aware optimization, and embodiment grounding.Source: C. CHORD Validation from Human Video · p. 20
D. Dexterous Manipulation Benchmark Details
This section details how human manipulation sequences are converted, retargeted, and checked for the benchmark, then describes a whole-body extension and its hardware setup. The extension plans whole-body motion from hand-only references and applies CHORD to whole-body trajectories.
D. Dexterous Manipulation Benchmark Details
- The task library unifies hand and object trajectories from seven motion-capture datasets; HOT3D segmentation expands roughly 294 recordings into approximately 4,045 atomic clips.For HOT3D, contact is detected below 2 cm fingertip-to-object distance, brief breaks up to 15 frames are bridged, and clips are split by dominant contacted object.Source: D. Dexterous Manipulation Benchmark… · p. 20
- Retargeting uses differential inverse kinematics to track wrist pose and fingertip residuals while enforcing robot joint-position and joint-velocity limits.Sequences are rejected for hand-object or hand-hand penetration above 2 cm in any frame, with a concavity exception for the hand-object check; accepted trajectories are replayed for 300 simulator steps.Source: D. Dexterous Manipulation Benchmark… · p. 20
- The benchmark characterizes human contact behavior using contact-event counts and Ferrari–Canny epsilon computed from a precomputed contact wrench matrix.Positive epsilon certifies force closure and measures disturbance resistance.Source: D. Dexterous Manipulation Benchmark… · p. 20
E. Extending CHORD to Whole Body Manipulation
- The whole-body extension uses a complete or synthetically generated whole-body reference trajectory and learns a residual following the CHORD recipe.The section covers generating a whole-body reference from hand-only data and applying CHORD to that reference.Source: E. Extending CHORD to Whole Body… · p. 21
E.1. Planning Whole Body Motion from End Effector Trajectories
- For hand-only references, an inpainting module generates whole-body motion from retargeted end-effector trajectories, using wrist constraints at every frame and training on retargeted BONES Seed data.The planner autoregressively generates full-body sequences in chunks using four context frames.Source: §E.1 · pp. 21–22
E.2. Applying CHORD
- Given a whole-body reference, CHORD learns a residual over a pretrained whole-body controller to align robot contact wrench with the demonstrated wrench and induce the desired object trajectory.The policy controls residual joint-position actions on top of the controller’s reference-tracking output.Source: §E.2 · p. 22
F. Hardware Setup
- Hardware deployment uses a 61-dimensional commanded joint vector, with policy inference at 20 Hz and a safety state that freezes joints when Vicon tracking is stale or drifts.The platform uses two Sharpa dexterous hands and a six-camera Vicon system for object and robot pose measurements.Source: F. Hardware Setup · p. 23
G. Baseline Reproduction Implementation Details
The authors describe porting DexMachina, ManipTrans, and Spider to Sharpa hands for baseline comparisons. Their reproduction details include embodiment-specific processing and task-set evaluation choices.
G. Baseline Reproduction Implementation Details
- For Table 1, DexMachina, ManipTrans, and Spider were extended with Sharpa hands using their original code bases.The authors note that DexMachina and ManipTrans claim embodiment-agnostic support for arbitrary robot embodiments.Source: G. Baseline Reproduction… · p. 24
G.1. DexMachina
- DexMachina’s Sharpa port added wrist degrees of freedom, MANO retargeting configurations, collision groups, and controller tuning before visual simulation checks.Four training variants were run with five seeds each, and the reported variant used tuned DexMachina gains with a Hybrid Action space.Source: §G.1 · p. 24
G.2. Maniptrans
- On 77 OakInk-v2 validation tasks, ManipTrans reproduction achieved 64.1% single-hand and 29.7% bimanual success.Sharpa porting required separate left- and right-hand calibration of the hand-to-MANO alignment rotation.Source: §G.2 · p. 24
G.3. Spider
- For Spider, several inverse-kinematics alternatives were tuned, but none produced consistently satisfactory motions, so generated motions were manually inspected before evaluation.The selected tasks were run with five random seeds each.Source: §G.3 · p. 24