PaperlayerView exact revision on arXiv

Source-linked AI summary

DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video

Priyanka Mandikal, Kristen Grauman

arXiv:2202.00164v1cs.ROcs.CV

TL;DR

Dexterous grasp learning must cope with formidable action spaces and unfamiliar objects, while conventional human demonstrations often depend on specialized setups. DexVIP learns from human hand poses in in-the-wild video, outperforming demonstration- and affordance-based methods while training 20% more efficiently.

  • Problem

    Dexterous hands’ 24–30-DoF action spaces and unfamiliar object shapes challenge learning, while conventional human demonstrations often require specialized setups.

    Source: §1 · p. 1

  • Method

    DexVIP extracts human hand poses from in-the-wild video and uses a hand-pose prior to shape deep reinforcement learning for grasping.

    Source: §1 · p. 2

  • Results

    20% more efficient to train, DexVIP outperformed state-of-the-art learning-from-demonstration and visual-affordance methods [2] [18].

    Source: §1 · p. 2

  • Takeaways & Limitations

    In-the-wild video provides expert information that scales well to new objects without conventional demonstration tools.

    Source: §5 · p. 8

    Some object orientations do not suit the target hand pose, constraining applicability to post-grasp functional tasks.

    Source: §4 · p. 8

Abstract

from arXiv · show

Dexterous multi-fingered robotic hands have a formidable action space, yet their morphological similarity to the human hand holds immense potential to accelerate robot learning. We propose DexVIP, an approach to learn dexterous robotic grasping from human-object interactions present in in-the-wild YouTube videos. We do this by curating grasp images from human-object interaction videos and imposing a prior over the agent's hand pose when learning to grasp with deep reinforcement learning. A key advantage of our method is that the learned policy is able to leverage free-form in-the-wild visual data. As a result, it can easily scale to new objects, and it sidesteps the standard practice of collecting human demonstrations in a lab -- a much more expensive and indirect way to capture human expertise. Through experiments on 27 objects with a 30-DoF simulated robot hand, we demonstrate that DexVIP compares favorably to existing approaches that lack a hand pose prior or rely on specialized tele-operation equipment to obtain human demonstrations, while also being faster to train. Project page: https://vision.cs.utexas.edu/projects/dexvip-dexterous-grasp-pose-prior

1 Introduction

Dexterous hands pose difficult learning and generalization challenges, while conventional human demonstrations require specialized setups and indirect proxies. DexVIP learns from hand poses in everyday videos, outperforming existing methods on simulated grasping while training more efficiently.

1 Introduction

  • Dexterous hands’ 24–30 DoF make RL exploration formidable, while unfamiliar object shapes demand generalization.Many exploratory hand-pose trajectories yield no reward, motivating the use of imitation learning to reduce sample complexity.Source: §1 · p. 1
  • Conventional human demonstrations require specialized setups and resources, repeated overhead for new objects, and proxy interactions removed from real object use.Examples include VR, motion capture, and kinesthetic guidance of the robot.Source: §1 · pp. 1–2
  • DexVIP extracts 3D human hand poses from in-the-wild video and rewards human-like poses at object contact alongside grasp success and affordance regions.The pose prior comes from people using objects in real-world activities rather than specialized demonstrations.Source: §1 · p. 2
  • Direct real-world interactions remove proxy-demonstration indirection and promote functional grasps; new object examples can be added by downloading video.The 3D hand-pose model also tolerates viewpoint variation and complex visual surroundings.Source: §1 · p. 2
  • Across 27 objects, the 30-DoF simulated-hand policy outperformed state-of-the-art learning-from-demonstration and visual-affordance methods [2] [18] and trained 20% more efficiently.The learned behavior also resembled natural human-object interactions.Source: §1 · p. 2

2 Related Work

Prior grasping research spans planning, reinforcement learning, imitation, visual affordances, and hand-pose estimation. DexVIP differs by using human hand-pose priors from in-the-wild video for functional dexterous grasping.

2 Related Work

  • Prior grasping work includes shape-based or model-based planning and closed-loop RL [8] [26] [27], but few studies explore RL for dexterous manipulation [5] [18].Many planning methods target simpler grippers and need not produce functional grasps.Source: §2 · p. 2
  • Unlike traditional learning from demonstration, DexVIP uses a visual prior for good hand states rather than full state-action trajectories, sourced from in-the-wild video.The cited prior demonstration methods include work in non-dexterous and dexterous settings [29] [30] [31] [32] [33] [15] [2] [4].Source: §2 · p. 3
  • Earlier visual-imitation methods relax state-sequence requirements, but often need paired robot-person data, shared visual environments, or object-specific setups [33] [15] [29] [32] [30] [31].The passage identifies in-the-wild video for dexterous grasping as the distinction of this work.Source: §2 · p. 3
  • Object-centric affordances are complementary to DexVIP’s hand-pose prior and can also influence grasping with dexterous hands [8] [18].The cited affordance work explores visual interaction regions and their use in grasping [36] [37] [38] [9] [24].Source: §2 · p. 3
  • DexVIP uses FrankMocap [44] to extract 3D hand poses from video; its contribution is the grasp-learning framework, not a new pose-parsing method.The passage describes FrankMocap as a state-of-the-art pose estimator.Source: §2 · p. 3

3 Approach

DexVIP formulates dexterous grasping as reinforcement learning and adds object-specific target hand poses inferred from human-object interaction videos as a learning signal. Its approach combines a 30-DoF robot hand, a video-derived pose prior, and rewards for grasping and pose matching.

3.1 Reinforcement Learning Framework for Dexterous Grasping

  • The 30-DoF Adroit hand learns a grasping policy from visual and motor observations through an actor-critic network, with experiments conducted in simulation.The visual and motor streams inform actions, while the hand-pose prior contributes an auxiliary reward.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Object-Specific Human Hand Pose Priors from Video

  • The method curates 715 HowTo100M grasp frames for 27 household objects, averaging 26 grasp images per object.The images depict objects in canonical upright use.Source: §3.2 · p. 5
  • FrankMocap estimates 3D hand poses from the frames, retaining right-hand detections and leaving bimanual grasps for future work.Alternative pose-estimation methods could also be used.Source: §3.2 · p. 5
  • For each object, k-medoid clustering selects the largest cluster's medoid as the consensus target, accounting for categories with multiple grasp styles.The video poses are often consistent when people use an object in its standard functional manner.Source: §3.2 · pp. 5–6
  • PPO training rewards successful lifts, affordance-region contact, pose matching, and action-distribution entropy; pose matching applies after 30% of touch sensors activate.The pose error ignores arm azimuth and elevation so the robot can orient its arm while matching the hand pose.Source: §3.1 · p. 4, §3.2 · p. 6
  • The video prior enables imitation without human state-action trajectories, and new-object priors can be added by obtaining new training images.The method transfers human poses to the robot using inverse kinematics before using them as grasp targets.Source: §3.2 · pp. 5–6

4 Experiments

Experiments evaluate DEXVIP, trained on 27 objects, against demonstration- and affordance-based methods across grasp quality, training speed, data scaling, reward ablations, and noisy conditions. DEXVIP performs strongly across these tests, including faster training and human-like grasp postures.

Grasping policy performance

  • On ContactDB, DEXVIP outperforms the compared methods on grasp metrics, with functionality similar to GRAFF and stronger posture results.The policy was trained on 27 objects; the comparison includes DAPG, which uses VR demonstrations, and GRAFF, which targets object affordances.Source: §4 · pp. 6–7, Figure 5 · p. 7
  • Example grasps show DEXVIP using natural human-like postures, while COM and GRAFF often produce unusual poses that can reduce post-grasp functionality.A failure case occurs when an object's orientation does not suit the target hand pose, such as a knife whose handle is on the left.Source: §4 · pp. 7–8, Figure 4 · p. 7

Training speed

  • 20% faster than GRAFF, DEXVIP learns successful grasping policies, and the authors attribute the gain to pose priors that improve sample efficiency.Figure 5 compares training curves; the caption also reports stable relative results under noisy sensing and actuation.Source: Figure 5 · p. 7, §4 · p. 8

Expert data: teleoperation vs. YouTube video

  • For 11 non-ContactDB objects, DEXVIP success falls from 68% to 65%, while DAPG falls from 59% to 50%.Video or image curation takes seconds, compared with five minutes to collect one DAPG demonstration on average; Figure 6 plots success against accumulated demonstration experience.Source: §4 · p. 8, Figure 6 · p. 8

Ablations

  • On ContactDB, adding touch to Raff raises success from 60% to 63%, and adding hand-pose priors raises it to 68%.The full model is reported as the most effective reward configuration.Source: §4 · p. 8

Noisy sensing and actuation

  • Under heavy simulated noise, DEXVIP achieves 64% grasp success, exceeding noise-free models of the other methods.The noise includes perturbations to sensing and actuation; the authors describe this result as support for potential real-world transfer, not a real-robot evaluation.Source: §4 · p. 8

5 Conclusion

DexVIP learns dexterous grasping from video-derived human hand-pose priors and outperforms methods without those priors. The authors identify direct observation and scalability to new objects as advantages, and propose extending the approach beyond grasping.

5 Conclusion

  • DexVIP uses human hand-pose priors from video to learn dexterous robotic grasping, outperforming methods without those priors.The authors describe this as an encouraging step toward learning manipulation from weakly supervised, scalable in-the-wild data.Source: §5 · p. 8
  • The approach observes people interacting directly with real objects, without conventional demonstration tools.The authors contrast this with demonstrations collected through tools that mediate human interaction with the robot.Source: §5 · p. 8
  • Video-derived expert information scales well to new objects; future work would extend the approach to fine-grained manipulation skills beyond grasping.Source: §5 · p. 8

A Noisy sensing and actuation

DexVIP retains high grasp success under substantial simulated sensing and actuation noise, while the authors note they could not test on a real robot. The reward details describe how grasp success, affordance proximity, pose matching, and exploration shape learning.

A Noisy sensing and actuation

  • The noise tests perturb joint angles and velocities, actuator commands, image pixels, and tracked object points, including randomly freezing tracking for 20 frames.Proprioceptive noise is Gaussian with standard deviation 0.01; tracking-point noise has standard deviation 1 cm.Source: A Noisy sensing and actuation · p. 12
  • Under substantial noise, DexVIP achieves high grasp success comparable to its noise-free performance and exceeds other methods’ noise-free models.The experiments are in simulation; the authors use noise induction to test compliance under realistic conditions.Source: A Noisy sensing and actuation · p. 12

B Reward function details

  • The success reward is positive when the hand contacts the object while the object is off the table; the affordance reward penalizes Chamfer distance from affordance regions.The affordance penalty falls as the hand approaches those regions, encouraging exploration of useful object areas.Source: B Reward function details · pp. 12–13
  • The pose reward penalizes per-joint angle error with greater weight on parent joints, encouraging pose alignment from parent joints to children.It is applied only after 30% of the robot’s touch sensors activate, encouraging the target pose when the hand is near and in contact with the object.Source: B Reward function details · p. 13
  • PPO training also maximizes entropy over the target action distribution to encourage exploration.Source: B Reward function details · p. 13

C TSNE on hand poses

Human and robot hand poses distribute across object morphologies, and the trained agent’s poses cluster more tightly because training uses a category-level target pose.

C TSNE on hand poses

  • Human and robot poses show morphology-linked distributions, while the trained agent’s successful-grasp poses are more clustered around category-level target poses.Examples include clenched-fist poses for mugs, pans, and teapots, versus loose-fist poses for apples and mice.Source: C TSNE on hand poses · p. 13

D Hand pose retargeting from FrankMocap to Adroit

The retargeting pipeline maps FrankMocap’s human hand poses into Adroit’s different joint structure, producing approximate poses that the authors find useful for training. The supplement also reports robustness to substantial changes in object mass and scale.

D Hand pose retargeting from FrankMocap to Adroit

  • Human and robot hand poses show morphology-linked distributions, including clenched fists for mugs, pans, and teapots and loose fists for apples and mice.Source: D Hand pose retargeting from… · p. 14
  • The FrankMocap-to-Adroit mapping is approximate because their kinematic chains differ, but produces poses that closely match human poses and work well for training.FrankMocap represents the hand with 15 three-DoF ball joints, while Adroit has 24 one-DoF revolute joints and a six-DoF arm.Source: D Hand pose retargeting from… · pp. 13–14
  • The retargeting pipeline converts 3D hand joints from world to wrist-relative coordinates, derives Adroit arm orientation from the palm plane, then maps parent-relative angles to Adroit joints.The palm plane is defined using the wrist and forefinger and ring-finger knuckles.Source: D Hand pose retargeting from… · pp. 14–15

E.1 Parameter settings

  • DexVIP remains fairly robust across large variations in object mass and scale.Source: Figure C · p. 16

E.2 Robustness to parameters

DexVIP remains fairly robust across tested object-mass and scale variations, while grasping lighter objects is easier. The hand-affordance distance uses tracked affordance points and sampled points on the hand.

E.2 Robustness to parameters

  • DexVIP remains fairly robust when object mass varies from 0.5kg to 1.5kg and scale from 0.8x to 1.2x of training size; lighter objects are easier to grasp.The evaluation tests robustness to these variations in physical properties.Source: §E.2 · p. 16

F Hand-Affordance distance

  • The hand-affordance distance tracks 20 sampled 3D affordance points and compares them with 10 sampled points on the palm and fingers.Noise experiments also induce affordance-point tracking failure.Source: F Hand-Affordance distance · p. 16

G Additional results

Policies trained with inferred hand poses perform comparably to policies trained with ground-truth poses. Clustering filters pose-prediction outliers to obtain representative poses for each object.

G.1 Effect of pose prediction on performance

  • Policies trained with inferred hand poses perform comparably to those trained with ground-truth poses, indicating robustness to pose-prediction errors.Ground-truth ContactPose poses captured using mocap provide an upper bound for grasp performance.Source: §G.1 · p. 16
  • Hand-pose clustering filters bad or outlier predictions to obtain a representative hand pose for each object.The outliers arise from pose-prediction errors.Source: §G.1 · p. 16

G.2 Effect of object shape variation on performance

DexVIP handles shape differences between video objects and simulator meshes, and maintains performance on additional object classes absent from ContactDB. Inferred-pose supervision also performs comparably to ground-truth supervision.

G.1 Effect of pose prediction on performance

  • Policies trained with inferred hand poses perform comparably to those trained with ground-truth poses, indicating robustness to pose errors.Ground-truth poses captured using mocap provide an upper bound for grasp performance.Source: §G.2 · p. 17

G.2 Effect of object shape variation on performance

  • DexVIP grasps objects fairly robustly despite shape differences between video objects and simulator meshes, including teapots, flashlights, and saucepans without exact simulator matches.The simulator uses a generic mesh from the same object category rather than requiring an exact instance match.Source: §G.2 · pp. 16–17

G.3 Ablation evaluation

  • The full DexVIP policy leverages a touch-informed pose prior to perform well across all reported metrics.The cited table is described as reporting metrics for DexVIP ablations.Source: Table C · p. 18

G.4 Performance on additional objects

  • On additional non-ContactDB objects, compared with performance on ContactDB, DexVIP experiences only a 4% drop, while DAPG and GRAFF drop by 15% and 12%, respectively.Source: §G.2 · p. 18

G.3 Ablation evaluation

On additional object classes absent from ContactDB, DexVIP has a smaller success-rate drop than DAPG and GRAFF. The full model improves success, stability, and posture over ablations while maintaining affordance-only functionality.

G.3 Ablation evaluation

  • The full DexVIP model gains substantially in success, stability, and posture metrics while maintaining the functionality score of the affordance-only policy.The evaluation reports metrics for DexVIP ablations.Source: §G.3 · p. 18

G.4 Performance on additional objects

  • On 11 non-ContactDB object classes, DexVIP’s success rate drops 4% from 68% to 65%, compared with DAPG’s 15% and GRAFF’s 12% drops.The results indicate that DexVIP can leverage hand poses for a variety of objects.Source: §G.4 · p. 18

Related papers

Loading 2202.00164v1…