PaperlayerView exact revision on arXiv

Source-linked AI summary

D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions

Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, Otmar Hilliges

arXiv:2112.03028v2cs.CVcs.LGcs.RO

TL;DR

Dynamic grasp synthesis asks how to generate physically plausible hand-object motions from a static grasp reference while moving an object to a target 6D pose. D-Grasp addresses this with hierarchical reinforcement learning in physics simulation, combining stable grasping and high-level motion synthesis. The method produces stable grasps and object motions, while correcting imperfect references, though its evaluation is constrained by difficult-to-grasp surface objects and simplified training meshes.

  • Problem

    Dynamic grasp synthesis requires generating smooth, stable hand-object trajectories from a single grasp reference while accounting for contacts, friction, collisions, and dynamics.

  • Method

    D-Grasp uses hierarchical reinforcement learning with physics simulation, separating a low-level grasping policy from high-level motion synthesis and using grasp-conditioned rewards.

  • Results

    D-Grasp learns stable, physically plausible grasps, generates motions to target poses, and improves performance over baselines across reported metrics, including 0 interpenetration loss in simulation.

  • Takeaways & Limitations

    The method can generate natural hand-object interaction sequences from static references and generalize to unseen objects when a static hand-pose reference is available.

  • Takeaways & Limitations

    The method has difficulty with objects that are difficult to grasp on a surface, and simplified training meshes limit interpenetration performance on original meshes.

Abstract

from arXiv · show

We introduce the dynamic grasp synthesis task: given an object with a known 6D pose and a grasp reference, our goal is to generate motions that move the object to a target 6D pose. This is challenging, because it requires reasoning about the complex articulation of the human hand and the intricate physical interaction with the object. We propose a novel method that frames this problem in the reinforcement learning framework and leverages a physics simulation, both to learn and to evaluate such dynamic interactions. A hierarchical approach decomposes the task into low-level grasping and high-level motion synthesis. It can be used to generate novel hand sequences that approach, grasp, and move an object to a desired location, while retaining human-likeness. We show that our approach leads to stable grasps and generates a wide range of motions. Furthermore, even imperfect labels can be corrected by our method to generate dynamic interaction sequences.

1. Introduction

The paper introduces dynamic grasp synthesis as generating physically plausible hand-object motions from a single static grasp reference. D-Grasp uses reinforcement learning, physics simulation, and hierarchical control to produce stable grasps and object motions from sparse input.

  • Dynamic grasp synthesis generates continuous, physically plausible hand-object interactions from a single static grasp reference.The task must maintain grasp stability while moving an object to a target 6D pose.
  • D-Grasp formulates the task as reinforcement learning with physics simulation to model contacts, forces, collisions, and object stability.Simulation supplies exploration and evaluation while discouraging interpenetration and slipping.
  • A hierarchical framework separates low-level stable grasping from high-level motion synthesis toward a user-specified target position.The grasping policy uses a static grasp label, while the motion module generates object-moving trajectories.
  • Motion-capture, static-grasp, and image-based labels may be unstable in physics simulation, but D-Grasp learns physically plausible stable grasps from them.The method also generates motions spanning a wide range of target poses.
  • The paper contributes the dynamic grasp synthesis task, an RL-based D-Grasp method, and generation from static references originating from several sources.These sources include motion capture, static grasp synthesis, and image-based pose estimation.

2. Related Work

Prior work studies hand-object pose estimation, static grasp synthesis, dexterous control, and physics-aware inference, but D-Grasp targets dynamic dexterous hand-object interactions from a single grasp label.

  • Hand-object research has emphasized pose estimation and static grasp synthesis from annotated data or RGB images.These approaches estimate hand and object poses or meshes but do not by themselves address continuous object-moving interaction.
  • Dexterous hand-control methods often use anchored hands or costly expert demonstrations, whereas D-Grasp requires only one frame-level grasp label per sequence.The cited contrast concerns the supervision and trajectory-generation requirements of prior approaches.
  • Physics-aware methods use simulation for plausibility validation, contact reasoning, pose correction, or policy regulation in related estimation and interaction tasks.These approaches motivate incorporating physical reasoning into learned hand-object systems.
  • D-Grasp differs by introducing fine-grained dynamic dexterous hand-object interactions that must respect object displacement, contact dynamics, and extensive collision queries.The paper characterizes this as a novel task and an important first step for human-object interaction.

3. Method

D-Grasp formulates dynamic grasp synthesis as hierarchical reinforcement learning in a physics simulation, separating stable grasp establishment from object motion toward a target pose. Its representation, residual control, reward design, and wrist-guidance technique encode hand-object state and grasp-reference information for physically plausible interaction.

  • 3. Method: The method uses a hierarchical framework that separates a grasping policy from motion synthesis while executing both concurrently during object movement.The grasping policy establishes and retains a stable grasp, while motion synthesis brings the object to the target pose.
  • 3.1. Task Setting: The task starts from a static grasp label and requires grasping an object before moving it into a specified 6D goal pose.The label contains the hand’s global and local pose together with the object pose.
  • 3.1. Task Setting: The physics simulation uses a controllable hand model with MANO-derived structure, actuated joints, joint limits, and dataset object meshes simplified for faster simulation.Simple objects may be approximated with primitives, while complex meshes are decimated.
  • 3.1. Task Setting: The reinforcement-learning formulation models states, actions, rewards, deterministic transitions, and initial states within a Markov decision process.The policy selects actions from the hand-object state to maximize discounted rewards.
  • 3.1. Task Setting: The action space controls 45 finger-joint actuators plus six global pose degrees of freedom through PD controllers, with residual actions modifying a bias configuration.Residual control produces smoother finger motion and more stable grasps than directly predicting reference joint angles.
  • 3.2.1 Feature Extraction for Grasping: Feature extraction transforms the state and grasp label into object- or wrist-relative inputs containing pose, velocity, contact, displacement, and goal information.The goal components encode target joint positions, target rotations, and target contacts; the reward combines position, pose, contact, and regularization terms.
  • 3.2.3 Wrist-Guidance Technique: Wrist-guidance biases the translational PD controller toward the hand position extracted from the grasp label, improving performance and convergence over using previous joint positions.The technique is applied during the grasping phase to guide the hand toward the object.

4. Experiments

Experiments evaluate stable grasp correction, unseen-object generalization, motion synthesis, and component importance using physics-based metrics and qualitative results.

  • Experimental Setup: Experiments use DexYCB and HO3D labels, PPO training in RaiSim, and success rate, interpenetration, simulated distance, and contact ratio metrics.Physical plausibility is evaluated on unseen grasp labels and unseen objects.
  • 4.3. Grasping Objects: Static grasp labels from motion capture, synthesis, or image-based estimation often fail in physics simulation, but the method corrects them into stable grasps.Qualitative results include correcting unstable grasps, interpenetration, and physically implausible poses after offline optimization.
  • 4.3. Grasping Objects: The method outperforms baselines across all reported grasping metrics and achieves 0 interpenetration loss in the physics simulation.Evaluation on the original MANO hand and detailed object meshes reports nonzero interpenetration because training uses simplified simulation models.
  • 4.3. Grasping Objects: Across six unseen-object splits, the method outperforms the baseline on all metrics, although overall success rate leaves room for improvement.Each test set contains three objects held out during training, and results are averaged across the six splits.
  • 4.4. Motion Synthesis: The hierarchical framework outperforms Flat-RL, while replacing active grasping with static pose maintenance reduces motion-synthesis performance.The evaluation uses 100 randomly sampled out-of-distribution target poses and compares several method variants.
  • 4.5. Ablations: Each ablated component improves performance significantly across metrics, with contact reward and feature representation especially important for stable grasps.The ablations remove the feature layer, goal space, contact reward, or wrist guidance.

5. Discussion and Conclusion

The paper introduces dynamic grasp synthesis and demonstrates physics-based reinforcement learning for stable grasps, object motion, and unseen-object generalization.

  • Discussion and Conclusion: D-Grasp frames dynamic grasp synthesis as reinforcement learning in physics simulation, learning from a single external grasp label.The method generates natural, physically plausible hand-object interaction sequences.
  • Discussion and Conclusion: The method learns stable grasps, moves objects without slipping, and generalizes to unseen objects when a static hand-pose reference is available.The authors describe this as a proof-of-concept that could scale to larger train/test sets.
  • Discussion and Conclusion: The approach assumes friction, inertia, and center of mass are known a priori, motivating future perceptual estimation of these dynamics properties.These quantities are often unavailable in real-world settings.

Supplementary Material

The supplementary material expands the paper with method, implementation, evaluation, societal-impact, and notation resources.

  • Supplementary Material: The supplement provides detailed method descriptions, implementation details, additional qualitative and quantitative results, societal-impact discussion, and a notation glossary.It also includes a video of the interactions.

A. Method Details

The method extracts object-relative grasp features so grasping is invariant to the object’s 6D pose, combining target hand geometry, joint rotations, and desired contacts.

  • A.1. Grasping Feature Extraction Details: Object-relative features make grasping invariant to the object’s 6D pose during the grasping phase.The representation is designed to remain flexible as the object moves.
  • A.1. Grasping Feature Extraction Details: Target joint positions are computed from object and hand 6D poses plus the target joint configuration using forward kinematics.Distances between current and target joint positions are then represented in object-relative and wrist-relative coordinates.
  • A.1. Grasping Feature Extraction Details: Target rotations encode angular distances between current and target joint and wrist rotations, with the wrist transformed into the object-relative frame.Local joint rotations are compared directly, while wrist orientation is made object-pose invariant.
  • A.1. Grasping Feature Extraction Details: Desired contacts are computed by measuring distances between hand-joint mesh vertices and object vertices, producing contact and active-contact vectors.A finger part is labeled in contact when any corresponding vertex pair is within 0.015m.

A.2. Motion Synthesis

Motion synthesis moves the hand toward estimated target poses through closed-loop control or a learned global-pose policy, using position and orientation errors to guide the object toward its target.

  • A.2. Motion Synthesis: The closed-loop controller recomputes the hand-to-target 6D pose error after each simulation update and sends the resulting target pose to a PD controller.This module replaces control of the grasping policy’s first 6DoF during motion synthesis.
  • A.2. Motion Synthesis: The learned motion policy uses features relevant to global hand-pose control, including the object’s current and target positions and orientations.Hand proprioception such as joint angles and angular velocities is excluded from these motion features.
  • A.2. Motion Synthesis: The motion reward combines position and angular terms, measuring distances between the object’s current and target position and orientation.The position and orientation components are weighted by αx and αq.
  • A.2. Motion Synthesis: A full-arm biomechanical model is identified as a future extension that could impose movement constraints and potentially make learned motion synthesis preferable to inverse kinematics.The current work directly controls the hand’s 6D pose, where an IK-based solution is expected to outperform the learning-based variant.
  • A.2. Motion Synthesis: RaiSim supplies contact and dynamics information for the physics-based hand simulation used by the method.The simulation provides the physical interaction signals needed for control and evaluation.

B. Implementation Details

Implementation uses RaiSim with controllable hand meshes, data-driven joint limits, PPO training, parallel environments, and mesh decimation to improve simulation efficiency.

  • B. Implementation Details: The implementation builds a controllable hand model in RaiSim, grouping skinned vertices into body-part meshes and limiting joint ranges from data.RaiSim is selected for non-convex mesh modeling and efficient parallel training.
  • B. Implementation Details: PPO policies are trained with a parallelized worker scheme, using one worker per grasp label and a single policy across training objects and grasps.The DexYCB setup uses 376 parallel environments, while other experiments adjust worker counts to match batch size.
  • B. Implementation Details: Mesh decimation reduces object-mesh vertices to speed physics simulation during training.The original and decimated meshes are shown as the comparison in Figure 6.

B.3. Metrics Details

Evaluation measures physical grasp stability, object-position accuracy, orientation accuracy, and baseline behavior using sequence-level metrics and implementation descriptions.

  • B.3. Metrics Details: Success rate is the primary physical-plausibility metric: the fraction of sequences maintaining a stable grasp without slipping or falling for 5 seconds.A score of 0.0 means no sequence succeeds, while 1.0 means all sequences succeed.
  • B.3. Metrics Details: Mean position error measures the average distance between the object’s 3D position and its target 3D position during motion synthesis.The metric uses the object-position error defined as go,x.
  • B.3. Metrics Details: Geodesic distance measures the angular difference between the object’s current and target orientations using their corresponding rotation matrices.The metric compares rotations associated with the current and target 6D object poses.
  • B.3. Metrics Details: The baseline descriptions include PD control initialized from grasp labels and offline IK optimization intended to correct label imperfections.These baselines provide alternative ways to maintain or refine the labeled grasp in simulation.

B.5. Experimental Details

The experiments use varied object sets and grasp-reference sources to evaluate grasping, motion synthesis, and ablations. Several settings modify simulation or sampling to accommodate differences in reference data and support fair comparison.

  • Grasping Objects: Grasping experiments use DexYCB objects, while evaluation also includes the HO3D subset and DexYCB objects for externally generated grasp references.The setup distinguishes training objects from evaluation objects and reports results across both datasets.
  • Grasping Objects: For references without physical object-placement information, the simulation disables hand-surface collision response and allows approaches from any direction.This modification is intended to ensure a fair comparison with motion-capture, synthesized, and image-based references.
  • Motion Synthesis: Motion-synthesis experiments use six YCB objects spanning cylindrical, box-shaped, and complex geometries, with failed grasps filtered before training and evaluation.The selected objects are two cans, two box-shaped objects, a pitcher base, and an extra-large clamp.
  • Ablations: Ablation experiments train per-object policies on one cylindrical, one box-shaped, and one complex YCB object using grasp labels extracted from DexYCB.The objects are the master chef can, sugarbox, and extra-large clamp.

C. Additional Results

Additional results show that policy design affects training speed and generalization, while D-Grasp improves physical plausibility and generalizes unevenly across unseen-object sets. The method remains dependent on realistic grasp references and is not yet indistinguishable from real dynamic grasps.

  • Policy Type Comparison: All-object policies generalize better on DexYCB, whereas per-object policies train faster and perform better on training labels.Per-object policies require approximately 3,000 rather than 10,000 epochs, while the all-object model has stronger generalization performance.
  • Quantitative Grasping Results: Thin objects remain difficult because successful pickup requires fine-grained finger motion or sliding against the surface to overcome static friction.The discussion identifies scissors and a large marker as examples with relatively low success rates.
  • Additional Qualitative Grasping Results: D-Grasp corrects interpenetration and produces more realistic contacts than baseline grasps on DexYCB training and test examples.The qualitative comparisons cover both dataset grasps and grasps generated by an external synthesis method.
  • Reference-Quality Limitation: The method performs best with grasp references that roughly approximate physically plausible human grasps, especially for generalization to unseen objects.Although it can correct some interpenetration and pose noise, it remains conditioned on the reference pose.
  • Generalization to Unseen Objects: On unseen objects, success reaches 0.83 for an easier test set but falls to 0.33 for a set containing scissors and a large marker.Across the different test sets, the method outperforms the static baselines, despite substantial variance.
  • Societal Impact: The generated dynamic grasps are not yet indistinguishable from real ones, and potential applications carry risks including deep-fakes and unequal access to costly systems.The paper discusses possible uses in AR/VR, HCI, and robotics alongside these societal concerns.
Loading 2112.03028v2…