Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands

1 Introduction

• Human grasp capture is laborious, while analytic synthesis relies on limiting assumptions that exclude important grasp types, motivating better synthetic grasp generation for vision systems.Vision-based grasp prediction uses visual inputs to estimate hand pose and joint angles, but depends on high-quality datasets of plausible, diverse grasps.Source: §1 · pp. 2–3
• The method addresses discontinuous contact forces, vanishing gradients before contact, and a rugged optimization landscape with coarse-to-fine SDF surface smoothing, leaky contact gradients, and relaxed force variables.The smoothed, padded surface gradually resolves to detailed geometry; force variables make physics violations a cost rather than a constraint.Source: §1 · p. 3
• Grasp’D produces more stable, plausible, high-contact grasps than analytic synthesis, which typically finds fingertip-only grasps; it also lowers interpenetration versus analytic baselines.The method is evaluated on synthetic ShapeNet objects and meshes reconstructed from the YCB RGB-D dataset, and also works with robotic hands.Source: §1 · p. 4, Summary of contributions · p. 4

Summary of contributions:

• Grasp’D scales to tens of thousands of dense contacts without simplifying assumptions, using differentiable simulation for gradient-based grasp optimization.Its approach is intended to avoid assumptions such as pre-specified contact locations or reduced search dimensionality that limit grasp plausibility and exclude key grasp types.Source: §1 · p. 3, Summary of contributions · p. 4

Grasp synthesis

• Multi-finger synthesis remains largely analytic, and dimensionality reduction or fixed contact locations limit discoverable grasps; Grasp’D optimizes the full grasp space.For parallel-jaw grippers, simulation-based metrics better align with human judgment and real-world performance, though they are more computationally costly.Source: §2 · pp. 4–5

Human grasp capture

• Captured human-grasp datasets are costly to collect and specific to human hands; contact-pattern transfer can fail across morphologies, while Grasp’D procedurally generates grasps for human or robotic hands.Existing collection may require calibrated cameras, lidar, special gloves, and human annotations; transferred contacts may not be achievable under different hand morphologies and articulation constraints.Source: §2 · p. 5

Vision-based grasp prediction

• Multi-finger vision-based systems use analytic or captured-human-grasp datasets, while Grasp’D can provide training grasps or directly synthesize grasps using reconstructed objects.Prior systems include GANHand, Grasping Field, and contact-model refinement; Grasp’D’s synthesis pipeline can generate higher-quality training data for vision-based grasp prediction.Source: §2 · pp. 5–6

Differentiable Grasping

• Unlike prior differentiable multi-finger methods [58], Grasp’D uses differentiable simulation, supports varying friction and contact forces, and scales to tens of thousands of contacts.Method [58] manually labels 45 potential contact points; assumes zero friction and uniform contact-force magnitudes. selects 3 contacts from 773 MANO vertices and takes about 40 minutes to find 5 acceptable grasps, while Grasp’D synthesizes an acceptable grasp in about 5 minutes.Source: §2 · p. 6

Differentiable Physics

• Prior differentiable simulators have collision or gradient limitations, whereas Grasp’D addresses these shortcomings and uses GPU parallelism to scale to tens of thousands of contacts.Brax and Tiny Differentiable Simulator support only collision primitives; Nimblephysics [92] supports mesh collisions but fails when contact-normal gradients are zero and runs only on CPU.Source: §2 · pp. 6–7

3.2 Object model with coarse-to-fine surface smoothing

• Coarse-to-fine smoothing starts contacts on a padded SDF level set and linearly reduces its offset to zero, enabling optimization across surface-normal discontinuities.The initial offset is approximately 10 cm; the schedule gradually restores the original surface and helps avoid overfitting to local geometric features.Source: §3 · p. 7, §3.2 · p. 9

3.1 Rigid body dynamics

• The simulator uses simple rigid-body dynamics and semi-implicit Euler time stepping to model hand-object interaction.The hand is kinematic and the object is dynamic, so contact forces affect the object but not the hand.Source: §3.1 · p. 8, §3.4 · p. 10

3.3 Contact dynamics with leaky gradient

• Penalty-based normal and Coulomb-friction forces support autodiff, while leaky normal-force gradients let inactive contacts contribute gradients instead of leaving optimization on a plateau.The leak parameter α is set to 0.1 in the experiments.Source: §3.3 · p. 9

3.4 Grasping metric and problem relaxation

• The grasp metric averages final object speeds across simulations with different initial velocities; prescribed contact wrenches relax the rugged problem into task-resistance and physics-mismatch losses.The task loss evaluates whether desired forces resist the initial velocities, while the physics loss measures mismatch between desired and pose-generated contact forces.Source: §3.4 · pp. 10–11
• Additional losses encourage joints toward range midpoints, penalize joint-limit violations, and penalize hand self-intersections.For self-intersection loss, contacts between neighboring links in the chain are ignored.Source: §3.4 · p. 11

3.5 Optimization

• Optimization uses Modified Differential Multiplier Method constraints on task and joint-limit losses while Adamax updates hand pose and desired contact forces.The method minimizes physics, joint-limit, and self-intersection losses subject to those constraints.Source: §3.5 · p. 11

4.2 Grasp synthesis with ShapeNet models

• Generated grasps achieve around 42cm2 of contact area—about 4× the ObMan [40] baseline—and top-2 grasps reduce simulation displacement by ∼3×, with modest gains in analytic stability.Table 2 summarizes over 4× denser contact area and 4× lower simulation displacement; the higher contact comes with higher interpenetration but a similar contact-area-to-intersection-volume ratio.Source: Table 2 · p. 12, §4.2 · pp. 12–13

4.1 Experimental setup

• Evaluation measures contact area, intersection volume and their ratio, analytic Ferrari-Canny and volume stability, and simulated displacement; MANO uses a 44-component PCA hand parameterization.The PCA parameterization serves as a prior for human-like poses.Source: §4.1 · p. 12

4.3 Grasp synthesis from RGB-D input of unknown objects

• On imperfect RGB-D reconstructions, simulation discovers plausible human grasps for many objects that align with real-world object poses, though results trail ground-truth models.The YCB reconstructions use only 5 views from 3 cameras (2.5% of the available captures); synthesis takes about 5 minutes per grasp.Source: §4.3 · pp. 13–14, Figure 4 · p. 13

4.4 Ablation study

• Without problem relaxation, grasps have much lower contact area and intersection volume; removing coarse-to-fine smoothing leaves most metrics comparable but raises simulation displacement by about 25%.The study omits a no-leaky-gradient variant because a hand not touching the object at initialization receives no gradient and remains stuck on a plateau.Source: §4.4 · p. 14, Table 3 · p. 14

• Differentiable simulation enables gradient-based optimization to generate high-quality, high-contact grasps for multi-finger grippers while scaling to thousands of dense contacts.The pipeline can generate large datasets of plausible grasps.Source: §5 · p. 14
• The approach outperforms the existing classical grasping algorithm both quantitatively and qualitatively.Source: §5 · p. 14
• The PyTorch-compatible approach can integrate into existing pipelines, and its grasps can benefit vision systems trained on synthetic data.Source: §5 · p. 14

Supplementary Material

• The supplement complements the main paper with additional implementation details and results.Source: Overview · p. 15

Overview

• Appendix A describes the implementation and experimental setting.Source: Overview · p. 15
• Appendix B provides additional YCB dataset results using the human MANO hand model.Source: Overview · p. 15
• Appendix B also provides additional YCB dataset results using a robotic Allegro hand model.Source: Overview · p. 15
• Appendix C visualizes MANO grasp optimization trajectories for YCB and ShapeNet objects, showing grasps improving as optimization progresses.Source: Overview · p. 15
• Appendix D provides additional validation results for the RGB-D reconstruction presented in Section 4.3 of the main paper.Source: Overview · p. 15

A.1 Dataset listings --; A.3 Mesh processing

• The mesh pipeline computes a discretized SDF from watertight meshes on a 256 × 256 × 256 grid padded by 1 cm.ManifoldPlus [47] makes meshes watertight before sampling and computing signed distances. Tables 1–3 list ShapeNet objects for the main experiment and YCB objects for the RGB-D and ablation experiments.Source: §A.1 · p. 15, §A.3 · p. 16

A.2 Initialization and smoothing schedule; Initialization.; Coarse-to-fine smoothing schedule.

• Different initial hand poses yield different final grasps, and a coarse-to-fine smoothing schedule transitions optimization from rough pose finding to detailed surface conformity.The initialization heuristic adapted from sets joints fully open and force variables bfc to zero, then samples a surface approach point and roll angle; the palm begins 10 cm from the approach point. The smoothing radius decreases linearly to zero over the first 5,000 of 7,000 steps, followed by 2,000 steps at zero.Source: Initialization · pp. 15–16, Coarse-to-fine smoothing schedule · p. 16

A.4 Reconstruction pipeline

• The RGB-D reconstruction registers 15 depth images into a point cloud, applies Poisson reconstruction at depth 5, and fills remaining mesh holes.The YCB objects are captured from five cameras at 120 angles; the selected depth images come from three cameras at five angles. PyMeshFix completes the unseen bottom and other remaining holes.Source: §A.4 · pp. 16–17

A.5 Simulation details; True signed distances.

• Grasp optimization uses a 1 × 10^-5-second simulation step, hand-specific contact samples, and fixed stiffness and friction parameters.MANO uses all 773 vertices as contact locations; Allegro uses approximately 3,000 sampled surface points. The settings are kn = 1 × 10^6, kf = 1 × 10^8, µ = 0.8, and leaky-gradient proportion α = 0.1. For evaluation metrics relying on an object SDF, the authors use libigl with the ground-truth mesh rather than the discrete grid approximation.Source: §A.5 · p. 17, True signed distances · p. 18

A.6 Evaluation details; Interpenetration volume; Contact area; Contact area to interpenetration volume ratio.; ϵ (Ferrari-Canny) metric; Volume metric

• Evaluation considers grasp contact patterns and stability, reporting interpenetration volume, contact area, contact-area-to-interpenetration ratio, the smallest-break-force stability measure, and Grasp Wrench Space (GWS) volume.Interpenetration volume is the hand-object intersection, computed by voxelizing the hand at 1 mm resolution and querying the object SDF; contact area is measured in cm2 by voxelizing only the hand surface as an empty shell. The ratio captures interpenetration conditional on contact, while the stability measure is the smallest force that breaks a grasp.Source: §A.6 · p. 17, Interpenetration volume · p. 17, Contact area · p. 17

Simulation displacement

• Simulation displacement is evaluated in PyBullet, a third-party simulator with different collision detection, contact, and time-stepping, to avoid evaluating with the optimization simulator.The evaluation uses GANHand’s implementation and its default physics parameters except for friction coefficient 1.2 instead of 3.0.Source: Simulation displacement · p. 18

Evaluating on top k grasps.

• The method reports its top 2 and top 5 grasps per object, ranked by simulation displacement, after generating 10 grasps with 7,000 optimization steps each.For ObMan, the top grasps are ranked by the heuristic in Appendix C.2 of, which encourages palm and phalange contact to counter the fingertip bias of analytic synthesis; its grasps are generated with about 70,000 simulated-annealing steps.Source: Evaluating on top k grasps · p. 18

A.8 Timing

• About 2 minutes are needed per MANO grasp when three are synthesized in parallel, compared with around 5 minutes for GraspIt! analytic synthesis.A single 7,000-step grasp takes about 5 minutes on a mobile Nvidia RTX 2070; the authors note this is not yet realtime performance.Source: §A.8 · p. 19

B YCB results

• The YCB examples apply the method to objects from the dataset using both Allegro robotic and MANO human hand models.Figures 1 and 2 show the Allegro and MANO examples, respectively.Source: B YCB results · p. 19
• The YCB results include examples using the Allegro robotic hand model.These examples are shown in Figure 1.Source: B YCB results · p. 19
• The YCB results also include examples using the MANO human hand model.These examples are shown in Figure 2.Source: B YCB results · p. 19

C Optimization trajectories

• Optimization trajectories are visualized by rendering hand poses at regular intervals as optimization progresses.The visualization covers ten grasps in total.Source: C Optimization trajectories · p. 19
• Five of the ten rendered optimization trajectories are for grasps on YCB objects.The YCB trajectories are shown in Figure 3.Source: C Optimization trajectories · p. 19
• The other five rendered optimization trajectories are for grasps on ShapeNet objects.The ShapeNet trajectories are shown in Figure 4.Source: C Optimization trajectories · p. 19

D Additional RGB-D results

• The YCB RGB-D results show three synthesized grasps per reconstructed object from two viewpoints.The grasps are described as visually plausible.Source: D Additional RGB-D results · p. 19, Table 4 · p. 20
• RGB-D reconstruction performance is poorer than with ground-truth object models, but metrics are comparable to Grasping Field [52] and GANHand [15].Small reconstruction errors may cause large grasp-synthesis errors; future work could optimize the reconstruction alongside the grasp.Source: Table 4 · p. 20

E Training on synthesized data

• Grasp’D fine-tuning outperforms additional GraspIt! data on unseen YCB objects, producing higher-contact grasps with a better contact-area-to-intersection ratio and simulation displacement despite slightly greater intersection volume.The study also reports significantly higher contact and improved simulation displacement.Source: E Training on synthesized data · p. 20, Table 5 · p. 20
• The network was trained for 1,400 epochs on ObMan, then fine-tuned for 100 epochs on 1,000 synthesized grasps of ShapeNet objects already in ObMan before testing on eight YCB objects.The fine-tuning data came from either the GraspIt! simulator or Grasp’D.Source: E Training on synthesized data · p. 20