Source-linked AI summary
Grasping Field: Learning Implicit Representations for Human Grasps
Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael Black, Krikamol Muandet, Siyu Tang
TL;DR
Realistic synthesis of human grasps remains difficult because human hands and object interactions are complex. The paper learns a Grasping Field that represents hand and object signed distances jointly, then uses it for grasp generation and reconstruction. Experiments produce natural, physically plausible grasps and comparable state-of-the-art hand reconstruction, while reducing reconstruction interpenetration.
Problem
Fully automatic synthesis of realistic human grasps remains unsolved, requiring natural conformity, semantic plausibility, and physical plausibility for complex hands and objects.
Method
The paper learns a deep-neural-network-parameterized Grasping Field mapping each 3D point to signed distances from hand and object surfaces in a shared representation.
Results
The model generates plausible grasps from object point clouds, improves contact physicality and reconstruction interpenetration, and achieves comparable 3D hand reconstruction to state-of-the-art methods.
Takeaways & Limitations
The learned signed-distance representation provides an expressive framework for modeling hand-object interaction across grasp generation and single-image reconstruction.
Takeaways & Limitations
The current representation does not explicitly model object functionality or human action intentions, which can produce different grasps for the same object.
Abstract
from arXiv · showhide
Robotic grasping of house-hold objects has made remarkable progress in recent years. Yet, human grasps are still difficult to synthesize realistically. There are several key reasons: (1) the human hand has many degrees of freedom (more than robotic manipulators); (2) the synthesized hand should conform to the surface of the object; and (3) it should interact with the object in a semantically and physically plausible manner. To make progress in this direction, we draw inspiration from the recent progress on learning-based implicit representations for 3D object reconstruction. Specifically, we propose an expressive representation for human grasp modelling that is efficient and easy to integrate with deep neural networks. Our insight is that every point in a three-dimensional space can be characterized by the signed distances to the surface of the hand and the object, respectively. Consequently, the hand, the object, and the contact area can be represented by implicit surfaces in a common space, in which the proximity between the hand and the object can be modelled explicitly. We name this 3D to 2D mapping as Grasping Field, parameterize it with a deep neural network, and learn it from data. We demonstrate that the proposed grasping field is an effective and expressive representation for human grasp generation. Specifically, our generative model is able to synthesize high-quality human grasps, given only on a 3D object point cloud. The extensive experiments demonstrate that our generative model compares favorably with a strong baseline and approaches the level of natural human grasps. Our method improves the physical plausibility of the hand-object contact reconstruction and achieves comparable performance for 3D hand reconstruction compared to state-of-the-art methods.
1. Introduction
The paper addresses realistic human grasp synthesis by introducing a learned representation that models hand-object contact explicitly. A generative model built on this representation produces plausible grasps from object point clouds, while reconstruction experiments improve contact physicality and hand reconstruction.
- Motivation: Realistic human grasp synthesis remains unsolved because hand anatomy, object variety, and natural physical interaction make posing difficult.Existing data-driven approaches are mainly limited to simple robotic end effectors.
- Representation: The Grasping Field maps each 3D point to signed distances from the hand and object surfaces, representing their interaction in a common space.A deep neural network parameterizes and learns this continuous function from data.
- Grasp generation: The proposed generative model synthesizes semantically and physically plausible human grasps from a 3D object point cloud, including for unseen objects.The generated grasps are reported as similar to ground truth.
- Reconstruction: The representation avoids first constructing hand and object meshes to infer contact, while enabling efficient enforcement of non-interpenetration and proper contact.The authors report reduced interpenetration and improved 3D hand reconstruction compared with Hasson et al. [29].
- Contributions: The paper contributes a Grasping Field representation, a grasp generator, a single-pass RGB reconstruction model, and experiments showing natural-looking grasps and improved hand reconstruction over baseline [29].These contributions target both grasp synthesis and 3D hand-object reconstruction.
2. Related work
Related work spans human grasp capture, grasp synthesis, hand pose estimation, object representations, and joint hand-object reconstruction. This paper focuses instead on learning an interaction representation that is efficient for deep neural networks and models hand and object in signed-distance space.
- Human grasp and contact: Human grasp research includes capture, recognition, and contact modeling from diverse modalities, although some methods interfere with natural movement.This work emphasizes learning an interaction representation rather than capturing contact directly.
- Grasp synthesis: Grasp synthesis approaches are commonly analytic, data-driven, or hybrid, with prior neural methods predicting hand parameters from object information.Analytic methods formulate constrained optimization, while hybrid methods combine analytic models and deep learning.
- Hand pose estimation: Hand pose estimation uses RGB, RGB-D, and depth inputs, increasingly recovering hand pose and surface with statistical hand models such as MANO.Synthetic data is often used because large-scale 3D ground truth is limited.
- Object model representation: Implicit functions such as Signed Distance Functions and Occupancy Networks model arbitrary object topology with adjustable resolution.The paper adopts implicit functions to represent hand-object interaction.
- Hand-object interaction: Prior joint reconstruction methods combine MANO with mesh-based object representations, whereas this work learns hand and object in signed-distance space.The signed-distance formulation supports arbitrary shape modeling and easier distance-field manipulation.
3. Method
The method represents hand-object interaction with a learned Grasping Field and uses it for grasp generation and single-image reconstruction. Signed distances make contact and interpenetration explicit, while task-specific networks recover plausible hand-object configurations.
- Grasping Field representation: The Grasping Field maps each 3D point to signed distances from the hand and object surfaces, explicitly representing contact and interpenetration.The contact manifold is defined where both signed distances are zero.
- Grasping Field representation: A deep neural network learns the Grasping Field from data, enabling interaction inference without explicit hand and object surfaces.The learned field serves as an interaction prior for inferring grasping poses from 3D objects.
- Human grasp synthesis: For grasp synthesis, a generative model takes an object point cloud and produces diverse hand grasps intended to interact naturally with the object.The model uses an encoder-decoder architecture and conditions the hand distribution on encoded object features.
- Human grasp synthesis: The synthesis model trains signed-distance reconstruction, latent-distribution, and hand-part classification objectives, then samples the hand latent vector from a standard normal distribution.The distance loss clips signed distances to the interval [−δ, δ], with δ set to 1cm; hand-part annotations follow MANO.
- RGB hand-object reconstruction: For RGB reconstruction, conditional Grasping Field networks recover hand and object representations in one pass using either separate branches or an interaction-aware one-branch design.Training includes reconstruction, interpenetration, contact, and classification losses.
- RGB hand-object reconstruction: The interpenetration loss penalizes negative sums of predicted hand and object signed distances across 3D space, imposing stronger constraints than losses limited to the intersection volume.After distance queries, surfaces are extracted near zero signed distance using sampled points and marching cubes; MANO fitting can recover a compact hand configuration.
4. Experiments
Experiments evaluate grasp synthesis and 3D hand-object reconstruction using physical, semantic, and geometric metrics across synthetic and real-world datasets. The grasping-field models improve interaction plausibility and reconstruction results relative to the baseline, while performance varies with architecture, losses, and object distribution.
- Evaluation setup: The evaluation covers human grasp generation from 3D object point clouds and 3D hand-object reconstruction from images.Experiments use ObMan, FHB, and HO3D, with reconstruction evaluated on ObMan and FHB.
- Dataset and baseline: The model is trained on synthetic ObMan data and tested on ObMan test objects, FHB, and HO3D, including unseen real-world objects.ObMan provides physics-simulated hand-object interactions across 2772 object meshes and 8 everyday-object classes.
- Evaluation setup: Physical grasp quality is measured by intersection volume and depth, contact ratio, and simulated grasp stability.Semantic quality is assessed through perceptual studies, while reconstruction uses Chamfer distance and hand-joint error.
- 3D hand-object reconstruction: Compared with the baseline, the one-decoder model reduces intersection volume from 6.25 to 0.65 and depth from 1.20 to 0.32.The two architectures achieve comparable hand-reconstruction performance, while the one-decoder design performs better on intersection error through joint hand-object modeling.
- Ablations: Contact and interpenetration losses improve interaction quality for the two-decoder model, whereas the one-decoder model does not require this auxiliary loss.MANO fitting has little effect on reconstruction quality, and predicted MANO joints improve from 2.94cm to 2.6cm against pseudo-ground-truth joints.
5. Conclusion and Discussion
The paper introduces the grasping field as a shared signed-distance representation for hand-object interaction and evaluates it on grasp generation and reconstruction. Experiments report natural, physically plausible grasps and hand reconstruction comparable to state of the art, while object functionality and human action remain outside the current representation.
- Conclusion: The grasping field models the joint distribution of hand and object shape in a common framework for hand-object interaction.The representation supports both grasp generation from objects and hand-object reconstruction from a single image.
- Conclusion: Generated grasps appear natural and physically plausible, while hand reconstruction achieves performance comparable to state-of-the-art methods.
- Discussion: The current representation does not explicitly model object functionality or human action, so it does not distinguish grasps driven by different intentions.The paper identifies action-specific grasp generation using human intention and object affordances as a future direction.
A. Implementation Details
The implementation-details section introduces the neural networks used for human grasp generation and 3D reconstruction before describing their implementation.
- Implementation Details: The paper uses neural networks for human grasp generation and reconstruction, with implementation details discussed after the task-specific network descriptions.
A.1. Architecture
The experiments use a shared decoder design with input-specific encoders for image and point-cloud data. Encoders and decoders are trained jointly end to end, with latent representations tailored to each task.
- Architecture: The same eight-layer fully connected decoder is used across image reconstruction and hand-generation tasks.Encoder architectures change according to the input type, and the encoder and decoder are jointly trained end to end.
- Image reconstruction: Image reconstruction uses an ImageNet-pretrained ResNet18 encoder whose final layer produces a 256-dimensional latent vector.
- Point-cloud reconstruction: Point-cloud reconstruction uses two separated PointNet encoders that produce 256-dimensional hand and object latents concatenated into a 512-dimensional code.Each encoder maps points to 512-dimensional features, applies five ResNet blocks, and outputs a 256-dimensional latent vector.
- Hand generation: For hand generation, the hand encoder produces a 256-dimensional vector per point and concatenates it with the 256-dimensional object latent vector.
- Known-object reconstruction: Image reconstruction with known objects combines a 128-dimensional object latent from sampled object surfaces with a 128-dimensional hand latent from ResNet18.The concatenated 256-dimensional code is passed to the decoder.
A.2. Data preparation
Training points are prepared by normalizing each hand-object mesh pair into a shared unit-cube coordinate system and sampling points from that cube.
- Each hand-object mesh pair is translated so the hand root joint is at the origin.
- Both meshes are scaled to fit within a unit cube using one dataset-wide scaling factor.The shared factor keeps hand normalization consistent across the dataset.
- After normalization, 40,000 points are sampled inside the unit cube for training.
A.3. Training
Training uses staged contact and KL objectives, Adam optimization, and Marching Cubes inference to learn and recover hand-object geometry.
- The contact loss starts disabled because applying it immediately or omitting its reconstruction mask causes dramatic degradation.
- KL annealing keeps the loss weight at 0 for 200 epochs, then increases it linearly to 0.1 over the next 200 epochs.Applying KL loss from the beginning causes posterior collapse in the generative network.
- Models use Adam with learning rate 10^-4, decay to 5 × 10^-5 after 600 epochs, and train for 1,300 total epochs.The final 100 epochs include hand-part classification loss.
- At inference, Marching Cubes at resolution 128 reconstructs meshes after a two-stage procedure dynamically scales the cube to the object.A second cube covers points with negative signed distances, which indicate mesh interiors.
- The decoder concatenates the latent code with each 3D query and again mid-network, while its final layer predicts object and hand distances plus hand-part scores.The latent vector has 256 or 512 dimensions depending on the task.
B. Dataset Analysis
FHB and HO-3D provide detailed annotations but limited object and grasp diversity and contain substantial hand-object mesh interpenetration, motivating ObMan for training.
- FHB contains 3 objects and HO-3D contains 10, limiting the diversity of objects and grasp types available for learning.
- 91.94% of HO-3D training examples contain contact, yet contacting examples average 10.91 cm3 intersection volume and 1,56 cm depth.
- The FHBc subset retains examples near the object and reports 97.1% contact, while similar hand-object intersection remains evident.The subset excludes milk-bottle examples and cases with hand-joint distance above 1 cm.
- Because these datasets show considerable intersection volume and depth, the study uses ObMan, whose contact ground truth is more suitable for learning physically plausible grasps.
- ObMan-trained grasps receive higher perceptual scores than HO3D ground-truth grasps, linking physical plausibility with judged naturalness.The paper specifically attributes plausibility to avoiding interpenetration and maintaining proper contact.
C. Details of the evaluation metrics
The evaluation combines physical stability and contact measures, perceptual ratings, and reconstruction errors to assess generated grasps and reconstructed geometry.
- Grasp synthesis is evaluated with physics metrics and perceptual studies, while reconstruction uses Chamfer distance and hand joint error.
- Intersection volume and depth quantify hand-object mesh overlap, with depth defined as zero when meshes do not overlap.Meshes are voxelized using 0.5 cm voxels for the intersection calculation.
- The contact ratio counts samples whose hand surface is on or inside the object surface; the ideal contact ratio is 100%.High contact ratio should accompany small intersection volume and depth.
- Simulation displacement measures grasp stability through the average object mass-center displacement while the hand is fixed.Smaller displacement indicates a more stable grasp.
- Perceptual fidelity is rated on Amazon Mechanical Turk from 1 to 5 using images rendered from six views.Each generated sample is scored for authenticity by participants.
- Hand reconstruction quality uses mean-square point-cloud Chamfer distance in cm2 and mean Euclidean joint distance.The MANO wrist is sealed to create a watertight mesh for fair comparison.
E. Qualitative results
Qualitative results show that the model reconstructs interacting hand meshes, generates varied grasps, and transfers to unseen objects, while some generated samples still exhibit artifacts or interpenetration.
- The model recovers hand meshes with proper interaction with objects in single-image reconstruction.
- MANO fitting matches inferred hand meshes even when rasterized reconstructions contain merged fingers.
- The VAE generates a variety of grasps conditioned on an object surface point cloud.
- The same model generates grasps for HO3D objects never seen during training, demonstrating qualitative transfer beyond ObMan.
- Some generated samples contain SDF artifacts and interpenetration between fitted MANO hands and objects.
- Baseline samples can show hand-object interpenetration and, in some cases, no contact with the object.