PaperlayerView exact revision on arXiv

Source-linked AI summary

Hand-Object Contact Consistency Reasoning for Human Grasps Generation

Hanwen Jiang, Shaowei Liu, Jiashun Wang, Xiaolong Wang

arXiv:2104.03304v1cs.CV

TL;DR

Natural human grasp generation is challenging because multi-finger hands require complex, physically plausible, and natural contact configurations. The paper models consistency between hand contact points and object contact regions through training objectives and a self-supervised test-time adaptation task. Experiments report significant gains over state-of-the-art methods, especially for unseen and out-of-domain objects.

  • Problem

    Human grasp generation is difficult because multi-finger hands have many degrees of freedom and require natural, physically plausible contact.

  • Method

    The framework combines GraspCVAE and ContactNet with contact-consistency losses and a self-supervised task for test-time grasp adaptation.

  • Results

    The method achieves significant improvements over state-of-the-art grasp-generation approaches, with larger gains on unseen and out-of-domain objects after test-time optimization.

  • Takeaways & Limitations

    Contact consistency supports more natural and stable grasps while strengthening cross-domain generalization.

Abstract

from arXiv · show

While predicting robot grasps with parallel jaw grippers have been well studied and widely applied in robot manipulation tasks, the study on natural human grasp generation with a multi-finger hand remains a very challenging problem. In this paper, we propose to generate human grasps given a 3D object in the world. Our key observation is that it is crucial to model the consistency between the hand contact points and object contact regions. That is, we encourage the prior hand contact points to be close to the object surface and the object common contact regions to be touched by the hand at the same time. Based on the hand-object contact consistency, we design novel objectives in training the human grasp generation model and also a new self-supervised task which allows the grasp generation network to be adjusted even during test time. Our experiments show significant improvement in human grasp generation over state-of-the-art approaches by a large margin. More interestingly, by optimizing the model during test time with the self-supervised task, it helps achieve larger gain on unseen and out-of-domain objects. Project page: https://hwjiang1510.github.io/GraspTTA/

1. Introduction

Human grasp generation is difficult because multi-finger hands create complex contact configurations, while grasps must also be physically plausible and natural. The paper addresses this by enforcing consistency between hand contact points and object contact regions during training and test-time adaptation.

  • Human grasp generation is challenging because multi-finger hands have many degrees of freedom and complex contact patterns.
  • Prior methods model hand contact points but overlook object regions that may also serve as contact sites.
  • The proposed framework enforces mutual agreement between hand contact points and object contact regions for grasp generation.
  • GraspCVAE uses two consistency losses: one brings the hand toward the object surface, while the other aligns predicted hand contact with object contact regions.
  • ContactNet predicts object contact maps and supplies a self-supervised consistency signal for test-time GraspCVAE adaptation without extra outside supervision.
  • Experiments on Obman, HO-3D, and FPHA report significant improvements, including larger gains on unseen and out-of-domain objects after test-time optimization.

2. Related Works

Related work spans hand-object interaction modeling, grasp generation, affordance prediction, and test-time learning. The paper builds on explicit physical constraints, object contact maps, affordance modeling, and self-supervised adaptation while integrating these ideas for grasp generation.

  • Hand-object interaction: Hand-object interaction research estimates poses jointly or learns interaction representations from multimodal data.
  • Hand-object interaction: Explicit hand-centric physical constraints have been used to avoid hand-object penetration during interaction modeling.
  • Grasp generation: Human grasp generation remains difficult because the hand has a higher degree of freedom than simpler gripper systems.
  • Grasp generation: Earlier grasp-generation work used object contact maps to filter generated grasps, treating contact maps as constraints rather than learning targets.
  • Affordance prediction: Affordance prediction has been applied to grasp learning and robotic manipulation, but the paper also targets perceptual naturalness and physical plausibility.
  • Learning on test instances: Test-time learning methods use self-supervision on individual test examples to adapt neural networks without conventional labels.

3. Approach

The approach models human grasp generation as mutual hand–object contact consistency, using GraspCVAE and ContactNet during training and test-time adaptation. Two contact-aware losses improve grasp plausibility, while a self-supervised consistency task refines grasps for novel objects.

  • Contact-consistent training: Two contact-consistency losses encourage prior hand contact vertices to approach the object surface and object contact regions to be touched by the hand.Together, the losses seek mutual agreement between the hand’s contact points and the object’s contact regions.
  • GraspCVAE: GraspCVAE generates MANO-parameterized hand grasps from object point clouds, using both hand and object inputs during training and only the decoder with object information at test time.The encoder reconstructs grasps during training, while test-time generation samples a latent code and conditions the decoder on object features.
  • Contact-consistent training: The baseline combines mesh and MANO-parameter reconstruction, latent-distribution regularization, and penetration losses before adding the contact-aware objectives.The reconstruction loss balances vertex, pose, and shape terms, while the KL-divergence and penetration terms complete the baseline objective.
  • ContactNet: ContactNet predicts an object contact map from hand and object point clouds by combining local object-point features with global hand-object features.Its output assigns a contact score to each object point.
  • Test-time contact reasoning: At test time, ContactNet supervises GraspCVAE refinement by matching the contact map computed from the generated hand with the map predicted from the hand-object pair.This cascade enables a self-supervised consistency loss when no ground-truth grasp is available.

4. Experiment

Experiments evaluate grasp quality, diversity, loss designs, ContactNet variants, and test-time adaptation across in-domain and out-of-domain datasets. The framework improves physical plausibility, stability, perceptual quality, and cross-domain generalization while balancing penetration against displacement.

  • Datasets and evaluation: The evaluation uses Obman for training and testing, while HO-3D and FPHA assess generalization to objects never seen during training.Obman contains 2772 object meshes across 8 everyday-object classes; HO-3D and FPHA are real datasets used for evaluation.
  • Qualitative results: Generated grasps are qualitatively stable and natural on in-domain and out-of-domain objects, while sampling different object poses produces diverse grasps.Figure 8 shows five generated grasps for each of four out-of-domain objects, with each example presented in two views.
  • Quantitative results: The framework significantly improves physical plausibility, grasp stability, and perceptual score over GF across all three datasets, with stronger cross-domain generalization.On HO-3D and FPHA, the method improves both penetration and stability relative to GF, providing a better balance between them.
  • Quantitative results: Perceptual scores are 3.54 on Obman, 3.50 on HO-3D, and 3.57 on FPHA, indicating similar grasp quality across in-domain and out-of-domain data.The reported scores are close to or sometimes better than ground truth, especially for stability and perceptual score.
  • Ablation studies: Adding the object-centric loss improves object-contact metrics and stability after the hand-centric loss reduces displacement and increases contact metrics.The object-centric loss acts as a regularizer on object contact regions, consistent with its intended design.
  • Ablation studies: The proposed losses outperform modified alternatives, while normalized contact maps outperform direct hand-object distance minimization.The direct-distance variant is dominated by distant point pairs, whereas normalization emphasizes hand vertices near the object surface.
  • ContactNet and TTA: Object local features reduce ContactNet contact-map error by 0.07, a 50% improvement, and online TTA is stronger than offline alternatives.All TTA methods improve results; online updating increases stability while reducing penetration depth.
  • Penetration and displacement: Test-time adaptation moves results toward greater stability without increasing penetration, keeping generated grasps near the desirable low-penetration, low-displacement region.The comparison is evaluated across Obman, HO-3D, and FPHA, with results comparable to or better than ground truth.

5. Conclusion

The paper concludes that consistent hand and object contact modeling improves human grasp generation. Its two-network framework supports both training-time grasp refinement and test-time adaptation, yielding natural, stable grasps with strong cross-domain generalization.

  • Conclusion: The framework reasons about hand-object contact consistency through separate hand- and object-based training targets that seek mutual agreement.The approach uses contact information from both the generated hand and the object.
  • Conclusion: A grasp-generation network and a contact-map prediction network provide outputs for a self-supervised test-time adaptation task on novel objects.The predicted outputs are required to be consistent during adaptation.
  • Conclusion: The method produces more natural and stable grasps and demonstrates strong generalization to cross-domain test inputs.These are the reported overall outcomes of the proposed framework.

Appendix A: Network Architectures

The appendix describes separate GraspCVAE training and testing architectures, plus ContactNet for predicting object contact maps from hand-object point clouds.

  • Architecture overview: The appendix presents separate tables for GraspCVAE training and test-time architectures and a dedicated table for ContactNet.These are identified as Tables 5, 6, and 7, respectively.
  • GraspCVAE: During training, GraspCVAE uses hand and object point clouds to reconstruct hand parameters and generate a hand mesh.Two PointNet encoders produce hand and object features that feed the CVAE encoder and decoder.
  • GraspCVAE: During testing, GraspCVAE receives only the object point cloud and samples its latent code from a standard Gaussian distribution.The test-time architecture omits the CVAE encoder because the latent code is randomly sampled.
  • ContactNet: ContactNet takes hand-object point clouds as input and regresses an object contact map using global and local object features.Local features maintain point correspondence.

B.1. Datasets

The evaluation follows prior work by using the HO-3D and FPHA datasets to assess generalization, with large FPHA objects excluded.

  • Datasets: The study evaluates generalization using the HO-3D and FPHA datasets following prior work.For FPHA, ground-truth hand meshes are fitted to the provided hand joints.
  • Datasets: The FPHA evaluation excludes huge objects, especially milk bottles.This exclusion follows the protocol adopted from prior work.

B.2. Evaluation Metrics

Evaluation combines perceptual judgments with physical penetration measures, while rejecting single-ground-truth reconstruction error as an appropriate grasp-quality metric.

  • Perceptual score: Perceptual score uses Amazon Mechanical Turk ratings of three views per sample, with each sample rated by three workers on a 1–5 scale.The online evaluation layout is shown in Figure 12.
  • Penetration: Penetration measures hand-object collision using maximum penetration depth and penetration volume.Maximum depth is the largest distance from penetrating hand vertices to the closed object surface.
  • Penetration: Penetration volume is computed from intersecting voxels after voxelizing hand and object meshes at 0.5 cm resolution.The reported volume equals voxel volume multiplied by the number of intersecting voxels.
  • Reconstruction error: Hand reconstruction error is not used because a reasonable grasp can differ substantially from the single ground-truth grasp available for each sample.The paper therefore considers reconstruction error unsuitable for evaluating grasp quality in this setting.

B.3. Experiments

The experiments section introduces additional details and results concerning GraspCVAE training targets and test-time adaptation.

  • Experiments: The experiments section covers GraspCVAE training targets and test-time adaptation.It introduces further experimental details and results for both topics.

B.3.1. GraspCVAE Training Targets

This section reports training-target comparisons and test-time adaptation paradigms for grasp and contact-map models. The proposed training targets outperform prior losses on the Obman test set, while adaptation choices affect stability and generalization.

  • GraspCVAE Training Targets: The proposed training targets significantly outperform losses from [46] when training GraspCVAE on the Obman test set.Table 8 compares the two training-loss settings.
  • TTA Details: During test-time adaptation, each test object is augmented into 32 samples using random translations between −5 and 5 cm for 10 iterations.Scaling and rotation are described as harmful because they alter the object's geometry features.
  • TTA Paradigms: TTA-optm optimizes only the 45-D hand-joint axis-angle rotation tensor because optimizing the 61-D full hand pose is unstable and can worsen results.The comparison concerns offline test-time optimization of hand pose parameters.
  • TTA Paradigms: Injecting noise into ContactNet inputs increases reconstruction error from 0.090 to 0.109, indicating that this noise harms contact-map learning.The noisy model is described as unable to learn to correct the injected noise.
  • TTA Paradigms: TTA-online adapts on HO-3D and FPHA video clips, using smoothly changing object poses to fit the test distribution continuously across incoming samples.The described procedure does not reinitialize network parameters between following test samples.

Appendix C: Additional Results

The appendix provides additional visualizations of generated grasps for in-domain and out-of-domain objects. Results are shown from multiple views, with several examples per object or row depending on the figure.

  • Additional Results: Figures 13 and 14 show generated grasps for in-domain Obman test objects and out-of-domain HO-3D objects, respectively.Each result is arranged in a row and includes the input object, output hand mesh, three views, and contact visualization.
  • Additional Results: Figure 15 presents randomly selected generated grasps for in-domain Obman test objects, with each result shown in three views.The figure focuses on additional in-domain examples.
  • Additional Results: Figure 16 visualizes generated grasps for out-of-domain HO-3D and FPHA objects using three views per result.The visualization includes 8 of 10 HO-3D objects and all 3 FPHA objects, with three results per object in each row.
Loading 2104.03304v1…