Source-linked AI summary
ContactOpt: Optimizing Contact to Improve Grasps
Patrick Grady, Chengcheng Tang, Christopher D. Twigg, Minh Vo, Samarth Brahmbhatt, Charles C. Kemp
TL;DR
Image-based hand poses can yield unrealistic hand-object contact because small pose errors matter physically and perceptually. ContactOpt predicts desirable contact from hand and object meshes, then refines hand pose with a differentiable contact model that tolerates limited interpenetration. Across evaluations, optimized grasps better matched ground truth contact, had lower kinematic error, and were preferred by human participants.
Problem
Image-based hand pose errors can create unrealistic hand-object contact, and submillimeter fingertip changes can affect whether an object is held or dropped.
Method
ContactOpt uses DeepContact to predict likely contact, DiffContact to model current contact, and gradient-based optimization to refine hand pose.
Results
Optimized grasps better matched ground truth contact, had lower kinematic error, and were preferred by human evaluators across dataset-quality and image-based pose refinements.
Takeaways & Limitations
Contact-aware mesh postprocessing can improve hand poses from existing image-based estimation methods, including on a novel object set.
Takeaways & Limitations
Multiple valid grasp modes can prevent recovery of the correct grasp from an inaccurate initial pose, leaving some kinematic error.
Abstract
from arXiv · showhide
Physical contact between hands and objects plays a critical role in human grasps. We show that optimizing the pose of a hand to achieve expected contact with an object can improve hand poses inferred via image-based methods. Given a hand mesh and an object mesh, a deep model trained on ground truth contact data infers desirable contact across the surfaces of the meshes. Then, ContactOpt efficiently optimizes the pose of the hand to achieve desirable contact using a differentiable contact model. Notably, our contact model encourages mesh interpenetration to approximate deformable soft tissue in the hand. In our evaluations, our methods result in grasps that better match ground truth contact, have lower kinematic error, and are significantly preferred by human participants. Code and models are available online.
1. Introduction
ContactOpt addresses unrealistic hand-object contact caused by pose-estimation errors by predicting desirable contact and refining hand pose. Its contact-aware optimization improves realism, kinematic accuracy, and agreement with observed contact.
- Motivation: Image-based pose errors can produce unrealistic contact, while submillimeter fingertip changes may determine whether an object is held or dropped.Small-scale contact errors also reduce the perceptual realism of rendered poses.
- Method: ContactOpt predicts likely contact from hand and object meshes, then optimizes hand pose to match that target.DeepContact estimates target contact, DiffContact computes current geometric contact, and gradient-based optimization refines MANO pose, translation, and rotation.
- Method: Up to 2 mm of hand-object interpenetration is permitted without penalty to approximate deformable soft tissue and encourage contact over wider hand areas.The hand’s finger pads can deform 2–3 mm and the palm 5 mm under normal grasping forces.
- Evaluation: Refined poses better matched ground truth contact and were preferred by human participants in evaluations using measured-contact data.The first evaluation refined pose estimates from ContactPose with respect to ground truth thermal contact.
- Evaluation: ContactOpt’s refined image-based grasps had lower kinematic error, stronger agreement with observed contact patterns, and higher human preference than initial estimates and RefineNet.The evaluation used an RGB pose-estimation network on HO-3D and tested ContactOpt on a novel object set.
- Method: DeepContact estimates likely contact across inaccurately aligned hand and object surfaces, while DiffContact provides a differentiable model of current contact.These components support contact-aware pose refinement rather than relying only on image-based pose estimates.
2. Related Work
Related work spans hand-object pose estimation, contact datasets, grasp synthesis, and contact-aware full-body pose methods. ContactOpt builds on these directions by using fine-grained contact to refine human hand poses from mesh inputs.
- Datasets of hand-object contact: ContactPose uniquely combines hand-object pose with ground truth thermal contact maps measured during static grasps.Its contact measurements show contact across large hand regions, not only at fingertips.
- Datasets of hand-object contact: ContactPose’s 3D hand-pose accuracy is bounded by the hand-pose estimator, which can create discrepancies between contact maps and the MANO mesh.This limitation motivates methods that use contact to improve pose alignment.
- Image-based hand-object pose estimation: Hand-object interaction research uses gloves, depth or RGB-D sensors, and RGB images to estimate hand and object pose.Image-based methods may predict hand and object representations jointly yet still produce relative pose errors and unrealistic grasps.
- Grasp synthesis: Robotic grasp-generation methods commonly seek stable grasps using depth or RGB input, physics simulation, or analytical heuristics.Much robotic grasping work focuses on simple grippers with sparse contact points.
- Grasp synthesis: Human-grasp synthesis includes affordance-based generation and methods such as ContactGrasp and GRAB.ContactGrasp uses pre-recorded dense contact maps but lacks ground truth hand poses, limiting comparison with ground truth and conditioning on images.
- Contact in human pose: Contact constraints have also been used to improve full-body pose estimation and generate human poses interacting with scenes.ContactOpt extends this emphasis to fine-grained hand-object contact.
3. Methods
ContactOpt refines an estimated MANO hand pose by predicting target hand-object contact and optimizing pose parameters so differentiable contact matches that target. Its contact model uses virtual capsules and controlled interpenetration to represent broad, deformable tissue contact, while optimization minimizes contact mismatch and excessive penetration.
- ContactOpt overview: ContactOpt starts from a noisy grasp parameterization and seeks a better hand pose by exploiting predicted or ground-truth hand-object contact.The grasp uses an object mesh and a MANO hand mesh with pose, shape, translation, and rotation parameters.
- DeepContact: Learning to Estimate Contact: DeepContact predicts target contact maps for the hand and object from their meshes, using point-cloud representations of 2048 object points and all 778 MANO vertices.It is trained on randomly perturbed ContactPose grasps, producing a 22K / 1.4K training/testing split.
- DiffContact: Differentiable Contact Model: DiffContact computes differentiable contact from the current hand and object geometry, enabling gradients for optimizing the hand parameters.Its contact maps have gradual dropoffs and diffuse edges, producing area-based contact rather than a single point.
- DiffContact: Differentiable Contact Model: Virtual capsules are placed at object vertices along surface normals, marking nearby hand vertices as contact and allowing attraction beyond the object surface.The capsule design approximates deformable hand tissue and supports broad contact regions.
4. Evaluation
ContactOpt is evaluated on millimeter-scale annotation errors and large inaccuracies from perturbed or image-based poses. Refinement improves contact agreement, kinematic accuracy, and perceptual grasp quality, while performance remains limited when multiple grasp modes make the correct location ambiguous.
- Evaluation setup: ContactOpt is evaluated on ContactPose and HO-3D poses with small and large inaccuracies, using contact, kinematic, and perceptual metrics.The evaluation includes intersection volume, MPJPE, contact coverage, precision/recall, and human perceptual judgments.
- Refining small inaccuracies: Both contact precision and recall increase after refining ContactPose poses, reducing unwanted and excess contact.The refinement improves self-consistency between mesh poses and measured thermal contact maps.
- Refining small inaccuracies: Participants favored refined ContactPose grasps at over a 2:1 ratio in a perceptual evaluation of natural-looking grasps.The contact maps were not shown to participants; refinement addressed millimetric penetration and undershoot.
- Refining large inaccuracies: Almost 70% lower MPJPE was achieved from heavily misaligned Perturbed ContactPose inputs, alongside improved perceptual grasp quality and contact-map consistency.Ground-truth contact maps were not provided to the algorithm during refinement.
- Refining image-based pose estimates: DeepContact transfers from Perturbed ContactPose to HO-3D despite different objects and dynamic grasps, but multiple valid grasp modes can make the correct location difficult to predict.The paper attributes the smaller geometry-domain gap to consistency of hand and object geometry across datasets.
- Refining image-based pose estimates: Human evaluators favored refined image-based grasps by almost 6:1, whose hand-contact frequencies resembled ground-truth patterns more closely than original estimates.The image-based evaluation used an RGB pose-estimation network on HO-3D, with object class and pose taken from ground truth.
5. Conclusion
ContactOpt refines coarsely aligned hand-object meshes by comparing learned target contact with contact estimated from the current geometry. The experiments show improved contact and pose quality, including on image-based estimates and novel object sets.
- 5. Conclusion: ContactOpt refines coarsely aligned hand and object meshes using DeepContact target contact and DiffContact current contact.The difference between the two contact estimates guides hand-pose optimization.
- 5. Conclusion: Optimized grasps improve dataset-quality meshes and image-based pose estimates, including tests on a novel object set.The reported improvements cover cases with measured thermal contact and image-derived poses.
- 5. Conclusion: Optimized grasps achieve lower kinematic error and are preferred by human evaluators.These outcomes are reported as the main experimental results of the method.