PaperlayerView exact revision on arXiv

Source-linked AI summary

ContactPose: A Dataset of Grasps with Object Contact and Hand Pose

Samarth Brahmbhatt, Chengcheng Tang, Christopher D. Twigg, Charles C. Kemp, James Hays

arXiv:2007.09545v1cs.CV

TL;DR

The paper addresses the underexplored problem of modeling hand-object contact, for which paired multimodal datasets are limited. It introduces ContactPose and evaluates representations, heuristics, and learning methods, finding that learning-based geometry methods capture nuanced contact details and richer hand-shape features perform better.

  • Problem

    The paper addresses the underexplored problem of predicting hand-object contact from information such as 3D hand pose and grasp images.

  • Method

    ContactPose pairs hand-object contact maps with 3D hand pose, object pose, and multi-view RGB-D grasp videos without instrumenting hands with sensors.

  • Results

    Learning-based geometry methods capture nuanced contact details missed by heuristic methods, while richer hand-shape features perform better and multi-view inputs improve image-based techniques.

  • Takeaways & Limitations

    ContactPose supports analysis and evaluation of contact modeling for hand models, computer interfaces, social interaction, object manipulation, and safety.

  • Takeaways & Limitations

    Plain visual textures on the 3D-printed objects limit the generalization ability of RGB-based methods, and the capture protocol assumes the hand remains fixed relative to the object.

Abstract

from arXiv · show

Grasping is natural for humans. However, it involves complex hand configurations and soft tissue deformation that can result in complicated regions of contact between the hand and the object. Understanding and modeling this contact can potentially improve hand models, AR/VR experiences, and robotic grasping. Yet, we currently lack datasets of hand-object contact paired with other data modalities, which is crucial for developing and evaluating contact modeling techniques. We introduce ContactPose, the first dataset of hand-object contact paired with hand pose, object pose, and RGB-D images. ContactPose has 2306 unique grasps of 25 household objects grasped with 2 functional intents by 50 participants, and more than 2.9 M RGB-D grasp images. Analysis of ContactPose data reveals interesting relationships between hand pose and contact. We use this data to rigorously evaluate various data representations, heuristics from the literature, and learning methods for contact modeling. Data, code, and trained models are available at https://contactpose.cc.gatech.edu.

1 Introduction

ContactPose addresses the underexplored problem of modeling hand-object contact by pairing ground-truth contact with hand pose, object pose, and RGB-D grasp imagery. It provides a dataset and evaluates contact-modeling representations, heuristics, and learning methods against this ground truth.

  • Dataset and motivation: Accurate contact modeling matters for computer interfaces, social-interaction understanding, object manipulation, safety, grasp dynamics, VR simulations, and soft robotic hands.The paper motivates contact modeling as useful across computer vision, graphics, and robotics applications.
  • Dataset and motivation: Thermal contact capture provides ground-truth contact but constrains ContactPose to plain-textured, static grasps and limits generalization of RGB-based methods.The authors note that depth images and synthetic textures may mitigate the texture limitation, while dynamic grasps remain future work.
  • Dataset and motivation: ContactPose captures hand-object contact alongside 3D hand joints, object pose, and multi-view RGB-D grasp videos.Its data collection avoids instrumenting the hands, helping preserve natural grasping behavior while supporting contact capture and hand-pose reconstruction.
  • Dataset and motivation: 50 participants grasp 25 household objects with 2 functional intents, producing high-quality contact maps and over 2.9 M RGB-D images from 3 viewpoints.The dataset also includes object pose and 3D hand joints for each frame.
  • Dataset and motivation: ContactPose analyzes relationships between hand pose and contact and rigorously evaluates geometric and learning-based contact models against ContactPose ground truth.The analysis reports surprising patterns and confirms common intuitions while comparing representations, heuristics, and network architectures.
  • Dataset and motivation: Existing datasets generally lack contact annotations and may omit hand pose, grasp images, object information, or realistic objects and interactions.These limitations make them unsuitable for developing applicable contact models and evaluating contact predictions comprehensively.

3 The ContactPose Dataset

ContactPose captures hand-object contact, hand pose, object pose, and RGB-D observations without instrumenting participants’ hands. Its pipeline combines thermal contact maps, multi-view RGB-D detections, known object tracking, and robust optimization to estimate object-frame 3D joints.

  • Dataset representation: ContactPose represents contact as a normalized object-surface map and hand pose as object-frame 3D joint locations from multi-view RGB-D video.Known camera calibration and object pose allow the joints to be projected into images.
  • Data capture protocol and equipment: The capture setup uses OptiTrack object tracking, Kinect v2 RGB-D cameras, a thermal camera, 3D-printed objects, and a turntable.The equipment supports simultaneous object tracking, visual observations, and thermal contact capture.
  • Contact and object capture: Thermal images are texture-mapped onto object meshes to produce contact maps, and the selected objects are 3D-printed in blue for visual contrast.The dataset uses objects applicable to both functional intents.
  • Hand pose estimation: 3D hand joints are estimated by aggregating noisy 2D detections across frames and cameras under a hand-object rigidity assumption.A robust Huber distance, RANSAC, and a second pass for inaccurate object poses mitigate noisy detections and tracking failures.
  • Hand pose estimation: MANO hand meshes are fit by optimizing hand pose and shape parameters to minimize distances from ContactPose’s 3D joint annotations.

4 Data Analysis

ContactPose analysis shows that contact is distributed across the hand and varies with grasp intent, while similar hand poses can produce different contact patterns. The dataset also supports automatic, hand-part-specific active-area discovery and reveals greater pose diversity for hand-off grasps.

  • Contact distribution: Thumb, index, and middle fingers are contacted most often, while fingertips, proximal phalanges, and the palm all contribute substantially.All three index-finger phalanges have higher contact probability than the pinky fingertip.
  • Contact distribution: 35.87 cm2 is the average contact area for use grasps, compared with 30.58 cm2 for hand-off grasps.
  • Contact distribution: Non-fingertip regions account for a significant portion of grasp contact, as shown by fingertip-versus-whole-hand area comparisons.
  • Active-area discovery: ContactPose estimates contact probabilities for specific hand parts at object points, enabling automatic segmentation of corresponding active areas.The method uses point–phalange association and thresholds the estimated probabilities.
  • Grasp diversity: Hand-off grasps show more diverse 3D joint locations than use grasps, and their pose variation is more continuous.The average intra-cluster distance for use grasps is 32.5% less than for hand-off grasps.
  • Pose–contact relationship: Grasps with similar hand pose can contact different object or hand regions, producing different contact characteristics.The comparison maximizes contact distance while minimizing hand-pose distance.

5 Contact Modeling Experiments

The contact-modeling experiments evaluate how object shape, hand pose, and grasp images can represent inputs for predicting object-surface contact. They compare multiple hand-pose encodings, image features, and a discretized learning target designed to address blurred predictions and imbalanced contact values.

  • Experimental setup: The experiments predict contact maps from hand pose or RGB grasp images while assuming known object geometry and pose.They focus on data representations, learning algorithms, and evaluation against ground-truth contact.
  • Hand-pose representations: Simple-joints represents each object point with the 21 hand joints in object coordinates, producing 63-dimensional features.For bi-manual grasps, each point uses the hand with the closest joint.
  • Hand-pose representations: Relative-joints augments relative joint vectors with surface normals, yielding 66-dimensional point features.The representation encodes both finger position relative to the object point and surface geometry.
  • Hand-pose representations: Skeleton features use distances and normal dot products for 20 phalanges modeled as line segments, while mesh features use nearest hand-mesh geometry and joint distances.Skeleton features are 40-dimensional and mesh features are 23-dimensional.
  • Image representations: A CNN encoder-decoder extracts dense 40-dimensional features from 256×256 RGB crops for one-view and three-view settings.For three views, the feature extractor is shared and features are transferred to 3D object points using pose and camera calibration.
  • Contact representation and learning: Contact values are discretized into 10 bins and learned with weighted cross entropy after mean squared error produced blurred, saturated predictions.The bin weights account for contact-value occurrence imbalance, and an annealed mean converts classification outputs to [0, 1] estimates.

6 Results

Contact prediction is evaluated on held-out objects and participants using re-balanced AuC. Richer hand-shape features and learning-based point-cloud methods outperform simpler features and heuristics, while multi-view images improve RGB-based results.

  • Evaluation: Re-balanced AuC (%) is evaluated on held-out objects and participants, with higher values indicating better contact prediction.The object split holds out mug, pan, and wine glass; the participant split holds out participants 5, 15, 25, 35, and 45.
  • Hand-pose prediction: Richer hand-shape features perform better than simple joint features across the evaluated contact-prediction methods.The comparison includes simple-joints, skeleton, and mesh representations.
  • Hand-pose prediction: Learning-based mesh point-cloud predictors outperform heuristic methods despite using lower-resolution object point clouds.Skeleton features also perform competitively with mesh-based heuristics despite accessing only 3D joint locations.
  • Image-based prediction: Image-based predictions contain fewer high-frequency details and exhibit depth ambiguity, while multi-view inputs provide a significant accuracy boost.The proximal index finger is incorrectly predicted as contacting the mug from RGB images; depth images may mitigate this ambiguity.
  • Hand-pose prediction: Mesh-PointNet++ captures finer palm, thumb, and lower-palm contact details than skeleton-PointNet++ on an unseen mug.Skeleton-PointNet++ still predicts plausible contact patterns for dropped-out hand parts and captures palm-contact nuances.

7 Conclusion and Future Work

ContactPose provides paired contact, hand pose, object pose, and RGB-D data for functional grasping, supports contact-modeling evaluation, and suggests future work on deformable hand meshes.

  • Conclusion: ContactPose is introduced as a dataset pairing hand-object contact with hand pose, object pose, and RGB-D images for functional grasping.The conclusion identifies it as the first dataset with this combination of modalities.
  • Conclusion: ContactPose analysis finds higher hand-contact concentration on the first three fingers in ‘hand-off’ grasps than in ‘use’ grasps.This is presented as one of the dataset’s surprising contact patterns.
  • Conclusion: Learning-based geometry contact models capture nuanced details that heuristic methods miss.The conclusion frames this result as evidence from ContactPose’s contact ground truth.
  • Future Work: ContactPose data can support development and evaluation of hand-mesh deformation algorithms because existing state-of-the-art models are rigid and do not explicitly reason about hand-object contact.The paper identifies deformable hand mesh modeling as an interesting future research direction.

Supplementary Material

The supplementary material documents data capture, ground-truth and hand-pose evaluation, model fitting, algorithm implementation, and dataset metadata.

  • Supplementary Material: The supplementary material discusses contact capture and evaluates the accuracy of hand-pose and contact ground truth.It also covers MANO hand-mesh fitting details, network architectures, and learning-algorithm implementation.
  • Supplementary Material: It provides object lists, ‘use’ instructions, and participant hand information included in ContactPose.The extended supplementary material is available through the ContactPose website.

A Contact Capture Discussion

Thermal readings are converted into continuous contact values between 0 and 1 through a calibrated sigmoid mapping of object temperature.

  • Contact value conversion: Raw thermal readings are mapped to continuous contact values in [0, 1] using a sigmoid.The warmest point maps to 0.95 and the coldest point maps to 0.05; the range approximately corresponds to room-to-body temperature.
  • Contact value conversion: The continuous values non-linearly encode object temperature and can support a hard contact-status decision when needed.Most experiments use the continuous representation rather than a hard decision.

B MANO Fitting

ContactPose data are fitted with the MANO hand model by optimizing shape and pose parameters to match annotated 3D joints while regularizing hand shape.

  • MANO represents the hand mesh and 3D joint locations as functions of shape parameters β and pose parameters θ.The model is modified with one additional joint at each fingertip to match ContactPose annotations.
  • The fitting objective combines L2 distance between predicted and annotated 3D joints with shape-parameter regularization.
  • σ is set to 10, and optimization uses the Dogleg optimizer after initializing β and θ at the mean shape and pose.Initialization follows 6-DOF wrist and five-palm-joint alignment.

C Dataset Accuracy

ContactPose accuracy is assessed by comparing thermal contact with MANO geometry and pressure sensing, while participant hand measurements and model specifications document the dataset’s fidelity and coverage.

  • Contact accuracy: 95.4% binary contact agreement with pressure images is achieved at a thermal contact threshold of 0.4.Agreement is evaluated against Sensel Morph pressure data, and the maximum occurs at threshold 0.4.
  • Contact accuracy: 4.17 mm and 4.06 mm are the average distances from contacted object points to their nearest MANO hand point using 10 and 15 pose parameters, respectively.
  • Dataset coverage and models: ContactPose includes participant palm contact maps, seven-gesture RGB-D videos, and hand-contact modeling specifications spanning point-cloud and image-based architectures.The supplementary figures show palm maps, participant gestures, and the image encoder-decoder architecture; PointNet++ outputs ten contact-value classes.

F Training and Evaluation Details

Training uses augmentation and controlled frame sampling, while evaluation separates object and participant generalization and documents the dataset’s object inventory.

  • Training: Point clouds and voxel grids are rotated around the up-axis at regularly spaced 30° intervals during training, with predictions averaged across rotations at evaluation.
  • Training and evaluation: Image-based models train on 50 equally spaced frames per grasp and evaluate on 12 equally spaced frames from those 50.The sampling reduces temporal redundancy from approximately 300 correlated RGB-D frames per grasp across three Kinects.
  • Dataset inventory: The dataset documentation lists all 25 objects together with their functional grasping categories and specific use instructions.
Loading 2007.09545v1…