Source-linked AI summary
ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging
Samarth Brahmbhatt, Cusuh Ham, Charles C. Kemp, James Hays
TL;DR
ContactDB addresses the limited ability of external sensors to observe hand-object contact by building a thermal-imaging dataset of functional human grasps. It records contact maps on 3D object meshes, analyzes how intent and size shape contact, and trains models to predict diverse patterns from object shape. The dataset reveals substantial non-fingertip contact and supports future contact-aware grasping research.
Problem
External visual sensing often cannot observe hand-object contact because contact regions are occluded, leaving fundamental aspects of grasping insufficiently characterized.
Method
The authors collect functional grasps of 3D-printed household objects with calibrated RGB-D and thermal imaging, then texture-map multiview thermal images onto object meshes and train shape-to-contact predictors.
Results
ContactDB reveals that functional intent and object size influence grasping, active areas are selectively touched or avoided, and palm and proximal-finger contact is frequent.
Takeaways & Limitations
Object-centric contact maps support detailed grasp analysis, diverse contact prediction, and contact-informed design of soft robotic manipulators.
Takeaways & Limitations
Contact-map accuracy varies across objects and sessions because calibration, printing, pose estimation, and texture-mapping errors can affect the process.
Abstract
from arXiv · showhide
Grasping and manipulating objects is an important human skill. Since hand-object contact is fundamental to grasping, capturing it can lead to important insights. However, observing contact through external sensors is challenging because of occlusion and the complexity of the human hand. We present ContactDB, a novel dataset of contact maps for household objects that captures the rich hand-object contact that occurs during grasping, enabled by use of a thermal camera. Participants in our study grasped 3D printed objects with a post-grasp functional intent. ContactDB includes 3750 3D meshes of 50 household objects textured with contact maps and 375K frames of synchronized RGB-D+thermal images. To the best of our knowledge, this is the first large-scale dataset that records detailed contact maps for human grasps. Analysis of this data shows the influence of functional intent and object size on grasping, the tendency to touch/avoid 'active areas', and the high frequency of palm and proximal finger contact. Finally, we train state-of-the-art image translation and 3D convolution algorithms to predict diverse contact patterns from object shape. Data, code and models are available at https://contactdb.cc.gatech.edu.
1. Introduction
ContactDB addresses the difficulty of observing hand-object contact by recording detailed, object-centric contact maps during functional human grasps. The dataset supports analysis of grasping preferences and prediction of contact from object shape.
- Motivation and approach: Post-grasp intent changes grasp behavior, such as grasping a knife by its handle for use but by its blunt blade side for handing off.The example illustrates why contact should be studied together with the intended action.
- Dataset contribution: ContactDB represents hand-object contact as textures on 3D object meshes, rather than primarily recording hand configuration.This object-centric representation is called a contact map.
- Motivation and approach: Object-centric contact maps enable analysis of grasp preferences by functional intent, shape, size, and semantic category.They also support learning object-shape features for grasp prediction and grasp retargeting to diverse hand models.
- Motivation and approach: Thermal imaging captures contact regions that are typically occluded in visual-light images, using a calibrated RGB-D and thermal camera rig.The procedure directly observes contact on the object surface at unprecedented detail.
- Dataset contribution: 3750 meshes and 375K paired RGBD-thermal frames record functional human grasping across household objects.Participants grasped objects with post-grasp intents involving use and handing them off.
2. Related Work
Prior grasp datasets commonly record hand configuration or related indirect signals, while ContactDB focuses on directly observed hand-object contact. The paper also distinguishes real functional grasps from preference annotations and frames grasp prediction as a diverse-output problem.
- Datasets of Human Grasps: Earlier grasp studies used data gloves, magnetic trackers, manual annotation, or arranged robotic hands to record grasping activity.These approaches primarily provide hand joint configuration or other indirect measurements.
- Datasets of Human Grasps: ContactDB directly observes where the hand contacted the object, addressing the coarse or speculative contact estimates of earlier approaches.The authors describe this observation as having unprecedented fidelity.
- Datasets of Human Grasps: Unlike crowdsourced tactile saliency, ContactDB records full observations of real human grasps performed with functional intent.Crowdsourced saliency instead aggregates pairwise preferences over sampled object-surface points.
- Predicting Grasp Contact: Grasping permits multiple equally correct outputs, motivating prediction methods that generate diverse and meaningful grasp configurations or contact patterns.Prior work developed theoretical frameworks for diverse neural predictions and applied similar techniques to parallel-jaw grasp configurations.
3. The ContactDB Dataset
ContactDB combines thermal sensing, controlled 3D-printed objects, functional grasp protocols, and multiview texture mapping to create object-surface contact maps. Its processing registers nine RGB-D and thermal views with reconstructed object meshes.
- Data collection: ContactDB contains 50 3D-printed household objects grasped by 50 participants with two post-grasp functional intents.The dataset design centers on observing contact through thermal imaging.
- Thermal contact sensing: Hand heat transfers to the object surface, allowing contact regions to remain visible in thermal images after release when the material dissipates heat slowly.Thermal intensity also relates to skin heat, contact duration, conduction, and pressure.
- Object selection: The object set emphasizes household interaction and includes primitive shapes at three scales to study how size influences grasping.Excluded categories included deformable, very small, and very large objects.
- Data collection: Objects were held for 5 seconds, then placed on a turntable while cameras recorded multiview RGB-D and thermal data.Participants used or handed off objects, and an insulating glove prevented experimenter heat transfer.
- Data processing: Nine turntable views were converted into point clouds, segmented, and aligned with ICP to estimate each object’s full 6D pose.The resulting poses and thermal images were used for mesh texture mapping.
- Data processing: A colormap optimization algorithm locally adjusts object poses to reduce photometric texture-projection error and generate coherent contact-map textures.The process combines the 3D mesh, nine pose estimates, and thermal images.
4. Analysis of Contact Maps
ContactDB analysis shows that grasp contact varies with functional intent and object size, while human grasps frequently involve substantial palm and non-fingertip contact.
- Effect of Functional Intent: Functional intent produces distinct dominant contact patterns for many objects, with active areas quantifying how often participants touched selected surface regions.Active-area contact is evidenced by map values greater than 0.4.
- Effect of Object Size: Small objects are commonly grasped with two or three fingertips, whereas larger objects involve more fingers and additional object contact.For large objects, grasps are bi-modal: bimanual full-hand grasps or single-handed fingertip grasps.
- Effect of Object Size: People with smaller hands prefer bimanual grasps for large objects when handing them off, while no bimanual grasps were observed for medium or small objects.The large objects considered were cube, cylinder, pyramid, and sphere.
- How much of the contact is fingertips?: Total contact area for many objects exceeds the upper bound for fingertip-only contact, indicating substantial use of palm and other soft-tissue regions.The fingertip-only upper bound was estimated from annotated fingertip regions on a palm print and doubled for observed bimanual grasps.
- How much of the contact is fingertips?: Average contact area differs across functional intents for some objects, including bowls, mugs, PS controllers, and toothbrushes.These differences reflect different kinds of grasps used for those objects.
5. Predicting Contact Maps
ContactDB frames contact-map prediction as a one-to-many problem influenced by functional intent, using single-view RGB-D and full 3D shape representations. Experiments compare image translation, point-cloud, voxel-grid, and diverse prediction strategies on held-out and unseen objects.
- Contact patterns are significantly influenced by functional intent, so separate models are trained for ‘hand-off’ and ‘use’.
- Single-view Prediction: Single-view prediction uses RGB-D input to generate a 2D contact map for the visible object surface.The representation is suited to robotics scenarios requiring grasping after observation from one view, but unseen object regions remain unavailable.
- Single-view Prediction: Unseen-object predictions vary with intent: mugs favor the handle for use and the top for handoff, while pans and wine glasses show analogous intent-specific patterns.
- 3D Prediction: Full 3D prediction addresses view-consistency issues, while DiverseNet generates multiple contact maps from one network by varying a control variable.sMCL instead uses an ensemble of predictors; PointNet and VoxNet represent shapes as point clouds and voxel occupancy grids, respectively.
- 3D Prediction: The voxel occupancy-grid representation performs better for this task, and single-prediction models do not capture ContactDB’s contact complexity.Predictions for unseen classes and shapes include body, handle, stem-intersection, and occasional bimanual contacts.
6. Conclusion and Future Work
The paper presents ContactDB as a large-scale functional-grasping contact-map dataset, analyzes grasp behavior, and explores prediction from object shape. It identifies future applications in soft-manipulator design and hand-pose assistance.
- ContactDB combines a large-scale functional-grasping dataset with analysis of grasp behavior and prediction strategies based on object shape.
- Future Work: Contact patterns could inform soft robotic manipulators by targeting object regions touched by humans.
- Future Work: Contact maps may help recover or assist prediction of hand pose in functional grasping.The paper presents this as an exciting problem for future research.
Supplementary Material
The supplementary material compares ContactDB with crowdsourced contact saliency, examines thermal dissipation, documents texture-mapping error sources, and lists the dataset’s objects and use instructions.
- Comparison with Crowdsourced Maps: Crowdsourced saliency maps lack clear finger marks and resemble averaged contact maps, whereas ContactDB records more detailed contact patterns.The supplementary comparison notes that crowdsourcing relies on self-reporting and gives a wine-glass example.
- Heat Dissipation During Data Collection: Thermal prints take more than 35 s to diffuse significantly, while scanning a 360° rotation takes 18 s.Comparable prints without strongly blurred edges support minimal heat-dissipation artifacts during the scan.
- Accuracy of Texture Mapping: Texture mapping combines thermal images from 9 views with object-pose estimates to produce a contact-textured mesh.
- Accuracy of Texture Mapping: The texture-mapping pipeline has multiple error sources, including camera calibration, 3D printing, depth-based pose estimation, and algorithmic artifacts.A heated-point test measured 4.4 mm geometric error for one instance, and accuracy can vary across objects and sessions.
- Dataset Inventory: The supplementary material lists all 50 ContactDB objects and identifies their functional grasping categories and specific ‘use’ instructions.