Source-linked AI summary
Reconstructing Hand-Object Interactions in the Wild
Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, Jitendra Malik
TL;DR
Single-image hand-object reconstruction must resolve 3D depth and scale ambiguity that can produce distant or interpenetrating configurations. RHO uses 2D cues, 3D contact priors, and joint optimization, achieving improvements over existing methods and supporting a new in-the-wild dataset.
Problem
Single-image inputs create depth and scale ambiguity, making hand-object configurations potentially distant or interpenetrating in 3D.
Method
RHO reconstructs hand-object interactions by combining 2D image cues and 3D contact priors through staged estimation, joint 3D optimization, and pose refinement.
Results
RHO achieves quantitative and qualitative improvements over existing methods, including lower HO3D hand joint error (9.7 mm vs. 14.7 mm) and object Chamfer distance (19.9 vs. 26.8).
Takeaways & Limitations
The method reconstructs a wider variety of objects in the wild and contributes the MOW dataset with 500 images spanning 121 object categories.
Takeaways & Limitations
Reconstructions can remain imperfect across viewpoints because of ambiguity, so MOW annotation includes iterative human verification.
Abstract
from arXiv · showhide
In this work we explore reconstructing hand-object interactions in the wild. The core challenge of this problem is the lack of appropriate 3D labeled data. To overcome this issue, we propose an optimization-based procedure which does not require direct 3D supervision. The general strategy we adopt is to exploit all available related data (2D bounding boxes, 2D hand keypoints, 2D instance masks, 3D object models, 3D in-the-lab MoCap) to provide constraints for the 3D reconstruction. Rather than optimizing the hand and object individually, we optimize them jointly which allows us to impose additional constraints based on hand-object contact, collision, and occlusion. Our method produces compelling reconstructions on the challenging in-the-wild data from the EPIC Kitchens and the 100 Days of Hands datasets, across a range of object categories. Quantitatively, we demonstrate that our approach compares favorably to existing approaches in the lab settings where ground truth 3D annotations are available.
1. Introduction
The paper addresses the reality gap and missing 3D annotations that make wild hand-object interactions difficult to reconstruct. It introduces RHO and MOW to reconstruct and annotate diverse interactions in the wild.
- Understanding hand-object interactions requires 3D reconstruction, but heavy occlusion, small daily objects, and complex contacts make single-image inference difficult.
- Existing 3D-labeled datasets focus mainly on laboratory settings, creating a large reality gap with diverse in-the-wild environments and interactions.
- RHO reconstructs 3D hands and objects from single in-the-wild images using 2D image cues and 3D contact priors.
- RHO combines hand keypoints, object masks and depth, joint 3D hand-object optimization, and contact-prior-based pose refinement.
- RHO recovers an order of magnitude more object variety than previous hand-object or human-object reconstruction work.
- The dataset supports evaluation on challenging in-the-wild data and analysis of human manipulation from images.
- MOW contains 500 in-the-wild images spanning 121 object categories, with annotations for instance category, 3D models, hand pose, and object pose.
2. Related Work
Prior work estimates 3D hands and objects using prediction or fitting, but existing hand-object datasets and interaction methods remain constrained by laboratory capture, limited categories, or extra inputs. RHO addresses these constraints with joint optimization, contact priors, and object-mask/depth fitting for diverse in-the-wild reconstruction.
- 3D hand pose estimation: Earlier hand-pose methods either predict 3D joints or fit parametric hand models to 2D keypoints.
- RHO method: RHO combines 2D keypoint fitting, differentiable rendering, joint optimization, and contact-prior refinement into a four-step reconstruction procedure.
- 3D object pose estimation: Object-pose methods either predict shape and pose with category-limited networks or estimate 6 DOF pose from known 3D models.
- 3D hand-object datasets: Existing hand-object datasets rely on manual annotation, depth tracking, motion capture, simulation, or multi-view capture, generally restricting collection to laboratory settings.
- Optimizing 3D interactions: RHO jointly reconstructs hand-object configurations and uses 3D MoCap contact priors without requiring manually labeled interaction vertices or other extra input.
3. Method
RHO reconstructs hand-object interactions through staged estimation and joint optimization, using image cues, differentiable rendering, distance and collision constraints, and learned contact priors. The procedure progresses from separate hand and object estimates to refined, physically more plausible interactions.
- Overview: RHO estimates hand pose, object pose, their joint 3D configuration, and final pose refinement using contact priors.Intermediate results visualize the separate estimates, joint optimization, and refinement stages.
- 3.1. Hand Pose Estimation: The hand stage initializes a MANO mesh from a single RGB image and fits its pose to detected 2D hand keypoints.A weak-perspective camera and hand-shape regularization are used during fitting.
- 3.2. Object Pose Estimation: The object stage optimizes scale, rotation, and translation by differentiably rendering a known 3D model and matching its mask and depth maps.Randomly initialized poses are optimized in the object-centered image region, and the lowest-loss result is selected.
- 3.2. Object Pose Estimation: For diverse unknown object categories, instance masks are obtained from recognition models by selecting the predicted instance with highest overlap with the detected hand box.This strategy supports masks for more than 100 daily object categories.
- 3.3. Joint Optimization: Joint optimization addresses depth and scale ambiguity by minimizing hand-object distance while penalizing mesh interpenetration.Chamfer distance brings the interaction pair closer, while a hand-mesh signed distance field penalizes object vertices inside the hand.
- 3.4. Pose Refinement: Pose refinement learns contact priors from 3D hand MoCap data and iteratively updates hand parameters using hand-object distance fields.The refinement stage produces more realistic interactions than the preceding joint-optimization result.
4. Method Evaluation
RHO is evaluated quantitatively in laboratory datasets and qualitatively on in-the-wild images. Its joint losses improve hand-object interaction geometry, while comparisons show stronger generalization to diverse wild objects.
- Evaluation setup: RHO is evaluated quantitatively on HO3D and FPHA in the lab, and qualitatively on in-the-wild images.The comparison uses monocular RGB input with known 3D object models and includes ablations of loss terms and refinement.
- Quantitative comparison: In HO3D, RHO reduces hand joint error to 9.7 mm versus 14.7 mm and object Chamfer distance to 19.9 versus 26.8.Errors are measured using mean average error over 21 hand joints and object Chamfer distance.
- Quantitative comparison: In FPHA, RHO achieves lower hand joint error, 14.2 mm versus 18.0 mm, but slightly higher object error than [14].The comparison is affected by [14] using an action split with the same objects represented across training and testing.
- Qualitative comparison in the wild: Qualitative comparisons show an advantage over [14] on in-the-wild images, where limited lab-category training makes generalization to unseen objects difficult.The cited comparison contrasts good laboratory performance with weaker performance in the wild.
- Ablation studies: Depth and collision losses reduce collision to 7.7 at 76.4 mm distance, while refinement further reduces collision to 6.5 and distance to 75.8 mm.The refinement stage makes only small adjustments to the final result.
5. Dataset
MOW is a semi-automatic 3D dataset built from in-the-wild interaction images, with human verification and annotations spanning varied objects and grasp types. Its evaluations indicate good annotation quality and broad dataset diversity.
- Dataset construction: The dataset uses static frames from EPIC Kitchens and 100 Days of Hands, covering varied objects, people, viewpoints, and interaction scenarios.Temporal information is not used when selecting source images.
- Dataset construction: Each annotation proceeds through 3D model selection, RHO reconstruction, and verification of the resulting hand-object configuration.The reconstruction step is semi-automatic and may require annotators to select loss weights.
- Dataset construction: Verification handles ambiguous or imperfect reconstructions by accepting good results, iterating promising cases, or removing unlikely cases.Annotators inspect outputs across viewpoints and iterate with reconstruction until reaching reasonable quality.
- Dataset outputs: For each image, MOW provides a 3D object model, 3D object pose, and 3D hand pose, from which amodal masks and contact maps can also be derived.Example annotations include 3D meshes, amodal masks, and contact maps.
- Dataset evaluation: Amodal masks show high overlap with human-labeled ground truth, with higher overlap for larger objects.The evaluation uses mean IoU for hands and objects across different object sizes.
- Dataset evaluation: User evaluation gives the annotations an average quality score of 4.16 out of 5, and the selected 3D model matches the true object in 92% of cases.Most mismatches are attributed to imprecise mesh topology, such as fitting a cylinder to a handled mug.
- Object variety: MOW contains 121 object categories with a long-tailed distribution and includes many interaction examples across diverse daily objects.The dataset was constructed from 500 images.
- Grasp variety: The grasp analysis suggests varied grasp types within object categories rather than only iconic grasps, with similar grasps across objects and different grasps for the same object.The 2D Isomap embedding shows a pen cluster but no other clear clusters.
6. Conclusion
The paper concludes that RHO reconstructs in-the-wild hand-object interactions using 2D image cues and 3D contact priors, while MOW provides a corresponding 3D dataset. The work also supplies mechanisms for handling unlabeled object categories and depth constraints.
- Conclusion: RHO reconstructs hand-object interactions using 2D image cues and 3D contact priors.The approach is presented as a technique for reconstructing interactions in the wild.
- Conclusion: MOW is constructed by combining the reconstruction technique with human intervention to create a new in-the-wild 3D hand-object interaction dataset.The conclusion identifies the dataset as a central outcome of the work.
- Unlabeled object categories: Models trained on labeled datasets can provide reasonable masks for unlabeled object categories even when category predictions are incorrect.This insight supports reconstructing categories absent from existing image segmentation datasets.
- Depth loss: The depth loss supplies object shape constraints that improve reconstruction results.The depth-loss effect is illustrated through predicted masks, depth maps, and object pose results with and without the loss.
Appendix
The appendix supplements the main paper with details about object masks, depth loss, implementation, failure cases, and additional qualitative examples.
- Appendix: The appendix provides further information on 2D object masks, depth loss, implementation details, failure cases, and qualitative examples.These materials supplement the results presented in the main text.
Object Masks
Recognition models can produce useful masks for object categories absent from their labels, supporting reconstruction across diverse objects.
- Predicted masks remain reasonable even when the model assigns an incorrect object category.For example, the model segments a spatula despite not recognizing its category name.
Depth Loss
The depth loss addresses pose ambiguity that remains when object pose estimation relies only on 2D masks.
- 2D mask loss alone can leave multiple object poses plausible because it does not capture geometry.The ambiguity is illustrated by different poses producing similar masks.
- The added depth loss compares rendered object depth with an estimated depth map to resolve pose ambiguity.The final result shows more accurate estimation after introducing this loss.
Implementation Details
Implementation uses simplified object meshes, many rotation initializations, confidence-weighted hand keypoints, and specified joint-optimization loss weights.
- Object models are preprocessed to approximately 800 faces to speed up optimization.
- Object rotation uses 1000 uniformly sampled initializations over [−π, π], with zero-initialized translation.The pose with the lowest loss is selected.
- Hand keypoint loss is weighted by each predicted joint’s confidence score.
- Joint optimization uses default loss weights beginning with λ1 = 200 and λ2, as specified in the implementation.
Failure Cases
Failures arise in hand estimation, object pose estimation, and joint optimization, including occlusion, mask ambiguity, imperfect models, and undesirable local minima.
- Hand pose estimation fails when the hand is highly occluded or cropped by the image.
- Object pose errors result from ambiguous 2D masks and imperfect object models.
- Joint optimization can converge to an undesirable local minimum.
- Additional qualitative results show strong performance across a range of interactions and objects.