PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning joint reconstruction of hands and manipulated objects

Yana Hasson, Gül Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, Cordelia Schmid

arXiv:1904.05767v1cs.CV

TL;DR

Jointly reconstructing hands and manipulated objects from RGB images is difficult because occlusion and interaction must be handled together. The paper introduces an end-to-end model with a contact loss and trains it with the large-scale ObMan synthetic dataset, demonstrating transfer to real images.

  • Problem

    Joint hand-object reconstruction remains difficult because manipulation causes substantial occlusion, while existing interaction datasets are small or incompletely annotated.

  • Method

    The paper jointly predicts hand and object meshes with a differentiable MANO model, relative object transformation, and a contact loss penalizing penetration while encouraging contact.

  • Results

    The model improves grasp quality metrics over baselines using RGB images, and ObMan-trained models transfer to real images.

  • Takeaways & Limitations

    ObMan and the proposed physically constrained reconstruction approach support research on modeling object manipulations from visual data.

  • Takeaways & Limitations

    Object annotations in the FHB dataset contain substantial errors, including 11.0mm average hand-skeleton penetration, limiting strong conclusions from object reconstruction metric fluctuations.

Abstract

from arXiv · show

Estimating hand-object manipulations is essential for interpreting and imitating human actions. Previous work has made significant progress towards reconstruction of hand poses and object shapes in isolation. Yet, reconstructing hands and objects during manipulation is a more challenging task due to significant occlusions of both the hand and object. While presenting challenges, manipulations may also simplify the problem since the physics of contact restricts the space of valid hand-object configurations. For example, during manipulation, the hand and object should be in contact but not interpenetrate. In this work, we regularize the joint reconstruction of hands and objects with manipulation constraints. We present an end-to-end learnable model that exploits a novel contact loss that favors physically plausible hand-object constellations. Our approach improves grasp quality metrics over baselines, using RGB images as input. To train and evaluate the model, we also propose a new large-scale synthetic dataset, ObMan, with hand-object manipulations. We demonstrate the transferability of ObMan-trained models to real data.

1. Introduction

The paper addresses joint hand-object reconstruction under occlusion by exploiting physical manipulation constraints. It introduces an end-to-end model, a contact loss, and the ObMan synthetic dataset for training and evaluation.

  • 1. Introduction: The model jointly reconstructs hand and object meshes from a monocular RGB image in a single forward pass.The framework is end-to-end and predicts both meshes together.
  • 1. Introduction: Joint reconstruction is challenging because hand-object interactions create significant occlusions, while contact physics restricts valid configurations.Stable grasps require surface contact, and solid objects prohibit penetration.
  • 1. Introduction: The proposed contact loss penalizes hand-object interpenetration and encourages contact between the hand and manipulated object.Its repulsion and attraction terms encode complementary physical constraints.
  • 1. Introduction: Real hand-object datasets are difficult to obtain at scale, and existing datasets are too small or provide only partial 3D annotations.These limitations make them unsuitable or less suitable for training deep neural networks.
  • 1. Introduction: ObMan provides large-scale synthetic hand-object manipulations using 2.7K everyday object models from 8 object categories.Automatic grasp generation supplies diversity and ground-truth data for training and ablation studies.
  • 1. Introduction: The paper contributes an end-to-end RGB reconstruction model, a novel contact loss, and the publicly available ObMan dataset, models, and code.The method is built around a differentiable MANO hand model.

2. Related work

Prior work commonly reconstructs hand pose or object shape separately, often using sparse joints, canonical object coordinates, or limited interaction data. This paper instead combines dense hand meshes, generic object reconstruction, and differentiable physical constraints in an end-to-end architecture.

  • Hand pose estimation: RGB hand-pose methods often predict sparse 3D joints, whereas this work outputs dense hand meshes for inferring hand-object interactions.Some prior methods obtain meshes through post-processing or fitting rather than direct end-to-end prediction.
  • Hand pose estimation: Existing synthetic hand datasets with objects often treat objects as occluders and are difficult to scale to many object instances.One cited approach renders hand poses while grasping 7 objects in a merged-reality environment.
  • Object reconstruction: Object-reconstruction methods use representations such as voxels, point clouds, or meshes, but typically predict category-specific objects in normalized canonical coordinates.This work uses meshes and a view-centered variant to handle generic object categories in images containing hands.
  • Hand-object reconstruction: Prior hand-object methods use proximity metrics, direct contact regression, instrumented measurements, or physical constraints for contact and penetration.The cited approaches differ in whether attraction is uni-directional or symmetric.
  • Hand-object reconstruction: The paper's main distinction is an end-to-end learnable CNN that combines a differentiable hand model with differentiable penetration and contact constraints.This design jointly addresses hand-object reconstruction and interaction modeling.

3. Hand-object reconstruction

The model jointly reconstructs hand meshes, object meshes, and their relative configuration from RGB crops, using differentiable components and manipulation-aware contact constraints. It combines hand and object reconstruction losses with attraction and repulsion terms to improve physical plausibility.

  • Architecture: The network reconstructs object shape, hand mesh, relative scale, and translation in a single forward pass from a cropped RGB image.It uses separate object and hand branches, each with a ResNet18 encoder.
  • Hand model: MANO maps predicted pose and shape parameters to hand vertices and joints through a differentiable network layer.The hand branch supervises vertex and joint positions with L2 losses and regularizes hand shape toward the average MANO shape.
  • Object model: AtlasNet predicts object surfaces in normalized coordinates, then applies predicted translation and scale in the hand-relative coordinate system.Object training includes vertex, translation, scale, Chamfer, edge-regularization, and curvature-regularizing losses.
  • Contact loss: The contact loss penalizes interpenetration and encourages contact between hand and object surfaces using repulsion and attraction terms.Repulsion detects hand vertices inside the watertight object mesh; attraction acts on exterior vertices near the object.
  • Contact loss: Contact regions are estimated from grasp statistics, yielding 6 connected palmar regions corresponding to the fingertips and part of the palm.Vertices are selected when they lie within 3mm of the object surface in at least 8% of grasps; attraction is set to decrease beyond a 1cm threshold.
  • Training: Balancing attraction and repulsion is important for physical quality, and training adds the contact loss after initial hand-and-object reconstruction training.The contact weighting coefficient λR controls the balance between the two terms.

4. ObMan dataset

ObMan is a large-scale synthetic dataset of hand-object interactions generated by automatically grasping diverse everyday object models and rendering varied scenes.

  • Objects: ObMan uses ShapeNet models from 8 everyday object categories to provide varied hand-object configurations.The selected categories include bottles, bowls, cans, jars, knives, and cellphones.
  • Grasp generation: Automatic grasp generation prioritizes simplicity and robustness, using MANO transformed into 16 rigid hand parts.GraspIt generates grasps for each object mesh from various initializations.
  • Rendering: The dataset varies body poses, body shapes, viewpoints, camera distance, textures, lighting, and backgrounds during rendering.Global rotations are sampled uniformly in SO(3), and camera distance is sampled between 50 and 80 cm.
  • Rendered outputs: For each hand-object configuration, ObMan provides object-only, hand-only, and combined images with corresponding segmentation and depth maps.These outputs supply multiple visual representations and ground-truth scene annotations.

5. Experiments

The experiments evaluate reconstruction under occlusion, physical contact constraints, synthetic-to-real transfer, and qualitative generalization. Results show that occlusion-aware training, balanced contact losses, and ObMan pre-training improve reconstruction or grasp quality in the reported settings.

  • Evaluation setup: The experiments use multiple metrics covering hand error, object error, penetration, intersection volume, and simulation displacement.Hand error is mean endpoint error over 21 joints; object error uses symmetric Chamfer distance, while contact quality uses physical collision and simulation measures.
  • Effect of occlusions: Occluded-image training reduces hand and object reconstruction errors by 12% and 25%, respectively, when both training and testing use occluded images.The results support the conclusion that occluded hand-object reconstruction is harder and benefits from matching occlusion conditions during training.
  • Effect of contact loss: Without explicit physical constraints, average penetration depth is 9mm on ObMan and 19mm on FHBC.The FHBC ground truth contains object annotation imprecision, with an average training-split penetration depth of 11.0mm and standard deviation 8.9mm.
  • Effect of contact loss: Equal weighting of attraction and repulsion improves both physical measures without negatively affecting reconstruction metrics on synthetic and real datasets.Repulsion alone reduces maximum penetration by 33% on ObMan and 68% on FHBC but sacrifices simulation stability; equal weighting avoids that trade-off in the reported experiments.
  • Synthetic to real transfer: ObMan pre-training is beneficial in low-data regimes, especially when fewer than 1000 real images are available for fine-tuning.The study compares randomly initialized models with ObMan-pre-trained models on reduced real-data subsets.
  • Qualitative real-data results: On CORe50, the ObMan-trained model qualitatively generalizes across object categories, including a lightbulb absent from its training categories.The global outline is recovered in the camera view, while larger errors occur in the perpendicular direction.

6. Conclusions

The paper presents an end-to-end approach for reconstructing hands and objects jointly from a single RGB image, supported by complementary analyses of hand-only reconstruction. It identifies future directions in learning grasping affordances and recognizing complex, dynamic hand actions.

  • The method jointly reconstructs hands and objects from a single RGB image using an end-to-end approach.
  • A novel contact loss enforces physical constraints on interactions between the hand and object meshes.
  • Future directions include learning grasping affordances from large-scale visual data and recognizing complex, dynamic hand actions.
  • The supplementary analysis covers MANO hand losses, latent hand representation, hand pose estimation, and object reconstruction.

A.1. Loss study on MANO

The MANO loss study finds that shape regularization is necessary for predicting hand shape from sparse supervision, while dense vertex supervision provides an additional improvement on ObMan. The selected 30-component pose representation performs competitively on StereoHands while producing a dense mesh.

  • Loss design: Shape regularization Lβ prevents extreme hand-mesh deformations when predicting β from sparse keypoint supervision.Without Lβ, β is fixed to the average hand shape.
  • Loss design: Predicting β with Lβ significantly improves mean end-point error on keypoints.
  • Loss design: On ObMan, adding vertex-position supervision LV yields a small additional improvement.
  • Loss design: The training losses depend on available supervision: all three losses are used with dense vertices, while LJ and Lβ are used with keypoints only.Dense vertex supervision is available on ObMan but not on FHB or StereoHands.
  • MANO pose representation: Thirty PCA components are used because lower-dimensional representations miss poses and performance does not improve beyond 30 components.The representation uses 30 of the 45 full pose dimensions.
  • Hand pose evaluation: On StereoHands, the differentiable hand model is on par with state-of-the-art methods while producing a dense hand mesh rather than only sparse keypoints.The benchmark is described as close to saturation.

B.1. Canonical versus camera view reconstruction

The object branch can reconstruct meshes directly in the camera coordinate frame without degrading performance relative to canonical-view reconstruction. Both settings achieve performance comparable to the released AtlasNet model.

  • Viewpoint comparison: Camera-view and canonical-view object reconstruction have comparable numerical and qualitative performance.
  • Viewpoint comparison: Table A.3 compares Chamfer loss for the released AtlasNet model, a canonical-view reimplementation, and a non-canonical camera-view implementation.The table uses a ×1000 scaling for Chamfer loss and 2500 points.
  • Qualitative results: Qualitative camera-view reconstructions are shown using ShapeNet models as references.
  • Viewpoint comparison: Predicting the reconstruction in camera view shows no degradation relative to canonical-view prediction.

B.2. Object mesh regularization

Object-mesh regularization improves perceptual quality by reducing irregularities, but it also worsens the final reconstruction score and can remove sharp object curvatures. Edge and Laplacian terms provide complementary visual benefits.

  • Motivation: Unregularized predicted meshes exhibit irregular triangles and sharp curvature discontinuities in regions that should be smooth.
  • Ablation setup: The ablation trains four models: no regularization, each term separately, and both terms together.Each model is trained for 200 epochs.
  • Regularization effects: Edge regularization LE improves mesh quality but does not eliminate unwanted bendings by itself.The cellphone reconstruction illustrates the remaining bending artifacts.
  • Regularization effects: Adding Laplacian smoothness LL resolves the unwanted bendings that remain with LE alone.
  • Accuracy-quality trade-off: 22%: introducing edge regularization increases the Chamfer loss while significantly improving perceptual mesh quality.
  • Accuracy-quality trade-off: Regularization makes reconstructions coarser, so sharp curvatures visible in the input may not be captured.

C.1. Training details

Training uses staged optimization: the object branch is pretrained, the full network is then trained, and real-data fine-tuning unfreezes object reconstruction. Dataset-specific schedules and loss settings are used for stability.

  • Synthetic-data training freezes batch-normalization layers, and all loss weights are normalized to sum to 1.The frozen batch-normalization weights retain their ImageNet-pretrained values.
  • The final ObMan model first trains the normalized object branch for 250 epochs before freezing its encoder and decoder.Training starts at 10^-4 and decreases to 10^-5 after epoch 200.
  • The full network is subsequently trained for 350 additional epochs with LHand + LObject supervision.The learning rate decreases from 10^-4 to 10^-5 after the first 200 epochs.
  • Fine-tuning on smaller real datasets unfreezes the object reconstruction branch, while contact-loss fine-tuning uses a learning rate of 10^-5.When contact loss is included, Adam momentum is set to zero because it negatively affects training stability.
  • FHBc training jointly optimizes all network parts for 400 epochs using LHand + LObject supervision.The learning rate changes from 10^-4 to 10^-5 at epoch 300.

C.2. Heuristic metric for sorting GraspIt grasps

The heuristic ranks GraspIt-generated grasps using grasp-wrench-space quality and contact coverage, retaining two high-quality, diverse grasps per object.

  • GraspIt generates varied grasps by exploring initial hand poses, after which a post-processing heuristic filters out low-quality grasps.The retained grasps are selected according to an engineered quality metric.
  • Each grasp supplies ε and v metrics derived from the grasp wrench space under rigid contacts with friction.The grasp wrench space is normalized by object scale, defined using the maximum radius centered at the object’s center of mass.
  • The heuristic uses the norm of the [ε, v] vector and weights it by Np, the number of phalanges contacting the object.This weighting favors grasps with broader phalange-object contact.
  • The final grasp quality score G is defined as the stated weighted heuristic used to sort candidate grasps.The score combines grasp-quality metrics with the number of contacting phalanges.
  • Keeping the two best grasps for each object yields grasps that are both diverse and good quality.

D. Qualitative results on CORe50 dataset

On CORe50, the ObMan-trained model qualitatively generalizes across object categories, while reconstructions remain limited by topology, coarse shape detail, spatial misalignment, collisions, and occasional invalid hand poses.

  • The final model, trained solely on ObMan, produces reconstructions for diverse CORe50 input images and object instances.
  • The model generalizes to light bulbs, although that object category is absent from ObMan’s synthetic object models.
  • Mug reconstructions expose the sphere baseline’s topological limitation: it cannot capture handles by construction.
  • Object shapes are often coarse, and fine details such as phone antennas are not reconstructed.
  • Some predictions misplace the object toward the hand’s palmar centroid, allow hand-object interpenetration, or violate anatomical hand constraints.Anatomical validity is not guaranteed even though the hand pose is predicted in a low-dimensional space.
Loading 1904.05767v1…