Source-linked AI summary
End-to-end Hand Mesh Recovery from a Monocular RGB Image
Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, Wen Zheng
TL;DR
HAMR addresses monocular hand understanding by recovering a full 3D hand mesh from one RGB image, rather than only estimating hand pose. It parameterizes a generic hand model and uses derived 2D and 3D joints for end-to-end training, recovering meshes under severe occlusion and outperforming state-of-the-art methods on both pose tasks.
Problem
Most existing hand image-understanding methods rely on depth cameras or multi-view images, while monocular RGB recovery is challenged by appearance variation, self-occlusion, articulation, perspective ambiguity, and limited annotated 3D data.
Method
HAMR parameterizes a generic 3D hand model by shape and relative joint angles, derives 2D and 3D joints from the mesh, and trains using available pose annotations.
Results
HAMR recovers reasonable 3D hand meshes under severe occlusions and outperforms state-of-the-art methods on monocular 2D and 3D hand pose estimation across benchmark datasets.
Takeaways & Limitations
A unified mesh representation supports hand mesh recovery together with 2D and 3D pose estimation from a single RGB image.
Takeaways & Limitations
Networks trained on synthetic data do not generalize well to real-world images because of domain shift.
Abstract
from arXiv · showhide
In this paper, we present a HAnd Mesh Recovery (HAMR) framework to tackle the problem of reconstructing the full 3D mesh of a human hand from a single RGB image. In contrast to existing research on 2D or 3D hand pose estimation from RGB or/and depth image data, HAMR can provide a more expressive and useful mesh representation for monocular hand image understanding. In particular, the mesh representation is achieved by parameterizing a generic 3D hand model with shape and relative 3D joint angles. By utilizing this mesh representation, we can easily compute the 3D joint locations via linear interpolations between the vertexes of the mesh, while obtain the 2D joint locations with a projection of the 3D joints.To this end, a differentiable re-projection loss can be defined in terms of the derived representations and the ground-truth labels, thus making our framework end-to-end trainable.Qualitative experiments show that our framework is capable of recovering appealing 3D hand mesh even in the presence of severe occlusions.Quantitatively, our approach also outperforms the state-of-the-art methods for both 2D and 3D hand pose estimation from a monocular RGB image on several benchmark datasets.
1. Introduction
Hand image understanding from monocular RGB images is difficult because of appearance variation, self-occlusion, complex articulation, missing depth, and limited annotated 3D data. HAMR addresses this gap by recovering a hand mesh while also producing 2D and 3D pose estimates.
- Hand configuration recovery is challenging because hands exhibit diverse appearances, self-occlusions, and complex articulations.
- Most existing markerless methods rely on depth cameras or multi-view images, limiting their practicality when only monocular RGB images are available.
- Monocular RGB hand-pose methods address 2D and 3D estimation, but single-image 2D predictions can be unreasonable and 3D estimation faces perspective ambiguity from missing depth.
- Learning strategies for monocular 3D pose include latent multimodal representations, intermediate 2.5D poses, and weak supervision using depth images during training.
- Synthetic training data can provide 3D annotations, but models trained on synthetic data do not generalize well to real-world images because of domain shift.
- HAMR parameterizes a generic 3D hand model by shape and relative joint angles, deriving mesh, 2D-pose, and 3D-pose representations for end-to-end learning.
2. Related Work
Prior work established monocular face and body mesh recovery using parametric models, while monocular hand reconstruction remained comparatively underexplored. HAMR extends this direction to hand mesh recovery and integrates pose estimation outputs.
- Single-image 3D reconstruction has been studied extensively in computer vision and computer graphics, including face and human-body reconstruction.
- Monocular Face Reconstruction: Monocular face reconstruction uses 3D morphable models and differentiable, end-to-end frameworks to estimate facial pose, shape, expression, reflectance, and illumination.
- Monocular Human Body Recovery: Monocular human-body methods recover pose and shape or full meshes by regressing parameters of parametric models such as SMPL.
- HAMR's framework architecture combines 2D heat-map estimation, iterative regression of mesh and camera parameters, and mesh-based recovery of 2D and 3D joints.
- Monocular hand reconstruction had received limited attention, and HAMR was presented as one of the first approaches targeting this problem among concurrent works.
3. Framework
HAMR reconstructs a hand mesh from monocular RGB by parameterizing a generic model and deriving 3D joints and 2D keypoints from it. Differentiable geometric, silhouette, pose, camera, and intermediate-supervision losses support end-to-end training, with iterative regression refining camera and mesh parameters.
- Framework: HAMR jointly represents a detailed hand mesh M, 2D keypoints Φ2D, and 3D joints Φ3D from a hand-centered RGB image.The mesh has N vertices, while the keypoint and joint representations contain K points.
- Hand Mesh Representation: MANO parameterizes the triangulated mesh using shape parameters β and pose parameters θ, modeling hand identity and articulation-dependent surface deformation.Shape includes properties such as finger slenderness and palm thickness; pose encodes relative 3D joint rotations.
- Hand Mesh Representation: The final mesh is produced by applying shape and pose deformations to a mean template, then posing finger parts with blend skinning.Both template deformation and skinning are differentiable with respect to the mesh parameters.
- Derived Hand Pose Representations: 3D joint locations are obtained by linear interpolation between mesh vertices, while 2D keypoints result from projecting the recovered 3D joints with a weakly perspective camera.The camera parameters are scale and two image-plane offsets, (s, tx, ty).
- Loss Function: HAMR combines 2D and 3D representation losses with geometric, silhouette, camera, and intermediate heatmap supervision to train from available annotations without ground-truth mesh parameters.The full process is differentiable, and hyper-parameters balance the different supervision terms.
- Iterative Regression Module: An iterative regression module refines camera and mesh parameters from image features and current estimates, using a fully convolutional encoder and fully connected layers.Camera supervision uses parameters computed from paired 3D and 2D annotations.
4. Experiments
HAMR is evaluated through qualitative mesh recovery, cross-dataset generalization, quantitative 3D and 2D pose performance, and ablation studies across STB, RHD, and Dexter.
- Experiment Settings: Experiments use RHD, STB, and Dexter, with RGB images and pose annotations, while Dexter supports cross-dataset evaluation because its hand annotations are incomplete.RHD provides RGB, depth, masks, and 2D/3D annotations; the experiments use RGB images and corresponding pose annotations.
- Experiment Settings: HAMR is trained end-to-end with RGB-centered hand crops, data augmentation, and a two-stack Stacked Hourglass pose module.The framework uses RMSprop optimization and combines pose, geometric, camera, heatmap, and silhouette-related losses during training.
- Visualization of Appealing Mesh: Qualitative results show high-quality mesh recovery on STB and RHD, including poor lighting, image truncation, self-occlusion, and exaggerated articulation.The recovered meshes are described as anthropometrically reasonable even under heavily self-occluded or otherwise difficult conditions.
- Quantitative Evaluation: On STB, HAMR performs competitively with comparison methods, while on RHD it achieves state-of-the-art 3D PCK and on Dexter it largely outperforms most state-of-the-art methods.The evaluation uses area under the curve on percentage of correct key points across varying thresholds.
- Generalization Performance Evaluation: On Dexter, cross-dataset testing without finetuning shows better 2D hand-pose results than methods reported in [17] [27] [63], despite occluded images.The model is trained on RHD and STB before testing on Dexter.
- Ablation Studies: Ablations show that the parametric mesh model substantially improves pose performance, while camera supervision and silhouette consistency also improve accuracy; geometric constraints add only marginal gains.The mesh model already captures inherent hand geometry, whereas camera supervision calibrates projection and silhouette loss refines hand shape and pose.
5. Conclusion
HAMR addresses hand mesh recovery from single RGB images by enriching pose estimation with a generic 3D hand-model representation. It recovers reasonable meshes under severe occlusions and outperforms state-of-the-art methods on 2D and 3D hand pose estimation tasks.
- HAMR uses a generic 3D hand model to produce a mesh representation from single RGB images.The mesh model also supports 2D and 3D hand pose estimation.
- HAMR recovered reasonable hand meshes from single RGB images even under severe occlusions.
- HAMR's superiority was empirically confirmed on 2D and 3D hand pose estimation tasks compared with state-of-the-art methods.
6. Supplementary Material
The supplementary material evaluates HAMR's reconstructed-hand segmentation on the RHD dataset. It presents example masks and reports that HAMR outperforms the compared state-of-the-art methods.
- Segmentation is evaluated with mean Intersection over Union on the RHD dataset.Ground-truth masks are unavailable for the STB and Dexter datasets, so the reported segmentation evaluation concerns RHD.
- Figure 10 presents RGB examples from the RHD testing set alongside their corresponding segmentation masks.
- HAMR outperforms all compared state-of-the-art methods in the quantitative segmentation comparison.