Source-linked AI summary
Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering
Seungryul Baek, Kwang In Kim, Tae-Kyun Kim
TL;DR
Recovering dense 3D hand shapes and poses from single RGB images remains difficult because of hand variation, self-occlusion, and limited direct 3D evidence. The paper proposes a CNN-based dense hand pose estimator with a neural renderer, iterative testing refinement, and self-supervised data augmentation, achieving state-of-the-art accuracy in 3D skeleton estimation and 2D hand segmentation while reconstructing dense hand shapes.
Problem
Estimating dense 3D hand poses from single RGB images is challenging, while dense shapes are important for understanding human-object interactions and robotic grasping.
Method
The paper uses a CNN-based dense hand pose estimator with a deformable 3D hand mesh, neural rendering, indirect 2D supervision, iterative gradient refinement, and view-based self-data augmentation.
Results
The framework achieves state-of-the-art accuracy in both 3D hand skeleton estimation and 2D hand segmentation while reconstructing 3D hand shapes and poses.
Takeaways & Limitations
Jointly estimating hand shapes and poses, together with iterative refinement and indirect supervision, supports accurate dense hand reconstruction from single RGB images.
Takeaways & Limitations
The authors do not explore generating new poses because this often produces implausible hand poses.
Abstract
from arXiv · showhide
Estimating 3D hand meshes from single RGB images is challenging, due to intrinsic 2D-3D mapping ambiguities and limited training data. We adopt a compact parametric 3D hand model that represents deformable and articulated hand meshes. To achieve the model fitting to RGB images, we investigate and contribute in three ways: 1) Neural rendering: inspired by recent work on human body, our hand mesh estimator (HME) is implemented by a neural network and a differentiable renderer, supervised by 2D segmentation masks and 3D skeletons. HME demonstrates good performance for estimating diverse hand shapes and improves pose estimation accuracies. 2) Iterative testing refinement: Our fitting function is differentiable. We iteratively refine the initial estimate using the gradients, in the spirit of iterative model fitting methods like ICP. The idea is supported by the latest research on human body. 3) Self-data augmentation: collecting sized RGB-mesh (or segmentation mask)-skeleton triplets for training is a big hurdle. Once the model is successfully fitted to input RGB images, its meshes i.e. shapes and articulations, are realistic, and we augment view-points on top of estimated dense hand poses. Experiments using three RGB-based benchmarks show that our framework offers beyond state-of-the-art accuracy in 3D pose estimation, as well as recovers dense 3D hand shapes. Each technical component above meaningfully improves the accuracy in the ablation study.
1. Introduction
Dense 3D hand pose estimation extends sparse skeletal prediction by recovering hand shape, articulation, and viewpoint from RGB images. The proposed DHPE combines neural rendering, indirect 2D supervision, iterative refinement, and fitted-mesh augmentation.
- Motivation: Dense hand modeling matters for applications such as human-object interaction and robotic grasping, where surface contacts are essential.
- Motivation: Dense hand pose estimation recovers 3D shapes alongside articulations and viewpoints, providing a richer hand description than coarse skeletal representations.Recovered shapes can be represented in a canonical articulation and viewpoint.
- Motivation: Existing CNN methods predict coarse skeletal poses without consistently enforcing explicit kinematic or geometric mesh constraints, while generative methods optimize personalized models against images.
- Method: DHPE uses a neural renderer and CNNs to estimate 2D evidences before 3D mesh parameters, with 2D segmentation masks and skeletons replacing unavailable RGB-mesh supervision.
- Method: At test time, DHPE iteratively refines the initial mesh using gradients from consistency between estimated meshes and predicted 2D evidences, while recursively improving 2D skeletons.
- Method: Fitted meshes provide close-to-real shapes and poses for varying viewpoints, enabling self-data augmentation that addresses scarce annotations and synthetic-real domain gaps.
2. Related work
Prior hand-pose work spans RGB and depth inputs with skeletal, mesh, voxel, and 2.5D representations. DHPE instead combines a deformable 3D mesh model, differentiable projection, and staged 2D-to-3D estimation for RGB inputs.
- Output and input representations: Hand-pose methods differ by input modality and output representation, including RGB or depth inputs and 2D skeleton, 3D skeleton, or 3D mesh outputs.
- 3D mesh reconstruction: RGB mesh fitting methods may fail when coarse 2D skeleton estimates are inaccurate, motivating denser geometric evidence.
- Depth and RGB settings: Depth-based methods benefit from inherently rich 3D information and million-scale datasets, whereas RGB data and automatic pose annotation remain scarce.
- Recovering 3D skeletons from RGB: Recent RGB methods estimate 2.5D depth representations before lifting them to 3D skeletons, while DHPE uses a 3D mesh model to address 2D-to-3D uncertainty.
- Proposed framework: The proposed DHPE decomposes estimation into 2D evidence and 3D mesh stages, connected through a projector that uses 3D skeleton and rendering operations.
- 3D mesh reconstruction: Differentiable renderers enable mesh reconstruction from 2D silhouettes, depth maps, or RGB maps without requiring full 3D supervision.
3. Proposed dense hand pose estimator
The proposed dense hand pose estimator decomposes RGB-to-mesh prediction into 2D evidence estimation and 3D mesh estimation, connected through a fixed differentiable projection operator. It uses indirect 2D supervision, iterative refinement, and auxiliary losses to recover articulated hand meshes and improve pose estimation.
- Dense hand mesh representation: The hand mesh estimator maps a 224×224 RGB image to a 63-dimensional MANO parameter vector representing a mesh with 778 vertices and 1,538 triangular faces.The corresponding mesh can produce 21 3D joint positions through a skeleton regressor.
- Estimator decomposition: DHPE decomposes RGB-to-mesh estimation into a 2D evidence estimator and a 3D mesh estimator connected by a fixed projection operator.The projection operator combines a skeleton regressor and differentiable renderer while respecting hand geometry and camera structure.
- 2D evidence estimator: The 2D evidence estimator predicts a 2,048-dimensional RGB feature vector and 21 2D joint heat maps, with segmentation-based masked images used as evidence.The heat maps provide 2D skeletal joint estimates, while the feature extractor uses ResNet-50 features.
- 3D mesh estimator: The 3D estimator predicts MANO pose, shape, and camera parameters from 2D evidence, then synthesizes the mesh through the parametric hand model.The mesh is initialized and transformed using pose and shape parameters together with global rotation, scale, and translation.
- Training objectives: Joint training uses articulation and shape losses derived from projected 3D skeletons and 2D masks, with experiments showing gains from combining 2D features, 2D skeletons, and the pose refiner.The articulation loss measures skeleton deviation, while the shape loss indirectly supervises hand shape through projected segmentation masks.
4. Experiments
Experiments evaluate the framework on three hand-pose datasets against neural and model-fitting baselines, with ablations of its design choices. The results show improvements in pose estimation, segmentation, and several framework components.
- Experimental settings: The evaluation uses SHD, RHD, and DO, with training combining the original SHD and RHD training sets.SHD contains 36,000 frames, RHD contains 43,986 synthetic images, and the combined training set contains 71,258 frames.
- Ablation study: Jointly estimating hand shape and pose yields higher pose-estimation accuracy than estimating pose alone.This comparison is reported for the RHD evaluation.
- Ablation study: Testing refinement significantly improves performance on DO, while its effect is smaller on RHD.The no-refinement results are considerably worse on DO but similar on RHD.
- Ablation study: Data augmentation and explicit 2D-loss supervision further improve accuracy over their corresponding ablated settings.The 2D losses supervise the 2D evidence estimator, while the augmentation strategy improves accuracy over cases without data augmentation.
- Ablation study: Hierarchical scheduling of the shape-loss weight produces more training instances with small articulation errors and faster test-error reduction than a constant weight.The comparison uses λ scheduled by Eq. 10 versus a constant λ value.
- Hand segmentation performance: The method achieves higher hand-segmentation accuracy than the compared state-of-the-art segmentation networks on RHD.The comparison uses four object-segmentation performance criteria.
5. Conclusions
The paper presents DHPE, a CNN-based framework for reconstructing dense 3D hand shapes and poses from single RGB images. Its projector enables indirect 2D supervision and data generation, while experiments report state-of-the-art 3D skeleton and 2D segmentation accuracy.
- Conclusions: DHPE reconstructs 3D hand shapes and poses from single RGB images using a 2D evidence estimator, 3D mesh estimator, and projector.The projector uses a neural renderer to connect the estimated mesh with image-based evidence.
- Conclusions: The neural renderer replaces insufficient full 3D supervision with indirect supervision from 2D segmentation masks and 3D joints.This design also enables generating new data for training.
- Conclusions: Jointly estimating 3D hand shapes and poses offers state-of-the-art accuracy in both 3D hand skeleton estimation and 2D hand segmentation.The conclusion reports this outcome across the evaluated tasks.