Source-linked AI summary
3D Hand Shape and Pose Estimation from a Single RGB Image
Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, Junsong Yuan
TL;DR
The paper addresses monocular RGB estimation of full 3D hand shape and pose, beyond methods focused mainly on sparse joints. It uses Graph CNN mesh reconstruction, synthetic full supervision, and depth-based weak supervision for real-world fine-tuning, reporting accurate real-time meshes and superior pose accuracy.
Problem
Monocular RGB methods mainly estimate sparse 3D hand joints, leaving full 3D hand shape insufficiently represented despite the needs of hand-analysis applications.
Method
A Graph CNN generates full hand-mesh vertices from RGB-derived image features, trained synthetically with mesh and pose ground truth and fine-tuned using rendered-versus-reference depth.
Results
Experiments on proposed and public datasets recover accurate 3D hand meshes and joint locations in real time while achieving superior 3D pose-estimation accuracy against state-of-the-art methods.
Takeaways & Limitations
Full mesh estimation can represent hand-shape variation and local surface detail while also improving the reported 3D hand pose-estimation performance.
Takeaways & Limitations
At test time, the implementation assumes that global hand scale and the absolute depth of the root joint are provided.
Abstract
from arXiv · showhide
This work addresses a novel and challenging problem of estimating the full 3D hand shape and pose from a single RGB image. Most current methods in 3D hand analysis from monocular RGB images only focus on estimating the 3D locations of hand keypoints, which cannot fully express the 3D shape of hand. In contrast, we propose a Graph Convolutional Neural Network (Graph CNN) based method to reconstruct a full 3D mesh of hand surface that contains richer information of both 3D hand shape and pose. To train networks with full supervision, we create a large-scale synthetic dataset containing both ground truth 3D meshes and 3D poses. When fine-tuning the networks on real-world datasets without 3D ground truth, we propose a weakly-supervised approach by leveraging the depth map as a weak supervision in training. Through extensive evaluations on our proposed new datasets and two public datasets, we show that our proposed method can produce accurate and reasonable 3D hand mesh, and can achieve superior 3D hand pose estimation accuracy when compared with state-of-the-art methods.
1. Introduction
The paper frames full 3D hand shape and pose estimation from one RGB image as an open problem beyond sparse joint prediction. It proposes a Graph CNN mesh generator, synthetic full-supervision data, and depth-based weak supervision for real-world fine-tuning.
- Motivation: Monocular RGB hand analysis has mainly estimated sparse 3D joints, while applications often require both 3D pose and full hand shape.The paper motivates dense surface recovery for VR and AR applications.
- Problem: The task jointly estimates 3D hand joint locations and all vertices of the hand-surface mesh from a single RGB image.This extends monocular analysis beyond keypoint-only representations.
- Challenges: Dense mesh prediction is difficult because its output space can contain about 1280 vertices, compared with approximately 21 skeletal joints.A predefined deformable model such as MANO is identified as one straightforward alternative.
- Method: The proposed end-to-end Graph CNN hierarchically applies graph convolutions, upsampling, and nonlinear activations to generate mesh vertices from image features.The graph representation exploits topological relationships among mesh vertices and is intended to capture shape variation and local detail.
- Training: A large-scale synthetic dataset supplies full mesh and pose supervision, while real-world fine-tuning renders predicted meshes into depth maps and minimizes loss against reference depth.Testing requires only an RGB image, without real-world 3D mesh or pose annotations.
- Results: Experiments on proposed and public datasets report accurate, reasonable real-time mesh recovery and superior 3D pose accuracy against state-of-the-art methods.The evaluations include the proposed synthetic and real-world datasets plus two public datasets.
2. Related Work
Prior work estimated hand shape mainly from depth or relied on constrained model-based representations. This paper instead targets direct Graph CNN mesh generation from monocular RGB images.
- 3D hand shape and pose from depth: Depth-based methods commonly fit deformable hand models through iterative optimization, while CNN approaches recover meshes through pose and shape parameters or linear blend skinning.The quality of CNN-recovered meshes can be restricted by a simple LBS model.
- 3D hand pose from RGB: Some prior RGB methods estimate hand pose from monocular images, but the supplied related-work passage does not describe full mesh recovery.The comparison motivates the paper's denser output target.
- 3D human body shape and pose: Human-body methods often regress SMPL parameters or predict volumetric representations, whereas this work estimates hand mesh vertices with Graph CNNs.The stated goal is to model nonlinear hand-shape variation and mesh-vertex relationships.
3. 3D Hand Shape and Pose Dataset Creation
The paper creates datasets that add full 3D hand-mesh annotations to RGB hand imagery, addressing the scarcity of real-world mesh supervision. The synthetic set provides broad rendered variation, while a smaller real-world set supports quantitative evaluation.
- Motivation: Existing hand RGB datasets generally annotate 2D or 3D joints but lack 3D hand-shape annotations needed for mesh-estimation training.Manual annotation of real-world meshes and joints is described as laborious and time-consuming.
- Synthetic dataset: The synthetic dataset uses a rigged 3D hand model, photorealistic textures, natural lighting, blend-shape variation, 500 gestures, and 1000 camera views.These construction choices are intended to cover hand-shape and pose variation.
- Synthetic dataset: 375,000 rendered hand RGB images comprise the synthetic dataset, split into 315,000 training and 60,000 validation images.Training also blends rendered hands with randomly sampled and cropped backgrounds from COCO, LSUN, and Flickr.
- Real-world dataset: The real-world dataset contains 583 hand RGB images annotated with 3D meshes and 3D joint locations using corresponding RealSense depth images.The annotations are produced by manually adjusting a 3D hand model in Maya with RGB and depth references.
4. Methodology
The method reconstructs a full 3D hand mesh and regresses 3D joint locations from a single RGB image using a hierarchical Graph CNN. It combines fully supervised synthetic training with weakly supervised real-world fine-tuning using depth and pseudo-ground-truth mesh supervision.
- Pipeline: A residual-network latent feature vector is passed to a Graph CNN that infers 3D mesh vertices, from which 3D joint locations are linearly regressed.The input image first produces feature maps and 2D heat-maps through a two-stacked hourglass network.
- Graph CNN architecture: The mesh-generation network outputs 3D coordinates for 1280 vertices after transforming the latent feature into an 80-vertex coarse graph and applying two upsampling layers.The coarse graph assigns 64-dimensional features to its 80 vertices before refinement.
- Graph convolution: Chebyshev spectral graph convolution updates each vertex using a K-localized polynomial of the graph Laplacian and trainable feature transformations.The operation affects the K-hop neighbors of each central node.
- Graph CNN architecture: The Graph CNN hierarchically generates mesh features from coarse to fine graphs using graph coarsening, feature upsampling, and graph convolutions.The predefined mesh topology and coarse-graph topologies remain fixed during training and testing.
- Fully-supervised training: Synthetic training uses heat-map, mesh, and 3D pose losses, with mesh supervision combining vertex, normal, edge, and Laplacian terms.The heat-map resolution is 64×64 px, and the Laplacian-loss weight is set to λl = 50.
- Weakly-supervised fine-tuning: Real-world fine-tuning does not require ground-truth 3D meshes or joints and uses depth-map supervision together with pseudo-ground-truth mesh supervision.The generated mesh is rendered to a depth map during training; this rendering step is not used during testing.
5. Experiments
Experiments evaluate mesh reconstruction and pose estimation across proposed and public datasets, comparing the Graph CNN method with baselines and state-of-the-art approaches.
- 3D hand mesh reconstruction: Mesh reconstruction is evaluated using average Euclidean vertex error on the proposed synthetic and real-world datasets.No other hand RGB dataset contains ground-truth 3D meshes.
- 3D hand mesh reconstruction: The Graph CNN method produces lower mesh error than direct LBS on both proposed datasets, whose predefined model cannot adapt to different hand shapes.Direct LBS also produces unrealistic joint deformations and serious artifacts.
- 3D hand mesh reconstruction: The MANO-based method performs worse than the proposed method for mesh reconstruction on the aligned meshes used for fair comparison.Its fixed model limits representation power for hand-shape variation.
- 3D hand pose estimation: On STB, the full model outperforms both pose-estimation baselines, while depth-map loss is especially beneficial without 3D pose supervision.The mesh reconstruction network also improves pose estimation relative to directly regressing joints.
- 3D hand pose estimation: The method outperforms state-of-the-art methods over most or all tested PCK thresholds on RHD and STB, including a large-margin comparison with Cai et al. without pose supervision.Figure 9 reports AUC values in parentheses for these comparisons.
- Runtime: The full model runs in 19.9ms on average, corresponding to over 50fps on an Nvidia GTX 1080 GPU.This includes mesh and pose output.
6. Conclusion
The conclusion presents a Graph CNN model and supervised synthetic-data training for single-image hand mesh and pose estimation. Weak depth supervision supports real-world fine-tuning, with accurate real-time recovery reported across proposed and public datasets.
- The paper develops a Graph CNN-based model that reconstructs a full 3D hand-surface mesh from a single RGB image.
- A large-scale synthetic RGB dataset provides ground-truth 3D joint locations and full 3D hand meshes for fully supervised training.
- Real-world fine-tuning uses depth maps rendered from generated meshes as weak supervision without 3D ground truth.
- Experiments show accurate 3D hand mesh and joint recovery in real time.
- Future work includes larger Mocap-based pose and shape data and hand-object or hand-hand interaction cases.
A. Qualitative Results
Additional qualitative results cover mesh reconstruction and pose estimation across the proposed datasets, STB, RHD, and Dexter+Object.
- Figure 11 presents qualitative mesh and pose results for five datasets, including Dexter+Object and both proposed datasets.
B. Details of Baseline Methods for 3D Hand Mesh Reconstruction
The baseline methods generate meshes from predicted joints or MANO parameters, whereas the proposed approach is contrasted with their representation limitations.
- Direct LBS: Direct LBS predicts 3D joints, applies inverse kinematics, and generates meshes with a predefined model and skinning weights.
- MANO-based Method: The MANO-based baseline regresses shape and pose parameters, then generates meshes through the differentiable MANO model during training.
- MANO-based Method: Because MANO is fixed and essentially uses LBS with blend shapes, its representation power is limited.
C. Details of the Task Transfer Method
The “full model, task transfer” method transfers a model trained for 3D hand mesh reconstruction to 3D hand pose estimation by regressing joints from its latent features.
- C. Details of the Task Transfer Method: “Full model, task transfer” directly regresses 3D hand joint locations from latent features extracted by the mesh-reconstruction model.An MLP with three fully connected layers performs the regression.
- C. Details of the Task Transfer Method: The MLP is first trained with 3D pose loss on the synthetic dataset, then fine-tuned or used directly on STB depending on pose supervision.With supervision, it is fine-tuned using 3D pose loss; without supervision, the synthetic-data-pretrained MLP is used directly.
- C. Details of the Task Transfer Method: On STB, the task-transfer method outperforms a baseline trained only for 3D hand pose estimation despite using the same pipeline.The result is attributed to the more discriminative latent feature learned through full-model mesh reconstruction training.
- C. Details of the Task Transfer Method: Figure 13 compares the proposed task-transfer pipeline with direct LBS and MANO-based baseline pipelines.The figure highlights differences between the baseline methods and the proposed method.