Source-linked AI summary
3D Hand Shape and Pose from Images in the Wild
Adnane Boukhayma, Rodrigo de Bem, Philip H. S. Torr
TL;DR
The paper addresses monocular 3D hand reconstruction despite depth and scale ambiguities, limited 3D annotations, and poor generalization to images in the wild. It combines a deep convolutional encoder with a differentiable hand-model decoder and weak 2D plus limited 3D supervision, achieving state-of-the-art benchmark performance and plausible reconstructions in challenging wild images.
Problem
Monocular 3D hand reconstruction is ill posed and remains limited by scarce reliable 3D annotations and inadequate generalization to challenging images in the wild.
Method
A deep convolutional encoder predicts hand and view parameters for a differentiable articulated hand model and weak-perspective reprojection, trained with both 2D and 3D joint annotations.
Results
The method achieves state-of-the-art 3D pose results on standard benchmarks and produces geometrically valid, plausible reconstructions on challenging images in the wild.
Takeaways & Limitations
Combining learned prediction with hand-model shape and pose priors supports good generalization to 3D hand reconstruction from uncontrolled RGB images without post-processing optimization.
Takeaways & Limitations
The training objective depends on masks that hand-skin segmentation methods cannot obtain reliably under occlusion.
Abstract
from arXiv · showhide
We present in this work the first end-to-end deep learning based method that predicts both 3D hand shape and pose from RGB images in the wild. Our network consists of the concatenation of a deep convolutional encoder, and a fixed model-based decoder. Given an input image, and optionally 2D joint detections obtained from an independent CNN, the encoder predicts a set of hand and view parameters. The decoder has two components: A pre-computed articulated mesh deformation hand model that generates a 3D mesh from the hand parameters, and a re-projection module controlled by the view parameters that projects the generated hand into the image domain. We show that using the shape and pose prior knowledge encoded in the hand model within a deep learning framework yields state-of-the-art performance in 3D pose prediction from images on standard benchmarks, and produces geometrically valid and plausible 3D reconstructions. Additionally, we show that training with weak supervision in the form of 2D joint annotations on datasets of images in the wild, in conjunction with full supervision in the form of 3D joint annotations on limited available datasets allows for good generalization to 3D shape and pose predictions on images in the wild.
1 University of Oxford, UK 2 Federal University of Rio Grande, Brazil
The listed affiliations identify the University of Oxford in the UK and the Federal University of Rio Grande in Brazil.
- The first listed affiliation is the University of Oxford, UK.
- The second listed affiliation is the Federal University of Rio Grande, Brazil.
- The affiliation line associates the authors with institutions in the United Kingdom and Brazil.
1. Introduction
The paper addresses difficult monocular 3D hand estimation, where limited 3D data and poor real-world generalization remain central obstacles. It uses a model-based deep-learning pipeline trained with both 2D and 3D supervision, reporting strong benchmark and in-the-wild performance.
- Motivation: Monocular 3D hand estimation is ill posed because of depth and scale ambiguities, uniform appearance, complex poses, and varied viewpoints.
- Problem: The main obstacles are limited reliable 3D annotations and poor generalization from existing 3D datasets to images in the wild.
- Approach: The method exploits a differentiable articulated hand model and combines 2D and 3D joint annotations instead of relying heavily on 3D training data.
- Contributions: The work proposes end-to-end prediction of both 3D hand shape and pose from a single RGB image.
- Results: The method achieves state-of-the-art results on standard benchmarks and strong qualitative reconstructions on challenging images in the wild.
2. Related work
Prior work spans depth, multi-view, monocular, synthetic-data, regularization, and hand-model approaches. This paper distinguishes itself by combining a CNN encoder with a fixed generative hand-model decoder for monocular reconstruction.
- Scope: Existing 3D hand reconstruction research uses depth, image-plus-depth, stereo, or multiple images, whereas this work focuses on a single color image.
- Monocular methods: Pre-deep-learning monocular methods generally had limited performance and depended on careful initialization or background knowledge.
- Deep learning: Deep-learning approaches used sequential prediction, synthetic-to-real image transformation, and other strategies to address monocular hand pose estimation.
- Depth regularization: Recent methods reduce depth ambiguity or annotation dependence using depth regularization and multimodal variational autoencoders.
- Hand models: Hand models represent geometry with primitives, Gaussians, sphere meshes, subdivision, or learned linear-blend-skinning shape variation.
- Model-based decoders: This paper is the first to combine a CNN encoder with a fixed generative hand model for 3D hand reconstruction from images.
3. Overview
The pipeline encodes an image and optional 2D detections into hand and view parameters, then decodes them into a 3D hand and its image projection. Training is end-to-end with joint annotations, without direct supervision of those parameters.
- Encoder: A deep convolutional encoder takes a hand image and optionally 2D joint heat-maps to predict shape, pose, and view parameters.
- Hand model: A differentiable articulated hand model converts the predicted shape and pose into a triangulated 3D mesh and underlying 3D skeleton.
- Re-projection: A weak-perspective camera re-projects the generated mesh and skeleton into the image domain using the predicted view parameters.
- Decoder: The fixed hand model and re-projection module form a model-based decoder that requires no decoder-parameter training.
- Training: End-to-end training uses 2D and 3D joint annotations without supervising hand or view parameters, aside from magnitude regularization on hand parameters.
4. Hand model
The method uses the differentiable MANO articulated mesh deformation model to generate hand geometry from shape and pose parameters, incorporating learned shape and pose variation and a 21-joint output.
- Hand model: MANO represents the hand as a differentiable function M(β, θ), where β controls shape and θ controls pose.
- Hand model: The model applies linear blend skinning to a triangulated template mesh rigged with a kinematic tree of K = 16 joints.Joint locations are learned as a sparse linear regressor from mesh vertices, and blend weights define the skinning function.
- Hand model: Shape and pose corrective blend shapes deform a mean mesh to reduce artifacts such as overly smooth outputs and mesh collapse around joints.
- Hand model: The model constants are learned from registered hand scans of 31 subjects performing roughly 51 hand poses.
- Hand model: Pose is represented using 10 principal-component coefficients retained from angle-axis joint data to encourage physically plausible hand poses.
- Hand model: The generated mesh contains N = 778 vertices and 1538 faces, while five mesh vertices are appended to produce 21 final 3D key-points.
5. Camera model
The camera model re-projects generated 3D hand geometry into the image using weak perspective, avoiding the need for camera intrinsics.
- Camera model: Weak perspective re-projects the 3D hand mesh and joints into the 2D image plane using rotation R, translation t, and scale s.The projection uses a global rotation matrix, a 2D translation, and a positive scale.
- Camera model: The approximation enables training on annotated images without camera intrinsics, including images in the wild from sources such as YouTube videos.
- Camera model: The projection uses Π as the orthographic projection operator.
6. Encoder
A ResNet-50 encoder predicts hand and camera parameters from RGB images, optionally augmented with 2D joint heat-maps, and is pre-trained using synthetic examples generated by the decoder model.
- Encoder: The encoder outputs v = {R, t, s, β, θ} ∈ IR26, including hand shape, hand pose, and camera parameters.Global rotation is encoded with three axis-angle parameters.
- Encoder: The encoder can receive 2D hand joint heat-maps from an independent detector as additional input channels alongside the RGB image.
- Encoder: Figure 2 presents examples from the synthetic dataset used to pre-train the encoder.
- Encoder: Synthetic paired hand images and ground-truth camera and hand parameters are created with the same generative model used as the decoder for encoder pre-training.
- Encoder: The synthetic dataset includes rendered hand appearances produced by transferring registered scan colors onto the model topology and rendering textured hands.
7. Training objective
Training combines 2D and 3D joint supervision with mask and parameter regularization losses, using GrabCut masks initialized from 2D joints to refine shape predictions.
- Training objective: The training objective combines 2D joint re-projection, 3D joint, hand mask, and model-parameter regularization losses.
- Training objective: The 2D joint re-projection loss aligns projected hand joints with ground-truth 2D coordinates using an L1 loss that accommodates annotation inaccuracies.
- Training objective: When 3D annotations are available, the 3D joint loss minimizes distance between generated and ground-truth 3D hand joints.
- Training objective: The novel hand mask loss accelerates convergence and refines shape predictions by penalizing projected vertices outside a pre-computed hand region.
- Training objective: The mask H is occlusion-aware, labeling pixels inside the hand region even when the hand is occluded.
- Training objective: Skin-segmentation methods cannot provide these masks because they are sensitive to occlusions.
- Training objective: GrabCut masks are initialized from 2D joint annotations by defining foreground, background, and undecided regions around the hand skeleton.
- Training objective: The parameter regularization loss reduces hand-parameter magnitudes to encourage physically plausible reconstructions and reduce mesh distortions.
8. Evaluation
The method is evaluated quantitatively on standard 3D and 2D hand-pose benchmarks and qualitatively on challenging images, including occlusion, clutter, blur, low resolution, and varying viewpoints. It achieves strong benchmark performance, outperforms 2D fitting, and produces visually plausible reconstructions.
- Datasets and conditions: The training set contains 31,729 images, including 15,000 with 3D joint annotations and 16,729 with 2D annotations only.The datasets are PANOPTIC, MPII+NZSL, and STEREO.
- Datasets and conditions: Evaluation spans third-person, egocentric, object-interaction, occluded, cluttered, and in-the-wild hand images.DEXTER+OBJECT evaluates object interaction, EGODEXTER evaluates egocentric interaction under occlusions and clutter, and MPII+NZSL evaluates blur, low resolution, viewpoints, and pose variation.
- In-the-wild evaluation: On MPII+NZSL, the method outperforms [68] by a substantial margin in 2D PCK and 2D joint error.The qualitative superiority is also visually confirmed on the challenging testing set.
- Ablation and fitting comparison: The proposed learning approach outperforms 2D fitting on all evaluated datasets, with the fitting strategy dropping considerably in 3D despite narrowing the gap in 2D.The comparison uses 3D PCK and joint error on STEREO, DEXTER+OBJECT, and EGODEXTER, plus 2D metrics on MPII+NZSL.
- Qualitative evaluation: Qualitative reconstructions on MPII+NZSL remain visually plausible in blurry, low-resolution, occluded, and viewpoint-varied images, while competing methods often fail to predict good 3D poses.The reconstructions are shown both overlaid on the input image and from alternative views.
9. Conclusion
The method predicts 3D hand pose and shape from a single RGB image using an end-to-end encoder and model-based decoder. It achieves state-of-the-art 3D pose results and plausible reconstructions in the wild.
- The network combines a deep convolutional encoder with a generative hand-model decoder and trains end-to-end using 2D and 3D joint annotations.The encoder predicts hand and view parameters; the decoder generates and re-projects the 3D hand.
- The method produces state-of-the-art results on 3D pose benchmarks and compelling 3D reconstructions on challenging images in the wild.
- Future improvements could add a hand appearance model for photometric-loss training or fine-tune MANO corrective blend shapes.