PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Estimate 3D Hand Pose from Single RGB Images

Christian Zimmermann, Thomas Brox

arXiv:1705.01389v3cs.CV

TL;DR

Single-image RGB hand-pose estimation is difficult because missing depth creates ambiguities, while suitable large-scale annotated hand data are scarce. The paper learns a 3D articulation prior from synthetic data and combines it with detected 2D keypoints, achieving reasonable pose estimates and demonstrating sign-language recognition. The authors report competitiveness with depth-based approaches, while identifying real-world dataset limitations.

  • Problem

    Single-RGB 3D hand pose estimation lacks depth information and sufficiently variable large-scale annotated training data.

  • Method

    The system uses deep networks for segmentation, 2D keypoint localization, and 3D pose estimation with a canonical representation and synthetic training data.

  • Results

    The system produces reasonable 3D hand poses from 2D keypoints in real-world images and supports sign language recognition, with performance competitive to depth-based approaches.

  • Takeaways & Limitations

    Single-color-image 3D hand pose estimation is feasible without special equipment, including for sign language recognition.

  • Takeaways & Limitations

    Performance is limited by the lack of a large annotated real-world dataset with diverse pose statistics, and the NYU dataset cannot be used for this color-based approach.

Abstract

from arXiv · show

Low-cost consumer depth cameras and deep learning have enabled reasonable 3D hand pose estimation from single depth images. In this paper, we present an approach that estimates 3D hand pose from regular RGB images. This task has far more ambiguities due to the missing depth information. To this end, we propose a deep network that learns a network-implicit 3D articulation prior. Together with detected keypoints in the images, this network yields good estimates of the 3D pose. We introduce a large scale 3D hand pose dataset based on synthetic hand models for training the involved networks. Experiments on a variety of test sets, including one on sign language recognition, demonstrate the feasibility of 3D hand pose estimation on single color images.

1. Introduction

The paper targets full 3D hand pose estimation from single RGB images, avoiding specialized equipment and addressing severe ambiguity, articulation, occlusion, and limited training data. It combines deep networks with a canonical pose representation and synthetic data, demonstrating promising results and sign-language recognition.

  • Single-image 3D hand pose estimation is difficult because of ambiguity, strong articulation, and heavy self-occlusion.
  • Regular color cameras offer a more accessible setting than depth cameras, which are less available and reliable mainly indoors.
  • The system uses three deep networks for hand segmentation, 2D keypoint localization, and 3D pose estimation.
  • A canonical pose representation makes learning 3D pose from image evidence feasible.
  • A synthetic dataset with data augmentation addresses the lack of sufficiently variable hand-pose data with 3D ground truth.
  • The resulting system achieves promising qualitative and quantitative results and is also applied to sign language recognition.

2. Related work

Prior hand-pose methods primarily use depth data, explicit pose matching, or tracking from an initial pose. This paper instead develops a learning-based RGB approach that lifts 2D detections into 3D using learned structure.

  • The paper’s pipeline segments the hand, localizes 2D keypoints, and estimates 3D structure from their score maps.
  • Human 3D pose estimation commonly detects 2D keypoints before lifting them into 3D with databases, probabilistic models, statistical models, or volumetric prediction.
  • Earlier hand-pose research focused largely on depth images enabled by low-cost consumer depth cameras.
  • Existing approaches include edge-map matching, particle-swarm optimization, reinitializable tracking, and CNN-based 2D keypoint detection.
  • No prior learning-based approach addressed 3D hand pose estimation from a single color image.

3. Hand pose representation

The representation removes scale and translation ambiguity by normalizing hand coordinates and expressing them relative to a stable root keypoint.

  • The pose is represented by 3D coordinates for J = 21 hand keypoints.
  • Single-view 3D inference is ill-posed, so the method estimates normalized coordinates to obtain a scale-invariant structure.
  • Normalization sets the distance between a selected keypoint pair to unit length using a sample-dependent scale factor.
  • Relative coordinates are obtained by subtracting a defined root keypoint, producing a translation-invariant representation.
  • The palm keypoint is used as the root because it was the most stable landmark in experiments.

4. Estimation of 3D hand pose

The approach estimates normalized 3D hand coordinates from single images through segmentation, 2D keypoint localization, and a PosePrior network using a canonical representation. Training combines canonical-coordinate and transformation losses.

  • 4.1. Hand segmentation with HandSegNet: HandSegNet segments the hand to crop and normalize inputs, simplifying subsequent PoseNet learning.
  • 4.2. 2D hand keypoint estimation: PoseNet formulates 2D keypoint localization as score-map prediction, with each map encoding the likelihood of a keypoint at each image location.
  • 4.3. 3D hand pose with the PosePrior network: PosePrior predicts relative, normalized 3D coordinates from potentially noisy or incomplete 2D score maps by learning the manifold of plausible hand articulations.
  • 4.3. 3D hand pose with the PosePrior network: The method predicts coordinates in a canonical frame and separately estimates the rotation into that frame, enforcing invariance to global hand orientation.
  • 4.3. 3D hand pose with the PosePrior network: The canonical transformation is constructed through rotations aligning a selected keypoint and then adjusting rotation around the canonical y-axis.
  • 4.4. Network training: Training uses a squared L2 loss for canonical coordinates and another for the canonical transformation matrix, combined as an unweighted sum.

5. Datasets for hand pose estimation

The paper combines existing RGB hand-pose datasets with a synthetic rendered dataset because available data have limited variation, sample counts, or annotation completeness. The rendered data provide diverse training examples and detailed hand annotations.

  • 5.1. Available datasets: Stereo Hand Pose Tracking Benchmark contains 18,000 stereo pairs with 2D and 3D annotations for 21 keypoints, split into 15,000 training and 3,000 evaluation images.
  • 5.1. Available datasets: Dexter contains 3,129 color images of two operators manipulating a cuboid, but incomplete hand annotation limits it to cross-dataset generalization evaluation.
  • 5.1. Available datasets: The NYU Hand Pose Dataset is unsuitable for this color-based approach because it provides only registered color images.
  • 5.2. Rendered hand pose dataset: The authors created a synthetic dataset using Mixamo human models and animations rendered with Blender to complement existing datasets for training.
  • 5.2. Rendered hand pose dataset: The rendered dataset uses 20 characters and 39 actions, with disjoint character or action assignments between training and validation splits.
  • 5.2. Rendered hand pose dataset: The dataset contains 41,258 training and 2,728 evaluation images, each annotated with a 21-keypoint skeleton, 33 segmentation masks, visibility information, and camera coordinates.

6. Experiments

Experiments evaluate the system’s 2D keypoint detection, 3D pose lifting and learned pose prior, then demonstrate its use for sign language recognition. Results show strong cross-dataset behavior, improved generalization from diverse training data, reasonable pose recovery under missing keypoints, and favorable comparison with depth-based methods.

  • Experiments: The experiments evaluate PoseNet with oracle and predicted hand crops, 3D lifting representations, the learned pose prior, and sign language recognition.The study also compares the complete system with prior approaches on S-val.
  • 6.1. Keypoint detection in 2D: HandSegNet is reliable in most cases, but occasional segmentation failures cause the complete 2D keypoint prediction to fail.The difference between the median and mean errors indicates these failures are intermittent rather than typical.
  • 6.1. Keypoint detection in 2D: The method performs equally well on R-val and S-val, while Dexter is harder because it differs from training data and includes frequent cube occlusions absent from training.The training data includes self-occlusion but not the handled-cube occlusions found in Dexter.
  • 6.1. Keypoint detection in 2D: Jointly training PoseNet on R-train and S-train yields the best 2D keypoint results on Dexter.Training on the more diverse synthetic R-train generalizes better than training only on limited S-train.
  • 6.2.1 Pose representation: The proposed pose representation generalizes well and performs best among the evaluated lifting approaches.The direct approach struggles to separate global hand rotation from articulation, while nearest-neighbor retrieval does not generalize well to new samples.
  • 6.2.2 Pose prior: With no keypoints, the network predicts its canonical 3D pose prior; adding keypoints adjusts the pose and still permits reasonable estimates under simulated occlusion.The experiment demonstrates that the learned prior supplies plausible structure when 2D evidence is incomplete.
  • 6.2.3 Comparison to literature: On S-val, the RGB-based approach largely outperforms the cited depth-based approaches after depth is used only to rescale and shift predictions into metric coordinates.The paper notes that RGB prediction retains ambiguities in scale and absolute depth before this adjustment.
  • 6.3. Sign language recognition: The system is also used to train a classifier for 30 static gestures in the RWTH German Fingerspelling Database subset.The benchmark contains 35 gestures overall, including dynamic gestures excluded from this experiment.

7. Conclusions

The paper presents a learning-based system for estimating 3D hand pose from a single RGB image and contributes a synthetic dataset for training it. The learned pose prior produces reasonable poses from 2D keypoints, while performance remains constrained by the lack of large, diverse real-world annotated data.

  • 7. Conclusions: The paper presents the first learning-based system for estimating 3D hand pose from a single image.It uses a large synthetic dataset to train the system successfully.
  • 7. Conclusions: The network learns a 3D pose prior that predicts reasonable hand poses from 2D keypoints in real-world images.The system’s performance is reported as competitive with approaches using depth maps.
  • 7. Conclusions: Performance is mostly limited by the lack of a large-scale annotated real-world dataset with diverse pose statistics.This conclusion identifies the principal data-related scope boundary for further improvement.

Supplementary Material: Learning to Estimate 3D Hand Pose from Single RGB Images

The supplementary material describes HandSegNet’s architecture and training setup. It specifies initialization, optimization, iteration schedule, cropping, and the limited color augmentation used for segmentation.

  • Supplementary Material: HandSegNet is trained on R-train with batch size 8, ADAM, and softmax cross-entropy loss for 40000 iterations.The network is initialized with Wei et al.’s weights for layers 1–16.
  • Supplementary Material: Training uses random 256×256 crops from 320×320 images and random color-hue augmentation of 0.1.No other data augmentation is reported for HandSegNet.

B. PoseNet architecture and learning schedule

PoseNet is trained to predict keypoint score maps from normalized hand crops using a staged optimization schedule. Gaussian score-map targets and crop, keypoint, and contrast perturbations provide the training inputs.

  • B. PoseNet architecture and learning schedule: PoseNet uses batch size 8, ADAM, pretrained initial layers, random initialization for later layers, and L2 loss over 30000 iterations.The initial 16 layers use Wei et al.’s weights.
  • B. PoseNet architecture and learning schedule: The learning rate is 1·10^-4 for the first 10000 iterations, 1 · 10^-5 for the next 10000, and 1 · 10^-6 until training ends.The schedule decreases the learning rate in two stages.
  • B. PoseNet architecture and learning schedule: Ground-truth score maps use normal distributions with variance 25 pixels centered at visible keypoints and are normalized from 0 to 1.Invisible keypoints receive maps that are zero everywhere.
  • B. PoseNet architecture and learning schedule: PoseNet trains on axis-aligned hand crops resized to 256×256 pixels, with noise added to crop centers and bounding-box sizes.The crop is selected to contain all keypoints of one hand.
  • B. PoseNet architecture and learning schedule: Training perturbs each keypoint independently with zero-mean variance-1.5 Gaussian noise and applies contrast scaling between 0.5 and 1.0.These augmentations modify score-map coordinates and image contrast.

C. PosePrior architecture

PosePrior uses convolutional and fully connected streams to estimate viewpoint and 3D coordinates, with hand-side information included before the fully connected layers. The experiments also specify architectures and training settings for the related networks.

  • PosePrior streams use six convolutional layers followed by two fully connected layers with ReLU activations.
  • Hand-side information is concatenated with the flattened convolutional representation before the first fully connected layer.
  • P = 3 parameters are estimated for viewpoint and P = 63 for coordinate estimation.
  • GestureNet is trained with Adam on randomly cropped 256×256 images using softmax cross-entropy for 30,000 iterations.The initial learning rate is 1 · 10−4 and drops by one decade at 15,000 and 20,000 iterations.
  • Figure 10 presents results from the proposed approach.

F. NYU Hand Pose Dataset

The NYU Hand Pose Dataset is a common 3D hand-pose benchmark, but its registered color images contain color only at pixels with valid depth data. Consequently, corrupted color images make it unsuitable for evaluating color-only approaches.

  • The NYU Hand Pose Dataset is identified as a commonly used benchmark for 3D hand pose estimation.
  • Registered color images provide color only for pixels with valid depth data, producing corrupted images.Figure 11 shows two samples exhibiting these artifacts.
  • These artifacts make the dataset infeasible for approaches that use only color information.
Loading 1705.01389v3…