Source-linked AI summary
Using a single RGB frame for real time 3D hand pose estimation in the wild
Paschalis Panteleris, Iason Oikonomidis, Antonis Argyros
TL;DR
Monocular RGB hand-pose estimation remains difficult because of articulation, motion, appearance variation, and occlusion, despite progress with RGBD input. The paper combines deep-learning detection and 2D localization with generative 3D model fitting, and reports robust real-time performance that outperforms state-of-the-art solutions in real-world settings.
Problem
Full 3D hand pose estimation from conventional RGB remains difficult because of hand articulation, fast motion, uniform appearance, and occlusions, while the problem remains unsolved in full generality.
Method
A hybrid pipeline uses deep-learning hand detection and 2D joint localization followed by non-linear least-squares fitting of a generative 3D hand model.
Results
The method outperforms state-of-the-art solutions in accuracy and supports effective real-time 3D hand pose estimation in real-world situations.
Takeaways & Limitations
The approach provides robust monocular 3D hand pose estimation using conventional RGB input for real-time or interactive applications.
Takeaways & Limitations
The method assumes that the camera intrinsic parameters are known, including focal length, camera center, and distortion.
Abstract
from arXiv · showhide
We present a method for the real-time estimation of the full 3D pose of one or more human hands using a single commodity RGB camera. Recent work in the area has displayed impressive progress using RGBD input. However, since the introduction of RGBD sensors, there has been little progress for the case of monocular color input. We capitalize on the latest advancements of deep learning, combining them with the power of generative hand pose estimation techniques to achieve real-time monocular 3D hand pose estimation in unrestricted scenarios. More specifically, given an RGB image and the relevant camera calibration information, we employ a state-of-the-art detector to localize hands. Given a crop of a hand in the image, we run the pretrained network of OpenPose for hands to estimate the 2D location of hand joints. Finally, non-linear least-squares minimization fits a 3D model of the hand to the estimated 2D joint positions, recovering the 3D hand pose. Extensive experimental results provide comparison to the state of the art as well as qualitative assessment of the method in the wild.
1. Introduction
The paper addresses real-time full 3D hand pose estimation from conventional RGB images, a problem that remains unresolved in general despite progress from RGBD sensing and deep learning. It combines discriminative detection and 2D localization with generative 3D model fitting to estimate articulated hand states robustly.
- Markerless hand pose estimation remains unsolved in its full generality despite substantial progress.
- The method targets real-time recovery of 21 3D key points for one or more hands from a calibrated conventional RGB camera.
- Single-camera 3D hand pose estimation is difficult because hands have over 20 articulation parameters, fast motion, uniform appearance, and frequent self- or object-occlusion.
- The proposed hybrid approach uses deep networks for 2D hand detection and joint localization, then fits a generative hand model through non-linear least-squares optimization.The optimization is solved with the Levenberg-Marquardt optimizer.
- Experiments on annotated datasets and YouTube videos show higher accuracy than state-of-the-art solutions and effective 3D estimation in real-world situations.
2. Related work
Prior monocular RGB methods suffered from low accuracy, high runtime, and limited handling of occlusion, while RGBD methods benefited from depth sensing. The paper combines learned 2D components with generative fitting in a three-step pipeline for robust real-time monocular 3D hand pose estimation.
- RGBD sensors improved hand-pose estimation, while monocular color methods did not achieve comparable advancement.
- Earlier monocular RGB approaches had large runtime, low accuracy, or limited support for hand self-occlusions.
- The proposed pipeline performs hand detection, 2D key-point localization, and 3D pose estimation from color input.
- A retrained detector and a state-of-the-art 2D key-point method provide observations for generative 3D model fitting.
- Unlike Zimmermann and Brox, the method estimates absolute 3D hand position rather than a scale-normalized pose anchored at the palm center.The paper states that this improves applicability in real-world scenarios and reports higher accuracy and faster execution time.
- The authors claim the method is the only one known to robustly address monocular, real-time 3D hand pose estimation for real-world applications.
3. 3D hand pose from a single RGB frame
The method combines CNN-based hand detection and 2D joint localization with generative 3D hand-model fitting from a single calibrated RGB image. It estimates full hand pose in real time while handling plausible articulations through kinematic constraints.
- Hand detection: A CNN-based detector identifies hands in the RGB image and produces likelihood estimates and bounding-box coordinates for each detected hand.The implementation follows YOLO v2 to maintain low runtime while targeting high-quality detection with few false positives.
- 2D keypoint localization: Each cropped hand image is processed by a feed-forward CNN that estimates 2D joint locations using heatmaps.The detector outputs 21 heatmaps corresponding to 20 hand keypoints and one wrist point; right hands are mirrored because the network was trained on left hands.
- Limitations: Local YOLO-based detection does not use contextual information, which hinders distinguishing left from right hands from local image evidence alone.The paper notes that incorporating image context could improve handedness classification.
- 3D pose estimation: The hand model constrains solutions to plausible articulations with joint limits and represents the pose using 27 parameters and 26 degrees of freedom.The parameters include global translation and rotation plus finger-joint articulations, with skeleton keypoints corresponding to detected 2D joints.
- Assumptions and performance: The method assumes known camera intrinsics, including focal length, camera center, and distortion, and can incorporate multiple views as additional constraints.Experiments report that more camera views considerably improve hand-pose accuracy; for a single hand, the reference implementation achieves 18 fps on an Intel i7 CPU and NVIDIA GTX 1070 GPU.
4. Experiments
The method is evaluated quantitatively and qualitatively across three datasets, including direct comparison with Zimmermann and Brox. Results show strong performance on B2RGB and HIC, while monocular depth uncertainty remains the main error source.
- Datasets: Three datasets—SHD, B2RGB, and HIC—support quantitative and qualitative evaluation of the proposed method.SHD enables direct comparison with Zimmermann and Brox, although its ground truth accuracy is limited; B2RGB provides perfectly accurate synthetic ground truth.
- State-of-the-art comparison: The proposed method largely outperforms Zimmermann and Brox on B2RGB and HIC under the aligned 3D-output comparison.For SHD, Zimmermann and Brox perform better below 30mm, partly because their CNN was trained on that dataset and the hand dimensions differ from the model used here.
- Error analysis: Below 10mm accuracy is achieved on the X and Y axes for 90% of frames, whereas depth estimation along Z is the main uncertainty source.The comparison with stereo input on B2RGB further shows that adding a second camera considerably increases 3D hand-pose accuracy.
- Qualitative results: Qualitative results agree with the visual data across dataset frames and YouTube videos, even when generic camera calibration is used.The experiments also indicate that the hand model need not exactly match the observed hand to produce a reasonably accurate 3D pose.
5. Summary
The method estimates 3D hand pose from monocular RGB input in the wild and is qualitatively validated across videos and datasets. Future work targets adapting model metrics, adding color cues, and automatically estimating camera calibration.
- The method estimates 3D hand pose from monocular RGB input in the wild and compares favorably to the state of the art.It combines discriminative 2D joint localization with generative lifting into 3D.
- Figure 7 showcases complex poses with self-occlusions and object manipulation from YouTube videos, egocentric views, HIC, and SHD.Original hand views are cropped and shown in the top corner of each image.
- Future directions include automatic adjustment of model metrics, incorporating color cues in the final step, and automatic camera-calibration estimation.