Source-linked AI summary
Hand Keypoint Detection in Single Images using Multiview Bootstrapping
Tomas Simon, Hanbyul Joo, Iain Matthews, Yaser Sheikh
TL;DR
Markerless hand keypoint detection in unconstrained RGB images is limited by occlusion and scarce reliable annotations. The paper uses multiview bootstrapping to triangulate weak detections, reproject reliable 3D keypoints, and iteratively retrain a detector. It reports a realtime RGB detector and markerless 3D capture for challenging interactions.
Problem
Markerless hand keypoint detection in RGB images in the wild lacks reliable large-scale annotations because heavy occlusions make manual labeling difficult.
Method
Multiview bootstrapping triangulates weak detections across camera views, reprojects reliable 3D keypoints into difficult views, and iteratively retrains the detector.
Results
The method produces a realtime RGB hand keypoint detector with accuracy comparable to depth-based methods and supports markerless 3D capture of challenging interactions.
Takeaways & Limitations
Multiview geometry can provide external supervision for building richer annotations and improving detectors for occlusion-prone keypoints.
Takeaways & Limitations
The procedure requires conservative selection of reliable triangulations because geometrically consistent labeling errors can persist across iterations.
Abstract
from arXiv · showhide
We present an approach that uses a multi-camera system to train fine-grained detectors for keypoints that are prone to occlusion, such as the joints of a hand. We call this procedure multiview bootstrapping: first, an initial keypoint detector is used to produce noisy labels in multiple views of the hand. The noisy detections are then triangulated in 3D using multiview geometry or marked as outliers. Finally, the reprojected triangulations are used as new labeled training data to improve the detector. We repeat this process, generating more labeled data in each iteration. We derive a result analytically relating the minimum number of views to achieve target true and false positive rates for a given detector. The method is used to train a hand keypoint detector for single images. The resulting keypoint detector runs in realtime on RGB images and has accuracy comparable to methods that use depth sensors. The single view detector, triangulated over multiple views, enables 3D markerless hand motion capture with complex object interactions.
1. Introduction
The paper addresses the lack of markerless hand keypoint detectors for RGB images by using multiview bootstrapping to overcome severe occlusions and create training data. The resulting approach supports realtime single-view tracking and improves detectors using geometrically consistent annotations.
- No markerless hand keypoint detectors worked on RGB images in the wild, despite hands’ importance in everyday activities.
- Heavy occlusions make hand keypoints difficult to annotate manually, especially when joints are hidden from view.Occlusions arise from hand articulations, viewing angle, or grasped objects.
- Multiview bootstrapping uses a multi-camera setup because an unoccluded view often exists even when another hand image is severely occluded.
- Weak detectors localize subsets of keypoints in good views, while robust 3D triangulation filters incorrect detections and labels difficult views by reprojection.
2. Related Work
Prior RGB hand-pose methods were brittle or restricted to controlled settings, while depth sensing became dominant. Multiview bootstrapping addresses RGB annotation scarcity by generating large annotated datasets from a weak detector.
- Early RGB hand-pose methods relied on complex 3D models, strong priors, or controlled cues, and some required specialized hardware such as a colored glove.
- Commodity depth sensors shifted hand-pose research toward single-view depth-based generative, discriminative, and hybrid methods.
- Depth-based methods still encounter difficult hand-hand and hand-object interactions, and discriminative or hybrid approaches often rely heavily on synthetic data.
- Deep discriminative methods require large annotated training sets, but RGB datasets are difficult to synthesize because rendering realistic appearance and lighting is challenging.
- Multiview bootstrapping generates large annotated RGB datasets using a weak initial detector, enabling a realtime hand keypoint detector for in-the-wild images.
3. Multiview Bootstrapped Training
The training procedure runs a detector across multiview frames, robustly triangulates reliable keypoints, selects trustworthy frames, and retrains with reprojected labels. Finger-level robustness and conservative frame selection reduce incorrect training annotations.
- Initialization and detection: An initial detector is trained from labeled image-keypoint pairs and then applied independently to every view of each unlabeled multiview frame.
- Triangulation: Detections from multiple views are robustly triangulated in 3D, using confidence thresholds and reprojection error to identify inliers.
- Scoring and retraining: The algorithm scores and sorts triangulated frames, then retrains the detector with the N-best reprojections added to the original training set.
- Triangulation: Calibrated-camera reprojections of triangulated 3D points provide new 2D labels in views where the detector failed.
- Robustness: Entire fingers are triangulated simultaneously using average reprojection error across four landmarks, reducing false positives despite requiring all finger points to be correct in a view.
- Scoring and retraining: Because geometrically consistent errors can survive later iterations, the procedure conservatively selects reliable triangulations and manually verifies training frames.
- Scoring and retraining: About 100 selected frames per three minutes of video yield roughly 100 · V training samples, one for each unoccluded viewpoint.
4. Detection Architecture
The detector uses multistage confidence-map prediction built on VGG-19 features to localize hand keypoints, with a separate body-pose-based crop supplying the input patch.
- Keypoint Detection via Confidence Maps: Convolutional Pose Machines are modified to predict one confidence map for each hand keypoint, with locations taken from map maxima.The detector follows the CPM architecture, while using confidence maps whose peaks determine final keypoint positions.
- Keypoint Detection via Confidence Maps: VGG-19 convolutional features feed six sequential prediction stages that refine one score map per keypoint.The feature extractor runs through conv4_4, and later stages concatenate prior score maps with image features.
- Keypoint Detection via Confidence Maps: The final score maps are resized to the original patch size, and each keypoint is extracted at the pixel with maximum confidence.This produces the detector’s 2D keypoint locations from the final-stage maps.
- Keypoint Detection via Confidence Maps: Missing keypoint annotations are permitted when triangulation for that point fails.This supports training examples in which only a subset of keypoints has valid annotations.
- Hand Bounding Box Detection: The detector assumes a crop around the right hand, generated from wrist and elbow estimates supplied by body-pose models.Training uses a square crop sized relative to the tight hand bounding box.
5. When does Multiview Bootstrapping Work?
The paper analyzes when multiview verification can accept correct triangulations while rejecting spurious ones. It relates detector quality, camera count, inlier requirements, and bootstrapping performance through probabilistic models and evaluation curves.
- Analytical Conditions: The analysis targets the camera count or initial detector quality needed for multiview bootstrapping to work.The derived results quantify erroneous triangulations for varying detector quality and camera setups.
- Detector Quality: PCK is the probability that a predicted keypoint lies within distance threshold σ of its true location.For a particular keypoint p, the paper denotes this quantity as PCK_p.
- Analytical Conditions: Multiview bootstrapping requires a low false-positive rate when erroneous triangulations are accepted as valid.The analysis therefore distinguishes correct detections from spurious triangulations.
- False Triangulations: The false-triangulation model combines the probability of a spurious two-view match with the probability that additional views support it as inliers.For a given view pair, q_n is approximated as q_2 multiplied by the probability of at least n−2 supporting inliers among the remaining views.
- False Triangulations: Result 1 approximates the probability of at least one false triangulation supported by at least n inliers across V views.The count of qualifying view pairs is modeled as a random variable Y.
- True and False Positives: Result 2 approximates true- and false-positive rates for a keypoint across V views when at least n inliers are required.The result provides a basis for selecting camera configurations and inlier thresholds.
- Empirical Evaluation: Figure 7 evaluates PCK across bootstrapping iterations, datasets, joint types, and SGD training iterations.Improvements are smaller on the successfully manually annotated subset, while larger gains are usually observed on its complement.
- True and False Positives: Figure 6 varies camera views V, minimum inliers n, and detector quality PCK_σ to read off TP, FP, and target false-discovery rates.The plotted setups include V=5 and V=31 cameras.
6. Evaluation
The evaluation uses manually annotated in-the-wild datasets, synthetic and manual initial detectors, and multiview reconstruction tests. Bootstrapping improves detection across difficult views, while the RGB detector performs comparably to depth-based methods in several scenarios and supports challenging 3D capture.
- Datasets: The evaluation datasets comprise manually annotated MPII and NZSL images, split into 2,000 training and 800 testing hands.The datasets target everyday hand gestures and activities in in-the-wild imagery.
- Experimental setup: Three initial detectors use rendered data, manual annotations, or their combination before multiview bootstrapping.The rendered set contains around 11,000 synthetic hand examples.
- 2D keypoint evaluation: PCK evaluation averages accuracy over all keypoints while varying normalized distance thresholds on combined MPII and NZSL test images.The normalization uses 0.7 times the person’s head size, approximately the length of an outstretched hand.
- Robustness to view angle: Bootstrapping improves robustness across viewing angles, including side and fingertip-to-wrist views that produce greater self-occlusion.Robustness is measured by the percentage of reconstruction outliers binned by azimuth and elevation.
- Comparison to depth-based methods: On Tzionas et al.’s dataset, the RGB detector shows comparable performance to a depth-based method for single-hand and hand-object scenarios and runs in realtime on GPUs.Performance degrades for hand-hand interaction when one hand heavily occludes the other.
- 3D motion capture: Triangulating detections from 31 HD camera views reconstructs hands during tool use, sports, musical performances, and interactions among multiple people.The reconstruction uses the Mix 3 detector and supports sequences beyond those used during bootstrapping.
7. Discussion
The paper presents multiview bootstrapping as a way to build richer annotations and enable realtime RGB hand detection and markerless 3D capture. It also identifies fewer cameras and less controlled capture environments as future directions.
- Discussion: Multiview bootstrapping improves annotation quality and quantity by using multiview geometry as external supervision.The approach is presented as applicable to occlusion-prone keypoint detectors beyond hands, including body and face detectors.
- Discussion: The resulting system provides realtime hand keypoint detection for in-the-wild RGB video and markerless 3D capture during hand-object interactions and musical performances.These are presented as the paper’s two innovations.
- Future work: Future work targets robustness with fewer cameras and less controlled environments, including configurations using multiple cellphones.The stated goal is to create richer datasets that more closely reflect real-world capture conditions.