Source-linked AI summary
A Dual-Source Approach for 3D Pose Estimation from a Single Image
Hashim Yasin, Umar Iqbal, Björn Krüger, Andreas Weber, Juergen Gall
TL;DR
Accurate 3D annotations for unconstrained single-image pose estimation are difficult to acquire. The paper combines independent 2D-annotated images and 3D motion-capture data through 2D estimation and 3D pose retrieval, achieving state-of-the-art results and competitiveness despite substantial skeleton differences.
Problem
Accurate 3D pose annotations for unconstrained images are difficult and expensive to acquire, limiting training data for single-image 3D pose estimation.
Method
The method independently trains 2D pose estimation on annotated images and prepares motion-capture poses for robust 3D retrieval, then combines them during inference.
Results
The approach achieves state-of-the-art results when training data come from the same dataset and remains competitive when the two sources have substantially different skeleton structures.
Takeaways & Limitations
Independent 2D image annotations and motion-capture data can be collected separately, making the approach practical when image–3D pose annotation is infeasible.
Abstract
from arXiv · showhide
One major challenge for 3D pose estimation from a single RGB image is the acquisition of sufficient training data. In particular, collecting large amounts of training data that contain unconstrained images and are annotated with accurate 3D poses is infeasible. We therefore propose to use two independent training sources. The first source consists of images with annotated 2D poses and the second source consists of accurate 3D motion capture data. To integrate both sources, we propose a dual-source approach that combines 2D pose estimation with efficient and robust 3D pose retrieval. In our experiments, we show that our approach achieves state-of-the-art results and is even competitive when the skeleton structure of the two sources differ substantially.
1. Introduction
Single-image 3D pose estimation is difficult because accurate 3D annotations for unconstrained images are costly and impractical to acquire. The paper therefore combines independent image-based 2D annotations with laboratory motion-capture data.
- Accurate 3D pose annotations for realistic images require expensive hardware and controlled studio conditions.Marker-based systems require synchronized cameras, a studio environment, and attached markers, preventing realistic image capture.
- The proposed approach incorporates 2D and 3D information from two different training sources instead of image–3D pose pairs.
- The image source provides manually annotated 2D poses without imposing constraints on the environment where images are captured.The paper notes that Internet images can be annotated and used.
- The second source is accurate 3D motion-capture data collected in a laboratory, and the two sources are treated as independent.
2. Related Work
Prior monocular 3D pose methods are limited by constrained capture settings, auxiliary visual assumptions, difficult training data, or known motion types. This paper addresses the training-data problem by exploiting motion-capture data alongside 2D pose information.
- Multi-camera and depth-based methods require controlled or indoor environments, limiting their use in unconstrained scenarios.
- A 3D pictorial structure model requires a known 3D bounding volume and takes a few minutes per frame for inference.
- Several approaches require sufficient training images annotated with 3D poses, while others additionally rely on bounding volumes, boxes, or silhouettes.
- Existing methods estimate 3D poses from 2D poses using nearest-neighbor retrieval, sparse representations, evolutionary sampling, or anthropometric constraints.
- Some prior approaches iterate 2D and 3D estimation or use 2D pose estimators, whereas the proposed method is described as faster and more accurate than the cited approach.
- Action-specific motion priors constrain pose tracking by assuming that the motion type is known in advance.
3. Overview
The approach independently preprocesses motion-capture data for 3D retrieval and trains an image-based pictorial structure model for 2D pose prediction. At inference, the estimated 2D pose retrieves candidate 3D poses, which guide projection-error minimization.
- An image-based pictorial structure model predicts 2D poses, while motion-capture data is prepared for efficient retrieval of corresponding 3D poses.
- Retrieved poses alone are insufficient, so the final pose minimizes projection error while remaining close to the retrieved poses.
- The 2D pose-estimation models and 3D pose-estimation source code are publicly available.
4. 2D Pose Estimation
The 2D pose estimator uses a tree-structured pictorial structure model with joint-specific image evidence and kinematic constraints. Random forests provide unary joint regressors, while Gaussian mixtures model relative offsets between adjacent joints.
- The pictorial structure model represents 2D body pose as a tree graph whose vertices are joints and whose edges encode kinematic constraints.The tree structure allows efficient inference.
- Unary potentials model joint templates and the probability of each joint location, while binary potentials encode deformation costs relative to parent joints.
- Separate random-forest joint regressors are trained for each body joint to model the unary potentials.
- Binary potentials are modeled as Gaussian mixtures learned by clustering relative offsets between adjacent joints with k-means.
- The joint-set visualization distinguishes upper-body, lower-body, left-side, right-side, and all-joint configurations.
5. 3D Pose Estimation
The method links independent 2D image annotations and 3D motion-capture data by retrieving candidate 3D poses, then refining the pose through projection, retrieval, and anthropometric constraints. It uses multiple joint subsets and an approximate optimization strategy to improve robustness and efficiency.
- 3D pose retrieval: The approach uses an estimated 2D pose as a query to retrieve candidate 3D poses from independently prepared motion-capture data.The image source trains a 2D pose estimator, while the motion-capture source is normalized and projected into a 2D pose space for retrieval.
- 3D pose retrieval: Multiple joint subsets compensate for 2D estimation errors by allowing retrieval to rely on different body regions.The sets include all joints and upper-body, lower-body, left-side, and right-side subsets.
- 3D pose estimation: The final 3D pose minimizes an energy combining projection error, proximity to retrieved poses, and anthropometric constraints.The projection term evaluates agreement with predicted 2D joints, the retrieval term keeps the estimate near neighboring 3D poses, and the anthropometric term constrains limb lengths.
- 3D pose estimation: Weighting nearest neighbors by w_k,s improves 3D pose estimation accuracy while the retrieval error remains computed over all joints.The nearest-neighbor set depends on the selected joint subset, but the deviation penalty evaluates all joints.
- 3D pose estimation: The method estimates the projection separately for each joint subset before selecting a subset and optimizing the 3D pose.This approximate procedure avoids jointly optimizing the discrete subset variable with continuous pose and projection parameters.
- Iterative refinement: Two iterations are sufficient: a refined 2D pose is reused for retrieval before the final pose estimate is computed.The implementation reuses previously computed terms and initializes later optimization from the preceding iteration.
6. Experiments
The experiments evaluate dual-source 3D pose estimation on HumanEva-I and Human3.6M protocols, including parameter sensitivity, iterative refinement, motion-capture choices, and comparisons with existing methods.
- Evaluation setup: The evaluation uses 14-joint skeletons and reports 3D pose error after rigid alignment on HumanEva-I and Human3.6M.Testing follows established protocols with training data for learning and validation data as the test set.
- Parameter analysis: Nearest-neighbour weighting reduces pose estimation error across motion-capture datasets; the default settings are K = 256 and Kw = 64.Without weighting, more nearest neighbours are required; optimizing the retrieved poses gives lower errors than simply averaging them.
- Parameter analysis: The reconstruction term is essential when the selected joint set omits joints, while projection and anthropometric terms provide refinement and smaller additional gains.The experiments use ωp = 0.55, ωr = 0.35, and ωa = 0.065.
- Iterative refinement: A second iteration decreases pose error on nearly all sequences, whereas a third iteration provides no further reduction.This comparison includes both weighted and unweighted nearest-neighbour retrieval.
- Comparison with state-of-the-art: The approach outperforms competing methods in average 3D pose error on HumanEva-I when using HumanEva-I motion-capture data.Its first iteration requires 19 seconds per image, with an additional 8 seconds for a second iteration.
- Cross-dataset evaluation: Using CMU motion-capture data raises error because the skeleton and datasets differ, but performance remains low or competitive relative to other methods.More than 60% of the error increase after retargeting can be attributed to skeleton differences.
7. Conclusion
The paper presents a dual-source approach that combines a MoCap dataset with 3D poses and independently annotated 2D image data. It achieves state-of-the-art results while requiring fewer assumptions and avoiding infeasible 3D annotations for training images.
- The approach uses a MoCap dataset with 3D poses and images with annotated 2D poses as independent training sources.The sources can be collected separately rather than pairing each training image with an accurate 3D pose.
- The method achieves state-of-the-art results when training and test data come from the same dataset.
- Separately collecting 2D image annotations and motion-capture data makes the approach practical because accurate 3D annotations for images are often infeasible.