Source-linked AI summary
Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
Denis Tome, Chris Russell, Lourdes Agapito
TL;DR
The paper addresses ambiguous 2D detection and ill-posed 3D lifting from a single RGB image. It integrates a probabilistic 3D pose model into a multistage CNN that jointly refines 2D and 3D estimates, achieving state-of-the-art results on Human3.6M and improving both 2D and 3D errors.
Problem
Single-image 3D pose estimation must resolve ambiguous 2D joint locations and an ill-posed mapping from 2D landmarks to 3D poses.
Method
A multistage CNN jointly estimates 2D and 3D pose by integrating a pretrained probabilistic 3D pose model that lifts coordinates into physically plausible 3D poses.
Results
The method achieves state-of-the-art Human3.6M results, with 2D error reduction of 0.59 pixels over [44].
Takeaways & Limitations
The approach shows that 3D pose information can improve 2D landmark estimation while enabling efficient single-image 3D inference.
Takeaways & Limitations
Small 2D pose errors can produce drastically different 3D poses, although additional RGB annotations may mitigate these inaccuracies without further 3D data.
Abstract
from arXiv · showhide
We propose a unified formulation for the problem of 3D human pose estimation from a single raw RGB image that reasons jointly about 2D joint estimation and 3D pose reconstruction to improve both tasks. We take an integrated approach that fuses probabilistic knowledge of 3D human pose with a multi-stage CNN architecture and uses the knowledge of plausible 3D landmark locations to refine the search for better 2D locations. The entire process is trained end-to-end, is extremely efficient and obtains state- of-the-art results on Human3.6M outperforming previous approaches both on 2D and 3D errors.
1. Introduction
Single-image 3D pose estimation requires resolving ambiguous 2D landmark detection and ill-posed 2D-to-3D lifting. The paper jointly reasons about both tasks by integrating image-based predictions with a pretrained 3D pose model.
- Single-image 3D pose estimation is challenging because 2D joints are ambiguous under viewpoint, occlusion, clothing, body-shape, and illumination changes.
- The 2D-to-3D lifting problem is ill-posed because infinitely many 3D poses can be consistent with one image's 2D landmarks.
- The proposed approach jointly estimates 2D landmarks and full 3D pose to improve both tasks.
- A pretrained 3D human-pose model is integrated into a CNN layer that lifts 2D coordinates and imposes physically plausible 3D poses.
- The architecture uses independent 2D image annotations and 3D mocap data, enabling each training source to be augmented separately.
2. Related Work
Prior work either reconstructs 3D pose from known 2D joints, regresses 3D pose directly from images, or uses pipelines that estimate 2D pose before imposing 3D structure. These approaches face limitations involving 3D annotation, measurement uncertainty, temporal dependence, or unmodeled 2D errors.
- Methods using known 2D joints recover 3D pose through anatomical constraints, joint limits, or statistical body models learned from mocap data.
- NRSfM methods avoid 3D training data but require substantial camera movement throughout a monocular sequence for accurate reconstruction.
- Sparse 3D-basis methods reconstruct single-image pose, but their best results often depend on temporal smoothness from video sequences.
- Direct image-to-3D regression requires annotating images with ground-truth 3D poses, whereas pipeline methods must account for uncertainty in 2D measurements.
- Earlier pipeline methods estimate 2D landmarks first and then impose 3D spatial relationships through structured learning or graphical models.
3. Network Architecture
The network is a differentiable multistage CNN that jointly estimates 2D and 3D landmarks. A probabilistic 3D pose layer lifts 2D predictions and propagates skeletal information back to the 2D convolutional layers.
- The architecture is trained end-to-end to estimate 2D and 3D joint locations jointly.
- Each stage uses a probabilistic 3D pose layer to lift 2D poses into 3D and pass skeletal information to 2D convolutional layers.
- The 3D pose model is trained from 3D mocap data, while evaluation includes Human3.6M and qualitative results on MPII and Leeds.
4. Probabilistic 3D Model of Human Pose
The probabilistic pose model aligns 3D training poses, removes confounding variation, and represents the resulting pose space with compact and multimodal probabilistic models. Candidate selection and mixture modeling are used to capture both common and low-density pose regions.
- The model addresses limited pose diversity by removing confounding factors including ground-plane rotation, limb length, and left-right symmetry.
- Ground-plane rotations are optimized so the aligned poses are closely approximated by a compact low-rank Gaussian distribution.
- Training poses are jointly modeled through an average pose, orthonormal basis matrices, noise variance, per-pose rotations, and basis coefficients.
- The optimization alternates probabilistic PCA updates with rotation updates, increasing basis size progressively to avoid encoding rotation-related deformations prematurely.
- Because the aligned pose data are multimodal, the model uses a mixture of probabilistic PCA models rather than a single Gaussian.
- Greedy selection chooses representative pose samples by minimizing distances from aligned poses to their nearest selected sample before clustering and EM fitting.
5. A New Convolutional Architecture for 2D and 3D Pose Inference
The architecture repeatedly combines CNN-based 2D landmark predictions with projected, physically plausible 3D pose information, refining both 2D and 3D estimates across stages. A final lifting step produces the 3D pose estimate.
- The multistage architecture repeatedly fuses and refines 2D and 3D poses, then lifts the final 2D landmarks into 3D.It uses a second module for the final 3D estimate after six refinement stages.
- The architecture modifies Convolutional Pose Machines by adding probabilistic 3D pose and fusion layers that propagate skeletal information to 2D convolutional layers.This allows 2D predictions to benefit from 3D information encoded by the pose model.
- The projected 3D pose reduces 2D belief-map uncertainty at each stage while increasing 3D pose accuracy.Figure 3 visualizes the evolution of 2D skeletons, landmark beliefs, and relative per-landmark 3D error.
- Each stage predicts CNN-based 2D belief maps, lifts them into 3D with a pretrained probabilistic pose model, projects the pose back to 2D, and fuses both map sets.The projected maps encode 3D dependencies between body parts, while the fusion layer learns their combination for the next stage.
- For lifting, the method searches over rotations and solves the remaining scale and pose coefficients by linear least squares for each fixed rotation.The fast implementation checks 80 rotations and achieves the same average accuracy as the global-optimum search, at around 3,000 frames per second.
- The final 3D pose results substantially outperform competing Human3.6M methods under Protocol #1, with a 4.7mm average improvement over the closest competitor.The table reports 3D errors in millimeters, and some competitors use video rather than a single frame.
6. Experimental evaluation
On Human3.6M, the approach improves 2D landmark accuracy and outperforms competing methods across multiple 3D evaluation protocols, while remaining sensitive to 2D errors on other datasets.
- 2D Evaluation: 0.59 pixels: the full approach reduces mean 2D landmark error relative to the estimates of [44].This reduction is comparable to the 0.81-pixel improvement from Zhou et al. [50] to [44].
- 3D Evaluation: 4.76mm: the mixture model improves over Sanzari et al., the closest competitor, under Protocol #1.The evaluation averages Euclidean 3D joint errors over 17 joints, using five training subjects and two test subjects.
- 3D Evaluation: The model outperforms competing methods under Protocol #2 despite training on one fewer subject than that protocol specifies.Protocol #2 evaluates similarity-aligned 3D pose error averaged over 14 joints.
- 3D Evaluation: Almost 3mm: the method outperforms Bogo et al. [7] under Protocol #3, despite Bogo et al. using a detailed statistical 3D body model.Protocol #3 evaluates similarity-aligned error on 14 joints from frontal-camera sequences.
- Cross-dataset results: On MPII and Leeds, the probabilistic 3D pose model produces anatomically plausible poses, but small 2D errors can yield drastically different 3D poses.The passage identifies additional RGB 2D annotations as a way to mitigate these inaccuracies without further 3D data.
7. Conclusion
The conclusion presents 3D proposals as iterative refinements of 2D estimates, yielding better 2D accuracy and an efficient 2D-to-3D upgrade.
- Conclusion: 3D proposals iteratively refine 2D estimates, improving 2D pose accuracy while estimating 3D pose from a single image.The approach is framed as thinking in 3D even for 2D pose estimation.
- Conclusion: Approximately 1,000 frames a second: the three-model 2D-to-3D upgrade runs at this speed in CPU-based Python code.The conclusion identifies reliable real-time 3D pose estimation as a natural future direction.
1. Computing derivatives for back-propagation through our lifted model
The lifted model can be differentiated through projected 3D proposals using sub-gradients, while training also remains possible without back-propagating lifting derivatives. The derivation accounts for discrete rotations, stage-wise notation, and gradients propagated from belief maps to 2D landmark predictions.
- Training alternative: Similar results can be obtained by using the 3D lifting only in the forward pass, without back-propagating its derivatives through the rest of the network.Avoiding these derivatives may be computationally preferable because the added layers use custom Python-based derivatives.
- Gradient propagation: A recurrent multi-stage architecture requires layer-specific losses because vanishing gradients otherwise hinder effective training.The additional objective at each layer locally strengthens gradients, making learning concentrate on accurate 2D predictions in each layer.
- Notation and assumptions: The derivation simplifies notation by treating variables as belonging to one stage and selecting the best mixture component during the forward pass for sub-gradient computation.The text also unwraps landmark matrices and projection tensors into compact vector and matrix forms.
- Landmark gradients: The mapping from 2D landmarks Y to projected 3D proposals ˆY uses sub-gradients by fixing the currently selected discrete rotation.Discrete rotations make the mapping piecewise smooth; R is exhaustively minimized over and J denotes the number of bases.
- Belief-map gradients: Projected landmark coordinates induce Gaussian belief maps, so changes in predicted coordinates produce corresponding belief-map updates that propagate back to the original 2D landmarks.The update is computed using a one-pixel δx and the sub-gradients from the projected proposals to Y.
- Belief-map gradients: Belief-map updates decrease belief at an incorrect predicted location and increase it elsewhere using a Gaussian-shaped update scaled by the induced landmark gradient.The construction uses a negative step at the predicted location and positive updates in the corresponding Gaussian quadrant.