Source-linked AI summary
Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video
Xiaowei Zhou, Menglong Zhu, Spyridon Leonardos, Kosta Derpanis, Kostas Daniilidis
TL;DR
The paper addresses ambiguous 3D full-body pose estimation from monocular video, including cases where 2D joints are unknown. It combines sparse 3D geometry, CNN heat maps, temporal smoothness, and EM inference, achieving higher accuracy than state-of-the-art baselines on Human3.6M and outperforming a public 2D baseline on PennAction.
Problem
The paper addresses the challenge of recovering 3D full-body human pose from monocular RGB image sequences, where 3D geometry is ambiguous and 2D, 3D, and temporal information are difficult to integrate.
Method
The framework combines CNN-based 2D joint heat maps, a sparse 3D pose dictionary, temporal smoothness, and an EM algorithm that can treat 2D joint locations as latent variables.
Results
The proposed approaches achieve greater 3D pose estimation accuracy than state-of-the-art baselines, while the framework achieves the best results on most Human3.6M actions.
Takeaways & Limitations
3D geometric priors and temporal coherence can improve both 3D reconstruction and 2D joint localization within the proposed framework.
Takeaways & Limitations
The EM inference problem is nonconvex and therefore requires a suitably chosen initialization, despite convergence of its non-increasing objective.
Abstract
from arXiv · showhide
This paper addresses the challenge of 3D full-body human pose estimation from a monocular image sequence. Here, two cases are considered: (i) the image locations of the human joints are provided and (ii) the image locations of joints are unknown. In the former case, a novel approach is introduced that integrates a sparsity-driven 3D geometric prior and temporal smoothness. In the latter case, the former case is extended by treating the image locations of the joints as latent variables. A deep fully convolutional network is trained to predict the uncertainty maps of the 2D joint locations. The 3D pose estimates are realized via an Expectation-Maximization algorithm over the entire sequence, where it is shown that the 2D joint location uncertainties can be conveniently marginalized out during inference. Empirical evaluation on the Human3.6M dataset shows that the proposed approaches achieve greater 3D pose estimation accuracy over state-of-the-art baselines. Further, the proposed approach outperforms a publicly available 2D pose estimation baseline on the challenging PennAction dataset.
1. Introduction
The paper targets 3D full-body pose recovery from monocular video by combining image-based 2D localization, 3D geometry, and temporal information. It jointly handles pose reconstruction and uncertain 2D joint localization.
- Monocular 3D pose recovery is ambiguous and further complicated by appearance variation, viewpoint changes, occlusion, and self-occlusion.
- Existing 2D pose methods provide strong image-based localization, but typically do not leverage 3D pose geometry.
- The paper identifies seamless integration of 2D, 3D, and temporal information as a limited but promising research direction for handling model and measurement uncertainty.
- The proposed framework jointly reasons about 2D part estimates and 3D reconstruction, imposing temporal smoothness on 3D pose and viewpoint parameters.
- CNN heat maps and a sparse 3D pose model are combined within an EM framework to recover the 3D pose sequence.
- Unlike two-stage approaches, the method treats 2D joint locations as latent variables, allowing 3D geometry to assist localization and uncertainty to be handled statistically.
2. Models
The model represents 3D poses sparsely over a learned motion-capture dictionary, projects them through a weak-perspective camera, and links image evidence through CNN heat maps. Sparse and temporal penalties regularize the inferred parameters.
- 2.1. Sparse representation of 3D poses: Each 3D pose is represented as a linear combination of predefined basis poses learned from motion-capture training data.
- 2.1. Sparse representation of 3D poses: An overcomplete pose dictionary uses sparse coefficients to model large human-pose variability.
- 2.2. Dependence between 2D and 3D poses: A weak-perspective camera model maps the 3D pose to 2D joint locations using camera rotation and translation parameters.
- 2.2. Dependence between 2D and 3D poses: The observation model assigns each 2D joint location a Gaussian distribution centered at the projection of its corresponding 3D joint.
- 2.3. Dependence between pose and image: Image-conditioned joint distributions are modeled as heat maps produced by CNN mappings from each image to joint-location probabilities.
- 2.4. Prior on model parameters: The parameter penalty combines coefficient sparsity with first-order temporal smoothness on pose coefficients and rotations.
3. 3D pose inference
3D pose inference treats provided 2D joints directly or treats unknown joint locations as latent variables. The approach alternates parameter updates, and the latent-joint case uses EM to replace difficult marginalization with expected joint locations.
- Inference cases: The framework distinguishes inference with provided 2D joint locations from inference where those locations are unknown latent variables.The unknown-joint case marginalizes the 2D poses during estimation.
- Given 2D poses: Given 2D poses, penalized maximum likelihood estimation recovers the pose parameters θ.The parameters are optimized by alternating updates to C, R, and T.
- Given 2D poses: The block coordinate descent updates C with accelerated proximal gradient, R with manifold optimization, and T with a closed-form solution.The C objective includes an ℓ1 penalty, while rotations are constrained to SO(3).
- Initialization and convergence: The given-2D-pose algorithm converges because each update is non-increasing in the objective, but its nonconvexity makes initialization important.A convex relaxation and robust estimation procedure initialize the parameters, using heat-map maxima when 2D poses are unknown.
- Unknown 2D poses: For unknown 2D poses, EM computes the expected penalized log-likelihood and then updates pose parameters using the given-2D-pose algorithm.Because the relevant probabilities factor over joints, each joint expectation can be approximated separately by sampling the pixel grid.
- Algorithms: The complete inference procedures are summarized in separate algorithms for block coordinate descent with known 2D poses and EM with unknown 2D poses.Both procedures iterate until convergence after parameter initialization.
4. CNN-based joint uncertainty regression
A fully convolutional CNN predicts joint-location uncertainty maps from cropped subject images. It jointly predicts all joint distributions using full-body information, with training augmentations and dense heat-map outputs.
- Network design: A fully convolutional neural network regresses distributions for all joints simultaneously from an input image.Jointly predicting the distributions uses full-body information rather than one network per joint.
- Training data: The training labels are multi-channel heat maps, with one channel representing the image-location uncertainty distribution for each joint.Images are cropped around the subject, resized to 256×256 pixels, and normalized by subtracting dataset RGB means.
- Optimization: Training minimizes l2 loss between predicted and target heat maps using stochastic gradient descent with momentum 0.9.Random crop shifts and channel-wise RGB noise provide data augmentation.
- Architecture: The network uses seven convolutional layers with 5×5 filters, ReLU layers, three early 2×2 max-pooling layers, and a final 1×1×p convolution.The final layer provides dense predictions for all joints.
- Inference input: At test time, a presumed subject bounding box is cropped from each frame and passed through the CNN to produce heat maps for every joint.The resulting heat maps represent the predicted joint-location distributions used by inference.
5. Empirical evaluation
The empirical evaluation tests 3D reconstruction with known and unknown 2D poses on Human3.6M, and 2D pose estimation on in-the-wild PennAction videos. Across these settings, the proposed method benefits from geometric pose priors, temporal smoothness, uncertainty modeling, and EM optimization.
- Datasets and protocols: Evaluation used Human3.6M for 3D pose recovery and PennAction for challenging in-the-wild 2D pose estimation.Human3.6M contains synchronized 2D–3D pose data; PennAction provides frame-level 2D annotations for 2326 consumer videos.
- Known 2D poses: With known 2D poses, the proposed method outperformed the NRSFM baseline on Human3.6M, especially under the dataset’s small camera motion.Artificial camera rotation substantially improved NRSFM, while the authors conclude that a structure prior is critical when neighboring views provide weak geometric constraints.
- Unknown poses: Human3.6M: With unknown 2D poses, the proposed method achieved the best results on most Human3.6M actions, except “walk” and “walk together”.Its across-action standard deviation was 28.75, compared with 37.80 for Tekin et al.
- Ablations: Removing temporal smoothness significantly increased average error, while unrestricted action-agnostic pose priors also increased estimation error.The latter effect is attributed to greater 3D reconstruction ambiguity when the pose prior is not restricted to an action class.
- Qualitative results: Human3.6M examples show EM correcting detector errors caused by occlusion, left-right ambiguity, and other uncertainty by leveraging pose and temporal priors.The visualization compares heat maps, greedy 2D localization, EM-estimated 2D poses, and novel-view 3D poses.
- Unknown poses: PennAction: On PennAction, CNN initialization alone outperformed the retrained Yang–Ramanan baseline, and EM optimization further improved results by integrating geometric and smoothness priors.The proposed approach recovered poses across varied subjects and viewpoints and avoided the baseline’s tree-model “double-counting” problem through a holistic 3D pose prior.
6. Summary
The presented video framework synthesizes deep learning-based 2D part regression, sparsity-driven 3D reconstruction, and temporal smoothness. Experiments show that geometric priors and temporal coherence improve both 3D reconstruction and 2D joint localization, while the framework remains modular.
- The framework combines a deep learning-based 2D part regressor, sparsity-driven 3D reconstruction, and a 3D temporal smoothness prior.
- This joint design combines discriminative 2D detectors, expressive 3D pose models, and temporal regularization across video.
- 3D geometric priors and temporal coherence improve both 3D reconstruction and 2D joint localization.
- Alternative joint detectors, pose representations, and temporal models can replace the framework’s original components.
- MATLAB code, HumanEva I evaluation, demonstration videos, and supplementary materials are publicly available.