Source-linked AI summary
MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Prior
Xiaowei Zhou, Menglong Zhu, Georgios Pavlakos, Spyridon Leonardos, Kostantinos G. Derpanis, Kostas Daniilidis
TL;DR
MonoCap addresses markerless 3D full-body pose recovery from monocular RGB sequences, where integrating uncertain 2D evidence with 3D geometry and temporal information is difficult. It combines CNN heat maps, a sparse 3D pose representation, and EM-based inference, achieving greater accuracy than state-of-the-art baselines while using separate 2D and 3D training sources.
Problem
Markerless monocular 3D pose recovery must infer 3D geometry from 2D appearance while integrating temporal information and uncertainty.
Method
MonoCap treats 2D joint locations as uncertain latent variables, models 3D poses sparsely, and estimates pose parameters with an Expectation-Maximization framework.
Results
MonoCap achieves greater accuracy than state-of-the-art baselines on benchmark datasets.
Takeaways & Limitations
The framework can train its CNN and 3D pose dictionary from separate 2D and 3D sources and applies to in-the-wild MPII images.
Takeaways & Limitations
Failures are mostly associated with heavy occlusion, left-right symmetry ambiguities, overlapping people, and rare poses beyond the learned dictionary.
Abstract
from arXiv · showhide
Recovering 3D full-body human pose is a challenging problem with many applications. It has been successfully addressed by motion capture systems with body worn markers and multiple cameras. In this paper, we address the more challenging case of not only using a single camera but also not leveraging markers: going directly from 2D appearance to 3D geometry. Deep learning approaches have shown remarkable abilities to discriminatively learn 2D appearance features. The missing piece is how to integrate 2D, 3D and temporal information to recover 3D geometry and account for the uncertainties arising from the discriminative model. We introduce a novel approach that treats 2D joint locations as latent variables whose uncertainty distributions are given by a deep fully convolutional neural network. The unknown 3D poses are modeled by a sparse representation and the 3D parameter estimates are realized via an Expectation-Maximization algorithm, where it is shown that the 2D joint location uncertainties can be conveniently marginalized out during inference. Extensive evaluation on benchmark datasets shows that the proposed approach achieves greater accuracy over state-of-the-art baselines. Notably, the proposed approach does not require synchronized 2D-3D data for training and is applicable to "in-the-wild" images, which is demonstrated with the MPII dataset.
1 INTRODUCTION
MonoCap targets markerless 3D full-body pose recovery from monocular RGB sequences by integrating uncertain 2D detections, 3D pose structure, and temporal information. Its framework combines CNN-based image evidence with sparse 3D reconstruction and reports improved accuracy and broader training-data flexibility.
- Problem: MonoCap recovers 3D full-body human pose from markerless monocular RGB image sequences.The approach avoids multiple cameras, reflective markers, and depth sensors.
- Motivation: The central challenge is integrating 2D appearance, 3D geometry, temporal information, and uncertainty from image-driven estimates.Prior 2D successes do not leverage 3D pose geometry, while combining all three information sources has received limited attention.
- Approach: MonoCap jointly reasons about CNN-based 2D joint heat maps and sparse 3D pose reconstruction within an Expectation-Maximization framework.Temporal smoothness is imposed on 3D pose and viewpoint parameters to improve robustness to detector error, occlusion, and reconstruction ambiguity.
- Contributions: The method uses separate annotated 2D imagery and 3D MoCap data rather than requiring synchronized 2D-3D training pairs.The paper demonstrates this flexibility with in-the-wild MPII imagery and a separate 3D pose dictionary source.
- Contributions: The extended work adds a perspective camera model, a corresponding optimization algorithm, a state-of-the-art 2D pose detector, and broader empirical evaluation.The manuscript identifies these as extensions over the preliminary CVPR 2016 version.
2 MODELS
MonoCap models 3D poses with a sparse learned dictionary, projects them into 2D under orthographic or perspective cameras, and combines geometric likelihoods with CNN image evidence. Temporal regularization constrains pose coefficients and rotations, while unknown 2D joints are treated as latent variables and marginalized.
- 3D pose model: A 3D pose is represented as a linear combination of predefined basis poses learned from training poses in a MoCap dataset.The representation uses an overcomplete dictionary with sparse coefficients to model large human-pose variability.
- Camera models: Under the orthographic model, 2D joint locations depend on 3D poses through camera rotation and translation parameters.W_t denotes the 2D pose, while R_t and T_t denote camera rotation and translation.
- Probabilistic model: Given 3D poses and camera parameters, each 2D joint is modeled with a Gaussian distribution centered at the projected 3D joint.The precision parameter ν represents the inverse variance of the observation model.
- Model extensions: Relative to earlier work, the model adds perspective projection, latent rather than fixed 2D poses, and temporal smoothness constraints.These extensions distinguish the formulation from the cited previous loss function.
- Camera models: The perspective model uses homogeneous 2D coordinates, 3D rotation and translation, and per-joint depths, with the root depth constrained to prevent a trivial zero solution.The root-joint depth constraint addresses scale ambiguity in the perspective formulation.
- Probabilistic model: When 2D poses are unknown, they are treated as latent variables and marginalized during estimation, while CNN heat maps provide image-conditioned joint-location likelihoods.The CNN maps each image to a heat map for every joint, representing its likelihood over image locations.
- Temporal prior: The parameter penalty promotes sparse pose coefficients and first-order temporal smoothness in both pose coefficients and rotations.The weights α, β, and γ control the penalty terms; translation smoothness was not included because it showed no obvious empirical difference.
3 3D POSE INFERENCE
3D pose inference alternates updates of pose parameters under orthographic or perspective projection, using block coordinate descent when 2D joint locations are given and EM when they are unknown. The procedure uses sparse pose representations, specialized parameter updates, and initialization because the optimization is nonconvex.
- Given 2D joint locations W, the model parameters θ are recovered by penalized maximum likelihood estimation.
- Block coordinate descent alternately updates parameter blocks while fixing the others.The updates include C, R, T, and, for perspective projection, Z.
- The C update combines differentiable objectives with an ℓ1 penalty and is solved by accelerated proximal gradient, with global optimality guaranteed by convexity.
- Rotation updates optimize variables constrained to SO(3) using manifold optimization with a trust-region solver.
- The inference algorithm terminates after objective convergence, but its nonconvex optimization requires suitably chosen initialization.A convex relaxation is used for initialization, and dictionary learning supplies pose dictionaries with empirically selected sizes.
- When 2D poses are unknown, W is treated as a latent variable and marginalized through an EM algorithm.The expectation step computes E[W|I,θ′], while the maximization step updates θ using that expectation.
- Each joint-location expectation is efficiently approximated by sampling over the pixel grid because the relevant probabilities factor into marginal terms.
4 CNN-BASED JOINT UNCERTAINTY REGRESSION
MonoCap learns joint-location uncertainty distributions from images with a fully convolutional CNN. Its multi-channel heat maps represent the spatial uncertainty of all joints while using full-body context, with subject bounding boxes assumed at test time.
- A fully convolutional neural network regresses all p joint uncertainty distributions simultaneously from an input image.Each output channel is a heat map representing one joint’s image-location uncertainty distribution.
- Training labels are multi-channel heat maps whose channels encode Gaussian uncertainty centered at annotated joint locations.
- The adopted Stacked Hourglass architecture combines downsampling, upsampling, contextual processing, and a second hourglass for heat-map refinement.
- At test time, a bounding box around the subject is assumed, and the cropped image patch is used to predict heat maps for all joints.
5 EMPIRICAL EVALUATION
Experiments across controlled and realistic datasets evaluate MonoCap using per-joint and reconstruction errors, showing strong performance and the value of its modeling choices.
- The evaluation uses Human3.6M, Human Eva I, KTH Football II, and MPII, with quantitative evaluation on the first three datasets and qualitative evaluation on MPII.
- Per-joint error depends on absolute pose, while reconstruction error evaluates structure after similarity alignment, ignoring scale and rigid pose.Scale and depth ambiguities are inherent to monocular reconstruction; the method scales output to the average training-subject limb length for fair comparison.
- 3D pose reconstruction with known 2D pose: With known 2D joints, the perspective model substantially outperforms the orthographic model in per-joint error, while their reconstruction errors differ much less.The gap is attributed mainly to more accurate rigid-pose estimation under perspective projection.
- 3D pose reconstruction with known 2D pose: MonoCap outperforms the NRSFM baseline because stationary-camera videos provide insufficient inter-frame geometric constraints for NRSFM.The proposed approach also improves over the initialization approach when temporal smoothness is used; given 2D joints, the orthographic variant reduces to that initialization plus smoothness.
- 3D pose reconstruction with unknown 2D pose: With unknown 2D poses, MonoCap achieves much lower average error than baselines on most actions and outperforms SMPLify in reconstruction error.The generic-detector configuration remains competitive, while fine-tuned 2D detection and action-specific 3D dictionaries produce a remarkable improvement.
- Human3.6M online test set: On the H36M NOS10 online test set, MonoCap achieves significant improvements for all actions compared with previous approaches.The evaluation uses the fine-tuned hourglass detector and action-specific pose dictionaries.
- Ablative analysis: Ablation results show that omitting EM significantly increases both 3D and 2D errors, while removing temporal smoothness also increases error, especially with noisier 2D detections.The perspective model consistently outperforms the orthographic model, with a larger gap for per-joint than reconstruction error.
- Ablative analysis: MAP estimation reaches 79.3 mm mean per-joint error, worse than EM at 77.8 mm with the fine-tuned detector.
5.4 HumanEva I
On HumanEva I, the proposed approach achieves the best overall accuracy across evaluation sequences, although no method dominates every sequence.
- Walking and jogging sequences from camera C1 of all subjects were used for evaluation.The 2D detector was fine-tuned per action, and action-specific pose dictionaries were learned separately for each subject.
- The proposed approach achieves the best overall accuracy across the HumanEva I evaluation sequences.Compared methods do not dominate across all individual sequences.
- Higher accuracies were generally obtained on HumanEva I than on Human3.6M for all approaches.The passage attributes this pattern to training-test overlap and lower pose variability.
5.5 KTH Football II
On KTH Multiview Football II, MonoCap improves state-of-the-art 3D part localization and also produces qualitatively reasonable single-image reconstructions, with failures concentrated in difficult visual and pose cases.
- The proposed approach achieves improved accuracy over the state-of-the-art on KTH Football II according to mean PCP scores.The evaluation uses PCP to measure 3D part localization.
- MonoCap produces visually reasonable 3D reconstructions from single images across varied activities and viewpoints.The pose dictionary is learned from another dataset, while MPII supplies the 2D detector.
- Failures are mostly associated with heavy occlusion, left-right symmetry ambiguities, overlapping people, and extremely rare 3D poses.Rare poses can exceed the representational capacity of the learned pose dictionary.
- Dictionary size trades reconstruction accuracy against computation time: 44.4 mm required 488 seconds at size 128, versus 48.1 mm in 91 seconds at size 32.Intermediate sizes 64 and 96 achieved 46.0 mm in 197 seconds and 45.6 mm in 317 seconds, respectively.
6 SUMMARY
MonoCap combines learned 2D evidence, sparse 3D geometry, and temporal smoothness in one monocular pose-estimation framework. The integrated priors can improve both 3D reconstruction and 2D part localization, including correction of uncertain heat-map detections.
- MonoCap synthesizes a deep learning-based 2D part regressor, sparsity-driven 3D reconstruction, and a 3D temporal smoothness prior.The framework targets monocular images or video.
- The 3D pose prior can correct errors in 2D heat maps after joint 3D pose reasoning.Comparative MPII examples contrast greedy heat-map localization with the proposed EM-estimated 2D pose.
- MPII failure cases are linked mainly to occlusion, symmetry ambiguity, overlapping people, and rare poses beyond the learned dictionary.These limitations constrain the robustness of the qualitative in-the-wild results.
- Temporal coherence and 3D geometric priors improve both 3D reconstruction and 2D part localization.The framework can integrate alternative part detectors, pose representations, and temporal models by replacing its original components.
PROOF OF EQUATION (19)
The supplied passages identify components of the notation used in the proof but do not provide the equation’s stated derivation.
- W is formed by stacking all W_t terms.
- M(θ) is formed by stacking all R_t terms.
- A squared term is omitted from the presentation for brevity.
DERIVATION OF EQUATION (20)
Equation (26) contains the quantities M, W, and dW, with M specified as a constant.
- Equation (26) includes M, W, and dW.
- The supplied passage does not state the derivation or operational role of these quantities.
- M is defined as a constant.