Source-linked AI summary
Towards Accurate Markerless Human Shape and Pose Estimation over Time
Yinghao Huang, Federica Bogo, Christoph Lassner, Angjoo Kanazawa, Peter V. Gehler, Ijaz Akhter, Michael J. Black
TL;DR
Markerless motion capture remains limited by assumptions about backgrounds, cameras, and motion priors. MuVS extends SMPLify with multi-view joints, CNN silhouettes, and a training-free DCT temporal prior, achieving comparable benchmark accuracy while producing realistic body meshes and promising monocular reconstructions.
Problem
Existing markerless motion-capture methods commonly rely on known static backgrounds or strong sequence-specific motion priors.
Method
MuVS extends SMPLify by fitting an SMPL body model to multi-view 2D joints and CNN-estimated silhouettes, with a robust DCT prior for temporal fitting.
Results
MuVS achieves error comparable to published results on HumanEva and Human3.6M while returning realistic meshes; it also gives promising monocular-video results.
Takeaways & Limitations
The method provides relatively accurate 3D pose and faithful human-body meshes for general activities with fewer restrictions than previous work.
Takeaways & Limitations
Future work identifies cluttered backgrounds, multiple people, extreme poses, and computational cost as remaining practical challenges.
Abstract
from arXiv · showhide
Existing marker-less motion capture methods often assume known backgrounds, static cameras, and sequence specific motion priors, which narrows its application scenarios. Here we propose a fully automatic method that given multi-view video, estimates 3D human motion and body shape. We take recent SMPLify \cite{bogo2016keep} as the base method, and extend it in several ways. First we fit the body to 2D features detected in multi-view images. Second, we use a CNN method to segment the person in each image and fit the 3D body model to the contours to further improves accuracy. Third we utilize a generic and robust DCT temporal prior to handle the left and right side swapping issue sometimes introduced by the 2D pose estimator. Validation on standard benchmarks shows our results are comparable to the state of the art and also provide a realistic 3D shape avatar. We also demonstrate accurate results on HumanEva and on challenging dance sequences from YouTube in monocular case.
I. INTRODUCTION
MuVS targets markerless motion capture beyond restricted environments by extending SMPLify with multi-view joints, CNN silhouettes, and a generic temporal prior. It achieves competitive benchmark accuracy, realistic meshes, and promising monocular reconstruction.
- Motivation: Existing markerless motion-capture methods commonly assume known static backgrounds or strong sequence-specific motion priors.Benchmark performance may depend on training with the same actor and motion used at test time.
- Approach: MuVS extends SMPLify from single-image estimation to multi-camera data using a realistic, low-dimensional SMPL body model.The model jointly represents human shape and pose while fitting observations from multiple views.
- Approach: CNN-estimated silhouettes are fitted alongside 2D joints, removing the need for a background image and improving mocap accuracy and mesh realism.The silhouette term complements joint-based fitting with body-contour evidence.
- Approach: A robust DCT temporal prior addresses occasional left-right joint swaps without requiring training data.The prior uses a low-dimensional basis to model human motion over time.
- Evaluation: MuVS has comparable error to published results on HumanEva and Human3.6M while producing more realistic meshes with fewer restrictions.The method is also evaluated on monocular video, including moving-camera, complex-background, and challenging dance sequences.
II. RELATED WORK
Prior work often separates 3D pose from realistic shape estimation or relies on restrictive inputs and initialization. MuVS combines explicit segmentation, temporal regularization, and SMPL to estimate pose and a detailed body mesh automatically from multi-view or monocular video.
- 3D pose estimation: Many prior methods infer 3D pose from 2D features without an explicit human body model.These approaches include multi-view pictorial structures, linear triangulation, latent-variable models, and pose sparsity priors.
- Body representations: Explicit intermediate body representations can assist pose estimation but often lack realistic shape detail.Examples include articulated models, pictorial structures, sum-of-Gaussians models, and triangulated meshes.
- Joint shape and pose estimation: Joint pose-and-mesh estimation often requires complex global optimization, known silhouettes, manual initialization, or a pre-scanned surface mesh.These requirements constrain fully automatic deployment.
- SMPLify and MuVS: SMPLify demonstrated automatic pose and convincing shape estimation from a single image by fitting projected SMPL joints to detected 2D joints.MuVS extends this model-fitting framework to multi-view and monocular videos.
- MuVS distinctions: Unlike related systems, MuVS uses explicit contours, a training-free DCT temporal prior, and SMPL’s holistic shape-pose representation to produce detailed realistic meshes.The framework therefore estimates joints and body shape together rather than returning only 3D joints or a coarse surface.
III. 2D JOINTS AND CONTOUR SEGMENTATION
The method uses automatically detected 2D joints and body segmentations as observations, first fitting SMPL independently across views and frames, then enforcing consistent shape and temporally coherent motion.
- Inputs: MuVS takes 2D body joints and CNN-estimated body segmentation as input.DeepCut is used for standard-dataset comparisons, while CPM is used for more complex Internet videos.
- Pipeline: The pipeline first extends SMPLify to multiple camera views and then extends it over time.It estimates 3D pose at each instant and combines views to obtain a consistent human shape across the sequence.
- Stage One: Per-frame fitting: During the first stage, a separate SMPL model is fit using information from all camera views at each time instant.Using as few as two views reduces ambiguities present in single-view fitting.
- Stage Two: Temporal fitting: The second stage estimates sequence-wide shape by taking the median of first-stage shape parameters before refining motion over consecutive frames.This separates consistent body shape estimation from frame-level pose initialization.
- Stage Two: Temporal fitting: A low-dimensional DCT basis regularizes 3D joint trajectories while minimizing projected joint error.Temporal smoothing reduces noise, spurious detections, and left-right ambiguities in CNN joint estimates.
A. Stage One: Per-frame Fitting
Stage one fits the SMPL body model independently across multiple camera views using 2D joints, robust priors, and silhouette evidence. A hierarchical strategy first obtains a coarse joint-based estimate, then refines it with contour fitting.
- Model representation: SMPL represents body shape and pose with separate identity and joint-rotation parameters.The model is written as M(β, θ; Φ), where β controls shape and θ controls pose.
- Multi-view fitting: The multi-view objective combines learned shape and pose priors with joint reprojection errors across camera views.The method removes several single-view priors because multiple views better constrain the solution.
- Multi-view fitting: Joint fitting uses detector confidence weights and a robust Geman-McClure error instead of standard squared error.The formulation accounts for inevitable detection noise and errors.
- Silhouette refinement: Silhouette fitting refines the joint-based estimate by comparing the rendered SMPL contour with the CNN-segmented body silhouette.The contour energy uses an L2 term for projected-model points and an L1 term for estimated-silhouette points, improving robustness to noise.
- Optimization strategy: Hierarchical optimization fits 2D joints first and then adds silhouette fitting to converge faster to better solutions.The first stage provides a coarse pose and shape estimate before contour refinement.
B. Stage Two: Temporal Fitting
Stage two estimates a sequence-consistent body shape and temporally regularized pose from consecutive frames. A low-dimensional DCT prior addresses noisy detections and left/right ambiguities such as leg swaps.
- Motivation: Single-frame fitting misses temporal relationships, while consecutive-frame processing can alleviate errors such as leg swaps.These errors arise from ambiguous or noisy 2D joint detection and may be difficult to correct frame by frame.
- Temporal fitting: The method retains the median shape from stage one and optimizes pose trajectories over consecutive frames using 2D joint fitting and DCT reconstruction.The second stage omits silhouettes for efficiency and uses the obtained median shape as initialization.
- Temporal prior: Each 3D joint-coordinate trajectory across N frames is encouraged to be approximated by a low-dimensional DCT basis.The trajectory is represented through DCT coefficients for each joint coordinate.
- Temporal prior: Temporal smoothness is applied directly to 3D SMPL joint locations rather than to image measurements.The temporal term uses the robust function introduced earlier.
C. Implementation Details
The implementation uses standard optimization and differentiable rendering tools, with fixed temporal-window and DCT settings. Figure 4 contrasts single-view SMPLify against multi-view MuVS using projected mesh contours.
- Implementation: The algorithm is implemented in Python with Powell’s dogleg method, OpenDR, and Chumpy.These tools are used for the two optimization problems.
- Implementation: Stage two processes 30 consecutive frames and uses the first 10 DCT components as temporal bases.The reported setup uses four views with 500x500 images on a 12GB-RAM, four-core PC.
- Visual comparison: Figure 4 places single-view SMPLify results in the left column and MuVS results in the right, with white contours showing projected meshes.The figure is intended to compare the two fitting settings visually.
V. EVALUATION
The evaluation uses HumanEva and Human3.6M to test the method’s design choices against established approaches. Both datasets are controlled laboratory collections with multiple subjects and motions.
- Datasets and comparisons: Experiments evaluate each method stage on HumanEva and Human3.6M against state-of-the-art methods.The datasets are used to assess the effectiveness of the design choices.
- Datasets and comparisons: HumanEva contains 4 subjects and 6 motions, while Human3.6M contains 11 subjects performing 15 motions.Both datasets were collected in controlled laboratory environments.
A. Ablation study
Ablation experiments on HumanEva show that additional views, silhouette fitting, and the DCT temporal prior improve reconstruction, with silhouettes also improving shape accuracy.
- Ablation study: Adding the silhouette term at the second optimization stage improves 3D pose estimation substantially but increases running time.
- Effect of multi-view: Adding more views consistently improves 3D pose estimation and helps eliminate incorrectly estimated orientations.
- Effect of silhouette fitting: Adding silhouettes consistently improves both 3D pose and shape estimation accuracy.
- Effect of DCT based temporal prior: The DCT temporal smoothness term boosts overall performance, but its effect diminishes as more views are added.
- Effect on shape estimation: Silhouette information and the DCT temporal prior consistently improve body shape estimation.With 3 views, average vertex-to-vertex distance is 12 mm without silhouettes and around 8 mm with them.
B. Quantitative comparison
MuVS is evaluated on HumanEva and Human3.6M, where it achieves strong multi-view joint accuracy while also producing realistic subject-faithful body meshes.
- HumanEva: With only 2 views in the General HumanEva case, MuVS is more accurate than other methods using all 3 views.
- HumanEva: With 3 views, MuVS improves HumanEva error over the second-best method, achieving 55.52 versus 63.25.
- HumanEva: MuVS achieves the lowest error in the Specific HumanEva case.
- Human3.6M: On Human3.6M, MuVS’s 3D joint estimation accuracy is quite close to the concurrent method [34].
- Human3.6M: Unlike methods focused only on 3D joints, MuVS simultaneously returns a realistic body shape model faithful to subjects and suitable for later modification and animation.
VI. POSE AND SHAPE FROM MONOCULAR VIDEO
MuVS extends markerless pose and shape estimation to monocular video while remaining fully automatic, with promising results under challenging conditions. The authors identify more complex scenarios and computational cost as directions for future work.
- MuVS applies to monocular video sequences without large modifications while remaining fully automatic.
- On videos without camera information, the method manually sets focal length and other imaging parameters to common values.
- Future work targets cluttered backgrounds, multiple people, extreme poses, and reduced computational costs.