Source-linked AI summary
MonoPerfCap: Human Performance Capture from Monocular Video
Weipeng Xu, Avishek Chatterjee, Michael Zollhöfer, Helge Rhodin, Dushyant Mehta, Hans-Peter Seidel, Christian Theobalt
TL;DR
Marker-less full-body performance capture from monocular video remains difficult because occlusions, missing depth, articulation, and clothing deformation make reconstruction ill-posed. MonoPerfCap combines sparse 2D/3D pose detections, batch-based motion optimization with a low-dimensional trajectory prior, and automatic silhouette refinement. It reports temporally coherent reconstruction in challenging scenes and applications including video editing and free-viewpoint video.
Problem
Monocular human performance capture is an ill-posed problem because strong occlusions and missing depth complicate reconstruction of articulated bodies with non-rigid clothing.
Method
MonoPerfCap jointly optimizes batch motion from sparse 2D/3D pose detections under a low-dimensional trajectory prior and refines surfaces using automatically extracted silhouettes.
Results
The approach produces temporally coherent 3D motion and medium-scale surface reconstructions for challenging monocular video sequences, including rotations, sitting, and object interaction.
Takeaways & Limitations
The reconstructions support video editing and free-viewpoint video from single RGB video, applications previously infeasible with this input.
Takeaways & Limitations
The method requires a person-specific actor rig in preprocessing, and automatically extracting such a rig from monocular video with general motion remains unsolved.
Abstract
from arXiv · showhide
We present the first marker-less approach for temporally coherent 3D performance capture of a human with general clothing from monocular video. Our approach reconstructs articulated human skeleton motion as well as medium-scale non-rigid surface deformations in general scenes. Human performance capture is a challenging problem due to the large range of articulation, potentially fast motion, and considerable non-rigid deformations, even from multi-view data. Reconstruction from monocular video alone is drastically more challenging, since strong occlusions and the inherent depth ambiguity lead to a highly ill-posed reconstruction problem. We tackle these challenges by a novel approach that employs sparse 2D and 3D human pose detections from a convolutional neural network using a batch-based pose estimation strategy. Joint recovery of per-batch motion allows to resolve the ambiguities of the monocular reconstruction problem based on a low dimensional trajectory subspace. In addition, we propose refinement of the surface geometry based on fully automatically extracted silhouettes to enable medium-scale non-rigid alignment. We demonstrate state-of-the-art performance capture results that enable exciting applications such as video editing and free viewpoint video, previously infeasible from monocular video. Our qualitative and quantitative evaluation demonstrates that our approach significantly outperforms previous monocular methods in terms of accuracy, robustness and scene complexity that can be handled.
1 INTRODUCTION
MonoPerfCap addresses marker-less human performance capture from monocular RGB video in general scenes, where occlusion and missing depth make reconstruction ill-posed. It combines 2D/3D pose detections, batch-based motion optimization, and silhouette-based surface refinement.
- Research gap: Monocular capture avoids the expensive camera setups and controlled studios required by many multi-view methods, while also addressing the limitations of depth cameras.The target setting includes outdoor scenes and existing RGB video footage.
- Core approach: The method combines discriminative 2D and 3D joint detections with batch-based motion optimization to resolve monocular pose ambiguities.A low-dimensional trajectory subspace provides the motion constraint, while 2D and 3D detections complement each other.
- Core approach: Automatically extracted silhouettes support recovery of medium-scale non-rigid surface deformations beyond articulated skeleton motion.This enables alignment of loose clothing and other deforming surface regions.
2 RELATED WORK
Prior performance-capture methods provide accuracy under restricted sensing or studio conditions, but monocular reconstruction must handle missing depth, non-rigid clothing, and temporal ambiguity. MonoPerfCap combines pose estimation and generative reconstruction to target these constraints with a single camera.
- Multi-view performance capture: Multi-view methods reconstruct detailed and temporally consistent surfaces, but generally require dense cameras, stereo constraints, and controlled studio environments.These requirements limit their applicability to general outdoor scenes and single-camera capture.
- Depth-based performance capture: Depth-based methods reduce monocular ambiguity and can support real-time reconstruction, but active sensors face sunlight, energy, distribution, and mobility constraints.RGB cameras are more widely available and already integrated into smartphones.
- Sparse skeletal pose reconstruction: Earlier skeletal pose methods often require manual initialization or annotations, while direct single-image 3D regression can produce temporally incoherent reconstructions.The cited manual sports-video approach requires joint annotations every 30 frames.
- Sparse skeletal pose reconstruction: Hybrid 2D/3D pose methods improve skeletal estimation but may require multiple views, whereas MonoPerfCap combines complementary monocular pose models with trajectory constraints.The paper additionally recovers medium-scale non-rigid surface deformations that skeleton-only models cannot represent.
- Dense monocular shape reconstruction: Dense monocular shape methods use low-dimensional constraints or foreground segmentation, but prior approaches may require manual correction or capture only skin-tight clothing without surface detail.MonoPerfCap instead targets strongly deforming humans in general apparel.
3 METHOD OVERVIEW
MonoPerfCap represents a person with a pre-acquired actor-specific template and reconstructs articulated motion plus medium-scale surface deformation from monocular video. The template is obtained from handheld footage of the actor in a static pose.
- Problem setting: Monocular non-rigid reconstruction is ill-posed because the subject is only partially visible at each time and depth cues are implicit.Full-body capture adds articulated motion and apparel deformation to the reconstruction challenge.
- Representation: A two-level deformation hierarchy parameterizes the actor’s articulated motion and medium-scale surface changes during monocular capture.The template provides the subject-specific shape representation for tracking.
- Template acquisition: The method uses a person-specific template mesh acquired by recording a handheld camera orbit around the actor in a static T-pose.The reconstruction pipeline samples 60 images and automatically produces a textured triangulated surface.
4 BATCH-BASED 3D HUMAN POSE ESTIMATION
The method jointly estimates human skeleton motion over overlapping video batches, combining CNN-based 2D/3D joint alignment with anatomical constraints and low-dimensional temporal regularization.
- Motion representation: The skeleton uses 33 deformation parameters: root translation, root rotation, and 27 joint angles for 16 joints, with the actor mesh rigged by dual quaternion skinning.The motion representation is based on a low-dimensional skeleton subspace.
- Batch-based pose estimation: The approach jointly recovers skeleton motion over continuous temporal windows because single-frame monocular 3D pose estimation is highly underconstrained.Each batch contains a continuous sequence of frames rather than independently estimated poses.
- Batch processing: Videos are processed in overlapping batches of |B| = 50 frames, then neighboring reconstructions are combined through linear blending in their 10-frame overlap.Each batch is processed independently before the overlap-region fusion.
- Batch-based pose estimation: Each batch is formulated as a constrained optimization problem whose joint-angle parameters remain within anatomically plausible lower and upper bounds.The objective combines data-fitting and regularization terms for pose estimation.
- Discriminative joint alignment: The data terms align the reconstructed skeleton with CNN-regressed 2D image joints and 3D joint positions.The method uses discriminative detections for 16 joints, including added toe detections to better constrain foot rotations.
- Batch-based motion regularization: Temporal motion is regularized by constraining each skeleton-parameter trajectory to a low-dimensional DCT subspace with K = 8 dimensions.This batch-level coupling addresses inaccuracies caused by the underconstrained monocular reconstruction problem.
5 SILHOUETTE-BASED REFINEMENT
Silhouette-based refinement addresses pose–silhouette misalignment and medium-scale clothing and skin deformation after batch-based pose estimation. The method uses automatic segmentation, ICP-like pose refinement, and an embedded deformation graph for surface alignment.
- Motivation: Batch-based pose optimization does not capture apparel and skin deformation, causing boundary misalignments between the skeleton-deformed mesh and input images.
- Automatic Silhouette Extraction: GrabCut segmentation is initialized automatically from rasterized skeleton and template masks, then strengthened with temporal motion cues.The masks distinguish known foreground, known background, and uncertain regions.
- Silhouette-based Pose Refinement: ICP-like pose refinement aligns projected model boundaries to silhouette points while softly constraining the solution near the batch-based pose estimate.The refinement typically performs three iterations and balances silhouette alignment against pose stabilization.
- Silhouette-based Pose Refinement: A second silhouette extraction after pose refinement improves segmentation accuracy and supports more accurate non-rigid surface alignment.The refinement improves both pose estimates and silhouette segmentation.
- Silhouette-based Non-Rigid Surface Refinement: Medium-scale surface refinement uses an embedded deformation graph whose local rotations and translations are blended across the dense actor model.The graph contains approximately 1000 nodes, and the optimization combines silhouette alignment with as-rigid-as-possible regularization.
- Optional Fine-scale Refinement: The reconstruction can optionally receive shading-based refinement to add fine-scale surface detail, beyond the paper’s main focus.
6 RESULTS
The evaluation shows that MonoPerfCap remains accurate and temporally coherent across diverse monocular sequences, while each major optimization stage improves reconstruction quality. It compares favorably with multi-view and monocular baselines, supports long and handheld sequences, and has clear failure boundaries under severe occlusion or pose errors.
- Benchmark: The benchmark contains 13 sequences totaling around 40k frames, spanning indoor and outdoor settings, camera types, environments, subjects, and clothing.A second camera and manually labeled silhouettes support alternative-view evaluation and silhouette-overlap measurement.
- Qualitative Results: The approach accurately captures actors with temporally coherent results across challenging motions, including 360-degree rotation, sitting, and object interaction.It also continuously tracks three seven-minute sequences without restarting.
- Qualitative Results: The reconstructions recover deforming pose and shape despite small monocular depth offsets and provide textured models renderable from arbitrary viewpoints.The models support free-viewpoint video; shading-based refinement can recover fine-scale detail, although that is not the work’s focus.
- Algorithmic Components: The three main processing steps progressively reduce per-frame surface-to-surface mean error, with the complete approach achieving the lowest error over almost all frames.The evaluation aligns reconstructions to ground truth by translation to remove global depth offsets.
- Failure Cases: Silhouette-based refinement cannot correct artifacts when strong occlusion or complicated poses produce an incorrect pose estimate and wrong segmentation.The correspondence-pruning strategy limits modification where segmentation is unreliable, and tracking recovers when occluded parts reappear.
- Comparisons: >3cm improvement is reported over state-of-the-art approaches, while surface refinement improves silhouette overlap accuracy by 11% over skinning results.The pose estimation also consistently outperforms prior methods in silhouette overlap accuracy.
- Comparisons: The method achieves similar accuracy to stereo-based capture using only monocular input and consistently outperforms prior monocular approaches in tracking and surface reconstruction.On the Pablo sequence, it is close to an eight-camera multi-view method and outperforms a template-based monocular method.
7 LIMITATIONS
The method still depends on a person-specific actor rig created during preprocessing and can fail under strong occlusion combined with fast motion. Foot capture is also less robust.
- Scope and Assumptions: The approach requires a person-specific actor rig in preprocessing, although the template can be generated from a circular-path video in about half a minute.Automatically extracting such a rig from general-motion monocular video remains unsolved.
- Failure Cases: Strong occlusion combined with fast motion can still cause tracking failure in the monocular setting.The method can instantly recover when occluded parts become visible again because of discriminative joint detections.
- Failure Cases: Foot capture is less robust than the rest of the reconstruction results.
8 CONCLUSION
The paper presents marker-less, temporally coherent human performance capture from monocular video, combining batch-based motion recovery with silhouette-driven surface refinement. It reports compelling reconstruction results, including comparable quality to binocular stereo, while noting failures under strong occlusion and fast motion.
- The approach jointly recovers motion from sparse 2D and 3D detections with a low-dimensional motion prior over frame batches.This design addresses ambiguities in the underconstrained monocular reconstruction problem.
- The monocular approach obtains comparable quality to binocular stereo without requiring explicit depth cues.Per-vertex differences are color coded in the comparison, and the method does not use full BRDF-based shading refinement.
- Strong occlusion and fast motion can cause tracking failure, although the approach instantly recovers when occluded parts become visible again.
- Automatically extracted silhouettes refine tracked surface geometry for medium-scale non-rigid alignment.
- The results support applications such as video editing and free-viewpoint video from single RGB video.