Source-linked AI summary
LiveCap: Real-time Human Performance Capture from Monocular Video
Marc Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt
TL;DR
Single-camera human performance capture lacked a real-time approach for dense, coherent geometry of people in everyday clothing. LiveCap fits a textured template in two stages and tracks non-rigid skin and apparel deformations with material-aware optimization and parallel solvers. The system runs above 25 Hz and reports accuracy close to offline capture methods while outperforming related monocular online methods.
Problem
Real-time dense performance capture from a single RGB camera remained unavailable, although such capture would simplify applications requiring live human models.
Method
LiveCap preprocesses a textured actor template, then jointly estimates skeletal motion and dense non-rigid skin and clothing deformation using monocular alignment terms and real-time optimization.
Results
The system captures dense, space-time coherent clothed human geometry in real time above 25 Hz, with reconstruction density and accuracy close to offline approaches and better than related monocular online methods.
Takeaways & Limitations
LiveCap provides a real-time monocular route to capturing entire humans and loose clothing for live 3D video and virtual try-on applications.
Takeaways & Limitations
Tracking depends on accurate template acquisition, cannot recover topological changes far from the template, and only partially recovers deformations along the viewing axis.
Abstract
from arXiv · showhide
We present the first real-time human performance capture approach that reconstructs dense, space-time coherent deforming geometry of entire humans in general everyday clothing from just a single RGB video. We propose a novel two-stage analysis-by-synthesis optimization whose formulation and implementation are designed for high performance. In the first stage, a skinned template model is jointly fitted to background subtracted input video, 2D and 3D skeleton joint positions found using a deep neural network, and a set of sparse facial landmark detections. In the second stage, dense non-rigid 3D deformations of skin and even loose apparel are captured based on a novel real-time capable algorithm for non-rigid tracking using dense photometric and silhouette constraints. Our novel energy formulation leverages automatically identified material regions on the template to model the differing non-rigid deformation behavior of skin and apparel. The two resulting non-linear optimization problems per-frame are solved with specially-tailored data-parallel Gauss-Newton solvers. In order to achieve real-time performance of over 25Hz, we design a pipelined parallel architecture using the CPU and two commodity GPUs. Our method is the first real-time monocular approach for full-body performance capture. Our method yields comparable accuracy with off-line performance capture techniques, while being orders of magnitude faster.
1 INTRODUCTION
LiveCap targets the lack of real-time dense human performance capture from a single RGB camera, reconstructing coherent geometry for people in everyday clothing. Its two-stage system combines template fitting, non-rigid tracking, and parallel optimization to achieve real-time capture.
- Single-RGB performance capture previously lacked a real-time method, despite applications in interactive VR, AR, gaming, virtual try-on, and telepresence.
- The method reconstructs dense, space-time coherent deforming geometry of humans wearing loose everyday clothing from one video camera.
- A preprocessing step builds a rigged surface and appearance template, automatically identifying skin and apparel regions through multi-view segmentation with deep learning.
- The first stage fits a skinned template using CNN-based 2D and 3D joints, sparse facial landmarks, and the foreground silhouette.
- The second stage captures dense non-rigid deformations using silhouette and photometric alignment, material-aware regularization, GPU-accelerated Gauss-Newton solvers, and a pipelined CPU–two-GPU architecture.The architecture is designed to exceed 25 Hz and supports high-quality reconstruction from a single monocular camera.
2 RELATED WORK
Earlier human performance capture relied mainly on multi-view or depth-based sensing, while monocular approaches generally produced skeletal or coarse body estimates. LiveCap addresses this gap by combining monocular pose cues with dense deformable surface reconstruction.
- Multi-view: Multi-view methods can capture detailed or arbitrary deforming shapes, but require multiple cameras, controlled setups, or expensive active stereo systems.
- Depth-based: Depth-based methods enabled dense reconstruction from affordable sensors, but RGB-D cameras have limited range, sunlight robustness, and resolution.
- Monocular 3D Pose and Shape Estimation: Monocular pose and shape methods commonly estimate 2D or 3D joints, fit constrained body models, or regress coarse volumes rather than dense deforming clothing.
3 METHOD
LiveCap captures articulated motion and dense non-rigid deformation from a single color video using a coarse-to-fine, two-stage template-fitting pipeline. It combines learned joint and facial detections, silhouette alignment, photometric tracking, and material-dependent deformation regularization.
- 3 METHOD: The method takes a single color video and a textured actor model, then estimates articulated motion and non-rigid skin and clothing deformation automatically.The actor model is acquired during preprocessing from a monocular video sequence.
- 3.3 Skeletal Pose Estimation: The first stage fits skeletal pose using 2D and 3D joint detections, facial landmarks, foreground silhouettes, temporal regularization, and anatomical constraints.The pose energy combines E2D, E3D, silhouette, temporal, and anatomical terms.
- 3.1 Actor Model Acquisition: The template is built from approximately 70 images, downsampled to about 5,000 vertices, rigged to a skeleton, and assigned material-dependent non-rigidity classes.Seven classes use different per-edge smoothness weights to model deformation differences across materials.
- 3.3 Skeletal Pose Estimation: The silhouette constraint selects the distance-field direction or its opposite according to the projected boundary normal, avoiding attraction to an incorrect contour.When n^T z < 0, the method follows p = −z; otherwise it follows z.
- 3.4 Non-rigid Surface Registration: The second stage tracks dense surface deformation using silhouette and photometric alignment with smoothness and material-dependent regularization.Photometric residuals use robust color-based correspondence pruning and a three-level image pyramid.
4 DATA PARALLEL GPU OPTIMIZATION
The paper formulates pose estimation and non-rigid registration as nonlinear least-squares problems and solves them with data-parallel Gauss–Newton methods tailored to each system's structure. A three-thread CPU–GPU pipeline supports real-time processing.
- Both pose estimation and non-rigid registration minimize a sum-of-squares objective over model parameters.The unknowns are kinematic-model parameters for pose estimation and vertex positions for non-rigid deformation.
- Gauss–Newton linearizes the residual vector and updates the parameters using local steps from sequential linear subproblems.The residual vector stacks all residual terms, and the Jacobian determines each linearized system.
- Gauss–Newton avoids recomputing residuals during iterations, enabling faster runtimes and more iterations within the real-time constraint.This runtime benefit follows from using Gauss–Newton rather than Levenberg–Marquardt.
- Pose optimization uses a small dense 36 × 36 system transferred to the CPU and solved by QR decomposition.The 36 variables comprise rotation, translation, joint angles, and additional translation parameters.
- Non-rigid registration produces a large sparse normal-equation system solved with GPU-based data-parallel Preconditioned Conjugate Gradient.Parallelization operates over the rows, or unknowns, of the sparse system matrix.
- A three-thread pipeline assigns preprocessing to the CPU, pose detection to one GPU, and optimization to another processing stage.The architecture combines data-parallel computation with two GPUs to meet real-time constraints.
5 RESULTS
The method achieves real-time monocular capture of articulated motion and dense clothing deformation, with reconstructions that remain accurate during challenging motions and competitive with offline approaches. Ablations show consistent gains from the pipeline components, non-rigid stage, and material-adaptive regularization.
- Around 25 FPS enables real-time human performance capture using one Intel Core i7 and two Geforce GTX 1080Ti GPUs.The implementation uses the same empirically determined parameters across experiments.
- Over 20 minutes across 11 sequences and seven subjects evaluate loose clothing, fast motions, and varied apparel.The dataset includes skirts, hooded sweaters, dresses, and baggy pants, alongside live webcam tests and existing sequences.
- The reconstructions precisely overlay input images and capture body pose, head orientation, loose-clothing deformation, and laterally displaced views.Fast 360° rotations and jumping motions are also successfully tracked, supporting robustness under challenging motion.
- 86.86% versus 86.89% IoU matches Xu et al. [2018] on a challenging sequence while remaining orders of magnitude faster.On another sequence, the method is only 4.3% worse in IoU than Xu et al. [2018], while outperforming Zhou et al. [2016] and Mehta et al. [2017] by 8.5% and 9.4%.
- The method comes very close to offline multi-view and monocular surface-reconstruction quality, while slightly lower accuracy is attributed to less accurate foreground estimates.The Pablo comparison reports close quality to Robertini et al. [2016] and Xu et al. [2018].
- A 5.8% total improvement in average 3D vertex error results from displacement warping, body-part masks, and vertex snapping.The non-rigid stage especially improves skirt and arm reconstruction, while material-adaptive weights preserve both head shape and skirt motion.
6 DISCUSSION AND LIMITATIONS
The method combines real-time pose and non-rigid tracking refinements to achieve reconstruction quality comparable to an offline approach, while remaining orders of magnitude faster. Its monocular setting imposes limitations involving template accuracy, unseen poses, occlusion, topology, and segmentation.
- Discussion: Real-time pose tracking improves through per-frame estimation, displacement warping, and a GPU-based solver.These changes improve pose accuracy while reducing computation time and supporting faster motions.
- Discussion: Dense photometric constraints for non-boundary vertices and adaptive material-based regularization improve non-rigid reconstruction.
- Limitations: Accurate template acquisition is required because template reconstruction errors and mislabeled material segments cannot be recovered during tracking.
- Limitations: Extreme unseen poses can cause joint-regression failures and tracking glitches, while monocular occlusions reduce accuracy for hidden parts and may mispose fully occluded limbs.Model fitting can often, but not always, correct wrong joint estimates; the approach recovers when difficult occlusions end.
- Limitations: The method cannot handle topological changes far from the template, and viewing-axis deformations are only partially recovered by the photometric term.Changing illumination, strong shadows, shading, or similarly colored objects can also cause suboptimal foreground/background segmentation.
7 CONCLUSION
The paper presents a monocular real-time approach for dense, space-time coherent capture of clothed humans. It combines material-aware deformation modeling with pipelined parallel optimization to support real-time operation and applications including immersive virtual and augmented reality and telepresence.
- 7 CONCLUSION: The approach reconstructs dense, space-time coherent deforming geometry of entire humans in loose everyday clothing from a single view.
- 7 CONCLUSION: Automatically identified template material regions distinguish the non-rigid deformation behavior of skin and apparel.
- 7 CONCLUSION: Two data-parallel Gauss-Newton solvers run in a pipelined implementation for pose estimation and non-rigid tracking at real-time speed.
- 7 CONCLUSION: The method is positioned as a first step toward general real-time single-view human capture for virtual and augmented reality, telepresence, and virtual try-on.