Source-linked AI summary
DeepCap: Monocular Human Performance Capture Using Weak Supervision
Marc Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt
TL;DR
DeepCap addresses the lack of dense, temporally coherent monocular human capture that avoids expensive multi-view studios and 3D ground-truth training data. It uses weak multi-view supervision with separate pose and deformation prediction, and reports higher-quality, more accurate, and more robust reconstructions than previous work.
Problem
Dense monocular performance capture must recover articulated pose and non-rigid surface deformation with space-time coherence, while prior methods require controlled multi-view setups or miss correspondences and deformation.
Method
DeepCap jointly predicts skeletal pose and dense non-rigid surface deformation with a differentiable mesh architecture trained by analysis-by-synthesis from multi-view video without explicit 3D ground truth.
Results
The method produces higher-accuracy and more 3D-stable reconstructions than related work, particularly in depth and under difficult poses.
Takeaways & Limitations
The mesh-based approach provides temporal surface correspondences, preserves skeletal structure and limbs, and handles loose clothing in monocular performance capture.
Abstract
from arXiv · showhide
Human performance capture is a highly important computer vision problem with many applications in movie production and virtual/augmented reality. Many previous performance capture approaches either required expensive multi-view setups or did not recover dense space-time coherent geometry with frame-to-frame correspondences. We propose a novel deep learning approach for monocular dense human performance capture. Our method is trained in a weakly supervised manner based on multi-view supervision completely removing the need for training data with 3D ground truth annotations. The network architecture is based on two separate networks that disentangle the task into a pose estimation and a non-rigid surface deformation step. Extensive qualitative and quantitative evaluations show that our approach outperforms the state of the art in terms of quality and robustness.
1. Introduction
DeepCap targets dense monocular human performance capture by jointly recovering articulated pose and space-time coherent non-rigid surface deformation without 3D ground-truth training annotations. Its weak multi-view, mesh-based approach improves reconstruction accuracy, 3D stability, and robustness, including difficult poses.
- Dense monocular capture must recover full-body pose alongside coherent skin and clothing deformation, beyond the sparse or articulated motion handled by most existing monocular methods.
- Existing multi-view methods depend on controlled multi-camera studios, limiting use in location shootings and living spaces.
- DeepCap jointly infers skeletal pose and non-rigid 3D deformation parameters in one feed-forward pass using a differentiable mesh template and embedded deformation graph.
- The method tracks surface vertices over time, preserves human limbs during occlusions and out-of-plane motion, and supports graphics operations such as texturing and rendering.
- Training uses weak multi-view supervision and analysis-by-synthesis, requiring only a personalized actor template and multi-view video rather than explicit 3D annotations.
- The model produces higher-accuracy and more 3D-stable reconstructions than related work, particularly in depth and under difficult poses.
2. Related Work
Related work spans parametric, depth-based, and monocular template-free approaches, each trading generality, clothing coverage, robustness, or temporal correspondence. DeepCap is positioned as a mesh-based alternative that explicitly preserves per-vertex correspondences over time.
- Capture using Parametric Models: Parametric models provide shape and deformation priors, but clothing variation is often omitted; non-rigid extensions handle tight garments better than loose apparel.
- Depth-based Template-free Capture: Depth-based template-free methods can capture non-rigid scenes, but reliable operation is generally limited to small, controlled, slow motions or enhanced sensing setups.
- Monocular Template-free Capture: Monocular template-free methods reconstruct clothed humans from images, including volumetric and textured representations, but their generalization to strong articulation is limited.
- Monocular Template-free Capture: Implicit representations do not recover frame-to-frame correspondences, whereas DeepCap’s mesh representation explicitly obtains per-vertex correspondences over time, with slightly less generality.
3. Method
The method captures dense deforming human surfaces from monocular images using separate pose and deformation networks trained through differentiable multi-view objectives. It adapts the networks to in-the-wild images with single-view finetuning.
- In-the-wild Domain Adaptation: Training uses a personalized template and multi-view studio video, while test-time adaptation finetunes the networks on monocular images to address the studio-to-wild domain gap.The adaptation uses the same 2D keypoint and silhouette losses for a small number of single-view iterations.
- Pose Network: PoseNet estimates skeletal joint angles and camera-relative rotation from a monocular image, using differentiable kinematics and multi-view 2D keypoint supervision.A pose prior constrains joint angles to anatomically valid ranges, while global alignment estimates the skeleton's world-space translation.
- Deformation Network: DefNet regresses embedded deformation-graph rotations and translations to model non-rigid surface changes after skeletal pose estimation.PoseNet weights remain fixed while DefNet is trained without direct supervision on deformation parameters.
- Deformation Network: The differentiable deformation layer applies graph-based non-rigid deformation, skeletal posing, inverse camera rotation, and global translation to template vertices.Vertex-to-node weights are based on geodesic distances, and dual quaternion skinning derives posed node transformations.
- Deformation Network: Multi-view silhouette, sparse keypoint, and as-rigid-as-possible losses supervise deformed geometry, constrain mesh-to-image assignments, and encourage local smoothness.Silhouette alignment uses distance transforms; sparse keypoints address wrong assignments during highly articulated motions, while rigidity weights vary by surface material.
4. Results
The method is evaluated on in-the-wild and calibrated multi-camera sequences, showing accurate overlays, plausible 3D reconstructions, and strong pose and surface deformation performance. Ablations show that additional training views and frames improve reconstruction, while DefNet improves deformation quality over pose-only estimation.
- Dataset: The evaluation covers four subjects with varied apparel, including 13 in-the-wild sequences and four calibrated multi-camera sequences with ground-truth annotations.The sequences span indoor and outdoor environments and diverse motions.
- Qualitative Comparisons: Reconstructions precisely overlay input images and remain plausible from arbitrary 3D viewpoints across varied clothing, poses, and environments.These qualitative results are shown on in-the-wild test sequences.
- Skeletal Pose Accuracy: The method outperforms related monocular approaches in pose accuracy and remains close to the multi-view baseline.Table 1 reports skeletal pose accuracy using metrics evaluated on commonly used joints.
- Surface Reconstruction Accuracy: The method consistently outperforms other approaches in AMVIoU and RVIoU, demonstrating accurate recovery of 3D surface geometry.The method is again close to the multi-view baseline.
- Ablation Study: Adding cameras and training frames consistently improves reconstruction quality, with the largest camera benefit occurring when increasing from one to two views.More diverse training poses and deformations improve reconstruction quality.
- Ablation Study: DefNet improves AMVIoU by around 4% over PoseNet-only estimation and especially improves fitting for loose clothing such as skirts.It also corrects slight pose errors and typical skinning artifacts.
5. Conclusion
The paper presents a weakly supervised monocular approach that directly regresses pose and surface deformation, tracks temporal correspondences, preserves body structure, and handles loose clothing. Across different scenarios, it reports more accurate 3D reconstruction than existing methods.
- The approach performs monocular dense human performance capture using only weak multi-view supervision.
- Neural networks directly regress poses and surface deformations while producing temporal surface correspondences.
- The method preserves the human skeletal structure and handles loose clothes.
- Qualitative and quantitative evaluations across different scenarios show more accurate 3D reconstruction of pose and non-rigid deformation than existing methods.
- Future work targets joint tracking of the body, facial expressions, and hand gestures, plus more physically correct multi-layered garment representations.