Source-linked AI summary
Video Based Reconstruction of 3D People Models
Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, Gerard Pons-Moll
TL;DR
Existing approaches struggle to reconstruct personalized, textured, and rigged human avatars from consumer-accessible monocular video. This paper fits a parametric body model and fuses motion-compensated silhouette information to reconstruct detailed, animatable models from a moving person.
Problem
Personalized 3D human models need person-specific geometry, hair, clothing, texture, and animation structure, but existing scanners and multi-view or RGB-D methods are costly, time-consuming, limited in availability, or lack a rigged skeleton.
Method
The method extends a canonical SMPL body model with surface offsets, estimates poses across video frames, and morphs and fuses dynamic silhouette cones into a common reference frame.
Results
The approach reconstructs personalized geometry of hair, body, and clothing, surface texture, and an underlying pose- and shape-adjustable model that is automatically rigged with a kinematic skeleton.
Takeaways & Limitations
Because the method uses monocular RGB video and consumer cameras, it supports realistic avatar creation for VR, entertainment, biometrics, virtual try-on, and image editing.
Takeaways & Limitations
Long open hair and skirts cannot be modeled as offsets from the body, while unseen concave regions and strong fabric motion reduce reconstruction detail.
Abstract
from arXiv · showhide
This paper describes how to obtain accurate 3D body models and texture of arbitrary people from a single, monocular video in which a person is moving. Based on a parametric body model, we present a robust processing pipeline achieving 3D model fits with 5mm accuracy also for clothed people. Our main contribution is a method to nonrigidly deform the silhouette cones corresponding to the dynamic human silhouettes, resulting in a visual hull in a common reference frame that enables surface reconstruction. This enables efficient estimation of a consensus 3D shape, texture and implanted animation skeleton based on a large number of frames. We present evaluation results for a number of test subjects and analyze overall performance. Requiring only a smartphone or webcam, our method enables everyone to create their own fully animatable digital double, e.g., for social VR applications or virtual try-on for online fashion shopping.
1. Introduction
The paper targets personalized, animatable 3D human models from monocular RGB video, avoiding expensive or specialized capture setups. Its key approach unposes motion-dependent silhouette rays into a canonical frame, enabling shape reconstruction from moving people.
- Personalized avatars need person-specific geometry, clothing and hair, coherent texture, and an underlying animation structure.
- Existing scanners are expensive, multi-view capture is time-consuming and error-prone, and RGB-D sensors are less available than video.
- The overview proceeds from segmented image sequences and SMPL pose estimation to ray unposing, canonical shape optimization, texture calculation, and personalized blend-shape generation.
- Fitting a body model to F images requires optimizing F poses simultaneously and storing F models in memory, making the process computationally expensive and impractical.
- The method fits SMPL poses frame by frame, associates silhouette points with 3D model points, and transforms projection rays using inverse deformation.
- Unposing rays across frames produces a visual hull constraining the subject’s shape in a canonical T-pose.
2. Related Work
Prior work spans free-form, model-based, clothing-aware, and learning-based reconstruction. The paper positions monocular reconstruction as a setting where parametric body priors are especially useful, while existing approaches often require multi-view data, pre-scanned templates, or restrict recovered geometry.
- Free-form methods: Free-form methods reconstruct dynamic geometry flexibly but typically require high-quality multi-view input, limiting their practicality.
- Model-based methods: Model-based methods use parametric body models to constrain ambiguous monocular reconstruction, while temporal methods optimize shared shape with multiple poses.
- Hybrid approaches: Some hybrid approaches pre-scan a template or use a skeleton to regularize dynamic fusion, whereas this work aligns visual hulls over time.
- Shape under clothing: Clothing-aware methods impose constraints such as placing the body inside clothing silhouettes, but results are usually restricted to naked model space.
- Learning-based methods: Learning-based methods are uncommon because annotated shape, pose, and clothing geometry are scarce; existing predictions can be over-smooth or dataset-limited.
3. Method
The method reconstructs a personalized, animatable human model by combining pose estimation, consensus shape optimization in a canonical frame, frame refinement, and texture generation. It extends SMPL with vertex offsets to represent clothing and personal surface detail while efficiently aggregating silhouette constraints across video frames.
- The pipeline comprises pose reconstruction, consensus shape estimation, and frame refinement with texture-map generation.Pose estimation produces frame-wise poses; consensus optimization aggregates silhouette information; refinement captures time-varying details before texture computation.
- 3.4. Frame Refinement and Texture Generation: The refined canonical model is warped back to each frame, image colors are back-projected to visible vertices, and aggregated texels form the texture map.The supplied method description also states that frame-wise deviations are optimized in a sliding-window approach before texture computation.
- 3.1. SMPL Body Model with Offsets: SMPL provides the parametric body representation, while auxiliary vertex offsets D deform it to explain clothing and personal surface details.The offsets are optimized during consensus shape estimation, extending the naked-human model to clothed subjects.
- 3.2. Pose Reconstruction: Pose initialization jointly optimizes one shape and five poses, incorporates silhouette alignment, and then estimates temporally coherent poses for all F frames.Known height can constrain scale; without it, reconstructions remain accurate up to a scale factor, with height estimation roughly off by 2–5 cm.
- 3.3. Consensus Shape: Silhouette rays from each moving frame are inverse-transformed into a canonical T-pose, allowing one shape model to explain constraints from F = 120 frames.Unposing removes per-frame blend-shape calculations and reduces memory use during optimization.
- 3.3. Consensus Shape: Consensus shape optimization minimizes point-to-ray distances while regularizing smoothness, proximity to the body model, and approximate bilateral symmetry.The objective combines a data term with Laplacian, body-model, and symmetry regularizers; symmetry remains soft to permit clothing wrinkles and body asymmetries.
4. Experiments
The experiments evaluate monocular reconstruction on rendered scan sequences, real RGB sequences, and a depth-based comparison. Results show accurate recovery of personalized geometry and robustness to pose-estimation errors, while capturing more detail than pure model-based methods.
- Datasets and protocol: The evaluation uses BUFF and DynamicFaust rendered scan sequences with ground-truth shapes, plus People-Snapshot real RGB sequences and KinectCap data.The datasets vary clothing, motion, subjects, lighting, and backgrounds; People-Snapshot contains 24 sequences of 11 subjects.
- Qualitative comparisons: Compared with pure model-based methods, the consensus shape captures medium-level geometry details from a single RGB camera.The comparison extends the monocular model-based baseline to multiple views for fairness.
- Quantitative evaluation: The method achieves mean average surface errors of 5.37mm on BUFF, 4.44mm on D-FAUST, and 3.97mm on KinectCap.Errors are computed as bidirectional vertex-to-surface distances after pose and scale alignment to ground-truth scans.
- Quantitative evaluation: 3.40mm on BUFF and 2.86mm on D-FAUST with ground-truth poses remain close to full-pipeline errors, demonstrating robustness to noisy 3D pose estimates.The comparison indicates that pose-estimation inaccuracies do not substantially degrade reconstruction accuracy on these datasets.
5. Discussion and Conclusions
The paper reconstructs personalized, textured, and animatable human models from a single moving-person video, including hair, body, and clothing. It reports millimeter-level accuracy and useful pose editing, while identifying limits for non-body-topological appearances and unseen or rapidly deforming surfaces.
- Contributions: The method reconstructs personalized geometry, texture, and an underlying kinematic skeleton from a single video of a moving person.It combines a parametric body model with surface displacements and fused, motion-normalized silhouette cones.
- Results: 4.5mm accuracy is reported for human body shape reconstruction, with robustness to noisy 3D pose estimates shown by ablation analysis.The reconstructed model includes hair, body, clothing, surface texture, and approximate pose-dependent deformation.
- Limitations: Long open hair and skirts cannot be modeled as offsets from the body because they do not share the body’s topology.Only surface details visible on the outline of at least one view can be captured; concave regions and strong fabric motion remain difficult.
- Animation: Because reconstructed templates share SMPL topology, SMPL can change their pose and shape, although clothing deformations are not modeled explicitly.The resulting deformed templates are described as plausible for several applications.
- Surface detail: The method captures body shape and medium-level geometry, while shape-from-shading can add finer details such as folds and wrinkles.The shading refinement depends on accurate model-to-image alignment.
- Applications: Monocular RGB capture could support realistic avatars for VR, entertainment, biometrics, virtual try-on, and image editing.The paper emphasizes that cameras are ubiquitous and low cost.