Source-linked AI summary
Detailed Human Avatars from Monocular Video
Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, Gerard Pons-Moll
TL;DR
Detailed personal avatars require recovering identity-bearing features such as facial details, hair, clothing, wrinkles, and texture from accessible monocular video. The paper refines a parameterized body model using facial landmarks and shape-from-shading, then stitches textures with semantic constraints. In user studies, the resulting avatars were preferred over state-of-the-art methods 89.64% of the times and judged more detailed 95.72% of the times.
Problem
Existing monocular RGB methods are typically limited to statistical body-model parameters, while prior free-form reconstructions are overly smooth, lack facial details, and have blurry textures.
Method
The method optimizes a high-resolution shape in canonical T-pose space using SMPL regularization, facial landmarks, shape-from-shading, and semantic graph-cut texture stitching.
Results
Users preferred the avatars over state-of-the-art methods 89.64% of the times and judged them more detailed 95.72% of the times.
Takeaways & Limitations
The method produces detailed personalized avatars with improved face reconstruction, identity preservation, coherent textures, and reduced color spilling between semantic parts.
Takeaways & Limitations
Full texturing requires subjects to be seen from all sides, and the method does not yet cover clothing with topology different from the body, such as skirts and coats.
Abstract
from arXiv · showhide
We present a novel method for high detail-preserving human avatar creation from monocular video. A parameterized body model is refined and optimized to maximally resemble subjects from a video showing them from all sides. Our avatars feature a natural face, hairstyle, clothes with garment wrinkles, and high-resolution texture. Our paper contributes facial landmark and shading-based human body shape refinement, a semantic texture prior, and a novel texture stitching strategy, resulting in the most sophisticated-looking human avatars obtained from a single video to date. Numerous results show the robustness and versatility of our method. A user study illustrates its superiority over the state-of-the-art in terms of identity preservation, level of detail, realism, and overall user preference.
1. Introduction
The paper targets detailed, identity-preserving human avatars from monocular video, addressing limitations of existing methods in facial detail, shape detail, and texture quality. It combines canonical-space refinement cues with efficient, semantically constrained texture stitching, and reports strong user preference for the resulting reconstructions.
- Motivation and gap: Monocular RGB methods generally predict statistical body-model parameters, while prior free-form reconstruction produces overly smooth shapes, lacks facial details, and has blurry textures.These limitations prevent avatars from fully retaining subjects’ identities.
- Shape and appearance refinement: Facial landmark detections are transformed into a joint T-pose space before temporal fusion, improving robustness to landmark misdetections.The landmarks are incorporated directly into the 3D reconstruction objective.
- Shape and appearance refinement: Shape-from-shading is applied at every frame to obtain partial 3D normal fields, which are fused in T-pose space to recover fine details such as wrinkles.Frame-wise cues cannot be directly fused while the person moves, so pose tracking and inverse pose transformation are used.
- Texture stitching: A graph-cut texture update energy efficiently stitches partial textures from incoming views without the intractable label growth of large multi-view assignments.The method addresses the computational cost of assigning each texel among many view labels.
- Texture stitching: The semantic appearance term penalizes texel colors unlikely for their semantic body parts, significantly reducing clothing-to-skin texture spilling.It also implicitly connects texels belonging to the same part.
- Evaluation: Users preferred the avatars over state-of-the-art methods 89.64% of the times and judged them more detailed 95.72% of the times.The study was used because metric-based scan-to-mesh distances do not capture perceptual quality.
2. Related work
Prior human reconstruction methods rely on expensive multi-camera or depth-based capture, static poses, or temporally limited representations. Related shape-from-shading and texture-generation work motivates the paper’s use of shading cues and semantic constraints for monocular nonrigid reconstruction.
- Multi-view capture: Multi-camera systems can reconstruct clothed bodies and related attributes, but they are expensive and require complicated calibration.The broader setup may also capture joint shape, clothing, pose, or facial expressions.
- Depth and monocular reconstruction: Monocular reconstruction is appealing but often requires many depth viewpoints around a static object, whereas humans cannot hold a static pose for long.Depth-based nonrigid fusion can additionally suffer from phantom surfaces during fast motion.
- Performance capture: Dynamic performance-capture methods reconstruct individual frames or frame windows, but these representations are not temporally coherent.Some approaches register a common model, use volumetric tracking, or assume a pre-built static template.
- Shape-from-shading: Shape-from-shading captures geometric details such as folds that are difficult to recover from silhouettes or photometric information alone.The paper applies this cue to human-body modeling.
- Texture generation: Texture-generation methods combine fragments from different views using blending, mosaicing, or graph cuts, but registration errors can cause blurring or ghosting.The paper uses semantic information to constrain texture stitching in the more difficult monocular nonrigid setting.
3. Method
The method reconstructs a detailed textured avatar from monocular video through coarse-to-fine shape estimation, fine-detail recovery, and texture stitching. It combines SMPL-based modeling with facial alignment, shape-from-shading, semantic priors, and graph-cut optimization.
- Overview: The pipeline first estimates medium-level clothing and skin geometry from segmentations, then adds fine details and computes texture.The three stages proceed from rough body shape to shape-from-shading details and semantic-prior-based texture computation.
- Subdivided SMPL body model: The SMPL body model is extended with template offsets and edge subdivision to represent clothing, wrinkles, and fine facial features.The finer model uses normal displacements and contains 110210 vertices and 220416 faces.
- Medium-level shape: Medium-level shape optimization combines silhouette, facial-landmark alignment, and regularization terms.Facial landmarks are detected for each key frame and mapped to mesh points through barycentric interpolation.
- Fine-level surface details: Fine surface details are recovered by estimating frame-wise shading normals, transforming them into canonical T-pose space, and optimizing a surface over merged evidence.The method uses key frames individually because global shape-from-shading can smooth away details under fabric movement and misalignment.
- Fine-level surface details: Regularization preserves local smoothness, mesh structure, and consensus shape during fine-detail optimization.The method uses anisotropic Laplacian smoothness, edge-length variation pruning, and pruning of large deviations from the consensus shape.
- Texture generation: The texture stage assigns colors per texel using graph-cut optimization and connects texels across texture seams.A semantic prior and part-specific HSV color models reduce spilling and artifacts caused by mesh-view misalignment.
4. Experiments
Experiments compare the method with monocular and RGB-D reconstruction approaches and evaluate identity, detail, texture quality, and user preference. The results show improved facial and geometric detail, reduced texture artifacts, comparable quality to RGB-D results, and strong user preference.
- Qualitative comparisons: The reconstructed avatars capture input-image detail, including improved faces, clothing wrinkles, and T-shirt stamps compared with [3].
- Qualitative comparisons: Our RGB-only results are comparable in quality to the RGB-D method despite using a more difficult input setting.
- Face similarity: FaceNet reconstruction-to-photo distance is 0.99 ± 0.11 for our method versus 1.09 ± 0.15 for [3].
- Ablation analysis: Shape-from-shading adds fine-grained geometric details such as wrinkles and buttons beyond the low-frequency mesh shape.
- Ablation analysis: The semantic texture prior removes color spills and reduces stitching artifacts along semantic boundaries.
- User study: Users preferred our avatars over [3] 89.64% of the time and judged them more detailed 95.72% of the time.The study compared randomized renderings across four aspects using 24 reconstructed subjects.
5. Discussion and Conclusion
The method combines high-detail reconstruction and semantic texture stitching to produce personalized avatars that users rate as more realistic, detailed, identity-preserving, and preferable than prior work. Its current scope is limited for clothing with topology different from the body and requires views from all sides for full texturing.
- 83.12% of the times, users judged the method better at preserving identity, while 95.72% judged it to capture more details.
- The binary texture-stitching optimization efficiently merges multiple frames into one coherent texture while reducing color spilling between semantic parts.
- Compared with the method of [3], the results look more natural and contain finer details.
- 92.7% of the times, users considered the results more realistic than state of the art, and 89.64% preferred the avatars.
- Future work should address clothing with topology different from the body, such as skirts and coats.