Source-linked AI summary
Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition
Chen Guo, Tianjian Jiang, Xu Chen, Jie Song, Otmar Hilliges
TL;DR
Reconstructing detailed, naturally moving humans from monocular in-the-wild videos requires separating people from arbitrary backgrounds while recovering 3D surfaces from short sequences. Vid2Avatar jointly models the human and background in 3D with self-supervised neural fields and optimized canonical shape, appearance, background, and pose; it achieves clean decomposition and high-quality reconstructions, performing best across several datasets and settings. The method is demonstrated on online and hand-held mobile-phone videos, though it relies on reasonable pose estimates and struggles with loose clothing.
Problem
Monocular in-the-wild videos make it difficult to separate dynamic humans from arbitrary backgrounds and reconstruct detailed 3D surfaces from short sequences.
Method
Vid2Avatar jointly models the human and background with separate neural fields, using a canonical human representation and global optimization over shape, appearance, background, and per-frame pose.
Results
Vid2Avatar produces clean scene decomposition and high-quality human reconstructions, performs best across several datasets and settings, and reconstructs people from online and hand-held mobile-phone videos.
Takeaways & Limitations
Self-supervised 3D scene decomposition supports robust and detailed human reconstruction from monocular in-the-wild videos without external segmentation methods.
Takeaways & Limitations
Vid2Avatar relies on reasonable pose estimates and faces significant challenges with loose clothing such as skirts or free-flowing garments.
Abstract
from arXiv · showhide
We present Vid2Avatar, a method to learn human avatars from monocular in-the-wild videos. Reconstructing humans that move naturally from monocular in-the-wild videos is difficult. Solving it requires accurately separating humans from arbitrary backgrounds. Moreover, it requires reconstructing detailed 3D surface from short video sequences, making it even more challenging. Despite these challenges, our method does not require any groundtruth supervision or priors extracted from large datasets of clothed human scans, nor do we rely on any external segmentation modules. Instead, it solves the tasks of scene decomposition and surface reconstruction directly in 3D by modeling both the human and the background in the scene jointly, parameterized via two separate neural fields. Specifically, we define a temporally consistent human representation in canonical space and formulate a global optimization over the background model, the canonical human shape and texture, and per-frame human pose parameters. A coarse-to-fine sampling strategy for volume rendering and novel objectives are introduced for a clean separation of dynamic human and static background, yielding detailed and robust 3D human geometry reconstructions. We evaluate our methods on publicly available datasets and show improvements over prior art.
1. Introduction
Vid2Avatar reconstructs detailed human avatars from monocular in-the-wild videos by jointly decomposing foreground and background in 3D without external segmentation or groundtruth supervision. Its self-supervised formulation combines temporally consistent canonical human modeling, global optimization, and scene-decomposition objectives, with evaluations showing strong performance across datasets and settings.
- Existing template-based, explicit-mesh, supervised, and pre-segmented neural methods face limitations involving templates, topology, out-of-distribution poses, temporal consistency, or segmentation quality.
- Vid2Avatar learns detailed 3D avatars from monocular in-the-wild videos without groundtruth supervision, clothed-human scan priors, or external segmentation modules.
- The method models the human foreground and static background with two separate neural fields to solve scene separation and surface reconstruction directly in 3D.
- A temporally consistent canonical human representation, global sequence-level optimization, and coarse-to-fine volume-rendering sampling form the core method.
- The paper evaluates segmentation, novel view synthesis, and reconstruction, reports the best performance across several datasets and settings, and introduces a semi-synthetic test set with accurate 3D geometry.
- The method demonstrates detailed reconstruction of different humans from online videos and hand-held mobile phone clips.
2. Related Work
Prior monocular human-reconstruction methods rely on scanned templates, explicit meshes, or learned image-level cues, while multi-view systems remain expensive and specialized. Neural scene-decomposition methods reduce some constraints but do not directly address articulated humans with intricate motions.
- Traditional monocular performance-capture methods require personalized pre-scanned templates and manual rigging, limiting deployment in real-life applications.
- Explicit mesh approaches avoid some scanning requirements but remain constrained by fixed representation properties.
- High-fidelity dynamic-human reconstruction traditionally depends on calibrated dense multi-view systems that are expensive, laborious, and expertise-intensive.
- Image-level moving-object segmentation uses motion cues or annotated video matting but lacks 3D knowledge and struggles with complicated backgrounds lacking sufficient color contrast.
- Methods that jointly optimize multiple neural fields can decompose simpler dynamic objects and static backgrounds but are not directly applicable to articulated humans with intricate motions.
3. Method
Vid2Avatar reconstructs humans and backgrounds jointly in 3D from monocular videos using separate neural fields and a temporally consistent canonical human representation. Global optimization, composited rendering, and scene-decomposition objectives separate dynamic subjects from backgrounds while producing detailed geometry.
- Vid2Avatar reconstructs detailed human geometry and appearance from challenging in-the-wild monocular videos without external segmentation or large clothed-human scan priors.
- Implicit Neural Avatar Representation: The human is represented by pose-conditioned implicit SDF and texture fields in canonical space, with pose inputs modeling local non-rigid changes such as clothing wrinkles.
- Implicit Neural Avatar Representation: Sampled foreground points are inverse-warped into canonical space, while skeletal deformation provides mappings between canonical and deformed points for temporally consistent reconstruction.
- Composited Volume Rendering: Canonical texture predicts colors from points, deformed-space normals, pose parameters, and geometry features, while surface-based foreground rendering is composited with standard background volume rendering.
- Composited Volume Rendering: The background uses a separate inverted-sphere neural radiance field with per-frame latent codes to accommodate lighting and other temporal background changes.
- Scene Decomposition Objectives: Scene decomposition combines opacity sparseness from the canonical human shape with binary ray classification, penalizing opacity on non-human rays and encouraging zero-or-one ray distributions.
4. Experiments
Experiments evaluate Vid2Avatar through ablations and comparisons spanning 2D segmentation, novel-view synthesis, and human-surface reconstruction. Across these settings, the method produces clean decomposition, detailed reconstructions, and strong qualitative results on in-the-wild videos.
- Experimental Setup: The experiments cover ablations, 2D segmentation, novel-view synthesis, reconstruction, and qualitative evaluation on videos from multiple sources.Evaluations use MonoPerfCap, NeuMan, 3DPW, SynWild, and online or self-captured videos.
- Ablation Study: Jointly optimizing pose, shape, and appearance improves global pose alignment and normal consistency.The ablation compares the full model against a version without joint pose optimization.
- Ablation Study: The proposed scene decomposition loss prevents noisy segmentation that includes background regions.Removing this loss causes the segmentation to become noisy and incorporate parts of the background.
- 2D Segmentation Comparisons: Vid2Avatar consistently outperforms 2D segmentation baselines on all MonoPerfCap metrics and generates plausible masks around visually difficult feet regions.The method uses ray opacity αH(r)=1 to extract human-mask pixels and handles low photometric contrast between shoes and stairs.
- View Synthesis Comparisons: On NeuMan test views, Vid2Avatar achieves comparable or better rendering quality than NeRF-based methods and produces cleaner human-background separation.HumanNeRF and NeuMan require additional segmentation inputs and show artifacts around feet and arms.
- Reconstruction Comparisons: On 3DPW and SynWild, Vid2Avatar outperforms all baselines across metrics and reconstructs complete, detailed, temporally consistent human bodies.Qualitative comparisons highlight recovered cloth wrinkles and facial features, while competing methods can produce missing limbs or physically incorrect bodies.
- Qualitative Results: Qualitative results show detailed reconstruction of complex cloth deformations and personalized facial features from online, dataset, and self-captured monocular videos.The results include hand-held mobile phone clips and other in-the-wild sources.
5. Conclusion
Vid2Avatar reconstructs detailed 3D avatars from monocular in-the-wild videos through self-supervised 3D scene decomposition, without groundtruth supervision, clothed-human-scan priors, or external segmentation. It jointly optimizes background, canonical human shape and appearance, and pose estimates, while requiring reasonable pose inputs and struggling with loose clothing.
- Vid2Avatar reconstructs detailed 3D avatars from monocular in-the-wild videos via self-supervised scene decomposition.
- The method avoids groundtruth supervision, clothed-human-scan priors, and external segmentation modules.
- A global differentiable volume-rendering optimization jointly updates the background field, canonical human shape and appearance, and human pose estimates across the sequence.
- The method achieves robust and high-fidelity human reconstruction from monocular videos.
- Vid2Avatar relies on reasonable pose estimates and faces significant challenges with loose clothing such as skirts or free-flowing garments.