Source-linked AI summary
Capturing and Animation of Body and Clothing from Monocular Video
Yao Feng, Jinlong Yang, Marc Pollefeys, Michael J. Black, Timo Bolkart
TL;DR
Existing clothed-avatar methods can require extensive 3D training data or jointly represent body and clothing, limiting their separation for uses such as virtual try-on. SCARF combines a mesh-based body with a clothing NeRF and optimizes from monocular video without 3D supervision. The authors report improved visual quality and support for animation, novel views, and clothing transfer.
Problem
Clothed-avatar methods may require large 3D datasets or represent body and clothing holistically, limiting their separation for applications such as virtual try-on.
Source: §1 · p. 2Method
SCARF combines an SMPL-X body mesh with a clothing NeRF and optimizes the avatar from monocular video without 3D supervision.
Source: §1 · p. 2, §3.1 · p. 4Results
SCARF outperforms existing video avatar extraction methods in visual quality and generality, according to the authors.
Source: §6 · p. 7Takeaways & Limitations
The separated representation supports body-pose and shape control, facial expression, novel-view synthesis, and clothing transfer between subjects for virtual try-on.
Source: §1 · p. 2, §6 · p. 7SCARF requires body and clothing segmentation, and segmentation errors can reduce visual quality or impair body–clothing separation.
Source: §5 · p. 7
Abstract
from arXiv · showhide
While recent work has shown progress on extracting clothed 3D human avatars from a single image, video, or a set of 3D scans, several limitations remain. Most methods use a holistic representation to jointly model the body and clothing, which means that the clothing and body cannot be separated for applications like virtual try-on. Other methods separately model the body and clothing, but they require training from a large set of 3D clothed human meshes obtained from 3D/4D scanners or physics simulations. Our insight is that the body and clothing have different modeling requirements. While the body is well represented by a mesh-based parametric 3D model, implicit representations and neural radiance fields are better suited to capturing the large variety in shape and appearance present in clothing. Building on this insight, we propose SCARF (Segmented Clothed Avatar Radiance Field), a hybrid model combining a mesh-based body with a neural radiance field. Integrating the mesh into the volumetric rendering in combination with a differentiable rasterizer enables us to optimize SCARF directly from monocular videos, without any 3D supervision. The hybrid modeling enables SCARF to (i) animate the clothed body avatar by changing body poses (including hand articulation and facial expressions), (ii) synthesize novel views of the avatar, and (iii) transfer clothing between avatars in virtual try-on applications. We demonstrate that SCARF reconstructs clothing with higher visual quality than existing methods, that the clothing deforms with changing body pose and body shape, and that clothing can be successfully transferred between avatars of different subjects. The code and models are available at https://github.com/YadiraF/SCARF.
1 INTRODUCTION
Image-based body models do not capture clothing variation, while holistic clothed-avatar methods limit body–clothing separation and articulated hands and faces. SCARF combines a parametric body mesh with a clothing NeRF to reconstruct, animate, and transfer clothing from monocular video.
1 INTRODUCTION
- Statistical body models trained on minimally clothed scans cannot capture the shape and appearance variation of clothing.Clothing therefore needs a more flexible representation than the body model.Source: §1 · p. 1
- Clothed-avatar methods often require large 3D scan datasets or use holistic representations that limit hand and face articulation and prevent clothing transfer.Layered methods also require 3D clothing scans and may lack clothing appearance or generalize poorly to diverse clothing.Source: §1 · pp. 1–2
- SCARF combines SMPL-X for the body with a NeRF for varied clothing, allowing the two layers to be modeled separately from monocular video.It refines body pose and shape during optimization and learns a non-rigid deformation field to correct clothing deviations from the body.Source: §1 · p. 2
- Clothing segmentation masks penalize the NeRF outside clothed regions, helping keep clothing separate from the body.Source: §1 · p. 2
- SCARF supports body-pose changes, including hand articulation and facial expressions, and enables clothing transfer between avatars for virtual try-on.The clothing is extracted separately from the body.Source: §1 · p. 2
2 RELATED WORK
Prior work spans parametric body models, non-parametric clothing representations, and video-based avatar reconstruction. SCARF addresses their limitations by separately representing the body and clothing, using a NeRF for clothing appearance and variation.
2 RELATED WORK
- Statistical body models recover pose and shape but do not capture clothing or hair, and rarely recover texture information.This work instead captures both a parametric body and non-parametric clothing from monocular video.Source: §2 · p. 2
- Non-parametric clothing models handle varied styles better than mesh offsets, but often recover hands and faces poorly and typically require many manually cleaned 3D scans.Mesh offsets also struggle with varied topologies such as skirts and dresses.Source: §2 · p. 2
- Video-based implicit avatar methods avoid 3D scans, but hand gestures and facial expressions remain difficult to capture and animate; SCARF models clothing separately for transfer and virtual try-on.Prior layered clothing methods may lack appearance fidelity or struggle with real clothing variation, while SCARF optimizes its NeRF clothing layer from scratch.Source: §2 · p. 3
3 METHOD
SCARF combines an SMPL-X body mesh with a clothing-only NeRF, keeping body and clothing separate. It canonicalizes observations, integrates the mesh into volume rendering, and jointly fits the model to monocular video with reconstruction and disentanglement objectives.
3.1 Hybrid Representation
- SCARF combines SMPL-X for the body with a NeRF for clothing, matching each component to its modeling needs and supporting pose, hand, and facial control.Clothing is represented separately, enabling transfer between avatars.Source: §3 · p. 3, §3.1 · pp. 3–4
3.2 Canonicalization
- The model maps observed points into a canonical, star-like body pose using the inverse SMPL-X transformation, generalized from surface vertices to the surrounding space.Nearby mesh vertices and their blend weights contribute to the generalized transformation.Source: §3.2 · p. 4
3.3 Mesh Integrated Volume Rendering
- When a camera ray intersects the body mesh, SCARF ends NeRF sampling at the intersection and uses the mesh color instead of the NeRF color for the terminal sample.A differentiable rasterizer separately renders the colored body mesh.Source: §3.3 · pp. 4–5
3.4 Objectives
- The sequence objective combines image reconstruction, clothing segmentation, and body losses; clothing masks constrain NeRF density to clothing regions.Body losses fit the body silhouette and visible body parts, while additional terms address body color and mesh regularization.Source: §3.4 · p. 5
- Figure 4 reports that SCARF reconstructs different clothing types more faithfully than SMPLicit and BCNet.The caption provides this qualitative comparison without reporting numerical measurements.Source: Figure 4 · p. 6
3.5 Implementation
- The implementation initializes body parameters per frame with PIXIE and optimizes in two stages: 100k iterations before segmentation, then 50k iterations for the full model.The first stage refines the canonical NeRF and SMPL-X pose; the second optimizes the full model.Source: §3.5 · p. 6
4 EXPERIMENTS
SCARF is evaluated for novel-view synthesis, separate body and garment reconstruction, animation, clothing transfer, and representation ablations. It performs better on most reported metrics and supports detailed reposing and cross-avatar clothing transfer.
4.1 Datasets
- The evaluation uses People Snapshot, iPER, SelfRecon, and self-captured sequences, with around 100–150 images per subject.For People Snapshot, provided SMPL poses initialize the method instead of PIXIE.Source: §4.1 · p. 6
4.2 Comparisons
- SCARF is more accurate than NeRF, SMPLpix, Neural Body, and Anim-NeRF under most metrics, with qualitative comparisons showing more detailed hand and face geometry.The evaluation follows the protocol of Chen et al. [2021b].Source: §4.2 · p. 6
4.3 Applications
- SCARF supports reposing with fine-grained body-pose control and transferring clothing between avatars.Figure 6 depicts reposing and dressing a source subject in target clothing.Source: §4.3 · p. 6, Figure 6 · p. 6
4.4 Ablation Experiments
- The hybrid NeRF-and-mesh representation estimates face, hand, and complex clothing geometry better than holistic NeRF or mesh-only variants.Unlike the compared body NeRF methods, the hybrid representation also supports clothing transfer between avatars.Source: §4.4 · p. 7, Figure 7 · p. 7
5 DISCUSSION AND LIMITATIONS
SCARF’s performance and fidelity are bounded by segmentation quality, pose initialization, and unmodeled appearance and motion factors.
5 DISCUSSION AND LIMITATIONS
- Segmentation errors reduce avatar visual quality, and clothing-mask errors impair body-clothing separation.SCARF requires body and cloth segmentation for training; optical-flow temporal consistency could improve segmentation quality.Source: §5 · p. 7
- A poor initial pose may cause SCARF to fail, extreme novel poses can produce artifacts, and clothing dynamics remain future work.The method also retains baked-in lighting and shading and relies on PIXIE expressions that do not capture the full spectrum of emotions.Source: §5 · p. 7
6 CONCLUSION
SCARF extracts animatable clothed 3D avatars from monocular video using a hybrid representation that separately models the body and clothing. It supports pose, shape, and expression changes, unseen views, and clothing transfer, with higher visual quality and generality than existing video-based avatar extraction methods.
6 CONCLUSION
- SCARF combines a mesh-based body model with a neural radiance field to separately model the body and clothing.The factored representation supports clothing transfer between avatars, animation including finger articulation, body-shape and facial-expression changes, and visualization from unseen viewing directions.Source: §6 · p. 7
- SCARF extracts animatable clothed 3D human avatars automatically from monocular video.Source: §6 · p. 7
- SCARF outperforms existing video-based avatar extraction methods in visual quality and generality.Source: §6 · p. 7
A APPENDIX
The supplementary material provides additional dataset and implementation details, along with further results.
A APPENDIX
- The supplementary material includes additional dataset and implementation details and presents further results.Source: A APPENDIX · p. 10
A.1 Implementation Details
The implementation uses staged NeRF training and non-rigid deformation to model clothing motion, with MRF loss capturing additional detail. Optimization takes around 40 hours on an NVIDIA V100.
A.1 Implementation Details
- 40 hours on an NVIDIA V100 is the reported overall optimization time.Source: §A.1 · p. 10
- The first NeRF training stage omits optimization of the non-rigid deformation model and sets the MRF loss weight to zero.Source: §A.1 · p. 10
- In the second stage, non-rigid deformation models clothing changes not explained by body transformation, while MRF loss captures additional details.Source: §A.1 · p. 10
- The implementation specifies loss weights and assigns the offset-loss weight across body, face, and hands in a 2:3:12 ratio.Source: §A.1 · p. 10
A.2 Datasets
The evaluation uses People Snapshot for qualitative and quantitative evaluation, and iPER, SelfRecon, and self-captured videos for qualitative evaluation. The datasets use specified subject, outfit, motion, and frame selections for optimization and testing.
A.2 Datasets
- Four People Snapshot subjects are used for qualitative and quantitative evaluation, following Anim-NeRF settings.The appendix specifies subject-specific train and test frame ranges and step sizes.Source: §A.2 · p. 10
- Four iPER subjects with outfit 1 and motion 1 are used for qualitative evaluation, optimizing frames 1–490 at step 4.Source: §A.2 · p. 10
- Four synthetic SelfRecon videos and one self-captured video are used, with 100 optimization frames per subject.The self-captured evaluation also uses six clothing types and frames 0–400 at step 2 from A-pose videos.Source: §A.2 · p. 10
A.3 Ablation Experiments
Pose refinement improves reconstruction image quality because per-frame pose estimates are not accurate.
A.3 Ablation Experiments
- Pose refinement improves image quality and recovers more detail when per-frame pose estimates are inaccurate.Figure 8 shows more texture details with pose refinement.Source: §A.3 · p. 10
A.4 More Qualitative results
The qualitative results show improved garment reconstruction, pose-refinement benefits, and clothing adaptation to changed body shape; they also identify segmentation, geometry, and unseen-pose limitations.
A.4 More Qualitative results
- SCARF reconstructs clothing with better visual quality than SMPLicit and BCNet, although their training settings differ.The compared methods reconstruct from single images and require 3D scans and manually designed cloth templates; SCARF learns from video without 3D supervision and captures garment appearance.Source: §A.4 · p. 10, Figure 13 · p. 11
- Pose refinement improves novel-view reconstruction quality by recovering more texture details.Figure 8 compares results without and with pose refinement.Source: Figure 8 · p. 10
- Changing SMPL-X shape parameters changes the underlying body shape, and the NeRF clothing adapts accordingly.Figure 9 illustrates clothing adaptation to altered body shape.Source: §A.4 · p. 10, Figure 9 · p. 10
A.5 Limitations
- Segmentation errors can mis-separate body and clothing, reconstructed clothing geometry can be noisy despite good renderings, and unseen poses can produce visual artifacts.The illustrated segmentation error omits a belt from clothing, leaving a visible gap.Source: Figure 10 · p. 10, §A.5 · p. 11, Figure 11 · p. 11