PaperlayerView exact revision on arXiv

Source-linked AI summary

SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video

Boyi Jiang, Yang Hong, Hujun Bao, Juyong Zhang

arXiv:2201.12792v2cs.CVcs.GR

TL;DR

High-fidelity clothed-avatar reconstruction from monocular video is difficult because explicit methods depend on templates while implicit methods lack consistent multi-frame registration. SelfRecon combines both representations with self-supervised losses and produces coherent mesh sequences on self-rotating videos, while remaining limited by optimization time, normal-map reliance, and motion scope.

  • Problem

    Monocular reconstruction must balance explicit methods’ template dependence and implicit methods’ limited space-time coherence for high-fidelity clothed avatars.

  • Method

    SelfRecon combines explicit meshes and implicit SDF geometry, using deformation, mask, neural-rendering, normal, and consistency losses during self-supervised optimization.

  • Results

    SelfRecon extracts space-time coherent mesh sequences without actor-specific templates and outperforms existing methods on self-rotating human videos.

  • Takeaways & Limitations

    The method reconstructs high-fidelity clothed body shapes and demonstrates avatar generation from self-rotating monocular videos.

  • Takeaways & Limitations

    SelfRecon requires relatively long optimization, relies on predicted normal maps for details, and mainly works for self-rotating motions.

Abstract

from arXiv · show

We propose SelfRecon, a clothed human body reconstruction method that combines implicit and explicit representations to recover space-time coherent geometries from a monocular self-rotating human video. Explicit methods require a predefined template mesh for a given sequence, while the template is hard to acquire for a specific subject. Meanwhile, the fixed topology limits the reconstruction accuracy and clothing types. Implicit representation supports arbitrary topology and can represent high-fidelity geometry shapes due to its continuous nature. However, it is difficult to integrate multi-frame information to produce a consistent registration sequence for downstream applications. We propose to combine the advantages of both representations. We utilize differential mask loss of the explicit mesh to obtain the coherent overall shape, while the details on the implicit surface are refined with the differentiable neural rendering. Meanwhile, the explicit mesh is updated periodically to adjust its topology changes, and a consistency loss is designed to match both representations. Compared with existing methods, SelfRecon can produce high-fidelity surfaces for arbitrary clothed humans with self-supervised optimization. Extensive experimental results demonstrate its effectiveness on real captured monocular videos. The source code is available at https://github.com/jby1993/SelfReconCode.

1. Introduction

SelfRecon addresses the difficulty of reconstructing high-fidelity clothed avatars from monocular video by combining explicit and implicit geometry representations. It reduces template dependence while targeting space-time coherent mesh sequences.

  • Monocular high-fidelity avatar reconstruction could broaden applications beyond studios requiring templates, multicamera systems, controlled environments, and extensive artistic work.The paper cites telepresence, AR/VR, anthropometry, and virtual try-on as target applications.
  • Existing explicit monocular capture methods require actor-specific rigged templates and estimate pose and deformation from joints and silhouettes.Explicit representations provide space-time coherence and compatibility with graphics control pipelines.
  • Implicit methods support varied clothing and hairstyles through arbitrary topologies but generally require high-quality 3D supervision and lack sequence-wide vertex coherence.These limitations complicate consistent registration across frames.
  • SelfRecon combines explicit and implicit representations to reconstruct high-fidelity digital avatars from monocular video.
  • The method uses a learnable SDF instead of fixed-topology templates, forward deformation, periodic explicit mesh extraction, and mask and smoothness losses for overall shape recovery.
  • SelfRecon alleviates actor-specific template dependence and extracts space-time coherent mesh sequences, outperforming existing methods on self-rotating human videos.

2. Related Work

Related work spans explicit human models and implicit representations, with prior methods trading representation flexibility, temporal coherence, supervision requirements, or template dependence.

  • Implicit Human Reconstruction: PIFu-style implicit methods represent clothed humans with occupancy fields, while challenging poses can produce incorrect body structures.
  • Implicit Human Reconstruction: Actor-specific implicit reconstructions have become popular, but methods such as NASA and SCANimate target learned representations of particular subjects or scans.
  • Implicit Human Reconstruction: SNARF improves deformation generalization for unseen poses, but these methods require 4D scan data and are difficult to apply broadly to general image data.
  • Implicit Human Reconstruction: NeuralBody and related methods reconstruct or render human motion with implicit fields, but geometry extracted from NeRF can suffer from noise.
  • Explicit Human Reconstruction: SMPL+D extends statistical body models with vertex displacements, yet supports only tight clothing and recovers coarse geometry.
  • Explicit Human Reconstruction: Template-based monocular performance capture methods track specific people effectively, but predefined templates limit broader use.

3. Method

SelfRecon jointly optimizes explicit and implicit geometries through forward deformation, using explicit losses for coherent overall shape and implicit rendering for surface detail. A consistency loss keeps the representations aligned.

  • SelfRecon decomposes forward deformation into learnable per-frame non-rigid deformation and skinning fields to generate space-time coherent explicit meshes.
  • Differentiable masks, regularization, and smoothness losses constrain the explicit mesh shape during optimization.
  • The pipeline maintains explicit and implicit geometry simultaneously while transforming canonical geometry into each frame with forward deformation fields.
  • Non-rigid ray casting finds differentiable intersections with the deformed implicit surface, enabling neural rendering to refine geometry from ray colors.
  • Predicted normal maps refine implicit geometric details, and a consistency loss matches the explicit and implicit representations.
  • Both representations are initialized from an SMPL body mesh generated using initial shape and per-frame pose parameters.

3.1. Canonical Implicit SDF

SelfRecon represents the canonical clothed shape as the zero isosurface of a learnable signed distance field rather than a fixed-topology displacement model. The field is initialized from an SMPL body mesh.

  • The canonical template shape is represented as the zero isosurface of an SDF expressed by an MLP with learnable weights.
  • This SDF representation is chosen because SMPL+D has limited resolution and cannot represent high-fidelity geometry or varied clothing types.
  • IGR initializes the implicit surface as the initial canonical body to avoid unexpected solutions.

3.2. Deformation Fields

SelfRecon combines skeleton skinning with a learnable non-rigid deformation field to map canonical points into each frame while preserving clothing motion.

  • Skinning transformation controls large-scale articulated body movements, but cannot fully represent garments’ non-rigid deformation.
  • A learnable MLP deformation field d models frame-specific non-rigid motion using conditional variable h_i.
  • The skinning field W expands SMPL vertex weights through a fixed sparse grid and interpolates weights for canonical-space points.
  • Composing d and W yields D = W(d(·)), which transforms canonical points into each frame using h_i and pose θ_i.

3.3. Differentiable Non-rigid Ray-casting

SelfRecon uses an explicit deformed mesh to initialize ray–surface intersections, then solves differentiable constraints so rendered points lie on both the implicit surface and camera ray.

  • Because deformation fields make standard sphere tracing infeasible, an explicit mesh guides intersection search on the deformed implicit surface.
  • The first ray intersection with the deformed mesh approximates the implicit intersection, while barycentric correspondence provides a canonical-space initialization.
  • Starting from the canonical initialization, optimization finds p whose deformation x = D_i(p) exactly intersects the ray and lies on S_η.
  • The solver constrains the canonical point toward S_η and its deformed point onto the ray, using ω = 3.05 and 10 gradient-descent iterations.
  • Differentiability is obtained by treating p as a function of camera, deformation, and SDF parameters and differentiating the surface and ray constraints.

3.4. Implicit Rendering Network

The implicit rendering network transfers current-frame geometry attributes into canonical space and predicts view-dependent color for differentiable neural rendering.

  • SelfRecon adapts IDR’s rendering design to non-rigid scenes by converting current-frame attributes into canonical space.
  • An MLP M with learnable weights γ predicts the color L_p of each surface point.
  • The color along a viewing direction depends on SDF and rendering-network weights, camera parameters, and deformation-field parameters.

3.5. Loss Function

SelfRecon jointly optimizes explicit and implicit representations against masks, RGB images, and predicted normals, while regularization and consistency losses coordinate geometry and motion.

  • The optimization variables include camera parameters, sequence-shared MLP weights, and per-frame pose and non-rigid deformation variables.
  • 3.5.1 Explicit Loss: Explicit losses update mesh geometry using mask overlap, deformation regularization, and skeleton smoothness terms.
  • 3.5.1 Explicit Loss: The mask loss renders each deformed mesh with a differentiable point-cloud renderer and compares it with the target mask using IoU.
  • 3.5.1 Explicit Loss: Skeleton smoothness constrains 30 consecutive frames toward a 10-dimensional subspace spanned by the lowest-frequency DCT bases.
  • 3.5.2 Implicit Loss: Implicit losses refine geometry with color and normal supervision plus rigidity and Eikonal regularization near the implicit surface.
  • 3.5.2 Implicit Loss: The normal loss transforms predicted current-frame normals into canonical space with the deformation Jacobian and downweights view-inconsistent normals.
  • 3.5.3 Explicit/Implicit Consistency: After explicit mesh updates, a consistency loss requires updated mesh vertices to match the implicit surface, and optimization alternates explicit and implicit gradients.

4. Experiments

Experiments evaluate SelfRecon quantitatively on synthetic self-rotating sequences and qualitatively on benchmark and smartphone videos. Results cover reconstruction quality, loss ablations, and avatar driving.

  • Quantitative Evaluation: Synthetic evaluation uses Blender and CLO3D sequences, comparing SelfRecon with VideoAvatar using reconstruction, GT, and Chamfer errors.The five sequences include male and female avatars with designed clothing and self-rotating motions.
  • Quantitative Evaluation: SelfRecon significantly reduces the reported error metrics relative to VideoAvatar on the synthetic sequences.
  • Qualitative Evaluation: On PeopleSnapshot comparisons, SelfRecon recovers detailed geometry including facial features and clothing folds, while NeuralBody exhibits noisy surfaces.VideoAvatar approximately captures overall shape but misses some facial and clothing details; NeuralBody alleviates deep ambiguity but has surface noise.
  • Qualitative Evaluation: Smartphone-video experiments produce high-fidelity geometry across varied clothing and body types, with neural renderings close to the input images.
  • Ablation Study: Mask loss alone yields concave, detail-poor geometry; color loss improves details and concavities, while normal loss further removes pits and preserves detail.The normal-loss benefit is especially visible where texture and multiview observations are limited.
  • Ablation Study: Adaptive weighting and a small λi1 are needed because otherwise the normal loss can produce artifacts.
  • Avatar Generation: The extracted texture mesh can be driven with three SMPL pose parameters to generate plausible avatar results.The method extracts a consistent-topology mesh sequence and a texture template with bound skinning weights.

5. Conclusion and Discussion

SelfRecon combines neural implicit representation, neural rendering, and explicit meshes for self-supervised, template-free reconstruction of space-time coherent clothed-body surfaces. Its main limitations are optimization time, reliance on predicted normal maps for geometric detail, and reduced scope beyond self-rotating motions.

  • SelfRecon combines neural implicit representation and neural rendering for self-supervised reconstruction from self-rotating video.
  • Forward deformation enables recovery of space-time coherent surfaces that support downstream applications involving body movement.
  • Combining explicit representation with non-rigid ray casting enables differentiable intersection with the deformed implicit surface.
  • SelfRecon reconstructs high-fidelity clothed body shapes from self-rotating video without pre-computed templates and demonstrates high-fidelity avatar generation.
  • Optimization requires relatively long time, geometric details rely on predicted normal maps, and the method mainly works for self-rotating motions.The paper identifies direct recovery of geometric details from self-supervised rendering loss and more general motion sequences as future directions.
Loading 2201.12792v2…