PaperlayerView exact revision on arXiv

Source-linked AI summary

ARAH: Animatable Volume Rendering of Articulated Human SDFs

Shaofei Wang, Katja Schwarz, Andreas Geiger, Siyu Tang

arXiv:2210.10036v1cs.CV

TL;DR

Sparse multi-view RGB videos leave clothed-human avatar geometry underconstrained and do not reliably support unseen poses. ARAH combines articulated SDFs with volume rendering and joint root-finding, achieving improved geometry, appearance, and out-of-distribution pose animation. The method remains relatively slow at inference and can overfit texture into geometry.

  • Problem

    Sparse multi-view RGB videos provide limited supervision, while existing NeRF approaches suffer inaccurate geometry and unreliable mapping to unseen poses.

  • Method

    ARAH combines articulated SDF geometry and volume rendering with joint root-finding for simultaneous ray-surface intersection and canonical correspondence search.

  • Results

    ARAH improves geometry reconstruction, novel-pose synthesis, and novel-view synthesis, including plausible results under extreme out-of-distribution poses.

  • Takeaways & Limitations

    The pipeline creates animatable clothed avatars with detailed pose-dependent geometry and appearance from sparse multi-view RGB videos.

  • Takeaways & Limitations

    Inference is relatively slow because iterative root-finding and volume rendering are major bottlenecks, and reconstruction can overfit texture into geometry.

Abstract

from arXiv · show

Combining human body models with differentiable rendering has recently enabled animatable avatars of clothed humans from sparse sets of multi-view RGB videos. While state-of-the-art approaches achieve realistic appearance with neural radiance fields (NeRF), the inferred geometry often lacks detail due to missing geometric constraints. Further, animating avatars in out-of-distribution poses is not yet possible because the mapping from observation space to canonical space does not generalize faithfully to unseen poses. In this work, we address these shortcomings and propose a model to create animatable clothed human avatars with detailed geometry that generalize well to out-of-distribution poses. To achieve detailed geometry, we combine an articulated implicit surface representation with volume rendering. For generalization, we propose a novel joint root-finding algorithm for simultaneous ray-surface intersection search and correspondence search. Our algorithm enables efficient point sampling and accurate point canonicalization while generalizing well to unseen poses. We demonstrate that our proposed pipeline can generate clothed avatars with high-quality pose-dependent geometry and appearance from a sparse set of multi-view RGB videos. Our method achieves state-of-the-art performance on geometry and appearance reconstruction while creating animatable avatars that generalize well to out-of-distribution poses beyond the small number of training poses.

1 Introduction

Sparse multi-view RGB videos offer accessible but weakly supervised input for clothed-human avatar reconstruction. ARAH addresses inaccurate geometry and poor unseen-pose generalization with articulated SDFs and joint root-finding.

  • Sparse RGB videos are accessible for avatar reconstruction but provide the least supervision, making clothed-human reconstruction and animation especially challenging.
  • NeRF-based approaches achieve visual fidelity but lack geometric regularization and condition representations on observation-space inputs, causing inaccurate geometry and poor pose generalization.
  • ARAH improves geometry by representing clothed humans with articulated signed-distance fields and improves canonicalization with an efficient joint root-finding algorithm.
  • The model combines forward LBS, canonical SDF, and color networks, using joint root-finding to locate observation-space iso-surface points and their canonical correspondences.
  • On ZJU-MoCap and H36M, ARAH improves novel-pose synthesis, geometry reconstruction, and novel-view synthesis over state-of-the-art methods.

2 Related Works

Prior work spans explicit meshes, implicit functions, and neural rendering for clothed-human modeling. ARAH extends personalized multi-view neural rendering with articulated SDF geometry and a joint root-finding pipeline.

  • Clothed Human Modeling with Explicit Representations: Explicit mesh methods support parametric-body compatibility but can struggle with large garment deformations or require accurate 3D surface reconstruction.
  • Clothed Humans as Implicit Functions: Implicit-function methods model clothed humans from diverse sensors, while some image-based methods require ground-truth geometry or omit pose-dependent cloth deformation.
  • Clothed Humans as Implicit Functions: Personalized per-subject avatar methods can use only 2D supervision, and ARAH follows this personalized-avatar setting.
  • Neural Rendering of Animatable Clothed Humans: Neural Body conditions NeRF on observation-space latent codes, limiting generalization to out-of-distribution poses; ARAH instead uses canonical-space articulated representations.
  • ARAH Pipeline: ARAH jointly searches ray-surface intersections and canonical correspondences before canonical SDF-based volume rendering and end-to-end training.

3 Method

ARAH represents articulated clothed humans with canonical-space neural SDF and color networks driven by learned skinning. Joint root-finding connects observation-space rays to canonical surface correspondences for SDF-based volume rendering.

  • Model Components: The pipeline uses a forward LBS network, canonical SDF network, and canonical color network to model articulated clothed humans.
  • Neural Linear Blend Skinning: Neural LBS predicts 24-dimensional skinning weights for canonical points and applies 24 rigid bone transformations to obtain observation-space points.
  • Neural Linear Blend Skinning: The forward LBS function maps each canonical point to observation space by blending transformed points according to predicted skinning weights.
  • Canonical SDF and Color Networks: The canonical SDF conditions on canonical position, local pose, body shape, and a per-frame latent code to capture pose-dependent and dynamic cloth deformation.
  • Canonical SDF and Color Networks: The color network predicts RGB values from canonical position, transformed normals, viewing direction, SDF features, and a per-frame latent code.
  • Joint Root-Finding: Joint root-finding simultaneously solves for a canonical point and ray depth satisfying the SDF iso-surface and observation-space ray constraints.
  • Joint Root-Finding: Broyden’s method avoids recomputing the Jacobian at every iteration, while joint root-finding has amortized complexity O(M) versus O(MN) for naive alternation and yields about 5× speedup.
  • Differentiable Volume Rendering: SDF-based volume rendering converts SDF values to densities, samples around surface points, and renders colors using standard transmittance-weighted volume rendering.

4 Experiments

The experiments evaluate generalization to unseen poses, geometry reconstruction, and novel-view synthesis using ZJU-MoCap and H36M, with comparisons against Neural Body, Ani-NeRF, and A-NeRF. The approach performs strongly across these evaluations, including limited-data and extreme out-of-distribution poses, while preserving fine geometric details.

  • Experimental Setup: The evaluation benchmarks unseen-pose generalization, geometry reconstruction, and novel-view synthesis against Neural Body, Ani-NeRF, and A-NeRF.Unseen poses use LPIPS, geometry uses L2 Chamfer distance and normal consistency, and novel views use PSNR, SSIM, and LPIPS.
  • Generalization to Unseen Poses: 60–300 repetitive training frames in ZJU-MoCap cause severe baseline overfitting, whereas the proposed method generalizes well to unseen poses despite limited training data.Table 1 reports LPIPS on synthesized images under unseen testing poses.
  • Generalization to Unseen Poses: Extreme AMASS and AIST++ poses produce plausible geometry and renderings with the proposed method, while all baselines show severe artifacts.These poses are out of distribution relative to models trained on ZJU-MoCap.
  • Geometry Reconstruction: The proposed method consistently outperforms NeRF-based human models in geometry reconstruction, producing cleaner surfaces while preserving high-frequency details such as wrinkles.Geometry is evaluated against pseudo-ground-truth using L2 Chamfer distance and normal consistency.
  • Novel View Synthesis on Training Poses: The method also outperforms existing methods on novel-view synthesis for training poses and produces more consistent rendering results across novel views.Table 3 reports PSNR, SSIM, and LPIPS; larger bounding boxes capture loose clothing but make the numbers vary slightly from Neural Body’s evaluation.

5 Conclusion

The paper proposes animatable clothed human avatars from sparse multi-view videos by modeling geometry as articulated SDFs and using joint root-finding for generalization to extreme unseen poses.

  • The approach models clothed-human geometry as articulated signed-distance fields to improve reconstruction over existing methods.
  • The joint root-finding algorithm supports avatar generalization to extreme out-of-distribution poses.
  • Loss Definition: The total training objective combines color, Eikonal, off-surface, inside-point, and skinning losses.
  • Loss Definition: The color loss compares predicted RGB values with ground-truth pixel colors across sampled image pixels.
  • Loss Definition: Eikonal regularization penalizes deviations of the SDF gradient norm from one at sampled canonical-space points.
  • Loss Definition: Off-surface, inside-point, and skinning losses regularize SDF signs and forward LBS weights using sampled canonical-space points.

B Network Architectures

The architecture combines canonical-space forward LBS, SDF, and color networks with differentiable root-finding, enabling gradients through canonical correspondences and pose or shape optimization.

  • Network Architectures: The forward LBS network operates only on canonical-space query points and uses no conditional inputs.
  • Network Architectures: The SDF network uses a hypernetwork conditioned on local pose and shape, while a mapping network modulates layers with per-frame latent codes.
  • Network Architectures: The color network receives canonicalized points, normals, viewing directions, SDF features, and per-frame latent codes.
  • Pose and Shape Optimization: Implicit gradients can also update SMPL pose and shape parameters, improving noisy pose estimates and rendering quality on unseen poses.
  • Implicit Gradients for Joint Root-finding: The joint root-finding formulation simultaneously enforces an SDF iso-surface condition and correspondence between canonical and observation-space points.
  • Implicit Gradients for Joint Root-finding: Implicit differentiation converts root-finding solutions into differentiable point and depth samples for the SDF and color networks.

D Implementation Details

Training uses Adam with per-frame latent codes, sampled foreground and background rays, and surface-focused point sampling; inference renders cropped person regions and reuses the last training latent code for unseen poses.

  • Training: Adam optimizes the models and per-frame latent codes, using learning rates of 1e−6 for the SDF network and 1e−4 for remaining models and codes.
  • Training: Each batch contains 2048 rays, split evenly between foreground-mask and background samples.
  • Training: Rays intersecting a surface receive 16 near and 16 far surface samples, while non-intersecting rays receive 64 samples.
  • Inference: Inference crops an enlarged projected-SMPL bounding box, and unseen poses use the latent code from the last training frame.
  • Inference: Rendering a 512 × 512 image takes about 10–20 seconds, including roughly 1 second for joint root-finding.

E Implementation Details for Baselines

Baseline evaluation uses official implementations where available and compares ray-sampling and canonicalization strategies, while noting important differences between training-pose and unseen-pose evaluation.

  • Baseline Evaluation: Baseline evaluation uses official results, code, and pretrained models when available, with default hyperparameters for remaining sequences.
  • Baseline Evaluation: Neural Body and Ani-NeRF use ground-truth training masks to remove extraneous geometry, but masks are disabled for unseen-pose testing.
  • A-NeRF: A-NeRF produces reasonable training-pose novel views but cannot generalize to unseen poses, with floating blobs attributed to limited poses and keypoint-based conditioning.
  • Ray Sampling: The ray-sampling ablation compares surface rendering, uniform sampling with 64 samples, and the proposed strategy on training-pose novel views.
  • Canonicalization: Canonicalization ablations replace learned forward LBS with learned backward LBS or deterministic nearest-neighbor SMPL skinning weights.
  • Ray Sampling: Surface rendering causes severe geometric artifacts, uniform sampling produces stratified artifacts, and the proposed approach avoids these problems.

F.3 Ablation on Root-finding Initialization

Adding two additional SMPL-bone initializations did not materially change rendering metrics, but substantially increased memory and time consumption.

  • F.3 Ablation on Root-finding Initialization: 31.6/0.973/0.050 with single initialization versus 31.5/0.972/0.049 with two additional initializations showed no noticeable metric change.The reported metrics are PSNR/SSIM/LPIPS, respectively.
  • F.3 Ablation on Root-finding Initialization: The method therefore uses a single root-finding initialization because additional initializations drastically increase memory and time consumption.

G Additional Quantitative Results

The supplementary evaluation reports results on ZJU-MoCap test poses and H36M, alongside complete novel-pose metrics and an ablation of learned LBS networks.

  • Additional Quantitative Results: Complete PSNR, SSIM, and LPIPS results are reported for ZJU-MoCap test poses in Table G.1.
  • Additional Quantitative Results: Quantitative results on H36M are reported following the testing protocols proposed by prior work.
  • Additional Quantitative Results: Backward LBS has difficulty learning weights far from the surface, causing artifacts under specific poses.
  • Additional Quantitative Results: Deterministic SMPL weights produce discretized artifacts on the cloth surface, whereas the proposed approach does not suffer from these problems.

H.1 Qualitative Results on ZJU-MoCap Training Poses

Additional visualizations show the approach preserving detailed appearance on training poses and avoiding artifacts in test and out-of-distribution pose renderings.

  • H.1 Qualitative Results on ZJU-MoCap Training Poses: On ZJU-MoCap training poses, the approach better captures cloth wrinkles, textures, and face details due to stronger geometry constraints.
  • H.1 Qualitative Results on ZJU-MoCap Training Poses: The approach avoids extraneous color blobs under novel views that affect all baselines.
  • H.1 Qualitative Results on ZJU-MoCap Training Poses: On ZJU-MoCap test poses, A-NeRF and Neural Body fail to generalize, while Ani-NeRF produces noisy rendering from inaccurate backward LBS.
  • H.1 Qualitative Results on ZJU-MoCap Training Poses: For out-of-distribution poses, the supplementary figures include comparisons among Neural Body, Ani-NeRF, the proposed method, and its geometry.
  • H.1 Qualitative Results on ZJU-MoCap Training Poses: The closest-training-pose visualization compares each out-of-distribution test pose with four images from its nearest training pose.

I Limitations

The limitations include slow inference and reconstruction bias from texture-driven geometry, while supplementary figures document qualitative behavior across views and poses.

  • I Limitations: Inference is relatively slow because iterative root-finding and volume rendering are the major bottlenecks.
  • I Limitations: Neural rendering methods can overfit geometry to texture, producing reconstruction bias.
  • I Limitations: SDF-based rendering can wrongly reconstruct shirt stripes as geometry, while failing to recover some detailed wrinkles.
  • I Limitations: The paper includes supplementary visualizations of novel-view synthesis, test-pose generalization, out-of-distribution poses, closest training poses, monocular-video generalization, and shape-appearance ambiguity.
Loading 2210.10036v1…