PaperlayerView exact revision on arXiv

Source-linked AI summary

Animatable Neural Radiance Fields from Monocular RGB Videos

Jianchuan Chen, Ying Zhang, Di Kang, Xuefei Zhe, Linchao Bao, Xu Jia, Huchuan Lu

arXiv:2106.13629v2cs.CV

TL;DR

Detailed human reconstruction from monocular video remains challenging, and existing NeRF methods are limited in handling large human movements. This paper combines NeRF with explicit SMPL-guided deformation and pose refinement to create a controllable canonical human representation. The authors report detailed reconstruction, photo-realistic novel views, and synthesis of novel poses.

  • Problem

    Detailed human geometry is challenging to reconstruct, and NeRF approaches described by the authors are limited to nearly static scenes or small movements rather than humans with large movements.

    Source: §I · p. 1

  • Method

    The method uses SMPL-guided deformation to map observations into a canonical NeRF space and jointly refines SMPL and NeRF parameters to compensate for inaccurate pose estimates.

    Source: §I · p. 1, B. Pose-guided Deformation · p. 2

  • Results

    The authors report detailed human reconstruction from monocular video, photo-realistic novel-view rendering, and synthesis of images with novel poses.

    Source: §I · p. 1

  • Takeaways & Limitations

    The learned canonical-space representation is controllable through SMPL deformation and can be animated with novel poses.

    Source: B. Pose-guided Deformation · p. 2

    The method has difficulty with extremely loose clothing or complex non-rigid garment deformation because its explicit pose-guided deformation associates spatial points with the SMPL mesh.

    Source: §VI · p. 10

Abstract

from arXiv · show

We present animatable neural radiance fields (animatable NeRF) for detailed human avatar creation from monocular videos. Our approach extends neural radiance fields (NeRF) to the dynamic scenes with human movements via introducing explicit pose-guided deformation while learning the scene representation network. In particular, we estimate the human pose for each frame and learn a constant canonical space for the detailed human template, which enables natural shape deformation from the observation space to the canonical space under the explicit control of the pose parameters. To compensate for inaccurate pose estimation, we introduce the pose refinement strategy that updates the initial pose during the learning process, which not only helps to learn more accurate human reconstruction but also accelerates the convergence. In experiments we show that the proposed approach achieves 1) implicit human geometry and appearance reconstruction with high-quality details, 2) photo-realistic rendering of the human from novel views, and 3) animation of the human with novel poses.

I. INTRODUCTION

The paper targets detailed, animatable human avatars from monocular RGB video, combining NeRF with explicit SMPL-guided deformation and pose refinement. It addresses challenges in capturing human detail and handling large movements, while distinguishing its approach from prior body-model and neural-rendering methods.

I. INTRODUCTION

  • High-quality human reconstruction often requires multiple synchronized cameras or RGB-D sensors, while modeling details such as hair, glasses, and cloth wrinkles remains challenging.The paper targets monocular RGB videos as an accessible input form.Source: §I · p. 1
  • The approach combines NeRF with SMPL-guided deformation into a canonical space and jointly optimizes NeRF and SMPL parameters to address inaccurate pose estimates.The authors state that joint optimization improves reconstruction and accelerates training convergence.Source: §I · p. 1
  • The method reconstructs detailed human geometry from monocular RGB video, renders photo-realistic novel views, and synthesizes images with novel poses.The authors report high-quality details including clothing and hair.Source: §I · p. 1

II. RELATED WORK

  • Parametric body-model methods produce animatable textured humans but struggle with complex geometry, while implicit surface methods handle complex poses and clothing but may look blurry and require registration for animation.The paper also notes that combining implicit representations and parametric models can improve robustness and support animation.Source: §II · pp. 1–2
  • Dynamic NeRFs use learned deformation fields but offer difficult-to-control human motion; this approach instead uses SMPL to explicitly deform points and fine-tunes its parameters.Related methods using similar explicit combinations commonly require multi-view images or accurately registered SMPL, making monocular video more challenging.Source: §II · p. 2
  • GAN-based human-motion-transfer methods can show convincing performance without recovering detailed 3D geometry, so generated-human geometry is usually inconsistent across views and motions; neural-rendering methods fail to produce photorealistic results for novel poses unseen during training.Source: §II · p. 2

III. METHOD

The method estimates camera and SMPL parameters from each frame, then renders a canonical-space NeRF after transforming observation-space points according to pose. Its explicit deformation uses nearby SMPL vertices to map points between poses and supports joint refinement of the scene and body parameters.

III. METHOD

  • The pipeline estimates camera intrinsics and SMPL parameters per frame, masks the foreground, renders the canonical-space NeRF, and jointly optimizes NeRF and SMPL parameters.Background and pose regularization are introduced to improve optimization robustness.Source: §III · p. 2, Figure 1 · p. 3

A. Animatable Neural Radiance Fields

  • Animatable NeRF maps 3D position, pose, and shape to color and density after deforming the position from observation space into canonical space.The method removes viewing direction from the input because human skin and clothing are mainly diffuse reflectors.Source: A. Animatable Neural Radiance Fields · p. 2

B. Pose-guided Deformation

  • SMPL explicitly transforms points from observation space to a constant canonical template pose, which supports a controllable representation and reduces reliance on diverse training poses for unseen-pose generalization.The canonical template pose is defined as an X-pose.Source: B. Pose-guided Deformation · pp. 2–3
  • For points near the mesh, deformation follows neighboring SMPL vertices; their influence weights use blend-skinning information and spatial distance.The transformation extends vertex-defined skinning to nearby space, where points may be influenced by multiple body parts.Source: B. Pose-guided Deformation · p. 3

C. Volume Rendering

The method renders images by accumulating color and density samples along camera rays, with a body-surface prior guiding sampling and deformation. It assumes density should vanish far from the human mesh.

C. Volume Rendering

  • For each frame, camera rays sample points between near and far planes, whose colors and densities are accumulated to render pixels.The continuous integration is approximated with N samples along each ray.Source: C. Volume Rendering · p. 3
  • The near-surface prior provides geometric guidance and addresses ambiguity during pose-guided deformation.It is based on weighted distance to nearby mesh vertices and a distance threshold.Source: C. Volume Rendering · p. 4
  • The method assumes density is zero for points far from the human-mesh surface.Source: C. Volume Rendering · p. 4
  • Hierarchical volume sampling uses coarse and fine networks to obtain rendered colors.Source: C. Volume Rendering · p. 4

D. Pose Refinement via Analysis-by-Synthesis

The method initializes per-frame SMPL parameters with VIBE estimates and refines them during training. The refined parameters fit input images better and produce clearer, sharper results.

D. Pose Refinement via Analysis-by-Synthesis

E. Objective Functions

The objective combines image reconstruction, pose regularization, and background regularization. These terms respectively fit rendered colors to observations, stabilize pose estimates, and align rendered density with segmentation masks.

E. Objective Functions

  • The objective combines reconstruction loss, pose regularization, and background regularization, with a weight balancing the background term.Source: E. Objective Functions · p. 4
  • Reconstruction loss minimizes the difference between rendered colors and observed image colors along camera rays.Source: E. Objective Functions · p. 4
  • Pose regularization keeps optimized poses close to their initial estimates and encourages similarity between adjacent frames.Source: E. Objective Functions · p. 4
  • Background regularization aligns rendered integral density with segmentation masks, encouraging density in the human region and zero density in the background.The stated aim is cleaner empty-space estimation and a more solid, clear person in canonical space.Source: E. Objective Functions · p. 4

F. Applications

The learned representation supports human reconstruction from monocular video, novel-view rendering, surface extraction, and animation in desired poses.

F. Applications

  • The animatable NeRF reconstructs implicit human geometry and appearance from monocular video of a person turning before a camera in an A-pose.Source: F. Applications · p. 4
  • The learned representation supports novel-view rendering and surface-geometry extraction with Marching Cubes.Source: F. Applications · p. 4
  • SMPL-guided deformation lets the neural radiance field render the reconstructed person in desired poses.Source: F. Applications · p. 4

IV. EXPERIMENTS

The experiments evaluate animatable NeRF on real and synthetic human data, testing novel-view synthesis, novel-pose synthesis, and 3D reconstruction. The setup includes fixed implementation choices and task-specific evaluation protocols.

A. Implementation Details

  • All experiments use coarse and fine networks, 64 coarse and 64 + 32 fine ray samples, 512×512 images, and Adam training on two RTX 3090 GPUs.Training takes about 13 hours; 90% of rays are sampled from the foreground and 10% from the background.Source: A. Implementation Details · pp. 4–5

B. Datasets and Evaluation

  • Figure 2 reports that pose refinement improves robustness when estimated SMPL poses are reasonably good, while inaccurate poses leave NeRF+U blurry.Table I is identified as a quantitative comparison of novel-view synthesis on People-Snapshot and iPER, but its values are not supplied here.Source: Figure 2 · p. 5, Table I · p. 6
  • People-Snapshot and iPER evaluate novel-view and novel-pose synthesis, while synthetic Multi-Garment data evaluate 3D reconstruction.People-Snapshot and iPER contain monocular videos; Multi-Garment provides scanned bodies, textures, and registered SMPLD models.Source: B. Datasets and Evaluation · p. 5
  • Evaluation uses A-pose frames for training, held-out A-pose frames for novel views, and random-pose frames for novel poses.Metrics are PSNR, SSIM, and LPIPS for image synthesis, and P2S and Chamfer distance for geometry; real-scene datasets lack ground-truth geometry.Source: B. Datasets and Evaluation · p. 5

C. Novel view synthesis

The novel-view experiments compare the method with NeRF, NeRF+U, NeuralBody, and SMPLpix on People-Snapshot and iPER. The reported results favor the proposed approach, while the view test uses only slightly different poses.

C. Novel view synthesis

  • The held-out A-pose frames test rendered views, but the views differ only slightly in human pose.SMPL parameters for test frames are optimized while the NeRF network remains fixed.Source: C. Novel view synthesis · p. 6
  • Figure 4 shows that NeRF and NeRF+U fail at 3D reconstruction, while the proposed method produces more reasonable geometry than NeRF+L.The NeRF+U(GT) reference uses ground-truth SMPL and learns geometry with very high precision; Table II compares reconstruction on Multi-Garment.Source: Figure 4 · p. 7, Table II · p. 7
  • The proposed approach achieves higher PSNR and SSIM than the compared approaches on People-Snapshot and iPER.The authors report more realistic and reliable results; pose refinement significantly improves synthesis quality over NeRF+U with inaccurate SMPL parameters.Source: C. Novel view synthesis · pp. 6–7

D. 3D human reconstruction

The experiments report improved 3D reconstruction and novel-pose synthesis, with detailed results depending on pose quality and clothing motion. The discussion identifies pose refinement, canonical poses, background regularization, and view direction as topics for analysis.

D. 3D human reconstruction

  • The proposed approach achieves lower P2S and Chamfer distances than the compared strategies on Multi-Garment reconstruction.Qualitative results capture cloth wrinkles, faces, and hair; ground-truth SMPL yields still lower distances than all approaches.Source: D. 3D human reconstruction · pp. 7–8
  • The iPER reconstruction captures details such as a hood and pigtail when clothing does not deform violently.The pose-guided deformation does not model clothing deformation.Source: D. 3D human reconstruction · p. 8

E. Novel pose synthesis

  • The method achieves much better novel-pose synthesis results than NeuralBody, preserving identity and cloth details on poses substantially different from training poses.Training uses A-pose videos and testing uses random-pose videos of the same person; Table III compares novel-pose synthesis on iPER.Source: E. Novel pose synthesis · p. 8, Table III · p. 8

V. DISCUSSION

  • The discussion examines pose refinement, canonical poses, background regularization, and view direction.Figures 7 and 8 show novel-pose results, including synthesis from models trained only on A-pose images.Source: Figure 7 · p. 8, §V · p. 9, Figure 8 · p. 9

A. Analysis of Pose Refinement

Pose refinement improves alignment between the SMPL model and input images, while canonical-pose choice affects canonical-space reconstruction and novel-pose synthesis.

A. Analysis of Pose Refinement

  • After pose refinement, the SMPL model fits the input image better than the initial VIBE estimate, including at the foot joints.The initial estimate can be misaligned even for a simple A-pose, reflecting depth ambiguity in monocular pose estimation.Source: A. Analysis of Pose Refinement · p. 9, Figure 9 · p. 9

B. Analysis of Canonical Poses

  • A-pose gives the best canonical-space NeRF learning, whereas T-pose and X-pose produce artifacts under the axilla and thighs.The paper describes A-pose as the average training-frame pose, T-pose as SMPL’s rest pose, and X-pose as a customized pose with more spread body parts; proximity between body parts complicates deformation because SMPL omits clothing offsets.Source: B. Analysis of Canonical Poses · p. 9, Figure 10 · p. 10
  • A-pose is best for reconstruction, but X-pose is better for novel-pose synthesis; A-pose and T-pose can produce multiple-leg artifacts.In the canonical space of A-pose or T-pose, nearby body parts can be deformed by one another.Source: B. Analysis of Canonical Poses · pp. 9–10, Figure 11 · p. 10

C. Impact of Background Regularization

Background regularization reduces empty-space artifacts, and excluding viewing direction produces more consistent novel views. Complex poses, loose clothing, and unseen body regions remain important limits.

C. Impact of Background Regularization

D. Impact of viewing direction

  • Removing viewing direction from the input produces more consistent novel-view results across views.The method assumes subject appearance is not view-dependent; including viewing direction during training produced unpredictable artifacts in the reported experiment.Source: Figure 13 · p. 10, D. Impact of viewing direction · p. 10

VI. LIMITATIONS

  • Complex-pose videos yield reasonable results but lose detail; extremely loose clothes, complex garment deformation, and invisible body regions also limit reconstruction.The authors recommend slow turns with simple poses and clothing that remains nearly still relative to the body, and videos covering the whole body as much as possible.Source: §VI · p. 10

VII. CONCLUSION

  • The approach learns an animatable NeRF through explicit pose-guided deformation and pose refinement, with experiments on synthetic and real data demonstrating its effectiveness.The conclusion describes a controllable canonical-space template and refinement that compensates for inaccurate pose estimates to provide more consistent geometry and appearance guidance.Source: §VII · p. 11

Related papers

Loading 2106.13629v2…