Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies

Sida Peng; Junting Dong; Qianqian Wang; Shangzhan Zhang; Qing Shuai; Xiaowei Zhou; Hujun Bao

Source-linked AI summary

Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies

Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, Hujun Bao

arXiv:2105.02872v2May 6, 2021cs.CV

TL;DR

Reconstructing animatable human models from multi-view video is difficult because existing deformation representations are under-constrained and cannot explicitly respond to input motions. This paper uses neural blend weight fields with 3D skeletons to build an animatable radiance field, achieving state-of-the-art novel-view and novel-pose synthesis across H36M and ZJU-MoCap.

Problem
Existing translational-vector and SE(3) deformation fields make optimization under-constrained and cannot explicitly synthesize new scenes from input motions.
Method
The method combines neural blend weight fields with 3D human skeletons to generate observation-to-canonical and canonical-to-observation deformation correspondences.
Results
Across H36M and ZJU-MoCap video sequences, the approach achieves state-of-the-art novel-view and novel-pose synthesis performance.
Takeaways & Limitations
The learned representation can synthesize free-viewpoint videos of performers given novel motion sequences.
Takeaways & Limitations
Performance degrades for performers wearing loose clothing because skeleton-driven deformation cannot express complex non-rigid garment deformations.

Abstract

from arXiv · show

This paper addresses the challenge of reconstructing an animatable human model from a multi-view video. Some recent works have proposed to decompose a non-rigidly deforming scene into a canonical neural radiance field and a set of deformation fields that map observation-space points to the canonical space, thereby enabling them to learn the dynamic scene from images. However, they represent the deformation field as translational vector field or SE(3) field, which makes the optimization highly under-constrained. Moreover, these representations cannot be explicitly controlled by input motions. Instead, we introduce neural blend weight fields to produce the deformation fields. Based on the skeleton-driven deformation, blend weight fields are used with 3D human skeletons to generate observation-to-canonical and canonical-to-observation correspondences. Since 3D human skeletons are more observable, they can regularize the learning of deformation fields. Moreover, the learned blend weight fields can be combined with input skeletal motions to generate new deformation fields to animate the human model. Experiments show that our approach significantly outperforms recent human synthesis methods. The code and supplementary materials are available at https://zju3dv.github.io/animatable_nerf/.

1. Introduction

The paper targets automatic reconstruction of animatable humans from multiview videos, replacing under-constrained deformation fields with neural blend weight fields guided by 3D skeletons. The approach supports novel-view and novel-pose synthesis while recovering canonical-space human geometry.

Motivation: Traditional animatable-human reconstruction is costly because it requires complex capture hardware and skilled manual animation.Applications include free-viewpoint video, telepresence, video games, and movies.
Problem: The work automatically reconstructs animatable humans from multiview videos, addressing representation and learning challenges.The stated goal is to reduce reconstruction and animation costs and enable digital-human creation at scale.
Limitations: Existing canonical-NeRF methods use translational or SE(3) deformation fields, making joint optimization without motion priors highly under-constrained and preventing explicit motion-controlled synthesis.These methods decompose dynamic scenes into a canonical NeRF and observation-to-canonical deformation fields.
Method: Neural blend weight fields combine with 3D human skeletons to generate deformation fields for animatable human reconstruction.Tracked skeletons regularize deformation learning without requiring joint optimization, while learned blend weights can support animation from input skeletal motions.
Results: On H36M and ZJU-MoCap, the approach achieves state-of-the-art novel-view and novel-pose synthesis across video sequences.It also reconstructs canonical-space 3D human shape and reposes the geometry.

2. Related work

Prior work spans hardware-intensive and differentiable-renderer-based human reconstruction, skeletal and SMPL-based animation, and neural rendering for pose synthesis and novel-view generation. NeRF and SMPL have also been combined to model dynamic humans.

Human reconstruction: Differentiable-renderer methods reduce the number of required camera views while achieving impressive image-based 3D reconstruction results.Earlier human reconstruction methods often rely on complicated hardware.
Human animation: Skeletal animation assigns mesh vertices blend weights that describe how their positions deform with a human skeleton.SMPL learns a skeleton regressor and blend weights from many ground-truth 3D meshes.
Neural rendering: Image-to-image methods can synthesize photorealistic images under novel human poses but struggle with novel-view rendering.These methods map 2D skeleton images to target rendering results.
Neural rendering: Neural radiance fields represent scenes with color and density fields, support volumetric rendering, and achieve state-of-the-art novel-view synthesis performance.Related approaches interpret 3D features as colors and accumulate them into 2D images; one method combines NeRF with SMPL for dynamic humans.

3. Method

The method decomposes a dynamic human into a canonical NeRF and per-frame neural blend weight fields, using 3D skeleton and statistical body-model priors to regularize deformation learning. These fields establish observation-to-canonical correspondences and enable animation under novel skeletal poses.

Problem Setup: Given synchronized, calibrated multi-view video, per-frame 3D skeletons and foreground masks, the method reconstructs an animatable performer model for free-viewpoint synthesis.Skeletons may come from marker-based or marker-less pose estimation, while foreground masks are extracted per image.
Dynamic Radiance Field: The representation combines a canonical human NeRF with per-frame blend weight fields that establish observation-to-canonical and canonical-to-observation correspondences.Each frame’s deformation field maps observation-space points into canonical space, while radiance fields model density and appearance.
Neural Blend Weight Fields: Skeleton-driven deformation and statistical body-model weights regularize the otherwise ill-posed blend-weight optimization through residual vectors predicted by a shared per-frame-conditioned MLP.Initial weights are computed from SMPL or another statistical human model, and the network learns residual corrections.
Neural Blend Weight Fields: Observation-space blend weights provide direct correspondences, avoiding the complex, time-consuming, and coarse interpolation required when only canonical-space weights are learned.Canonical-space weights are additionally learned for template animation using consistency between corresponding canonical and observation points.
Learning and Animation: The canonical radiance, color, deformation, and latent-code parameters are jointly optimized over multi-view video with rendering loss, while novel poses generate new deformation fields from updated SMPL weights.For a novel pose, the method updates SMPL parameters, predicts new neural blend weights with a new latent code, and constructs the deformation field.

4. Implementation details

The method uses NeRF-style radiance-field networks with neural blend weight fields and 128-dimensional appearance and blend-weight codes. Training proceeds in two stages with Adam, exponentially decaying learning rates, and four 2080 Ti GPUs.

Network architecture: NeRF-style networks use single-level sampling with 64 points per camera ray, while F∆w outputs 24 channels and applies exp(·).Fσ and Fc closely follow the original NeRF architecture, and F∆w otherwise follows Fσ.
Network architecture: The appearance code ℓi and blend weight field code ψi each have dimension 128.
Training: Training jointly optimizes Fσ, Fc, F∆w, {ℓi}, and {ψi} in the first stage, then learns neural blend weight fields under novel poses using equation (9).The method follows a two-stage training pipeline.
Training: Adam training starts at 5e−4 and exponentially decays to 5e−5 during optimization on four 2080 Ti GPUs.

5. Experiments

Experiments on H36M and ZJU-MoCap evaluate novel-view and novel-pose synthesis against SMPL-based baselines, while ablations examine pose accuracy, video length, and input views. The method achieves stronger quantitative and qualitative results, with fast rendering and interpretable residual deformations.

Experimental setup: Experiments use H36M multi-view videos with three cameras for training and one held-out camera for testing across seven subjects performing representative complex actions.SMPL parameters are obtained from the 3D human poses.
Novel pose synthesis: The method achieves the best PSNR and SSIM against Neural Textures and NHR for novel-pose synthesis and outperforms the latest method on ZJU-MoCap.Qualitatively, competing methods produce blurry or distorted images for complex poses, whereas this method renders higher-quality results with better controllability.
Ablation studies: More accurate human poses improve rendering quality, while training on 150∼300 video frames is empirically suitable for most subjects.Training helps representation learning, but very long videos are difficult to fit.
Ablation studies: Models trained on one, two, and three views have similar quantitative performance, although three views render more details and one view already gives reasonable results.The ablation uses one view for testing and selects nearest views for training.
Efficiency: Rendering a 512 × 512 image takes 1.09s, including 0.39s for color and density prediction, 0.63s for blend-weight prediction, and 0.07s for volume rendering.The reported speed uses an Intel i7 3.7GHz CPU and GTX 1080 Ti GPU.

6. Limitations

The method has limitations in modeling loose-clothing deformations, relying on estimated skeletons, requiring per-scene training, and rendering relatively slowly. Future work includes refining poses during training, cross-video generalization, shorter training, and caching-based acceleration.

Modeling limitations: The skeleton-driven deformation model cannot express complex non-rigid garment deformations, degrading reconstruction for performers wearing loose clothes.The paper suggests augmenting neural radiance fields with deformation modeling to address this limitation.
Pose limitations: The method relies on estimated human skeletons, motivating future work on refining human poses during training.
Training limitations: The model is trained per-scene, requiring substantial time to produce animatable human models; cross-video generalization and reduced training time remain future work.
Rendering limitations: Rendering is relatively slow, though recent caching-based techniques could potentially address this limitation.

7. Conclusion

The paper introduces a dynamic human representation that augments a neural radiance field with skeleton-driven deformation fields for modeling animatable human characters from multi-view videos.

7. Conclusion: The method learns neural blend weight fields to generate observation-to-canonical and canonical-to-observation correspondences within a skeleton-driven deformation framework.These deformation fields augment a neural radiance field by transforming observation-space points to canonical space.

1. Derivation of transformation matrices

The method represents a human skeleton using joint locations and relative axis-angle rotations, then derives part transformations from a canonical pose to a target pose. It uses Rodrigues-converted rotations and joint centers along the kinematic tree, adopting the 24-part SMPL skeleton in practice.

Derivation of transformation matrices: The human skeleton is represented as joint locations J and relative axis-angle rotations θ for K joints in a kinematic tree.J contains K three-dimensional joint locations, while θ contains K + 1 relative rotations with respect to parent parts.
Derivation of transformation matrices: Part k’s transformation from canonical pose θc to target pose θt is derived from the ordered parent-joint chain.The transformation uses the kinematic structure connecting joint k to its parent joints.
Derivation of transformation matrices: Each relative rotation is converted to a rotation matrix R(ωi) using the Rodrigues formula and combined with the corresponding joint centers ji.The derivation also uses P(k), the ordered set of parent joints of joint k.
Derivation of transformation matrices: The implementation adopts the SMPL skeleton with K = 24 parts, while the transformation derivation also applies to other human skeletons.The paper explicitly instantiates the method with SMPL but notes that the idea is not limited to that skeleton.

2. Network architectures

The paper presents the architectures of its NeRF network and neural blend weight field network in Figures 9 and 10, respectively.

The NeRF network architecture is presented in Figure 9.
The neural blend weight field network architecture is presented in Figure 10.

3. Training and test data

Training uses 150–300 frames per subject, with the remaining video frames reserved for testing to accommodate different video lengths.

3. Training and test data: Each subject uses an appropriate 150–300 video frames for training, while the remaining frames are used for testing.The exact training and test frame counts for each H36M subject are reported in Table 8.

4. 3D reconstruction

The method reconstructs human geometry in canonical space and animates it using the learned blend weight field with input human poses. Reconstructions may be rough because of inaccurate segmentation, so Gaussian smoothing is additionally applied.

Canonical reconstruction and animation: The learned blend weight field is used to animate the reconstructed geometry.This demonstrates that the reconstructed model supports deformation through the learned blend-weight representation.
Reconstruction refinement: Rough reconstruction geometry may result from inaccurate segmentation, and Gaussian smoothing is applied to address this issue.Additional results are provided in the supplementary video.
Canonical reconstruction and animation: Reconstructed geometries are represented in canonical space and can be animated according to input human poses.Figure 11 shows both canonical reconstructions and their reposed geometries.