PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering

Youngjoong Kwon, Dahun Kim, Duygu Ceylan, Henry Fuchs

arXiv:2109.07448v1cs.CVcs.GR

TL;DR

Sparse-camera free-viewpoint rendering of arbitrary human performances is difficult because human motion causes heavy occlusions and dynamic articulations. Neural Human Performer uses a parametric body model with temporal and multi-view transformers, achieving state-of-the-art generalization on unseen poses and identities across ZJU-MoCap and AIST.

  • Problem

    Free-viewpoint rendering must handle arbitrary human performances from sparse multi-view cameras, but occlusions and dynamic articulations make generalizable radiance fields challenging.

  • Method

    Neural Human Performer uses a parametric human body model with temporal and multi-view transformers to aggregate tracked features across time and views.

  • Results

    The method achieves state-of-the-art performance on ZJU-MoCap and AIST, outperforming body-model-based per-scene optimization and generalizable NeRF methods on unseen poses and identities.

  • Takeaways & Limitations

    The feed-forward representation synthesizes photo-realistic novel-view videos for arbitrary performers from sparse camera views.

  • Takeaways & Limitations

    Cross-dataset generalization is limited when dataset distributions differ substantially, and performance is affected by inaccurate SMPL parameters or moving-camera input-estimation challenges.

Abstract

from arXiv · show

In this paper, we aim at synthesizing a free-viewpoint video of an arbitrary human performance using sparse multi-view cameras. Recently, several works have addressed this problem by learning person-specific neural radiance fields (NeRF) to capture the appearance of a particular human. In parallel, some work proposed to use pixel-aligned features to generalize radiance fields to arbitrary new scenes and objects. Adopting such generalization approaches to humans, however, is highly challenging due to the heavy occlusions and dynamic articulations of body parts. To tackle this, we propose Neural Human Performer, a novel approach that learns generalizable neural radiance fields based on a parametric human body model for robust performance capture. Specifically, we first introduce a temporal transformer that aggregates tracked visual features based on the skeletal body motion over time. Moreover, a multi-view transformer is proposed to perform cross-attention between the temporally-fused features and the pixel-aligned features at each time step to integrate observations on the fly from multiple views. Experiments on the ZJU-MoCap and AIST datasets show that our method significantly outperforms recent generalizable NeRF methods on unseen identities and poses. The video results and code are available at https://youngjoongunc.github.io/nhp.

1 Introduction

Neural Human Performer targets free-viewpoint human performance rendering from sparse cameras while generalizing across performers, addressing occlusions and dynamic articulations with a body-model-based radiance field and transformer aggregation.

  • Motivation: Sparse-camera free-viewpoint rendering must generalize across human performers despite heavy self-occlusions and dynamic body articulations.The target setting uses only three to four views and seeks a scalable alternative to dense camera rigs or accurate depth sensors.
  • Motivation: Pixel-aligned generalizable NeRFs enable feed-forward synthesis from sparse views, but direct application to non-rigid human motion produces significant artifacts.Person-specific methods avoid this generalization problem but cannot generalize to new performers.
  • Approach: Neural Human Performer learns generalizable radiance fields from a parametric 3D body model for robust performance capture.Its temporal and multi-view transformers aggregate spatio-temporal observations to compute query-point density and color.
  • Approach: A temporal transformer aggregates trackable visual features over skeletal motion, while a multi-view transformer cross-attends to pixel-aligned features at each time step.Together, these modules adaptively aggregate information across multiple times and views.
  • Results: Experiments on ZJU-MoCap and AIST show significant gains over generalizable NeRF methods on unseen identities and poses.The method also achieves better rendering quality than person-specific body-model methods on novel poses.

2 Related works

Human performance capture has progressed from specialized hardware and template-based tracking toward neural scene representations, but dynamic human rendering remains challenging and existing methods often generalize poorly to unseen scenarios.

  • Human performance capture: Traditional capture uses dense camera rigs or accurate depth sensors, while template-based methods enable sparse-view synthesis but depend on pre-scanned models and lack photorealism.These approaches address sparse-view capture through tracked human templates, but their synthesis quality and model availability remain limited.
  • Neural 3D representations: Neural 3D representations learn scene shape and appearance from 2D images using differentiable rendering networks.Prior representations include voxel-grid features, point clouds, textured meshes, and multi-plane images.
  • Neural 3D representations: NeRF learns implicit density and color fields with volume rendering and achieves photorealistic view synthesis, while later work extends NeRF to dynamic scenes.Dynamic human rendering remains especially under-constrained because deformation fields and NeRF must be learned jointly.
  • Human-centered NeRFs: Neural Body regularizes NeRF training with a deformable human body model, such as SMPL, but human-specific and general NeRF methods require per-video optimization or generalize poorly.Generalizable NeRFs instead use image conditioning with pixel-aligned features to reduce expensive per-scene optimization.

3 Method

Neural Human Performer learns generalizable radiance fields for arbitrary human performances from sparse multi-view and temporal observations, using a parametric body model and temporal and multi-view transformers. The fused query-point representation is converted into color and density and rendered along target-view rays.

  • Problem setup: The method targets novel-view video synthesis for arbitrary human performances from sparse input cameras and given per-frame 3D body-model fits.It augments each queried timestep with sampled memory frames to learn from multi-time and multi-view observations.
  • Feature construction: SMPL vertices are projected into input images to sample pixel-aligned features, forming skeletal feature banks across views and memory frames.The skeletal features are sampled at projected SMPL vertices and organized over the available temporal observations.
  • Multi-view aggregation: For each query 3D point, the multi-view Transformer cross-attends from time-augmented skeletal features to pixel-aligned features and adaptively fuses observations across views.Pixel-aligned features represent the exact query location at the current time, while skeletal features are time-augmented and include geometric deviations from the body model.
  • Radiance-field rendering: The fused query representation is passed to radiance-field MLPs to predict time-dependent density and color, with viewing direction included through positional encoding.Target-view pixels are rendered by marching rays and accumulating radiance and opacity along sampled points.
  • Temporal aggregation: The temporal Transformer dynamically weights each skeletal vertex’s previous and next memory-frame representations instead of using simple average pooling.Its query, key, and value embeddings produce softmax attention over the vertex’s memory bank, allowing helpful information to be incorporated and irrelevant information ignored.
  • Training: Training uses a photometric reconstruction loss between the predicted and ground-truth target images.The loss trains both the radiance field and the feature extraction network.

4 Experiments

Experiments evaluate Neural Human Performer against body model-based per-scene methods and generalizable NeRFs across pose, identity, dataset, reconstruction, and ablation settings. The method consistently improves generalization and benefits from its transformer modules and complementary feature types.

  • Experimental setup: Experiments compare Neural Human Performer with body model-based per-scene methods and generalizable NeRF methods on ZJU-MoCap and AIST.The evaluations cover novel-view synthesis, 3D reconstruction, and multiple generalization scenarios.
  • Pose generalization: 26.94 PSNR and 0.929 SSIM are achieved by Neural Human Performer on source models’ unseen poses, exceeding Neural Body, NHR, and Neural Textures.The comparison uses source-model training and tests unseen poses from those same models.
  • Identity-and-pose generalization: Neural Human Performer significantly outperforms generalizable NeRF baselines on unseen identities and poses, with +1.6 PSNR and +0.037 SSIM across datasets and metrics.The compared generalizable methods include Pixel-NeRF and PVA.
  • 3D reconstruction: The method produces more reliable 3D reconstructions than competing generalizable methods and even the per-person Neural Body method in the reported examples.The reconstruction evaluation uses three input views on ZJU-MoCap and AIST.
  • Cross-dataset generalization: Eight-minute target-dataset fine-tuning outperforms baselines fully trained on the target dataset, while 16-minute fine-tuning performs on par with the fully trained model.The cross-dataset setting is challenging because the datasets differ in color distribution, lighting, background, and camera distance.
  • Ablation studies: The full model reaches 24.75 PSNR and 0.9058 SSIM, with temporal and multi-view transformers improving over average-pooling variants.The multi-view transformer gives the largest individual gain, while skeletal and pixel-aligned features are complementary.

5 Limitations

The method’s generalization is limited when dataset distributions differ substantially, SMPL fits become inaccurate, or capture inputs are difficult to estimate with moving cameras.

  • Cross-dataset generalization is limited when the source and target dataset distributions differ significantly.
  • Rendering quality is affected by degraded SMPL parameter accuracy, with lower scores reported on the more motion-complex AIST dataset than on ZJU-MoCap.
  • Moving-camera capture may make SMPL fits, camera parameters, and foreground masks difficult to estimate because of blur, backgrounds, lighting, and synchronization.

6 Societal impact

The paper identifies telepresence as a positive application while recognizing that sparse-camera reconstruction could also facilitate person identification by surveillance organizations.

  • Human performance synthesis could support telepresence between physically separated people using inexpensive webcams and AR/VR headsets.
  • The technology could also make it easier for organizations to identify people by reconstructing them from a small number of surveillance cameras.
  • The authors express hope that the research will be used in positive directions.

7 Conclusion

Neural Human Performer combines a parametric body model with temporal and multi-view Transformers to synthesize photo-realistic free-viewpoint videos for unseen human poses and identities from sparse views.

  • Neural Human Performer is a generalizable radiance-field network for arbitrary performers using sparse camera views and a parametric body model.
  • A temporal Transformer and multi-view Transformer integrate multi-time and multi-view observations in a feed-forward manner.
  • The method produces photo-realistic novel-view synthesis for unseen poses and identities at test time.
  • Across generalization settings on ZJU-MoCap and AIST, it outperforms body-model-based per-scene optimization and generalizable NeRF methods.

B.1 Implementation details.

The implementation extracts multi-scale image features, constructs time-augmented skeletal features, samples query-point features, and fuses them with a multi-view Transformer before NeRF rendering.

  • Image feature extractor: An ImageNet-pretrained ResNet18 extracts a four-scale feature pyramid, whose maps are upsampled and concatenated into a 512 × H/2 × W/2 representation.
  • Temporal Transformer: The temporal Transformer constructs time-augmented skeletal features by processing all L vertices batch-wise with per-vertex attention, using d = 64.
  • Skeletal feature sampling: Skeletal features are sampled at a query point using SparseConvNet after voxelizing the SMPL-based body volume at 5mm × 5mm × 5mm resolution.
  • Multi-view Transformer: The multi-view Transformer cross-attends time-augmented skeletal features with time-specific pixel-aligned features to adaptively fuse observations across views.
  • NeRF network: The fused query representation is passed to a NeRF network composed of fully connected layers.
  • Query point sampling: Query points are sampled inside an SMPL-derived bounding box enlarged by 2.5%, with 1024 training rays and 64 points per ray for training and inference.

C Datasets

The study uses ZJU-MoCap for research-only evaluation, with controlled unseen-identity splits and 512 × 512 videos.

  • ZJU-MoCap contains no personally identifiable information or offensive content and is intended for research purposes.
  • ZJU-MoCap contains 10 human subjects, with 7 reserved for training and 3 for testing on unseen identities.
  • Experiments use 512 × 512 videos for training and testing, following the original Neural Body setup.
  • The dataset provides SMPL parameters from EasyMocap and foreground masks extracted using PGN.

C.2 AIST

AIST provides high-resolution motion videos with SMPL parameters but requires preprocessing for foreground segmentation and balanced image framing.

  • AIST provides 60 fps videos at 1080 × 1920 resolution with corresponding SMPL parameters obtained using the AIST++ API.
  • Because AIST lacks foreground masks, the study obtains them using PGN.
  • The videos are center-cropped to 800 × 800 and resized to 512 × 512 for training and evaluation because most image area is background.
  • AIST contains 30 human subjects.
Loading 2109.07448v1…