PaperlayerView exact revision on arXiv

Source-linked AI summary

HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs

Fuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, Lan Xu

arXiv:2112.02789v3cs.CVcs.GR

TL;DR

HumanNeRF addresses the dense-view and costly per-scene training demands that constrain dynamic human neural rendering. It combines aggregated image features, pose-embedded deformation, and appearance blending; experiments report photo-realistic free-view synthesis on varied datasets, including challenging motions and sparse camera inputs.

  • Problem

    Existing neural human representations require dense multi-view inputs or costly per-scene training, limiting their practicality for dynamic scenes.

    Source: §1 · p. 1

  • Method

    HumanNeRF uses aggregated image-conditioned features and pose-embedded hybrid deformation in a generalizable dynamic radiance field, with implicit appearance blending to add input-image texture detail.

    Source: §1 · p. 2

  • Results

    Experiments on various datasets demonstrate photo-realistic free-view synthesis under challenging motions and very sparse camera inputs.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The method supports efficient generation of high-quality novel views of dynamic humans from a sparse set of cameras.

    Source: §5 · p. 8

    The regressed parametric human model constrains HumanNeRF to single-person scenes and does not handle multi-person or human-object interactions.

    Source: A. Limitations · p. 12

Abstract

from arXiv · show

Recent neural human representations can produce high-quality multi-view rendering but require using dense multi-view inputs and costly training. They are hence largely limited to static models as training each frame is infeasible. We present HumanNeRF - a generalizable neural representation - for high-fidelity free-view synthesis of dynamic humans. Analogous to how IBRNet assists NeRF by avoiding per-scene training, HumanNeRF employs an aggregated pixel-alignment feature across multi-view inputs along with a pose embedded non-rigid deformation field for tackling dynamic motions. The raw HumanNeRF can already produce reasonable rendering on sparse video inputs of unseen subjects and camera settings. To further improve the rendering quality, we augment our solution with an appearance blending module for combining the benefits of both neural volumetric rendering and neural texture blending. Extensive experiments on various multi-view dynamic human datasets demonstrate the generalizability and effectiveness of our approach in synthesizing photo-realistic free-view humans under challenging motions and with very sparse camera view inputs.

1. Introduction

HumanNeRF targets high-quality free-view synthesis of dynamic humans from sparse RGB streams, addressing the dense capture and costly per-scene training demands of prior neural rendering. It combines a generalizable dynamic radiance field with hybrid deformation and appearance blending.

1. Introduction

  • Prior neural rendering methods still require dense capture views or time-consuming per-scene training, limiting practicality for dynamic scenes.Earlier approaches also depended on dome-based multi-view capture or depth sensors.Source: §1 · p. 1
  • HumanNeRF extends generalizable NeRF to dynamic human settings, where few prior methods had explored such representations.A prior generalizable human-rendering approach [45] suffers artifacts near occluded regions because it lacks global inherent geometry and texture modeling.Source: §1 · p. 2
  • The method aggregates image-conditioned features and uses pose-embedded hybrid deformation to generalize motion and appearance across unseen identities and poses.The deformation combines explicit model-based warping with implicit subtle displacement modeling in a canonical space.Source: §1 · p. 2
  • An implicit blending scheme combines volumetric rendering with input-image texture detail for photo-realistic appearance rendering.It learns implicit, occlusion-aware blending weights to draw detail from adjacent input images.Source: §1 · p. 2
  • HumanNeRF generates high-quality free-view synthesis for arbitrary performers from sparse RGB streams and reports superiority to existing state of the art.The approach is described as efficiently generating a generalizable radiance field for unseen performers.Source: §1 · p. 2

2. Related work

Prior human capture approaches trade capture complexity against rendering fidelity, while neural rendering and image-based rendering each have limitations. HumanNeRF combines pixel-aligned features and implicit blending in a lightweight multi-RGB representation.

2. Related work

  • Lightweight multi-view human capture offers a compromise between demanding hardware and reconstruction fidelity, but comparable solutions still rely on 3–8 RGB-D streams.Other approaches require pre-scanned templates, dense cameras, controlled imaging, or cannot capture motion in occluded regions.Source: §2 · p. 2
  • Generalizable human methods use pixel-aligned source-image features [5] [21] [36] [37] [45] [54], but cited approaches report blur or geometric discontinuity [37] [45].Temporally aggregated features [21] still suffer blur when generalizing unseen identities with complex motions due to self-occlusion.Source: §2 · p. 2
  • Image-based rendering blends reference pixels without detailed 3D geometry, but its renderable viewpoint range is limited and proxy-geometry methods are sensitive to reconstruction quality [37].HumanNeRF embraces image blending within an implicit representation pipeline under lightweight multi-RGB capture.Source: §2 · p. 3

3. The HumanNeRF Approach

HumanNeRF targets high-fidelity free-viewpoint video synthesis from synchronized multi-view performer videos, combining a generalizable dynamic radiance field with appearance blending. It uses pose-conditioned deformation and aggregated image features to model motion and refine texture detail.

3. The HumanNeRF Approach

  • HumanNeRF takes synchronized multi-view performer videos as input and aims to synthesize high-fidelity free-viewpoint video and generalize motion to arbitrary people.Its core is an efficient generalized Neural Radiance Field adapted for dynamic humans.Source: §3 · p. 3

3.1. Generalizable Dynamic Neural Radiance Field

  • A pose-dependent deformation field warps the body to a canonical pose, while aggregated pixel-alignment features from projected multi-view images support generalizable radiance prediction.The model combines SMPL inverse skinning with learned subtle displacements and uses the transformed location, view direction, and aggregated feature to predict density and color.Source: §3.1 · pp. 3–4

3.2. Neural Appearance Blending

  • Neural appearance blending refines sparse-view texture by combining colors from two neighboring views with the NeRF rendering, using learned weights informed by image features and visibility.The method back-projects target-view depth points into neighboring views and blends their colors with the rendered color.Source: §3.2 · p. 5, Figure 4 · p. 5

3.3. Implementation Details

  • Training uses 1,820 Twindom scans and six-view videos of 26 subjects; the generalizable NeRF takes about two days to train, with subject fine-tuning taking 30–90 minutes.The generalizable NeRF and appearance blending networks are trained separately.Source: §3.3 · pp. 5–6

4. Experimental Results

HumanNeRF is evaluated on challenging human appearances and poses, with comparisons against per-scene and generalizable methods. The results report strong rendering accuracy and show how fine-tuning, appearance refinement, camera count, and pose generalization affect its outputs.

4.1. Comparison

  • Against per-scene methods, HumanNeRF produces better textures and complete, accurate geometry with short fine-tuning; against generalizable methods, it better addresses self-occlusion.The qualitative comparisons cover Taichi and Batman for per-scene methods, and IBRNet and NeuralHumanFVV for generalizable methods.Source: §4.1 · p. 6, Figure 6 · p. 7, Figure 7 · p. 7
  • HumanNeRF achieves the best PSNR, SSIM, LPIPS, and MAE performance among the compared methods.Table 1 reports the quantitative comparison on real testing data.Source: §4.1 · p. 6, Table 1 · p. 7
  • Even without per-scene fine-tuning, HumanNeRF achieves comparable results.This finding accompanies the comparison with per-scene optimization methods.Source: §4.1 · pp. 6–7

4.2. Ablation Study

  • Without fine-tuning, results are low-detail and blurry; omitting appearance refinement causes blurring artifacts, especially near boundaries.The complete approach produces photo-realistic results with better decomposition for various entities.Source: §4.2 · p. 7, Figure 8 · p. 8
  • Fewer than two input views lead to severe geometric and rendering artifacts.The evaluation varies camera count; PSNR, SSIM, and LPIPS are calculated against results using all cameras as reference.Source: §4.2 · pp. 7–8, Figure 9 · p. 8
  • On unseen poses, HumanNeRF produces visually good results and shows good metrics after fine-tuning on 400 of 500 Swing frames.The remaining 100 frames are used for testing.Source: §4.2 · p. 8

5. Conclusion

HumanNeRF synthesizes dynamic-human views from sparse cameras using fused image features, pose-embedded deformation, and appearance blending. Experiments report photo-realistic results on challenging poses and motions, and the authors suggest possible relevance to VR/AR applications.

5. Conclusion

  • The method combines fused image features and pose-embedded human deformation with appearance blending that borrows fine detail from two adjacent views.The conclusion describes blending as refining volumetric-rendering results.Source: §5 · p. 8
  • The authors present a lightweight method for efficiently generating high-quality novel-view synthesis of dynamic humans from sparse cameras, with experiments on various datasets demonstrating its effectiveness.Figure 10 qualitatively evaluates pose generalization; Table 2 quantitatively compares results on seen and unseen poses.Source: §5 · p. 8, Figure 10 · p. 8, Table 2 · p. 8
  • The authors suggest the efficient generation may inform VR/AR applications including gaming, entertainment, education, and immersive telepresence.This is stated as a potential application, not as a demonstrated deployment.Source: §5 · p. 8

A. Limitations.

HumanNeRF has limitations in supported subjects, motion and appearance distributions, and lighting variation. These constraints can produce artifacts in multi-person interactions and across views with significant brightness differences.

A. Limitations.

  • The regressed parametric human model limits the method to single-person scenes, excluding multi-person and human-object interaction settings.It is used to handle large pose deformation and complex motions.Source: A. Limitations · p. 12
  • The method’s generalization is limited by human datasets that cover only a small portion of human dynamics and appearances.Source: A. Limitations · p. 12
  • Because lighting is not explicitly modeled, large brightness or color differences between views may cause severe artifacts, including jumping artifacts when the nearest views switch during appearance blending.The authors say this artifact is alleviated when illumination is almost consistent across views.Source: A. Limitations · p. 12

B. Components ablation study.

The ablation analysis compares pipeline variants and input-view counts. The full module achieves the best results, while average error increases as the number of cameras decreases.

B. Components ablation study.

C. Discussion about our generalizability .

HumanNeRF’s generalizable radiance-field module produces meaningful but blurry results on unseen identities. With efficient fine-tuning in hours, the method reports state-of-the-art performance even for unseen poses.

C. Discussion about our generalizability.

D. Training time comparison.

HumanNeRF is reported to be more efficient than other per-scene training methods in training or fine-tuning time. The comparison is on the 300-frame “batman” video.

D. Training time comparison.

E. Network Architectures.

The architecture comprises an image feature extractor, feature blending and deformation networks, a generalizable dynamic radiance field, and an appearance blending network.

E. Network Architectures.

  • The architecture includes an image feature extractor U, feature blending network MLPB, deformation network MLPd, dynamic radiance field Φ, and appearance blending network MLPA.The supplied network descriptions identify these components without specifying further architecture details for Φ.Source: E. Network Architectures · p. 12
  • The feature extractor table specifies kernel size, stride, dilation, and input and output channel counts for each layer.CRB2D denotes ConvBnReLU2D.Source: Table 6 · p. 13
  • The feature blending network MLPB is described using positional encoding and LinearRelu layer structures.PE/LR refers to positional encoding and LinearRelu, respectively.Source: Table 7 · p. 13
  • The deformation network MLPd uses distances and directions from sample point p to the 24 SMPL joints, alongside the blended feature F.Rd and Rv denote those distances and directions.Source: Table 8 · p. 13
  • A separate table provides network details for the generalizable dynamic neural radiance field Φ.Source: Table 9 · p. 13
  • In Table 10's network details for appearance blending network MLPA, fr, 0r, fl, and 0l are described as two adjacent image features and occlusion maps.Source: Table 10 · p. 13
Loading 2112.02789v3…