Source-linked AI summary
H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion
Hongyi Xu, Thiemo Alldieck, Cristian Sminchisescu
TL;DR
Sparse-view human performance capture requires both detailed temporal geometry and novel-view rendering. H-NeRF couples a radiance field with an imGHUM-based signed distance model and reports robust reconstruction and rendering that generalize to novel views, poses, and shapes, while remaining sensitive to input estimates and segmentation quality.
Problem
Sparse-view human capture must learn detailed temporal geometry and render novel camera views and human poses, while unconstrained radiance fields are poorly regularized under sparse views.
Method
H-NeRF co-learns a neural radiance field and an imGHUM-based signed distance function, using geometric constraints and pose- and shape-conditioned representations.
Results
H-NeRF produces robust sparse-view reconstruction and rendering, generalizing to novel camera views, human poses, and body shapes across the evaluated experiments.
Takeaways & Limitations
The approach extends dynamic human rendering and temporal reconstruction beyond the sparse views, poses, and shapes observed during training.
Takeaways & Limitations
The method assumes full-body views of a single person in everyday clothing and is sensitive to estimated poses, shapes, and image segmentation quality.
Abstract
from arXiv · showhide
We present neural radiance fields for rendering and temporal (4D) reconstruction of humans in motion (H-NeRF), as captured by a sparse set of cameras or even from a monocular video. Our approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions. Instead of learning a radiance field with a uniform occupancy prior, we constrain it by a structured implicit human body model, represented using signed distance functions. This allows us to robustly fuse information from sparse views and generalize well beyond the poses or views observed in training. Moreover, we apply geometric constraints to co-learn the structure of the observed subject -- including both body and clothing -- and to regularize the radiance field to geometrically plausible solutions. Extensive experiments on multiple datasets demonstrate the robustness and the accuracy of our approach, its generalization capabilities significantly outside a small training set of poses and views, and statistical extrapolation beyond the observed shape.
1 Introduction
H-NeRF targets free-viewpoint video of humans in motion from sparse views, combining neural rendering with structured implicit human geometry. The system supports rendering and reconstruction across novel views, poses, and body shapes.
- Free-viewpoint video of humans in motion from sparse views is challenging but has applications in immersive visualization, virtual clothing, fitness, AR, and VR.
- Existing approaches emphasize either accurate geometric reconstruction with classical rendering or image production quality without guaranteeing accurate scene geometry.
- H-NeRF combines neural radiance fields with an implicit statistical human body model and signed distance functions for sparse-view reconstruction.
- Conditioning on an implicit human body model enables rendering from viewpoints, body poses, and body shapes substantially different from those observed during training.
- The framework co-learns radiance and geometry with losses designed to support photo-realistic free-viewpoint temporal rendering and plausible reconstructions.
2 Related Work
Prior human performance-capture methods reconstruct motion using template-based or image-derived cues, while neural scene representations support novel-view synthesis. NeRF provides photorealistic rendering but has slow rendering time, motivating faster variants.
- Human performance capture: Human performance-capture methods deform rigged template meshes using silhouettes, skeleton key-points, or image correspondences from multiple views.Later systems also use RGB-D video streams or monocular capture.
- Neural scene representation: Voxel-grid scene representations embed scenes in grids and render views by ray probing with linear interpolation of voxel values.
- Neural Radiance Fields: Neural Radiance Fields represent scenes with fully connected networks that map spatial query points and viewing directions to volume density and emitted radiance.Ray tracing this representation enables photo-realistic images from novel views.
- Neural Radiance Fields: NeRF’s excellent image quality is offset by slow rendering time, prompting approaches based on sparse grids or viewing-direction-independent spherical harmonics.
3 Background
The paper aims to learn detailed temporal human geometry and render dynamic subjects from novel views and poses using sparse calibrated cameras, including the monocular limit. Its background combines implicit 3D human representations with volumetric radiance fields.
- Problem formulation: The task is to learn a moving human’s detailed temporal geometry and render the sequence from novel camera views and different poses.The observations come from sparse calibrated camera views, potentially down to a monocular camera.
- Problem formulation: The approach unifies implicit 3D human representations with volumetric radiance fields.
- Neural Radiance Fields: A NeRF models a 3D scene as a continuous function of color volume densities, mapping spatial points and viewing directions to density and radiance.Its inputs can be encoded with sinusoidal positional encoding.
- Neural Radiance Fields: The differentiable NeRF function can be optimized with image reconstruction loss, and approximate scene geometry can be extracted from its opacity field using Marching Cubes.
- Implicit Generative Human Models: Implicit human surfaces are commonly represented with binary occupancy classifiers or signed distance functions.
- Implicit Generative Human Models: The imGHUM model maps an unposed spatial point, body shape, and pose parameters to signed distance values and additional surface-related outputs.
4 Method
H-NeRF co-learns a volumetric radiance field and signed-distance geometry, using imGHUM as a structured prior for reconstructing and rendering humans from sparse observations. Its losses constrain scene structure, opacity, geometry, and temporal correspondence to support novel views, poses, and shapes.
- Core framework: H-NeRF co-learns a radiance field and dressed-subject signed distance function, while imGHUM supplies an articulated human pose-and-shape prior.The framework jointly reconstructs geometry and appearance rather than learning an unconstrained occupancy field.
- Static scene structuring: Coarse scene structuring places the person inside an imGHUM-derived 3D bounding box and applies a segmentation loss to constrain radiance points.The box includes a margin for geometry not modeled by imGHUM, and mask supervision excludes remaining scene geometry.
- Radiance-field coupling: SDF-based pseudo alpha values couple NeRF opacity with the subject’s zero-isosurface, while blending and geometric losses regularize surface and background rendering.At test time, learned NeRF alpha values remain available for transparency and complex structures such as hair.
- Geometry refinement: The personalized SDF combines imGHUM geometry with a residual network that models surface details such as clothing and hair.The final signed distance is formed by adding the residual SDF to the imGHUM-conditioned distance.
- Dynamic reconstruction: For motion, H-NeRF learns a continuous function across frames and warps observations into a canonical reference frame using imGHUM-based semantic correspondences.Spatial points are mapped to canonical descriptors before conditioning the radiance and residual-SDF functions.
- Dynamic reconstruction: Conditioning on semantic body parameters enables interpolation and generalization to novel human poses and body shapes.The articulated imGHUM prior provides the geometric volume deformation used for this generalization.
5 Experiments
Experiments evaluate H-NeRF across four datasets, sparse-view settings, dynamic sequences, and pose or shape generalization. The results show stronger rendering and geometric reconstruction, particularly when training data or camera views are sparse, while performance remains subject to input-quality assumptions.
- Evaluation setup: Evaluation covers RenderPeople, GHS3D, PeopleSnapshot, and Human3.6M using image and geometric metrics where ground truth is available.The study includes eight RenderPeople, fourteen GHS3D, seven PeopleSnapshot, and five Human3.6M sequences.
- Static reconstruction: H-NeRF produces good static reconstruction quality with only 2–8 training cameras, unlike NeRF and IDR under sparse views.Both image quality and geometric accuracy improve with more views, but H-NeRF remains robust in the sparse regime.
- Dynamic reconstruction: H-NeRF outperforms NeuralBody across dynamic datasets and metrics, especially for geometric reconstruction.The paper attributes this advantage partly to explicit conditioning on pose and root transformation for pose-dependent geometry and appearance.
- Frame-number ablation: H-NeRF degrades less than NeuralBody when trained with fewer temporal frames, especially for geometric reconstruction.The method can be robustly trained with as little as 10 temporal frames per camera in a four-camera setup.
- Qualitative comparison: Qualitative comparisons show sharper synthesized poses and more complete, smooth, and detailed geometry than NeuralBody.Reported details include reduced noise and preservation of clothing folds in reconstructed geometry.
- Generalization: H-NeRF synthesizes images and reconstructs geometry from novel viewpoints, poses, and modified body shapes across the evaluated datasets.Figure 5 distinguishes test and training configurations for camera, pose, and shape variations.
- Limitations: The current setup assumes full-body views of one person in everyday clothing and is sensitive to estimated pose, shape, and image segmentation quality.Multiple people, performer occlusion, or increasingly degraded inputs can eventually break the method.
6 Conclusions
H-NeRF extends neural radiance fields to photo-realistic rendering and temporal reconstruction of humans in motion from sparse views. It combines signed-distance implicit human pose and shape modeling with radiance fields to support novel views, poses, and body-shape extrapolation.
- Conclusion: H-NeRF targets photo-realistic rendering and temporal reconstruction of humans in motion from sparse views.The approach is designed to generalize to novel camera views, poses, and body shapes.
- Conclusion: The model constrains a generic NeRF with a compatible implicit statistical human pose-and-shape model represented by signed distance functions.Its formulation also represents hair and clothing through an implicit residual network and uses multiple losses trained end-to-end.
- Conclusion: H-NeRF achieves good results for both 3D reconstruction and photorealistic rendering.The conclusion presents this as the outcome of combining radiance fields, implicit geometry, and end-to-end losses.
A Additional Results
H-NeRF demonstrates accurate dynamic reconstruction and rendering across sparse-view, monocular, novel-pose, novel-view, and novel-shape settings. Its generalization remains strong, but depends on sufficient pose-dependent training examples and accurate pose estimation.
- Dynamic Human Reconstruction and Rendering: H-NeRF achieves very good accuracy for novel-view synthesis and 3D geometric reconstruction across four dynamic datasets.The evaluation includes RenderPeople, GHS3D, PeopleSnapshot, and Human3.6M sequences.
- Dynamic Human Reconstruction and Rendering: H-NeRF generates novel geometry for poses absent from training by altering the imGHUM pose control.Novel-pose image errors can arise from both NeRF rendering and implicit geometric surface skinning.
- Generalization: H-NeRF produces high-quality rendering and geometric reconstruction for substantially altered body shapes through changes to the imGHUM shape latent code.The method propagates shape changes through both the radiance field and signed distance function.
- View Generalization: H-NeRF consistently outperforms NeRF and IDR as test viewpoints diverge from training cameras, although all methods degrade at large view differences.The increased error is largely attributed to rendering body parts absent from training images.
- Pose Generalization: Pose-generalization quality shows no strong correlation with pose distance, but depends on imGHUM prediction accuracy and sufficient examples of pose-dependent geometry and appearance.The study evaluates image and geometric metrics on novel RenderPeople poses.
- Monocular Videos: With one training camera, H-NeRF remains robust for novel views, geometry, and pose extrapolation, though quality is lower than with four cameras.GHS3D-S47 declines more because the subject’s back is not fully visible during training; monocular pose estimation also required filtering some PeopleSnapshot frames.
B Ablation Studies
Ablations show that the complete loss configuration provides the best balance between rendering quality and geometric accuracy. Different losses affect image and geometry metrics differently, with Eikonal regularization especially important under insufficient views or lower resolution.
- Ablation Studies: Using all proposed losses achieves the best performance for most metrics while balancing image rendering quality and geometric reconstruction accuracy.The ablation covers four datasets.
- Appearance Conditioning: Removing root-transformation conditioning from NeRF appearance causes significant image-quality drops.This condition is represented by the root transformation T.
- Loss Contributions: Lblend and Lmask strongly affect image metrics, whereas Lgeom, Lreg, and Leik primarily affect geometric metrics.Removing Lblend slightly improves a geometric metric but substantially worsens SSIM for real-world videos.
- Loss Contributions: Eikonal regularization smooths surface reconstruction and is critical for videos with insufficient views or lower image resolution.The cited cases are PeopleSnapshot and Human3.6M.
- Loss Contributions: Removing Lseg degrades both rendering and geometric performance.
C Training, Memory Consumption and Timings
H-NeRF training and inference require substantial computation, with imGHUM warping adding overhead and limiting query capacity under GPU memory constraints. The reported hyperparameters use fixed loss weights with dataset-specific η values.
- Training: Training each dynamic sequence takes about 6–8 hours using 10k iterations, 4k-ray batches, and 8 Nvidia V100 GPUs.
- Inference: Inference at 512×512 resolution takes about 9.1 seconds on one Nvidia V100 GPU, compared with about 6.5 seconds for original NeRF.The additional computation mainly comes from imGHUM warping.
- Memory Consumption: imGHUM warping limits the maximum number of query points to 643 because of memory constraints.
- Parameters: The loss weights are fixed across training, while η is set to 1.0 for RenderPeople and GHS3D and 0.25 for the other two datasets.The reported weights include Lrec, Lmask, Lblend, Lgeom, Lseg, Lreg, Leik, Lfit, and Linc.
D imGHUM Fitting
The additional evaluations compare H-NeRF with alternative neural representations across camera, pose, dataset, and training-data settings. Results emphasize stronger view synthesis and reconstruction under sparse observations, while tabulated comparisons define the evaluated metrics and scopes.
- Novel View Synthesis: Figure 8 evaluates novel-view image quality against angular deviation from training cameras using static scenes trained with four cameras.H-NeRF is compared with NeRF and IDR using ground truth and rendered views.
- Pose Generalization: Figure 9 evaluates image and geometric metrics against pose difference from the closest training pose for two RenderPeople sequences.The reported trend is that reconstruction quality does not significantly degrade with pose difference when imGHUM predicts test poses well.
- Quantitative Comparisons: Table 3 compares H-NeRF with Nerfie on dynamic sequences, reporting geometric metrics only where ground truth exists and excluding Nerfie scenes without Marching Cubes outputs.Nerfie does not support novel-pose rendering, so its metrics use training poses.
- Monocular Videos: Figure 10 qualitatively evaluates four dynamic monocular sequences, including novel views for training poses and novel poses under novel viewpoints.The sequences include RenderPeople-Tina, RenderPeople-Nagy, and GHS3D S36 and S47.
- Static Scenes: Figure 11 shows that novel-view synthesis and geometric reconstruction improve for all methods as the number of training cameras grows.NeRF and H-NeRF converge, while IDR retains differences associated with its different image-formation process.
- Sparse Video Comparison: Figure 12 qualitatively compares H-NeRF and NeuralBody on sparse video frames, reporting consistent H-NeRF superiority for RenderPeople and GHS3D sequences.The examples use 10 training frames for RenderPeople and 20 for GHS3D.