Source-linked AI summary
NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads
Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, Matthias Nießner
TL;DR
High-fidelity rendering of dynamic human heads requires temporally consistent reconstruction despite complex motion and fine facial detail. The paper introduces NeRSemble and a large synchronized multi-view dataset, and reports detailed reconstructions that outperform existing dynamic NeRF baselines, especially under complex motion.
Problem
Reconstructing human heads from dynamic multi-view recordings requires temporally consistent representations that preserve fine detail for photorealistic rendering from arbitrary viewpoints and times.
Method
NeRSemble combines a deformation field with an ensemble of multi-resolution hash-grid encodings, supported by a 16-camera, 7.1 MP, 73 fps capture setup and a new benchmark dataset.
Results
NeRSemble achieves temporally coherent, highly detailed volumetric reconstructions and significantly outperforms existing baselines, particularly on sequences with complex motions.
Takeaways & Limitations
The dataset and benchmark provide a foundation for further research on dynamic novel-view synthesis and digital human reconstruction.
Takeaways & Limitations
NeRSemble cannot perfectly capture fast hair motion and optimizes each sequence separately, so it does not learn priors that generalize across sequences.
Abstract
from arXiv · showhide
We focus on reconstructing high-fidelity radiance fields of human heads, capturing their animations over time, and synthesizing re-renderings from novel viewpoints at arbitrary time steps. To this end, we propose a new multi-view capture setup composed of 16 calibrated machine vision cameras that record time-synchronized images at 7.1 MP resolution and 73 frames per second. With our setup, we collect a new dataset of over 4700 high-resolution, high-framerate sequences of more than 220 human heads, from which we introduce a new human head reconstruction benchmark. The recorded sequences cover a wide range of facial dynamics, including head motions, natural expressions, emotions, and spoken language. In order to reconstruct high-fidelity human heads, we propose Dynamic Neural Radiance Fields using Hash Ensembles (NeRSemble). We represent scene dynamics by combining a deformation field and an ensemble of 3D multi-resolution hash encodings. The deformation field allows for precise modeling of simple scene movements, while the ensemble of hash encodings helps to represent complex dynamics. As a result, we obtain radiance field representations of human heads that capture motion over time and facilitate re-rendering of arbitrary novel viewpoints. In a series of experiments, we explore the design choices of our method and demonstrate that our approach outperforms state-of-the-art dynamic radiance field approaches by a significant margin.
1 INTRODUCTION
The paper targets temporally consistent, high-fidelity 3D reconstruction of dynamic human heads for photorealistic rendering from arbitrary viewpoints and times. It introduces NeRSemble alongside a high-resolution multi-view dataset and benchmark.
- Motivation: Dynamic human-head reconstruction must jointly capture static appearance, motion over time, and compact scene representations for novel-view rendering.Fine facial detail, hair, reflectance variation, skin elasticity, wrinkles, and non-rigid deformation make the task especially challenging.
- Method: NeRSemble combines a deformation field with an ensemble of multi-resolution hash encodings to represent coarse dynamics, fine details, and complex movements.The deformation field aligns hash-grid coordinates before feature queries, while blended hash grids model time-dependent detail.
- Dataset and Benchmark: The capture setup records 7.1 MP video at 73 fps using 16 machine vision cameras, producing 4734 sequences of 222 human heads and 31.7 million frames.These recordings support training and evaluation on diverse dynamic human-head content.
- Dataset and Benchmark: The work introduces a public benchmark and dataset for dynamic novel-view synthesis of human heads.The benchmark is intended to support further research in this area.
2 RELATED WORK
Prior work spans mesh-based morphable models, static and dynamic neural radiance fields, and non-radiance-field video view synthesis. NeRSemble differentiates itself through a template-free hash-based decomposition combined with deformation modeling.
- 3D Morphable Models: 3D morphable models use unified mesh topology to represent identity and expression but are limited in modeling hair and fine identity-specific details.Neural fields alleviate the constraint of relying on topologically uniform meshes.
- Dynamic NeRF: Dynamic NeRF methods model time-varying content using canonical-space deformations, latent conditioning, Fourier-based grid compression, or image-to-volume generation.These approaches differ in whether dynamics are represented explicitly through deformation or implicitly through latent or feature representations.
- NeRSemble: NeRSemble proposes a hash-based decomposition together with a deformation field, contrasting with concurrent methods that combine fast NeRF backbones or tensor decompositions with temporal modeling.Its face-reconstruction approach is template-free rather than dependent on tracked 3DMM parameters.
- Video View Synthesis: Other dynamic view-synthesis methods use geometry-assisted image rendering, multi-sphere images with layered meshes, or related non-radiance-field representations.These methods target novel-view video synthesis without relying on a radiance-field backbone.
- NeRSemble: Compared with feature blending methods using tracked 3DMMs, NeRSemble jointly optimizes blend weights and model parameters and places deformation before hash-grid blending.The authors report significant improvements from including the deformation field before blending.
3 MULTI-VIEW VIDEO DATASET OF HUMAN FACES
The dataset combines high-resolution, high-framerate multi-view recordings of diverse human heads with standardized facial-motion sequences, processing support, and a public benchmark plan.
- Acquisition: 4734 recordings of 222 subjects were captured with 16 machine vision cameras at 7.1 megapixels and 73 frames per second.The rig covers 93° horizontally and 32° vertically, targeting fine facial detail and subtle motion.
- Acquisition: Each recording session contains 25 short sequences, totaling approximately three minutes of multi-view video per person.
- Acquisition: The capture script spans expression, hair, emotion, speech, and free-deformation sequences to maximize motion variety.It includes 9 expression sequences, 1 hair sequence, 4 emotion sequences, 10 spoken sentences, and 1 free sequence.
- Acquisition: Short exposure, small apertures, strong diffuse lighting, and synchronized cameras support sharp, low-blur recordings of facial motion.The setup uses a 3 ms shutter speed, 8 LED panels, and diffusor plates to reduce skin specularities.
- Benchmark: The dataset is intended for photo-realistic human-head reconstruction and a representative benchmark designed to improve comparability across methods.The authors also identify applications including generalization, immersive conferencing, avatar rendering, microexpression study, reenactment, and animation.
- Data Privacy: Participants signed GDPR-compliant agreements, including a right to request timely deletion of their captured data.
4 DYNAMIC NERF USING HASH ENSEMBLES
NeRSemble models dynamic human-head radiance fields by combining deformation-based warping with multi-resolution hash features, decoded for volume-rendered novel views.
- Method Overview: The method targets highly realistic novel-view synthesis of human heads undergoing complex non-rigid deformations.
- Method Overview: NeRSemble combines a deformation field with an ensemble of 3D feature volumes that decompose the temporal scene representation.
- Method Overview: The method’s overview warps a point into canonical space, queries ensemble hash grids, blends their features, and predicts density and view-dependent color.
- Preliminaries: NeRF represents a scene through a density field and view-dependent color field evaluated along camera rays for volume rendering.
- Hash Encoding: Instant NGP-style multi-resolution hash features provide a memory-efficient 3D encoding that small MLPs decode into radiance-field quantities.
4.2 Multi-Resolution Hash Ensemble
The multi-resolution hash ensemble represents dynamic scene states by blending features from several hash grids, allowing complex motion to be decoded into radiance fields.
- Representation: The representation assumes each scene state can be expressed as a combination of feature vectors from an ensemble of multi-resolution hash grids.
- Representation: The ensemble forms a linear combination of grid features to produce a blended radiance field for each time step.
- Representation: Blend weights are optimized jointly with the hash ensemble and the shared MLPbase and MLPcolor networks.
- Representation: Feature-space blending allows the model to represent complex movements before the blended features are decoded by MLPbase and MLPcolor.
4.3 Spatial Alignment of Features
NeRSemble aligns dynamic head features in a shared canonical space using deformation fields, then progressively blends hash-grid features to represent increasingly detailed motion.
- The deformation field maps observed-space points to corresponding canonical-space points conditioned on the current expression, establishing correspondences across time.
- Canonical alignment makes it easier to blend features belonging to the same moving point at different timesteps.
- The deformation field and hash ensemble can compete to explain facial dynamics, creating local minima where the deformation field learns no meaningful correspondences.
- Warm-Up Phase: During the first E_init epochs, only one hash grid is active, forcing the deformation field and deformation codes to explain dynamic behavior.
- Transition Phase: After warm-up, remaining hash tables are gradually added over E_trans epochs, extending representation from low-frequency dynamics to fine-scale motion and detail.
- Transition Phase: The blend schedule increases s linearly from 1 to N while keeping the first hash table continuously active for a smooth transition.
4.5 Depth Supervision
The method uses incomplete multi-view depth observations and line-of-sight priors to supervise geometry and improve the geometric fidelity of reconstructed heads.
- Depth supervision uses known ground-truth ray depths, computing the depth loss only for rays whose depth observations are available.
- The method adopts two line-of-sight priors from Urban Radiance Fields to further exploit depth constraints.
- One prior carves empty space in front of a surface while exponentially decaying epsilon during training.
- Another prior encourages density near the depth observation to follow a narrowing Gaussian distribution.
- Together with L_depth, the three priors target improved geometric fidelity in the reconstruction.
4.6 Background Removal
Background removal uses continuous alpha maps and an L1 sparsity loss to discourage density on rays associated with background pixels.
- Continuous-valued alpha maps M(r) discourage the model from reconstructing parts of the background.
- An L1 sparsity loss penalizes density on rays that hit background pixels, using each ray’s total transmittance and corresponding alpha value.
4.7 Optimization Objective
The optimization objective combines reconstruction, background, depth, proximity, and empty-space terms, with distortion regularization adapted to roughly convex human-head scenes.
- The final loss combines L_rgb, L_mask, L_depth, L_near, and L_empty.
- L_rgb is a standard MSE color loss computed only on foreground rays, while L_dist penalizes isolated low-density islands.
- Because human heads are roughly convex, L_dist is also evaluated on random rays toward the center, extending regularization behind the head.
- The loss weights are λ_depth, λ_dist, λ_near, λ_empty = 1e-4 and λ_mask = 1e-2.
4.8 Discussion on Dynamic Scene Representations
NeRSemble relates its temporal hash-grid representation to 4D tensor decompositions and HyperNeRF, while using deformation fields to align spatial features across time.
- NeRSemble Representation: The deformation field before hash-grid queries aligns spatial features across timesteps by explaining part of the motion.
- Relation to Tensor Decomposition: Concurrent approaches decompose dynamic scenes using six 2D feature planes rather than NeRSemble’s four 1D–3D outer-products.
- Relation to HyperNeRF: NeRSemble models canonical space with multiple hash grids, functioning as a discrete ambient dimension analogous to HyperNeRF’s continuous ambient dimensions.
5 EXPERIMENTAL RESULTS
Experiments evaluate NeRSemble on diverse multi-view facial-motion sequences and show strong image quality, temporal coherence, and robustness across architectural choices, while identifying motion and generalization limits.
- Evaluation Setup: 10 validation sequences contain 300–500 frames at 73 fps and evaluate four held-out viewpoints from 12 input cameras.Sequences cover rotations, expressions, blinking, talking, hair shaking, and mouth movements.
- Comparison to State of the Art: NeRSemble outperforms all baselines in every image metric, with especially strong improvements in SSIM and LPIPS for high-frequency facial detail.
- Comparison to State of the Art: NeRSemble provides smooth temporal renderings, whereas the time-agnostic Instant NGP baseline produces considerable flickering artifacts.
- Ablations: NeRSemble’s deformation field and hash ensemble complement each other: deformation sharpens simple motion, while hash blending represents challenging dynamics.Neither component alone matches the final model’s performance.
- Ablations: 16 hash tables perform negligibly worse than 32, indicating effective information sharing across tables.
- Ablations: Warm-up improves performance by helping hash grids learn correspondences with the deformation field before all grids become active.Without warm-up, encodings align less effectively and renderings become slightly blurrier.
- Ablations: Removing depth supervision only slightly impairs performance, suggesting that the 12 RGB input views already provide substantial geometric supervision.
- Ablations: Additional losses significantly improve metrics, while view-frustum culling and occupancy-grid filtering mainly improve visual quality outside the evaluation protocol.
6 CONCLUSION
The paper contributes a multi-view benchmark and a spatio-temporal NeRF representation for reconstructing animated human heads, with detailed and temporally coherent novel-view renderings.
- Contributions: The benchmark contains over 220 identities and 4700 sequences of diverse human heads captured in multi-view video.
- Method: NeRSemble factors out coarse movements with deformation fields and models fine deformations with an ensemble of hash-grid encodings.
- Results: Experiments show temporally coherent, highly detailed volumetric reconstructions that outperform existing baselines, especially for complex motions.