PaperlayerView exact revision on arXiv

Source-linked AI summary

HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion

Mustafa Işık, Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, Matthias Nießner

arXiv:2305.06356v2cs.CVcs.GRcs.LG

TL;DR

High-fidelity reconstruction of humans in complex motion remains difficult, especially at production-scale resolution and over long sequences. HumanRF combines a temporal low-rank representation with ActorsHQ, a 12MP multi-view dataset, and reports high-quality novel-view synthesis with a compact representation. The method still relies on a separate radiance field for each sequence and does not explicitly control articulation outside training poses.

  • Problem

    Reconstructing photo-realistic humans with fine detail and fast, complex motion at sufficient resolution remains an open production problem.

  • Method

    HumanRF uses low-rank space-time decomposition, adaptive temporal sequence splitting, shared MLPs, and ActorsHQ multi-view recordings to represent humans in motion.

  • Results

    HumanRF significantly improves over existing state-of-the-art methods on ActorsHQ and produces high-quality free-viewpoint video at 12MP.

  • Takeaways & Limitations

    ActorsHQ and HumanRF make a significant step toward production-level novel-view synthesis for high-resolution human performances.

  • Takeaways & Limitations

    HumanRF optimizes a separate radiance field for each sequence and lacks explicit articulation control outside the training poses.

Abstract

from arXiv · show

Representing human performance at high-fidelity is an essential building block in diverse applications, such as film production, computer games or videoconferencing. To close the gap to production-level quality, we introduce HumanRF, a 4D dynamic neural scene representation that captures full-body appearance in motion from multi-view video input, and enables playback from novel, unseen viewpoints. Our novel representation acts as a dynamic video encoding that captures fine details at high compression rates by factorizing space-time into a temporal matrix-vector decomposition. This allows us to obtain temporally coherent reconstructions of human actors for long sequences, while representing high-resolution details even in the context of challenging motion. While most research focuses on synthesizing at resolutions of 4MP or lower, we address the challenge of operating at 12MP. To this end, we introduce ActorsHQ, a novel multi-view dataset that provides 12MP footage from 160 cameras for 16 sequences with high-fidelity, per-frame mesh reconstructions. We demonstrate challenges that emerge from using such high-resolution data and show that our newly introduced HumanRF effectively leverages this data, making a significant step towards production-level quality novel view synthesis.

1 INTRODUCTION

HumanRF addresses the difficulty of reconstructing photo-realistic humans in complex motion at high resolution by combining a new dynamic representation with the ActorsHQ dataset. The paper targets long sequences, fine detail, and production-level novel-view synthesis.

  • Novel-view synthesis of recorded humans supports applications including movie and game production and immersive telepresence.
  • Fast, complex motion and photo-realistic detail remain open reconstruction challenges because human appearance and motion involve many degrees of freedom.
  • HumanRF extends hash encodings into time with low-rank space-time decomposition and splits sequences into segments for long-sequence representation.
  • ActorsHQ records clothed humans using 160 cameras at 12MP resolution and provides high-fidelity data for photo-realistic novel-view synthesis.

2 RELATED WORK

Related work spans neural, explicit, tensor-, hash-, and deformation-based representations for static scenes, dynamic scenes, and human rendering. HumanRF extends low-rank representations into time while using a 3D-1D decomposition and a template-free strategy for high visual quality.

  • 3D Neural Representations: Neural radiance fields encode scenes with MLPs, while sparse grids trade larger memory footprints for faster training and inference.
  • 3D Neural Representations: TensoRF reduces memory inefficiency through low-rank tensor decomposition, while Instant-NGP uses hash structures with small MLPs.
  • 4D Dynamic Representations: HumanRF represents 4D feature grids with four 3D hash grids and four 1D dense grids, unlike competing 2D-2D decompositions.
  • 4D Dynamic Representations: Dynamic NeRF methods model time through conditioning, deformation fields, flow supervision, fused static representations, or low-rank 4D decompositions.
  • Human Performance Capture: Human-rendering methods commonly deform canonical representations using templates such as SMPL or sparse skeletons, whereas HumanRF is template-free.

3 METHOD

HumanRF learns a temporally consistent 4D radiance-field representation for long, moving-human sequences from multiview RGB images and masks. It combines compact spatio-temporal decomposition, adaptive temporal partitioning, shared MLPs, volumetric rendering, and mask-based regularization.

  • Representation and training: HumanRF learns a 4D scene representation from multiview RGB images and foreground masks for temporally consistent novel-view synthesis.Differentiable volumetric rendering is supervised by photometric and mask losses.
  • Adaptive Temporal Partitioning: Moderate segment sizes outperform single-segment and per-frame representations, while stronger motion favors shorter segments and moderate motion favors larger ones.Adaptive partitioning avoids searching for one global segment size and performs better on average than fixed sizes.
  • 4D feature grid decomposition: Each temporal segment uses a compact 4D feature grid formed by decomposing spatio-temporal data into four 3D and four 1D grids.The 3D components use multiresolution hash grids, while the 1D components use dense vectors.
  • Adaptive Temporal Partitioning: Adaptive temporal partitioning selects segment boundaries using occupancy expansion so each segment covers a similar amount of 3D volume.A greedy heuristic spawns a segment when expansion exceeds a threshold, set to 1.25 in the experiments.
  • Shared MLPs and volume rendering: Shared shallow MLPs transform segment features into density, geometry features, and view-dependent radiance for volumetric rendering.The density network is three layers, the radiance network is four layers, and both are shared across the sequence.
  • Losses: RGB supervision uses a Huber color loss, while binary cross-entropy between foreground masks and accumulated rendering weights regularizes occupancy.The occupancy loss helps prune empty space early in training and accelerates training iterations.

4 DATASET

ActorsHQ is a high-fidelity multi-view dataset of clothed humans in motion, combining 12MP recordings from 160 synchronized cameras with per-frame mesh reconstructions.

  • Dataset Composition: ActorsHQ contains 39,765 frames of dynamic human motion captured using multi-view video.The dataset uses 4 female and 4 male actors performing two 100-second motion sequences in everyday clothing.
  • Capture System: Each sequence is recorded by 160 12MP cameras operating at 25fps within a 1.6m-diameter, 2.2m-high capture volume.A synchronized 420-LED programmable lighting array and 650us shutter speed support controlled illumination and reduced motion blur.
  • Geometry: The dataset provides independently reconstructed meshes with approximately 500k faces per frame alongside the recorded images.These reconstructions are generated using state-of-the-art multi-view stereo.
  • Benchmark Scope: Compared with standard multi-view human video benchmarks, ActorsHQ provides longer sequences, more cameras, and higher image resolution.The benchmark comparison is summarized in Table 1.

5 EVALUATION

HumanRF is evaluated on long, challenging motion sequences against general-scene and human-specific baselines, with analyses of quality, compression, and representation design.

  • 5.2 Quality vs Number of Frames: HumanRF consistently outperforms the baselines across sequence lengths using PSNR, LPIPS, SSIM, and VMAF.VMAF evaluates perceptual video quality and correlates with temporal consistency; deformation-based methods struggle with long sequences and complex motion.
  • 5.2 Quality vs Number of Frames: Instant-NGP achieves better LPIPS but lacks temporal stability and uses 20× more trainable parameters than HumanRF.Instant-NGP is trained independently for each frame, trading temporal consistency and memory footprint for per-frame quality.
  • 5.3 Design Choices: HumanRF’s 3D-1D feature-grid decomposition consistently outperforms tNGP and Hex4D under both moderate and strong motion.Hex4D benefits from compression under moderate motion but degrades substantially under rapid motion.
  • 5.3.2 Grid Resolution and Feature Dimensionality: HumanRF uses K_max = 2048 and F = 2 for its feature grid after searching finest grid resolution and feature dimensionality at fixed parameter count.The search fixes the number of parameters while varying grid resolution and feature dimensionality.
  • 5.3.2 Grid Resolution and Feature Dimensionality: HumanRF uses 5.2% of the parameters of per-frame Instant-NGP while retaining higher reported quality.Adaptive temporal partitioning keeps parameters per frame approximately constant, so total parameters scale linearly with sequence length.
  • 5.3.3 Segment Sizes and Expansion Factors: Rapid motion increases the frequency of spawning new temporal segments, producing smaller average segment sizes.Larger expansion-factor thresholds produce larger segments, without a striking quality difference as average segment size changes.

5.4 Input Resolution

HumanRF benefits from high-resolution training data when rendering at full resolution, recovering finer details as input resolution increases beyond prior benchmark resolutions.

  • 5.4 Input Resolution: HumanRF recovers finer details in full-resolution renderings as the training input resolution increases.The comparison uses 4× downscaled, 2× downscaled, and full-resolution inputs; numerical results are provided in the supplement.

5.5 Dynamic Furry Animal Dataset

HumanRF also applies to dynamic non-human scenes, achieving state-of-the-art results on the Dynamic Furry Animal dataset while adapting its grid settings for fur detail.

  • 5.5 Dynamic Furry Animal Dataset: HumanRF is a template-free method that can be applied beyond humans to foreground objects with masks.The method is evaluated on the Dynamic Furry Animal dataset using default settings except for L = 20.
  • 5.5 Dynamic Furry Animal Dataset: HumanRF surpasses the state of the art on dynamic furry-animal sequences despite most baselines using skeleton information.The authors increase the number of resolution levels to L = 20 to account for high-frequency fur details.
  • 5.6 Limitations and Future Work: HumanRF relies on ActorsHQ and optimizes a separate radiance field for each sequence, limiting direct use as a general pretrained avatar.The paper identifies model training on high-end recordings for monocular-only test sequences as future work.

6 CONCLUSION

HumanRF reconstructs high-fidelity human performance with a memory-efficient spatio-temporal radiance field, while ActorsHQ provides 12MP, 160-camera multi-view data for evaluation.

  • 6 CONCLUSION: HumanRF reconstructs human performance as a high-fidelity spatio-temporal radiance field.Its intra-frame 4D decomposition captures details, while adaptive splitting shares features across frames for memory-efficient long-sequence representation.
  • 6 CONCLUSION: ActorsHQ is the first publicly available multi-view dataset captured with 160 cameras recording 12MP footage.The dataset supports demonstrating HumanRF’s performance on high-resolution human-performance reconstruction.
  • 6 CONCLUSION: HumanRF produces high-quality free-viewpoint video and represents an important step towards production-level novel view synthesis.The authors also release ActorsHQ and HumanRF source code to support further research on photo-realistic virtual-human reconstruction.

HumanRF: Supplementary Material

The supplementary material is identified as HumanRF’s companion publication and provides its bibliographic and author information.

  • HumanRF: Supplementary Material: The paper’s authors are affiliated with Synthesia, University College London, and the Technical University of Munich.Most authors are affiliated with Synthesia, while Lourdes Agapito is at University College London and Matthias Nießner is at the Technical University of Munich.
  • HumanRF: Supplementary Material: The supplementary material is associated with arXiv preprint arXiv:2305.06356v2 dated 11 May 2023.The passage identifies the computer-vision arXiv category and version.
  • HumanRF: Supplementary Material: The supplementary material is titled HumanRF: Supplementary Material and was published in ACM Transactions on Graphics in 2023.The supplied passage gives the venue, volume, article number, publication month, page count, and DOI.

1 IMPLEMENTATION DETAILS

HumanRF’s implementation combines CUDA-optimized neural rendering with on-the-fly ray sampling, mixed precision, adaptive batching, and temporal segmentation to process large multi-view datasets.

  • 1 IMPLEMENTATION DETAILS: The implementation uses PyTorch, CUDA, Tiny CUDA neural networks, four 3D hash grids, torch-ngp, and NerfAcc.A custom CUDA kernel samples four 1D dense grids and composes them with sampled hash-grid features to reduce intermediate memory and improve performance.
  • 1 IMPLEMENTATION DETAILS: Light annotations exclude light-bloom regions from training to avoid their impact on photometric consistency.The annotations identify image regions that should not contribute to training.
  • 1 IMPLEMENTATION DETAILS: Training data are sampled on the fly from image pools across cameras and time frames instead of storing or precomputing terabyte-scale batches.This data-loader design avoids impractical memory and disk requirements associated with large chunks of training data.
  • 1 IMPLEMENTATION DETAILS: Training uses ADAM, learning-rate decay, FP16 operations, and iteration counts scaled by sequence length and input resolution.The supplied settings use an initial learning rate of 10^-2, decay to 5 · 10^-3, and more iterations for full-resolution training.
  • 1 IMPLEMENTATION DETAILS: Dynamic ray batching targets maximum sample counts of 640K, 576K, and 512K for 4× downscaled, 2× downscaled, and full-resolution inputs.The batch begins with 8192 rays and adjusts dynamically to maintain high GPU utilization.
  • 1 IMPLEMENTATION DETAILS: Temporal segmentation enables scaling to thousands of frames because each iteration samples rays from only eight time frames.On average, segments contain 12 frames and require about 64 million parameters, or 256MB, in GPU memory at a time instance.

2 EVALUATION

The evaluation measures novel-view quality across held-out cameras, actors, resolutions, motion complexities, and non-human scenes, with additional supplementary analyses.

  • 2 EVALUATION: The input-resolution experiment evaluates representation quality when training at different resolutions and rendering at full resolution.Table 1 provides additional results for this experiment, while Fig. 10 examines the effect of downscaled training data.
  • 2 EVALUATION: The protocol computes foreground-aware SSIM and LPIPS from tightly cropped images, while VMAF uses every third frame rendered from hero camera 127.The evaluation combines image metrics with a video-quality metric.
  • 2 EVALUATION: Additional experiments analyze motion complexity, per-sequence performance, and the effect of increasing sequence length.These results are reported in Tables 2 and 3 and illustrated in Fig. 3.
  • 2 EVALUATION: HumanRF is also evaluated on four non-human scenes from the DFA dataset.The authors infer from these visual results that HumanRF can produce high-fidelity results for non-human subjects.

3 HEX4D AND TNGP FORMULATIONS

The Hex4D formulation represents spatio-temporal features through decomposed 2D planes, while tNGP uses an undecomposed 4D hash grid. The section also shows HumanRF maintaining quality over longer sequences, unlike deformation-based baselines.

  • HEX4D formulation: Hex4D represents the spatio-temporal tensor with six 2D planes using multi-resolution dense grids.Its formulation combines plane features through addition rather than concatenation.
  • TNGP formulation: tNGP represents the spatio-temporal tensor directly with a 4D hash grid without decomposition.
  • Sequence-length influence: HumanRF consistently outperforms deformation-based baselines as sequence length increases and motion becomes more complex.The baselines struggle to capture complex motion over long sequences.
  • Sequence-length impact: HumanRF maintains constant synthesis quality across sequence lengths, whereas deformation-based baselines degrade when rendering the same pose and frame.
Loading 2305.06356v2…