Source-linked AI summary
HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling
Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O'Toole, Changil Kim
TL;DR
Existing volumetric approaches do not simultaneously provide high-quality, real-time, memory-efficient rendering for challenging 6-DoF video. HyperReel combines ray-conditioned sample prediction with a compact dynamic volume representation, achieving strong quality and memory performance while rendering at up to 18 frames per second at megapixel resolution.
Problem
Existing 6-DoF video methods struggle to combine high-quality rendering, real-time speed, and a small memory footprint.
Method
HyperReel combines a ray-conditioned sample prediction network with a compact keyframe-based dynamic volume representation using trainable scene flow.
Results
HyperReel outperforms prior and contemporary 6-DoF video representations along multiple axes and renders at up to 18 frames per second at megapixel resolution without custom CUDA code.
Takeaways & Limitations
HyperReel provides a high-fidelity 6-DoF video representation balancing rendering quality, speed, and memory efficiency.
Takeaways & Limitations
The method falls short of the rendering speed ideally required for VR, specified as 72 FPS in stereo.
Abstract
from arXiv · showhide
Volumetric scene representations enable photorealistic view synthesis for static scenes and form the basis of several existing 6-DoF video techniques. However, the volume rendering procedures that drive these representations necessitate careful trade-offs in terms of quality, rendering speed, and memory efficiency. In particular, existing methods fail to simultaneously achieve real-time performance, small memory footprint, and high-quality rendering for challenging real-world scenes. To address these issues, we present HyperReel -- a novel 6-DoF video representation. The two core components of HyperReel are: (1) a ray-conditioned sample prediction network that enables high-fidelity, high frame rate rendering at high resolutions and (2) a compact and memory-efficient dynamic volume representation. Our 6-DoF video pipeline achieves the best performance compared to prior and contemporary approaches in terms of visual quality with small memory requirements, while also rendering at up to 18 frames-per-second at megapixel resolution without any custom CUDA code.
1. Introduction
HyperReel targets the difficult balance among quality, rendering speed, and memory for high-resolution 6-DoF video. It combines ray-conditioned sample prediction with a compact dynamic volume representation to enable high-quality real-time rendering.
- Motivation: Existing 6-DoF video methods struggle to achieve high quality, fast rendering, and a small memory footprint simultaneously.Some memory-efficient approaches take nearly a minute to render a megapixel image, while fast dynamic-volume methods may require gigabytes of storage for short clips.
- Approach: HyperReel combines a ray-conditioned sample prediction network with a memory-efficient dynamic volume representation.The sample network predicts sparse point samples, while the dynamic representation exploits spatio-temporal redundancy through compact keyframes and trainable scene flow.
- Approach: The sample prediction network accelerates volume rendering while improving quality for challenging view-dependent scenes.This distinguishes it from prior sample networks that primarily reduce rendering cost.
- Approach: The dynamic volume representation achieves high compression by exploiting spatio-temporal redundancy in dynamic scenes.It extends Tensorial Radiance Fields to compactly represent volumetric keyframes and uses trainable scene flow for intermediate frames.
- Results: HyperReel outperforms existing works while rendering high-quality megapixel images at up to 18 frames per second without custom CUDA code.The representation also handles scenes with challenging non-Lambertian appearances.
2. Related Work
Related work spans neural and volumetric scene representations, adaptive sampling, and dynamic 6-DoF video from monocular or multi-view captures. These approaches make different trade-offs among quality, speed, memory, and reconstruction constraints.
- Novel View Synthesis: Novel-view synthesis renders new scene views from posed input images using image-based or optimized 3D scene representations.Examples include point clouds, voxel grids, meshes, and layered multi-plane or multi-sphere representations.
- Neural Radiance Fields: Neural radiance fields provide high-quality view synthesis with small memory cost but are not naturally suited to real-time rendering.Rendering a ray requires integrating color and opacity from many points, often involving hundreds of MLP evaluations per pixel.
- Adaptive Sampling for Neural Volume Rendering: Adaptive-sampling methods accelerate neural volume rendering by reducing the number of volume queries or predicting sample locations along each ray.Some methods achieve near-real-time rendering, while others learn integrated opacity and color over a small number of ray segments.
- 6-Degrees-of-Freedom Video: 6-DoF video systems let users explore new views within videos, using multi-view camera rigs or time-varying multi-sphere images.Multi-sphere images can provide better visual quality but require higher training cost.
- 6-DoF from Monocular Captures: Monocular 6-DoF video reconstruction is highly under-constrained because it must decouple camera motion from object motion.Neural-radiance-field priors provide natural signal priors during reconstruction.
- 6-DoF from Multi-View Captures: Multi-view 6-DoF video remains challenging because an ideal format must jointly provide high visual quality, fast rendering, and memory efficiency.Existing methods may trade storage against speed or visual quality, whereas HyperReel targets all three properties.
3. Method
HyperReel combines ray-conditioned sampling with compact volumetric representations to render static and dynamic scenes efficiently while modeling challenging view-dependent appearance. Its dynamic representation uses keyframes and scene flow to reduce memory while supporting time-dependent rendering.
- Sample Networks for Volume Rendering: The sample prediction network maps each ray to sample points used for volume rendering, using either two-plane or Pl¨ucker ray parameterization.It can predict primitive parameters and derive samples by differentiable ray–primitive intersections.
- Sample Networks for Volume Rendering: Geometric primitives constrain predicted samples to begin along the ray, supporting multi-view consistency and easier interpolation across rays.World-space primitives allow distinct rays observing the same scene point to share a primitive.
- Flexible Sampling for Challenging Appearance: Tanh-activated per-sample offsets let HyperReel warp samples to represent complex view-dependent effects such as distorted refractions.Scalar gates modulate the offsets, while negative initialization discourages their use unless needed.
- Flexible Sampling for Challenging Appearance: The sample network enables real-time rendering with low sample counts while improving modeling of reflections and refractions without costly multi-step ODE solvers.Its flexibility is not guaranteed to be physically interpretable, but it is designed for efficient inference.
- Volume Representation: Static scenes use memory-efficient Tensorial Radiance Fields, while dynamic scenes extend TensoRF with keyframe-based volumes and trainable scene flow.The dynamic representation exploits spatio-temporal redundancy by storing volumetric snapshots at discrete time steps.
- Volume Representation: A dynamic keyframe factorization can have a memory footprint similar to static TensoRF for one frame when the number of keyframes is small relative to spatial resolution.The factorization stores spatial and temporal dimensions in separate tensor components.
- Rendering from Keyframe-Based Volumes: For dynamic rendering, samples are evaluated at the nearest keyframe time, with time-dependent colors, densities, and rendering weights forming HyperReel’s video representation.The pipeline combines sampling, keyframe-based volume representation, and the corresponding rendering scheme.
4. Experiments
Experiments evaluate HyperReel on static and dynamic datasets, comparing quality, speed, memory, and design choices. Across these tests, HyperReel achieves strong quality while retaining efficient training and rendering.
- Dynamic comparisons: HyperReel exceeds Neural 3D Video quality, trains in 1.5 GPU hours per sequence rather than 1000+ GPU hours, and renders faster on Technicolor.The evaluation covers five 50-frame sequences at full image resolution.
- Dynamic comparisons: HyperReel quantitatively outperforms NeRFPlayer while rendering approximately 40 times faster on the Neural 3D Video dataset.HyperReel also uses less memory per frame than StreamRF and NeRFPlayer on average.
- Dynamic comparisons: HyperReel exceeds NeRFPlayer by a 3 dB margin and renders more quickly on the Google Immersive dataset.The comparison evaluates seven scenes while holding out the central view.
- Ablation studies: One keyframe every four frames provides the best overall performance by balancing temporal resolution and spatial rank.Increasing keyframes captures more complex motion but distributes volume capacity across more time steps.
- Ablation studies: The Tiny model reaches 18 FPS and the Small model 9 FPS at megapixel resolution without custom CUDA code; the Small model matches Neural 3D Video overall quality more closely.The Tiny model has lower quality than Neural 3D Video on Technicolor, while the Small model achieves comparable overall performance.
- Ablation studies: Using the sample prediction network improves performance substantially over a no-network variant with four times as many samples.The comparison uses 128 samples without the network versus 32 with it, and every frame as a keyframe for the no-network variant.
- Ablation studies: Point offsets improve quality on both diffuse and highly refractive static scenes.The result suggests improved capacity allocation in addition to modeling view dependence.
5. Conclusion
HyperReel combines ray-conditioned sampling with a keyframe-based dynamic volume representation for balanced quality, speed, and memory efficiency. The paper reports broad comparisons and identifies limitations in extrapolated views, streaming, and VR-rate rendering.
- HyperReel combines a ray-conditioned sampling network with a keyframe-based dynamic volume representation.
- HyperReel outperforms prior and contemporary 6-DoF video representations along multiple qualitative and quantitative axes.
- The approach can lose quality for extrapolated or sparsely observed views, and its speed remains below the 72 FPS target for stereo VR.
- The keyframe-based representation is memory efficient but cannot be streamed like NeRFPlayer or StreamRF.
- The appendix adds training, evaluation, design, comparison, and per-scene metric details across static and dynamic datasets.
- The supplemental website provides code, real-time high-resolution demos, dynamic results, and qualitative comparisons on static and view-dependent scenes.
C. Additional Training & Evaluation Details
The training and evaluation details specify the ray-subsampling procedure used to make training more memory efficient and the LPIPS implementation used for comparisons.
- The ray-subsampling scheme is provided as pseudocode and is used to enable more memory-efficient training.
- All main-paper LPIPS comparisons use the AlexNet LPIPS variant, matching the baseline methods.
C.3. SSIM Evaluation Details
SSIM evaluation uses normalized images and a data range of 1, while some baselines use inconsistent implementations or an incorrect default range. The authors therefore treat aggregated SSIM scores as less reliable.
- SSIM is computed with scikit-image on images normalized to [0, 1] and with the data range parameter set to 1.
- Some methods use SSIM implementations inconsistent with the standard implementation, such as R2L.
- Some methods fail to set the data range, causing it to default to 2.0, as reported for Neural 3D Video.
- Inconsistent SSIM settings can return higher-than-intended values, making aggregated SSIM scores somewhat less reliable.
D.1. Additional Training Details
The additional training details describe optimization settings, smooth ray- and time-conditioned sample outputs, scene-specific ray parameterizations, and differentiable ray–primitive intersections. The method also applies per-sample color scaling for inconsistent multi-view color correction.
- Training uses 16,384 rays per batch, learning rates of 0.02 for the volume and 0.0075 for the sample network, plus scheduled regularization weights.
- Low-frequency positional encoding for rays and time encourages the sample network outputs to vary smoothly.
- Forward-facing scenes use normalized device coordinates and z-axis planes, while outward-facing scenes use Plücker rays and origin-centered spheres.
- Differentiable ray–primitive intersections allow color-loss gradients to propagate to the sample network.
- The sample network predicts per-sample color scales to handle inconsistent color correction and white balancing across views.
- The dynamic volume is initialized at 128^3 and progressively upsampled to a final 640^3 grid.
F.1. LLFF Dataset
On LLFF, HyperReel outperforms several baselines but does not surpass NeRF, and its quality is affected by calibration errors and sparse input views.
- Quantitative comparison: HyperReel outperforms DoNeRF, AdaNeRF, TermiNeRF, and InstantNGP on the real-world LLFF dataset.LLFF contains eight sequences with 1008×756-pixel images.
- Quantitative comparison: HyperReel achieves slightly worse quality than NeRF on LLFF.
- Evaluation setup: Table F.1 reports LLFF comparisons with FPS normalized per megapixel and memory measured in megabytes.
- Dataset challenges: Erroneous camera calibration and sparse input views make LLFF challenging for explicit volume representations.Explicit representations have more parameters and can more easily overfit the training images.
- Quantitative comparison: On downsampled 504×378 LLFF, HyperReel performs slightly worse in quality than R2L.
F.2. DeepView Dataset
HyperReel is compared with DeepView as a quantitative proxy for Immersive Light Field Video, achieving higher quality and lower memory use while rendering more slowly.
- Quality: HyperReel outperforms DeepView by a large margin in image quality on the static Spaces dataset.DeepView is used as a proxy because Immersive Light Field Video does not provide quantitative image-quality benchmarks.
- Memory and speed: 1.2 MB per frame versus 8.87 MB per frame, HyperReel uses less memory than Immersive Light Field Video’s baked layered mesh representation.The comparison is calculated from reported bitrate numbers.
- Memory and speed: More than 100 FPS versus a little over 4 FPS, the layered mesh renders faster than HyperReel on commodity hardware.HyperReel is implemented in vanilla PyTorch and could be optimized with custom CUDA kernels or baked into a real-time representation.
H. Additional Results
Additional results demonstrate HyperReel’s support for wide fields of view and improved handling of refractive appearance, alongside qualitative and per-scene comparisons.
- Wide-field rendering: HyperReel supports an unlimited field of view, including panoramic rendering of a synthetic 360-degree scene using spherical primitives.
- Refractive scenes: Point offsets noticeably improve quality for refractive scenes by allowing the sample network to capture appearance that violates epipolar constraints.
- Qualitative comparisons: Additional qualitative comparisons with Neural 3D Video Synthesis show more accurate and detailed recovered appearance on the Technicolor dataset.
- Per-scene results: Table H.1 reports per-scene Technicolor results and notes that SSIM metrics require discussion of their reliability.
- Per-scene results: Tables H.2 and H.3 provide per-scene comparisons with NeRFPlayer for the Neural 3D Video and Google Immersive Light Field Video datasets.These tables list results available for the indicated methods and datasets.