Source-linked AI summary
VR-NeRF: High-Fidelity Virtualized Walkable Spaces
Linning Xu, Vasu Agrawal, William Laney, Tony Garcia, Aayush Bansal, Changil Kim, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Aljaž Božič, Dahua Lin, Michael Zollhöfer, Christian Richardt
TL;DR
VR-NeRF addresses the challenge of high-fidelity, scene-scale exploration of walkable spaces in VR. It combines dense HDR capture with neural radiance field reconstruction and real-time rendering; on a custom demo machine, it renders full dual 2K×2K eye buffers at 36 Hz.
Problem
Existing methods offer either high-fidelity view synthesis within a headbox under 1 m in diameter or scene-scale synthesis at lower quality or frame rate.
Source: §1 · p. 2Method
VR-NeRF combines a custom multi-camera capture rig with a neural radiance field model using perceptual HDR color and mip-mapping for LOD rendering, plus a multi-GPU VR renderer.
Source: §1 · p. 2Results
Two 2K×2K eye buffers (~8 megapixel) render at 36 Hz on the custom demo machine.
Source: §1 · p. 2Takeaways & Limitations
The system provides high-fidelity free exploration of walkable spaces in VR on the custom demo machine.
Source: §1 · p. 2Aggressive pruning does not work well for complex real-world scenes such as those with reflective surfaces, transparent objects, or unbounded scenes.
Source: §7 · p. 8
Abstract
from arXiv · showhide
We present an end-to-end system for the high-fidelity capture, model reconstruction, and real-time rendering of walkable spaces in virtual reality using neural radiance fields. To this end, we designed and built a custom multi-camera rig to densely capture walkable spaces in high fidelity and with multi-view high dynamic range images in unprecedented quality and density. We extend instant neural graphics primitives with a novel perceptual color space for learning accurate HDR appearance, and an efficient mip-mapping mechanism for level-of-detail rendering with anti-aliasing, while carefully optimizing the trade-off between quality and speed. Our multi-GPU renderer enables high-fidelity volume rendering of our neural radiance field model at the full VR resolution of dual 2K$\times$2K at 36 Hz on our custom demo machine. We demonstrate the quality of our results on our challenging high-fidelity datasets, and compare our method and datasets to existing baselines. We release our dataset on our project website.
1 INTRODUCTION
VR-NeRF targets high-fidelity free-viewpoint exploration of walkable spaces, where existing methods trade small-headbox fidelity against scene-scale quality or frame rate. Its capture, reconstruction, and rendering pipeline culminates in full-resolution VR rendering at 36 Hz on a custom demo machine.
1 INTRODUCTION
- VR-NeRF combines dense high-fidelity capture, high-fidelity neural radiance field reconstruction, and real-time VR rendering for walkable spaces.The system targets a gap between high-fidelity synthesis limited to a headbox under 1 m in diameter and scene-scale synthesis with lower quality or frame rate.Source: §1 · p. 2
- The custom rig captures thousands of 50-megapixel HDR images, with several datasets exceeding 100 gigapixels.The paper describes these datasets as two orders of magnitude larger than existing datasets.Source: §1 · p. 2
- A perceptually based color space represents HDR radiance up to 10,000 cd/m2, while mip-mapping supports real-time LOD rendering with aliasing suppression.The model learns up to 22 stops of dynamic range; a pruning stage also produces an occupancy grid to speed rendering while focusing on geometry estimation.Source: §1 · p. 2
- Two 2K×2K eye buffers (~8 megapixel) render at 36 Hz on the custom demo machine, enabling free exploration of walkable spaces in high fidelity.The reported performance is for the full resolution of the Quest Pro VR headset.Source: §1 · p. 2
2 RELATED WORK
Prior approaches to walkable-space capture and view synthesis leave limitations in coverage, scale, HDR, or level of detail. VR-NeRF builds on neural radiance fields and instant neural graphics primitives, adding capture and rendering approaches aimed at dense, high-fidelity, real-time VR exploration.
High-Resolution Multi-View Capture System
- Existing multi-view captures are limited to small headboxes or are sparse, restricting freedom of motion; the authors build a rig for dense, high-resolution HDR capture of walkable spaces.The rig is described as capturing 50-megapixel details.Source: §2 · p. 2
Large-scale Novel View Synthesis
- NeRFs offer versatility for complex scenes, but instant neural graphics primitives lack level-of-detail and perceptually based HDR support for high-fidelity walkable spaces.The paper uses iNGP as its foundation because it supports real-time rendering without model baking and offers extensible capacity through its hash grid.Source: §2 · pp. 2–3
High Dynamic Range (HDR)
- VR-NeRF trains directly on HDR input images in a perceptually uniform color space, without custom losses or tonemapping modules.Prior approaches described here either learn linear radiance from raw sensor measurements or reconstruct it from low-dynamic-range images using differentiable tonemapping.Source: §2 · p. 3
Level of Detail (LOD)
- The proposed efficient LOD approach is designed for iNGP and enables real-time VR rendering with anti-aliasing.The paper contrasts it with Mip-NeRF's encoding, which is incompatible with efficient grid-based NeRF approaches, and Zip-NeRF's supersampling approximation, whose multi-second inference prevents real-time rendering.Source: §2 · p. 3
3 THE “EYEFUL TOWER” CAPTURE RIG
The Eyeful Tower system is designed to capture walkable scenes densely and in high fidelity, then process and calibrate the resulting images into datasets. The capture procedure tiles floor space with rig positions, and the resulting datasets span 6–120 m2.
3 THE “EYEFUL TOWER” CAPTURE RIG
- The Eyeful Tower design addresses capture coverage and fidelity alongside mobility, rigidity for exposure bracketing, and on-camera storage.Its stated design considerations also include parallel capture and offloading photos over a single network cable.Source: §3 · p. 3
3.1 Capture Rig Design
- The rig uses 22 Sony α1 cameras distributed across seven levels on a 1.8 m pole, with a 1.5 kWh battery powering cameras and control hardware.The cameras provide 50-megapixel raw images with 14 stops of dynamic range.Source: §3.1 · p. 3
3.2.1 Desirable Capture Density.
- Walkable-scene capture needs dense viewpoint coverage because scene-scale datasets lack the distribution needed to extrapolate to novel viewpoints for 6-degree-of-freedom head movement..Source: §3.2.1 · p. 3
3.2.2 Capture Procedure.
- The capture procedure tiles available floor area with rig positions roughly 30 cm apart, usually recording forward- and backward-facing views.Near walls, the procedure may capture only the direction facing away from the wall.Source: §3.2.2 · p. 3
3.3.1 HDR Image Merging.
- Nine exposures are merged into an HDR image using a robustified Poisson photon-noise estimator to estimate scene radiance.The process tracks minimum radiance per pixel and color channel to ignore outliers, and assigns fully saturated pixels the lowest radiance that saturates in all images.Source: §3.3.1 · pp. 3–4
3.3.2 Camera Calibration.
- Agisoft Metashape Pro 2.0 estimates camera poses and intrinsics using rig calibration, while scale-bar markers place camera poses in metric space for 1:1 VR rendering.The software handles datasets of up to 6,300 photos at 50-megapixel resolution.Source: §3.3.2 · p. 4
3.3.3 Captured Datasets.
- The resulting datasets contain 29–303 billion pixels or rays and cover spaces of 6–120 m2.Table 1 summarizes cameras, rig positions, images, capture time, surface area, and ray counts at full and typical 2K training/rendering resolutions.Source: §3.3.3 · p. 4, Table 1 · p. 4
4 HIGH-FIDELITY NEURAL RADIANCE FIELDS
The section builds on Instant NGP to improve high-fidelity NeRF rendering through perceptual HDR modeling, feature-grid level-of-detail, and pruning. These mechanisms target visual quality while preserving rendering speed for large-scale scenes.
4.1 Perceptual Modeling of High Dynamic Range
- A perceptual color space enables HDR image optimization with a simple L1 loss, avoiding the bright-area dominance of linear-space losses.The capture pipeline spans 22 stops, and mapping luminance into PQ space makes L1 or L2 errors track human visual perception.Source: §4 · p. 4, §4.1 · pp. 4–5
4.2 Feature Grid Mip-Mapping for Level-of-Detail
- A real-time-capable mip-mapping scheme for multi-level hash grids supports continuous LOD and reduces aliasing in large-scale scenes.Prior grid-based mip-mapping was unsuitable for real-time rendering, at 1.1 FPS on 8×V100.Source: §4.2 · p. 5
4.2.1 Feature Grid Mip-Mapping.
- Ray-cone footprints determine which grid levels are resolvable; finer levels are downweighted or ignored when they would introduce aliasing.The method estimates footprint size along the ray and in contracted space, then uses the grid configuration and a Nyquist-based threshold to select the LOD.Source: §4.2.1 · p. 5
4.2.2 LOD Bias.
- An adjustable LOD bias continuously shifts detail sharpness, and annealing it from negative values toward zero recasts coarse-to-fine training as a feature-scale schedule.A negative bias initially limits the model to low-frequency features.Source: §4.2.2 · p. 5
4.2.3 Distance-aware Features.
- Distance-aware features can reduce artifacts from rig shadows in ambiguously captured locations and encourage capacity allocation to fine-grained detail.Mip-mapping allows the model to learn distance-dependent appearance where limited training views can otherwise lead to incorrect geometry or appearance.Source: §4.2.3 · p. 5
4.3 Optimizing the Quality–Speed Trade-off
- An explicit binary occupancy grid skips free space to reduce sample queries and MLP evaluations while balancing quality against rendering speed.The grid is initialized from known rig positions to carve out free space inside approximated capture-rig cylinders.Source: §4.3 · p. 6, §4.3.1 · p. 6
4.3.1 Cylinder Pruning.
- Cylinder pruning marks voxels inside the capture-rig cylinders as free, preventing floaters in front of cameras and accelerating early training.The resulting view-consistent models also benefit from improved convergence speed.Source: §4.3.1 · p. 6
4.3.2 Joint History- and Grid-based Pruning.
- Joint history- and grid-based pruning combines observed training-ray densities with dense voxel sampling to identify low-density regions more conservatively.Pruning begins after 100K iterations, uses the closest-camera minimum step size for a worst-case density estimate, and upsamples the grid over training.Source: §4.3.2 · p. 6
5 VR NERF RENDERING
The VR renderer combines GPU-specific MLP evaluation, dynamic work allocation, and a multi-GPU workstation. Its dynamic work split increases frame rate relative to an equal static split.
5 VR NERF RENDERING
- The target is native Quest Pro rendering at dual 2K×2K eye buffers and ideally 72 FPS, addressed with hardware, software, and model optimizations.The renderer combines compute-efficient LOD, automatic work distribution, and a custom 20-GPU workstation.Source: §5 · p. 6
- 49% higher FPS is achieved by dynamic work splitting compared with a static split across three GPUs.The renderer assigns contiguous image rows according to each GPU's measured throughput.Source: Figure 4 · p. 6, §5 · p. 6
- The specialized small-network MLP implementation keeps layer activations in registers and interleaves network evaluation with ray marching and hash-grid sampling.This limits memory traffic and avoids staging neural features through global memory or multiple kernels.Source: §5 · p. 6
- Dynamic GPU allocation responds to uneven ray workloads by measuring each GPU's rays-per-second throughput and distributing rows proportionally.Ray costs vary with scene pruning, GPU caching, and overhead; each GPU stores a separate scene copy of about 700 MB VRAM.Source: §5 · p. 6
- A custom workstation connects 20 Nvidia A40 GPUs through a PCIe switching solution for high-fidelity VR rendering.It uses a Dell R7515 server with an AMD Epyc 7313P CPU and 256 GB RAM.Source: §5 · p. 6
6 RESULTS AND EVALUATION
The evaluation finds that PQ color modeling improves reconstruction metrics across test scenes, while mip-mapped features improve visual quality and geometry. Runtime results show that full-resolution VR performance clears the ASW threshold for four of six scenes.
6 RESULTS AND EVALUATION
- The evaluation holds out a fixed camera and trains ablations for 110K iterations on 1K-resolution Eyeful Tower datasets.The demo videos instead use 2K-resolution images and models trained for more than 200K iterations.Source: §6 · p. 7
6.1 Comparative Evaluation
- The PQ color space consistently outperforms linear color space across all test scenes and metrics, while LOD features further improve rendering quality and geometry.Table 2 reports average PSNR, SSIM, and LPIPS in both sRGB and PQ spaces; the qualitative improvements from mip-mapped features include reduced aliasing and cleaner geometry.Source: §6.1 · p. 7, Table 2 · p. 7
- Mip-mapped features address rig-shadow appearance, ambiguous geometry, distant aliasing, and detail loss in different viewing scenarios.The authors report similar quantitative metrics to baselines, but describe the visual improvements as more apparent and important for high-fidelity VR rendering.Source: §6.1 · p. 7
6.2 Performance Evaluation
- The runtime evaluation uses mean and standard deviation of frame rates alongside p99 FPS as a proxy for worst-case performance.The paper identifies p99 as the more useful indicator of VR experience because falling below 36 FPS can prevent ASW from maintaining smooth playback.Source: §6.2 · p. 7, Figure 5 · p. 8
- 4 of 6 scenes exceed the 36 FPS ASW threshold at full VR resolution; at half resolution, all but one achieve a p99 of 72 FPS.The full-resolution result is for the six scenes shown in Figure 5; the half-resolution results support smooth VR without ASW for all but one scene.Source: §6.2 · p. 7, Figure 5 · p. 8
7 DISCUSSION
The discussion identifies boundaries for aggressive pruning and distance-aware feature weighting. Both can improve rendering in some settings but have limitations in complex scenes or unseen viewpoints.
7 DISCUSSION
- Aggressive pruning speeds rendering in bounded scenes with mostly solid surfaces, but can produce box-like artifacts in reflective, transparent, or unbounded scenes.The artifacts may be more apparent in VR than in rendered 2D images or videos.Source: §7 · p. 8
- Distance-aware features improve representation of distance-dependent appearance but reduce extrapolation to unseen viewpoints or viewing distances.The authors work around possible density appearing in free space by pruning as much free space as possible.Source: §7 · p. 8
8 CONCLUSION
VR-NeRF combines a multi-camera capture rig, HDR neural reconstruction, and multi-GPU rendering for high-fidelity walkable spaces. The capture system evolved from handheld photography to a 22-camera rig that reduced the required scene passes.
8 CONCLUSION
- VR-NeRF presents a holistic system for capturing, reconstructing, and rendering high-fidelity walkable spaces in virtual reality.Its contributions include a multi-camera rig for thousands of HDR photos, perceptual color modeling, feature mip-mapping, and a multi-GPU renderer.Source: §8 · p. 8
A.1 “Rig” v0 — Handheld DSLR
- The handheld DSLR workflow typically produced 100–200 photos per scene in 1–2 hours, but achieving sufficient coverage was difficult and exposure could vary across the scene.These limitations motivated a rigid multi-camera rig that could capture at different heights simultaneously and take exposure brackets.Source: §A.1 · p. 13, §A.2 · p. 13
A.2 Eyeful Tower v1 — 9 fisheye cameras
- Eyeful Tower v1 used nine cameras at different heights and directions, but still required four scene orientations for 360-degree horizontal coverage.Its cameras could take nine-image exposure brackets, one stop apart from −4 EV to +4 EV.Source: §A.2 · pp. 13, 15
A ADDITIONAL CAPTURE DETAILS
- The v1 camera rig used a 1.5 kWh battery to power the cameras and capture equipment for 6 to 8 hours.A Raspberry Pi ran the capture software, and a network switch connected the cameras.Source: §A.2 · pp. 13, 15
A.3 Eyeful Tower v2 — 22 rectilinear lenses
- Eyeful Tower v2 increased the camera count to 22 and enabled two scene passes instead of v1’s four.The replacement rectilinear lenses were selected for 2–3× MTF50 sharpness and less than half the chromatic aberration, in pixels.Source: §A.3 · p. 15
B MODEL IMPLEMENTATION DETAILS
The implementation models camera vignetting, uses geometry-related regularization during training, and samples pixels with greater emphasis on high-frequency detail. It leaves lens distortion optimization for future work.
B.1 Vignetting and Lens Distortion
- The model learns per-camera vignetting parameters, while lens-distortion optimization is left for future work.Explicit vignetting modeling helps avoid black floaters caused by explaining edge brightness falloff through scene content; a test camera’s vignetting can be optimized before inference.Source: §B.1 · p. 15, Figure 15 · p. 15
B.2 Loss Design
- Depth guidance from a reconstructed mesh is used early in training, and depth-variance regularization can help suppress reflection-related geometry errors without ground-truth depth.The method also uses distortion loss to encourage sparse ray weights; weight decay can improve geometry and flat surfaces, but slows convergence and can lose detail.Source: §B.2 · p. 16, Figure 16 · p. 16, Figure 17 · p. 16
B.3 Weighted Sampling
- Weighted sampling uses image Laplacian pyramids to prioritize high-frequency details, thresholding importance at the 75th percentile.For perspective images, sampling weights also account for wide-angle pixel footprints by weighting inversely to their radii.Source: §B.3 · pp. 16–17
C ADDITIONAL RESULTS AND ANALYSIS
The supplemental analyses examine weighted sampling, HDR color learning, and level-of-detail effects. They report stabilized learning in PQ space and finer detail with LOD.
C ADDITIONAL RESULTS AND ANALYSIS
- The supplement reports a quantitative breakdown of complete ablations across all 11 Eyeful Tower datasets in Table 3.Source: C ADDITIONAL RESULTS AND ANALYSIS · p. 17
- Weighted sampling reveals finer scene details early in training, including the “SHARP” sign at 2K iterations.The pixel weights are derived using the idea of a Laplacian pyramid.Source: Figure 19 · p. 17, Figure 18 · p. 17
C.1 Learning of linear RGB space
- PQ-space learning keeps RGB values bounded and stabilized, unlike linear RGB learning with a safe exponential activation, whose maximum predictions fluctuate and grow.The PQ conversion preserves very bright and dark regions; its values lie in [0,1] and use sigmoid activation.Source: §C.1 · pp. 17–18, Figure 20 · p. 17, Figure 21 · p. 18
C.2 Effects of LOD
- LOD improves the revelation of fine details on the Inria dataset; the authors conjecture that masking high-resolution grid features reserves them for close observations.Figure 22 compares the two ablated modules on the Inria dataset.Source: §C.2 · p. 18
C.3 Ablation on pruning strategy
The pruning ablation compares history-based, grid-based, and joint strategies. Joint pruning combines their strengths and yields accurate occupancy grids with limited computation.
C.3 Ablation on pruning strategy
- Joint pruning combines training-history observations with dense voxel-grid sampling, producing accurate occupancy grids with limited computation using four grid samples per voxel.The resulting depth and normal maps indicate well-preserved geometry compared with either individual strategy.Source: §C.3 · p. 18
- History pruning can leave holes because stochastic samples may not visit every voxel during the pruning period.Source: §C.3 · p. 18
- Grid pruning accuracy depends on samples per voxel, but increasing samples raises computation and even sampling can produce box-like geometry artifacts.Source: §C.3 · p. 18
- On the raw Inria dataset, iNGP with PQ and LOD better preserves geometry and details than the ablated baselines.Exposure values are adjusted around the bright window areas.Source: Figure 22 · p. 18
C.4 Mip-NeRF 360 & Inria datasets
The supplemental experiments extend the method to Mip-NeRF 360 and Inria scenes, compare 1K and 2K training, and examine per-image appearance variation. Shadow compensation remains unresolved in challenging captures.
C.4 Mip-NeRF 360 & Inria datasets
- The method recovers fine details and accurate occupancy grids on all nine Mip-NeRF 360 scenes trained at 2K resolution, and shows three Inria scenes trained with HDR inputs and PQ color space.Source: §C.4 · p. 18, Figure 25 · p. 20, Figure 26 · p. 21
C.5 High-resolution rendering
- Validation PSNR is generally close between paired 1K- and 2K-resolution training runs, with a slight SSIM drop at 2K; a 4K example shows fine-grained details.The comparison is presented as evidence of potential for training on higher-resolution images sufficiently long.Source: Figure 23 · p. 18, §C.5 · p. 21
C.6 Handling per-image variations
- Shadow handling remains an open problem: interpolated per-frame latent codes and shadow fields yield suboptimal solutions, especially when few observations are mostly shadow-cast.Interpolated latent codes move shadows smoothly, but the overall appearance remains noisy with dark floats.Source: §C.6 · p. 21, Figure 28 · p. 22
C.7 Nerfstudio results (nerfacto)
The supplement compares VR-NeRF with Nerfstudio’s nerfacto model, describes the custom 20-GPU workstation, and explains how LOD masking reduces rendering time.
C.7 Nerfstudio results (nerfacto)
- The shadow-field prediction looks reasonable overall but is not accurate enough to compensate properly for the affected appearance.The implementation uses low-frequency grid features for shadow-field prediction.Source: Figure 29 · p. 22
- Nerfacto handles far-field views and captures many scene details, but commonly misses fine textures such as those on carpets and floors.The comparison uses sRGB nerfacto results against VR-NeRF results trained on HDR and converted to sRGB.Source: §C.7 · pp. 22–23, Figure 30 · p. 23
D.1 Design of our 20-GPU Workstation Machine
- The custom workstation targets at least 16 directly attached dual-slot GPUs, a single CPU, Windows 10, and a movable, quiet enclosure; commercial options did not meet all requirements.The enclosure must fit through a standard 32-inch door and be quieter than 55 dB; the built system uses a sound-dampening rack.Source: §D.1 · p. 23
D.2 Efficient Level-of-Detail Rendering
- LOD masking reduces rendering time by skipping expensive finest hash-grid feature samples and conditionally replacing them with zeros.Profiling found substantial frame time spent waiting for hash-grid feature sampling, especially at the finest levels.Source: §D.2 · p. 23