Source-linked AI summary
FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality
Nianchen Deng, Zhenyi He, Jiannan Ye, Budmonde Duinkharjav, Praneeth Chakravarthula, Xubo Yang, Qi Sun
TL;DR
VR NeRF rendering must satisfy high-field-of-view, high-resolution, stereoscopic, egocentric viewing with low latency, but conventional inference can be slow or low quality. The paper introduces a gaze-contingent, foveated neural representation that incorporates visual and stereo acuity and jointly optimizes perceptual quality and latency. It reports responsiveness improvements without loss of fidelity relative to ground truth, while noting constraints on translation range and optimization over sphere radii.
Problem
VR imposes high-field-of-view, high-resolution, stereoscopic, and egocentric requirements that make NeRF rendering computationally heavy, with high latency or low quality.
Method
The method uses an egocentric concentric-spherical neural representation with gaze-contingent foveated synthesis matched to visual and stereoscopic acuity, plus spatial-temporal quality-latency optimization.
Results
The method improves responsiveness from 9 s to 20 ms without loss of visual fidelity compared with rendered or captured ground truth.
Takeaways & Limitations
The system delivers high-field-of-view, high-fidelity neural radiance fields perceived as identical to an arbitrarily high-quality rendered 3D scene under optimized quality-latency performance.
Takeaways & Limitations
Optimal quality is constrained by the innermost sphere’s radius, while enlarging that sphere indefinitely decreases synthesized image quality.
Abstract
from arXiv · showhide
Virtual Reality (VR) is becoming ubiquitous with the rise of consumer displays and commercial VR platforms. Such displays require low latency and high quality rendering of synthetic imagery with reduced compute overheads. Recent advances in neural rendering showed promise of unlocking new possibilities in 3D computer graphics via image-based representations of virtual or physical environments. Specifically, the neural radiance fields (NeRF) demonstrated that photo-realistic quality and continuous view changes of 3D scenes can be achieved without loss of view-dependent effects. While NeRF can significantly benefit rendering for VR applications, it faces unique challenges posed by high field-of-view, high resolution, and stereoscopic/egocentric viewing, typically causing low quality and high latency of the rendered images. In VR, this not only harms the interaction experience but may also cause sickness. To tackle these problems toward six-degrees-of-freedom, egocentric, and stereo NeRF in VR, we present the first gaze-contingent 3D neural representation and view synthesis method. We incorporate the human psychophysics of visual- and stereo-acuity into an egocentric neural representation of 3D scenery. We then jointly optimize the latency/performance and visual quality while mutually bridging human perception and neural scene synthesis to achieve perceptually high-quality immersive interaction. We conducted both objective analysis and subjective studies to evaluate the effectiveness of our approach. We find that our method significantly reduces latency (up to 99% time reduction compared with NeRF) without loss of high-fidelity rendering (perceptually identical to full-resolution ground truth). The presented approach may serve as the first step toward future VR/AR systems that capture, teleport, and visualize remote environments in real-time.
GT NeRF
The figure presents a gaze-contingent neural radiance field for VR, contrasting its foveal rendering quality with NeRF and full-quality ground truth.
- The method uses an egocentric-viewing-tailored coordinate and neural scene representation for VR.
- Its neural encoding and synthesis match human visual and stereoscopic acuity while balancing quality against run-time perceptual latency.
- Foveal zooms highlight the region where human vision is most sensitive to image quality and stereopsis.
- The method is compared against NeRF, with full-quality rendered ground truth serving as the reference.
1 INTRODUCTION
The introduction identifies a mismatch between VR’s demanding viewing requirements and conventional neural rendering, then presents a gaze-contingent NeRF system that jointly targets perceptual quality and latency. The authors report improved responsiveness without loss of visual fidelity and validate the system through multiple evaluations.
- VR requires high field-of-view, low latency, stereoscopic, and egocentric viewing, while NeRF inference is computationally heavy and can produce high latency or low quality.
- Existing image-based neural representations can suffer from high latency, low fidelity, or inconsistent robustness in complex scenes with occlusion.
- The method represents radiance fields with concentric spherical coordinates and models human visual, stereoscopic, and temporal sensitivities.
- The system was evaluated through psychophysical experiments, numerical analysis, and case studies on commercially available VR displays.
- Its contributions include foveated high-resolution, high-field-of-view first-person VR rendering and a spatial-temporal model for jointly optimizing latency and perceptual quality.
2 RELATED WORK
Related work spans image-based rendering, implicit neural representations, panorama-based 6DoF methods, and foveated rendering. These approaches address quality, viewing range, or computational cost, but retain constraints relevant to immersive egocentric VR.
- Image-based rendering provides pixel-wise quality and performance, but sparse and small viewing ranges limit smooth VR viewing with continuous head and gaze motion.
- Implicit neural representations approximate object appearance from camera pose, while prior work also represents shapes and surfaces.
- Existing implicit representations primarily target locally outside-in views and have limitations in field-of-view, speed, and resolution.
- Panorama imagery supports 360 FoV VR coverage, but its single projection center limits free-form camera translation and 6DoF viewing.
- Foveated rendering accelerates or enhances perceptual cues through mesh-, image-, and optical-based methods using high-FoV displays and eye tracking.
3 METHOD
The method combines an egocentric concentric-spherical neural representation with gaze- and acuity-adaptive elemental-image synthesis for stereoscopic VR. It jointly balances perceptual quality and latency through spatial-temporal modeling and image-based compositing.
- Runtime Rendering: The system synthesizes visual- and stereoscopic-acuity-adaptive elemental images, then uses image-based rendering to composite displayed stereo frames.Runtime synthesis covers foveal, mid-peripheral, and far-peripheral regions before blending the outputs.
- Egocentric Neural Representation: Concentric spherical coordinates target egocentric, inside-out VR viewing and parameterize representation sparsity through the number of spheres per network.The representation addresses viewpoint variation in large immersive environments and supports 6DoF interaction.
- Neural Representation: The neural network concatenates all N sphere-intersection coordinates for one ray and predicts them with a single inference instead of NeRF’s N per-ray inferences.Denser spherical sampling improves precision but slows runtime performance.
- Visual Acuity Adaptation: Gaze and eccentricity determine three elemental-image ranges: fovea at 0–20 degrees, mid-eccentricity at 0–45 degrees, and the full field at 0–110 degrees.The foveal image depends on gaze, while the entire-field image is gaze-independent; resolutions are 256², 256², and 230×256, respectively.
- Adaptive Stereoscopic Acuity: Adaptive stereo acuity reduces inference by computing foveal left/right views and shared midpoint views rather than six independent elemental images.Midpoint images are shifted by half the inter-gaze difference for each eye to reduce blending misalignment.
- Latency-Quality Joint Optimization: The latency-quality model analyzes how representation density and network settings affect image error, with fixed sphere-radius ranges showing lower Eimage as N increases.The method uses this analysis to balance latency against perceptual quality during optimization.
4 IMPLEMENTATION
The implementation uses separate foveal and periphery datasets and neural networks, with parameters selected to balance rendering quality and latency. The system is implemented with GPU-accelerated graphics software and trained across the evaluated scenes.
- Optimized network parameters: The foveal network uses Nm = 4, Nc = 512, and N = 16, while the periphery network uses Nm = 4, Nc = 256, and N = 32.These parameters were selected through latency-quality joint optimization and tested across 64 spheres depicting the scenes.
- Environment: The system uses OpenGL with CUDA and TensorRT on an Intel Xeon Gold 6230R CPU, 256GB RAM, and one NVIDIA GTX 3090 graphics card.Both networks were trained for 200 epochs per scene.
5 EVALUATION
The evaluation compares OURS with NeRF, ground-truth-derived conditions, and panorama-based synthesis across subjective preference, static and dynamic perceptual quality, dis-occlusion, view-dependent effects, and performance. Across these tests, OURS maintains high perceptual quality while supporting dynamic viewing, broader scene translation, realistic view-dependent effects, and interactive frame rates.
- 5.1 User Study: 89.6% of participants preferred OURS over NeRF, while 42.4% preferred OURS over GT in the 2AFC study.The OURS-versus-GT result was close to random guessing, whereas both GT and OURS were significantly preferred over NeRF.
- 5.2 Static Visual Quality: OURS achieved lower LPIPS than NeRF from the fovea through near periphery, with the pattern extending to ≤35 deg in lobby and barbershop.Across four scenes, near-peripheral results favored OURS over NeRF; comparisons with F-GT varied by scene in the nearest eccentricities.
- 5.2 Static Visual Quality: At eccentricities beyond 20 deg, OURS had lower LPIPS than F-GT but higher LPIPS than NeRF, with the pattern applying across all four validated scenes.In the reported lobby example at 40 deg, the stimulus effect was significant with ω2 = 0.32 and large pairwise effect sizes.
- 5.3 Dynamic Viewing Quality: OURS achieved FovVideoVDP M = 8.26, exceeding M-GT M = 6.77 and F-GT M = 7.99 under dynamic viewing.The difference was statistically significant, with F(11,22) = 14.15 and p < .001***.
- 5.4 Qualitative Comparison with Panorama-Based View Synthesis: OURS performed better than PANO for large translations facing occluded geometry and produced stronger view-dependent effects in glossy scenes.The comparison showed a marginal difference for mostly flat geometries, while the glossy-clock example demonstrated stronger view-dependent effects for OURS.
- 5.5 Performance: The full method achieved 50 FPS for 1440 × 1600 stereo VR frames, while spatial foveation alone reached about 30 FPS.The complete system combined spatial and stereoscopic foveation and maintained perceived resolution and quality.
6 CONCLUSION
The method targets egocentric and stereoscopic VR by encoding gaze-contingent visual acuity and stereopsis into neural scene representation and view synthesis. It balances quality and latency for high-resolution, high-field-of-view immersive viewing.
- The method individually synthesizes foveal, mid-, and far-periphery retinal images, then blends them into a wide-field-of-view image.
- It derives an analytical quality-latency model and optimizes both factors using psychophysical study data.
- The representation supports synthesis of both virtual content and physically captured scenes.
- Compared with NeRF, the method produces faster and higher-perceptual-fidelity results for high-resolution, high-field-of-view immersive viewing.
NeRF OURS
The evaluation compares OURS with NeRF for neural rendering, including physically captured scenes and challenging perceptual conditions. The reported limitations identify constraints on camera translation, peripheral sampling, and spatial-temporal optimization.
- OURS enhances visual quality over NeRF in high-depth-disparity and high-frequency regions of physically captured scenes.The comparison uses fovea-only network parameters because the available data have limited field-of-view coverage.
- The method remains robust to occluded scenes with complex geometry, addressing a challenge of panorama-based synthesis approaches.
- Camera translation is constrained by the radius of the innermost sphere, although the method supports a larger translation scale than panorama-plus-neural-disocclusion approaches.Enlarging the sphere indefinitely decreases synthesized image quality.
- Peripheral aliasing remains because of low sampling rates, but the user study found no significant perceptual-quality degradation.Traditional anti-aliasing may improve results but introduces a significant performance penalty with neural synthesis.
- The sampling strategy models eccentricity, acuity, and stereopsis but does not yet incorporate finer-grained luminance or depth sensitivity.Such analyses are proposed as possible routes to higher quality or faster performance.
- The spatial-temporal optimization links output precision and latency to sphere count rather than sphere radii for simplicity.Content-aware sphere-radius distributions and joint parameter training are proposed to improve synthesis quality and performance.