Source-linked AI summary
NeRF-SR: High-Quality Neural Radiance Fields using Supersampling
Chen Wang, Xian Wu, Yuan-Chen Guo, Song-Hai Zhang, Yu-Wing Tai, Shi-Min Hu
TL;DR
NeRF struggles to generalize to resolutions higher than its input images, limiting high-resolution novel-view synthesis from mostly low-resolution observations. NeRF-SR addresses this with sub-pixel supersampling and depth-guided reference-patch refinement, producing photorealistic high-resolution views without external data. The method is reported to outperform baselines requiring LR-HR training pairs, while losing NeRF’s arbitrary-scale property.
Problem
NeRF can render arbitrary resolutions but struggles when the target resolution exceeds that of the input images, making high-resolution synthesis from mostly low-resolution views difficult.
Method
NeRF-SR uses sub-pixel supersampling to enforce multi-view consistency and depth-guided refinement to transfer details from relevant patches in an HR reference.
Results
NeRF-SR produces photorealistic high-resolution novel views from mostly low-resolution inputs without external data and outperforms baselines requiring LR-HR training pairs.
Takeaways & Limitations
The pipeline provides high-resolution novel view synthesis using posed multi-view images of the target scene rather than external priors.
Takeaways & Limitations
NeRF-SR does not retain NeRF’s arbitrary-scale property and introduces extra computation, although its training time is no greater than training an HR NeRF.
Abstract
from arXiv · showhide
We present NeRF-SR, a solution for high-resolution (HR) novel view synthesis with mostly low-resolution (LR) inputs. Our method is built upon Neural Radiance Fields (NeRF) that predicts per-point density and color with a multi-layer perceptron. While producing images at arbitrary scales, NeRF struggles with resolutions that go beyond observed images. Our key insight is that NeRF benefits from 3D consistency, which means an observed pixel absorbs information from nearby views. We first exploit it by a supersampling strategy that shoots multiple rays at each image pixel, which further enforces multi-view constraint at a sub-pixel level. Then, we show that NeRF-SR can further boost the performance of supersampling by a refinement network that leverages the estimated depth at hand to hallucinate details from related patches on only one HR reference image. Experiment results demonstrate that NeRF-SR generates high-quality results for novel view synthesis at HR on both synthetic and real-world datasets without any external information.
1 INTRODUCTION
NeRF-SR targets high-resolution novel view synthesis when available input images are mostly low resolution. It combines sub-pixel supersampling with depth-guided refinement to produce high-quality views without external information.
- Novel view synthesis generates photorealistic views from posed images and supports applications including VR, navigation, and telepresence.
- NeRF represents scenes continuously with MLPs, enabling rendering at arbitrary resolutions but struggling beyond the resolution of training images.
- NeRF-SR addresses the higher-resolution gap by enforcing multi-view consistency through supersampling and generating super-resolved images and depth maps.
- NeRF-SR is reported to produce visually pleasing novel views from mainly low-resolution inputs and outperform baselines requiring LR-HR training pairs.
- The framework adds a refinement network that blends details from relevant patches in an HR reference using available depth maps.
2 RELATED WORK
Prior novel-view and super-resolution methods use image warping, learned components, geometry, or reference images. NeRF-SR instead combines 3D scene geometry with reference-based detail transfer to serve all novel views.
- Image-based rendering warps and blends source frames but depends heavily on dense observations and accurate proxy geometry.
- Learning-based methods predict blending weights or view-dependent effects, while single-image approaches often require substantial training data.
- Geometry-based methods reconstruct 3D models or use representations such as meshes, multiplane images, voxel grids, and depth.
- NeRF represents static scenes with implicit neural functions and differentiable rendering to synthesize novel views from posed images.
- Reference-based super-resolution matches HR references to LR inputs, whereas NeRF-SR uses 3D geometry to bring details to all novel views.
3 BACKGROUND
NeRF encodes a scene as a continuous radiance field that maps 3D positions and viewing directions to color and density. It renders rays through volume integration and trains by matching predicted and ground-truth pixel colors.
- NeRF encodes a 3D scene with an MLP that predicts view-dependent radiance and position-dependent volume density.
- The model represents each camera ray as originating at the camera center and traveling along its viewing direction.
- Accumulated transmittance represents the probability that a ray travels between sampled depths without hitting a particle.
- NeRF is trained by minimizing mean-squared error between rendered pixel colors and corresponding ground-truth colors.
- Volume-rendering integrals are approximated numerically by sampling finite points along rays and using coarse-to-fine MLP queries.
4 APPROACH
NeRF-SR addresses high-resolution rendering from mostly low-resolution inputs by combining sub-pixel supersampling with depth-guided patch refinement. Supersampling improves multi-view constraints, while refinement adds details from high-resolution references while preserving synthesized structure.
- Motivation: NeRF struggles to generalize beyond training-image resolution, producing blurry high-resolution views from low-resolution inputs.The training–testing sampling gap arises because higher-resolution rendering queries ray directions and smaller 3D regions that were not observed during training.
- 4.1 Supersampling: Supersampling splits each training pixel into an s×s sub-pixel grid and samples one ray per sub-pixel instead of a single ray.At inference, these sub-pixels are directly rendered and organized into an sH × sW image, eliminating the sampling gap between training and testing.
- 4.1 Supersampling: Supersampling exploits cross-view consistency at sub-pixel level by corresponding and constraining more 3D points than vanilla NeRF.The strategy improves visual quality, but insufficient sub-pixel correspondence can leave high-resolution synthesis without enough detail.
- 4.1 Supersampling: NeRF-SR averages sub-pixel radiances before comparing them with the available low-resolution pixel color for supervision.This uses the average-kernel view of low-resolution images as downsampled high-resolution images and avoids complex assumptions about degradation.
- 4.2 Patch-Based Refinement: The patch-based refinement network blends synthesized patches with depth-associated high-resolution reference patches to recover local details.Depth and camera parameters backproject synthesized-patch pixels into the reference image; multiple patches address occlusion and depth-estimation inaccuracies.
- 4.2 Patch-Based Refinement: A U-Net-based encoder–decoder separately encodes synthesized and reference patches, then combines their features to produce refined patches.Training combines reconstruction and perceptual losses, while perspective transformations and nearby reference patches improve robustness to testing-time viewpoint differences.
5 EXPERIMENTS
Experiments evaluate NeRF-SR on synthetic Blender and real-world LLFF scenes using PSNR, SSIM, and LPIPS, comparing supersampling and refinement with several baselines. NeRF-SR recovers details from low-resolution inputs, outperforms competing methods, and benefits substantially from refinement at larger upscale factors.
- Experimental Setup: NeRF-SR is evaluated on Blender and LLFF using PSNR, SSIM, and LPIPS for 2× and 4× upscaling.Blender uses 100 × 100 and 200 × 200 inputs; LLFF uses 504 × 378 inputs.
- Baselines: The comparison includes Vanilla NeRF, bicubic upsampling, and learning-based single-image super-resolution baselines, including LIIF and SwinIR.LIIF and SwinIR rely on externally trained LR-HR pairs, unlike the proposed approach.
- Blender Results: NeRF-SR outperforms all baselines across Blender scenarios, while NeRF-LIIF and NeRF-Swin obtain strong LPIPS but lower PSNR and SSIM than NeRF.The authors attribute false predictions by NeRF-Swin partly to domain differences between its training data and the synthetic Blender dataset.
- LLFF Results: On LLFF, NeRF-SR fills details in complex real-world scenes and significantly outperforms other methods, whereas 2D baselines can produce oversmoothing or false color and geometry.The reported qualitative examples include leaves, textures, and fissures in scene objects.
- Degradation Sensitivity: Under symmetric average-kernel degradation, Blender results improve to PSNR/SSIM/LPIPS values of 30.94 dB/0.956/0.023 at 2× and 28.28 dB/0.925/0.061 at 4×.The results indicate sensitivity to the degradation process used for downscaling and upscaling.
- Refinement Network: Refinement improves metrics substantially at 4×, while at 2× it mainly improves LPIPS because supersampling already learns a reasonably detailed field.The refinement network adds texture details and edge sharpness; its extra inference time is about 1.3 seconds after roughly 48 seconds of NeRF rendering for a 1008 × 756 image.
6 LIMITATIONS AND CONCLUSION
NeRF-SR delivers photorealistic high-resolution novel views from mostly low-resolution inputs without external data, while sacrificing arbitrary-scale rendering and adding computation.
- Limitations: NeRF-SR does not retain NeRF’s arbitrary-scale rendering property.
- Limitations: The method introduces extra computation, although its training time does not exceed that of training an HR NeRF.
- Conclusion: NeRF-SR produces photorealistic high-resolution novel views with mostly low-resolution inputs and no external data.
- Conclusion: The method exploits 3D consistency through multi-view sub-pixel supersampling and depth-guided refinement from relevant patches on an HR reference image.