Source-linked AI summary
NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction
Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, Zexiang Xu
TL;DR
NeRFusion addresses the difficulty of reconstructing large indoor radiance fields when NeRF is slow and classical fusion lacks realistic rendering. It predicts local fields directly from image sequences, recurrently fuses them into a sparse global volume, and achieves strong quality with substantially faster reconstruction than NeRF and other methods.
Problem
NeRF has limited MLP capacity and slow per-scene optimization for large-scale indoor scenes, while TSDF fusion does not produce realistic renderings.
Method
NeRFusion predicts local radiance fields from RGB sequences and recurrently fuses them into a sparse neural volume for direct, cross-scene-generalized reconstruction.
Results
After one hour of per-scene fine-tuning, NeRFusion reaches state-of-the-art quality and outperforms NeRF and NVSF, while direct inference can match NeRF on large indoor scenes.
Takeaways & Limitations
NeRFusion improves the efficiency and scalability of radiance-field reconstruction toward more practical neural scene reconstruction and rendering.
Abstract
from arXiv · showhide
While NeRF has shown great success for neural reconstruction and rendering, its limited MLP capacity and long per-scene optimization times make it challenging to model large-scale indoor scenes. In contrast, classical 3D reconstruction methods can handle large-scale scenes but do not produce realistic renderings. We propose NeRFusion, a method that combines the advantages of NeRF and TSDF-based fusion techniques to achieve efficient large-scale reconstruction and photo-realistic rendering. We process the input image sequence to predict per-frame local radiance fields via direct network inference. These are then fused using a novel recurrent neural network that incrementally reconstructs a global, sparse scene representation in real-time at 22 fps. This global volume can be further fine-tuned to boost rendering quality. We demonstrate that NeRFusion achieves state-of-the-art quality on both large-scale indoor and small-scale object scenes, with substantially faster reconstruction than NeRF and other recent methods.
1 University of California, San Diego 2 Adobe Research
The paper is affiliated with the University of California, San Diego and Adobe Research; Figure 1 presents its fast large-scale radiance-field reconstruction pipeline.
- Figure 1 shows a recurrent network reconstructing a large indoor radiance field from an input image sequence for realistic novel-view rendering.The direct result is comparable to NeRF, while short fine-tuning further improves quality.
- Xiaoshuai’s research was partially conducted during an internship at Adobe Research.
1. Introduction
NeRFusion targets fast radiance-field reconstruction for full-size indoor scenes by combining neural radiance fields with TSDF-inspired sequential fusion. Its recurrent, cross-scene-trained framework reconstructs sparse neural volumes directly from RGB sequences and achieves competitive or state-of-the-art rendering with brief fine-tuning.
- 1. Introduction: NeRF struggles with large-scale indoor scenes because global MLP capacity is limited and per-scene optimization is impractically slow, while TSDF methods lack realistic image synthesis.
- 1. Introduction: NeRFusion incrementally reconstructs a large sparse radiance field from long RGB image sequences using recurrent neural modules trained across scenes for direct inference.
- 1. Introduction: The pipeline adapts TSDF fusion’s global-volume workflow but reconstructs sparse neural voxels for photo-realistic rendering instead of geometry alone.
- 1. Introduction: Local radiance fields are estimated per key frame and sequentially fused across frames by a recurrent neural fusion module.Local fields use deep multi-view stereo techniques and sparse 3D convolutions on world-space cost volumes.
- 1. Introduction: After one hour of per-scene fine-tuning, NeRFusion reaches state-of-the-art quality and outperforms NeRF and NVSF, while direct inference can match NeRF on large indoor scenes.The framework is trained end to end with rendering losses across ScanNet, DTU, and Google Scanned Object scenes.
2. Related Work
Related work spans multi-view geometry and neural radiance-field reconstruction. NeRFusion differs by modeling appearance with neural volumetric fields that generalize across scene scales and support efficient reconstruction.
- 2. Related Work: Multi-view reconstruction methods commonly use multi-view stereo or depth sensors to acquire scene geometry from multi-view data.
- 2. Related Work: Neural radiance-field methods model scene appearance volumetrically, with NeRF using a global MLP to regress density and view-dependent radiance.
- 2. Related Work: PixelNeRF and IBRNet extend radiance-field generalization, but PixelNeRF targets object rendering and dataset-specific training while IBRNet supports broader scene scales.
3. Method
NeRFusion reconstructs local sparse radiance fields from neighboring image views, then recurrently fuses them in canonical world space into a global sparse volume for realistic rendering.
- 3.1. Sparse Volumes for Radiance Fields: The sparse neural volume stores per-voxel features that an MLP converts into volume density and view-dependent radiance for differentiable novel-view rendering.Features are trilinearly interpolated at queried locations before the MLP decoder predicts rendering properties.
- 3.2. Reconstructing Local Volumes: The framework processes posed image sequences into per-frame local sparse volumes and recurrently fuses them into a global feature volume.A 2D CNN extracts image features, sparse 3D convolutions reconstruct local volumes, and a recurrent network builds the global representation.
- 3.2. Reconstructing Local Volumes: Each local volume uses neighboring views to aggregate image and viewing-direction features in canonical world space, improving correspondence and geometry reasoning.Per-voxel feature means and variances provide appearance fusion and correspondence cues while handling varying numbers of visible viewpoints.
- 3.3. Fusing Volumes for Global Reconstruction: The fusion module uses sparse 3D CNNs within a GRU to update the global volume from each incoming local reconstruction while preserving uncovered regions.Only voxels covered by the current local volume are updated; other global voxels remain unchanged, while features can be refined and holes filled.
- 3.4. Training and optimization: The resulting global sparse radiance field can be fine-tuned per scene to further improve rendering quality after direct network inference.The sparse volume features and decoder can be optimized after the pretrained network produces the initial reconstruction.
4. Implementation Details.
The model is trained on mixed indoor and object-centric datasets, using sampled key frames and neighboring views. During inference, it processes ScanNet sequences in real time at 22 FPS.
- Training uses 100 ScanNet scenes, 88 DTU scenes, and 1,023 Google Scanned Objects models.
- Each input frame uses three neighboring views, selected temporally for ScanNet and spatially by camera location and direction elsewhere.
- 22 FPS inference enables real-time processing of ScanNet sequences, while rendering a 640 × 480 image takes 38 seconds on average.
5. Results
Across ScanNet, NeRF Synthetic, and DTU, NeRFusion compares favorably with generalizable and per-scene NeRF methods. Direct inference is efficient and competitive, while short fine-tuning further improves quality.
- The evaluation compares direct inference and fine-tuned outputs against per-scene optimization and generalizable NeRF baselines under prior-paper settings.Baselines include NeRF, NVSF, NerfingMVS, PixelNeRF, IBRNet, and MVSNeRF.
- Large-scale scenes in ScanNet: Fine-tuned NeRFusion achieves the best PSNR, SSIM, and LPIPS on ScanNet, outperforming slower per-scene optimization methods.Direct inference already outperforms IBRNet, and one hour of fine-tuning further boosts quality.
- NeRF Synthetic: On NeRF Synthetic, one hour of fine-tuning reaches performance comparable to NeRF and NVSF while exceeding fine-tuned IBRNet.Without fine-tuning, NeRFusion is comparable to IBRNet.
- DTU: On DTU, NeRFusion generalizes with only three input views for direct inference, and fine-tuning outperforms other methods across all three metrics.The comparison follows MVSNeRF’s setting, with 16 additional views in the fine-tuning setting.
- Large-scale scenes in ScanNet: NeRFusion learns a unified 3D representation that aggregates information across all input views, improving rendering quality and cross-view consistency.This avoids the tearing artifacts caused by IBRNet’s limited neighboring-view aggregation.
A. Additional Details
The implementation uses sparse convolutional networks and feature volumes with dataset-dependent voxel sizes. Positional encoding and bounded view frustums support volume rendering and scene reconstruction.
- The 2D encoder is a modified ImageNet-pretrained MnasNet, while the direction encoder and sparse volume modules use multilayer perceptrons and sparse convolutions.The direction encoder has five MLP layers, and the sparse convolutional modules use five or three layers depending on the component.
- Feature volumes use 16 channels, with positional encoding up to maximum frequency L = 5 before volume rendering.
- Initial voxel size is 40 mm for large-scale datasets and 4 mm for object-centric datasets.
- Unprojected 2D features form local view frustums bounded by a maximum depth of dmax = 3 m.
B. Additional Results
NeRFusion’s additional results show that its learned fusion benefits from more input views, while fine-tuning adds rendering detail and its geometry reconstruction surpasses other neural rendering methods.
- Effect of input view numbers: More input views produce sharper rendered details that are very close to the reference.Figure 6 evaluates different neighboring-view counts using the pretrained model without fine-tuning.
- Effect of fine-tuning: Direct inference already outperforms NeRF, while fine-tuning produces significantly more rendering details.Figure 5 compares multiple fine-tuning durations and reports PSNR values on the rendered images.
- Geometry reconstruction: NeRFusion significantly outperforms PixelNeRF and IBRNet in DTU depth reconstruction at both input and novel views.Table 4 reports depth values for input/novel views and high depth accuracy for NeRFusion.