NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction

1. Introduction

• TSDF fusion reconstructs geometry efficiently but often relies on depth sensors and cannot render realistic images, while NeRF’s global MLPs struggle with large scenes because of limited capacity and slow per-scene optimization.The paper frames large-scale indoor reconstruction and rendering as important for applications including AR/VR, e-commerce, and robotics.Source: §1 · p. 1
• NeRFusion uses a generalizable network pretrained across scenes for direct inference, defining large-scale scenes as full-size indoor environments with multiple rooms, objects, and complex geometry and appearance.The paper illustrates reconstruction from a monocular ScanNet video.Source: §1 · pp. 1–2
• The global field is a sparse volume grid whose per-voxel features are trilinearly interpolated and decoded by an MLP into density and view-dependent radiance for differentiable rendering.Unlike prior similar representations that require slow per-scene optimization, NeRFusion trains across scenes and generalizes to unseen scenes.Source: §1 · p. 2
• For posed RGB sequences, deep MVS builds renderable local radiance fields from neighboring-frame features, then recurrent neural fusion adds and updates voxels to form a global field.The local fields use sparse 3D convolutions on a world-space cost volume; the end-to-end model handles arbitrary scene scales and numbers of input images.Source: §1 · p. 2
• Across datasets, NeRFusion outperforms prior methods including IBRNet [43]; direct inference can match optimized NeRF on large indoor scenes, and one-hour fine-tuning outperforms NeRF [27] and NVSF [24].The paper reports that fine-tuning boosts rendering quality to state of the art while requiring much less per-scene optimization time than those methods.Source: §1 · p. 2

Multi-view scene reconstruction

• Depth-based point-cloud and mesh methods can produce holes and blurry renderings, while learned opacity or signed-distance volumes focus on geometry rather than realistic appearance.Screen-space neural networks can mitigate geometry errors but require long per-scene optimization and may produce temporally unstable flicker; NeRFusion instead models neural radiance fields for faithful, photo-realistic views and generalization across scales.Source: §2 · pp. 2–3

Neural Radiance Fields

• NeRFusion directly infers a sparse radiance volume from a pretrained network, avoiding per-scene optimization and enabling faster large-scale reconstruction than NeRF and NSVF.The volume can be fine-tuned briefly to improve rendering quality; the paper reports better quality than NeRF and NSVF.Source: §2 · p. 3
• Unlike PixelNeRF [49], IBRNet [43], and MVSNeRF [7], NeRFusion reconstructs a 3D per-voxel neural volume and fuses local reconstructions from arbitrary numbers of images.The cited approaches are described as limited by dataset-specific training, blurry or flickering results from sparse view-dependent inputs, or reconstruction from a fixed number of nearby views; NeRFusion’s volume models geometry and appearance consistently for efficient large-scale rendering.Source: §2 · p. 3

3.1. Sparse Volumes for Radiance Fields

• A sparse neural volume stores per-voxel features, which an MLP maps at any 3D location to density and view-dependent radiance for differentiable ray-marched rendering.The same MLP is shared across local and global volumes, and neural volumes are predicted by networks trained across scenes rather than by per-scene optimization.Source: §3.1 · p. 3

3.2. Reconstructing Local Volumes

• For each frame, the local reconstruction network aggregates image features from neighboring views into a sparse volume in canonical world coordinates, aligned for global fusion.It encodes viewing directions and processes per-voxel mean and variance features, which combine appearance and correspondence cues and are invariant to the number and order of views.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Fusing Volumes for Global Reconstruction

• A GRU fusion network with sparse 3D CNNs incrementally combines each local volume with the prior global volume to build a global neural radiance field.The update is restricted to voxels covered by the incoming local volume; other global voxels remain unchanged. The gates adaptively incorporate information from the prior and current volumes, filling holes and refining features while preserving consistency.Source: §3.3 · p. 5

3.4. Training and optimization

• The full pipeline uses ground-truth rendering supervision without extra geometry supervision, and its reconstructed field can be fine-tuned per scene in less than 25k iterations and less than 1 hour.Local-volume pretraining initializes the shared radiance-field decoder before end-to-end training; fine-tuning optimizes voxel features and the decoder, with pruning and voxel subdivision every 10k iterations to increase spatial resolution.Source: §3.4 · pp. 5–6

Training datasets

• Training uses 100 ScanNet scenes [11], 88 DTU training scenes, and 1,023 Google Scanned Objects models.The training data spans large-scale indoor scenes and small objects.Source: §4 · p. 6

Training details

• For object-centric datasets, training samples 16 views per scene; for ScanNet, it samples 2%–5% of each sequence as key frames and uses all other frames for supervision.Each input frame uses K = 3 neighboring views, selected temporally for monocular ScanNet sequences and by spatial proximity and viewing direction for other datasets.Source: §4 · p. 6
• Training uses Adam with an initial learning rate of 0.003 on 2 NVIDIA 2080Ti GPUs for 3 days; ScanNet inference runs at 22 FPS and renders a 640 × 480 image in 38 seconds on average.Sparse volumes and networks are implemented with torchsparse [42].Source: §4 · p. 6

Results setup

• Comparisons follow prior experiment settings where possible; NSVF quantitative results are included only when previously reported, and MVSNeRF [7] is compared only on DTU because it requires three nearby input views.The authors report that NSVF was impractical to run with their available GPU resources, while MVSNeRF cannot support large-baseline rendering or arbitrary input-view counts.Source: §5 · p. 6

Large-scale scenes in ScanNet

• Direct inference beats IBRNet [43], and after 1 hour of fine-tuning NeRFusion achieves the best PSNR, SSIM, and LPIPS on ScanNet, despite longer per-scene optimization by other methods.The comparison uses the ScanNet testing scenes and settings described for NerfingMVS [45].Source: §5 · pp. 6–7, Table 1 · p. 7

NeRF Synthetic

• On NeRF Synthetic, direct and fine-tuned results beat IBRNet [43], while 1-hour fine-tuning is comparable to per-scene methods NeRF [28] and NVSF [24].The results concern the NeRF Synthetic 360° dataset under the evaluation setting used in [28].Source: Table 2 · p. 7, §5 · p. 7

Large-scale scenes in ScanNet

• On ScanNet, both settings look better than IBRNet [43]; recurrent cross-view fusion yields consistent, realistic outputs, and direct renderings are arguably comparable to NeRF [28].The model aggregates per-view information in a unified canonical-space representation; the paper describes direct renderings as having few noticeable artifacts.Source: §5 · p. 7, Figure 4 · p. 8

DTU

• On DTU, with 3 input views for direct inference and 16 additional views for fine-tuning, fine-tuned NeRFusion outperforms NeRF [28], PixelNeRF [49], IBRNet [43], and MVSNeRF [7].The table caption says fine-tuning results outperform other methods in all three metrics.Source: Table 3 · p. 7, §5 · p. 7

6. Limitations.

• A uniform grid may be inefficient for scenes with foreground objects and distant backgrounds, such as unbounded outdoor scenes [24].Potential remedies include per-view reconstruction in disparity space or spherical coordinates for distant regions [50].Source: §6 · p. 8
• Extreme camera poses without sufficient parallax can cause problems because the method relies on multiview correspondence.The passage notes that this limitation cannot be addressed by MVS-based techniques.Source: §6 · p. 8
• Uniform input-frame sampling suits ScanNet’s translational camera motion, but other motions may require view selection that accounts for relative camera poses.The current pipeline samples input frames uniformly.Source: §6 · p. 8

• NeRFusion achieves state-of-the-art rendering quality for large-scale indoor scenes from ScanNet while taking substantially less reconstruction time.Source: §7 · p. 8
• A recurrent neural network processes the input image sequence and incrementally reconstructs a global large-scale radiance field by reconstructing and fusing per-frame local radiance fields.Source: §7 · p. 8
• Unlike traditional TSDF-based reconstruction, NeRFusion represents scenes as volumetric radiance fields, enabling photo-realistic view synthesis.Source: §7 · p. 8

A. Additional Details

• The 2D encoder is a modified MnasNet pretrained on ImageNet [36] [41] following [40], paired with a 5-layer direction MLP and a 5-layer SparseConvNet J implemented with torchsparse [42].Each of Mz, Mr, and Mt has 3 SparseConv layers, and all networks use ReLU activations.Source: A. Additional Details · p. 11
• The 2D encoder outputs 64 channels; G outputs 16, all local and global volume features have 16 channels, and renderer inputs use positional encoding with maximum frequency L = 5.Source: A. Additional Details · p. 11
• Initial voxel sizes are 40mm for large-scale datasets such as ScanNet [11] and 4mm for object-centric datasets such as NeRF Synthetic [28] and Google Scanned Objects.Local feature-volume unprojection uses maximum depth dmax = 3m; the pruning threshold γ is 0.6 for all experiments, and coarse-to-fine fine-tuning uses nearest-neighbor interpolation.Source: A. Additional Details · p. 11

B. Additional Results

• Using more neighboring input views enables sharper rendered details that are very close to the reference.Figure 6 shows results from the pretrained model without further fine-tuning.Source: B. Additional Results · p. 11, Figure 6 · p. 11
• NeRFusion’s direct inference outperforms NeRF, and fine-tuning produces rendered images with significantly more detail.Figure 5 shows visual quality at different fine-tuning durations; PSNR values appear at the top right of each image.Source: B. Additional Results · p. 11
• On DTU depth reconstruction, NeRFusion achieves significantly more accurate geometry than PixelNeRF [49] and IBRNet [43].The evaluation uses 16 input views and compares depth on input and novel views, deriving depth from volume density by a weighted sum along marched rays.Source: Table 4 · p. 11, B. Additional Results · p. 11