PaperlayerView exact revision on arXiv

Source-linked AI summary

Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs

Haithem Turki, Deva Ramanan, Mahadev Satyanarayanan

arXiv:2112.10703v2cs.CVcs.GRcs.LG

TL;DR

Mega-NeRF addresses the difficulty of training and rendering NeRFs for massive urban captures, where images cover limited regions and model capacity becomes impractical. It uses spatially specialized submodules, visibility-based pixel partitioning, and temporal coherence, achieving faster training and near-interactive rendering while preserving reconstruction fidelity.

  • Problem

    Mega-NeRF addresses the challenge of scaling NeRFs from single-object scenes to massive urban environments with limited per-image visibility, large model capacities, and interactive rendering demands.

  • Method

    Mega-NeRF uses sparse spatially specialized NeRF submodules, geometric pixel partitioning for parallel training, appearance embeddings, and temporal coherence for rendering.

  • Results

    3x faster training and significantly improved reconstruction quality are reported, while temporal-coherent rendering reaches 40x the speed of conventional NeRF within 0.8 db PSNR.

  • Takeaways & Limitations

    Mega-NeRF demonstrates a modular route to building NeRFs at previously unexplored scale, while interactive rendering at scale remains an open research question.

  • Takeaways & Limitations

    Interactive NeRF-based rendering at scale remains an open research question, and sparse voxel-octree preprocessing can require up to 55GB of memory at higher resolution.

Abstract

from arXiv · show

We use neural radiance fields (NeRFs) to build interactive 3D environments from large-scale visual captures spanning buildings or even multiple city blocks collected primarily from drones. In contrast to single object scenes (on which NeRFs are traditionally evaluated), our scale poses multiple challenges including (1) the need to model thousands of images with varying lighting conditions, each of which capture only a small subset of the scene, (2) prohibitively large model capacities that make it infeasible to train on a single GPU, and (3) significant challenges for fast rendering that would enable interactive fly-throughs. To address these challenges, we begin by analyzing visibility statistics for large-scale scenes, motivating a sparse network structure where parameters are specialized to different regions of the scene. We introduce a simple geometric clustering algorithm for data parallelism that partitions training images (or rather pixels) into different NeRF submodules that can be trained in parallel. We evaluate our approach on existing datasets (Quad 6k and UrbanScene3D) as well as against our own drone footage, improving training speed by 3x and PSNR by 12%. We also evaluate recent NeRF fast renderers on top of Mega-NeRF and introduce a novel method that exploits temporal coherence. Our technique achieves a 40x speedup over conventional NeRF rendering while remaining within 0.8 db in PSNR quality, exceeding the fidelity of existing fast renderers.

1. Introduction

Mega-NeRF targets urban-scale NeRF reconstruction and interactive visualization, addressing scene scale, limited per-image visibility, training capacity, and rendering demands through spatial modularity and parallel data partitioning.

  • Mega-NeRF: Interactive fly-throughs use spatially local queries and temporal coherence, caching occupancy and color values from nearby previous views.The renderer uses standard raycasting and point sampling while querying the encompassing submodule for each sampled point.
  • Scale: Existing NeRF evaluations largely focus on single-object scenes, whereas Mega-NeRF targets urban environments spanning 150,000 to over 1,300,000 m2 per scene.Tanks and Temples spans 463 m2 on average, while the target scenes are orders of magnitude larger.
  • Search and Rescue: Search-and-rescue motivates interactive 3D inspection because drone footage is commonly reconstructed into 2D birds-eye-view maps for post-hoc analysis.The paper envisions neural rendering for inspecting fields as if flying a drone, at greater detail than classic SfM.
  • Challenges: Large-scale captures pose time-sensitive training, massive model-capacity, and limited-visibility challenges because each image covers only a small part of the scene.The target datasets contain an order-of-magnitude more pixels and rays than prior work, while each image captures significantly less of the scene.
  • Mega-NeRF: Mega-NeRF decomposes scenes into spatial cells with separate NeRF submodules and partitions pixels according to the cells intersected by their camera rays.This spatially aware structure specializes parameters by region and supports parallel training of submodules.

2. Related work

Related work accelerates NeRF rendering through partitioning, caching, sampling, or learned priors, while Mega-NeRF applies spatial modularity primarily to scalable training and combines it with rendering methods for urban scenes.

  • Fast rendering: Plenoctree, SNeRG, and FastNeRF cache non-view-dependent outputs in structures such as sparse voxel octrees to enable interactive rendering.They then compute view-dependent radiance using a smaller MLP or spherical basis computation, but finite cache capacity limits quality.
  • Spatial partitioning: DeRF and KiloNeRF partition scenes into multiple smaller networks for faster inference, whereas Mega-NeRF uses spatial partitioning to enable data-parallel training.DeRF reports 3x rendering acceleration over NeRF, while both methods initialize from a single large pretrained network.
  • Fast rendering: DONeRF reduces samples per ray and concentrates them near the first surface, but relies on a separate depth-oracle network trained with ground-truth depth.Mega-NeRF similarly uses guided sampling but does not use DONeRF’s depth-oracle setup.
  • Unbounded scenes: NeRF++ and NeRF in the Wild address unbounded scenes or appearance variation through separate spatial regions, transient components, and per-image embeddings.Mega-NeRF adopts similar foreground/background partitioning and appearance embeddings for large outdoor scenes.
  • Urban-scale environments: URF, CityNeRF, and BlockNeRF target urban-scale environments using lidar, multiscale modeling, or fixed city-block cells, respectively.Mega-NeRF instead uses geometry-based visibility reasoning to partition training pixels, allowing far-away cameras to influence a spatial cell.
  • Training speed: PixelNeRF, IBRNet, GRF, and Tancik et al. accelerate training through predicted image features or learned initialization, approaches viewed as complementary to Mega-NeRF.These methods incorporate priors from similar datasets or use metalearning to find quickly convergent weights.

3. Approach

Mega-NeRF scales NeRF training and rendering by specializing model parameters spatially, partitioning pixels for parallel training, and exploiting temporal coherence during fly-throughs. Its architecture tightens ray bounds, while visibility-based assignment reduces each submodule’s training data and dynamic caching accelerates successive views.

  • 3.1. Model Architecture: Mega-NeRF assigns spatially specialized MLP submodules to scene regions and uses appearance embeddings to model lighting differences across images.Each query uses the weights nearest to its position, with opacity and color conditioned on position, direction, and image appearance.
  • 3.1. Model Architecture: A top-down 2D centroid grid provides simple, low-preprocessing spatial partitioning and efficient inference-time point assignment.Centroid heights are fixed because altitude variation is small relative to latitude and longitude differences.
  • 3.1. Model Architecture: Mega-NeRF separates foreground and background into distinct models, replacing NeRF++’s unit sphere with a tighter ellipsoid and altitude-constrained sampling.The bounds avoid querying underground regions while preserving the complementary outer-volume parameterization.
  • 3.2. Training: Each self-contained submodule trains independently, while visibility partitioning assigns pixels only to intersected cells and reduces each trainset by 10x.The initial assignment uses camera positions rather than scene geometry; later pruning can remove rays blocked by inferred occluders.
  • 3.3. Interactive Rendering: Full-scene cached octrees are costly at Mega-NeRF scale: an 8-level tree takes an hour and 1–12 GB, while one extra level takes 10 hours and 55 GB.These costs motivate dynamic, selectively refined caching rather than simply increasing fixed-octree resolution.
  • 3.3. Interactive Rendering: Mega-NeRF-Dynamic dynamically expands an octree during fly-throughs, reusing cached information from temporally coherent views for subsequent frames.Guided sampling then places a small number of samples near surfaces using weights stored in the refined octree, unlike NeRF’s two-stage sampling.

4. Experiments

The experiments evaluate Mega-NeRF’s reconstruction quality, scalable training, and interactive rendering against neural and non-neural baselines. Mega-NeRF achieves strong reconstructions and its dynamic renderer substantially reduces rendering latency while preserving PSNR.

  • Interactive rendering: Mega-NeRF-Dynamic uses cached geometry to skip empty space and sample sparsely near surfaces.This guided-sampling strategy contrasts with standard NeRF’s coarse uniform sampling followed by a second weight-guided pass.
  • Scalable training: Mega-NeRF captures high detail while avoiding artifacts observed in competing view-synthesis methods.Qualitative comparisons report the best reconstructions across the evaluated cases.
  • Scalable training: Mega-NeRF consistently outperforms NeRF, NeRF++, Stable View Synthesis, DeepView, and Multi-View Stereo after completion.The comparison uses the same evaluation framework and allows competing approaches to train beyond 24 hours.
  • Diagnostics: 10x rendering acceleration follows from Mega-NeRF’s spatially aware sparsity relative to fully connected MLPs.Ablations show that appearance embeddings and foreground/background decomposition also significantly affect performance.

5. Limitations

The supplied limitation material gives limited detail about scope and evaluation caveats. It indicates that interactive rendering at scale remains unresolved despite Mega-NeRF’s improvements.

  • Scope: The paper states that further limitations and societal impacts are discussed in supplementary material.No more specific limitation is supplied in the listed main-text passage.
  • Open challenges: Interactive NeRF-based rendering at scale remains an open research question.The conclusion frames the evaluated fast renderers as evidence that the problem is not fully solved.

6. Conclusion

Mega-NeRF scales NeRF construction through spatially aware modularity and parallel training, then explores temporal smoothness for reducing redundant computation during fly-throughs.

  • Conclusion: Mega-NeRF combines sparse spatially aware connections with geometric partitioning of training pixels into parallel NeRF submodules.The approach is presented as a modular method for building NeRFs at previously unexplored scale.
  • Conclusion: Temporal smoothness is proposed as a valuable first step toward minimizing redundant computation between views.This conclusion concerns the remaining challenge of interactive rendering at scale.

A. Data Pruning

Data pruning refines Mega-NeRF’s initial camera-based pixel assignments using coarse scene geometry and cached opacity information. Scaling experiments show a quality–speed trade-off between submodule capacity and channel count.

  • Data pruning: Initial pixel assignments use camera positions rather than scene geometry because geometry is unavailable at initialization.Later 3D knowledge can prune rays that cannot influence a submodule because of occlusion.
  • Data pruning: Early pruning occurs after 100,000 iterations using a fixed-resolution opacity structure instead of prohibitively querying NeRF depths.The method uses coarse scene understanding to remove irrelevant data-partition entries.
  • Scaling properties: Increasing either the number of submodules or channels improves rendering quality, but channel increases severely reduce training and rendering speed.The number of submodules has less impact on speed than the channel count.
  • Dynamic structure: Mega-NeRF-Dynamic bounds its cache by GPU memory and subdivides the 16,384 nodes receiving the most pixels.Observed tree depths are roughly 12 in practice, with stale entries pruned to reclaim space.

E. Limitations

Mega-NeRF remains constrained by camera-pose accuracy, dynamic objects, urban-scale partitioning trade-offs, rendering throughput, training time, and privacy considerations. The authors identify these as obstacles to practical deployment and areas for future work.

  • Pose accuracy: Pose accuracy is a major deployment limitation: raw drone GPS and IMU poses produced extremely blurry initial models.PixSFM poses performed best, with a PSNR gap of over 6 db relative to Pix4DMapper.
  • Dynamic objects: Dynamic scenes are not explicitly addressed, and scaling existing dynamic-NeRF approaches to large urban scenes may require additional work.
  • Scale: Spatial partitioning reduces PSNR by about 1 db relative to NeRF on Synthetic-NeRF scenes, despite ray bounds and appearance embeddings not harming quality.
  • Rendering speed: The renderer does not yet reach throughput needed for truly interactive applications, and further sampling-efficiency improvements remain challenging.
  • Training speed: NeRF training time remains a significant bottleneck for rapid deployment despite being several factors quicker than previous works.Conditional-prior methods could complement Mega-NeRF but require similar data for target scenes.
  • Privacy: Drone footage can inadvertently capture privacy-sensitive faces and license plates, while the effect of removing sensitive pixels on the resulting model remains unclear.The authors are investigating policy-guided pixel removal through denaturing.

H. Dataset statistics

The paper derives scene visibility statistics by measuring how image rays intersect surface voxels, and reports additional dataset-level statistics for view-synthesis collections.

  • Visibility: Visibility statistics are generated from trained scene NeRFs by projecting all camera rays and measuring each image’s intersected surface-voxel proportion.
  • Dataset statistics: Additional top-level statistics for commonly used view-synthesis datasets are provided in Table 7 to complement Table 1.

I. Additional results

The appendix adds interactive rendering results across all evaluated datasets and contextualizes those evaluations against commonly used view-synthesis datasets.

  • Additional results: Additional interactive rendering results are reported across all datasets in Figure 13.
Loading 2112.10703v2…