Source-linked AI summary
Grid-guided Neural Radiance Fields for Large Urban Scenes
Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, Dahua Lin
TL;DR
Large urban scenes challenge NeRFs with underfitting and challenge feature grids with noisy, weakly constrained solutions. The paper combines a compact multiresolution ground feature grid with a grid-guided NeRF branch in joint learning, yielding high-fidelity rendering while refining the grid representation. The method supports fast grid rendering or more detailed, smoother NeRF rendering, with slower joint-stage training and challenges for large high-resolution image sets.
Problem
Large-scale urban rendering lacks a representation that simultaneously avoids NeRF underfitting and feature-grid noise while remaining computationally efficient.
Method
The method combines a compact multiresolution ground feature-plane grid with a positional-encoding NeRF branch, using the grid to guide sampling and jointly refine both representations.
Results
The method achieves high visual-fidelity rendering on extremely large urban scenes, with sharper geometry and more delicate details than purely MLP-based baselines across reported metrics.
Takeaways & Limitations
The unified representation provides complementary operating points: faster grid rendering or slower NeRF rendering with more high-frequency detail and spatial smoothness.
Takeaways & Limitations
Joint learning has slow training, and shuffled-ray batch sampling is ineffective for many high-resolution images without distributed training.
Abstract
from arXiv · showhide
Purely MLP-based neural radiance fields (NeRF-based methods) often suffer from underfitting with blurred renderings on large-scale scenes due to limited model capacity. Recent approaches propose to geographically divide the scene and adopt multiple sub-NeRFs to model each region individually, leading to linear scale-up in training costs and the number of sub-NeRFs as the scene expands. An alternative solution is to use a feature grid representation, which is computationally efficient and can naturally scale to a large scene with increased grid resolutions. However, the feature grid tends to be less constrained and often reaches suboptimal solutions, producing noisy artifacts in renderings, especially in regions with complex geometry and texture. In this work, we present a new framework that realizes high-fidelity rendering on large urban scenes while being computationally efficient. We propose to use a compact multiresolution ground feature plane representation to coarsely capture the scene, and complement it with positional encoding inputs through another NeRF branch for rendering in a joint learning fashion. We show that such an integration can utilize the advantages of two alternative solutions: a light-weighted NeRF is sufficient, under the guidance of the feature grid representation, to render photorealistic novel views with fine details; and the jointly optimized ground feature planes, can meanwhile gain further refinements, forming a more accurate and compact feature space and output much more natural rendering results.
1. Introduction
Large urban scenes expose complementary weaknesses in MLP-based NeRFs and feature grids. The proposed two-branch model combines their global continuity, local fitting, compactness, and detail-recovery strengths through joint learning.
- 1. Introduction: MLP-based NeRFs underfit large, complex scenes, while geographically partitioned sub-NeRFs increase model count and capacity demands as scene scale grows.Feature-grid methods offer an alternative but can produce noisy artifacts because their features are less constrained.
- 1. Introduction: The framework integrates grid-based and NeRF-based representations so explicit local features complement globally shared MLP continuity and positional-encoding detail.The joint design uses each representation for the properties it models most effectively.
- 1. Introduction: A pre-trained feature grid guides NeRF sampling toward scene surfaces and supplies geometry and appearance features alongside positional encodings.This lets a lightweight NeRF focus on finer details within a compressed sampling space.
- 1. Introduction: Ground-plane factorization reduces feature-grid memory from O(N^3) to O(N^2) while explicitly encoding compact urban layouts.Users can choose the faster grid branch or the more detailed, smoother NeRF branch at a higher rendering cost.
2. Related Works and Background
Large-scale scene reconstruction traditionally combines geometric correspondence pipelines with newer learned rendering methods. NeRF represents scenes as density and color fields, while later work adds efficient data structures and other extensions.
- Large-scale Scene Reconstruction and Rendering: Classical large-scale reconstruction detects and matches image points, estimates camera poses and 3D points, then jointly optimizes them with bundle adjustment.These methods support free-viewpoint navigation but can leave artifacts or holes that limit rendering quality.
- Large-scale Scene Reconstruction and Rendering: NeRF models a 3D scene as density and color fields optimized from images through volume rendering, with later methods introducing efficient data structures for faster rendering.Related extensions also combine NeRF with generative or motion fields for 3D-aware image generation and dynamic representations.
3. Grid-guided Neural Radiance Fields
The method pre-trains a compact multiresolution ground feature grid, then jointly trains a grid branch and a grid-guided NeRF branch. The grid supplies scene structure and sampling guidance, while NeRF adds detail and regularizes the grid.
- Multi-resolution Feature Grid Pre-train: The ground feature representation factorizes 3D density and appearance grids into xy-plane features and shared z-axis vectors using channel-wise outer products.This plane-vector construction provides a compact representation tailored to urban layouts.
- Grid-guided Neural Radiance Fields: The framework first pre-trains a multiresolution feature grid, then uses finer grid-guided sampling and joint supervision in a two-branch renderer.The grid branch remains supervised alongside NeRF during joint training.
- Multi-resolution Feature Grid Pre-train: The grid branch predicts coarse pixel colors from sampled ray points with a shallow MLP and squared reconstruction loss.This pre-training stage constructs informative density and appearance planes before NeRF joint learning.
- Multi-resolution Feature Grid Pre-train: A multiresolution grid captures scene content at different granularities, matching the varied scales of objects and structures in urban environments.Features are concatenated across resolutions to represent local complexity.
- Grid-guided Neural Radiance Field: Grid features guide NeRF toward approximated scene surfaces and complement positional encodings, allowing efficient sampling while recovering high-frequency details.The NeRF branch predicts density and color from grid features, coordinate encodings, and view-direction encodings.
- Grid-guided Neural Radiance Field: Two-stage training is necessary because random grids are initially uninformative, grid-only pre-training is faster, and joint learning lets the grid gain refinement rather than remaining fixed.The grid branch’s reconstruction supervision encourages it to continue capturing information while NeRF focuses on missing details.
- Refined Grid Feature Planes from NeRF: Joint NeRF optimization improves grid features through point-wise guidance and global shared-MLP regularization, addressing noisy artifacts and weak spatial continuity in independently learned grids.The grid can otherwise require high resolution and substantial memory to capture within-voxel variations accurately.
4. Experiments
Experiments on three real-world urban scenes show that the joint grid–NeRF framework improves rendering quality while retaining a compact representation. Ablations attribute these gains to NeRF supervision, grid guidance, and sufficient renderer capacity.
- Results Analysis: The final model reaches photorealistic novel-view quality, whereas NeRF and Mega-NeRF are blurry and TensoRF is noisy with inaccurate shapes.The pre-trained grid improves over single high-resolution grids but remains suboptimal; the final joint model is preferred for sharper details and smoother video continuity.
- Results Analysis: The final model produces sharper geometry and finer details than NeRF and Mega-NeRF, with significant improvements across all reported metrics.Figure 3 shows the qualitative comparison, while Table 1 reports PSNR, LPIPS, and SSIM on three large urban scene datasets.
- Results Analysis: Joint learning refines feature planes by reducing noise, sharpening boundaries, and recovering more detailed geometry and appearance structure.Residential-scene visualizations show cleaner density and appearance features after NeRF-guided refinement.
- Results Analysis: The plane-vector representation uses 3e8 parameters versus TensoRF’s 4e8 while achieving similar PSNR: 20.915 versus 21.075.The learned ground plane is reported as sharper, more informative, and better aligned with the physical scene layout.
- Ablations: Single-resolution grids are inferior: low resolution causes blur, high resolution causes noise, and NeRF supervision improves both regimes.At high resolution, combining NeRF raises PSNR by ∼2db on the Campus scene.
- Ablations: Supplying grid features to NeRF improves PSNR by ∼1db, while tuning the feature grid increases the gain to ∼2.5db.These results measure the contribution of local grid features and their joint optimization with the NeRF branch.
5. Discussion and Conclusion
The framework integrates MLP-based NeRF with an explicit feature grid to encode local and global scene information for large urban rendering. It achieves high visual fidelity on extremely large urban scenes, while joint learning remains limited by slow training and inefficient shuffled-ray sampling without distributed training.
- Integrating MLP-based NeRF with an explicit feature grid enables local and global scene information to be encoded jointly for large urban rendering.
- The model achieves high visual fidelity rendering on extremely large-scale urban scenes relevant to real-world application scenarios.
- Joint learning retains slow NeRF-based training, while shuffled-ray batch sampling is highly ineffective for large collections of high-resolution images without distributed training.