Source-linked AI summary
Block-NeRF: Scalable Large Scene Neural View Synthesis
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, Henrik Kretzschmar
TL;DR
Large-scale NeRF reconstruction must handle limited capacity, changing scene appearance, and costly whole-environment updates. Block-NeRF decomposes environments into independently trained, dynamically composited blocks with appearance, pose, exposure, visibility, and alignment adaptations, and demonstrates the approach on a 2.8M-image San Francisco neighborhood reconstruction.
Problem
NeRF methods do not naively scale to city-scale environments because of artifacts, limited model capacity, changing capture conditions, and rendering, memory, and update costs.
Method
Block-NeRF independently trains regional NeRFs, renders and composites only relevant blocks, and adapts them with appearance embeddings, pose refinement, exposure conditioning, visibility filtering, and appearance matching.
Results
2.8 million images were used to build an entire San Francisco neighborhood, forming the largest neural scene representation to date.
Takeaways & Limitations
Independent blocks allow rendering to scale to arbitrarily large environments and enable piecewise updates or new regions without retraining the entire environment.
Takeaways & Limitations
Dynamic objects, seasonal or moving vegetation, and unhandled temporal inconsistencies can produce artifacts or require manual retraining, limiting closed-loop simulation applicability.
Abstract
from arXiv · showhide
We present Block-NeRF, a variant of Neural Radiance Fields that can represent large-scale environments. Specifically, we demonstrate that when scaling NeRF to render city-scale scenes spanning multiple blocks, it is vital to decompose the scene into individually trained NeRFs. This decomposition decouples rendering time from scene size, enables rendering to scale to arbitrarily large environments, and allows per-block updates of the environment. We adopt several architectural changes to make NeRF robust to data captured over months under different environmental conditions. We add appearance embeddings, learned pose refinement, and controllable exposure to each individual NeRF, and introduce a procedure for aligning appearance between adjacent NeRFs so that they can be seamlessly combined. We build a grid of Block-NeRFs from 2.8 million images to create the largest neural scene representation to date, capable of rendering an entire neighborhood of San Francisco.
1. Introduction
Neural rendering methods have not naively scaled from small scenes to city-scale environments because of model-capacity, artifact, and data-collection challenges. Block-NeRF addresses these issues by independently training and dynamically combining regional NeRFs while adapting to changing appearance and pose.
- City-scale NeRF reconstruction is limited by model capacity, producing significant artifacts and low visual fidelity when applied to large environments.
- Large-scale reconstructions support mapping, localization, navigation, collision avoidance, and closed-loop robotic simulation.
- Training data across large environments varies in geometry and appearance because it is collected during different efforts, weather conditions, and times of day.
- A single enlarged Block-NeRF still has rendering, device-memory, and whole-environment retraining limitations.
- Block-NeRF divides environments into independently trained blocks, renders only nearby blocks, composites them geographically, and aligns their appearance embeddings.
2. Related Work
Prior reconstruction and view-synthesis pipelines face scaling, artifact, and data-collection limitations. Block-NeRF instead uses independently trained regional NeRFs and supports real driving data collected across multiple environmental conditions.
- Traditional large-scale reconstruction pipelines jointly optimize image features, camera poses, and 3D points, then require dense reconstruction and postprocessing.
- Geometry-based view synthesis reprojects pixels through point-cloud or mesh proxies, but limited texture and specular reflections can create artifacts or holes.
- NeRF unifies reconstruction and rendering with a neural volumetric representation, but its rendering compute scales poorly to large scenes.
- Prior partitioned-NeRF ensembles require joint training, whereas Block-NeRF trains regional networks independently and combines them at inference.
- Block-NeRF models camera data from multiple drives under varying weather and time-of-day conditions rather than relying on a single drive.
3. Background
NeRF represents scenes with coordinate-based multilayer perceptrons and differentiable volume rendering. Mip-NeRF reduces aliasing by modeling sampled ray frustums as Gaussian distributions and using integrated positional encoding.
- NeRF uses a differentiable rendering loss to optimize a coordinate-based neural scene representation from posed images and render unseen viewpoints.
- The NeRF representation uses one MLP for position-dependent density and features, followed by a color MLP conditioned on viewing direction.
- NeRF samples points along each camera ray, evaluates density and color, and repeatedly resamples using weights to concentrate samples in high-density regions.
- Sinusoidal positional encoding maps position and direction inputs to multiple frequency levels, with L specifying the number of levels.
- Mip-NeRF approximates conical frustums using Gaussian parameters and replaces positional encoding with its expectation over the Gaussian to reduce aliasing.
4. Method
Block-NeRF scales city-sized scene reconstruction by independently training spatial blocks and compositing only relevant blocks at inference. The method adds mechanisms for variable appearance, pose, exposure, visibility, and cross-block alignment.
- Block decomposition and rendering: Block-NeRF splits the environment into independently trained models that are dynamically selected and composited during inference.This enables parallel training, piecewise updates, and expansion without retraining the entire environment.
- Block decomposition and rendering: Blocks are placed to provide complete coverage, typically at intersections with overlapping street regions that facilitate alignment.Alternative placement heuristics are possible provided every location is covered by at least one Block-NeRF.
- Appearance and pose handling: Per-image appearance embeddings model varying weather and lighting and can interpolate between observed conditions.The embeddings are also optimized at test time to match the appearance of adjacent blocks.
- Appearance and pose handling: Learned, regularized translation and rotation offsets refine camera poses jointly with the NeRF.The offsets are learned per driving segment and strongly regularized early in training.
- Appearance and pose handling: Exposure conditioning lets the model compensate for exposure variation and enables human-interpretable exposure changes during inference.Exposure is encoded from shutter speed and analog gain using a four-level sinusoidal positional encoding.
- Visibility and compositing: A visibility network estimates whether sampled locations were observed by each block, supporting block filtering and appearance-matching location selection.It runs independently from the color and density networks, allowing efficient visibility computation.
- Visibility and compositing: For a target view, nearby blocks with sufficient mean visibility are rendered and blended in image space using inverse-distance weights.Typically one to three blocks remain after filtering, producing smooth transitions between renders.
- Visibility and compositing: Appearance matching freezes network weights and optimizes target-block appearance codes to reduce render differences between adjacent blocks.The procedure converges within 100 iterations and aligns global attributes such as time of day, color balance, and weather.
5. Results and Experiments
Experiments evaluate Block-NeRF’s datasets, component ablations, scalability with block count, and interpolation strategies. Results show that component choices and scene decomposition improve reconstruction quality while preserving scalable rendering, with interpolation methods trading compute, sharpness, and temporal consistency.
- 5.1. Datasets: 2,818,745 training images compose the Alamo Square dataset, divided into 35 Block-NeRFs across approximately 960 m × 570 m.The dataset contains 13.4 h of driving data from 1,330 collection runs.
- 5.3. Block-NeRF Size and Placement: Increasing the number of Block-NeRFs improves reconstruction metrics, including when the total number of weights is held constant.At inference, only nearby models are rendered, typically one to three NeRFs, reducing overall computational cost.
- 5.3. Block-NeRF Size and Placement: 54 m to 544 m Block-NeRF sizes are evaluated on Mission Bay with adjacent blocks overlapping by 50%.The experiments compare different scene granularities and block placements on held-out images spanning the trajectory.
- 5.2. Model Ablations: Appearance embeddings reduce cloudy geometry and exposure conditioning enables inference-time exposure control, while pose optimization sharpens reconstructions and removes ghosting.Removing exposure slightly decreases accuracy, whereas pose misalignment can produce blur and duplicated objects.
- 5.4. Interpolation Methods: 2D inverse distance weighting is selected for flythroughs because it produces temporally consistent results, whereas visibility-based weighting can be temporally inconsistent.Nearest-block rendering uses the least compute but creates harsh transitions; inverse distance weighting smooths them.
- 5.4. Interpolation Methods: Depth-based interpolation suffers from artifacts and temporal incoherence when predicted depth is incorrect.This method interpolates over projected 3D points predicted from expected Block-NeRF depth.
6. Limitations and Future Work
The method remains limited by transient and changing scene content, blur in distant objects, and expensive rendering. Future work includes dynamic-object modeling and acceleration techniques.
- 6. Limitations and Future Work: Unmasked transient objects can cause artifacts, while seasonal or moving vegetation produces blurred representations.Construction changes require manual retraining of affected blocks, and dynamic objects limit closed-loop robotics simulation.
- 6. Limitations and Future Work: Distant objects are reconstructed more blurily because unbounded volumetric representations do not sample them with the same density as nearby objects.The authors identify NeRF++ and Mip-NeRF 360 as potential ways to sharpen distant renderings.
- 6. Limitations and Future Work: NeRF rendering can require up to multiple seconds per image, limiting real-time applications.NeRF caching, sparse voxel grids, and faster NeRF-style training methods are suggested as possible remedies.
7. Conclusion
Block-NeRF represents large environments by combining independently optimized neural blocks, while architectural changes address transient objects and appearance variation. The paper demonstrates this approach at neighborhood scale using 2.8M images.
- Conclusion: 2.8M images were used to build an entire San Francisco neighborhood, forming the largest neural scene representation to date.The representation is split into multiple blocks that can be optimized independently.
- Conclusion: Each Block-NeRF incorporates appearance embeddings, learned pose refinement, and controllable exposure for changing capture conditions.The network also follows the mip-NeRF structure and uses learned pose corrections.
- Conclusion: Independently trained blocks support scalable representation of large environments and piecewise optimization.The supplied conclusion passage states the scale demonstration and independent block optimization, while the implementation passages specify per-block training resources and duration.
- Conclusion: A single Block-NeRF takes 9–24 hours to train, and rendering a 1200 × 900px image takes approximately 5.9 seconds.Training uses 32 TPU v3 cores, and multiple blocks can be processed in parallel during inference.
C. Block-NeRF Overlap Comparison
The overlap study finds that lower overlap improves image-quality metrics, likely because it reduces block size, while overlap remains important for temporally smooth interpolation. Image-space inverse distance weighting gives the most appealing videos.
- Block-NeRF Overlap Comparison: Reducing block overlap improves image-quality metrics, likely because the resulting blocks are smaller.Table 4 extends the eight-block comparison while changing spatial block size as overlap varies.
- Block-NeRF Overlap Comparison: Overlap remains important in practice because it helps avoid temporal artifacts when interpolating between Block-NeRFs.The overlap comparison distinguishes metric gains from the practical need for smooth transitions.
- Block-NeRF Interpolation: Simple image-space inverse distance weighting produces the most appealing videos because of temporal smoothness.The experiments use power 4 for Alamo Square renderings and power 1 for Mission Bay renderings.
- Block-NeRF Interpolation: Visibility-based interpolation produces sharper results but causes temporal inconsistencies, while depth-based interpolation introduces artifacts from noisy depth predictions.The comparison includes imagewise and pixelwise visibility as well as 3D inverse distance weighting.
E. Structure from Motion (COLMAP)
The COLMAP baseline reconstructs Mission Bay by splitting the dataset into overlapping blocks and fusing dense point clouds. Its rendered results remain sparse and fail on reflective surfaces and the sky.
- Structure from Motion: Mission Bay is reconstructed as 8 overlapping blocks with 97 m radius, based on camera positions.Each block has roughly 25% overlap with its adjacent block, and movable objects are masked during feature extraction.
- Structure from Motion: Dense multi-view stereo produces depth and normal maps that are fused into a dense scene point cloud.Poisson meshing was tested but failed to produce reasonably looking textured meshes because of challenging geometry and depth errors.
- Structure from Motion: Two rendering options are quantitatively compared using PSNR after discarding invisible pixels.This makes the point-cloud results comparable to the Block-NeRF setting.
- Structure from Motion: The COLMAP reconstruction is sparse and fails to represent reflective surfaces and the sky.Figure 8 presents qualitative comparisons for the two point-cloud rendering options.
F. Examples from our Datasets
The dataset examples show camera imagery for Mission Bay and paired camera images with segmentation masks for Alamo Square.
- Examples from our Datasets: Mission Bay is illustrated with camera images.Figure 10 presents the camera images from the Mission Bay dataset.
- Examples from our Datasets: Alamo Square is illustrated with camera images and corresponding segmentation masks.Figure 11 shows both modalities for the Alamo Square dataset.
G.1. Methodological
The method acknowledges that applying NeRF at unprecedented scale retains a heavy compute footprint, with energy use potentially worsening environmental damage. It identifies caching as a possible mitigation.
- Block-NeRF inherits NeRF’s heavy compute footprint when applied at unprecedented scale.
- At sufficient scale, compute energy use can increase carbon emissions and environmental damage.
- The paper points to future caching methods as a way to reduce compute demands and mitigate environmental damage.
G.2. Application
The method is applied to real city environments using datasets containing selected Mission Bay and Alamo Square imagery, including front-facing images with transient-object masks. The application also raises privacy concerns around repeated camera-based scans.
- The method is applied to real city environments while data collection is limited to public roads.
- Repeated and more regular scans of public environments could increase privacy concerns, despite blurring faces and license plates during collection.
- The Mission Bay Dataset and Alamo Square Dataset provide selected image collections for evaluating the method.
- The Alamo Square dataset includes front-facing images alongside transient-object masks predicted by a pretrained semantic segmentation model.