Source-linked AI summary
Urban Radiance Fields
Konstantinos Rematas, Andrew Liu, Pratul P. Srinivasan, Jonathan T. Barron, Andrea Tagliasacchi, Thomas Funkhouser, Vittorio Ferrari
TL;DR
Urban Radiance Fields addresses 3D reconstruction and novel view synthesis for large, complex urban scenes captured by mobile mapping platforms. By extending NeRF with lidar fusion, exposure compensation, and sky supervision, it achieves higher-quality novel views and 3D surface reconstructions on Street View data.
Problem
Neural scene representations have been evaluated mainly on synthetic or small controlled scenes, leaving large, complex, sparsely viewed urban outdoor mapping settings less addressed.
Method
The system extends NeRF with lidar-based losses, sky segmentation and a separate sky dome, and per-camera affine exposure compensation.
Results
+19% PSNR over [39] for synthesized novel views and +0.35 F-score over [30] for 3D surface reconstructions.
Takeaways & Limitations
The system demonstrates improved novel-view synthesis and 3D surface reconstruction from mobile scanning data in urban environments.
Takeaways & Limitations
The system assumes good camera parameters from structure-from-motion and has been demonstrated only on snippets requiring stitching for a continuous world model.
Abstract
from arXiv · showhide
The goal of this work is to perform 3D reconstruction and novel view synthesis from data captured by scanning platforms commonly deployed for world mapping in urban outdoor environments (e.g., Street View). Given a sequence of posed RGB images and lidar sweeps acquired by cameras and scanners moving through an outdoor scene, we produce a model from which 3D surfaces can be extracted and novel RGB images can be synthesized. Our approach extends Neural Radiance Fields, which has been demonstrated to synthesize realistic novel images for small scenes in controlled settings, with new methods for leveraging asynchronously captured lidar data, for addressing exposure variation between captured images, and for leveraging predicted image segmentations to supervise densities on rays pointing at the sky. Each of these three extensions provides significant performance improvements in experiments on Street View data. Our system produces state-of-the-art 3D surface reconstructions and synthesizes higher quality novel views in comparison to both traditional methods (e.g.~COLMAP) and recent neural representations (e.g.~Mip-NeRF).
1 Google Research 2University of Toronto · 1. Introduction
The paper adapts neural scene representations for large-scale, outdoor street-level mapping using panoramas, posed RGB images, and lidar to support novel view synthesis and 3D reconstruction. It extends NeRF with lidar fusion, sky supervision, and per-camera exposure compensation, improving both synthesized views and reconstructed surfaces on Street View data.
- 1. Introduction: The work targets 3D reconstruction and novel view synthesis for urban world mapping from data captured by platforms such as Street View.The setting uses a moving camera rig with a lidar sensor in large outdoor scenes under natural illumination.
- 1. Introduction: Street-level mapping challenges neural representations because scenes span hundreds of square meters and contain diverse buildings, vehicles, signs, trees, and vegetation.The outdoor scenes are substantially larger and more varied than the synthetic data or small real-scene regions commonly studied previously.
- 1. Introduction: Walking-pattern camera trajectories leave some scene regions observed by only a small number of cameras, creating sparse and uneven viewpoint coverage.This coverage differs from datasets designed to capture scenes from more focused or varied viewpoints.
- 1. Introduction: The method extends NeRF by fusing lidar with RGB signals and introducing lidar-based losses for accurate surfaces in buildings and volumetric vegetation.The lidar integration is designed to compensate for sparse viewpoints in large scenes while supporting both solid structures and volumetric formations.
- 1. Introduction: The model automatically segments sky pixels, uses a separate dome-like structure for sky-ray supervision, and estimates an affine color transformation per camera to compensate for exposure variation.These components address sky supervision and differing exposures across captured images.
- 1. Introduction: +19% PSNR over [39] and +0.35 F-score over [30] are reported for synthesized novel views and 3D surface reconstructions, respectively.Experiments on real-world Street View data found that the three NeRF extensions significantly improved both evaluation outcomes.
2. Related Works
Prior urban reconstruction methods rely on partial point clouds or explicit scene representations that make high-quality novel-view rendering difficult. NeRF extensions address representation and deployment limitations, but outdoor mapping systems generally do not combine lidar with 3D surface extraction.
- Novel View Synthesis: Urban reconstruction commonly uses raw SfM or lidar point clouds, while textured meshes and other explicit representations aggregate these data for rendering.Raw point clouds provide sampled, partial scene coverage, making high-quality novel views difficult to render.
- NeRF: NeRF represents scenes with an MLP mapping 3D position and viewing direction to density and radiance for volumetric novel-view synthesis.Training typically uses photometric consistency with posed RGB images.
- NeRF in vitro: Many NeRF extensions target aliasing, rendering speed, scene-scale, and surface extraction, but most are demonstrated only in synthetic or laboratory-controlled settings.These “in vitro” settings control lighting, viewpoint, and scene composition, limiting direct use in real-world applications with constrained sensor trajectories.
- NeRF for world mapping: Outdoor mapping work using NeRF remains limited: existing systems synthesize views from KITTI images or internet photos but neither leverages lidar nor extracts 3D surfaces.The passage identifies these omissions relative to outdoor mapping platforms where lidar is commonly available.
3. Data
The experiments use Trekker Street View data to reconstruct urban 3D surfaces and synthesize novel views at scale. The dataset combines posed fisheye-camera imagery with timestamped lidar sweeps, while semantic masking addresses prominent moving objects.
- Dataset: Experiments focus on Trekker data from Street View, acquired from Google with permission via personal communication.The broader goal is urban 3D surface reconstruction and novel-view synthesis from data collected for autonomous driving and world mapping.
- Image Data: Street View imagery comes from multiple calibrated fisheye cameras mounted on a trekker and captured at approximately 2Hz during motion.Each camera has estimated intrinsic parameters and a pose relative to the trekker.
- Image Data: Images are automatically posed in a global coordinate system using structure-from-motion and GPS, enabling camera rays with origin o and direction d for each pixel.The resulting rays support image-based scene reconstruction and rendering.
- Dynamic Objects: A pre-trained semantic segmentation model masks people in every image to reduce ghosting and blurring from moving objects.Urban scenes also contain other dynamic categories, including cars and bicyclists, whose positions change during capture.
- Lidar Data: VLP16 time-of-flight lidar sensors provide timestamped 3D line segments, each recording an origin and termination position.A lidar segment indicates that the emitted ray’s traversed space did not intersect an opaque surface at that timestamp.
4. Method
Section 4 defines Urban Radiance Fields as scene-level neural parameters jointly optimized with per-image exposure parameters, while extending rendering and training with sky-aware supervision and lidar constraints. The method uses restrictive affine exposure mappings, a learned environment map with segmentation-derived sky masks, and lidar depth and line-of-sight losses.
- URF optimization: URF jointly optimizes scene-level neural parameters θ and per-image exposure parameters {βi} from the scene’s image and lidar data.Training minimizes a combined loss over the image and lidar observations.
- Exposure variation: A decoded 3x3 affine radiance mapping models per-image white-balance and exposure variation while reducing entanglement with shared scene radiance parameters.Unlike per-image latent codes, the restrictive mapping is less able to compensate for non-exposure errors during joint optimization.
- Sky supervision: A coordinate-based spherical radiance map supplies direction-dependent background color for sky rays, while segmentation-derived sky masks encourage zero density along those rays.When a ray is marked as sky, the environment map explains its pixel color.
- Lidar supervision: Lidar supervision separates expected-depth matching from free-space constraints along each sensor-to-measurement line of sight.The expected rendered depth is trained to match the lidar depth, while the intervening line of sight is encouraged to contain no surface.
- Lidar supervision: A bounded truncated-Gaussian kernel around each lidar point makes the line-of-sight penalty numerically tractable and encourages density near the measured surface.The smooth kernel also guarantees continuity between the resulting loss intervals.
5. Experimental Evaluation
Experiments on Street View scenes evaluate novel-view synthesis and 3D reconstruction under held-out viewpoints and held-out buildings. The full model outperforms compared methods, with exposure and sky modeling improving renderings and lidar further improving geometric detail and accuracy.
- Dataset and evaluation settings: The dataset contains 10 urban scenes across six continents, with approximately 20 panoramas and 6 million lidar points per scene on average.Quantitative metrics average results over Taipei, Zurich, New York, and Rome.
- Dataset and evaluation settings: Evaluation uses held-out viewpoints and held-out buildings to test novel views and reconstruction of surfaces without lidar.Held-out viewpoints reserve 20% of simultaneous image captures, while held-out buildings remove lidar rays terminating on a selected building.
- Novel view synthesis: The full model outperforms all compared methods for novel view synthesis, including NeRF-W and lidar-based DS-NeRF, on sparse Street View data with exposure variation.Exposure and sky modeling improve the Mip-NeRF base model, and adding lidar improves renderings further; evaluation uses PSNR, SSIM, and LPIPS on held-out views.
- 3D reconstruction: Lidar supervision reconstructs finer depth detail and enables sharper images by improving surface placement and suppressing semi-transparent density floaters.The improvement is especially visible in distant areas and on difficult surfaces such as cars and window frames.
- 3D reconstruction: The method produces accurate, denser meshes than COLMAP and raw lidar, retaining fine details while avoiding limited-sampling artifacts.Meshes are reconstructed from dense image-based point clouds using Poisson Surface Reconstruction; images provide higher-resolution observations and broader coverage than raw lidar.
- Component analysis: Ablations show consistent 3D reconstruction gains from exposure compensation and sky modeling, while the near-surface loss provides the strongest lidar-related contribution.The best setup uses all proposed losses simultaneously; disabling the near-surface loss causes a considerable performance drop.
6. Conclusion
The paper presents a system for 3D reconstruction and novel view synthesis from mobile-scanning data in urban environments, extending Neural Radiance Fields with three targeted methods.
- Conclusion: The system reconstructs 3D scenes and synthesizes novel views from data captured by mobile scanning platforms in urban environments.It is designed for data combining posed RGB images with lidar sweeps acquired during platform motion.
- Conclusion: The approach leverages asynchronously captured lidar data, accounts for exposure differences between images, and uses predicted segmentations to supervise sky-ray densities.These are the three methodological extensions to Neural Radiance Fields described in the conclusion.
URF: Urban Radiance Fields · A. Additional Implementation Details
The supplemental material provides additional implementation details, ablation studies, and qualitative results for Urban Radiance Fields.
- URF: Urban Radiance Fields: The supplemental material includes additional implementation details.
- URF: Urban Radiance Fields: It reports ablation studies.
- URF: Urban Radiance Fields: It presents qualitative results.
A.1. Network architecture … B.1. Effect of margin ϵ
The system combines a NeRF-like radiance-field network with a separate sky network, using specified positional encodings and a fixed ray-sampling protocol. Margin ablations indicate that gradually decreasing the lidar-loss margin outperforms keeping it fixed.
- A.1. Network architecture: The radiance-field network uses fully connected layers of width 256 to map 3D position and viewing direction to RGB color and density.The architecture also includes a separate sky network that takes the direction of a ray pointing at the sky.
- A.1. Network architecture: A separate sky network processes the direction dx, dy, dz of rays pointing at the sky.
- A.2. Training Protocol: Training uses Adam with a two-stage learning-rate schedule: 50 warm-up epochs from 0.0005 to 0.005, followed by 500 main-stage epochs.Networks are trained separately for each baseline, model variant, and scene using TPU v2 hardware with 128 cores and TensorFlow 2.
- A.2. Training Protocol: Each ray batch contains 2048 rays per core, and training takes about one day per network.
- A.2. Training Protocol: Ray sampling uses evenly spaced log-scale intervals with 1024 samples per ray, without hierarchical sampling.Batches randomly sample rays from all images and lidar points.
- A.2. Training Protocol: The point location uses integrated positional encoding with L = 10 frequencies, while viewing directions use the original positional encoding with 4 frequencies.
- B. Additional Ablation Studies: Gradually decreasing the margin ϵ performs best in the Rome-scene ablation, whereas keeping it fixed produces lower performance.The margin controls the contribution of the lidar losses Lnear and Lempty.
- B.1. Effect of margin ϵ: The empty-space loss can reduce 3D reconstruction performance by strongly preferring empty space, while the complementary near-surface loss alleviates this effect.The margin ϵ is varied during training using fixed and stepwise strategies, among others.
B.2. Effect of exposure handling
The exposure-handling ablation compares an affine transformation model with directly providing exposure latent codes to the network for the Rome scene. The affine approach performs better across all 3D reconstruction metrics, while both approaches improve with lower-dimensional exposure codes.
- Exposure handling: Explicit color transformation for different exposures produces better reconstruction than directly providing the exposure code to the network.This is the central comparison reported in Table 5.
- Exposure handling: The affine transformation model outperforms direct exposure-code input on all three 3D reconstruction metrics for both tested code dimensions.The comparison uses D = 48, matching NeRF-W [39], and D = 4.
- Exposure handling: Both affine and direct approaches perform better when the exposure latent code has smaller dimensionality.Reducing dimensionality also reduces the latent code’s capacity.
C. Additional Qualitative Results
Additional qualitative results visualize colored meshes across multiple urban scenes and show that the extracted representation supports standard 3D editing and real-time browser rendering. The section also includes novel-view visualizations.
- Colored Meshes: Colored meshes are visualized for different scenes, with each vertex color estimated by querying the radiance field at that location.These meshes are shown in Figure 10.
- Applications: The extracted representation can be edited in Blender and rendered in real time in a browser using ThreeJS.Blender is used for the first and second Figure 10 columns, while ThreeJS is used for the third column.
- Scenes: The qualitative visualizations include scenes from Zurich, Taipei, Rome, and New York.
- Novel Views: Novel views are presented as an additional qualitative visualization.This result is shown in Figure 11.