PaperlayerView exact revision on arXiv

Source-linked AI summary

S-NeRF: Neural Radiance Fields for Street Views

Ziyang Xie, Junge Zhang, Wenye Li, Feihu Zhang, Li Zhang

arXiv:2303.00749v1cs.CV

TL;DR

Conventional NeRFs are poorly matched to large-scale street views because self-driving cameras provide sparse, weakly overlapping observations. S-NeRF adapts scene and pose representations, exploits noisy sparse LiDAR with learned confidence, and jointly reconstructs backgrounds and moving vehicles, outperforming prior methods on driving datasets.

  • Problem

    Conventional NeRFs assume object-centric views with large overlaps, unlike sparse-overlap panoramic observations in unbounded street scenes.

  • Method

    S-NeRF jointly synthesizes large-scale backgrounds and moving vehicles using improved scene parameterization and camera poses, noisy sparse LiDAR supervision, and geometry- and reprojection-based confidence.

  • Results

    S-NeRF outperforms prior methods across static street-scene and moving-vehicle rendering, reducing mean-squared error by 7 ∼40% and improving moving-vehicle PSNR by 45%.

  • Takeaways & Limitations

    S-NeRF enables novel-view synthesis on standard self-driving datasets with fewer cameras and noisy sparse LiDAR, including moving vehicles that conventional NeRFs cannot reconstruct.

  • Takeaways & Limitations

    S-NeRF still produces depth artifacts in reflective windows and renders side views worse when fast motion leaves only a few observations.

Abstract

from arXiv · show

Neural Radiance Fields (NeRFs) aim to synthesize novel views of objects and scenes, given the object-centric camera views with large overlaps. However, we conjugate that this paradigm does not fit the nature of the street views that are collected by many self-driving cars from the large-scale unbounded scenes. Also, the onboard cameras perceive scenes without much overlapping. Thus, existing NeRFs often produce blurs, 'floaters' and other artifacts on street-view synthesis. In this paper, we propose a new street-view NeRF (S-NeRF) that considers novel view synthesis of both the large-scale background scenes and the foreground moving vehicles jointly. Specifically, we improve the scene parameterization function and the camera poses for learning better neural representations from street views. We also use the the noisy and sparse LiDAR points to boost the training and learn a robust geometry and reprojection based confidence to address the depth outliers. Moreover, we extend our S-NeRF for reconstructing moving vehicles that is impracticable for conventional NeRFs. Thorough experiments on the large-scale driving datasets (e.g., nuScenes and Waymo) demonstrate that our method beats the state-of-the-art rivals by reducing 7% to 40% of the mean-squared error in the street-view synthesis and a 45% PSNR gain for the moving vehicles rendering.

1 INTRODUCTION

S-NeRF addresses the mismatch between conventional NeRF assumptions and sparse-overlap, unbounded street-view data, jointly rendering backgrounds and moving vehicles. It improves representation and pose processing, uses noisy sparse LiDAR with confidence modeling, and outperforms prior methods on driving datasets.

  • Street-view data violate conventional NeRF assumptions because panoramic cameras observe unbounded scenes with little overlap, and many vehicles appear in only 2 ∼6 views.
  • Existing large-scale NeRFs still require intersected rays or specially designed multi-camera acquisition systems and can produce blurs, floaters, and other artifacts.
  • S-NeRF improves scene parameterization and camera poses, then uses noisy sparse LiDAR and a confidence metric based on geometry and reprojection consistency.
  • The method extends to high-quality reconstruction of foreground moving vehicles through a virtual camera transformation.
  • S-NeRF reduces mean-squared error by 7 ∼40% for static street scenes and improves moving-vehicle rendering by 45% PSNR and 18% structure similarity.These results are reported on standard nuScenes and Waymo driving datasets against large-scale NeRF and mesh-based baselines.

2 RELATED WORK

Related work spans traditional and learning-based reconstruction, large-scale NeRFs, and depth-supervised methods. S-NeRF targets the remaining setting of large-scale street views with noisy sparse LiDAR rather than expensive dense depth.

  • Prior learning-based reconstruction methods represent geometry with voxels, patches, or meshes, while NeRF methods learn implicit representations for novel-view synthesis.
  • Large-scale NeRFs address outdoor scenes through appearance embeddings, plenoptic sampling, conical-frustum encoding, or block combinations.
  • Depth-supervised NeRFs use sparse reconstructed depth, point clouds, depth priors, or confidence-guided ray sampling to improve training and rendering.
  • Urban-NeRF requires accurate dense LiDAR that is expensive to collect, whereas common self-driving datasets provide noisy sparse points that S-NeRF handles with learnable confidence.

3 NERF FOR STREET VIEWS

S-NeRF adapts NeRF to street views by handling sparse camera overlap, unbounded scenes, dynamic vehicles, and noisy LiDAR supervision. It combines pose processing, scene parameterization, virtual-camera transformation, depth propagation, and learnable confidence estimation.

  • Overview: S-NeRF reconstructs large-scale background scenes and foreground moving vehicles from standard self-driving datasets with sparse cameras and noisy LiDAR.The design targets nuScenes- and Waymo-style data without specially designed acquisition platforms.
  • Camera pose processing: Limited camera overlap makes conventional SfM-based pose estimation unreliable for street-view data, motivating separate pose processing for backgrounds and moving vehicles.Background poses use sensor-fusion SLAM and IMU with learned refinement, while vehicle poses use relative camera positions.
  • Camera pose processing: The moving-vehicle transformation treats each target vehicle as static and converts ego-camera motion into relative camera poses estimated from 3D object detections.This leaves only the camera moving, which is favorable for NeRF training.
  • Representation of street scenes: S-NeRF constrains long street-view sequences with an improved scene parameterization whose radius-controlled mapping preserves more detail for nearby objects.It also samples along rays in a logarithmic scale to obtain more points near the near plane.
  • Moving vehicles: For vehicles observed in only 2∼6 views, S-NeRF creates dense depth supervision from coarse meshes, sparse LiDAR, and differentiable rendering while masking backgrounds.This supplies additional supervision when view coverage is limited.
  • Depth supervision: S-NeRF propagates sparse LiDAR depths and learns confidence from reprojection and geometry consistency to suppress depth outliers during joint depth and color training.Reprojection confidence compares warped views using RGB, SSIM, and VGG similarities; confidence weights are jointly optimized.

4 EXPERIMENTS

Experiments evaluate S-NeRF on nuScenes and Waymo foreground vehicles and large-scale background scenes, including quantitative comparisons, visual renderings, and ablations.

  • Novel view rendering for foreground vehicles: 45–65% improvements in PSNR and SSIM over NeRF and GeoSim show that S-NeRF preserves finer details and more accurate static-vehicle shapes.The comparison uses only 4–7 image views.
  • Novel view rendering for foreground vehicles: 45% PSNR and 18% SSIM gains over GeoSim demonstrate stronger moving-vehicle rendering by S-NeRF.S-NeRF also supports broader novel-view ranges and lighting changes without heavily relying on accurate mesh priors.
  • Novel view rendering for background scenes: 40% higher PSNR, 26% higher SSIM, and 45% lower LPIPS than Mip-NeRF establish S-NeRF’s strongest background-scene comparison.Against Mip-NeRF 360, S-NeRF improves PSNR by 7.5%, SSIM by 4.5%, and LPIPS by 5%.
  • Ablation study: 56% lower mean-squared error and 42% higher structure similarity than Mip-NeRF are achieved by the RGB baseline in background rendering.Adding confidence improves PSNR and SSIM by about 3%, while smoothness suppresses floaters and depth outliers despite a slight PSNR drop.
  • Ablation study: Ablations show that depth supervision and confidence improve moving-vehicle PSNR by 18% and 8%, respectively, while smoothness mainly improves structure similarity and LPIPS.The study evaluates four foreground vehicles and two background scenes under different settings.

5 CONCLUSION, LIMITATIONS AND FUTURE WORK

The conclusion presents S-NeRF as a method for rendering large-scale scenes and moving vehicles, while acknowledging residual depth artifacts and demonstrating additional dynamic-object renderings.

  • Conclusion and future work: S-NeRF renders both large-scale scenes and foreground moving vehicles from self-driving datasets, with higher-quality RGB and depth renderings than prior NeRFs.The paper’s future direction is a larger city-level neural representation using block merging.
  • Limitations: Reflective windows remain a source of depth errors despite S-NeRF’s improvements over MipNeRF and other prior work.The authors identify this as an artifact that remains in depth rendering.
  • Confidence visualization: Confidence visualizations separate geometry consistency from photometric and feature consistency in the street-view rendering pipeline.Geometry confidence uses flow and depth, while projection confidence uses photometric and feature consistency.
  • Additional dynamic objects: A moving truck reconstructed from only four image views produces novel-view quality sufficient for driving simulation.Additional figures show novel-view and novel-pose rendering for dynamic persons and their fusion into rendered S-NeRF scenes.

B IMPLEMENTATION DETAILS

Implementation uses LiDAR-frame accumulation, depth completion, confidence handling, and separate training configurations for foreground vehicles and background street scenes.

  • Depth completion: Accumulating 5–10 neighboring LiDAR frames densifies nuScenes inputs because its 32-channel LiDAR is too sparse for direct depth completion.NLSPN performs well with denser 64-channel LiDAR but not with nuScenes’ sparse measurements.
  • Depth completion: Depth completion and reprojection still leave outliers from optical-flow errors, ill poses, moving objects, and occlusions, motivating learned confidence supervision.The completion algorithm can also introduce new outliers into the dense depth map.
  • Foreground vehicles: Foreground vehicles use four 256-unit MLP layers, 64 ray samples, 30k Adam iterations, and roughly two hours of training per vehicle on an RTX3090.Each vehicle uses around 2–8 training views and 1–3 held-out views.
  • Background street scenes: Background scenes use an eight-layer 1024-unit density MLP, a three-layer 128-unit color MLP, and 100k Adam iterations with batch size 2048.The confidence threshold is τ = 20% and the scene-parameterization radius is r = 3.

C EXPERIMENTS

The experiments evaluate S-NeRF against related methods on standard driving data, using matched encoding and training settings while reporting results for street-view rendering and foreground vehicles.

  • S-NeRF is quantitatively compared with state-of-the-art methods on two Waymo scenes, with average PSNR, SSIM, and LPIPS reported.
  • Foreground-car novel-view synthesis is evaluated against NeRF and GeoSim baselines using PSNR, SSIM, and LPIPS.
  • C.1 PARAMETERS AND EFFICIENCY: All methods use the same MLP encoding settings, and S-NeRF has 8.76M parameters versus 8.7–9.9M for comparable methods.
  • C.1 PARAMETERS AND EFFICIENCY: All methods are trained for 100k iterations, while one S-NeRF street scene takes about 17 hours to train.

C.2 WAYMO RESULTS

On Waymo street-view sequences, S-NeRF outperforms competing methods for both large-scale scene rendering and foreground vehicle synthesis, while producing more accurate depths.

  • 40% higher PSNR, 80% higher SSIM, and 44% lower LPIPS than Mip-NeRF are achieved on Waymo street views.
  • 6.8% higher PSNR and 5.2% lower LPIPS than Mip-NeRF 360 are achieved on the Waymo scenes.
  • S-NeRF exceeds Urban-NeRF by 32–50% across PSNR, SSIM, and LPIPS on Waymo street-view rendering.
  • 63% higher PSNR, 18% higher SSIM, and 51% lower LPIPS than NeRF are obtained for Waymo vehicle synthesis.
  • S-NeRF improves vehicle-synthesis PSNR by 62% and reduces LPIPS error by 16% relative to the mesh-based method of Chen et al. (2021b).

C.4 MORE ABLATION STUDY

The ablations indicate that S-NeRF is relatively insensitive to loss-balance weights, with specified ranges maintaining strong background and vehicle rendering.

  • The ablations evaluate different loss-balance weights on two nuScenes scenes and three foreground vehicles after 30k training iterations.
  • Changing λ1 or λ2 slightly reduces background PSNR and SSIM by 0.2–1%, indicating limited sensitivity to loss-balance weights.
  • The reasonable background training ranges are [0.1, 0.4] for λ1 and (0.005, 0.02) for λ2.

Confidence components

Confidence, depth quality, and supervision choices affect S-NeRF differently across background and vehicle rendering, while confidence-guided supervision remains robust to depth-completion quality.

  • Confidence components: Removing reprojection confidence lowers PSNR by 0.4%, while removing geometry confidence lowers PSNR and SSIM by about 0.7%.
  • Geometry confidence is insensitive to threshold τ, with [10%, 40%] identified as a reasonable range.
  • Confidence components: For foreground vehicles, RGB-only or SSIM-only confidence raises PSNR by 1–2% but worsens LPIPS by 6.6–11%.
  • Using sparse depth alone improves PSNR by 5% over RGB-only S-NeRF but is 2% worse in PSNR and 8% worse in LPIPS than the default dense-depth confidence setting.
  • Replacing NLSPN with Ku et al. depth completion changes rendering quality only from 24.41 to 24.40, showing limited dependence on NLSPN quality.

Scene and ray parameterization

S-NeRF uses scene parameterization, depth supervision, pose refinement, and confidence modeling to handle noisy sparse depth and limited overlap in street-view data. Its design also considers efficiency and robustness to input quality.

  • Scene parameterization: The scene-parameterization radius controls distance mapping so close objects retain more detail while the large-scale scene remains bounded.The paper studies this radius through an ablation of the scene parameterization function.
  • Pose refinement: Pose refinement keeps rendering quality from being strongly affected by 3D detection quality across boxes with mIoU 0.79, 0.67, and 0.55.Depth supervision and visual multiview constraints guide refinement of inaccurate virtual-camera pose estimates.
  • Pose refinement: NeRF– pose refinement achieves the best quality among the tested refinements for self-driving data.The paper attributes this to directly learning translation and rotation shifts, which are easier to learn under depth supervision.
  • Depth supervision: S-NeRF remains robust when depth inputs contain Gaussian noise and outliers.The study measures noise strength with PSNR and error rates representing the number of introduced outliers.
  • Training and inference time: 30% faster training and 40% faster inference result when the MipNeRF-360 distillation strategy adds a light proposal MLP.The comparison uses the same other settings as the default configuration.

D LIMITATIONS AND SOCIAL IMPACTS

The paper reports remaining quality limitations for side views and identifies potential misuse risks from realistic street-view and vehicle synthesis. Its visual comparisons cover foreground cars, state-of-the-art baselines, and Waymo panorama views.

  • Failure cases: Side-view rendering is worse than front and back views because fewer images are captured when the ego vehicle moves quickly.Left and right views show 5–12% lower PSNR, SSIM, and LPIPS, with depth densification and semantic supervision proposed as future remedies.
  • Social impacts: Realistic street-view and vehicle synthesis could be misused to create fake photos or videos.The authors call for cautious use and consideration of potential negative social influence.
  • Foreground vehicle rendering: S-NeRF significantly reduces floats, blurs, and other artifacts in foreground-car rendering across four novel views for five cars.The comparison is against the NeRF baseline.
  • Visual comparisons: The paper compares S-NeRF with Mip-NeRF 360 and Urban-NeRF for street-view synthesis.The comparison is presented as a state-of-the-art visual evaluation.
Loading 2303.00749v1…