StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Views

1 Introduction

• Existing street-view methods prioritize novel-view synthesis or rely heavily on dense LiDAR for reconstruction, leaving multi-view implicit surface reconstruction without LiDAR unaddressed..Source: §1 · p. 2
• Street-view reconstruction is challenging because scenes are unbounded and captured along non-object-centric, long and narrow camera trajectories.Source: §1 · p. 2
• StreetSurf delimits close-range and distant-view spaces with trajectory-aligned cuboids, assigning them separate cuboid NeuS and hyper-cuboid NeRF++ models.With optional sky masks, a directional MLP can model sky separately from the distant view.Source: §1 · p. 2
• Road-surface initialization pre-trains the SDF roughly aligned with the road, while entropy regularization encourages crisp disentanglement of the close-range and distant-view models.Enclosed initial shapes can hinder disentanglement because street-view geometry is usually open.Source: §1 · p. 2
• Monocular estimates guide the close-range model’s volume-rendered normals and depths to address geometry errors on textureless surfaces and under insufficient viewing-angle coverage.Source: §1 · p. 2
• StreetSurf combines occupancy-grid acceleration with multi-stage hierarchical sampling for fine-grained ray marching in long, narrow spaces with complex occlusions.CUDA pack-like operations support sampling granularity without sacrificing efficiency.Source: §1 · pp. 2–3
• StreetSurf reports state-of-the-art geometry and appearance quality on street views, with training taking one to two hours on a single RTX3090 GPU, without necessarily requiring LiDAR.The work also provides a benchmark on challenging real-world autonomous-driving datasets.Source: §1 · p. 3

NeRF on street views

• Street-view NeRF work addresses large scenes with local blocks or scalable representations, while autonomous-driving-dataset studies often focus on novel-view synthesis [63].UrbanNeRF and FEGR reconstruct surfaces but rely on dense LiDAR, while StreetSurf targets static close-range backgrounds without necessarily requiring LiDAR.Source: §2 · pp. 3, 5

Spatial scene representations

• Scene representations include MLPs, grids, voxels, hash-indexing, and specialized voxel forms; StreetSurf uses cuboid and hyper-cuboid hash-grids for close-range and distant-view spaces, respectively.Rather than full-space warping [29], it delimits trajectory-aligned, long and narrow cuboid spaces.Source: §2 · p. 5

Multi-view implicit surface reconstruction

• Implicit-surface reconstruction methods use SDFs and volume rendering, with later work exploring local grids, sparse voxels, MLP blocks, and geometric priors [56].Prior outdoor extensions still use object-centric views; StreetSurf addresses street views.Source: §2 · p. 5

3.1 Multi-shell networks

• StreetSurf assigns close-range, distant-view, and sky regions to separate models and jointly renders their contributions along each ray.The close-range model uses a cuboid NeuS representation, the distant-view model a hyper-cuboid NeRF++ representation, and the optional sky model a directional MLP.Source: §3.1 · pp. 5–6

3.2 Cuboid space, cuboid hash-grids and hyper-cuboid distant-view models

• Trajectory-aligned cuboid boundaries and cuboid or hyper-cuboid hash-grids adapt scene capacity to street geometry and viewing distance.The close-range grid stretches voxels proportionally to the AABB dimensions; the distant-view model samples cuboid shells with inverse cuboid warping.Source: §3.2 · pp. 6–7, Figure 5 · p. 7

3.3.1 Disentanglement of close-range, distant-view and sky

• Road-surface initialization pre-trains the close-range SDF with a zero-level set aligned roughly to the road, estimated from ego tracks and ego height.This replaces enclosed initial shapes, which can hinder model disentanglement or sky-mask convergence in street scenes.Source: §3.3.1 · pp. 7–8
• Entropy regularization encourages opaque close-range occupancy, while optional sky-mask supervision distinguishes sky from the close-range and distant-view rendering.Sky masks can be extracted with SegFormer when annotations are unavailable; the design also supports omitting both masks and the sky model.Source: §3.3.1 · p. 8

3.3.2 All supervisions

• L1 photometric loss matches rendered and ground-truth pixel colors, while geometry supervision uses LiDAR depths or monocular normals and depths when LiDAR is absent.The monocular priors are estimated with Omnidata; L1 is reported to reduce reconstruction noise and benefit pose refinement, with a small appearance-performance trade-off against MSE and Huber losses.Source: §3.3.2 · p. 8
• Eikonal regularization constrains the close-range SDF gradients, and sparsity regularization penalizes free geometry in unobserved regions.The paper also includes entropy regularization in its combined objective.Source: §3.3.2 · pp. 8–9

4.1 Implementation details

• The proposed ray marcher applies multi-stage hierarchical sampling to occupancy-grid samples for the close-range model, with occupancy periodically updated from its SDF network.The approach does not require prior scene reconstruction with COLMAP or LiDAR point clouds to precompute occupied voxels; CUDA pack-like operations support efficiency.Source: §4.1 · p. 9

4.2 Street-view reconstruction

• The benchmark uses 32 manually selected Waymo sequences, with street-view images from three frontal cameras for fair comparison.Sequences were selected to minimize dynamic vehicles, ensure an unobstructed field of view, and avoid extremely adverse weather.Source: §4.2 · p. 10

4.3 Geometric metrics

• Geometry is evaluated using Chamfer Distance between rendered and original LiDAR point clouds and RMSE of rendered depth, despite possible noise or inaccuracy in LiDAR-derived information.The authors note that having a metric is preferable to having none when ground-truth street-view geometry is unavailable or difficult to obtain.Source: §4.3 · p. 10

4.4 Ablation

• Disentangling close-range and distant-view models separates distant landscapes and objects beyond the capture route, which are not distinguishable even with semantic segmentation.The reported disentanglement requires no supervision.Source: §4.4 · p. 13, Figure 10 · p. 13
• Road-surface initialization avoids the failed disentanglement that enclosed initial shapes may produce, where close-range geometry encases the ego trajectory.Figure 11 qualitatively compares the two initialization schemes.Source: §4.4 · p. 13, Figure 11 · p. 14
• Cuboid boundaries and hash-grids provide finer granularity with comparable performance while using half the parameter size and hash-grid levels of cubic ones.The close-range cuboid boundaries are automatically computed to align with long, narrow camera trajectories.Source: §4.4 · p. 14, Table 2 · p. 14
• PSNR improvement is greater for longer trajectories when switching from cubic boundaries and hash-grids to cuboid ones.The authors present this trend as validating cuboid space delimitation and models for street-view scenarios.Source: §4.4 · p. 14, Figure 12 · p. 14

5.1 LiDAR simulation

• Implicit surfaces provide definite intersection depth for sphere tracing, addressing volume rendering’s indefinite and often inaccurate depth; the paper qualitatively demonstrates simulated LiDAR point clouds.The simulation uses a large mapping parameter and evaluates its point clouds with a segmentation model trained on original Waymo data, using coordinate features because intensity is unavailable.Source: §5.1 · p. 15, Figure 13 · p. 15

5.2 High-resolution mesh and occupancy grid extraction

• The continuous reconstructed geometry can be extracted as high-resolution meshes or occupancy grids, including a demonstrated grid at 0.1 m resolution reconstructed without LiDAR.The occupancy-grid example uses Waymo Open Dataset data.Source: Figure 14 · p. 16, §5.2 · p. 16

5.3 Implicit-surface-guided ray tracing

• The reconstructed implicit surface provides definite geometry and high-fidelity normals for shading or relighting, with preliminary sphere-tracing and environment-map examples.These properties support graphics applications that require surface geometry and normals for further radiance factorization.Source: §5.3 · pp. 16–17, Figure 15 · p. 16

6 Conclusion, limitations and future work

• StreetSurf targets street-view implicit surface reconstruction, focusing on close-range-versus-distant-view disentanglement and geometric errors in non-object-centric, unbounded scenes.Source: §6 · p. 17
• StreetSurf currently handles only a small proportion of scenarios in autonomous-driving datasets, which still pose many challenges.Source: §6 · p. 17
• Dynamic foregrounds are not reconstructed in StreetSurf; future work will study multi-object reconstruction and rendering with StreetSurf as the background model.Source: §6 · p. 17

• The supplemental document is titled “Supplemental Material for StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Views.”Source: Supplemental Material for StreetSurf:… · p. 22
• Section A presents more street-view reconstruction results.Source: Supplemental Material for StreetSurf:… · p. 22
• Section B provides further implementation details over the optimization process.Source: Supplemental Material for StreetSurf:… · p. 22

A Additional results

• Experiments use three frontal cameras, with some Waymo sequences trimmed for limited ego-motion or high dynamic-object density.Sequence selection and trimming details are listed in Table 3.Source: A Additional results · p. 22, Table 3 · p. 22

A.1 Street-view reconstruction without LiDAR

• Without LiDAR, StreetSurf is compared with COLMAP and F2-NeRF using qualitative examples and quantitative results.The qualitative comparisons appear in Figures 16 and 17, and quantitative results in Table 4.Source: §A.1 · p. 22, Figure 16 · p. 23, Figure 17 · p. 24

A.2 Street-view reconstruction with LiDAR

• With LiDAR, comparisons include NeuS and NGP+L under auxiliary-four-LiDAR and all-five-LiDAR settings, evaluated using the top LiDAR.Qualitative examples appear in Figures 18 and 19, with quantitative comparisons in Table 5.Source: §A.2 · p. 22, Figure 18 · p. 25, Figure 19 · p. 26

B.1 Volume rendering of close-range implicit surfaces

• The close-range scene geometry is represented by a signed distance function (SDF).SDF values at sample points are used for volume rendering.Source: §B.1 · p. 27
• A learnable Sigmoid scaling parameter maps SDF values to color-contribution visibility weights while respecting occlusion.The mapping is described as ensuring unbiased calculation of color contribution.Source: §B.1 · p. 27
• Without LiDAR, the Sigmoid scale may not converge sufficiently, so it is manually increased linearly after the initial training stage, typically around 5k iterations.Source: §B.1 · p. 27

B.2 Optional sky masks

• Unlike prior outdoor-scene methods that typically require sky masks, StreetSurf works effectively with or without them.When masks are used, they supervise rendered occupancy; a distant-view model remains necessary for landscapes and objects beyond the close-range boundary.Source: §B.2 · p. 28
• Without sky masks, the distant-view model inclusively represents the sky, allowing close-range and distant-view disentanglement without extra supervision.Source: §B.2 · p. 29

B.3 Ray marching with multi-stage hierarchical sampling based on occupancy information

• Multi-stage hierarchical sampling is applied to occupancy-grid ray samples, with occupancy information updated in a bootstrap manner without precomputed geometry.The described alternatives that require geometry use a prior COLMAP reconstruction or LiDAR point clouds to precompute occupied voxels.Source: §B.3 · p. 29

B.4 Additional details

• Training uses up to 48 close-range and 64 distant-view meaningful samples per ray, with 12k iterations without LiDAR and 15k with LiDAR.Both settings take one or two hours on a single RTX3090 GPU.Source: §B.4 · p. 29
• An annealing binary mask gradually enables higher resolutions of the close-range hash-grid during the initial training phase.Source: §B.4 · p. 29

B.5 Geometry evaluation metrics

• Rendered results are evaluated against Waymo LiDAR data on provided beams, used as an indirect geometry reference when ground-truth scene geometry is unavailable.The authors describe the Waymo-perception LiDAR as a reliable evaluation basis despite possible sensor noise and inaccuracy.Source: §B.5 · p. 30
• RMSE is used as a depth metric by treating first-return LiDAR measurements as sparse depth observations.Source: §B.5 · p. 30
• Chamfer Distance compares rendered and original LiDAR point clouds because depth metrics are sensitive to beams near object boundaries.The calculation uses the closest 97% of ground-truth points and discards the remaining 3% to limit outlier effects.Source: §B.5 · p. 30