Source-linked AI summary
Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, Sida Peng
TL;DR
Dynamic urban-scene modeling needs efficient reconstruction and real-time, high-quality rendering, but prior neural approaches are costly. Street Gaussians uses explicit Gaussian point clouds for backgrounds and vehicles, with optimized poses and time-varying appearance; it reports state-of-the-art rendering quality across Waymo and KITTI while rendering over 100 times faster than previous methods.
Problem
Dynamic urban-scene modeling requires efficient reconstruction and high-quality real-time rendering for applications including autonomous driving, while prior methods face high training and rendering costs.
Method
Street Gaussians explicitly represents static backgrounds and moving vehicles with separate 3D-Gaussian point clouds, optimized tracked poses, and 4D spherical harmonics for time-varying appearance.
Results
Across Waymo and KITTI, Street Gaussians achieves state-of-the-art rendering quality while rendering over 100 times faster than previous methods.
Takeaways & Limitations
The explicit representation supports scene editing and real-time rendering within half an hour of training, while using off-the-shelf tracker poses can achieve comparable performance to precise ground-truth poses.
Takeaways & Limitations
Street Gaussians is limited to rigid dynamic scenes, depends on off-the-shelf tracker recall, and still requires per-scene optimization.
Abstract
from arXiv · showhide
This paper aims to tackle the problem of modeling dynamic urban streets for autonomous driving scenes. Recent methods extend NeRF by incorporating tracked vehicle poses to animate vehicles, enabling photo-realistic view synthesis of dynamic urban street scenes. However, significant limitations are their slow training and rendering speed. We introduce Street Gaussians, a new explicit scene representation that tackles these limitations. Specifically, the dynamic urban scene is represented as a set of point clouds equipped with semantic logits and 3D Gaussians, each associated with either a foreground vehicle or the background. To model the dynamics of foreground object vehicles, each object point cloud is optimized with optimizable tracked poses, along with a 4D spherical harmonics model for the dynamic appearance. The explicit representation allows easy composition of object vehicles and background, which in turn allows for scene editing operations and rendering at 135 FPS (1066 $\times$ 1600 resolution) within half an hour of training. The proposed method is evaluated on multiple challenging benchmarks, including KITTI and Waymo Open datasets. Experiments show that the proposed method consistently outperforms state-of-the-art methods across all datasets. The code will be released to ensure reproducibility.
1 Introduction
Street Gaussians addresses the high training and rendering costs of dynamic urban-scene modeling with an explicit Gaussian-based representation. It reconstructs dynamic streets efficiently, achieves real-time rendering, and is evaluated across Waymo and KITTI.
- Dynamic urban street models support city simulation, autonomous driving, and gaming, requiring efficient reconstruction and high-quality real-time novel-view rendering.
- NeRF-based urban methods can require long training, while compositional neural representations for moving vehicles remain costly to train and slow to render.
- Street Gaussians decomposes scenes into static backgrounds and moving vehicles represented separately with 3D Gaussians and point clouds.
- Tracked vehicle poses are optimized for foreground geometry, while a 4D spherical harmonics model represents time-varying appearance.
- 135 FPS at 1066x1600 resolution is achieved after half an hour of training, with rendering quality evaluated on Waymo and KITTI.
- The work presents Street Gaussians as a contribution for efficiently reconstructing and rendering high-fidelity dynamic urban scenes.
2 Related work
Prior work spans neural and point-based representations for static, dynamic, and urban scenes. These approaches improve scene modeling but face scalability, density, memory, or real-time-rendering limitations.
- Neural scene representations use differentiable rendering to model complex scenes from images, with NeRF representing continuous volumes through MLP networks.
- Point-based rendering applies learned descriptors to point clouds with differentiable rasterization, but sparse regions can produce blurry results and dense inputs are required.
- 3D Gaussian Splatting represents scenes with anisotropic 3D Gaussians, while related work extends point-based rendering to dynamic scenes.
- Compositional neural models combine moving-object and background representations but suffer from high memory cost on large scenes and lack real-time rendering.
- Dynamic-scene methods introduce deformation fields, physical priors, or 4D parameterizations to extend 3D Gaussian models.
3 Method
Street Gaussians represents dynamic urban streets with separate explicit Gaussian point clouds for the background and moving vehicles. Optimizable vehicle poses, time-dependent appearance, and compositional rasterization support dynamic reconstruction, rendering, semantic outputs, and editing.
- Street Gaussians: Street Gaussians decomposes each dynamic street scene into separate point clouds for the static background and moving vehicles.Each point cloud is built with 3D Gaussians and supports independent object-background composition.
- Background model: Background Gaussians use positions, opacity, covariance, spherical harmonics appearance, and semantic logits to represent geometry, view-dependent color, and semantics.The covariance is parameterized by scale and rotation, while semantic logits encode one of M classes.
- Object model: Each foreground vehicle combines local Gaussian geometry and semantic logits with optimizable tracked poses that transform object points into world coordinates.The tracked poses are learned because off-the-shelf vehicle tracks can be noisy.
- Object model: The 4D spherical harmonics model encodes time-dependent vehicle appearance through Fourier coefficients, avoiding separate spherical harmonics parameters for every timestep.Without this model, environmental lighting interactions can produce significant rendering artifacts.
- Rendering: Street Gaussians renders by projecting all point clouds into 2D and alpha-blending their contributions, producing color, depth, opacity, and semantic outputs.Semantic maps are obtained by replacing rendered color with semantic logits in the alpha-blending equation.
- Training: Training combines color, depth, sky, optional semantic, and regularization losses to supervise reconstruction and improve scene decomposition.The semantic loss uses input 2D segmentation predictions when available.
4 Implementation details
Street Gaussians is trained with fixed optimization settings and adaptive Gaussian control. Initialization and pruning constrain the models using scene geometry, bounding boxes, and visibility.
- Optimization: Street Gaussians is trained for 30000 iterations with Adam on one RTX 4090 GPU.Translation and rotation pose updates use exponentially decayed learning rates.
- Gaussian control: Adaptive control fixes the background scale at 20 meters and determines each object scale from its bounding-box dimensions.Object Gaussians sampled outside their bounding boxes are pruned during optimization.
5 Experiments
Street Gaussians is evaluated on Waymo, KITTI, and VKITTI2 for rendering quality, speed, ablations, segmentation, decomposition, and editing. It achieves strong visual and quantitative results, while tracked-pose optimization, 4D spherical harmonics, and LiDAR improve reconstruction quality.
- Experimental setup: Experiments use Waymo Open, KITTI, and VKITTI2 benchmarks, including challenging scenes with moving objects, ego-car motion, and complex lighting.Waymo evaluation uses eight sequences of approximately 100 frames, with every fourth image held out for testing.
- Quantitative results: 12.1% higher PSNR and 13.9% higher PSNR* are achieved than the compared methods, while rendering is two magnitudes faster than NeRF-based methods.PSNR* evaluates pixels inside projected moving-object boxes.
- Rendering quality: Qualitative Waymo comparisons show sharper, higher-fidelity novel views than NSG, MARS, 3D GS, and EmerNeRF, which exhibit blur, distortion, or ghosting.The reported failures are especially apparent in dynamic regions and moving-object areas.
- Ablation studies: Optimizing tracked poses substantially improves object rendering, adding texture details and reducing artifacts compared with models without pose optimization.The improvement includes details such as vehicle rears and logos.
- Ablation studies: 4D spherical harmonics and LiDAR inputs improve rendering quality by refining appearance and recovering more accurate scene geometry, although the method remains strong without LiDAR.LiDAR reduces blurry artifacts in both background and moving objects.
- Additional tasks: The instance-aware representation supports vehicle rotation, translation, swapping, object decomposition, and semantic segmentation.Reported qualitative results include cleaner foreground decomposition and improved segmentation in ambiguous shadow regions.
6 Conclusion
The conclusion presents Street Gaussians as an explicit representation that separately models background and foreground vehicles with neural point clouds. This design supports scene editing and real-time rendering after brief training while achieving competitive performance.
- 6 Conclusion: Street Gaussians separately models background and foreground vehicles as neural point clouds for dynamic urban street scenes.The representation is explicit and supports composition of object vehicles with the background.
- 6 Conclusion: Scene editing and real-time rendering are enabled within half an hour of training by the explicit representation.The conclusion identifies compositing of object vehicles and background as the basis for these capabilities.
- 6 Conclusion: The proposed scene representation achieves comparable performance while supporting the stated editing and rendering capabilities.
A.1 Street Gaussians implementations.
The implementation constructs background and object point clouds from SfM and LiDAR data, then optimizes color, depth, sky, semantic, and regularization losses. It also specifies practical settings and baseline implementations for reproducible evaluation.
- Point cloud initialization: Background point clouds are initialized from SfM triangulation while moving-object regions are masked, and SfM and LiDAR points can be concatenated in world coordinates.The masking addresses moving objects’ violation of multi-view consistency.
- Loss functions: The total loss combines color, depth, sky, semantic, and regularization terms.The color term uses L1 and D-SSIM, while depth supervision compares rendered depth with LiDAR depth.
- Loss functions: The depth loss uses LiDAR measurements and backpropagates through 95% of pixels with the smallest depth errors to reduce noisy-observation effects.The method sets λ1 to 0.01.
- Loss functions: Sky supervision uses a Grounded SAM-generated mask, while semantic supervision applies per-pixel softmax cross-entropy to rendered semantic logits.The sky mask is obtained from Grounding Dino boxes prompted into SAM.
- Loss functions: An entropy regularizer is added after adaptive control to better distinguish foreground and background, with λ4 set to 0.1.The regularizer acts on accumulated alpha values of decomposed foreground objects.
- Hyperparameters and baselines: Implementation settings use five Fourier coefficients, spherical-harmonics degree 1, and 0.15 m LiDAR voxel size, alongside official baseline implementations and variants.The settings balance performance, storage cost, view-dependent modeling, and overfitting control.
A.3 Evaluations
The evaluation appendix clarifies how moving-object PSNR* is computed and shows the effect of regularization on foreground-background decomposition. Both procedures use object-aware masks or alpha values to target specific scene components.
- PSNR* evaluation: PSNR* is computed from MSE over pixels inside projected 3D tracked-box masks, with boxes expanded 1.5 times in length and width.The same dataset-provided object-tracklet masks are used for the method and baselines.
- Decomposition evaluation: The regularization loss improves decomposition by removing ghosty artifacts around vehicles.The term is an entropy loss on accumulated alpha values of decomposed foreground objects.
B Additional experiments
Additional studies examine pose optimization, point-cloud initialization, sky modeling, extrapolated views, and qualitative comparisons on KITTI. The results support the method’s rendering quality and the benefits of its design choices.
- Tracking pose optimization: Pose optimization improves outcomes even with an implicit representation, but a noticeable performance gap remains.The study reports PSNR* on two Waymo scenes; “opt.” denotes optimization.
- Point-cloud initialization: Combining LiDAR and SfM points leverages their complementary coverage, while omitting SfM points produces worse rendering than omitting LiDAR points.SfM points cover structures such as distant road signs and trees beyond LiDAR coverage.
- Sky modeling: Separate cubemap sky modeling recovers detail areas better and reduces foreground objects being obscured by sky-region Gaussians.The effect is reported in Figure 13 and Table 6.
- Extrapolation results: Under Waymo lane-change extrapolation, the method produces high-quality novel views despite viewpoints being far from the input sequence.These qualitative results are shown in Figure 14.
- KITTI qualitative results: KITTI qualitative comparisons evaluate Street Gaussians against NSG and MARS for novel view synthesis, and against NSG and Panoptic Neural Fields for decomposition.The comparisons are presented in Figures 15 and 16.
C Limitations
Street Gaussians is scoped to rigid dynamic scenes with moving vehicles and has additional dependencies on tracker recall and per-scene optimization. The supplied figures provide qualitative context but do not add further limitations.
- Scope: The method cannot reconstruct non-rigid dynamic objects such as walking pedestrians.Its supported setting is rigid dynamic scenes, including static streets with moving vehicles.
- Tracking dependency: Missed vehicles from off-the-shelf trackers cannot be compensated for by the pose optimization strategy.Continuous tracklets from methods such as 2D tracking are suggested as a mitigation.
- Optimization scope: Street Gaussians still requires per-scene optimization rather than feed-forward prediction of generalizable 3D Gaussians.Feed-forward prediction is identified as future work.