Source-linked AI summary
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes
Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, Ming-Hsuan Yang
TL;DR
Large-scale dynamic driving scenes remain difficult to reconstruct from sparse, high-speed sensing, especially with multiple moving objects, occlusion, and view or lighting variation. DrivingGaussian decomposes scenes into incrementally built static backgrounds and graph-modeled dynamic objects, using LiDAR to initialize Gaussian geometry. It achieves state-of-the-art performance on two autonomous-driving datasets while enabling high-quality surrounding-view synthesis and dynamic scene reconstruction.
Problem
Sparse views, high-speed motion, multiple objects, occlusion, and lighting variation make large-scale dynamic driving-scene reconstruction challenging for existing methods.
Method
DrivingGaussian incrementally reconstructs static backgrounds, models each moving object with a composite dynamic Gaussian graph, integrates objects into the scene, and uses LiDAR as a Gaussian geometry prior.
Results
DrivingGaussian achieves state-of-the-art performance on two autonomous-driving datasets and outperforms previous methods in challenging benchmarks.
Takeaways & Limitations
The framework supports high-quality surrounding-view synthesis, dynamic scene reconstruction, and corner-case simulation for autonomous-driving evaluation.
Takeaways & Limitations
The method primarily struggles with extremely small numerous objects and totally reflective materials such as glass, mirrors, and water surfaces.
Abstract
from arXiv · showhide
We present DrivingGaussian, an efficient and effective framework for surrounding dynamic autonomous driving scenes. For complex scenes with moving objects, we first sequentially and progressively model the static background of the entire scene with incremental static 3D Gaussians. We then leverage a composite dynamic Gaussian graph to handle multiple moving objects, individually reconstructing each object and restoring their accurate positions and occlusion relationships within the scene. We further use a LiDAR prior for Gaussian Splatting to reconstruct scenes with greater details and maintain panoramic consistency. DrivingGaussian outperforms existing methods in dynamic driving scene reconstruction and enables photorealistic surround-view synthesis with high-fidelity and multi-camera consistency. Our project page is at: https://github.com/VDIGPKU/DrivingGaussian.
1. Introduction
DrivingGaussian addresses the difficulty of reconstructing large-scale dynamic driving scenes from sparse, high-speed multi-sensor data by decomposing scenes into static backgrounds and moving objects. It combines incremental static reconstruction, dynamic Gaussian graphs, and LiDAR priors, achieving state-of-the-art results and supporting corner-case simulation.
- Large-scale dynamic scene modeling supports autonomous-driving understanding, perception, detection, planning, view synthesis, and safety-oriented corner-case generation.These applications include BEV perception, 3D detection, motion planning, controllable simulation, and lower-cost safety validation.
- Sparse vehicle-mounted sensing and high-speed ego and object motion make surrounding driving scenes difficult to reconstruct.Objects can move rapidly across cameras while static backgrounds and dynamic objects undergo substantial changes.
- NeRF methods require intensive computation, densely overlapping views, and consistent lighting, while limited network capacity can blur long-term multi-object scenes.These constraints are particularly problematic for outward-facing multi-camera driving setups.
- Existing 3D-GS methods struggle with large-scale dynamic driving scenes because fixed Gaussians and limited capacity cannot jointly represent static-dynamic regions and multiple fast-moving objects.Prior dynamic extensions focus on individual objects or construct Gaussians per timestamp rather than handling complex combined scenes.
- DrivingGaussian uses Composite Gaussian Splatting to reconstruct the static background incrementally, model each moving object with a dynamic Gaussian graph, and integrate objects into the scene.Global Gaussian-Splatting rendering preserves real-world occlusion relationships, while LiDAR initialization improves geometry and multi-view consistency.
- Extensive experiments report state-of-the-art performance on public autonomous-driving datasets and support dynamic scene construction and corner-case simulation.The framework is presented as the first large-scale dynamic driving representation based on Composite Gaussian Splatting.
2. Related Work
Prior neural and Gaussian scene representations face challenges from unbounded scale, sparse surrounding views, high-speed motion, multiple objects, and lighting variation. DrivingGaussian responds with Composite Gaussian Splatting and uses LiDAR as a geometric initialization prior rather than only as depth supervision.
- NeRF for Bounded and Unbounded Scenes: NeRF was extended from bounded object or room scenes toward unbounded and dynamic settings, but these approaches remain constrained by scene scale, view configuration, poses, or computational demands.Relevant extensions address training speed, pose optimization, scene editing, unbounded scenes, and dynamic representations.
- Dynamic Scene Reconstruction: Prior methods use LiDAR for camera-pose refinement, depth supervision, or static-dynamic decomposition, but do not fully exploit LiDAR as a geometric reconstruction prior.The cited approaches combine LiDAR with surrounding views, optical flow, or depth supervision.
- Limitations of Existing Methods: NeRF-based view synthesis quality deteriorates with multiple dynamic objects and lighting variations because of its dependency on ray sampling.Its use of LiDAR is also described as limited to auxiliary depth supervision.
- DrivingGaussian: DrivingGaussian addresses these limitations by incrementally reconstructing static backgrounds, integrating multiple dynamic objects through Gaussian graphs, and initializing Gaussians with LiDAR geometry.This uses LiDAR for a more accurate shape prior and comprehensive scene description rather than solely for image-depth supervision.
- 3D Gaussian Splatting: 3D Gaussian Splatting provides explicit representations, fast splat-based rendering, and efficient modeling of complex shapes, but standard 3D-GS is designed for static scenes.Dynamic Gaussian extensions include deformation networks and evolving Gaussians, with some requiring dense multi-view camera arrays.
- Challenges in Autonomous Driving Scenes: Surrounding autonomous-driving data are sparse, often only 2-4 views, while platform motion, fast objects, spatial changes, and occlusion create complex reconstruction conditions.These factors collectively challenge existing methods.
3. Method
DrivingGaussian models large-scale dynamic driving scenes by incrementally reconstructing static backgrounds, separately representing multiple moving objects, and integrating them through composite Gaussian rendering. LiDAR provides geometric initialization and supports accurate multi-camera alignment.
- Composite Gaussian Splatting: Composite Gaussian Splatting separates large-scale driving scenes into static backgrounds and dynamic objects for integrated rendering.The framework targets mixed scenes that challenge standard 3D-GS because fixed Gaussians and limited representation capacity cannot adequately model static backgrounds alongside multiple moving objects.
- Incremental Static 3D Gaussians: Incremental Static 3D Gaussians reconstruct the background sequentially by dividing depth into chronological bins and propagating position priors across overlapping regions.The first bin is initialized with LiDAR; later bins use previous Gaussians as position priors and are optimized with surrounding frames as supervision.
- Incremental Static 3D Gaussians: Weighted multi-camera color averaging addresses inconsistent sampling between front and rear cameras during Gaussian projection.The procedure uses view weights, differential splatting, and view-matrix alignment to optimize pixel colors across surrounding views.
- Composite Dynamic Gaussian Graph: The Composite Dynamic Gaussian Graph represents multiple moving objects individually using object identities, timestamps, bounding boxes, orientations, and object-to-world transforms.Dynamic objects are extracted from static backgrounds, and each object’s Gaussians are transformed into the world coordinate system before integration.
- Composite Dynamic Gaussian Graph: Optimized dynamic Gaussians are concatenated with the static field chronologically, while closer objects receive higher opacity under occlusion.This integration restores object positions and handles visibility relationships between multiple dynamic objects.
- LiDAR Prior with Surrounding Views: LiDAR priors initialize Gaussian geometry and support multi-camera registration by projecting points into surrounding images and refining them with multi-camera bundle adjustment.The method uses LiDAR-derived point positions as geometric priors rather than only as image-depth supervision.
4. Experiments
Experiments on nuScenes and KITTI-360 evaluate surrounding-view and monocular synthesis, initialization choices, module contributions, and controllable dynamic-scene simulation. DrivingGaussian consistently outperforms the compared methods while LiDAR initialization and the proposed modules improve geometric and reconstruction quality.
- Datasets: Experiments use six challenging nuScenes scenes with six surrounding cameras and optional LiDAR, plus single-camera KITTI-360 input.nuScenes provides multi-sensor driving data, while KITTI-360 tests monocular performance despite offering stereo images.
- Surrounding-view synthesis: DrivingGaussian significantly surpasses Instant-NGP, Mip-NeRF, and Mip-NeRF360 across all evaluated nuScenes surrounding-view synthesis metrics.The comparison includes NeRF-based methods and 3D-GS-based baselines.
- Comparisons: DrivingGaussian outperforms EmerNeRF, S-NeRF, SUDS, and 3D-GS across evaluated metrics, while avoiding scene-flow estimation against EmerNeRF.Qualitative results show photorealistic multi-camera renderings, view consistency, and fewer artifacts in challenging regions.
- Monocular synthesis: On KITTI-360, DrivingGaussian achieves optimal monocular driving-scene performance and surpasses existing methods by a large margin.Compared methods include NeRF, Mip-NeRF360, Point-NeRF, NSG, SUDS, and DNMP.
- Initialization ablation: LiDAR initialization preserves more accurate geometric structures than SfM initialization, although overly dense point clouds can interfere with Gaussian optimization.Downsampling loses some local geometric information, while dense points may store redundant features.
- Ablation and simulation: The Composite Dynamic Gaussian Graph is crucial for dynamic reconstruction, Incremental Static 3D Gaussians improve large-scale backgrounds, and the loss terms enhance details and reduce artifacts.The framework also inserts arbitrary dynamic objects while maintaining temporal coherence and inter-sensor consistency for corner-case simulation.
5. Conclusion
DrivingGaussian progressively models large-scale dynamic driving scenes with incremental static Gaussians and a composite dynamic Gaussian graph. It also uses LiDAR priors and supports corner-case simulation.
- DrivingGaussian progressively models the static background with incremental static 3D Gaussians.
- DrivingGaussian supports corner-case simulation involving sudden pedestrian and vehicle interactions.
- A composite dynamic Gaussian graph captures multiple moving objects individually.
- LiDAR priors improve geometric detail and multi-view consistency in the reconstructed scenes.
Supplementary Material
The nuScenes surrounding-view setup uses multiple cameras with limited spatial overlap and substantial temporal gaps. These properties characterize the sparse surround-view input.
- The nuScenes visualization covers surrounding multi-camera views.
- Neighboring camera views have small overlaps.
- The surrounding views have large gaps across time.
7. Implementation Details
The implementation uses synchronized multi-camera evaluation, static LiDAR preprocessing, adaptive filtering, bin overlap for background alignment, and explicit object handling for global rendering.
- Experimental Details: NuScenes evaluation uses synchronized images from 6 surrounding cameras and reports average PSNR, SSIM, and LPIPS across selected camera frames.Full-resolution 1600 × 900 images are rendered with 360-degree horizontal FOV per time-step.
- LiDAR Prior: Dynamic LiDAR points are removed using segmentation labels before multi-frame aggregation constructs a static scene prior.
- LiDAR Prior: The LiDAR point cloud is voxelized, depth-divided along camera rays, distance-weighted, and filtered for isolated outliers.
- Incremental Static 3D Gaussians: Small overlaps between neighboring bins align their static backgrounds.
- Global Rendering vis GS: Global rendering re-renders multiple dynamic objects and static backgrounds in a shared driving scene using fast Gaussian splatting.
- Dynamic Object Modeling: Object bounding boxes are projected onto surrounding-view images, while object identities are aligned across frames using labels.
8. Additional Results on nuScenes
On nuScenes, DrivingGaussian is compared with graph-based, dynamic-Gaussian, NeRF-based, and other state-of-the-art methods. The reported results emphasize quality, speed, and modeling of both static and dynamic content.
- Quantitative Comparison: DrivingGaussian boosts NSG performance across three metrics on nuScenes.
- Rendering Speed: DrivingGaussian balances rendering quality and speed, achieving higher results with faster rendering than Instant-NGP.
- Rendering Speed: DrivingGaussian achieves optimal quality with less rendering time than NeRF-based methods for unbounded large-scale scenes.
- Qualitative Comparison: Qualitatively, DrivingGaussian surpasses existing works in modeling both static backgrounds and dynamic objects.
- Qualitative Comparison: Figure 10 compares DrivingGaussian with NSG, EmerNeRF, and 3D-GS on nuScenes driving-scene reconstruction.
9. Additional Results on KITTI-360
DrivingGaussian shows strong reconstruction quality on KITTI-360, producing more realistic details than competing methods and outperforming existing state-of-the-art approaches in quantitative evaluation.
- DrivingGaussian achieves better performance than all other methods on the KITTI-360 leaderboard.The comparison uses sequential images from a single camera as input.
- DrivingGaussian produces more realistic reconstruction and finer geometry than DNMP on challenging traffic signs, vehicles, and people.
- Compared with 3D-GS, DrivingGaussian avoids artifacts, blurring, and unnatural colors in detailed regions.
- The KITTI-360 evaluation includes qualitative comparison against DNMP and 3D-GS for driving-scene reconstruction.
10. Additional Ablation Study and Analysis
Additional analyses attribute reconstruction improvements to LiDAR initialization, incremental static Gaussians, and the composite dynamic Gaussian graph, while bin density affects performance through overlap and scene scale.
- Ablation Study: The ablation study combines quantitative results from Table 4 with qualitative comparisons of individual modules.
- Initialization Methods: LiDAR-prior initialization produces higher-quality geometry and preserves small-object details better than random or SfM initialization.
- Incremental Static 3D Gaussians: Incremental Static 3D Gaussians improve static-background geometry and topological integrity in large-scale driving scenes.
- Composite Dynamic Gaussian Graph: The Composite Dynamic Gaussian Graph supports reconstruction of vehicles, bicycles, and pedestrians moving at arbitrary speeds.
- Density of Bins: Sparse bin distributions reduce performance because bins lack overlapping regions and may create overly large scene scales.The analysis divides a roughly 400-meter scene into 3–7 bins, each containing multiple surrounding-view frames.
11. Failure Cases
DrivingGaussian’s main failure cases involve extremely small, numerous objects and totally reflective materials such as glass mirrors and water surfaces.
- The primary limitation is modeling extremely small, numerous objects and materials with total reflection properties.The authors attribute distortions to limitations in representing densely reflected light and estimating density on fully reflective surfaces.