PaperlayerView exact revision on arXiv

Source-linked AI summary

$\textit{S}^3$Gaussian: Self-Supervised Street Gaussians for Autonomous Driving

Nan Huang, Xiaobao Wei, Wenzhao Zheng, Pengju An, Ming Lu, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, Shanghang Zhang

arXiv:2405.20323v1cs.CVcs.AI

TL;DR

Street-scene reconstruction for closed-loop autonomous-driving evaluation needs controllable sensor views, while existing methods may require annotations or fail to decompose dynamic and static scenes automatically. S3Gaussian learns that decomposition without extra annotations using a spatial-temporal field network, and reports state-of-the-art rendering quality across tasks.

  • Problem

    Existing methods may require costly annotations or fail to divide dynamic and static scenes automatically, while closed-loop evaluation needs controllable sensor views from high-quality reconstructions.

    Source: §1 · p. 1, §2 · p. 3, §3 · p. 3

  • Method

    S3Gaussian uses a spatial-temporal field network with a multi-resolution Hexplane encoder and multi-head Gaussian decoder to decompose scenes without extra annotations.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    S3Gaussian achieves state-of-the-art rendering quality on scene reconstruction and novel view synthesis tasks, with superior scene decomposition ability.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The method performs high-fidelity, real-time neural rendering of dynamic urban street scenes, which is described as crucial for autonomous-driving simulation.

    Source: §2 · p. 3

    S3Gaussian has difficulty modeling high-speed objects, whose views are typically sparse.

    Source: §A.3 · p. 10

Abstract

from arXiv · show

Photorealistic 3D reconstruction of street scenes is a critical technique for developing real-world simulators for autonomous driving. Despite the efficacy of Neural Radiance Fields (NeRF) for driving scenes, 3D Gaussian Splatting (3DGS) emerges as a promising direction due to its faster speed and more explicit representation. However, most existing street 3DGS methods require tracked 3D vehicle bounding boxes to decompose the static and dynamic elements for effective reconstruction, limiting their applications for in-the-wild scenarios. To facilitate efficient 3D scene reconstruction without costly annotations, we propose a self-supervised street Gaussian ($\textit{S}^3$Gaussian) method to decompose dynamic and static elements from 4D consistency. We represent each scene with 3D Gaussians to preserve the explicitness and further accompany them with a spatial-temporal field network to compactly model the 4D dynamics. We conduct extensive experiments on the challenging Waymo-Open dataset to evaluate the effectiveness of our method. Our $\textit{S}^3$Gaussian demonstrates the ability to decompose static and dynamic scenes and achieves the best performance without using 3D annotations. Code is available at: https://github.com/nnanhuang/S3Gaussian/.

1 Introduction

High-quality street-scene reconstruction supports closed-loop autonomous-driving evaluation, which needs controllable sensor views. S3Gaussian addresses the annotation burden in dynamic street-scene modeling with self-supervised decomposition and a spatial-temporal field network.

1 Introduction

  • Closed-loop autonomous-driving evaluation requires controllable sensor views, motivating high-quality scene reconstruction.Source: §1 · p. 1
  • Tracked 3D bounding boxes are commonly used to separate static and dynamic elements, but the cost of 3D tracklet annotations limits modeling from in-the-wild data.Source: §1 · p. 1
  • The method uses a spatial-temporal field with a multi-resolution Hexplane encoder and compact multi-head Gaussian decoder to separate static and dynamic scene features and model Gaussian deformations.Dynamic-related features are stored in spatial-temporal planes, while static-related features remain in spatial-only planes.Source: §1 · p. 2
  • S3Gaussian reports state-of-the-art rendering quality for scene reconstruction and novel-view synthesis on challenging datasets, including NOTR and Waymo.Its decomposition uses no extra manually annotated data.Source: §1 · p. 2

2 Related Work

Prior work spans street-scene reconstruction and dynamic-scene modeling, but the cited approaches leave a need for automatic, self-supervised separation of static and dynamic street content without real annotations.

2 Related Work

  • Prior dynamic-scene methods use deformation networks, HexPlanes, semantic point clouds, or separate Gaussian models, while some require supervised scene-type classification.Source: §2 · p. 3
  • Street-scene work includes block-based modeling, LiDAR-informed NeRF, deformable mesh primitives, and methods tailored to urban surfaces or dynamic vehicles.Source: §2 · p. 3
  • The authors identify prolonged training, slow rendering, and lack of automatic static/dynamic division as shortcomings of prior approaches, motivating self-supervised decomposition without extra annotations.Source: §2 · p. 3

3 Proposed Approach

S3Gaussian learns static–dynamic street-scene decomposition without extra annotations by representing scenes with 3D Gaussians and a spatial-temporal field network. The approach lifts Gaussians to 4D, models their deformations efficiently, and optimizes the representation with self-supervised losses.

3 Proposed Approach

  • The proposed approach targets fully self-supervised decomposition of static and dynamic street-scene components without extra annotations.The representation combines 3D Gaussians with a spatial-temporal field network.Source: §3 · p. 3

3.1 4D Gaussian Representations

  • The 4D Gaussian representation uses canonical Gaussian geometry and predicts time-dependent displacement and semantic features before rendering deformed Gaussians.The renderer projects deformed Gaussians into 2D and computes pixel colors by α-blending ordered points.Source: §3.1 · pp. 3–4

3.2 Spatial-temporal Field Network

  • The spatial-temporal field uses multi-resolution Hexplane features and a compact decoder to model Gaussian motion and appearance over time.Spatial-only planes represent static content, while spatial-temporal planes capture dynamic variation; separate decoder heads predict position changes, semantic features, and SH appearance changes.Source: §3.2 · p. 5

3.3 Self-supervised Optimization

  • Self-supervised optimization combines image, depth, semantic-feature, similarity, total-variation, and position-regularization losses, with LiDAR point clouds used to initialize Gaussian positions.The method also uses static 3D Gaussian warm-up; regularization encourages offsets only when needed.Source: §3.3 · pp. 5–6, Figure 3 · p. 6

4 Experiments

The experiments compare S3Gaussian with state-of-the-art methods across driving-scene datasets and examine the effects of its components. The reported comparisons and ablations assess reconstruction quality, novel-view synthesis, and static/dynamic scene modeling.

4.1 Experimental Setup

  • Evaluation uses 64 NOTR static32 and dynamic32 scenes plus six complex, motion-heavy Waymo scenes, with NeRF- and 3DGS-based baselines.The NOTR scenes cover varied driving conditions; baseline comparisons include LiDAR initialization for 3DGS and depth regularization for 3DGS and MARS.Source: §4.1 · p. 6

4.2 Comparisons with the State-of-the-art

  • On Waymo-NOTR, S3Gaussian consistently outperforms other methods in scene reconstruction and novel-view synthesis; on StreetGaussian’s dataset, it performs similarly to StreetGaussian without explicit supervision.The NOTR comparison reports static and dynamic-object metrics; the StreetGaussian comparison notes that its method uses additional bounding boxes.Source: Table 1 · p. 7, §4.2 · pp. 7–8, Table 2 · p. 7
  • In novel-view synthesis on Waymo-Street, S3Gaussian better reconstructs distant dynamic objects and captures scene-detail changes than StreetGaussian.The comparison also reports noise in StreetGaussian’s sky renderings.Source: Figure 4 · p. 8, §4.2 · p. 8

4.3 Ablation and Analysis

  • Removing the multi-resolution HexPlane encoder sharply reduces rendering quality and prevents accurate spatial-temporal fields and Gaussian deformations.Its voxel features concentrate around moving scene parts, with spatial and temporal patterns corresponding to vehicle trajectories.Source: §4.3 · p. 8, Figure 6 · p. 9
  • Disabling the multi-head Gaussian decoder harms rendering quality, while omitting the semantic-feature and color-deformation decoders blurs truck geometry.The visual ablations also show noise from undeformed LiDAR-initialized Gaussians along moving-vehicle trajectories.Source: §4.3 · p. 8, Figure 6 · p. 9
  • Training 4D Gaussians without static-Gaussian warm-up slows convergence and reduces final rendering quality.Warm-up yields basic static-scene reconstruction, letting the spatial-temporal network focus on dynamic parts and avoiding early numerical errors.Source: §4.3 · p. 9

5 Conclusion

S3Gaussian decomposes static and dynamic street-scene elements using a self-supervised spatial-temporal field network. It models deformation with 4D Gaussians and reports state-of-the-art rendering quality across tasks on NOTR and Waymo.

  • S3Gaussian differentiates dynamic and static elements in complex driving scenes without extra annotations.Source: §5 · p. 9
  • Its Spatial-temporal Field Network uses a Multi-resolution Hexplane Structure Encoder and Multi-head Gaussian Decoder to transform 4D Gaussians.Source: §5 · p. 9
  • Experiments on NOTR and Waymo report superior scene decomposition and state-of-the-art rendering quality across different tasks.Source: §5 · p. 9

A.1 Additional Implementation Details

The experiments use specified camera, frame-selection, and image-resolution settings for Waymo-NOTR and Waymo-Street. Feature extraction uses DINOv2 ViT-B/14, with PCA reducing feature dimensions.

Datasets Details

  • Waymo-NOTR uses three frontal cameras at 640 × 960 for 100-frame sequences, with every tenth frame held out for testing.Source: §A.1 · p. 10
  • Waymo-Street uses frontal cameras at 1066 × 1600, with sequences of approximately 100 frames and every fourth frame held out for testing.Source: §A.1 · p. 10

Feature Extraction

  • DINOv2 ViT-B/14 features are extracted at 644×966 with stride 7, then PCA reduces their dimension from 768 to 3 and they are normalized to [0,1].Source: §A.1 · p. 10

A.2 More Related Work (Advances in Neural Radiance Fields)

The passage contrasts NeRF’s capabilities with its training and rendering speed challenges, and notes that S3Gaussian uses 4D Gaussian representations for faster rendering.

  • NeRF supports novel-scene generation and street-scene separation using vehicle bounding boxes, but training and rendering speed remain challenges.Source: §A.2 · p. 10
  • S3Gaussian uses 4D Gaussian representations for dynamic scenes, which the passage says significantly boost rendering speed compared with NeRF-based methods.Source: §A.2 · p. 10
  • The passage also describes NeRF research advances in training speed, pose optimization, scene editing, object generation, and dynamic-scene representation.Source: §A.2 · p. 10

A.3 Limitations

S3Gaussian has difficulty modeling high-speed moving objects. The authors suspect deformation-field variance and sparse observations contribute to the difficulty.

  • High-speed moving objects remain difficult for S3Gaussian to model, as they do for similar methods.Source: §A.3 · p. 10
  • The authors suspect high deformation-field variance may prevent accurate modeling of rapid motion.Source: §A.3 · p. 10
  • Sparse views of rapidly moving objects, often limited to a few observations, make reconstruction more challenging.Source: §A.3 · p. 10
Loading 2405.20323v1…