PaperlayerView exact revision on arXiv

Source-linked AI summary

Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Rendering

Yurui Chen, Chun Gu, Junzhe Jiang, Xiatian Zhu, Li Zhang

arXiv:2311.18561v3cs.CV

TL;DR

Large-scale dynamic urban-scene reconstruction must handle complex geometry and unconstrained spatial-temporal dynamics without adequately capturing interactions between static and dynamic elements. PVG extends 3D Gaussian Splatting with periodic vibration dynamics, temporal smoothing, and position-aware control, and outperforms prior methods across reconstruction and novel-view synthesis benchmarks while rendering substantially faster.

  • Problem

    Dynamic urban scenes combine intricate geometry with unconstrained spatial-temporal dynamics, while prior approaches often separate static and dynamic elements and miss their interactions.

  • Method

    PVG unifies static and dynamic scene representation by adding periodic vibration-based dynamics to 3D Gaussian Splatting, temporal smoothing, and position-aware point control.

  • Results

    PVG outperforms prior methods across reconstruction and novel-view synthesis on Waymo and KITTI, with 12.6% higher PSNR than SUDS for image reconstruction and 8.4% higher PSNR than EmerNeRF for novel-view synthesis.

  • Takeaways & Limitations

    PVG provides a unified dynamic-urban-scene representation with an efficiency advantage for reconstruction and novel-view synthesis.

  • Takeaways & Limitations

    The underlying 3D Gaussian Splatting representation lacks the ability to capture dynamic changes over time without the proposed PVG extension.

Abstract

from arXiv · show

Modeling dynamic, large-scale urban scenes is challenging due to their highly intricate geometric structures and unconstrained dynamics in both space and time. Prior methods often employ high-level architectural priors, separating static and dynamic elements, resulting in suboptimal capture of their synergistic interactions. To address this challenge, we present a unified representation model, called Periodic Vibration Gaussian (PVG). PVG builds upon the efficient 3D Gaussian splatting technique, originally designed for static scene representation, by introducing periodic vibration-based temporal dynamics. This innovation enables PVG to elegantly and uniformly represent the characteristics of various objects and elements in dynamic urban scenes. To enhance temporally coherent and large scene representation learning with sparse training data, we introduce a novel temporal smoothing mechanism and a position-aware adaptive control strategy respectively. Extensive experiments on Waymo Open Dataset and KITTI benchmarks demonstrate that PVG surpasses state-of-the-art alternatives in both reconstruction and novel view synthesis for both dynamic and static scenes. Notably, PVG achieves this without relying on manually labeled object bounding boxes or expensive optical flow estimation. Moreover, PVG exhibits 900-fold acceleration in rendering over the best alternative.

1 Introduction

Dynamic urban reconstruction must represent complex spatial and temporal changes while scaling beyond object-level supervision and separated scene components. PVG addresses these limitations with a unified Gaussian-based formulation and complementary temporal and spatial controls.

  • Dynamic urban scenes challenge methods designed primarily for static representation because they contain complex changes across both space and time.
  • Object- and background-decomposition methods face scalability problems because accurate object-level supervision is difficult and per-object representation increases complexity with scene size.
  • Optical-flow-assisted and self-supervised dynamic methods reduce annotation demands but retain efficiency bottlenecks for large-scale reconstruction and rendering.
  • PVG unifies static and dynamic elements in one formulation by extending 3D Gaussian Splatting with periodic vibration-based temporal dynamics.
  • PVG adds temporal smoothing and position-aware point adaptive control to improve learning continuity with limited data and distant-view fitting.
  • Experiments on KITTI and Waymo show that PVG outperforms prior alternatives in novel view synthesis while improving training and rendering efficiency, including 900-fold faster rendering than EmerNeRF.

2 Related work

Prior dynamic-scene methods extend implicit neural representations or explicitly separate scene components, but they face efficiency, scalability, or annotation limitations. PVG extends 3D Gaussian Splatting with time to provide a uniform and efficient representation without manual annotations or pretrained models.

  • Neural rendering: NeRF-based novel view synthesis provides high-quality views but suffers from slow training and rendering and high memory usage.
  • Neural rendering: 3D Gaussian Splatting introduces a point-based 3D scene representation aimed at improving efficiency over implicit volumetric approaches.
  • Dynamic scene models: Dynamic NeRF extensions that add time as an input can couple temporal positional changes with radiance without geometric priors for temporal influence.
  • Dynamic scene models: Optical-flow-based and self-supervised dynamic methods reduce annotation dependence to different degrees but can still suffer from low image quality or rendering speed.
  • Dynamic scene models: PVG adds a time dimension to 3D Gaussian Splatting, enabling uniform and efficient large-scale dynamic urban reconstruction without manual annotations or pretrained models.

3 Method

PVG extends 3D Gaussian Splatting with periodic, time-dependent point motion and opacity, providing one representation for static and dynamic scene elements. Temporal smoothing and position-aware control support coherent learning and scalable rendering from sparse urban-scene observations.

  • 3.1 Preliminary: PVG represents sequential multi-sensor images and timestamped LiDAR points for 3D reconstruction and novel-view synthesis at desired timestamps.The inputs include calibrated camera intrinsics and extrinsics, image timestamps, and spatially coordinated LiDAR points.
  • 3.1 Preliminary: 3DGS represents scenes with Gaussian points carrying means, covariances, opacities, and colors, then rasterizes projected points for real-time alpha blending.Rendered images use camera extrinsics and intrinsics, while depth and opacity can replace color as rendering targets.
  • 3.2 Periodic Vibration Gaussian (PVG): PVG makes Gaussian means and opacities time-dependent around a learnable life peak τ, allowing each point to adjust its motion and visibility over time.The model adds τ, β, and v to the conventional Gaussian parameters; β controls lifespan and v denotes instantaneous velocity at τ.
  • 3.2 Periodic Vibration Gaussian (PVG): Temporal smoothing translates the PVG point set queried at t − ∆t by its 3D flow translation before rendering supervision at timestamp t.This connects consecutive observations and helps points learn motion trends from adjacent frames.
  • 3.3 Position-aware point adaptive control: Position-aware adaptive control uses scene-radius scaling to decide whether high-gradient PVGs are cloned or split and whether oversized points are pruned.The strategy clones when max(s) ≤ g · γ(µ), splits otherwise, and prunes when max(s) > b · γ(µ).

4 Experiments

PVG is evaluated against state-of-the-art methods across dynamic and static urban-scene reconstruction and novel view synthesis benchmarks. The experiments show strong performance, efficient rendering, and benefits from unified temporal modeling and adaptive controls.

  • Comparison with state of the art: PVG is evaluated on Waymo and KITTI for image reconstruction and novel view synthesis against both dynamic- and static-scene methods.The evaluation includes comparisons with S-NeRF, StreetSurf, Mars, 3DGS, NSG, SUDS, and EmerNeRF.
  • Comparison with state of the art: 12.6% higher PSNR, 13.0% higher SSIM, and 20.8% lower LPIPS than SUDS are reported for Waymo image reconstruction.For Waymo novel view synthesis, PVG exceeds EmerNeRF by 8.4% in PSNR, 11.3% in SSIM, and 27.3% in LPIPS.
  • Comparison with state of the art: PVG outperforms static-scene baselines on Waymo across three metrics and achieves significant PSNR gains under the StreetSurf evaluation setup.These experiments assess whether the unified representation also handles static scenes effectively.
  • Comparison with state of the art: 16.0% higher PSNR, 7.0% higher SSIM, and 62.2% lower LPIPS than SUDS are reported for KITTI image reconstruction.The method also surpasses EmerNeRF for KITTI novel view synthesis, while temporal smoothing supports a concise representation under many dynamic objects.
  • Comparison with concurrent Gaussian Splatting models: PVG achieves the best overall performance among concurrent Gaussian Splatting methods without object bounding boxes, including on distant and occluded objects.Its primitive-level dynamics use concepts such as life peak and cycle to represent static and dynamic elements together.
  • Ablation study: The sine-based vibration formulation balances static and dynamic reconstruction better than constant or linear alternatives, while temporal smoothing and position-aware control improve robustness.The best cycle length is l = 0.2; temporal smoothing improves novel view synthesis, and camera perturbation tests show stable rendering quality.

5 Conclusions

PVG unifies static and dynamic urban-scene representation by combining periodic vibration, opacity decay, and scene-flow temporal smoothing within 3D Gaussian Splatting. It outperforms state-of-the-art methods on Waymo and KITTI, while retaining efficiency advantages but facing geometric-accuracy limitations.

  • PVG unifies diverse objects and materials in dynamic urban scenes within one formulation.
  • PVG integrates periodic vibration, time-dependent opacity decay, and scene-flow-based temporal smoothing into 3D Gaussian Splatting.
  • PVG significantly outperforms state-of-the-art methods on Waymo Open Dataset and KITTI in dynamic-scene reconstruction and novel view synthesis.
  • PVG’s highly adaptable independent-point design limits precise geometric representation and can challenge coherence for fast-moving objects.

Representation

PVG uses Gaussian points with different lifespans to represent static and unconstrained dynamic scene components. Long-lived points model static elements, while short-lived points distributed across space and time model dynamics.

  • Long-lived PVG points represent static scene elements similarly to conventional 3D Gaussian points.
  • Short-lived PVG points distributed over space and time represent unconstrained dynamic scene components.

Temporal smoothing by intrinsic motion

Temporal smoothing uses intrinsic motion consistency to constrain PVG states between sparse training frames. It measures consistency through rendered outputs and samples time shifts efficiently during training.

  • PVG enforces that a state at time t resembles the state translated from t − ∆t by velocity v∆t.
  • Each PVG trajectory uses a piecewise sine function, with static and dynamic behavior differentiated by slope, staticness coefficient ρ, and lifespan.
  • Diff(·) measures differences between two sets of 3D Gaussian points, while Render(·) provides an indirect way to compute that consistency.
  • The temporal-smoothing objective compares two rendering pathways across camera parameters, timestamps, and small time shifts.
  • Training samples timestamps, camera parameters, and ∆t uniformly, replaces one render with the ground-truth image, and therefore renders only once per step.

Appendix B More implementation details

The appendix describes implementation choices for training, point adaptation, temporal smoothing, and fair comparison across datasets and methods. These settings include Gaussian perturbation, dataset-specific training adjustments, and pose handling for StreetSurf.

  • KITTI and whole Waymo experiments use adjusted parameters to accommodate more training frames, with 40,000 training iterations and densification until iteration 20,000.
  • PVG point splitting shrinks new-point scales by 0.8 and perturbs temporal and spatial parameters using Gaussian noise tied to velocity.
  • Clone operations copy every parameter from the source PVG point to the new point.
  • StreetSurf pose refinement is disabled because refined poses cannot align with ground-truth poses, producing low novel-view-synthesis PSNR.

Appendix C Visualization for Waymo

PVG reconstructs both near and distant regions in Waymo scenes and produces novel views with ground-truth-like quality, unlike methods with narrower strengths.

  • PVG reconstructs both near and distant regions well in Waymo scenes.
  • PVG renders novel views with quality comparable to the ground truth.

Appendix D Visualization for KITTI

The KITTI appendix presents qualitative image-reconstruction and novel-view-synthesis results, while noting that the evaluated scenarios largely involve a stationary ego vehicle and that depth quality depends on the point cloud.

  • The KITTI scenarios used in are almost entirely cases where the ego vehicle is not moving.
  • Depth-reconstruction quality depends on the point cloud.
  • Fig. D3 shows qualitative image-reconstruction results on KITTI.
  • Fig. D4 shows qualitative novel-view-synthesis results on KITTI.
  • The appendix also identifies qualitative novel-view-synthesis visualizations for Waymo in Figs. D5 and D6.
Loading 2311.18561v3…