PaperlayerView exact revision on arXiv

Source-linked AI summary

Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Rendering

Yurui Chen, Chun Gu, Junzhe Jiang, Xiatian Zhu, Li Zhang

arXiv:2311.18561v3cs.CV

TL;DR

Reconstructing large urban scenes is challenging because they are complex and dynamic, while existing representations may overlook dynamic elements or fail to capture interactions among scene components. PVG unifies static and dynamic scene representation through periodic vibration dynamics, and experiments report stronger results than prior methods across reconstruction and novel-view synthesis.

  • Problem

    Existing representations focus on static scenes or separate static and dynamic components, limiting their capture of dynamic urban elements and interactions.

    Source: §1 · pp. 2–3

  • Method

    PVG unifies static and dynamic scene representation using periodic vibration-based temporal dynamics, with temporal smoothing and position-aware point control.

    Source: §1 · p. 3

  • Results

    PVG outperforms baselines across reconstruction and novel-view synthesis metrics; for novel-view synthesis, it exceeds EmerNeRF by 8.4% in PSNR.

    Source: §4.1 · p. 11

  • Takeaways & Limitations

    PVG is reported to represent diverse object and material characteristics in dynamic urban scenes within a unified formulation.

    Source: §5 · p. 20

    The underlying 3DGS model represents static points and lacks the ability to capture temporal changes needed for dynamic urban scenes.

    Source: §3.1 · p. 6

Abstract

from arXiv · show

Modeling dynamic, large-scale urban scenes is challenging due to their highly intricate geometric structures and unconstrained dynamics in both space and time. Prior methods often employ high-level architectural priors, separating static and dynamic elements, resulting in suboptimal capture of their synergistic interactions. To address this challenge, we present a unified representation model, called Periodic Vibration Gaussian (PVG). PVG builds upon the efficient 3D Gaussian splatting technique, originally designed for static scene representation, by introducing periodic vibration-based temporal dynamics. This innovation enables PVG to elegantly and uniformly represent the characteristics of various objects and elements in dynamic urban scenes. To enhance temporally coherent and large scene representation learning with sparse training data, we introduce a novel temporal smoothing mechanism and a position-aware adaptive control strategy respectively. Extensive experiments on Waymo Open Dataset and KITTI benchmarks demonstrate that PVG surpasses state-of-the-art alternatives in both reconstruction and novel view synthesis for both dynamic and static scenes. Notably, PVG achieves this without relying on manually labeled object bounding boxes or expensive optical flow estimation. Moreover, PVG exhibits 900-fold acceleration in rendering over the best alternative.

1 Introduction

PVG introduces a unified 3D Gaussian Splatting representation with periodic temporal dynamics for large-scale urban scenes. The paper reports improved novel-view synthesis on KITTI and Waymo and 900-fold faster rendering than EmerNeRF.

1 Introduction

  • Urban scene reconstruction supports digital maps, navigation, and autonomous driving, while static-focused NeRF methods overlook dynamic scene elements.Source: §1 · p. 2
  • NSG uses scene graphs and PNF decomposes scenes into objects and backgrounds, but object-level supervision is difficult to obtain and per-object modeling scales complexity with object count.PNF also uses panoptic segmentation as an auxiliary task.Source: §1 · p. 2
  • SUDS uses optical flow and EmerNeRF reduces that dependence through self-supervision, but implicit NeRF methods are inefficient and separating scene parts limits modeling their intrinsic interactions.Explicit object-level supervision and optical flow are challenges the paper identifies in prior approaches.Source: §1 · pp. 2–3
  • PVG extends 3D Gaussian Splatting with periodic vibration dynamics to unify static and dynamic elements, while modeling velocity, staticness, distant views, and temporal continuity.It introduces position-aware point adaptive control for distant views and temporal smoothing to improve continuity with limited training data.Source: §1 · p. 3
  • PVG outperforms previous state-of-the-art alternatives in novel view synthesis on KITTI and Waymo, with 900-fold rendering acceleration over EmerNeRF.The introduction also reports that PVG is superior in visual quality and rendering efficiency to concurrent 3DGS-based models.Source: §1 · p. 3

2 Related work

Prior work advances neural rendering and dynamic-scene reconstruction but faces efficiency, temporal-modeling, or supervision challenges; PVG extends 3D Gaussian Splatting with a uniform representation for dynamic, large-scale urban scenes.

Neural rendering

  • NeRF produces high-quality novel views but is slow to train and render and uses substantial memory; 3DGS combines NeRF-like volume-rendering quality with rasterization speed.Other work accelerates training with voxel grids, hash encoding, or tensor factorization, converts implicit volumes to textured meshes for faster rendering, or improves antialiasing and reflection modeling.Source: §2 · p. 3

Dynamic scene models

  • Dynamic-scene methods extend NeRF with time inputs or model deformation, while 3DGS-based methods use deformable Gaussians or 4D formulations, each with limitations for large scenes or static-dynamic separation.PVG introduces a per-point staticity coefficient and temporal smoothing; its autonomous-driving designs include LiDAR-projected depth supervision, cube map-based sky modeling, and position-aware control for distant structures.Source: §2 · p. 4

Urban scene reconstruction

  • Urban-scene methods use scalable or high-fidelity static-scene representations, but dynamic-scene approaches may require extra inputs, object-level boxes, or optical flow, while explicit object-background separation can reduce consistency.PVG extends 3DGS with time to represent dynamic, large-scale urban scenes without manual annotations or pre-trained models; bounding-box supervision can be noisy for distant or occluded objects, and prior optical-flow-reducing work reports low image quality and rendering speed.Source: §2 · pp. 4–5

3 Method

PVG extends 3D Gaussian splatting with time-dependent point dynamics to reconstruct urban scenes and render novel views at requested times and camera poses. Its method combines position-aware point control with temporal self-supervision and scene-specific training components.

3 Method

  • Given sequential calibrated camera images and timestamped LiDAR points, the framework aims to reconstruct 3D scenes and render images at any requested timestamp and camera pose.It represents this task as a rendering function of the output camera pose and time.Source: §3 · p. 5

3.1 Preliminary

  • 3DGS represents scenes with Gaussian points and supports real-time alpha blending, but its static-point model cannot capture urban scene changes over time.PVG is introduced to address this limitation.Source: §3.1 · pp. 5–6

3.2 Periodic Vibration Gaussian (PVG)

  • PVG gives each Gaussian a life peak τ and time-varying mean and opacity; β controls lifespan, v specifies vibration direction and peak-time velocity, and ρ = β/l measures staticness.The periodic formulation is a primitive for fitting complex motion, not an assumption that scene motion is inherently periodic.Source: §3.2 · pp. 6–7

3.3 Position-aware point adaptive control

  • Position-aware adaptive control scales densification by point location, cloning smaller PVGs and splitting larger ones when view-space gradients exceed a threshold, and pruning oversized points.The strategy favors larger points at distant locations and smaller points nearby.Source: §3.3 · p. 8

3.4 Model training

  • Temporal self-supervision translates point states between adjacent timestamps using average velocity, improving coherence with sparse data while avoiding optical-flow estimation.Training sometimes uses no time offset and otherwise samples one uniformly from a bounded interval; this helps learn motion trends and reduce overfitting.Source: §3.4 · pp. 8–9, Figure 3 · p. 8
  • Training combines RGB L1/SSIM, sparse LiDAR inverse-depth, sky-opacity, and average-velocity losses; a directional cube map models the sky background without affecting rendering speed.The average-velocity loss encourages sparse motion and larger β for static scene components.Source: §3.4 · pp. 9–10

4 Experiments

PVG is evaluated on dynamic and static urban scenes, concurrent Gaussian-splatting methods, dynamic-element behavior, ablations, camera perturbations, and Plenoptic videos. It outperforms leading baselines on Waymo and KITTI, while the ablations and robustness tests examine how its temporal and positional mechanisms affect results.

4 Experiments

  • Experiments evaluate reconstruction and novel view synthesis, using every fourth timestamp as the novel-view test set; runs use one RTX A6000 for 30,000 iterations, take about an hour, and render at 50 FPS.The metrics are PSNR, SSIM, and LPIPS. The comparisons include dynamic-scene methods, static-scene-focused methods, and concurrent box-based or Gaussian-splatting approaches.Source: §4 · pp. 10–11

4.1 Comparison with state of the art

  • On Waymo, PVG improves image reconstruction over SUDS by 12.6% PSNR, 13.0% SSIM, and 20.8% lower LPIPS, and novel view synthesis over EmerNeRF by 8.4%, 11.3%, and 27.3%.On KITTI, PVG improves image reconstruction over SUDS by 16.0% PSNR, 7.0% SSIM, and 62.2% lower LPIPS, and novel view synthesis over EmerNeRF by 8.7%, 11.9%, and 51.9% lower LPIPS. Table 1 reports dynamic-scene results for Waymo and KITTI.Source: §4.1 · pp. 11, 13, Figure 4 · p. 11, Table 1 · p. 12
  • PVG also outperforms static-scene baselines across three metrics and reports significant PSNR gains under StreetSurf’s setup; it is superior to EmerNeRF under matched training settings.The Waymo comparison uses four randomly selected scenes under EmerNeRF’s training setup; static comparisons use S-NeRF’s and StreetSurf’s respective settings.Source: §4.1 · pp. 11–12, Table 3 · p. 13, Table 2 · p. 13

4.2 Comparison with concurrent Gaussian Splatting models

  • PVG achieves the best overall performance among concurrent Gaussian-splatting competitors without object bounding boxes, using primitive-level life peak and cycle concepts to model static and dynamic elements together.The comparison covers eight StreetGaussian scenes and four paper scenes, each captured with three cameras at 960×640 resolution.Source: Table 4 · p. 13, §4.2 · pp. 14–15

4.3 Dynamic element analysis

  • PVG captures moving objects, light, and shadows while keeping static regions visually stable and accurately rendering dynamic regions such as vehicles and trees.The paper uses velocity and staticness maps to analyze this separation; small ρ indicates dynamic areas and large ρ indicates static ones.Source: Figure 9 · p. 15, §4.3 · p. 16, Figure 10 · p. 16

4.4 Ablation study

  • Ablations show temporal smoothing improves novel-view smoothness, consistency, and motion learning; LiDAR supervision and sky refinement improve geometry, while positional control improves distant-view reconstruction.Velocity loss encourages sparser velocity and simpler dynamic/static segmentation. The sine-vibration model balances static and dynamic representation: constant motion handles static elements but performs poorly on dynamic ones, while linear motion captures dynamics but can misrepresent static elements and cause ghosting. The reported PSNR values for constant, linear, and proposed dynamics are 27.77, 27.09, and 28.11, respectively.Source: §4.4 · pp. 16–17, Figure 12 · p. 17, Figure 13 · p. 17

4.5 View synthesis of different camera settings

  • PVG remains stable and compact under zoom changes and perturbed camera intrinsics and extrinsics; depth loss improves geometry and generalization beyond the driving path.The paper illustrates the beyond-path comparison in Fig. 14d versus Fig. 14e.Source: §4.5 · p. 19, Figure 14 · p. 19

4.6 Evaluation on Plenoptic video dataset

  • On the Plenoptic video dataset, with one view held out per scene, PVG achieves competitive performance, particularly for complex dynamic scenes with backgrounds such as water flow.The dataset has six real-world scenes with diverse human motions; the reported baselines are taken from their original papers.Source: §4.6 · p. 19, Table 10 · p. 19

5 Conclusions

PVG combines periodic temporal dynamics with 3D Gaussian Splatting for unified dynamic urban-scene reconstruction, outperforming state-of-the-art methods on Waymo and KITTI. Its flexible point-based design also leaves challenges in geometric accuracy, coherence for fast-moving objects, and dynamic-object editing.

5 Conclusions

  • PVG significantly outperforms state-of-the-art methods on Waymo Open Dataset and KITTI, with an efficiency advantage in dynamic-scene reconstruction and novel view synthesis.The conclusion does not quantify the advantage.Source: §5 · p. 20
  • PVG integrates periodic vibration, time-dependent opacity decay, and scene-flow-based temporal smoothing into 3D Gaussian Splatting to represent diverse urban objects and materials uniformly.Source: §5 · p. 20
  • PVG’s adaptable design limits precise geometric representation, while independent discrete Gaussian points make spatial and temporal coherence challenging for fast-moving objects.The independent points also make efficient editing of dynamic objects difficult.Source: §5 · p. 20
  • Future work will improve geometric accuracy and urban-scene depiction, using finer-grained motion modeling and structured priors to address coherence and editing challenges.Source: §5 · p. 20

Appendix A More model interpretation · Representation

PVG represents static and unconstrained dynamic scene components through points with different lifespan patterns, illustrated in a simplified scene containing both types.

Representation

  • PVG uses long-lifespan points to quantify static scene elements, like conventional 3D Gaussians.The interpretation is illustrated using a simplified scene with static and dynamic components.Source: Representation · p. 20
  • Short-lifespan PVG points distributed across space and time represent unconstrained dynamic scene components.Source: Representation · p. 20

Temporal smoothing by intrinsic motion

To address novel-timestamp under-fitting from sparse training frames, PVG regularizes renderings according to intrinsic motion consistency and uses an efficient single-render training procedure.

Temporal smoothing by intrinsic motion

  • PVG regularizes sparse-frame training by requiring status Hi(t) to resemble the status translated to t −∆t by distance v∆t.This intrinsic-consistency law is intended to make Hi temporally and spatially consistent with its motion.Source: Temporal smoothing by intrinsic motion · p. 20
  • PVG models each point’s trajectory with a piecewise sine function, using small ρ and short lifespans for dynamic parts and large ρ and long lifespans for static parts.Collections of PVG points can represent unconstrained motion, such as a moving car.Source: Figure A1 · p. 21
  • Because directly measuring differences between Gaussian-point sets is difficult, PVG measures them indirectly through rendered-image differences across camera parameters and timestamps.The objective is expressed as an expectation over camera extrinsics E, intrinsics I, timestamp t, and a small shift ∆t.Source: Temporal smoothing by intrinsic motion · p. 21
  • To avoid rendering twice per sampled camera pose and timestamp, PVG samples training-set {t, E, I}, draws ∆t from U(−δ, +δ), and substitutes the ground-truth image for one rendering.This yields one render per optimization step.Source: Temporal smoothing by intrinsic motion · p. 21

Appendix B More implementation details

Appendix B details training and Gaussian-update settings for KITTI and whole Waymo sequences, as well as a StreetSurf pose-refinement adjustment for fairer comparison. It also illustrates the temporal smoothness constraint used between observed training frames.

Appendix B More implementation details

  • For KITTI and whole Waymo sequences, PVG trains for 40,000 iterations and densifies and prunes Gaussians every 200 iterations through iteration 20,000.These settings accommodate more training frames than the other experiments.Source: Appendix B · p. 21
  • When splitting a PVG point, the method decays each new point’s scales by 0.8, perturbs τ and position, and shrinks β to 0.8β during iterations 10,000–20,000.The τ perturbation is sampled from N(0, β2), and position is disturbed by ∆τ·v̄; β is not shrunk during the first 10,000 iterations.Source: Appendix B · pp. 21–22
  • PVG’s temporal smoothness constraint connects training-frame knots smoothly by constraining the slope of v between observed frames.The figure explains that intermediate times between adjacent training frames lack corresponding data to constrain the model directly.Source: Figure A2 · p. 22
  • In clone operations, PVG copies every parameter of the original point to the new point.Source: Appendix B · p. 22
  • For StreetSurf, PVG deactivates pose refinement because refined poses do not align with ground-truth poses and lower novel-view-synthesis PSNR.This adjustment is made for a fairer comparison.Source: Appendix B · p. 22

Appendix C Visualization for Waymo

Waymo visualizations compare novel-view synthesis and depth against LiDAR-projected ground truth, showing PVG’s ability to reconstruct near and distant views and render ground-truth-quality images.

Appendix C Visualization for Waymo

  • PVG reconstructs near and distant views and renders ground-truth-quality images, while SUDS is poor at novel-view synthesis and 3DGS struggles with distant views and dynamic objects.LiDAR points are projected into the camera world to provide the ground-truth depth map.Source: Appendix C · p. 22

Appendix D Visualization for KITTI

The appendix presents qualitative KITTI image-reconstruction and novel-view-synthesis results, noting that the scenarios used in are almost all cases where the ego vehicle is stationary and that depth reconstruction quality depends on the point cloud.

Appendix D Visualization for KITTI

Related papers

Loading 2311.18561v3…