Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis

Jonathon Luiten; Georgios Kopanas; Bastian Leibe; Deva Ramanan

Source-linked AI summary

Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis

Jonathon Luiten, Georgios Kopanas, Bastian Leibe, Deva Ramanan

arXiv:2308.09713v1Aug 18, 2023cs.CV

TL;DR

Persistent dynamic 3D scene modeling aims to represent scene elements over time for applications such as robotics, augmented reality, and self-driving. This paper optimizes moving, rotating 3D Gaussians with persistent appearance and local-rigidity constraints, reporting 1.57 normalized-pixel error on 2D tracking metrics—10x better than previous state-of-the-art—and a 1.3 PSNR gain over 3GS-O.

Problem
Persistent dynamic 3D world modeling is an important goal for metric reconstruction of scene elements over time and applications such as robotics, augmented reality, and self-driving.
Source: §1 · p. 1
Method
The method optimizes moving, rotating 3D Gaussians with persistent color, opacity, and size, using local-rigidity constraints to maintain temporal consistency and corresponding scene points.
Source: §3 · pp. 3, 5
Results
1.57 normalized-pixel error on 2D tracking metrics, 10x better than previous state-of-the-art, accompanies 2.21cm average 3D L2 error over 150 timesteps and a 1.3 PSNR gain over 3GS-O.
Source: §1 · p. 2, §4 · p. 9
Takeaways & Limitations
The representation supports dynamic-scene editing by combining components from different scenes and backgrounds and propagating edits across frames.
Source: §4 · p. 9
The method tracks only scene parts visible in the initial frame, cannot reconstruct newly entering objects, and requires multiple cameras rather than monocular video.
Source: §5 · p. 9

Abstract

from arXiv · show

We present a method that simultaneously addresses the tasks of dynamic scene novel-view synthesis and six degree-of-freedom (6-DOF) tracking of all dense scene elements. We follow an analysis-by-synthesis framework, inspired by recent work that models scenes as a collection of 3D Gaussians which are optimized to reconstruct input images via differentiable rendering. To model dynamic scenes, we allow Gaussians to move and rotate over time while enforcing that they have persistent color, opacity, and size. By regularizing Gaussians' motion and rotation with local-rigidity constraints, we show that our Dynamic 3D Gaussians correctly model the same area of physical space over time, including the rotation of that space. Dense 6-DOF tracking and dynamic reconstruction emerges naturally from persistent dynamic view synthesis, without requiring any correspondence or flow as input. We demonstrate a large number of downstream applications enabled by our representation, including first-person view synthesis, dynamic compositional scene synthesis, and 4D video editing.

1. Introduction

Dynamic 3D Gaussians jointly model dynamic scenes for novel-view synthesis and dense long-term 6-DOF tracking. They use a persistent particle representation optimized to match images across time and viewpoints.

1. Introduction

Persistent dynamic world models could support metric scene reconstruction for robotics, augmented reality, and self-driving, as well as controllable editable 3D content, but prior methods had not combined photorealistic arbitrary-scene reconstruction, accurate tracking, appealing novel views, fast training, and real-time rendering.The paper frames scalability to high-resolution imagery as an important requirement.Source: §1 · p. 1
The method extends 3D Gaussian Splatting [17] from static to dynamic scenes, optimizing a persistent 3D representation against images from multiple timesteps and cameras so tracking emerges from scene modeling.Differentiable splatting-based rendering supports image reconstruction optimization, while spatial-consistency priors encourage physically plausible motion.Source: §1 · p. 2
Each Gaussian retains its number, color, opacity, and size over time while its position and orientation vary, with local rigidity, rotational-similarity, and long-term local-isometry priors regularizing motion.The priors encourage neighborhoods to move approximately rigidly between timesteps and nearby particles to remain close over time.Source: §1 · p. 2
Unlike Eulerian fixed-grid approaches and prior point-based representations [1], Dynamic 3D Gaussians use oriented Lagrangian particles with richer physical priors to reconstruct all 3D points' 6-DOF motion.The representation directly tracks both position and rotation.Source: §1 · p. 2
On CMU Panoptic Studio, the method reaches 28.7 PSNR at 850 FPS, 2.21cm average 3D tracking error, and 1.57 normalized-pixel error, 10x better than prior state-of-the-art.It trains on 150 timesteps with 27 training cameras per timestep in 2 hours on one RTX 3090 GPU, and tracks each 3D point's rotation.Source: §1 · p. 2
Tracking arises from rendering per-frame images without optical flow, pose skeletons, or other correspondence input, enabling scene edits across time, dynamic object addition or removal, and cameras that follow scene elements.The method's fast rendering and training also support real-time applications.Source: §1 · p. 2

2. Related Work

The section positions the work as a unified multi-camera analysis-by-synthesis approach to dynamic novel-view synthesis, long-term point tracking, and dynamic reconstruction. It contrasts prior methods by their correspondence capabilities, representation limits, and input requirements.

Dynamic Novel-View Synthesis

Prior dynamic-NeRF methods use per-timestep or Eulerian-grid [5] [10] [36] representations without correspondence, while canonical-warp methods [8] [21] [22] [28] [32] [40] provide backward but not default forward mappings.Some canonical warps use invertible skinning [35] or reversible networks [40], but dependence on one canonical view restricts dynamic representation.Source: §2 · pp. 2–3
Template-guided methods [20] constrain scenes to predefined motion models and require prior scene knowledge, whereas point-based methods [1] offer potential for correspondence through their Lagrangian representation.The paper notes that point-based rendering has underperformed MLP- and grid-based approaches [25].Source: §2 · p. 3
The authors describe their method as the first dynamic Gaussian reconstruction, building on static 3D Gaussian splatting [17] and using Gaussian rotation to model full 6-DOF motion and support local-rigidity losses.They note that earlier Gaussian view-synthesis work [17] [18] addressed static scenes.Source: §2 · p. 3

Long-Term Point Tracking

Unlike OmniMotion [40], which uses monocular video, pseudo-3D tracks, and dense pairwise optical-flow targets, this method uses multi-camera metric 3D and no correspondence input.Prior long-term dense trackers [6] [7] [12] [16] are generally trained on datasets of ground-truth point tracks; traditional tracking often targets objects or only two-frame point correspondences.Source: §2 · p. 3

Dynamic Reconstruction

Other dynamic-reconstruction approaches may require accurate depth cameras, ground-truth object point clouds [31], or domain-specific scenes [23], while uses 480 cameras and precomputed optical flow.The paper evaluates multi-camera 3D tracking using Panoptic Studio data with 27 cameras.Source: §2 · p. 3

3. Method

Dynamic 3D Gaussians reconstruct scenes online from multiview images using differentiable rendering and temporally persistent representations. Local physical priors make dense 6-DOF tracking emerge from the reconstruction, without correspondence or flow input.

Overview

Each timestep is optimized from multiview images, initialized from the previous representation, with only motion parameters updated after the first timestep.The first timestep optimizes all Gaussian properties; subsequent timesteps are reconstructed online by gradient-based optimization through a differentiable renderer.Source: §3 · p. 3