Source-linked AI summary
Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, Deva Ramanan
TL;DR
Persistent dynamic 3D modeling must support both dense 6-DOF tracking and dynamic novel-view synthesis from changing scenes. The paper builds a persistent Gaussian representation with motion and rotation regularization, achieving accurate tracking and improved view synthesis without correspondence or optical-flow inputs.
Problem
Persistent dynamic world models need to reconstruct and track scene elements over time while supporting scalable, high-resolution dynamic content creation.
Method
The method optimizes moving and rotating 3D Gaussians with persistent color, opacity, and size, using local rigidity, rotation, and long-term isometry constraints within an analysis-by-synthesis framework.
Results
The approach achieves 2.21cm average 3D tracking error over 150 timesteps, 1.57 normalized-pixel error, and a 1.3 PSNR improvement for view synthesis without correspondence or optical-flow inputs.
Takeaways & Limitations
Persistent dynamic view synthesis yields dense non-rigid 6-DOF tracking and supports dynamic reconstruction and downstream applications such as first-person view synthesis, compositional scene synthesis, and 4D video editing.
Takeaways & Limitations
The method tracks only scene parts visible in the initial frame, requires multiple cameras, and does not work off-the-shelf on monocular video.
Abstract
from arXiv · showhide
We present a method that simultaneously addresses the tasks of dynamic scene novel-view synthesis and six degree-of-freedom (6-DOF) tracking of all dense scene elements. We follow an analysis-by-synthesis framework, inspired by recent work that models scenes as a collection of 3D Gaussians which are optimized to reconstruct input images via differentiable rendering. To model dynamic scenes, we allow Gaussians to move and rotate over time while enforcing that they have persistent color, opacity, and size. By regularizing Gaussians' motion and rotation with local-rigidity constraints, we show that our Dynamic 3D Gaussians correctly model the same area of physical space over time, including the rotation of that space. Dense 6-DOF tracking and dynamic reconstruction emerges naturally from persistent dynamic view synthesis, without requiring any correspondence or flow as input. We demonstrate a large number of downstream applications enabled by our representation, including first-person view synthesis, dynamic compositional scene synthesis, and 4D video editing.
1. Introduction
The paper targets persistent dynamic 3D modeling for reconstruction, tracking, and generative scene editing. It introduces an approach intended to produce realistic dynamic reconstructions, accurate tracks, appealing novel views, and fast training and rendering.
- Persistent dynamic 3D world models could reconstruct every scene part over time for robotics, augmented reality, and self-driving applications.
- They could also support controllable, editable high-resolution dynamic 3D assets for movies, video games, and the metaverse.
- The paper presents one approach combining dynamic reconstruction, dense non-rigid long-term 6-DOF tracking, and dynamic novel-view synthesis.
- Tracking emerges from fitting a persistent dynamic 3D representation to images from different timesteps and cameras under physically plausible spatial-consistency priors.
- 2 hours of training on a single RTX 3090 GPU supports reconstruction from multi-camera imagery, while the method targets real-time rendering.
2. Related Work
Prior dynamic scene methods commonly rely on restricted representations, canonical views, templates, or correspondence inputs. The paper positions its approach as avoiding correspondence inputs while jointly modeling dynamic reconstruction, tracking, and novel-view synthesis.
- The paper frames dynamic novel-view synthesis, long-term point tracking, and dynamic reconstruction as a unified analysis-by-synthesis problem.
- Some methods use invertible warp fields to obtain correspondences, but requiring a single canonical view restricts dynamic representation ability.
- Template-guided methods model restricted environments using predefined motion structures such as human-pose skeleton transformations.
- Optical-flow-based approaches may require dense flow between every timestep pair, making optimization scale with the square of the frame count.
- In contrast, the proposed method takes no correspondences as input and obtains tracking by fitting a persistent representation with physically based priors.
3. Method
The method reconstructs dynamic scenes online as temporally persistent 3D Gaussian representations optimized through differentiable rendering. Motion and rotation are regularized locally so tracking and consistent 3D correspondences emerge from the reconstruction.
- Online reconstruction: The system reconstructs one timestep at a time from multi-camera images, initializing each timestep with the preceding scene representation.The first timestep optimizes all Gaussian properties; later timesteps optimize only scene motion while rendering is compared against input images.
- Emergent tracking: The resulting temporally consistent representations provide one-to-one 3D point correspondences and associated rotation changes across timesteps.Tracking therefore emerges from fitting a persistent representation to the observed images rather than from a separate correspondence stage.
- Dynamic 3D Gaussians: Each Dynamic 3D Gaussian has time-varying center and quaternion rotation, while size, color, opacity, and background remain temporally consistent.The Gaussian representation also uses a covariance formed from rotation and scaling, with quaternion-based rotations.
- Differentiable rendering: Differentiable rendering projects Gaussian centers and covariances into each camera, then combines depth-sorted Gaussian influences with front-to-back volume rendering.Gaussian influence is evaluated in the image plane and weighted by color and transmittance.
- Physical regularization: The local-rigidity loss requires nearby Gaussians to follow the rigid-body transform of a reference Gaussian between consecutive timesteps.This couples neighboring translations and rotations while permitting global non-rigid reconstruction.
- Physical regularization: Short-term local rigidity and rotation losses are supplemented by a long-term isometry constraint to reduce drift between scene elements.The method uses the same nearest-neighbor structure and weighting scheme for the local constraints, while the isometry loss preserves inter-Gaussian distances over longer horizons.
4. Experiments
Experiments evaluate Dynamic 3D Gaussians on PanopticSports and Particle-NeRF across view synthesis and 2D/3D tracking, including ablations and visual comparisons. The method achieves accurate long-term tracking while supporting dynamic-scene editing and compositional applications.
- Dataset Preparation: PanopticSports contains six evaluated 150-frame sequences captured at 30 FPS with 31 cameras, split into 27 training and 4 testing cameras.Three additional sequences are used for visual results but excluded from evaluation because 3D tracking ground truth is unavailable.
- Comparisons: The evaluation compares Dynamic 3D Gaussians with online 3D Gaussian Splatting on view synthesis, 3D tracking, and 2D tracking, and with Particle-NeRF, InstantNGP, and TiNeuVox on Particle-NeRF.The Particle-NeRF comparison uses a wall-clock-time criterion because the methods do not share the same optimization procedure.
- PanopticSports Results: 2.21cm median trajectory error, 100% survival, and 71.4 accuracy are reported for 3D tracking across all PanopticSports trajectories.The method is evaluated over 150 timesteps and does not lose the tracked points.
- PanopticSports Results: 1.57 pixels median trajectory error, 78.4 accuracy, and 100% survival are reported for 2D tracking, compared with PIPs at 15.7 pixels, 39.6 accuracy, and 79% survival.The paper notes that this comparison is not one-to-one fair because the methods receive different data and training histories.
- Ablation Study: The full method improves view synthesis by 1.3 PSNR over online 3GS-O, while rigidity, segmentation, parameter fixing, and forward initialization support both tracking and view synthesis.Rotation and isometric losses provide small metric gains but improve reconstruction coherence visually.
- Further Applications: Dynamic 3D Gaussians support removing, duplicating, combining, and editing dynamic objects, propagating edits over time, and attaching moving first-person cameras or objects.These applications rely on independent Gaussians and full 6-DOF tracking.
5. Conclusion and Limitations
The paper presents Dynamic 3D Gaussians for dynamic scene modeling, novel-view synthesis, and 6-DOF tracking, with applications in entertainment, robotics, VR, and AR. Its scope is limited to content visible initially and requires multi-camera input.
- Conclusion: The method models dynamic scenes with Gaussian elements that capture movement and rotation while supporting view synthesis and 6-DOF tracking.The conclusion connects the representation to entertainment, robotics, VR, and AR applications.
- Conclusion: The representation supports real-time rendering and creative scene editing as directions for practical 3D modeling and tracking applications.The conclusion describes these implications as new avenues extending beyond the immediate results.
- Limitations: The method tracks only scene parts visible in the initial frame and cannot reconstruct new objects entering the scene.This is a design limitation of the representation.
- Limitations: The method requires a multi-camera setup and does not work off-the-shelf on monocular video.The paper identifies this constraint as a direction for future research.