Source-linked AI summary
Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting
Zeyu Yang, Hongye Yang, Zijie Pan, Li Zhang
TL;DR
Dynamic novel view synthesis requires reconstructing complex scene structure and motion from 2D images without relying on poorly scaling deformation models. The paper represents the full spatio-temporal volume with optimized 4D Gaussian primitives and tailored rendering, reporting state-of-the-art quality with real-time, high-fidelity video synthesis. Its main scope boundary is that distant static backgrounds are difficult to capture when initial points are absent.
Problem
Dynamic scene reconstruction from 2D images is limited by difficulties learning spatial-temporal structure through a 6D plenoptic function and by the poor scalability of explicit deformation modeling.
Method
4DGS approximates a dynamic scene’s spatio-temporal 4D volume with 4D Gaussian primitives that jointly model geometry, motion, and time-evolving view-dependent appearance.
Results
Across synthetic and real, monocular and multi-view benchmarks, 4DGS reports superior visual quality and efficiency, including real-time high-fidelity video synthesis for complex dynamic scenes.
Takeaways & Limitations
A unified 4D Gaussian representation supports photorealistic dynamic novel-view synthesis while remaining efficient for real-time rendering.
Takeaways & Limitations
Without initial points, the method has difficulty capturing distant background areas, including static regions.
Abstract
from arXiv · showhide
Reconstructing dynamic 3D scenes from 2D images and generating diverse views over time is challenging due to scene complexity and temporal dynamics. Despite advancements in neural implicit models, limitations persist: (i) Inadequate Scene Structure: Existing methods struggle to reveal the spatial and temporal structure of dynamic scenes from directly learning the complex 6D plenoptic function. (ii) Scaling Deformation Modeling: Explicitly modeling scene element deformation becomes impractical for complex dynamics. To address these issues, we consider the spacetime as an entirety and propose to approximate the underlying spatio-temporal 4D volume of a dynamic scene by optimizing a collection of 4D primitives, with explicit geometry and appearance modeling. Learning to optimize the 4D primitives enables us to synthesize novel views at any desired time with our tailored rendering routine. Our model is conceptually simple, consisting of a 4D Gaussian parameterized by anisotropic ellipses that can rotate arbitrarily in space and time, as well as view-dependent and time-evolved appearance represented by the coefficient of 4D spherindrical harmonics. This approach offers simplicity, flexibility for variable-length video and end-to-end training, and efficient real-time rendering, making it suitable for capturing complex dynamic scene motions. Experiments across various benchmarks, including monocular and multi-view scenarios, demonstrate our 4DGS model's superior visual quality and efficiency.
1 INTRODUCTION
Dynamic novel view synthesis must capture both spatial structure and temporal motion from 2D observations, while existing implicit and deformation-based approaches face representation or scalability limits. 4DGS addresses this by modeling the spatio-temporal volume with 4D Gaussian primitives and a dedicated rendering pipeline.
- Dynamic scene reconstruction is difficult because object motion and temporal dynamics complicate modeling from 2D images, especially monocular videos.
- Implicit methods learn a 6D plenoptic function with MLPs, grids, or low-rank structures, but their ability to capture spatial-temporal correlations depends on the chosen representation.
- The framework conceptually maps a 4D Gaussian to a 2D planar Gaussian through temporal orthogonal projection and spatial perspective projection.
- Deformation-based methods explicitly model motion and can exploit space-time correlations, yet they have reduced flexibility and scalability in complex real-world scenes.
- 4DGS approximates a dynamic scene’s underlying spatio-temporal 4D volume using a collection of 4D Gaussians with coherent space-time modeling.
- 4D rotations capture intrinsic scene motion, while Spherindrical Harmonics model time-evolving, view-dependent appearance.
- Experiments across synthetic and real, monocular and multi-view datasets report superior visual quality and efficiency, including photorealistic high-resolution video beyond real-time speeds.
2 RELATED WORK
Prior dynamic novel view synthesis methods either learn implicit or explicit 6D representations without direct motion modeling or explicitly model deformation of static structures. These approaches face coupling, scaling, or topology-prior limitations that motivate 4D Gaussian representations.
- Novel view synthesis for static scenes: Static-scene novel view synthesis methods improved rendering quality and speed, with 3D Gaussian Splatting combining high-fidelity synthesis and real-time rasterization.
- Novel view synthesis for dynamic scenes: Dynamic novel view synthesis must capture correlations across timesteps and cannot be treated as independently reconstructing each static frame.
- Novel view synthesis for dynamic scenes: One research line learns a 6D plenoptic function through implicit or explicit structures without directly modeling scene motion, but struggles with parameter coupling.
- Dynamic 3D Gaussians: Another research line models continuous motion or deformation of static structures, but such approaches can impose assumptions that limit flexibility or scalability.
- Dynamic 3D Gaussians: Dynamic 3D Gaussian methods commonly optimize canonical-space Gaussians together with deformation fields or factorized neural trajectories, incorporating priors about scene structure.
3 METHOD
4DGS represents dynamic scenes with coherent 4D Gaussian primitives whose geometry, temporal behavior, and appearance are optimized jointly for novel-view rendering. Its pipeline conditions each 4D Gaussian on time, projects the spatial component to a 2D splat, and blends it with time-dependent color.
- Rendering pipeline: At time t, the 4D Gaussian is decomposed into a conditional 3D Gaussian and a marginal 1D temporal Gaussian before spatial projection.The conditional distribution yields a planar Gaussian after projection, while the marginal temporal Gaussian contributes during rasterization and color-opacity accumulation.
- Rendering pipeline: The rendering routine projects the conditional 3D Gaussian to a 2D splat and blends its planar density, temporal marginal, opacity, and view-dependent color.This adapts the tile-based rasterizer used for 3D Gaussian Splatting to dynamic rendering.
- 4D Gaussian representation: Each 4D Gaussian has a four-scalar mean (µx, µy, µz, µt), providing a joint spatial-temporal representation.Its covariance uses diagonal scaling elements (sx, sy, sz, st) and a 4D rotation constructed from paired quaternion rotations.
- Time-evolving appearance: 4DGS models dynamic appearance with 4D spherindrical harmonics, allowing each Gaussian’s view-dependent color to evolve over time.The representation combines spherical harmonics with one-dimensional basis functions; Fourier series are used for computational convenience.
4 EXPERIMENTS
Experiments evaluate 4D Gaussian Splatting on multi-view real and monocular synthetic dynamic scenes, including benchmark comparisons, qualitative results, motion capture, and ablations. The method achieves strong rendering quality and efficiency while its unconstrained space-time representation, rotations, appearance modeling, and temporal densification contribute to performance.
- Datasets: The evaluation covers the Plenoptic Video and D-NeRF datasets, spanning real multi-view scenes and monocular synthetic videos.Plenoptic Video contains six real-world scenes; D-NeRF contains eight synthetic monocular videos.
- Results on multi-view real scenes: On Plenoptic Video, the method surpasses previous methods in rendering quality and speed and is the only benchmark method providing real-time high-quality synthesis.Qualitative comparisons particularly highlight dynamic regions in the flame salmon scene.
- Results on monocular synthetic videos: On monocular synthetic videos, the method surpasses all competing methods without introducing architectural priors for scene topology.The authors attribute this result to efficient information exchange across time steps.
- Ablation and analysis: Unconstrained 4D Gaussians clearly outperform a constrained baseline that separates spatial and temporal dimensions.The comparison uses two representative real scenes.
- Ablation and analysis: 4D rotation captures 3D motion by producing displacement, whose frame-to-frame projections are rendered as estimated optical flow.The evaluation visualizes these flows on Plenoptic Video test views without explicit motion supervision or regularization.
- Ablation and analysis: Ablations show that removing 4D Spherindrical Harmonics lowers rendering quality, while temporal densification improves results.Temporal densification samples new positions from the complete 4D Gaussian.
5 CONCLUSION
The paper represents dynamic scenes as collections of 4D Gaussians aligned with the scene’s spatio-temporal volume. Across diverse scenes, this representation achieves high rendering quality and substantial speed improvements, including real-time high-fidelity synthesis for complex real-world dynamics.
- Conclusion: The method approximates a dynamic scene’s underlying spatio-temporal 4D volume with a collection of 4D Gaussians.The representation aligns the rendering process with dynamic-scene imaging.
- Conclusion: Experiments across diverse scenes demonstrate state-of-the-art rendering quality and substantial speed improvements over existing alternatives.The reported scope includes complex, real-world dynamic scenes.
- Conclusion: The method is presented as the first capable of real-time, high-fidelity video synthesis for complex real-world dynamic scenes.
A LIMITATIONS
The method struggles to reconstruct distant background areas when initial points are absent, limiting convenience in some scenes despite strong foreground recovery.
- Without initial points, the approach has difficulty capturing distant background areas, including static regions.Spherical initialization can partially mitigate this issue.
- Spherical initialization learns a background sphere of Gaussians rather than necessarily recovering correct geometry.
- These background-reconstruction issues may constrain the method’s convenience in some scenes.
B PROOFS
The proofs establish that the unnormalized Gaussian can be treated as a special probability distribution and that Gaussian conditional-probability formulas extend to the paper’s formulations.
- The appendix proves properties used when treating the unnormalized Gaussian in Equation 1 as a special probability distribution.
- The proof of Equation 12 relies on the conditional-probability formula for the normalized Gaussian.
- The result extends to p(x, y, z, t) = p(t)p(x, y, z|t).
C ADDITIONAL QUANTITATIVE RESULTS AND VISUALIZATIONS
Additional Plenoptic Video results report per-scene PSNR breakdowns and qualitative evidence that 4DGS captures subtle motion and large body movements with high fidelity.
- Table 4 provides a per-scene PSNR breakdown on the Plenoptic Video dataset.The table identifies the best, second, and third results for each scene.
- 4DGS faithfully captures subtle cookware movement across different timesteps.
D RESULTS IN THE URBAN SCENES
On urban scenes, the method uses calibrated multi-camera images, LiDAR supervision, and a unified representation to render dynamic and static regions with high fidelity.
- Urban-scene reconstruction has applications in autonomous-driving-model training and offline perception.
- Urban-scene reconstruction is evaluated on Waymo segments containing dynamic objects, calibrated images from five cameras, and LiDAR point clouds.Three frontal cameras provide the images used following prior work.
- Sparse LiDAR depth supervision and deactivated temporal 4DSH coefficients help mitigate overfitting under sparse observation.A cube-map background models the sky at infinite distance.
- 4D Gaussian Splatting achieves high-fidelity rendering for both dynamic and static urban regions.
- Figure 7 evaluates novel-view synthesis by holding out one frame from every ten frames as a test view.The method represents dynamic and static regions jointly rather than relying on 3D bounding boxes and dynamic-object segmentations.
E THE TEMPORAL CHARACTERISTIC OF 4D GAUSSIANS
4D Gaussians maintain restrained growth in total representation size as videos lengthen, while temporal locality keeps per-frame rendering costs nearly stable. Their temporal variance also supports motion fitting and reduces artifacts compared with removing 4D rotations.
- Temporal distribution: Anisotropic Gaussians help avoid intractable growth with video length, motivating an initial time scaling set to half the scene duration.This uses the anisotropic characteristic of Gaussian primitives to extend temporal support.
- Temporal distribution: Background Gaussians with large temporal variance remain active across the scene’s full duration, restraining growth in the total number of Gaussians as video length increases.Their temporal statistics also delineate dynamic and static regions.
- Rendering efficiency: Filtering Gaussians by marginal temporal probability before frustum culling keeps the number participating in each frame nearly constant, stabilizing rendering speed for longer videos.The filtering adds negligible time cost before rasterization.
- Rendering efficiency: For videos with hundreds of frames, the total number of fitted 4D Gaussians is not essentially larger than 3D Gaussians fitted on one frame, while average per-frame usage remains stable.Figure 10 compares total fitted Gaussians with those actually involved in rasterization under different video lengths.
- Effect of 4D rotation: Removing 4D rotations produces poorer wheel-rim reconstruction, fewer active rendering Gaussians, and more flicker and blur despite fitting more Gaussians overall.The No-4DRot variant has less temporal variance, impairing motion fitting and information exchange between successive frames.