Source-linked AI summary
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis
Zhan Li, Zhang Chen, Zhong Li, Yi Xu
TL;DR
Dynamic view synthesis must achieve photorealistic, high-resolution rendering in real time while remaining compact. The paper combines Spacetime Gaussians, splatted neural features, and guided Gaussian sampling, reporting state-of-the-art quality and speed with 8K rendering at 60 FPS in its lite version.
Problem
Dynamic view synthesis still faces the challenge of jointly achieving high-resolution photorealism, real-time rendering, and compact storage.
Method
The method enhances 3D Gaussians with temporal opacity and parametric motion and rotation, replaces spherical harmonics with splatted neural features, and guides Gaussian sampling with training error and coarse depth.
Results
8K rendering at 60 FPS: Experiments on real-world datasets show state-of-the-art rendering quality and speed while maintaining small model size.
Takeaways & Limitations
The representation supports compact dynamic-scene modeling across static, dynamic, and transient content while targeting high-resolution real-time view synthesis.
Takeaways & Limitations
The method cannot be trained on-the-fly and currently focuses on multi-view video inputs rather than monocular videos.
Abstract
from arXiv · showhide
Novel view synthesis of dynamic scenes has been an intriguing yet challenging problem. Despite recent advancements, simultaneously achieving high-resolution photorealistic results, real-time rendering, and compact storage remains a formidable task. To address these challenges, we propose Spacetime Gaussian Feature Splatting as a novel dynamic scene representation, composed of three pivotal components. First, we formulate expressive Spacetime Gaussians by enhancing 3D Gaussians with temporal opacity and parametric motion/rotation. This enables Spacetime Gaussians to capture static, dynamic, as well as transient content within a scene. Second, we introduce splatted feature rendering, which replaces spherical harmonics with neural features. These features facilitate the modeling of view- and time-dependent appearance while maintaining small size. Third, we leverage the guidance of training error and coarse depth to sample new Gaussians in areas that are challenging to converge with existing pipelines. Experiments on several established real-world datasets demonstrate that our method achieves state-of-the-art rendering quality and speed, while retaining compact storage. At 8K resolution, our lite-version model can render at 60 FPS on an Nvidia RTX 4090 GPU. Our code is available at https://github.com/oppo-us-research/SpacetimeGaussians.
1 OPPO US Research Center 2 Portland State University
Figure 1 presents the representation’s photorealistic quality, real-time high-resolution rendering, and compact model size, including 8K 6-DoF video at 66 FPS.
- 66 FPS: The lite-version model renders 8K 6-DoF video on an Nvidia RTX 4090 GPU.
1. Introduction
The paper targets dynamic view synthesis that combines photorealism, real-time high-resolution rendering, and compact storage. It introduces Spacetime Gaussians, splatted feature rendering, and guided Gaussian sampling to address this challenge.
- Dynamic-scene view synthesis must balance photorealistic quality, real-time high-resolution rendering, and compact model size.
- Spacetime Gaussians extend 3D Gaussians with temporal opacity and polynomial motion and rotation for static, dynamic, and transient content.
- Splatted feature rendering replaces spherical harmonics with compact neural features that model view- and time-dependent appearance through a lightweight MLP.
- Guided sampling uses training error and coarse depth to add Gaussians where distant, sparsely covered areas produce blurry rendering.
- 8K rendering at 60 FPS: Experiments report state-of-the-art rendering quality and speed while maintaining small model size.
2. Related Work
Related work spans image-based, neural, Gaussian, and dynamic novel-view synthesis methods. Dynamic approaches commonly use holistic temporal representations, while this paper focuses on multi-view video inputs rather than monocular videos.
- Novel View Synthesis: Image-based rendering methods use proxy geometry, depth, RGBD sensors, or volumetric voxels to synthesize novel views.
- Neural Scene Representations: Neural scene representations allocate features to volumes, textures, or point clouds, with NeRF using differentiable volume rendering and MLPs.
- Neural Scene Representations: 3D Gaussian Splatting uses anisotropic Gaussians and differentiable rasterization for fast, high-resolution static-scene rendering.
- Dynamic Novel View Synthesis: Dynamic view synthesis methods represent multiple frames jointly using temporal sharing, interpolation, time-variant latents, or factorized spacetime planes.
- Dynamic Novel View Synthesis: The paper focuses on dynamic representation from multi-view input videos, whereas monocular methods address sparse supervision with motion, scene-flow, or depth priors.
- Dynamic Novel View Synthesis: Concurrent dynamic methods include moving 3D Gaussians, 4D Gaussians, and image-based rendering, with reported trade-offs involving flickering artifacts and rendering quality.
3. Preliminary: 3D Gaussian Splatting
3D Gaussian Splatting represents static scenes with anisotropic Gaussians optimized through differentiable rasterization. Gaussians are projected into image space, depth-sorted, and volume-rendered, with spherical harmonics producing color.
- 3D Gaussian Representation: 3DGS optimizes anisotropic 3D Gaussians from multi-view images with known camera poses to represent static 3D scenes.
- 3D Gaussian Representation: Each Gaussian stores position, covariance, opacity, and spherical-harmonics coefficients; covariance decomposes into scaling and rotation.
- Differentiable Rasterization: Perspective projection approximates each 3D Gaussian as a 2D Gaussian using the viewing and projection matrices and the projective Jacobian.
- Differentiable Rasterization: After depth sorting, volume rendering combines projected Gaussian contributions to obtain each pixel’s color.
- Spacetime Extension: Figure 2 extends this pipeline with Spacetime Gaussians, temporal components, and splatted feature maps converted to color by an MLP.
4. Method
The method represents dynamic scenes with Spacetime Gaussians, renders compact view- and time-dependent features, and adds Gaussians using training errors and coarse depth.
- Spacetime Gaussians: Spacetime Gaussians extend 3D Gaussians with temporal opacity and time-conditioned position and rotation for dynamic-scene modeling.Temporal opacity models emerging or vanishing content, while parametric position and rotation model motion and deformation.
- Spacetime Gaussians: Temporal radial basis functions encode when each Gaussian is visible and its effective duration, with spatial opacity variation across Gaussians.
- Spacetime Gaussians: Polynomial trajectories represent Gaussian motion, while polynomial quaternions represent rotation; the implementation uses degree 3 for motion and degree 1 for rotation.The authors report that multiple short segments can represent complex and long motion, and keep scaling time-independent because conditioning it showed no quality improvement.
- Splatted Feature Rendering: Feature splatting stores 9-dimensional features instead of spherical-harmonic coefficients, rasterizes them, and uses a 2-layer MLP to produce RGB color.The feature channels encode base color, view-related information, and time-related information.
- Splatted Feature Rendering: 9 versus 48 parameters per Gaussian reduces representation size relative to 3-degree spherical harmonics while retaining fast rendering with a shallow, narrow MLP.The lite version drops the MLP and keeps only base features during training and rendering.
- Guided Sampling: Guided sampling selects high-error patches after loss stabilizes, samples rays through their centers, and uses coarse depth to narrow candidate depths.It targets distant, sparsely initialized regions and complements density-control splitting by adding Gaussians where existing coverage is sparse or absent.
5. Implementation Details
Implementation initializes Spacetime Gaussians from sparse structure-from-motion point clouds across timestamps and uses aggressive pruning to reduce model size.
- Spacetime Gaussians are initialized from structure-from-motion sparse point clouds across all available timestamps.The implementation uses Adam optimization and adapts splatting to different camera models in real-world datasets.
- 40–60 minutes trains a 50-frame sequence on a single NVIDIA A6000 GPU.More aggressive pruning than [41] is used to keep the model relatively small.
6. Experiments
The method is evaluated on three real-world dynamic-scene benchmarks using image-quality and rendering-speed metrics. It achieves strong quality and speed across datasets, while ablations identify the contributions of its components and training remains offline.
- Benchmarks and metrics: The evaluation covers Neural 3D Video, Google Immersive, and Technicolor datasets, with PSNR, DSSIM, LPIPS, and FPS as metrics.Metrics are averaged across six Neural 3D Video scenes except where noted.
- Neural 3D Video Dataset: 140 FPS is achieved on the Neural 3D Video Dataset, with the best LPIPS in all comparisons and the best PSNR/DSSIM in most cases.The qualitative results also show more vivid details and fewer rendering artifacts than baselines.
- Google Immersive Dataset: The method outperforms NeRFPlayer and HyperReel in both speed and quality on the Google Immersive Dataset.It is over 10 times faster than HyperReel; PSNR is 0.4 dB higher, with significant DSSIM and LPIPS improvements.
- Technicolor Dataset: The method achieves noticeable gains in quality and speed on the Technicolor Dataset.Evaluation uses five scenes at full resolution with a held-out camera from the 4×4 array.
- Ablation study: Removing temporal opacity, polynomial motion, polynomial rotation, or feature splatting reduces performance, while guided sampling matters especially for challenging scenes.Guided sampling has little effect on the textured Technicolor scenes because SfM points provide adequate coverage.
- Limitations: The representation renders quickly but cannot be trained on-the-fly, and the current method focuses on multi-view video inputs.The authors suggest advanced initialization for faster training and adaptation to monocular settings as future directions.
7. Conclusion
The paper presents Spacetime Gaussian Feature Splatting, combining spacetime Gaussians, splatted neural features, and guided Gaussian sampling for dynamic view synthesis. Experiments report state-of-the-art quality and speed with compact models.
- Spacetime Gaussians use temporal opacity and parametric motion and rotation to model complex 4D content.
- Splatted feature rendering replaces spherical harmonics with neural features and a lightweight MLP to improve compactness and model time-varying appearance.
- Guided Gaussian sampling uses training error and coarse depth to improve rendering quality in complex scenes.
- Experiments on real-world datasets show state-of-the-art rendering quality and speed while maintaining compact model size.
- The supplementary material provides comparisons, ablations, discussions, experiment details, per-scene results, and real-time demonstrations.
B. Comparisons with Concurrent Work
On the Neural 3D Video Dataset, the method is compared with concurrent dynamic Gaussian approaches and alternative sampling strategies. It combines strong quality, speed, compactness, and temporal consistency, while guided sampling reduces blur.
- The full model achieves the best PSNR, DSSIM, and LPIPS among comparable Gaussian Splatting methods.
- Both the lite model and Dynamic 3DGS run at over 300 FPS on the Neural 3D Video Dataset.
- The lite model uses about six percent of Dynamic 3DGS's model size and achieves 0.6 dB higher PSNR.
- Compared with Im4D and 4K4D, both the full and lite models achieve higher rendering quality and speed.
- Guided sampling sharpens distant, sparsely covered areas in both training and novel views.
- The proposed guided sampling strategy provides over 0.7 dB PSNR improvement over random and far-point Gaussian addition strategies.
C.2. Analysis on More Scenes
Additional ablations examine the method across scenes, features, rendering choices, sequence lengths, and motion modeling. The components generally improve quality, compactness, or training efficiency, while guided sampling has a stated limitation in difficult depth-uncertain regions.
- Analysis on More Scenes: The proposed components remain effective across additional scenes, camera setups, and scene content.
- Feature Ablation: Each feature component—f_base, f_dir, and f_time—boosts rendering quality.
- Rendering Comparison: Ours-Full provides better quantitative and visual quality with smaller model size than the spherical-harmonics variant Ours-SH.
- Rendering Comparison: The method has much smaller size and better rendering quality than per-frame-trained 3DGS.
- Longer Video Sequence: A single 300-frame model reduces per-frame training time by around 80% and model size by around 30% versus six 50-frame models, with comparable rendering quality.
- Guided Sampling Limitation: Guided sampling alleviates but cannot fully eliminate artifacts in challenging regions lacking accurate depth.
E. Experiment Details
The experiments use sparse reconstruction points, aggressive density control, and dataset-specific camera and rendering configurations. Training and rendering settings are reported alongside sequence, initialization, and model-size details.
- Camera Models: The method supports centered, uncentered, and distorted camera models across the evaluated real-world datasets.
- Initialization: Sparse point clouds from COLMAP initialize the models, with selected scenes using only 25 percent of SfM points from each frame.
- Feature Initialization: Features f_base and f_dir are initialized from SfM-point colors, while f_time is initialized to zeros.
- Rendering Setup: Rendering speed comparisons use an NVIDIA RTX 3090, while real-time demonstrations use an NVIDIA RTX 4090.
F. More Results
The method is evaluated across three real-world dynamic-scene datasets with quantitative and visual comparisons against several baselines. Results show strong per-scene performance and sharp detail preservation, while the lite model is slightly blurrier than the full model.
- The method outperforms other baselines on most scenes across the Neural 3D Video, Google Immersive, and Technicolor datasets.Per-scene quantitative comparisons are reported in Tables 11–13.
- Visual comparisons on the Neural 3D Video Dataset include the full and lite models against NeRFPlayer, HyperReel, K-Planes, MixVoxels-L, and Dynamic 3DGS.
- Visual comparisons on the Google Immersive Dataset compare both proposed models with NeRFPlayer and HyperReel.
- Visual comparisons on the Technicolor Dataset compare both proposed models with HyperReel.
- The method preserves sharp details with fewer artifacts, while the lite-version results are slightly blurrier than the full model.
- Per-scene Gaussian counts are reported for 50-frame models across the three datasets, with some prior methods reporting only part of the scenes.