Source-linked AI summary
4D-Rotor Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic Scenes
Yuanxing Duan, Fangyin Wei, Qiyu Dai, Yuhang He, Wenzheng Chen, Baoquan Chen
TL;DR
Dynamic novel-view synthesis is difficult because existing approaches struggle with temporal variation, diverse motions, and high-fidelity rendering. 4DRotorGS represents scenes with anisotropic 4D Gaussians, temporally slices them into renderable 3D Gaussians, and accelerates splatting with CUDA. The method reports unified static and dynamic reconstruction, high-quality rendering, and up to 583 FPS on an RTX 4090 GPU.
Problem
Dynamic NVS remains challenging because temporal dimensions and diverse motion patterns complicate efficient and accurate rendering.
Method
4DRotorGS represents dynamic scenes with 4D XYZT Gaussians, temporally slices them into 3D Gaussians, and uses rotor-based rotation with spatial-temporal regularization.
Results
4DRotorGS outperforms prior methods in reconstruction quality while reaching 583 FPS rendering on an RTX 4090 GPU.
Takeaways & Limitations
The framework provides a unified representation for static 3D and dynamic 4D reconstruction, including complicated dynamics such as abrupt appearances or disappearances.
Takeaways & Limitations
4D Gaussians remain difficult to constrain, and artifacts including floaters and inconsistent motions still exist.
Abstract
from arXiv · showhide
We consider the problem of novel-view synthesis (NVS) for dynamic scenes. Recent neural approaches have accomplished exceptional NVS results for static 3D scenes, but extensions to 4D time-varying scenes remain non-trivial. Prior efforts often encode dynamics by learning a canonical space plus implicit or explicit deformation fields, which struggle in challenging scenarios like sudden movements or generating high-fidelity renderings. In this paper, we introduce 4D Gaussian Splatting (4DRotorGS), a novel method that represents dynamic scenes with anisotropic 4D XYZT Gaussians, inspired by the success of 3D Gaussian Splatting in static scenes. We model dynamics at each timestamp by temporally slicing the 4D Gaussians, which naturally compose dynamic 3D Gaussians and can be seamlessly projected into images. As an explicit spatial-temporal representation, 4DRotorGS demonstrates powerful capabilities for modeling complicated dynamics and fine details--especially for scenes with abrupt motions. We further implement our temporal slicing and splatting techniques in a highly optimized CUDA acceleration framework, achieving real-time inference rendering speeds of up to 277 FPS on an RTX 3090 GPU and 583 FPS on an RTX 4090 GPU. Rigorous evaluations on scenes with diverse motions showcase the superior efficiency and effectiveness of 4DRotorGS, which consistently outperforms existing methods both quantitatively and qualitatively.
1 INTRODUCTION
Dynamic novel-view synthesis remains difficult because temporal variation and diverse motion patterns complicate efficient, accurate rendering. 4DRotorGS addresses this with explicit 4D Gaussian representations, temporal slicing, specialized rotation, regularization, and optimized CUDA rendering.
- Dynamic NVS remains challenging because temporal dimensions and diverse motion patterns complicate efficient, accurate rendering.
- Canonical-space deformation methods and joint spatial-temporal models can struggle with fine-detail preservation and challenging dynamics.
- 4DRotorGS models dynamic 3D scenes as 4D XYZT Gaussians whose temporal slices compose 3D Gaussians for image projection.
- Rotor-based 4D rotation supports spatial-temporal modeling while reducing to quaternion-based 3D rotation when the temporal dimensions are closed.
- Entropy and 4D consistency losses stabilize optimization by reducing floaters and encouraging consistent Gaussian motion.
- 583 FPS rendering is achieved for 1352×1014 videos on an RTX 4090 GPU, while 277 FPS is achieved on an RTX 3090 GPU.
2 RELATED WORK
Related work spans static and dynamic novel-view synthesis, including classical geometric methods, neural representations, and Gaussian-based renderers. 4DRotorGS extends the fast 3D Gaussian Splatting paradigm into a temporal 4D representation.
- Novel-view synthesis takes posed images as input and produces new appearance from a novel viewpoint.
- Static NVS: Static NVS methods include light-field interpolation, geometric reprojection, and neural radiance fields, trading capture density, memory, view range, quality, or speed.
- Dynamic NVS: Dynamic NVS accounts for temporal variation using changing geometry or neural representations with temporal inputs, latent codes, or deformation mechanisms.
- Gaussian-Based NVS: 3D Gaussian Splatting enables fast training and real-time static rendering through Gaussian split/clone operations and splatting-based rendering.
- Gaussian-Based NVS: 4DRotorGS lifts 3D Gaussians into 4D space to model dynamic scenes with explicit spatial-temporal structure.
3 METHOD
4DRotorGS lifts Gaussian Splatting into 4D XYZT space, uses rotor-based rotations and temporal slicing to produce dynamic 3D Gaussians for efficient rendering. Its design supports static and dynamic scenes while modeling temporal appearance, motion, and high-fidelity details.
- Temporal Slicing and Rendering: The renderer follows 3DGS by projecting sliced Gaussians to the image plane in depth order and optimizing image reconstruction with L1 and SSIM losses.The framework uses differentiable rasterization and back-propagated image gradients for adaptive Gaussian density control.
- 4D Gaussian Representation: 4DRotorGS represents dynamic scenes with 4D Gaussians defined by a 4D center, scaling, covariance, and rotor-based rotation.The 4D covariance is factorized into scaling and rotation components, with rotors providing the rotation representation.
- 4D Gaussian Representation: The rotor representation separates spatial and spatio-temporal rotation, while zeroing its temporal components reduces it to a quaternion for static 3D rotation.This allows the same framework to model both static and dynamic scenes and match 3DGS results on static scenes.
- Temporal Slicing and Rendering: At each timestamp, temporal slicing converts 4D Gaussians into 3D Gaussians that are projected to 2D through differentiable rasterization.The slicing operation introduces temporal decay and a motion term, allowing Gaussians to appear, grow, shrink, or disappear over time.
4 EXPERIMENTS
Experiments evaluate 4DRotorGS on real-world multi-view and synthetic monocular dynamic scenes, including abrupt motion, transparency, reflection, and sparse views. The method delivers strong rendering quality, high inference speed, and improved reconstruction through regularization and batch training.
- 4.1 Datasets: The evaluation uses the Plenoptic Video Dataset with six real-world multi-view scenes and the D-NeRF Dataset with eight synthetic monocular scenes.The Plenoptic videos include abrupt motion, transparent materials, and reflective materials; D-NeRF uses 400×400 resolution.
- 4.4 Results: 277 FPS at 1352×1014 resolution on an RTX 3090 GPU is over 10x faster than NeRF-based methods and over 2x faster than Gaussian-based methods.The method also achieves the highest PSNR of 31.62 versus the previous best 30.85, with 60 minutes average training time.
- 4.4.1 Evaluation on Plenoptic Video Dataset: The method reconstructs finer dynamic details and sharper static regions than baselines on the Plenoptic Video Dataset.Qualitative comparisons report sharper human heads and hand regions under frequent or fast motion, as well as a sharper hook.
- 4.4.2 Evaluation on D-NeRF Dataset: On D-NeRF, the method achieves the best rendering quality among the other methods at 1258 FPS, with training taking around 5 minutes.The qualitative results report reduced floaters and clearer details such as bulldozer blades, fingers, helmet patterns, facial features, teeth, and Hook’s hand.
- 4.5 Ablation Studies: Entropy loss removes floaters and reduces the number of points by an order of magnitude while maintaining overall PSNR and SSIM quality.Its authors recommend using it for opaque surfaces and sparse views because it can degrade PSNR on the dense-view Plenoptic dataset with many transparent objects.
- 4.5 Ablation Studies: 4D Consistency loss improves motion consistency and rendering quality, while batch training reduces gradient noise and improves quality and geometry consistency.Batch training is especially beneficial in sparse-view settings because it jointly optimizes over multiple views.
5 CONCLUSION
The paper presents 4D Gaussian Splatting as a unified framework for static and dynamic reconstruction, combining high-quality modeling with real-time rendering. It also notes that increased dimensionality still causes floaters and inconsistent motions despite the proposed regularizers.
- 5 CONCLUSION: 4D Gaussian Splatting enables high-quality 4D dynamic scene modeling and provides a unified framework for 3D static and 4D dynamic reconstruction.The method achieves 583 FPS rendering speed on an RTX 4090 GPU.
- 5 CONCLUSION: Increased dimensionality makes 4D Gaussians difficult to constrain, causing artifacts such as floaters and inconsistent motions.Entropy loss and 4D consistency loss mitigate these issues, but artifacts still remain.
Lego Jumping Jacks Stand Up T-Rex Hook
The method compares favorably with NeRF- and Gaussian-based baselines in qualitative D-NeRF results, while defining 4D rotor representations and temporally slicing 4D Gaussians into 3D Gaussians for rendering.
- Qualitative Comparison: Our method reconstructs D-NeRF scenes with less noise and more detail than TiNeuVox, Deformable4DGS, and RealTime4DGS.Highlighted examples include the bulldozer blade, cuffs and hands, T-Rex teeth, and helmet and facial details.
- 4D Rotor Representation: A 4D rotor represents rotation using eight components defined over scalar, bivector, and pseudoscalar basis elements.The four orthonormal basis axes correspond to the x, y, z, and t dimensions.
- 4D Rotor Normalization: Rotor normalization enforces the two conditions derived from rr†, then produces coefficients suitable for 4D rotation.The procedure applies a gradient-based update and subsequent scaling before obtaining the normalized rotor.
- Numerical Stability: Block-matrix computation avoids directly inverting the covariance sub-block when calculating the sliced 3D covariance.This approach leverages computationally feasible sub-blocks of the 4D covariance matrix.
B.1 Datasets Details
The evaluation uses real-world and synthetic datasets spanning sparse views, monocular input, multi-camera capture, fast motion, and challenging appearance changes.
- Plenoptic Video Dataset: The Plenoptic Video Dataset contains six real-world multi-view scenes with 17–20 training views and one central evaluation view.Its scenes include sudden flames, moving shadows, translucent materials, and reflective materials.
- D-NeRF Dataset: D-NeRF is a synthetic monocular-video dataset with eight scenes, where each timestamp provides only a single camera viewpoint.Scenes contain 50–200 training images, 10 or 20 validation images, and 20 testing images.
- HyperNeRF Dataset: HyperNeRF is captured with two cameras per scene, using 163–512 photos from each camera and evaluating four scenes.The evaluation follows Deformable4DGS settings.
- Total-Recon Dataset: Total-Recon contains 11 RGBD sequences with cats, a dog, and humans across four indoor environments.Its sequences are subsampled at 10 FPS, span 550–641 frames, and feature sparse perspectives and varied motion.
- Evaluation: On HyperNeRF, our method surpasses the baseline in PSNR for most scenes and on average.Table 4 compares against the state-of-the-art method on that dataset.
B.2 Additional Implementation Details
Implementation details specify Gaussian initialization, temporal ranges and scales, optimization settings, and learning-rate schedules for D-NeRF and Plenoptic Video experiments.
- Spatial and Temporal Initialization: The method initializes 100,000 Gaussian positions by random sampling inside dataset-specific spatial bounding boxes.The Plenoptic box follows the range of SfM points, while D-NeRF uses box dimensions [−1.3, 1.3]^3.
- Spatial and Temporal Initialization: Gaussian time means are uniformly initialized over [0, 1] for D-NeRF and [0, 10] for the Plenoptic Video Dataset.The corresponding initial time scales are 0.1414 and 1.414.
- Optimization: Training uses Adam with separate learning rates for positions, times, scales, rotation, spherical harmonics, and opacity.The listed rates range from 1.25e−4 for high-degree SH to 0.05 for opacity, with exponential decay for position-related rates.
B.3 Additional Real-World Dataset Results
Additional real-world evaluations report stronger reconstruction quality than baselines on HyperNeRF and Total-Recon, while ablations and comparisons examine entropy loss, consistency, and rotation choices.
- HyperNeRF Results: On HyperNeRF, our method provides superior rendering quality across most scenes and on average compared with the baseline.Qualitative results show more detail in static regions and reduced blurring.
- Total-Recon Results: In Total-Recon’s extremely sparse-view and fast-motion setting, our method outperforms RGB-only baselines by a large margin and is on par with RGB-plus-depth variants.The best-performing Total-Recon baseline additionally uses mask, depth, and flow supervision with multi-stage training.
- Total-Recon Qualitative Results: Sparse views and rapidly changing camera poses substantially reduce rendering quality in Total-Recon scenes.Regions observed across multiple frames remain relatively clear, whereas fast dog motion with camera following severely affects rendering.
- Entropy Loss: Entropy loss pushes Gaussian opacities toward extreme values, densifying Gaussians near surfaces and removing floaters without improving PSNR.The paper uses it to improve sparse-view reconstruction and distinguishes this role from prior surface-recovery use.
- Rotation Comparison: Rotor-based and dual-quaternion 4D rotations show no significant difference in training loss across three runs per representation.The comparison is reported in Figure 11 using RGB image loss curves.
- Total-Recon Results: Our method achieves the highest PSNR among RGB-only methods on Total-Recon and results comparable with baselines using added depth supervision.Table 5 reports the comparison with HyperNeRF, D2NeRF, and Total-Recon.
B.5 Additional Ablation Studies
The ablation compares rotor-based and dual-quaternion 4D rotations, finding equivalent testing PSNR while highlighting additional interpretability and modeling flexibility for rotors.
- Testing PSNRs showed no difference between rotor-based and dual-quaternion rotation representations.All other settings were kept the same as in the main-paper Tab. 2 experiments.
- Rotor-based rotation unifies static and dynamic scene modeling.
- Rotor temporal-component magnitude indicates how still a Gaussian is, supporting potential tracking or static/dynamic content separation.
B.6 Influence of Background Colors on D-NeRF Dataset.
Background color affects D-NeRF rendering quality, with black generally favored, especially for darker foregrounds. The method outperforms baselines under both black and white backgrounds.
- D-NeRF experiments compare black and white backgrounds because the dataset provides synthetic images without backgrounds.Deformable3DGS and RealTime4DGS use black backgrounds, while other methods use white backgrounds.
- PSNR 34.26 with black backgrounds exceeded PSNR 33.06 with white backgrounds for the proposed method.
- White backgrounds performed better on Jumping Jacks, Bouncing Balls, and Lego, whose foregrounds are brighter.
- The method outperformed baselines on both background colors.The table reports per-scene and average PSNR for each method using their default background settings.