PaperlayerView exact revision on arXiv

Source-linked AI summary

4D Gaussian Splatting for Real-Time Dynamic Scene Rendering

Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, Xinggang Wang

arXiv:2310.08528v3cs.CVcs.GR

TL;DR

Dynamic scenes require accurate modeling of complex motion without sacrificing training, storage, or rendering efficiency. 4D-GS combines canonical 3D Gaussians with 4D neural voxels and a lightweight deformation decoder, achieving real-time rendering at high resolution while maintaining comparable or better quality than prior methods.

  • Problem

    Dynamic-scene rendering must model complex motions while retaining high training, storage, and rendering efficiency.

  • Method

    4D-GS represents dynamic scenes with canonical 3D Gaussians, 4D neural voxels, decomposed voxel encoding, and a lightweight MLP for timestamped Gaussian deformations.

  • Results

    82 FPS at 800×800 on synthetic datasets and 30 FPS at 1352×1014 on real datasets are achieved with comparable or superior performance to prior SOTA methods.

  • Takeaways & Limitations

    4D-GS supports real-time dynamic-scene rendering while retaining low storage consumption and enabling potential 4D tracking and editing.

  • Takeaways & Limitations

    Large motions, missing background points, inaccurate camera poses, monocular joint motion, and urbanscale reconstruction remain challenging.

Abstract

from arXiv · show

Representing and rendering dynamic scenes has been an important but challenging task. Especially, to accurately model complex motions, high efficiency is usually hard to guarantee. To achieve real-time dynamic scene rendering while also enjoying high training and storage efficiency, we propose 4D Gaussian Splatting (4D-GS) as a holistic representation for dynamic scenes rather than applying 3D-GS for each individual frame. In 4D-GS, a novel explicit representation containing both 3D Gaussians and 4D neural voxels is proposed. A decomposed neural voxel encoding algorithm inspired by HexPlane is proposed to efficiently build Gaussian features from 4D neural voxels and then a lightweight MLP is applied to predict Gaussian deformations at novel timestamps. Our 4D-GS method achieves real-time rendering under high resolutions, 82 FPS at an 800$\times$800 resolution on an RTX 3090 GPU while maintaining comparable or better quality than previous state-of-the-art methods. More demos and code are available at https://guanjunwu.github.io/4dgs/.

1. Introduction

Dynamic novel-view synthesis must model complex spatial and temporal motion from sparse inputs while remaining efficient. 4D-GS addresses this by deforming one canonical set of 3D Gaussians over time, achieving real-time rendering with high quality and low overhead.

  • Dynamic-scene rendering is challenging because complex motions must be modeled from spatially and temporally sparse input.
  • NeRF-based methods reduce training time but retain non-negligible rendering latency.
  • 3D-GS enables real-time rendering through explicit 3D Gaussians and differentiable splatting, but it primarily targets static scenes.
  • 4D-GS models Gaussian motion and shape changes with a Gaussian deformation field while maintaining only one canonical set of 3D Gaussians.
  • 82 FPS at 800×800 on synthetic datasets and 30 FPS at 1352×1014 on real datasets are achieved with comparable or superior performance to prior SOTA methods.

2. Related Works

Dynamic neural-rendering methods range from deformation and flow-based NeRFs to decomposed voxel representations and point-cloud approaches. 4D-GS extends explicit Gaussian splatting to dynamic scenes with a compact deformation network rather than storing separate Gaussian states for every timestamp.

  • 2.1. Novel View Synthesis: Deformation-based NeRF methods map sampled ray points toward canonical space before volume rendering.
  • 2.1. Novel View Synthesis: Time-aware volume rendering computes each point’s features directly without changing the rendering path.
  • 2.1. Novel View Synthesis: Dynamic novel-view synthesis methods include deformation-based, flow-based, and time-aware neural-rendering approaches.
  • 2.2. Neural Rendering with Point Clouds: 3D-GS uses explicit point-based splatting for real-time novel-view rendering, while related point-cloud methods target dynamic-scene rendering.
  • 2.2. Neural Rendering with Point Clouds: Dynamic3DGS stores each Gaussian at every timestamp with O(tN) memory, whereas 4D-GS uses O(N + F) memory for Gaussians and its deformation network.
  • 2.2. Neural Rendering with Point Clouds: 4D-GS models Gaussian motion with a compact deformation network to support efficient training and real-time rendering.

3. Preliminary

The preliminary section contrasts explicit 3D Gaussian splatting with deformation-field formulations for dynamic NeRFs. 3D Gaussians support differentiable rendering, while dynamic NeRFs map spatiotemporal points to density and color through learned functions.

  • 3D Gaussian Splatting: 3D Gaussians represent scenes as point-like primitives with centers and covariance matrices decomposed into scaling and rotation.Their attributes also include spherical-harmonic color, opacity, rotation, and scaling factors.
  • 3D Gaussian Splatting: Differential splatting projects 3D Gaussians into camera planes using viewing and projective-transformation Jacobian matrices.The transformed covariance supports differentiable rendering of novel views.
  • Dynamic NeRFs with Deformation Fields: Dynamic NeRFs map spatial points, view directions, timestamps, and optional codes to color and density.The mapping uses an 8D input space and 4D output space, with optional codes modeling topological or appearance changes.
  • Dynamic NeRFs with Deformation Fields: Deformation-based dynamic NeRFs first estimate a world-to-canonical displacement and then use a network to predict volume density and view-dependent RGB color.Rendering follows the learned deformation before the NeRF pipeline evaluates each ray.
  • Dynamic NeRFs with Deformation Fields: 4D Gaussian splatting instead computes canonical-to-world Gaussian motion directly at time t and then applies differential splatting.This design also supports backward flow computation and 3D Gaussian tracking.

4. Method

The method combines canonical 3D Gaussians with a spatial-temporal deformation network that produces timestamp-specific Gaussians for differential splatting. A decomposed multi-resolution encoder and multi-head decoder support compact dynamic representation and optimization.

  • 4.1. 4D Gaussian Splatting Framework: 4D-GS uses 3D Gaussians and a Gaussian deformation field network to render timestamp-specific novel views with differential splatting.The rendered image is S(M, G′), where G′ is obtained by deforming the canonical Gaussians.
  • 4.2. Gaussian Deformation Field Network: The deformation field encodes spatial-temporal Gaussian features and decodes them into Gaussian deformations at time t.The encoder H computes features from G and t, while decoder D predicts ΔG.
  • 4.2. Gaussian Deformation Field Network: The encoder decomposes a 4D neural voxel into six multi-resolution planes covering spatial and temporal coordinate pairs.The planes represent (x,y), (x,z), (y,z), (x,t), (y,t), and (z,t) relationships.
  • 4.2. Gaussian Deformation Field Network: Bilinear interpolation queries plane features, and a tiny MLP merges them into a Gaussian feature used by the deformation decoder.The resulting feature is fd = ϕd(fh).
  • 4.2. Gaussian Deformation Field Network: Separate decoder heads predict position, rotation, and scaling changes, which are added to the original Gaussian parameters.The deformed Gaussians retain position, scaling, rotation, opacity, and color attributes.
  • 4.3. Optimization: The optimization warm-up trains 3D Gaussians for 3000 iterations before rendering with the deformation network.The method uses L1 color loss together with a grid-based total-variation loss.

5. Experiment

Experiments evaluate 4D-GS on synthetic and real-world dynamic-scene datasets, measuring rendering quality, speed, training time, and storage. The method achieves strong quality and efficiency, while ablations identify the roles of its encoder, deformation decoder, and initialization.

  • Experimental Settings: Experiments use synthetic D-NeRF scenes and real-world HyperNeRF and Neu3D datasets under distinct camera and frame settings.Synthetic scenes contain 50–200 dynamic frames; HyperNeRF uses one or two cameras, while Neu3D uses 15–20 static cameras.
  • Experimental Settings: Evaluation covers PSNR, LPIPS, SSIM variants, FPS, training time, and storage against multiple state-of-the-art methods.Rendering speed and storage for several baselines are estimated from official implementations.
  • Results: 4D-GS achieves the highest rendering quality on the synthetic dataset while also providing fast rendering, low storage consumption, and short convergence time.The comparison uses 800×800 rendering resolution.
  • Results: 4D-GS delivers comparable rendering quality, fast convergence, and strong free-view rendering speed on real-world datasets.Some competing methods converge slowly, capture intricate details poorly, or limit free-view speed and storage; Im4D requires multi-camera setups.
  • Ablation Study: Removing the spatial-temporal encoder reduces rendering quality despite minimal memory costs, because a shallow deformation MLP cannot model complex deformations adequately.The encoder retains spatial and temporal information while reducing storage relative to a purely explicit method.
  • Ablation Study: Modeling Gaussian motion and shape changes is necessary for fitting dynamic scenes and preserving details during stretching and twisting.The deformation decoder separates Gaussian changes into motion, rotation, and scaling components.
  • Ablation Study: Removing 3D Gaussian warm-up harms rendering quality because initialization improves convergence stability and focuses deformation learning on dynamic regions.Warm-up also helps avoid numerical errors in the deformation network.
  • Discussions: 90 FPS is attainable below 30,000 rendered Gaussians at 800×800 on one RTX 3090, subject to balancing resolution, representation size, network capacity, and hardware.The speed analysis is conducted on the synthetic dataset.

6. Conclusion

The conclusion presents 4D-GS as a real-time dynamic-scene representation combining Gaussian deformation modeling with spatial-temporal connections. It also identifies potential applications in tracking and editing.

  • Conclusion: 4D-GS combines an efficient deformation-field network with a spatial-temporal encoder connecting adjacent Gaussians.The framework models Gaussian motion and shape deformation for real-time dynamic-scene rendering.
  • Conclusion: Connections between Gaussians produce more complete deformed geometry and effectively avoid avulsion.The explicit representation also supports potential 4D object tracking and editing.

A. Appendix

The appendix provides additional ablation studies and hyperparameter settings, then discusses limitations of 4D-GS.

  • Appendix: The supplementary material expands the hyperparameter settings, ablation studies, and limitations discussion.

A.1. Hyperparameter Settings

The appendix details dataset-specific training settings and implementation choices, including HexPlane resolutions, learning rates, batch size, and decoder configuration. It also records dataset-dependent convergence behavior and a HyperNeRF optimization limitation.

  • Hyperparameter Settings: The multi-resolution HexPlane module starts at resolution 64 and is upsampled by factors of 2 and 4, while the deformation decoder uses a tiny MLP.Learning rates are separately specified for the HexPlane module and decoder, and batch size is 1.
  • Dataset-Specific Settings: Synthetic D-NeRF scenes use simpler structure settings, including one HexPlane upsampling rate, because each timestamp contains a single monocular image without background.
  • Dataset-Specific Settings: Neu3D uses dense SfM point clouds downsampled below 100k points, and 14,000 iterations suffice for high-quality images because scene motion is small.
  • Dataset-Specific Settings: HyperNeRF uses upsampling resolutions and decoder hidden dimension 128, but deformation fields can fall into local minima linking camera and object motion.The authors defer separating these relationships to future work.

A.2. More Ablation Studies

The ablations examine editing, deformation choices, failure cases, and feature encoding in 4D-GS. They show that removing position deformation limits motion modeling, while large motions and color or opacity changes remain challenging.

  • Editing with 4D Gaussians: 4D-GS supports editing dynamic scenes, but scene rotations require adjusting Gaussian rotation quaternions and scaling coefficients.Interpolation is also needed to enlarge or reduce 4D Gaussians.
  • Position Deformation: Removing the position-deformation head can still model object motion by scaling small dynamic-part Gaussians at selected timestamps.
  • Position Deformation: The position-deformation ablation models only coarse object motion and loses potential for tracking.
  • 3D Gaussians may fail to model large motions or dramatic scene changes, producing blurred results in such cases.The paper identifies exploring stronger priors as future work.
  • Color and Opacity’s Deformation: Color and opacity deformation uses two MLP decoders for fluid or non-rigid motion but may cause irrational shape changes in novel views.Surface Gaussians may move inconsistently with neighboring Gaussians.
  • Spatial-temporal Structure Encoder: The spatial-temporal encoder stores spatial structure in spatial planes and integrated scene motion in temporal voxel grids.

A.3. More Discussions

The discussions identify sparse monocular observations and large motions as important boundaries for 4D-GS. Novel-view failure can arise from overfitting, while short training times are insufficient for some large-motion scenes.

  • Monocular Dynamic Scene Novel View Synthesis: In monocular dynamic scenes, sparse camera poses and timestamps can cause overfitting to training images and failure on novel views.Depth supervision or optical-flow priors may be needed.
  • Large Motion Modeling with Multi-Camera Settings: With multi-camera input, 4D-GS performs relatively well when motion is small and views densely sample the scene.
  • Large Motion Modeling with Multi-Camera Settings: 4D-GS cannot fit large-motion sports scenes well within short training times.The paper suggests online training or information from other views as possible approaches.
  • Large Motion Modeling with Monocular Settings: In monocular settings, 4D-GS uses a deformation field network to model scene motion.
Loading 2310.08528v3…