PaperlayerView exact revision on arXiv

Source-linked AI summary

GauFRe: Gaussian Deformation Fields for Real-time Dynamic Novel View Synthesis

Yiqing Liang, Numair Khan, Zhengqin Li, Thu Nguyen-Phuoc, Douglas Lanman, James Tompkin, Lei Xiao

arXiv:2312.11458v3cs.CV

TL;DR

Monocular dynamic-scene reconstruction is underconstrained because geometry and motion must be estimated together. GauFRe uses canonical 3D Gaussians, a forward-warping deformation field, and a separate static component; it reports competitive results and 96 FPS real-world rendering on an RTX 3090.

  • Problem

    Monocular dynamic-scene reconstruction is ill-posed because geometry and motion must be estimated simultaneously, including continual non-rigid deformation.

    Source: §1 · pp. 1–2

  • Method

    GauFRe forward-warps canonical Gaussian primitives over time and combines them with a separate static Gaussian set, optimized end-to-end using self-supervised rendering loss.

    Source: §1 · p. 2

  • Results

    GauFRe achieves competitive qualitative and quantitative results against previous state-of-the-art methods, with 96 FPS real-world rendering on an RTX 3090 GPU.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The Gaussian-based approach combines competitive reconstruction results with real-time rendering for real-world scenes.

    Source: §5 · p. 8

    Gaussian methods can overfit sparse observations, and dynamic Gaussian Splatting may struggle with large motions or thin structures.

    Source: §5 · p. 8

Abstract

from arXiv · show

We propose a method that achieves state-of-the-art rendering quality and efficiency on monocular dynamic scene reconstruction using deformable 3D Gaussians. Implicit deformable representations commonly model motion with a canonical space and time-dependent backward-warping deformation field. Our method, GauFRe, uses a forward-warping deformation to explicitly model non-rigid transformations of scene geometry. Specifically, we propose a template set of 3D Gaussians residing in a canonical space, and a time-dependent forward-warping deformation field to model dynamic objects. Additionally, we tailor a 3D Gaussian-specific static component supported by an inductive bias-aware initialization approach which allows the deformation field to focus on moving scene regions, improving the rendering of complex real-world motion. The differentiable pipeline is optimized end-to-end with a self-supervised rendering loss. Experiments show our method achieves competitive results and higher efficiency than both previous state-of-the-art NeRF and Gaussian-based methods. For real-world scenes, GauFRe can train in ~20 mins and offer 96 FPS real-time rendering on an RTX 3090 GPU. Project website: https://lynl7130.github.io/gaufre/index.html

1. Introduction

Monocular dynamic-scene reconstruction is underconstrained because geometry and motion must be estimated together, and non-rigid motion exceeds what rigid-scene methods capture. GauFRe uses forward-warped Gaussians and a separate static component for dynamic view synthesis.

1. Introduction

  • Monocular dynamic-scene reconstruction is ill-posed because scene geometry and motion must be estimated simultaneously, including continual non-rigid deformation.Novel view synthesis relaxes the task to reconstructing scene appearance for editing, such as re-posing or re-timing.Source: §1 · pp. 1–2
  • Backward-warping fields query canonical space at sampled ray points, but Gaussian rendering needs all primitives’ states at once, making this approach inefficient.Source: §1 · p. 2
  • GauFRe predicts a time-conditioned deformed Gaussian set from canonical-space primitives using a forward-warping deformation field.The method relates this field to how a physical 3D point changes state over time and parameterizes deformation for Gaussian attributes.Source: §1 · p. 2
  • A separate non-deforming Gaussian set represents static regions, while inductive-bias-aware initialization helps the deformation field focus on moving regions.The canonical and static Gaussian sets and forward-warping field are optimized end-to-end with self-supervised rendering loss.Source: §1 · p. 2
  • GauFRe reports competitive qualitative and quantitative results against prior state-of-the-art methods, with 96 FPS rendering on an RTX 3090.The method uses forward-deformation-based Gaussian primitives for real-time monocular dynamic view synthesis.Source: §1 · p. 2

2. Related Work

Prior dynamic-scene work extends implicit neural fields or explicit spatial representations, with different efficiency and memory trade-offs. GauFRe applies deformable 3D Gaussians and a Gaussian-compatible static component to dynamic scenes.

2. Related Work

  • Dynamic NeRF methods add motion networks to implicit scene representations, making them more expensive to train and evaluate than static radiance fields.Source: §2 · p. 2
  • Explicit spatial representations accelerate optimization and rendering, while four-dimensional grids can become prohibitively memory-intensive even for short sequences.Prior grid approaches reduce memory complexity or factorize spatio-temporal grids; examples include 4D hash grids and low-dimensional decompositions.Source: §2 · pp. 2–3
  • GauFRe models dynamic scenes with deformable 3D Gaussians and reports superior rendering quality, training speed, and real-time rendering on real-world data in comparison.Source: §2 · p. 3
  • GauFRe’s static component is compatible with Gaussian Splatting, requires only color images, and avoids assumptions or extra supervision used by some static-dynamic NeRF methods.For the closest-setting comparison noted in the passage, GauFRe trains in minutes rather than days and renders in real time.Source: §2 · p. 3
  • Earlier Gaussian deformation-field methods [57] do not model static regions, while GauFRe analyzes forward-warping design choices for competitive quality and faster training and rendering.Source: §2 · p. 3

3. Method

GauFRe represents scenes with canonical-space deformable Gaussians and a time-conditioned forward-warping field, alongside a separate static Gaussian set. Its method combines Gaussian-specific initialization with self-supervised image reconstruction optimization.

3.1. Preliminaries: 3D Gaussian Splatting

  • 3D Gaussians encode each point with covariance, density, and view-dependent color; covariance is parameterized by rotation and scale, then projected and rasterized as a 2D Gaussian.Initial positions can come from an SfM point cloud or random initialization, and primitives are periodically cloned, split, and pruned.Source: §3.1 · p. 3

3.2. Forward-warping Deformation Field

  • A time-conditioned forward-warping field maps canonical Gaussians into each frame, deforming position, rotation, and scale to represent rigid and non-rigid motion.Opacity and color remain canonical because deforming them caused training-set overfitting and more novel-view artifacts than it fixed.Source: §3.2 · pp. 3–4
  • Pre-exponentiation scale deltas simplify estimation and allow negative changes; the authors report improved reconstruction quality, while quaternion normalization stabilizes deformation.Quaternion addition was faster but produced more artifacts at novel viewpoints and timesteps.Source: §3.2 · p. 4

3.3. Static Component for Gaussian Splatting

  • Static and deformed Gaussians are concatenated for rendering, while the two sets are densified and pruned separately.The static set is initialized from SfM points and the deformable set from uniform samples.Source: §3.3 · pp. 4–5
  • The static set uses SfM points while the deformable set uses uniform samples, reducing competition over scene occupancy and assigning unchanging regions to the static set.This lets the deformable set focus on dynamic regions.Source: §3.3 · p. 5

3.4. Implementation Details

  • The deformation field uses positional and temporal encoding for its position and time inputs, with encoding bases Lx = 10 and Lt = 10.Its MLP has 6 layers and width 256.Source: §3.4 · p. 5
  • Training uses self-supervised image reconstruction, switching from L2 to L1 after 20 K iterations and combining the selected loss with SSIM.Both Gaussian sets are densified until 20 K iterations, and optimization continues through 40 K iterations.Source: §3.4 · p. 5

4. Experiments

Across the evaluated datasets, GauFRe combines high reconstruction quality with fast training and rendering. The experiments compare it with deformation-based, decomposition-aware, and Gaussian-based methods, and assess design choices through ablations.

4. Experiments

  • Evaluation uses PSNR, SSIM, MS-SSIM, and LPIPS across synthetic D-NeRF and real-world NeRF-DS and HyperNeRF datasets, with all running times measured on one NVIDIA 3090 GPU.Methods are trained and rendered at 400×400 on D-NeRF to accommodate all baselines; LPIPS is excluded on HyperNeRF where some prior methods do not report it.Source: §4 · p. 5

4.1. Results

  • GauFRe combines high reconstruction quality with efficient training and rendering, unlike approaches that trade image quality for rendering speed or retain quality at the cost of slow training.The comparison includes voxel-grid and plane-based representations, MLP-based Nerfies and HyperNeRF, and V4D.Source: §4.1 · p. 6
  • Fixing Gaussian scale limits reconstruction, while deforming opacity or spherical harmonics increases overfitting and lowers reconstruction quality.Pre-activation for scale is marginally better than post-activation, and quaternion addition is assessed against geometrically correct multiplication.Source: §4.1 · p. 6
  • Removing either static Gaussians or inductive bias-aware initialization lowers performance, and removing the initialization performs worse than removing the static set.Removing the L2/L1 loss transition marginally hurts performance.Source: §4.1 · p. 6, Table 4 · p. 8
  • GauFRe delivers top image quality and the fastest training among deformation-based methods, with real-time rendering; it also achieves high reconstruction quality with efficient training and rendering.Training takes 10–20 minutes on D-NeRF and NeRF-DS; the comparison covers D-NeRF, NeRF-DS, and HyperNeRF.Source: §4.1 · p. 5, Table 1 · p. 7
  • On real-world datasets, GauFRe achieves comparable results to D3G while training almost 4×/2× faster and rendering 3×/2× faster; it is comparable to or better than 4DGS and EffG on HyperNeRF.EffG suffers on reconstruction quality in the underconstrained monocular setting; 4DGS lags in quality and trains slower on challenging NeRF-DS scenes.Source: §4.1 · pp. 7–8, Figure 7 · p. 7
  • At 400×400 on D-NeRF, GauFRe achieves higher quality than RTGS and 4DGS; the qualitative comparison describes its images as the sharpest among the methods shown.On NeRF-DS, the results highlight finer details on dynamic objects such as hands and high quality in static regions; on HyperNeRF, GauFRe is comparable to or better than 4DGS, D3G, and EffG.Source: Figure 5 · p. 6, Figure 6 · p. 7, Figure 7 · p. 7
  • Compared with NeuralDiff and D2NeRF, GauFRe produces better novel-view rendering while training and rendering significantly faster.The comparisons use real-world datasets and also examine dynamic masks and static-scene renderings.Source: Table 2 · p. 7, §4.1 · p. 8, Table 5 · p. 8

5. Conclusion

GauFRe combines a canonical Gaussian set, a time-conditioned deformation field, and a static component guided by inductive bias-aware initialization. It achieves competitive results and efficiency, while retaining limitations shared by Gaussian-based methods.

5. Conclusion

  • GauFRe uses a canonical Gaussian set affected by an MLP-based deformation field, with static primitives and inductive bias-aware initialization guiding deformation toward moving regions.The end-to-end pipeline uses a self-supervised image-based rendering loss.Source: §5 · p. 8
  • GauFRe achieves competitive qualitative and quantitative results and efficiency compared with previous state-of-the-art methods, enabling 96 FPS rendering of real-world scenes on one RTX 3090 GPU.Source: §5 · p. 8
  • Gaussian-based methods remain prone to overfitting, and dynamic Gaussian Splatting may struggle with large motions, thin structures, and densification under motion.The paper notes these issues are especially relevant with sparse observations and complex motion.Source: §5 · p. 8

Related papers

Loading 2312.11458v3…