Source-linked AI summary
3D Geometry-aware Deformable Gaussian Splatting for Dynamic View Synthesis
Zhicheng Lu, Xiang Guo, Le Hui, Tianrui Chen, Min Yang, Xiao Tang, Feng Zhu, Yuchao Dai
TL;DR
Dynamic view synthesis from monocular video is challenging because motion and shape are ambiguous, and incorporating local 3D structure into deformation learning is a key problem. The paper represents scenes with deforming 3D Gaussians and uses sparse convolution to extract geometry-aware features. It reports superiority over competing methods on synthetic and real datasets.
Problem
Monocular dynamic scenes have motion/shape ambiguity, and learning locally smooth, consistent deformation requires incorporating local geometric information.
Source: §1 · p. 1Method
The method represents a dynamic scene with deforming 3D Gaussians and uses sparse convolution to extract local geometry-aware features for deformation learning.
Source: §1 · p. 2Results
The method surpasses competing methods by a wide margin on synthetic and real datasets.
Source: §1 · p. 2Takeaways & Limitations
Experiments across synthetic and real datasets demonstrate the method’s superiority in dynamic view synthesis and 3D reconstruction.
Source: §5 · p. 8The method requires precise camera poses in advance and struggles with excessively complex motions and long videos, such as rapid object movements.
Source: §7.3 · p. 11
Abstract
from arXiv · showhide
In this paper, we propose a 3D geometry-aware deformable Gaussian Splatting method for dynamic view synthesis. Existing neural radiance fields (NeRF) based solutions learn the deformation in an implicit manner, which cannot incorporate 3D scene geometry. Therefore, the learned deformation is not necessarily geometrically coherent, which results in unsatisfactory dynamic view synthesis and 3D dynamic reconstruction. Recently, 3D Gaussian Splatting provides a new representation of the 3D scene, building upon which the 3D geometry could be exploited in learning the complex 3D deformation. Specifically, the scenes are represented as a collection of 3D Gaussian, where each 3D Gaussian is optimized to move and rotate over time to model the deformation. To enforce the 3D scene geometry constraint during deformation, we explicitly extract 3D geometry features and integrate them in learning the 3D deformation. In this way, our solution achieves 3D geometry-aware deformation modeling, which enables improved dynamic view synthesis and 3D dynamic reconstruction. Extensive experimental results on both synthetic and real datasets prove the superiority of our solution, which achieves new state-of-the-art performance. The project is available at https://npucvr.github.io/GaGS/
1. Introduction
Dynamic view synthesis from monocular video is difficult because motion and shape are ambiguous, and local geometric structure matters to coherent deformation. The paper proposes geometry-aware features and deformable 3D Gaussians, reporting strong results on synthetic and real datasets.
1. Introduction
- Monocular dynamic scenes have motion/shape ambiguity, while neighboring points’ motions are coupled and real-world deformations correlate with 3D structure.The paper identifies learning locally smooth and consistent deformation from local geometry as a research focus.Source: §1 · p. 1
- The method extracts local geometry-aware features from 3D Gaussians with sparse convolution to model deformation, rather than relying on pointwise features or simple interpolation.The authors describe sparse convolution as offering a larger receptive field for local information; Figure 1 contrasts these feature-extraction approaches.Source: Figure 1 · p. 1, §1 · p. 2
- The method uses continuous 6D rotation representation because quaternion rotation parameters are discontinuous in parameter space for neural-network learning.The stated aim is to let the network learn a continuous function representing Gaussian rotations over time.Source: §1 · p. 2
- The framework combines a canonical field of 3D Gaussians with a deformation field that transforms them over time for rendering.The canonical field includes geometry-aware feature learning, and 3D Gaussian splatting renders images at the given timestamp.Source: §1 · p. 2
- Experiments on synthetic and real datasets show that the method surpasses competing methods by a wide margin.The paper lists geometry-aware feature extraction and continuous 6D rotation with modified density control among its contributions.Source: §1 · p. 2
2. Related Work
The related work covers explicit and implicit approaches to novel view synthesis, then reviews dynamic methods spanning 4D representations, canonical deformation fields, and Gaussian-based models. It positions the paper’s canonical approach as leveraging 3D information for deformation learning.
2.1. Novel View Synthesis
- Explicit novel-view methods represent scenes as point clouds, voxels, or meshes, while other approaches estimate depth from multiple views and aggregate co-visible-frame features.NeRF represents scenes implicitly with MLPs, and 3D Gaussian representations are described as promising but mainly applicable to static scenes.Source: §2.1 · p. 2
2.2. Dynamic View Synthesis
- Some dynamic methods use single-view and multi-view depth warping, time-invariant and time-varying scene components, or continuously changing scene functions to model motion and appearance.Other approaches use a 4D radiance field or project 4D space onto multiple 2D planes.Source: §2.2 · pp. 2–3
- Dynamic view synthesis methods include 4D spatial-temporal representations and canonical-field/deformation-field pipelines, with examples using scene flow, radiance fields, and plane projections.DNeRF introduced the canonical-based framework; later work includes coarse-to-fine optimization, higher-dimensional canonical fields, and multiscale interpolation.Source: §2.2 · p. 3
- Dynamic Gaussian methods include deformable 3DGS and 4D Gaussian Splatting, but the reviewed projection-based representation compresses 3D space into 2D and loses geometric information for deformation learning.The paper contrasts this with its canonical-based method, which it says can leverage 3D information in 3D space.Source: §2.2 · p. 3
3. Method
The method represents a dynamic scene with canonical 3D Gaussians and a deformation field, using learned local geometry features to estimate time-dependent Gaussian transformations before rendering. It also adapts rotation representation, optimization, and density control for dynamic scenes.
3.1. Preliminary
- 3D-GS represents scenes with sparse Gaussians whose covariance is parameterized by scale and rotation, with position, color, opacity, and other attributes used for splatting.The covariance factorization is described as ensuring positive semi-definiteness while reducing learning difficulty.Source: §3.1 · p. 4
3.2. Gaussian Canonical Field
- The canonical field combines sparse-convolution features from Gaussian-point voxels with an identity branch to retain high-resolution point-level features.A sparse 3D U-Net aggregates local features, which are fused back onto corresponding points.Source: §3.2 · pp. 4–5
3.3. Deformation Field
- A deformation MLP uses geometry features, Gaussian position, and time to predict position, rotation, and scale changes for each canonical Gaussian.The method uses continuous 6D rotation representations; canonical color and opacity remain fixed over time.Source: §3.2 · p. 4, §3.3 · p. 5
3.4. Rasterization
- A differentiable tile rasterizer renders transformed Gaussian attributes at the desired viewpoint using camera intrinsics and extrinsics.The rendered attributes include position, color, rotation, scale, and opacity.Source: §3.4 · p. 5
3.5. Optimization
- The objective combines photometric and motion losses, with ω balancing components; regularization assumes dynamic points are fewer and motions modest.Photometric loss combines L1 and D-SSIM; density control uses current timestamp attributes and maps split or cloned Gaussians back to canonical space.Source: §3.5 · pp. 5–6, Figure 3 · p. 5
4. Experiments
Experiments on synthetic and real scenes compare the method with existing approaches, then examine its visual results and components. The reported findings favor the method across quantitative, qualitative, and ablation evaluations.
4.1. Dataset
- Evaluation uses eight D-NeRF synthetic scenes and HyperNeRF real scenes, reporting PSNR, SSIM, and LPIPS.The listed HyperNeRF scenes include interp-cut-lemon, chicken, 3D printer, and split cookie.Source: §4.1 · p. 6
4.2. Implementation Details
- Training runs for 40,000 iterations, beginning with 3,000 static-scene iterations before deformation fields are added.D-NeRF points are randomly initialized, while HyperNeRF uses its provided point cloud; tests use one RTX 4090 GPU.Source: §4.2 · p. 6
4.3. Quantitative Results
- On synthetic scenes, the method outperforms physical canonical-based methods on PSNR, SSIM, and LPIPS, while real-scene results are reported as strong against state-of-the-art methods.Compared with static 3D-GS, average PSNR improves on the synthetic dataset; no numerical values are supplied here.Source: §4.3 · p. 7
4.4. Visualization Results
- Rendered synthetic results are closer to ground truth, while real-scene comparisons show more recovered detail than TiNeuVox for objects such as chicken and banana.Figure 4 compares synthetic scenes from the same viewpoint; Figure 5 shows HyperNeRF real scenes.Source: §4.4 · p. 7, Figure 4 · p. 7, Figure 5 · p. 7
- The method’s point cloud has clear local geometric structure, and the visualization of its learned Gaussians is colored using position coordinates.Source: §4.4 · p. 8, Figure 7 · p. 8
4.5. Ablation Study
- The geometry branch has a clear advantage over its removal and is reported as the most important studied ablation component.The ablation table reports average PSNR, SSIM, and LPIPS; learned features also reflect local structure and distinguish moving from static objects.Source: §4.5 · p. 8, Table 4 · p. 8
- The geometry-feature method gains performance over PointNet-like and plane-projection alternatives, while visualized features reflect local geometric patterns.The T-rex tail’s spines and smoother bone regions show different feature patterns.Source: §4.5 · p. 8
- Quaternion rotation and canonical-only density control each show a performance drop relative to the proposed settings.The canonical-only density setting cannot reflect over- or under-reconstruction information across timestamps.Source: §4.5 · p. 8
5. Conclusion
The conclusion presents a geometry-aware Gaussian Splatting method for dynamic view synthesis. It reports results across synthetic and real datasets and identifies foreground/background motion cues as future work.
5. Conclusion
- The method uses 3D sparse convolution to extract local structure for deformation learning and deforming 3D Gaussians optimized to move, rotate, and scale over time.The authors report superiority on dynamic view synthesis and 3D reconstruction across synthetic and real datasets.Source: §5 · p. 8
6. Implementation Details
The method combines photometric supervision with motion regularization, then uses geometric and identity features to predict each Gaussian’s deformation. Its network architecture extracts local geometry with a 3D U-Net and feeds fused features, Gaussian positions, and time to a decoder.
6.1. Loss Function
- The total loss combines photometric loss with an L1 motion penalty on position displacement.The photometric term mixes RGB L1 loss and structural-similarity loss; the reported weights are λ = 0.2 and ω = 0.01.Source: §6.1 · p. 9
6.2. Network Architecture
- The geometric branch uses DownVoxelBlock, ResidualBlock, and UpVoxelBlock to extract geometric features from voxel-point positions.Figure 8 shows the branch’s overall architecture, while Figure 9 details its three blocks.Source: §6.2 · p. 9, Figure 8 · p. 9, Figure 9 · p. 9
- An identity MLP produces independent point features, which are concatenated with geometric features before fusion and deformation decoding.The decoder takes fused features, Gaussian positions, and time to predict position, rotation, and scale deformations.Source: §6.2 · p. 9
- The network applies positional encoding with L = 10 for position and L = 6 for timestamp, and concatenates the input to the decoder’s third layer.The layers are fully connected, with ReLU activations between them.Source: §6.2 · p. 9
7. Results and Discussions
On Neural 3D Video, the method retains high-fidelity restoration in static regions but struggles with long sequences and dynamic regions. Additional visualizations show canonical point-cloud reconstruction and temporal interpolation, while the paper identifies limitations and future directions.
7.1. Results on Neural 3D Video dataset
- On Neural 3D Video, the method struggles with long-time series and has severely limited capability in dynamic regions despite high-fidelity restoration in static regions.The evaluation covers Cook Spinach, Cut Roast Beef, Flame Steak, and Sear Steak, reporting per-scene PSNR, SSIM, and LPIPS; the supplied text does not give metric values.Source: §7.1 · p. 9, Table 5 · p. 10
7.2. More Visualization Results
- Canonical-space scene reconstruction is shown from randomly initialized points, including a HyperNeRF scene whose input point cloud has no dynamic parts.The paper also reports temporal interpolation on synthetic and real datasets, and demonstrates novel viewpoints at fixed times.Source: §7.2 · p. 10, Figure 10 · p. 10
7.3. Limitations and Impacts
- The method cannot represent points that abruptly appear or disappear, requires precise camera poses in advance, and struggles with excessively complex or long-sequence motion.The authors report difficulty with fine-grained movement over long sequences, while general curtain swinging and human-body motion remain describable.Source: §7.3 · p. 11
- The authors identify visual-effects synthesis, game modeling, and autonomous-driving simulation as potential application areas.For autonomous-driving simulation, they propose that additional viewpoints could contribute data.Source: §7.3 · p. 11
7.4. Future Work
- Future work proposes using motion masks to distinguish dynamic from static points and investigating explicit motion modeling with foreground-background segmentation cues.The stated aim of motion masks is to reduce computing resources by estimating deformation only for dynamic points.Source: §7.4 · p. 11