Source-linked AI summary
SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes
Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, Xiaojuan Qi
TL;DR
Dynamic monocular novel view synthesis must model motion while preserving rendering quality and efficiency. SC-GS separates motion and appearance using sparse control points and dense Gaussians, with adaptive control and rigidity constraints. The method reports superior dynamic-view quality, high rendering speeds, and appearance-preserved motion editing, but remains sensitive to inaccurate camera poses and specular effects.
Problem
Dynamic-scene novel view synthesis from monocular videos must recover geometry, appearance, and motion from limited observations while maintaining efficient rendering.
Method
SC-GS drives dense 3D Gaussians with approximately 512 sparse control points versus approximately 100K Gaussians, using interpolated transformations, adaptive point allocation, and ARAP regularization.
Results
SC-GS surpasses existing methods quantitatively and qualitatively while maintaining high rendering speeds and enabling motion editing with high-fidelity appearances.
Takeaways & Limitations
The explicit sparse motion representation supports user-controlled motion editing through control-point manipulation while retaining high-fidelity appearances.
Takeaways & Limitations
Performance is prone to inaccurate camera poses, and specular effects limit improvement on NeRF-DS datasets.
Abstract
from arXiv · showhide
Novel view synthesis for dynamic scenes is still a challenging problem in computer vision and graphics. Recently, Gaussian splatting has emerged as a robust technique to represent static scenes and enable high-quality and real-time novel view synthesis. Building upon this technique, we propose a new representation that explicitly decomposes the motion and appearance of dynamic scenes into sparse control points and dense Gaussians, respectively. Our key idea is to use sparse control points, significantly fewer in number than the Gaussians, to learn compact 6 DoF transformation bases, which can be locally interpolated through learned interpolation weights to yield the motion field of 3D Gaussians. We employ a deformation MLP to predict time-varying 6 DoF transformations for each control point, which reduces learning complexities, enhances learning abilities, and facilitates obtaining temporal and spatial coherent motion patterns. Then, we jointly learn the 3D Gaussians, the canonical space locations of control points, and the deformation MLP to reconstruct the appearance, geometry, and dynamics of 3D scenes. During learning, the location and number of control points are adaptively adjusted to accommodate varying motion complexities in different regions, and an ARAP loss following the principle of as rigid as possible is developed to enforce spatial continuity and local rigidity of learned motions. Finally, thanks to the explicit sparse motion representation and its decomposition from appearance, our method can enable user-controlled motion editing while retaining high-fidelity appearances. Extensive experiments demonstrate that our approach outperforms existing approaches on novel view synthesis with a high rendering speed and enables novel appearance-preserved motion editing applications. Project page: https://yihua7.github.io/SC-GS-web/
1 The University of Hong Kong 2 VAST 3 Zhejiang University
Figure 1 summarizes SC-GS: sparse control points represent motion, drive 3D Gaussians for rendering, and support dynamic view synthesis and motion editing.
- Sparse control points represent motion in a monocular dynamic-video pipeline and drive 3D Gaussians for high-fidelity rendering.
- The representation supports both dynamic view synthesis and motion editing.
1. Introduction
Dynamic monocular novel view synthesis requires accurate geometry, appearance, and motion from limited observations, while existing representations face speed or dynamic-motion challenges. SC-GS addresses these issues with sparse control-point motion, adaptive learning, and rigid-motion regularization.
- Monocular dynamic novel view synthesis must recover scene geometry and appearance from limited observations while representing object movements accurately.
- NeRF-based representations struggle with low rendering speeds and high memory usage, especially at high resolutions.
- Gaussian splatting provides fast training, real-time inference, and strong visual quality, but existing formulations target static scenes.
- SC-GS models dynamics with approximately 512 sparse control points versus approximately 100K Gaussians, using interpolated 6 DoF transformations predicted by a time- and location-conditioned MLP.
- Adaptive control-point learning accommodates regional motion complexity, while ARAP regularization encourages locally rigid motion.
- Experiments report quantitatively and qualitatively superior results, high rendering speeds, and motion editing through control-point manipulation while maintaining high-fidelity appearances.
2. Related Work
Related work spans dynamic NeRF representations, accelerated dynamic-scene methods, and Gaussian-splatting adaptations, with differing computational costs and motion-modeling strategies.
- Dynamic NeRF: Dynamic NeRF methods extend static neural representations to dynamic scenes using deformation fields, while 4D radiance fields incur substantial computational costs.
- Dynamic NeRF: Acceleration strategies include grid representations and image-based rendering methods for dynamic-scene modeling.
- Dynamic Gaussian Splatting: Dynamic Gaussian-splatting methods include frame-by-frame training, deformation fields, and hexplane-based alternatives, but some incur slow training from extra MLPs.
- Scene-editing approaches target geometric detail preservation or edited-scene rendering quality; SC-GS belongs to the latter category while using Gaussian splatting instead of implicit NeRFs.
3. Preliminaries
Gaussian splatting represents scenes with colored 3D Gaussians and renders images by projecting and alpha-blending them. SC-GS extends this representation to dynamic scenes using sparse control points.
- Each Gaussian is parameterized by a center, covariance, rotation, scale, opacity, and spherical-harmonic appearance coefficients.
- Rendering projects Gaussians onto the image plane and aggregates their contributions through fast alpha-blending.
- Optimizing Gaussian parameters and adapting Gaussian density enables high-quality real-time image synthesis.
- SC-GS introduces sparse control points to adapt Gaussian splatting for dynamic scenes while preserving rendering quality and speed.
4. Method
The method represents dynamic-scene appearance and geometry with canonical-space Gaussians while modeling motion through sparse control points and a deformation MLP. Learned transformations are interpolated to deform Gaussians, with adaptive control-point density, ARAP regularization, and rendering loss used for optimization.
- Dynamic scene representation: Canonical-space Gaussians represent scene geometry and appearance, while control points and a deformation MLP model time-varying motion.The model is first pretrained for coarse motion with fixed Gaussians, then optimized jointly.
- Sparse control points: Each sparse control point has a learnable canonical coordinate and RBF radius, forming compact motion bases far fewer than the scene’s Gaussians.The radius controls how a control point’s influence decreases with distance.
- Control-point transformations: The MLP predicts time-varying 6 DoF transformations for control points, represented with quaternions for stable optimization and interpolation.Canonical control-point coordinates and time are queried to obtain transformations at each timestep.
- Dynamic Gaussian rendering: Each Gaussian’s motion is computed by KNN-based linear blend skinning, using learned RBF interpolation weights from its neighboring control points.The implementation uses K=4 neighboring control points, and the learned radii adapt interpolation to complex motions.
- Optimization: Training combines rendering loss with ARAP regularization, encouraging control-point motion to remain locally rigid while fitting reference frames.Rendering loss combines L1 and D-SSIM terms; ARAP evaluates deviations from local rigidity.
- Adaptive control points: Adaptive density adjustment prunes low-impact control points and clones points in regions with poor reconstruction or complex motion.Pruning uses accumulated influence, while cloning responds to large Gaussian-gradient norms.
5. Motion Editing
The method enables motion editing by deforming a trajectory-based control graph under user constraints, then transferring the resulting 6 DoF transformations to the Gaussians.
- Control Point Graph: Motion editing manipulates sparse control points after predicting their trajectories, determining neighborhoods, constructing a rigid control graph, and deforming that graph.The graph is built from control-point trajectories rather than a single timestep, helping avoid unreasonable edge connections.
- Control Point Graph: Control-point connections use trajectory-based neighborhoods within a predetermined radius, with edge weights computed for connected vertices.Using the overall motion sequence accounts for temporal behavior when forming graph edges.
- Motion Editing: User-specified handle constraints drive ARAP deformation of the control graph to maintain local rigidity during editing.The deformed positions satisfy fixed handle constraints p′_l = h_l for l ∈ H.
- Motion Editing: Alternating optimization solves local rotations and deformed control-point positions, producing rotations and translations for each control point.The resulting translation is defined as T̂_i = p′_i − p_i, and together with R̂_i forms a 6 DoF transformation.
- Motion Editing: Replacing the original control-point transformations with the deformed ones warps Gaussians into high-quality edited images, including motions outside the training sequence.The edited results are visualized in Fig. 5.
6. Experiment
Experiments on D-NeRF and NeRF-DS evaluate dynamic novel-view synthesis quantitatively and qualitatively, including control-point and ARAP-loss ablations. SC-GS delivers high visual quality, preserves details, handles challenging specular scenes, and supports motion editing beyond training sequences.
- Quantitative Comparisons: Experiments cover eight D-NeRF scenes and seven NeRF-DS videos using PSNR, SSIM, MS-SSIM, and LPIPS metrics.D-NeRF uses 360° viewpoints, while NeRF-DS uses monocular videos with camera poses estimated by COLMAP.
- Quantitative Comparisons: Despite camera-pose estimation errors in NeRF-DS, SC-GS achieves the best reported visual quality and higher average performance without specialized specular-surface processing.NeRF-DS can outperform SC-GS on certain datasets because it uses a specialized design for modeling specular parts.
- Qualitative Comparison: SC-GS achieves the best visual quality on D-NeRF comparisons and produces results closest to ground truth while preserving scene details.The comparisons include D-NeRF, TiNeuVox, Tensor4D, K-Planes, FF-NVS, and 4D-GS.
- Ablation study: The control-point representation reduces noisy Gaussian trajectories and optimization difficulties compared with directly predicting each Gaussian’s motion.The direct Gaussian baseline is theoretically more flexible but tends toward local minima rather than the global optimum.
- Ablation study: ARAP loss removes motion violations in which Gaussians move toward the torso instead of following the ascending arm.Control points regularize motion, but ARAP additionally constrains inter-control-point relationships to encourage local rigidity.
- Ablation study: Manipulating learned control nodes enables appearance-preserving motion editing that generalizes to motion beyond the training sequence.The reconstructed and edited sequences are visualized in Fig. 5, supported by locally rigid motion modeling and learned Gaussian-control-point correlations.
7. Conclusion and Future Works
The method enables high-quality dynamic scene reconstruction and motion editing, but its performance is sensitive to inaccurate camera poses and common specular effects.
- The method drives 3D Gaussians with control points and a deformation MLP for dynamic scene reconstruction and motion editing.
- Experiments show improved visual quality for synthesized dynamic novel views compared with existing approaches.
- Inaccurate camera poses can cause reconstruction failures on datasets such as HyperNeRF.
- Specular effects limit improvement on NeRF-DS datasets because common highlights and mirror effects are not modeled accurately.
- Future work could incorporate specialized specular modeling and video deblurring to improve robustness for highlights, mirror effects, and dynamic-object blur.