Source-linked AI summary
Swift4D:Adaptive divide-and-conquer Gaussian Splatting for compact and efficient reconstruction of dynamic scene
Jiahao Wu, Rui Peng, Zhiyan Wang, Lu Xiao, Luyang Tang, Jinbo Yan, Kaiqiang Xiong, Ronggang Wang
TL;DR
Dynamic-scene novel view synthesis remains challenging, and existing approaches still face training-time and storage demands. Swift4D separates static and dynamic Gaussians and applies temporal modeling only to dynamic points; it reports state-of-the-art speed, storage efficiency, and rendering quality.
Problem
Dynamic-scene novel view synthesis is challenging, and existing methods still face long training times and heavy storage requirements.
Source: §1 · p. 2Method
Swift4D decomposes Gaussian points into static and dynamic groups and uses a compact multi-resolution 4DHash to model temporal changes for dynamic points.
Source: §1 · p. 3Results
Swift4D reports state-of-the-art performance in training and rendering speed, storage efficiency, and rendering quality.
Source: §1 · p. 3Takeaways & Limitations
The dynamic-static decomposition is presented as applicable to most existing dynamic-scene reconstruction methods to enhance quality and accelerate convergence.
Source: §6 · p. 10With nearly 6,000 images, loading images during training empirically wastes around 40% of training time on disk I/O.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Novel view synthesis has long been a practical but challenging task, although the introduction of numerous methods to solve this problem, even combining advanced representations like 3D Gaussian Splatting, they still struggle to recover high-quality results and often consume too much storage memory and training time. In this paper we propose Swift4D, a divide-and-conquer 3D Gaussian Splatting method that can handle static and dynamic primitives separately, achieving a good trade-off between rendering quality and efficiency, motivated by the fact that most of the scene is the static primitive and does not require additional dynamic properties. Concretely, we focus on modeling dynamic transformations only for the dynamic primitives which benefits both efficiency and quality. We first employ a learnable decomposition strategy to separate the primitives, which relies on an additional parameter to classify primitives as static or dynamic. For the dynamic primitives, we employ a compact multi-resolution 4D Hash mapper to transform these primitives from canonical space into deformation space at each timestamp, and then mix the static and dynamic primitives to produce the final output. This divide-and-conquer method facilitates efficient training and reduces storage redundancy. Our method not only achieves state-of-the-art rendering quality while being 20X faster in training than previous SOTA methods with a minimum storage requirement of only 30MB on real-world datasets. Code is available at https://github.com/WuJH2001/swift4d.
1 INTRODUCTION
Dynamic-scene novel view synthesis remains difficult, while existing methods face training-time and storage costs. Swift4D separates static and dynamic Gaussians, modeling temporal changes only for dynamic points, and reports state-of-the-art performance across speed, storage, and rendering quality.
1 INTRODUCTION
- Static points make up most of a scene and need little or no deformation, but uniform modeling can blur dynamic regions and adds computational and storage demands.The authors motivate partitioning static and dynamic points to avoid applying complex dynamic modeling to static regions.Source: §1 · p. 2
- Swift4D decomposes Gaussian points into static and dynamic groups using an additional parameter, applies temporal modeling only to dynamic points, and combines both groups for rendering.The decomposition is based on 2D multi-view images.Source: §1 · p. 3
- The compact multi-resolution 4DHash models the spatio-temporal domain with a footprint as small as 8MB.The authors report that it enhances rendering quality, accelerates training, and supports compact storage.Source: §1 · p. 3
- Swift4D reports state-of-the-art performance in training and rendering speed, storage efficiency, and rendering quality.The authors also report effectiveness in scenarios involving large movements.Source: §1 · p. 3
2 RELATED WORK
Prior novel-view-synthesis work spans grid and hash representations, point-based rendering, and several approaches to dynamic-scene modeling. Swift4D’s pipeline initializes canonical points, separates static and dynamic points, deforms the dynamic group, and recombines them for rendering.
2 RELATED WORK
- Static-scene NVS methods use grid-based representations, hash encoders, and integrated positional encoding, while 3DGS introduced real-time, high-quality point-based rendering.The passage also notes anti-aliasing, mesh-extraction, and large-scale-scene advances.Source: §2 · p. 3
- Dynamic-scene methods model moving-point trajectories, deform a canonical space, or represent the 6D light field; MLP- and low-rank-plane deformers can train slowly and limit complex-deformation capacity.Other approaches use sparse control points or Fourier series and polynomial fitting.Source: §2 · p. 3
- Swift4D’s pipeline initializes a canonical point cloud from first-frame images, classifies points using a learned dynamic parameter, deforms dynamic points, then mixes them with static points for rendering.Source: Figure 3 · p. 4
- Prior efforts to separate dynamic and static Gaussians use external foreground-background segmentation or adaptive separation, while reported output quality remains suboptimal for the former efforts.Source: §2 · p. 4
3 METHOD
Swift4D first establishes canonical Gaussians, learns a dynamic/static decomposition, then optimizes the remaining Gaussian parameters with a spatio-temporal structure. Its pipeline applies temporal deformation only to dynamic points and prunes low-importance Gaussians.
3.1 GAUSSIAN SPLATTING PRELIMINARY
- 3D Gaussian points are represented by a mean, covariance, opacity, and view-dependent color, with the covariance decomposed into rotation and scaling for implementation.The mean and covariance are projected into 2D for rendering.Source: §3.1 · pp. 4–5
3.2 EFFICIENT DYNAMIC AND STATIC DECOMPOSITION
- Dynamic/static decomposition learns a per-Gaussian motion score from temporal pixel masks, then thresholds it to assign deformation only to dynamic points.Pixels are classified using temporal intensity variation with γ = 0.02; Gaussian scores above ζ = 7.0 are dynamic. Other views can correct assignments affected by occlusion.Source: §3.2 · pp. 5–6
3.3 SPATIO-TEMPORAL STRUCTURE
- A multi-resolution 4D hash encodes dynamic Gaussians over space and time, and deformation MLPs predict parameter changes before static and deformed points are combined for rendering.The hash uses 16 resolution levels typically; encoding only dynamic points reduces the required hashing space.Source: §3.3 · p. 6
3.4 DENSITY CONTROL
- Temporal importance pruning removes Gaussians whose visibility-based importance falls below 0.02, addressing floaters across canonical and deformation spaces.The method also omits opacity resets because they create excessive coupling between those spaces.Source: §3.4 · pp. 6–7
3.5 OPTIMIZATION PIPELINE
- The optimization pipeline initializes the point cloud from first frames, trains canonical Gaussians for 5,000 iterations, then trains dynamic attributes for about one minute before the spatio-temporal structure.The loss remains simple and adds no additional terms.Source: §3.5 · p. 7
4 EXPERIMENT
The experiments report strong rendering quality alongside fast training and compact storage, then examine the effects of decomposition, encoder choice, pruning, and the dynamic threshold.
4.1 IMPLEMENTATION DETAILS
- Training proceeds from a 5,000-epoch canonical-space stage through 3,000 epochs for the dynamic parameter, then about 14,000 epochs for the spatio-temporal structure.Dynamic-parameter training takes under one minute; experiments used an NVIDIA RTX 3090 GPU.Source: §4.1 · pp. 7–8
4.2 DATASET
- The evaluation covers N3DV, MeetRoom, and the challenging basketball court, using established train/test splits or view assignments and assessing image quality, training time, FPS, and model size.N3DV includes complex scenes and is downsampled by a factor of two; MeetRoom uses 12 training views and one test view.Source: §4.2 · p. 8, §4.3 · p. 8
4.3 EVALUATION
- On N3DV, Swift4D surpasses prior methods in rendering quality and is at least 20 times faster than methods with similar quality; on MeetRoom, it achieves state-of-the-art quality, training time, and storage efficiency.On MeetRoom, 3DGStream requires 30 times more storage than Swift4D.Source: §4.3 · p. 8
- On the basketball court, the decomposition separates athletes from the scene, illustrating adaptability to occlusion.The evaluation also presents basketball-court results in Figure 10 and supplementary videos.Source: §4.3 · pp. 8–9
4.4 ABLATION AND ANALYSIS
- Treating all points as dynamic increased computation time, reduced rendering quality, and blurred regions with large motion amplitudes.The ablation was conducted on sear steak and flame steak.Source: §4.4 · p. 9
- The multi-plane encoder slightly trails the 4D Hash in rendering speed and quality but outperforms methods without dynamic-static decomposition.The comparison is reported for the sear steak ablation.Source: §4.4 · p. 9
- In Fig. 9, images rendered with temporal importance pruning (our pruning strategy, (c)) appear much cleaner than those in (a) and (b), which exhibit severe artifacts.Source: §4.4 · p. 9
5 DISCUSSION
The discussion identifies incomplete static-dynamic decomposition as a limitation, while describing cases where its classification choices reduce dynamic points or capture motion-related visual changes.
5 DISCUSSION
- Pixel-level supervision does not fully separate static and dynamic points: textureless regions of small moving objects may be labeled static, while nearby motion can make static objects appear dynamic.The discussion notes both misclassification patterns as consequences of incomplete decomposition.Source: §5 · pp. 9–10
- Treating textureless moving regions such as clothing and a table as static can reduce dynamic-point count and improve rendering quality when only motion-visible edges are labeled dynamic.The authors report this outcome for the table comparison against the no-decomposition setting.Source: §5 · p. 10
- For the cook spinach scene, thresholds from 3 to 9 did not significantly change PSNR or the percentage of dynamic points; the authors set the threshold to 7.This result is presented as evidence of robustness to threshold choice.Source: §5 · p. 10
- Loading images from a dataset of nearly 6,000 images wastes around 40% of training time on disk I/O; removing that overhead could reduce training to 10 minutes while maintaining high-quality reconstruction.The 10-minute figure is a stated potential reduction, not the reported current training time.Source: §5 · p. 10
6 CONCLUSION
Swift4D combines dynamic-static decomposition, compact temporal modeling, and temporal pruning. The authors report competitive results after five minutes, with the method scoped to multi-view scenes rather than monocular or human reconstruction.
6 CONCLUSION
- 5 minutes of training yields competitive results, and the method is described as providing fast convergence, compact storage, and high-quality real-time rendering.The conclusion also reports temporal pruning to remove floaters.Source: §6 · p. 10
- Swift4D combines dynamic-static decomposition with a 4D Hash encoder and multi-head decoder for temporal modeling, and uses temporal pruning to remove floaters.Source: §6 · p. 10
- The stated scope excludes monocular dynamic-scene datasets and human reconstruction.The authors describe the method as focused on multi-view scenes and scene reconstruction.Source: §6 · pp. 10–11
- The basketball-court figure clarifies that its black floaters are Gaussian points from the dynamic background.Source: Figure 10 · p. 11