Source-linked AI summary
FreeTimeGS: Free Gaussian Primitives at Anytime and Anywhere for Dynamic Scene Reconstruction
Yifan Wang, Peishan Yang, Zhen Xu, Jiaming Sun, Zhanhua Zhang, Yong Chen, Hujun Bao, Sida Peng, Xiaowei Zhou
TL;DR
Dynamic scene reconstruction remains difficult for complex motions because canonical Gaussian methods require challenging deformation correspondences and optimization. FreeTimeGS introduces flexible 4D Gaussian primitives with explicit motion, temporal opacity, and regularization, achieving higher rendering quality and speed across multiple datasets while still requiring lengthy per-scene reconstruction and lacking relighting support.
Problem
Canonical Gaussian methods struggle with complex motions because deformation fields must recover difficult long-range correspondences between canonical and observation spaces.
Method
FreeTimeGS allows Gaussian primitives to appear at arbitrary positions and times, assigns each an explicit motion and temporal opacity function, and regularizes high opacity during optimization.
Results
FreeTimeGS achieves the highest quality on public datasets and the best quality and efficiency by a large margin on a dataset with faster, more complex motions.
Takeaways & Limitations
Flexible primitives and explicit motion support more faithful modeling of complex dynamic scenes while improving rendering quality and speed across multiple datasets.
Takeaways & Limitations
FreeTimeGS still requires a lengthy reconstruction process for each dynamic scene and currently supports novel view synthesis but not relighting.
Abstract
from arXiv · showhide
This paper addresses the challenge of reconstructing dynamic 3D scenes with complex motions. Some recent works define 3D Gaussian primitives in the canonical space and use deformation fields to map canonical primitives to observation spaces, achieving real-time dynamic view synthesis. However, these methods often struggle to handle scenes with complex motions due to the difficulty of optimizing deformation fields. To overcome this problem, we propose FreeTimeGS, a novel 4D representation that allows Gaussian primitives to appear at arbitrary time and locations. In contrast to canonical Gaussian primitives, our representation possesses the strong flexibility, thus improving the ability to model dynamic 3D scenes. In addition, we endow each Gaussian primitive with an motion function, allowing it to move to neighboring regions over time, which reduces the temporal redundancy. Experiments results on several datasets show that the rendering quality of our method outperforms recent methods by a large margin. Project page: https://zju3dv.github.io/freetimegs/ .
1. Introduction
Existing dynamic view-synthesis methods often rely on canonical Gaussian primitives and deformation fields, but complex motions make long-range correspondences and deformation optimization difficult. FreeTimeGS instead uses a flexible 4D representation with explicit primitive motion, regularization, and strong benchmark results.
- Canonical Gaussian methods use MLP deformation fields to map primitives from canonical space to observation-space scenes over time.
- Complex motions challenge these methods because large object movements require difficult long-range correspondences between canonical and observation spaces.
- FreeTimeGS lets Gaussian primitives appear at arbitrary positions and time steps, while explicit motion functions move them between neighboring regions.This supports temporal reuse, reduces representation redundancy, and limits motion modeling to short-range correspondences.
- A high-opacity regularization strategy penalizes Gaussian opacity early in optimization to mitigate local minima caused by blocked gradient backpropagation.
- FreeTimeGS achieves the highest quality on Neural3DV and ENeRF-Outdoor, and the best quality and efficiency by a large margin on a faster, more complex-motion dataset.
2. Related Work
Dynamic scene reconstruction has progressed from RGB-D and NeRF-based systems to Gaussian representations, but existing approaches retain limitations in robustness, efficiency, or complex-motion modeling. FreeTimeGS combines flexible Gaussian primitives, explicit motion, temporal opacity control, and regularization for scalable dynamic reconstruction.
- Dynamic scene reconstruction with RGB-D cameras: RGB-D reconstruction methods fuse depth observations in canonical space using non-rigid tracking, but tracking robustness and correspondence accuracy remain bottlenecks.
- NeRF-based dynamic scene reconstruction: NeRF-based methods model dynamics with canonical fields, body priors, or direct 4D representations, while hybrid and factorized methods target computational efficiency.
- NeRF-based dynamic scene reconstruction: NeRF-based reconstruction still faces slow rendering, poor rendering quality, and high storage requirements, motivating more feasible scalable solutions.
- Gaussian-based dynamic scene reconstruction: Gaussian-based methods offer real-time speed and sharp quality, but many model scene motion through deformation networks from canonical space.
- Gaussian-based dynamic scene reconstruction: 4DGS and STGS avoid deformation fields yet still struggle with complex motions because their motion representations create difficult optimization interactions.For 4DGS, geometry and velocity are entangled through transformations between spatial and temporal scales.
3. Method
FreeTimeGS represents dynamic scenes with Gaussian primitives that can appear at arbitrary positions and times, each moving through neighboring regions over time. Training combines rendering objectives with opacity regularization, periodic relocation, motion scheduling, and correspondence-based initialization.
- 3.1. Gaussian primitives at anytime anywhere: FreeTimeGS uses Gaussian primitives at arbitrary spatial positions and time steps to represent dynamic scene content.Each primitive also has a motion function that adjusts its position over time to neighboring regions.
- 3.1. Gaussian primitives at anytime anywhere: Each primitive is parameterized by position, time, duration, velocity, scale, orientation, opacity, and spherical harmonics coefficients.Its position is first moved according to the motion function before opacity and color are evaluated.
- 3.2. Training: The model optimizes image, SSIM, and perceptual losses between observed and rendered images.The rendering objective is L_render = λ_imgL_img + λ_ssimL_ssim + λ_percL_perc.
- 3.2. Training: High-opacity primitives can hinder optimization in fast-moving or complex-motion regions, motivating a regularization loss that constrains opacity.The regularization is weighted by temporal opacity, while stop-gradient prevents it from minimizing temporal opacity.
- 3.2. Training: Periodic relocation moves low-opacity primitives toward regions with high sampling scores to mitigate the primitive-count increase caused by regularization.The sampling score combines spatial gradient and opacity, and relocation occurs every N iterations for primitives below an opacity threshold.
- 3.2. Training: Initialization uses ROMA matches and 3D triangulation to initialize primitive positions and times, while velocity optimization is annealed during training.An annealing motion scheduler helps model fast motions early and complex motion later.
4. Experiments
FreeTimeGS is evaluated on public datasets and a challenging self-collected dataset, using image-quality metrics and qualitative comparisons. It achieves the strongest reported rendering quality, particularly for fast and complex motion, while ablations show its motion representation, regularization, relocation, and initialization contribute to performance.
- Datasets: SelfCap contains eight fast, complex-motion scenes captured from 22–24 cameras, complementing public datasets with less challenging motion.The dataset includes dancing, playing with pets, and bicycle repair, with 60 frames per scene at 60 FPS.
- Metrics: PSNR, DSSIM1, DSSIM2, and LPIPS evaluate rendered-image quality, with higher PSNR and DSSIM and lower LPIPS indicating better results.DSSIM1 and DSSIM2 use data ranges of 1.0 and 2.0, respectively.
- Comparison Experiments: FreeTimeGS outperforms all baselines across reported metrics on Neural3DV, ENeRF-Outdoor, and SelfCap.The paper reports the highest performance on the two public datasets and the best performance on SelfCap, including challenging dynamic scenes.
- Comparison Experiments: Qualitative comparisons show clearer details in fast-moving regions, including arms, dolls, fingers, faces, clothing text, hands, and rotating pedals.The method also improves distant static regions such as forests visible through windows and retains details during dance and bicycle sequences.
- Ablation Studies: Removing the proposed motion representation or 4D initialization visibly degrades modeling of fast and complex motion.The ablations compare against 4DGS motion representation and zero-velocity initialization.
- Ablation Studies: Without 4D regularization, high-opacity Gaussians hinder optimization; without periodic relocation, limited Gaussian counts produce suboptimal results.The regularization ablation links large opacity to optimization difficulty, while relocation encourages using fewer high-opacity Gaussians.
5. Conclusions
FreeTimeGS is a flexible 4D representation for dynamic scenes that supports Gaussian primitives across arbitrary times and locations, while explicit motion and temporal opacity functions improve representation. The method achieves higher rendering quality and speed, but still requires lengthy scene reconstruction and does not support relighting.
- FreeTimeGS allows Gaussian primitives to emerge at any time and location, improving flexibility for modeling complex motions.Each primitive also has optimizable explicit motion and temporal opacity functions.
- A high-opacity regularization strategy mitigates local minima during optimization.The strategy penalizes high opacity in Gaussian primitives.
- FreeTimeGS achieves higher rendering quality and rendering speed on multiple widely used dynamic view synthesis datasets.
- The method still requires a lengthy reconstruction process for each dynamic scene.
- The current representation supports novel view synthesis but not relighting.Future extensions could add surface normals and material properties.
A.1. More Quantitative Comparison on Neural3DV
Additional Neural3DV comparisons examine rendering speed, storage cost, and the quality–storage trade-off obtained by controlling the number of Gaussian primitives.
- Additional Neural3DV comparisons include rendering speed and storage cost.
- Controlling the number of Gaussian primitives demonstrates the trade-off between rendering quality and storage cost.
A.2. More Quantitative Comparison on SelfCap
Additional SelfCap experiments compare rendering speed and storage cost, while showing that 4DSH adds storage without improving rendering quality and that long-range correspondences challenge complex-motion reconstruction.
- SelfCap comparisons report rendering speed and storage cost for the evaluated methods.
- 4DSH introduces additional storage cost without improving rendering quality on SelfCap, so it is omitted from the main-paper SelfCap experiments.
- The difficulty of building long-range correspondences makes complex dynamic scenes challenging to handle.This difficulty is shown in the SelfCap quantitative and qualitative comparisons.
A.3. More Qualitative Results
Additional SelfCap results provide qualitative comparisons and per-scene quantitative breakdowns across Neural3DV, ENeRF-Outdoor, and SelfCap.
- More qualitative comparisons on SelfCap are shown in Figures 8 and 9.
- Per-scene quantitative results are provided for Neural3DV, ENeRF-Outdoor, and SelfCap.
B.1. Evaluation details on SelfCap
The evaluation reports rendering-quality metrics on SelfCap and Neural 3D Video, including separate analysis of entire images and dynamic regions on SelfCap. Additional studies examine quantitative comparisons and how reducing primitive count affects storage cost.
- Dataset comparisons: Neural 3D Video comparisons report PSNR, DSSIM1, DSSIM2, and LPIPSAlex, with table variants defined by included scenes and primitive-count limits.The variants include Flame Salmon only, exclude Coffee Martini, or constrain the method to no more than 500k primitives.
- Primitive-count ablations: Ablations on Neural 3D Video vary the number of primitives while tracking PSNR, DSSIM1, DSSIM2, and LPIPSAlex.The method retains high-quality results with storage cost reduced to 8.3MB.
- SelfCap metrics: SelfCap evaluation reports PSNR, DSSIM2, and LPIPSV GG for entire images and dynamic regions, with FPS measured on an NVIDIA RTX 4090 GPU.The dynamic-region evaluation masks and crops the dynamic area before computing metrics.
- Result presentation: SelfCap tables identify the best and second-best results through green and yellow cell colors, respectively.The tables also include a variant constrained to no more than 500k primitives.