Source-linked AI summary
DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splatting
Agelos Kratimenos, Jiahui Lei, Kostas Daniilidis
TL;DR
Dynamic novel view synthesis must handle temporally complex, underconstrained motion while remaining accurate and efficient. DynMF decomposes scene motion into a few shared neural trajectories with per-point coefficients, achieving high-quality real-time rendering and strong benchmark performance. The approach also supports interpretable motion disentanglement and control, within the stated canonical-space point-cloud assumption.
Problem
Dynamic scene rendering must resolve displacements and overlapping non-rigid motions while providing fast training, real-time rendering, and accurate novel views.
Method
DynMF factorizes dynamic-scene motion into a few globally shared learned trajectories, with points selecting them through motion coefficients and a time-only queried MLP.
Results
DynMF significantly surpasses state-of-the-art benchmarks on D-NeRF and achieves high-quality real-time dynamic rendering above 120 FPS.
Takeaways & Limitations
Sparse motion factorization enables interpretable motion disentanglement, independent motion control, and novel motion combinations for dynamic scenes.
Takeaways & Limitations
The motion representation assumes a dynamic scene represented as a point cloud in canonical space over a frame interval.
Abstract
from arXiv · showhide
Accurately and efficiently modeling dynamic scenes and motions is considered so challenging a task due to temporal dynamics and motion complexity. To address these challenges, we propose DynMF, a compact and efficient representation that decomposes a dynamic scene into a few neural trajectories. We argue that the per-point motions of a dynamic scene can be decomposed into a small set of explicit or learned trajectories. Our carefully designed neural framework consisting of a tiny set of learned basis queried only in time allows for rendering speed similar to 3D Gaussian Splatting, surpassing 120 FPS, while at the same time, requiring only double the storage compared to static scenes. Our neural representation adequately constrains the inherently underconstrained motion field of a dynamic scene leading to effective and fast optimization. This is done by biding each point to motion coefficients that enforce the per-point sharing of basis trajectories. By carefully applying a sparsity loss to the motion coefficients, we are able to disentangle the motions that comprise the scene, independently control them, and generate novel motion combinations that have never been seen before. We can reach state-of-the-art render quality within just 5 minutes of training and in less than half an hour, we can synthesize novel views of dynamic scenes with superior photorealistic quality. Our representation is interpretable, efficient, and expressive enough to offer real-time view synthesis of complex dynamic scene motions, in monocular and multi-view scenarios.
1 Introduction
DynMF addresses the challenge of fast, accurate dynamic novel view synthesis by decomposing scene motion into a few sparse basis trajectories. The representation supports efficient training, real-time rendering, motion disentanglement, and strong benchmark performance.
- 1 Introduction: Dynamic novel view synthesis remains difficult because accurate, efficient, visually appealing rendering must disambiguate displacements and overlapping non-rigid motions.The challenge is especially pronounced for monocular captures and non-rigid objects.
- 1 Introduction: DynMF factorizes scene motion into a few sparse basis trajectories, mapping each point to one trajectory or a linear combination of them.The basis motions are learned through a small MLP queried only in time.
- 1 Introduction: The representation reaches photorealistic results in under 5 minutes, superior-to-state-of-the-art synthesis in under 30 minutes, and rendering above 120 FPS at 1K resolution.The framework is designed for fast convergence and real-time inference.
- 1 Introduction: Motion factorization enables disentanglement and independent control of scene motions, supporting novel motion combinations and applications such as video editing and interactive control.The paper illustrates selectively moving scene elements in Jumpingjacks and Bouncingballs.
- 1 Introduction: DynMF significantly surpasses state-of-the-art benchmarks on D-NeRF while achieving high-quality real-time dynamic rendering without prior scene knowledge.The method achieves comparable results on real-scene datasets.
2 Related Work
DynMF builds on dynamic neural rendering and motion-factorization research while relaxing the dense-correspondence requirements of prior point-based methods. Its overview depicts a compact factorization of dense motion into shared bases and per-Gaussian coefficients.
- 2 Related Work: Prior dynamic rendering methods include coordinate-based neural fields and point-based approaches for modeling deformation and temporally consistent scenes.The related work distinguishes several approaches to dynamic neural rendering.
- 2 Related Work: DynMF relaxes the dense-correspondence restriction of prior point-based methods while constructing an efficient and expressive deformation field.The method is designed for dynamic scenes without dense correspondences across frames or views.
- 2 Related Work: Motion factorization traditionally represents complex motion as linear combinations of basis trajectories, providing context for DynMF's trajectory decomposition.Earlier work applied trajectory factorization to non-rigid structure from motion.
- 2 Related Work: Compared with Dynamic 3D Gaussians, DynMF targets monocular scenes and random initialization rather than requiring multiple cameras and depth for first-frame reconstruction.The cited comparison also notes limitations of Dynamic 3D Gaussians with newly appearing objects.
- 2 Related Work: DynMF factorizes a dense motion field into globally shared learnable bases and motion coefficients stored on each Gaussian.At query time, a single global basis evaluation is blended by the coefficients to recover the deformed scene.
3 Method
DynMF represents dynamic scenes with a small set of shared trajectories whose coefficients control each point’s motion, then combines this representation with 3D Gaussian Splatting for efficient rendering. Learned time-only basis functions and sparsity regularization constrain motion, support decoupling, and preserve real-time rendering.
- Motion representation: DynMF models each point trajectory as a canonical position plus a linear combination of shared basis trajectories.The factorized matrix form uses point-specific coefficients with a basis matrix much smaller than the numbers of points or time steps.
- Motion representation: Sharing bases across points jointly constrains motion trajectories, encouraging physically plausible frame-consistent behavior and soft locality or rigidity.Neighboring Gaussians and rigid regions are encouraged to select similar trajectories without requiring dense correspondences.
- Motion representation: A small MLP queried only over normalized time learns compact basis trajectories spanning scene deformations.The learned functions are not strictly bases, but empirically cover sufficient deformation variation and converge quickly.
- 3D Gaussian Splatting: DynMF turns static Gaussian Splatting into a dynamic model by decomposing Gaussian means and rotations while keeping density and spherical-harmonic appearance fixed over time.The same motion coefficients are shared between translational and rotational decompositions.
- Optimization framework: An L1-based sparsity objective and a stronger coefficient ratio loss penalize unnecessary motion bases and force each Gaussian to use very few trajectories.Strict sparsity can decouple dynamic motions and enable novel motion synthesis, while the reported rigidity loss does not further improve rendering quality.
4 Experiments
DynMF is evaluated on synthetic monocular, real monocular, and real multi-view dynamic-scene datasets using quantitative, qualitative, motion-control, and ablation studies. The experiments report strong rendering quality, real-time performance, motion decomposition, and effective modeling with few trajectories.
- Results on Synthetic Data: More than 300 FPS at 400 × 400 resolution, while surpassing all state-of-the-art methods across metrics on the synthetic D-NeRF dataset.After 10 minutes of training, DynMF achieves state-of-the-art results; after 30 minutes, it synthesizes most scenes almost perfectly.
- Motion Decomposition: The representation enables independent control of scene motions, including moving the right foot or left hand separately in the Jumpingjacks and Mutant scenes.Sparsity encourages each Gaussian to select one trajectory, while nearby Gaussians tend to share trajectories.
- Time Encoding and Number of Trajectories: Scenes can be adequately modeled with very few trajectories, while increasing their number only subtly improves performance; higher time-encoding frequencies provide more expressive trajectories.The ablation varies trajectory count B and positional-encoding frequencies F on synthetic and real scenes.
- Fourier Series: A Fourier-series motion basis satisfactorily expresses dynamic motion but lacks sufficient expressiveness and smoothness for complicated scene details.The comparison evaluates non-learned motion and rotation bases on the Mutant and flame-steak scenes.
5 Conclusion & Future Work
DynMF enables real-time dynamic-scene rendering with fast convergence, state-of-the-art novel-view quality, and motion decomposition that supports independent motion control. The framework also offers a potential foundation for efficient and accurate point tracking.
- More than 120 FPS rendering and half-hour training convergence make DynMF suitable for real-time dynamic-scene view synthesis.The representation uses a small MLP queried only by time.
- Shared motion bases support locality and rigidity while retaining state-of-the-art novel-view rendering quality.
- Motion decomposition enables independently enabling or disabling scene motions to create novel instances.
- Future Work: The authors identify efficient and accurate tracking as a potential future application because the framework tracks scene points over time.