Source-linked AI summary
SMG: Semantic Motion Graph for Monocular Dynamic Gaussian Splatting
Haozheng Yu, Xinyu Yang, Rundong Luo, Jennifer J. Sun, Bharath Hariharan
TL;DR
Monocular dynamic Gaussian Splatting lacks reliable regularization in occluded and weakly constrained regions, making reconstruction vulnerable to overfitting. SMG models Gaussian motion through semantic graph structure and uncertainty-aware propagation, and it outperforms state-of-the-art methods on real-world dynamic novel-view synthesis benchmarks. Its scope remains limited by dependence on upstream priors and per-scene optimization.
Problem
Monocular dynamic Gaussian Splatting is ill-posed in unobserved regions because sparse viewpoints, occlusions, and unreliable priors provide insufficient motion constraints.
Method
SMG models Gaussian motion as deformation of a semantic motion graph and uses C-ARAP with LRM to guide unreliable nodes using reliable nodes.
Results
SMG outperforms state-of-the-art methods for dynamic novel-view synthesis on real-world benchmarks.
Takeaways & Limitations
Semantic motion modeling provides a structured regularization strategy for dynamic scene reconstruction and supports consistent semantics and 3D tracks.
Takeaways & Limitations
SMG depends on off-the-shelf priors and per-scene optimization, limiting robustness to inaccurate upstream predictions and scalability to large-scale or real-time applications.
Abstract
from arXiv · showhide
We study dynamic Gaussian Splatting from monocular videos. While recent advancements in dynamic Gaussian splatting offer a promising foundation for modeling dynamic scenes, they often overfit to the training views and fail under occlusion or complex scene motion due to the lack of reliable regularization signals in under-constrained regions. We propose Semantic Motion Graph (SMG), a novel approach models the Gaussian motion as the low-rank semantic motion. Our key insight is that the real-world scene motion is often structured by semantic coherence: regions that are spatially close and semantically related tend to exhibit consistent dynamics. To leverage this prior, we construct SMG to model structured motion of the scene. The Gaussian motion is driven by the motion of SMG nodes. We further observe that the uncertainty of Gaussian motion arises from both unreliable off-the-shelf priors and weakly constrained regions during optimization. SMG addresses this by using reliable graph nodes to guide the motion of nearby unreliable nodes. To evaluate dynamic Gaussian splatting under challenging real-world scenarios, we introduce a new multiview dataset collected under an ego-exo setup. Extensive experiments demonstrate that SMG achieves state-of-the-art performance on monocular dynamic Gaussian splatting across challenging real-world benchmarks. Project page: https://smg-gaussian.github.io/.
1 Introduction
Monocular dynamic Gaussian Splatting is under-constrained by sparse viewpoints, unpredictable motion, occlusions, and missing observations. SMG regularizes Gaussian motion through semantic structure and uncertainty-aware propagation, while introducing a challenging multiview benchmark.
- Motivation: Sparse viewpoints, unpredictable motion, and occlusions make monocular dynamic reconstruction ill-posed, causing overfitting, surface drifting, and inconsistent novel-view geometry.Gaussian colors and opacities can fit observed views while hallucinating geometry in unobserved regions.
- Method: SMG models Gaussian motion as deformation of a semantic motion graph built from lifted semantic tracks.The graph restricts propagation within local semantic groups, reflecting coherent motion among semantically related Gaussians.
- Method: Confidence-aware ARAP propagates motion from reliable nodes to less visible or less reliable nodes during dynamic Gaussian optimization.This mechanism addresses uncertainty from unreliable priors and unconstrained regions while maintaining graph topology.
- Method: Local rigid motion control handles uncontrollable node velocity changes when observations are limited, complementing C-ARAP regularization.Together, the mechanisms support motion propagation and generalization to challenging novel views.
- Evaluation: The SMG dataset is a custom multiview dataset captured in an ego-exo setup for complex camera and object motion.It is intended to evaluate dynamic Gaussian Splatting under challenging real-world conditions.
- Results: SMG demonstrates state-of-the-art performance for novel-view synthesis on challenging scenes and benefits related tasks such as 3D tracking.The introduction presents these outcomes as part of the method’s reported contributions.
2 Related Work
Prior work uses neural fields, Gaussian representations, semantic distillation, and data-driven priors for dynamic reconstruction and scene understanding. SMG instead uses semantic cues to regularize Gaussian deformation under sparse observations and supports broader 3D understanding.
- Dynamic Scene Representation: NeRF-based dynamic representations model deformation fields, time-conditioned radiance, or hybrid scene representations, but MLP optimization and rendering can be time-consuming.Follow-up work improves rendering speed or quality while extending neural fields to dynamic scenes.
- Dynamic Scene Representation: 3D Gaussian Splatting offers efficient rendering, fast training, and strong visual effects, and later methods encode Gaussian motion over time.Other approaches use feed-forward transformers or strong priors to regularize dynamic Gaussian optimization.
- Semantic Scene Understanding: Semantic-field methods add labels or foundation-model features to neural fields and Gaussian splatting for 3D queries, segmentation, and understanding.These approaches distill features such as CLIP or DINO into scene representations.
- Semantic Scene Understanding: Dynamic Gaussian semantic fields support semantic-driven tracking, editing, and interactive 4D VQA, but generally treat semantics as auxiliary outputs.SMG instead exploits semantic structure to regularize Gaussian deformation under sparse observations.
- 3D Scene Understanding: 3D scene understanding includes grounding, question answering, captioning, segmentation, and tracking, but monocular data and scarce 3D data remain challenging.SMG aims to support robust spatial understanding from monocular videos, including 3D object segmentation and tracking.
3 Method
SMG models dynamic Gaussian motion through locally connected semantic groups and stabilizes uncertain deformation using confidence-aware propagation and local rigidity controls.
- 3.1 Semantic Motion Graph (SMG): SMG initializes a semantic motion graph from lifted tracks, restricting motion propagation to nearby semantic groups.The graph represents Gaussian motion through deforming SMG nodes.
- 3.1 Semantic Motion Graph (SMG): Graph edges require both spatial neighborhood membership and semantic-feature similarity, preventing propagation across unrelated nearby objects.Trajectory-level KNN uses jointly visible frames, while a semantic threshold gates connectivity.
- 3.2 Modeling Motion Uncertainty in Dynamic Gaussians: C-ARAP uses node confidence to propagate motion from reliable nodes while suppressing unreliable nodes within local semantic groups.Confidence reflects visibility, motion consistency, and local rigidity; asymmetric edge weights preserve a nonzero lower bound.
- 3.2 Modeling Motion Uncertainty in Dynamic Gaussians: C-ARAP penalizes changes in relative displacement direction and length between neighboring nodes.The corresponding terms control vector consistency and length preservation through λc and λl.
- 3.2 Modeling Motion Uncertainty in Dynamic Gaussians: LRM constrains local velocity fields to rigid twists, penalizing velocities that cannot be explained by estimated local rigid motion.This control addresses locally inconsistent velocities and structural drift when geometric constraints are weak.
- 3.3 SMG-guided Dynamic Gaussian Optimization: Dynamic Gaussians are initialized from valid-depth dynamic pixels and driven by blended motions from K nearby semantic SMG nodes.Distance-aware weights sum to one, and dual-quaternion blending provides stable, smooth articulated deformation.
4 Experiments
Experiments evaluate SMG for novel-view synthesis and 3D tracking across Dycheck, NVIDIA, and the challenging SMG dataset, with ablations isolating its motion-regularization components.
- 4.2 Evaluations on Dycheck Dataset: On Dycheck, SMG outperforms baselines in novel-view synthesis, reducing geometry drift and producing more reliable object motion under wide-baseline views.Dycheck evaluates seven scenes using view 0 for training and views 1 and 2 for testing; metrics are computed on covisible mask areas.
- 4.3 Evaluations on NVIDIA Dataset: On NVIDIA, SMG achieves comparable results to state-of-the-art methods while improving motion propagation and preserving cleaner geometry.The narrow-baseline dataset contains mostly observed and temporally stable motion, leaving fewer under-constrained regions for semantic guidance.
- 4.4 Evaluations on SMG Dataset: The SMG dataset uses a moving egocentric camera paired with a static exocentric camera to capture complex camera motion and human–object interactions.These conditions create harder geometric challenges than Dycheck, whose camera motion is relatively mild and dynamics are often simple rotations.
- 4.4 Evaluations on SMG Dataset: SMG consistently outperforms MoSca and OriGS across all four SMG scenes, with larger margins under large viewpoint changes and heavy occlusions.The performance gap is smaller in exo_ball, where the camera baseline is small and the egocentric camera remains largely front-facing.
- 4.5 3D Tracking: SMG produces more accurate and semantically consistent 3D motion tracks than MoSca and OriGS on all reported tracking metrics.The evaluation uses 3D end-point error and the percentage of points within 5 cm and 10 cm of ground-truth 3D locations across five Dycheck scenes.
- 4.6 Ablation Study: Ablations show that SMG-only improves over vanilla dynamic Gaussian Splatting, C-ARAP reduces topology drift, and the full model achieves the best quantitative and qualitative results.LRM alone provides limited improvement, while its combination with C-ARAP complements uncertainty-aware motion regularization; without LRM, occasional floating artifacts can remain.
5 Discussion
SMG achieves state-of-the-art novel-view synthesis, but its reliance on off-the-shelf priors and per-scene optimization limits robustness and scalability. The authors identify improving upstream predictions and feed-forward reconstruction as directions for addressing these constraints.
- Limitations: SMG relies on off-the-shelf priors for initialization, so inaccuracies in those predictions may affect model performance.The authors state that performance should improve as upstream predictions become more accurate.
- Limitations: Per-scene optimization limits SMG’s scalability to large-scale or real-time applications.The authors point to feed-forward 4D Gaussian Splatting as a promising direction for more efficient and flexible reconstruction.
- Discussion: SMG achieves state-of-the-art performance on dynamic novel-view synthesis benchmarks and evaluates challenging real-world camera and scene motion with a multiview dataset.The framework models Gaussian motion as structured semantic motion and uses dynamic uncertainty components to regularize unconstrained regions.