Source-linked AI summary
MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds
Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, Kostas Daniilidis
TL;DR
Monocular casual videos make dynamic-scene reconstruction difficult because observations are limited and camera parameters may be unknown. MoSca uses foundational-model priors, a compact 4D Motion Scaffold, global Gaussian fusion, and joint camera solving; it achieves state-of-the-art benchmark performance and works on some in-the-wild videos, while retaining stated limitations.
Problem
Dynamic-scene reconstruction from casual monocular video is highly challenging and ill-posed because such footage provides limited multi-view stereo cues.
Method
MoSca lifts tracking and depth priors into a compact deformation scaffold, globally fuses anchored Gaussians, and estimates camera poses and focal lengths through bundle adjustment and photometric optimization.
Results
MoSca achieves state-of-the-art performance on standard dynamic-rendering benchmarks and can operate on some in-the-wild videos.
Takeaways & Limitations
The system provides automated 4D reconstruction and rendering from pose-free monocular videos while integrating camera solving and dense correspondence.
Takeaways & Limitations
MoSca relies on accurate 2D long-term tracks and depth estimation, reconstructs only areas visible in the video, and does not model several lighting effects.
Abstract
from arXiv · showhide
We introduce 4D Motion Scaffolds (MoSca), a modern 4D reconstruction system designed to reconstruct and synthesize novel views of dynamic scenes from monocular videos captured casually in the wild. To address such a challenging and ill-posed inverse problem, we leverage prior knowledge from foundational vision models and lift the video data to a novel Motion Scaffold (MoSca) representation, which compactly and smoothly encodes the underlying motions/deformations. The scene geometry and appearance are then disentangled from the deformation field and are encoded by globally fusing the Gaussians anchored onto the MoSca and optimized via Gaussian Splatting. Additionally, camera focal length and poses can be solved using bundle adjustment without the need of any other pose estimation tools. Experiments demonstrate state-of-the-art performance on dynamic rendering benchmarks and its effectiveness on real videos.
1 University of Pennsylvania 2 Stanford University 3 Archimedes, Athena RC
The listed affiliations include the University of Pennsylvania, Stanford University, and Archimedes, Athena RC; Figure 1 presents MoSca reconstructing renderable dynamic scenes from monocular casual videos.
- Figure 1 depicts MoSca reconstructing renderable dynamic scenes from monocular casual videos.
1. Introduction
MoSca addresses the challenging, ill-posed problem of reconstructing dynamic scenes from unposed monocular casual videos by combining foundational-model priors with a structured deformation representation and global Gaussian fusion.
- MoSca is an automated system for reconstructing and rendering dynamic scenes from casual monocular videos with unknown camera parameters.
- The task matters for datasets, spatial-computing content, and embodied agents, but limited multi-view stereo cues make it highly challenging and ill-posed.
- Foundational vision models provide tracking and depth priors, while MoSca adds a physically inspired deformation representation to address their noisy, local, and partial information.
- ARAP regularization supports MoSca, whose 3D lifting enables global fusion of observations across timesteps for rendering at query times.
- The system estimates camera poses and focal lengths through bundle adjustment and photometric optimization without requiring other pose estimators such as COLMAP.
2. Related Works
Related work spans dynamic novel-view synthesis, Gaussian-based scene representations, non-rigid reconstruction, and 2D vision foundation models; MoSca connects these directions in an automated monocular system.
- Dynamic Novel-View Synthesis: Dynamic novel-view synthesis includes synchronized multi-view methods and monocular methods, with monocular inputs introducing greater ambiguity from limited observations.
- Dynamic Novel-View Synthesis: MoSca uses 3D Gaussians for long-term global aggregation and targets unposed RGB videos in a full-stack automated system.
- Non-Rigid Structure-from-Motion: Non-rigid reconstruction methods include category-specific template registration, generic scan fusion, and embedded deformation graphs for modeling dense transformations.
- Non-Rigid Structure-from-Motion: MoSca extends classic embedded graphs by connecting 2D foundation-model priors to dynamic Gaussian splatting.
- 2D Vision Foundation Models: Large-scale pretrained vision models provide priors for segmentation, dense tracking, and monocular depth estimation that help disambiguate partial observations.
3. Method
MoSca reconstructs dynamic scenes from casual monocular video by lifting foundation-model predictions into a structured 4D deformation scaffold, solving camera parameters, and globally fusing time-varying Gaussians. Its deformation field combines sparse trajectory-graph motions, manifold interpolation, physics-inspired regularization, and optimized Gaussian rendering.
- Overview: MoSca lifts monocular video into a 4D representation that globally fuses observations and reconstructs geometry and appearance with dynamic Gaussians.The system also recovers unknown camera focal length and poses.
- Motion Scaffold representation: The Motion Scaffold is an explicit, compact graph whose nodes encode local low-rank motions and interpolate the scene’s dense deformation field.Each node represents a 6-DoF trajectory with per-timestep rigid transformations and a spatial control radius.
- Deformation field: MoSca derives a dense deformation field by interpolating neighboring trajectory motions with Dual Quaternion Blending on the SE(3) manifold.The neighborhood is determined from the scaffold topology, while radial-basis-function skinning weights control local influence.
- Foundation-model priors: Foundation models provide relatively consistent depth, long-term 2D tracks, and epipolar error maps, but MoSca fuses and optimizes these noisy, partial cues into a coherent reconstruction.The inferred tracks and epipolar errors support camera initialization and scaffold construction.
- Camera initialization: Tracklet-based bundle adjustment jointly optimizes camera poses, intrinsics, and depth corrections using reprojection and depth-alignment losses.Confident background tracklets are selected by thresholding their maximum epipolar error, and solved poses are refined during rendering.
- Geometric and Gaussian optimization: Physics-inspired ARAP regularization propagates visible information to invisible scaffold elements and addresses unknown local rotations while enforcing coherent deformation.The final optimized scaffold deforms Gaussians initialized from back-projected foreground depth points at all timesteps, which are then fused at query time.
4. Experiments
MoSca is evaluated on challenging dynamic-view synthesis benchmarks, camera-pose estimation, correspondence tracking, ablations, and real-world videos. It outperforms competing methods on DyCheck, performs strongly on NVIDIA, and supports diverse 4D scene applications.
- Novel View Synthesis: MoSca outperforms all other methods by a large margin on the challenging DyCheck benchmark.DyCheck uses handheld monocular videos with realistic camera motions and test views far from the training trajectory.
- Novel View Synthesis: MoSca achieves high PSNR and competitive LPIPS on the NVIDIA video dataset.The easier facing-forward, small-baseline setting does not fully expose MoSca’s advantages in occluded-area fusion.
- Camera and Correspondence: MoSca achieves comparable or superior camera-pose accuracy to camera-pose-tailored SLAM-based and DuST3R-based methods without requiring known camera intrinsics.Camera solving is evaluated on TUM-dynamics and Sintel.
- Camera and Correspondence: MoSca achieves state-of-the-art correspondence accuracy and substantially improves over the raw BootsTAPIR tracker after reconstruction optimization.The evaluation follows DyCheck and Gaussian Marbles protocols.
- Ablation Study: Both geometric and photometric optimization are critical, while DQB, the topology pyramid, node control, learnable skinning, and global fusion each improve reconstruction quality.The ablation study attributes smoothness, rigidity, shape, expressiveness, and multi-frame benefits to these components.
- Applications: MoSca supports foreground removal, occluder removal, novel-view semantic videos, and flexible editing of reconstructed 4D scenes.Applications are demonstrated on in-the-wild videos, including a challenging cup-game sequence.
5. Limitations and Conclusion
MoSca advances reconstruction and rendering from monocular in-the-wild casual videos, but its performance and completeness remain bounded by tracking, depth, visibility, and lighting limitations.
- MoSca relies on accurate 2D long-term tracks and depth estimation, making improvements in both areas important for better performance.
- The framework reconstructs only areas visible at some point in the video, leaving never-visible regions for future hallucination with large-scale 2D or video diffusion priors.
- Lighting effects including shadows, reflections, liquids, and exposure changes are not explained by deformation alone and may cause background artifacts.
- MoSca takes a step toward reconstruction and rendering from monocular in-the-wild casual videos and aims to inspire further study of dynamic physical-world understanding.