Source-linked AI summary
Shape of Motion: 4D Reconstruction from a Single Video
Qianqian Wang, Vickie Ye, Hang Gao, Weijia Zeng, Jake Austin, Zhengqi Li, Angjoo Kanazawa
TL;DR
Monocular dynamic reconstruction is highly ill-posed, especially for generic scenes captured from a single casually recorded video. The paper represents persistent 3D motion with compact SE(3) bases and fuses monocular depth with long-range 2D tracks, achieving strong performance across long-range tracking and novel view synthesis while retaining practical limitations.
Problem
Generic monocular 4D reconstruction is highly ill-posed, while existing approaches may require templates, quasi-static scenes, multiple views, or additional depth sensing.
Method
The method represents dynamic scenes as persistent 3D Gaussians with shared SE(3) motion bases and consolidates monocular depth estimates with long-range 2D tracks.
Results
The method significantly outperforms prior methods on long-range 2D/3D tracking and novel view synthesis across synthetic and real-world dynamic video datasets.
Takeaways & Limitations
Persistent world-coordinate trajectories provide globally consistent 3D tracking together with real-time novel view synthesis from a single monocular video.
Takeaways & Limitations
The method requires per-scene test-time optimization, depends on off-the-shelf predictions that may degrade in textureless regions or under large motions, and requires user masks for moving objects.
Abstract
from arXiv · showhide
Monocular dynamic reconstruction is a challenging and long-standing vision problem due to the highly ill-posed nature of the task. Existing approaches depend on templates, are effective only in quasi-static scenes, or fail to model 3D motion explicitly. We introduce a method for reconstructing generic dynamic scenes, featuring explicit, persistent 3D motion trajectories in the world coordinate frame, from casually captured monocular videos. We tackle the problem with two key insights: First, we exploit the low-dimensional structure of 3D motion by representing scene motion with a compact set of SE(3) motion bases. Each point's motion is expressed as a linear combination of these bases, facilitating soft decomposition of the scene into multiple rigidly-moving groups. Second, we take advantage of off-the-shelf data-driven priors such as monocular depth maps and long-range 2D tracks, and devise a method to effectively consolidate these noisy supervisory signals, resulting in a globally consistent representation of the dynamic scene. Experiments show that our method achieves state-of-the-art performance for both long-range 3D/2D motion estimation and novel view synthesis on dynamic scenes. Project Page: https://shape-of-motion.github.io/
1. Introduction
The paper addresses monocular 4D reconstruction by modeling persistent 3D motion with compact SE(3) bases and fusing noisy depth and 2D-track priors into a coherent representation.
- Monocular reconstruction of complex dynamic 3D scenes remains challenging because the problem is highly ill-posed and prior methods often require synchronized views or additional sensors.
- The method exploits the insight that complex image dynamics can arise from continuous, simple rigid motions in 3D.
- The system fuses complementary but noisy monocular depth estimates and long-range 2D tracks into a globally coherent scene geometry and motion representation.
- Dynamic scenes are represented with persistent 3D Gaussians whose motion is modeled using compact shared SE(3) motion bases.
- The approach recovers full-video 3D trajectories rather than only consecutive-frame scene flow, supporting long-range tracking and novel view synthesis.
2. Related Work
Related work spans 2D correspondence and tracking methods, monocular 3D trajectory estimation, and dynamic reconstruction or view synthesis approaches using additional sensing or implicit motion representations.
- Correspondences and Tracking: Prior correspondence methods estimate dense optical-flow fields or generate sparse long trajectories through keypoint matching.
- Correspondences and Tracking: Monocular 3D motion methods include self-supervised and test-time optimization approaches, followed by feed-forward methods for general-scene trajectory estimation.
- Correspondences and Tracking: Feed-forward monocular trajectories remain in frame space, entangling object and camera motion, whereas this work recovers persistent trajectories in world coordinates.
- Dynamic Reconstruction and View Synthesis: Dynamic reconstruction and view synthesis have also included non-rigid approaches that require RGB-D sensors and dynamic NeRFs that represent motion implicitly.
3. Method
The method reconstructs a dynamic scene as persistent 3D Gaussians whose geometry and motion are optimized from monocular video, depth, tracks, and masks. Shared SE(3) motion bases provide compact world-coordinate trajectories, while differentiable rendering and temporal consistency losses consolidate noisy priors into a coherent representation.
- Inputs: The input is a monocular video with camera parameters, monocular depth maps, long-range 2D tracks, and moving-object masks.Camera parameters are estimated by MegaSaM, depth maps by methods such as Depth Anything, and tracks by TAPIR.
- Scene representation: The scene is represented by persistent canonical 3D Gaussians whose appearance and geometry are shared across time.Each Gaussian stores a canonical mean, orientation, scale, opacity, and color, while its position and orientation vary over time.
- Rendering and tracking: Differentiable rasterization renders RGB images, depth maps, masks, and pixelwise world-coordinate trajectories for supervision and tracking.Rendered outputs are compared with per-frame color, depth, and mask inputs, while temporal correspondence losses supervise 2D tracks and track depths.
- Motion representation: Each Gaussian’s motion is a weighted combination of globally shared SE(3) basis trajectories, with coefficients learned per Gaussian.The compact basis representation regularizes trajectories toward low-dimensional motion and encourages similarly moving Gaussians to have similar coefficients.
- Optimization: The optimization jointly learns canonical Gaussian geometry, motion coefficients, and basis trajectories by fitting noisy 3D tracks under temporal smoothness constraints.Motion bases are initialized by clustering noisy 3D-track velocities and weighted Procrustes alignment before optimization.
4. Experiment
The experiments evaluate long-range 3D and 2D tracking and novel view synthesis across dynamic-scene datasets, with the iPhone benchmark providing especially broad real-world evaluation. The method outperforms baselines across the three tasks and produces globally consistent tracks and high-quality novel views.
- Task Specification: The evaluation covers long-range 3D tracking, long-range 2D tracking, and novel view synthesis on scenes with substantial motion.The iPhone dataset includes casual real-world captures and annotations supporting all three tasks.
- Evaluation on iPhone Dataset: The method produces dense, full-length 3D tracks, although Figure 3 displays only 50-frame trail segments for selected grid queries.TAPIR + DA tracks are overlaid on the method’s renderings because that baseline cannot produce novel views.
- Evaluation on iPhone Dataset: Table 1 reports state-of-the-art performance across 3D point tracking, 2D point tracking, and novel view synthesis on the iPhone dataset.Tracking-focused baselines cannot synthesize novel views, while view-synthesis methods struggle with tracking.
- Baselines: Motion-coefficient PCA correlates with rigid groups, producing constant color for an example block undergoing rigid motion.The visualization also includes predicted depth maps.
- Evaluation on iPhone Dataset: The predicted 3D tracks are described as smoother and more accurate than competing qualitative results, while D-3DGS and HyperNeRF degrade under significant motion.TAPIR + DA handles large motions but produces noisy and erroneous 3D tracks.
- Evaluation on Kubric Dataset: Across all reported 3D tracking metrics on Kubric, the method outperforms the baselines.The paper also reports a comparable result to Dynamic Gaussian Marbles on the NVIDIA dataset.
- Ablation Studies: Removing SE(3) initialization noticeably hurts performance, while removing 2D track supervision causes a significant drop in 3D tracking performance.These ablations test initialization and supervision strategies.
5. Discussion
The method remains constrained by per-scene test-time optimization, reliance on off-the-shelf predictions, and user-provided moving-object masks. The discussion identifies feed-forward reconstruction, improved priors, and automated segmentation as potential directions beyond these constraints.
- Discussion: Per-scene test-time optimization hinders streamable applications.The limitation is shared with most prior monocular view synthesis methods.
- Discussion: Camera-pose, geometry, and motion predictions may degrade in textureless regions or under large motions.The method can also benefit from advances in these components.
- Discussion: The pipeline requires user input to mask moving objects.Recent moving-object segmentation methods are identified as possible replacements for the current manual solution.
- Discussion: Feed-forward methods for joint reconstruction and tracking offer a potential alternative to per-scene test-time optimization.The paper characterizes their progress as encouraging rather than definitive.
B. Additional Training Details
Initialization estimates motion bases from temporally observed point clusters, then optimization fits these bases and per-track coefficients to lifted 3D and 2D tracking evidence while enforcing smooth motion.
- Per-cluster Procrustes alignment estimates SE(3) transformations between canonical and observed point sets, excluding occluded pairs and weighting matches by TAPIR uncertainty.
- Motion coefficients are initialized from exponentially decayed distances between each track’s canonical position and cluster centers.
- Optimization fits canonical means, motion coefficients, and motion bases to lifted 3D tracks using ℓ1 losses while regularizing quaternion and translation acceleration with ℓ2 penalties.Adam optimization runs for 2k steps, with learning rates exponentially decayed to one-tenth of their initial values.
B.2. Training Details
Training jointly optimizes canonical Gaussian properties, rendering consistency, geometry regularization, motion supervision, rigidity, smoothness, and monocular normal alignment.
- Gaussian means, opacity, scale, rotation, color, SE(3) motion bases, and motion coefficients are optimized with Adam using separately specified learning rates.Training randomly samples batches of 8 query frames.
- Rendering losses match per-frame pixelwise color, depth, and mask inputs, while depth-gradient and isotropic-scale penalties regularize surface geometry.The depth and mask loss weights are 0.5 and 1.0, respectively.
- Motion losses supervise normalized 2D tracks and track depth, preserve within-part neighbor distances, and regularize acceleration of translation and quaternion bases.Foreground part masks come from independent per-frame SAM segmentation, with 4 masks, 32 centers, and 16 neighbors sampled per query frame.
- Replacing 3D Gaussian Splatting with 2D Gaussian Splatting adds monocular normal supervision for rendered and depth-derived normals.The normal loss blends splat normals using ray-intersection weights and aligns them with monocular predictions.
C. Additional Evaluation Details
Evaluation uses ground-truth camera poses for Kubric and refines inaccurate iPhone ARKit poses with COLMAP while preserving metric scale through a global SIM(3) transformation.
- Kubric evaluation directly uses ground-truth camera poses, whereas iPhone evaluation refines ARKit poses with COLMAP while fixing camera intrinsics.A global SIM(3) transformation is computed per scene to maintain metric scale after refinement.
D. Visualization of Kubric Experiment
The visualization projects optimized motion coefficients into their first three PCA components to inspect how motion representations organize across evaluation scenes.
- Motion coefficients are qualitatively grouped coherently with individual moving objects in the scene.Figure 6 visualizes the first three PCA components of the optimized coefficients.
E. Visualization of Complex Dynamic Scene
The method handles complex scenes with multiple moving objects and visualizes both novel views and motion-coefficient PCA over time.
- The method handles scenes containing multiple moving objects.
- Figure 7 visualizes novel views of a real-world complex scene alongside motion-coefficient PCA.
- The school-girl DAVIS sequence is shown at time steps 0 and 54 using novel-view and motion-coefficient PCA visualizations.
F. NVIDIA Dataset Evaluation
The NVIDIA Dynamic Scenes evaluation uses one static-camera video for training and three other camera views for evaluation across seven scenes, with a specified resolution and evaluation protocol.
- Experiments cover seven scenes from the NVIDIA Dynamic Scenes dataset.
- Video from static camera 4 is used for training, while cameras 3, 5, and 6 provide evaluation views.
- All seven experiments use half-resolution images for consistency with Dynamic Gaussian Marbles, although the method is compatible with high-resolution images.
- The evaluation applies covisibility masks between training and test views, whereas Dynamic Gaussian Marbles inpaints unobserved test-view regions.
G. DynMF Implementation Detail
The DynMF implementation represents motion bases with MLPs, uses ten bases with positional encoding, and applies shared coefficients to Gaussian translations and rotations.
- Each DynMF motion basis is represented as an MLP.
- The implementation uses 10 motion bases and positional encoding with 10 frequency bandwidths for normalized time input t/T.
- Displacement motion is applied to Gaussian means and rotation quaternions using translation and rotation motion bases.
- The motion coefficients are shared between mean and rotation motions.