Source-linked AI summary
MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion
Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, Ming-Hsuan Yang
TL;DR
Dynamic-scene geometry is difficult because existing approaches often use complex multi-stage pipelines, while suitable posed videos with depth labels are scarce. MonST3R adapts DUSt3R by estimating per-timestep pointmaps and fine-tuning on limited dynamic data, achieving strong video depth and camera-pose results with promising feed-forward 4D reconstruction.
Problem
Dynamic-scene geometry remains challenging because common methods decompose the task into multiple subtasks, and suitable dynamic posed videos with depth labels are scarce.
Method
MonST3R directly estimates dynamic-scene geometry as per-timestep pointmaps and adapts DUSt3R through fine-tuning on suitable small-scale datasets.
Results
MonST3R achieves strong performance on video depth and camera pose estimation, with promising results for primarily feed-forward 4D reconstruction.
Takeaways & Limitations
A geometry-first pointmap representation can support dynamic-scene reconstruction and downstream video tasks without an explicit motion representation.
Takeaways & Limitations
The approach depends on scarce dynamic, posed training data with depth labels, and the original DUSt3R model has static-training limitations on moving scenes.
Abstract
from arXiv · showhide
Estimating geometry from dynamic scenes, where objects move and deform over time, remains a core challenge in computer vision. Current approaches often rely on multi-stage pipelines or global optimizations that decompose the problem into subtasks, like depth and flow, leading to complex systems prone to errors. In this paper, we present Motion DUSt3R (MonST3R), a novel geometry-first approach that directly estimates per-timestep geometry from dynamic scenes. Our key insight is that by simply estimating a pointmap for each timestep, we can effectively adapt DUST3R's representation, previously only used for static scenes, to dynamic scenes. However, this approach presents a significant challenge: the scarcity of suitable training data, namely dynamic, posed videos with depth labels. Despite this, we show that by posing the problem as a fine-tuning task, identifying several suitable datasets, and strategically training the model on this limited data, we can surprisingly enable the model to handle dynamics, even without an explicit motion representation. Based on this, we introduce new optimizations for several downstream video-specific tasks and demonstrate strong performance on video depth and camera pose estimation, outperforming prior work in terms of robustness and efficiency. Moreover, MonST3R shows promising results for primarily feed-forward 4D reconstruction.
1 INTRODUCTION
Dynamic-scene geometry remains difficult because existing systems decompose estimation into slow, brittle stages, while suitable posed video data with depth labels is scarce. MonST3R adapts DUSt3R’s pointmap representation through small-scale fine-tuning and reports strong downstream results.
- Motivation: Existing dynamic-scene methods decompose geometry into depth, flow, or trajectory estimation and combine them through multi-stage algorithms or global optimization.These methods are described as slow, brittle, and prone to errors at each step.
- Motivation: End-to-end dynamic-scene geometry requires a representation for camera motion, object motion, and deformations, plus annotated training datasets.The paper explores geometry-only representation because motion is difficult to supervise with limited annotations.
- Approach: MonST3R estimates per-timestep pointmaps in a shared camera coordinate frame, allowing dynamic objects to appear at multiple locations according to their motion.Pairs of pointmaps can be aligned using static scene elements, generalizing DUSt3R to dynamic scenes.
- Limitations of DUSt3R: DUSt3R struggles on dynamic scenes because static-scene training causes incorrect pointmap alignment and incorrect foreground depth estimation.It may align moving foreground objects instead of static backgrounds and place foreground subjects in the background.
- Training and Results: Dynamic posed videos with depth labels are scarce, so the paper identifies several small-scale datasets and uses training strategies that maximize their value during fine-tuning.The authors report successful adaptation of DUSt3R despite this limited data.
- Training and Results: MonST3R reports promising performance on video depth and camera pose estimation, with robustness and speed advantages over prior work.The paper also reports competitive results with specialized techniques and promising primarily feed-forward 4D reconstruction.
2 RELATED WORK
Related work addresses dynamic-scene geometry through structure-from-motion, visual SLAM, joint depth-and-motion estimation, learned visual odometry, and video depth or 4D reconstruction. These approaches vary from self-supervision and trajectory modeling to test-time optimization with generative or Gaussian representations.
- Dynamic geometry and localization: Structure-from-motion and visual SLAM estimate 3D structure and camera localization but struggle when moving objects violate the epipolar constraint.Recent methods therefore explore joint depth, pose, and residual-motion estimation, sometimes using motion segmentation.
- Static 3D reconstruction: DUSt3R learns strong 3D priors through pointmaps that implicitly infer camera intrinsics, relative pose, and two-view geometry, but targets static scenes.MonST3R shares this pointmap representation while extending it to dynamic scenes.
- Learning-based visual odometry: Learning-based visual odometry replaces hand-designed components and supports large-scale training for generalization with moving objects.Trajectory-based methods estimate long-term trajectories, classify dynamic and static motion, and localize the camera through bundle adjustment.
- Video depth estimation: Monocular depth methods can generalize across domains, but video estimates may flicker because they process single frames with invariant training objectives.Earlier video-depth methods improve temporal consistency through test-time fine-tuning or self-supervision.
- 4D reconstruction: Recent 4D reconstruction methods use precomputed motion, depth, camera, and intrinsic estimates before optimizing deformable 3D Gaussians for space-time reconstruction.These systems target novel-view synthesis in both space and time.
3 METHOD
MonST3R adapts DUSt3R for dynamic scenes by estimating per-timestep pointmaps, then uses fine-tuning and video-specific optimization to support depth and pose estimation.
- Main idea: MonST3R estimates pointmaps for dynamic scenes over time, including moving scene elements.Each image or image pair produces pointmaps with associated confidence maps.
- Training for dynamics: The dynamic-scene training setup requires synchronized images, camera poses, and depth, but suitable real-world data are scarce.The authors therefore use several datasets, including synthetic and real-world sources, for fine-tuning.
- Training strategies: Fine-tuning freezes the encoder and updates the prediction head and decoder to improve data efficiency on the limited dataset mixture.Training also samples frame pairs from videos, with asymmetric dataset weighting and image downsampling to 512 pixels on the largest dimension.
- Downstream applications: Relative camera pose is recovered with per-pixel 2D-3D correspondences and PnP because dynamic objects violate correspondence assumptions used by epipolar and 3D alignment methods.RANSAC improves robustness to outliers, while confidence thresholds define valid correspondences.
- Dynamic global point clouds and camera pose: A sliding temporal window and strided sampling reduce the cost of processing pairwise frame pointmaps for video.The resulting intermediates support global point-cloud and camera-pose optimization.
- Video depth: Video depth is obtained directly from the per-frame depthmaps parameterizing the globally optimized pointmaps.This unified representation also supports camera-pose estimation and related video-specific processing.
4 EXPERIMENTS
MonST3R is evaluated on video depth, single-frame depth, camera pose, qualitative reconstruction, and ablations. It achieves competitive or better results while retaining efficiency and showing robustness across dynamic and static settings, though practical limitations remain.
- Experimental setup: The evaluation compares MonST3R with specialized video-depth, single-frame-depth, joint depth-and-pose, and visual-odometry methods.
- Experimental setup: MonST3R is evaluated on video depth using Sintel, Bonn, and KITTI, covering dynamic and static, indoor and outdoor, realistic and synthetic data.
- Depth estimation: MonST3R achieves competitive or better video-depth results, including performance that surpasses the specialized DepthCrafter method, while remaining competitive with DUSt3R on single-frame depth.
- Camera pose estimation: MonST3R achieves the best accuracy on Sintel and ScanNet among joint depth-and-pose methods and remains competitive with pose-only methods without ground-truth camera intrinsics.
- Qualitative comparison: Qualitative comparisons show reliable camera trajectories and dynamic-scene geometry, whereas DUSt3R struggles with moving foreground objects and CasualSAM can fail in some cases.
- Ablation study: The ablation finds that all datasets improve camera-pose estimation, decoder-and-head fine-tuning performs best, and the proposed losses improve pose with minimal video-depth impact.
- Limitations: Dynamic camera intrinsics require careful tuning or manual constraints, small sliding windows are vulnerable to long-term occlusion, and out-of-distribution inputs remain difficult.
5 CONCLUSIONS
MonST3R directly estimates dynamic-scene geometry with per-timestep pointmaps and extracts camera pose and video depth from a relatively small fine-tuning dataset. It achieves strong downstream results and promising qualitative camera-pose performance, including challenging cases.
- MonST3R directly estimates dynamic-scene geometry and extracts downstream camera pose and video depth from per-timestep pointmaps.
- Despite fine-tuning on a relatively small training dataset, MonST3R achieves strong downstream results that surpass specialized techniques.
- MonST3R significantly outperforms baselines on challenging Sintel cases and performs comparably to or better than baselines on the remaining qualitative results.
A.3 JOINT DEPTH & CAMERA POSE RESULTS
MonST3R jointly estimates camera pose and scene geometry more reliably than the compared methods across additional dynamic video scenes.
- On mbike-trick, MonST3R avoids the large geometry and camera-pose errors made by CasualSAM and the late-video pose errors made by DUSt3R.
- MonST3R correctly estimates both geometry and camera pose in the Davis scenes mbike-trick, train, and dog.CasualSAM and DUSt3R each show geometry, pose, or alignment errors in these examples.
- On train, MonST3R correctly aligns the train and track while recovering camera pose, unlike the compared methods.
- On dog, MonST3R avoids the smeared geometry of CasualSAM and the frame-misalignment errors of DUSt3R.
A.4 PAIRWISE POINTMAPS
MonST3R produces useful pairwise pointmaps and dynamic-scene estimates directly from frames, retaining robustness to difficult camera and motion conditions.
- Fine-tuned MonST3R handles changing camera intrinsics in direct pairwise pointmap predictions without optimization or post-processing.
- MonST3R handles nearly non-overlapping frame pairs with motion, whereas DUSt3R misaligns them using the foreground object.
- Fine-tuning improves representation of large-scale scenes, where DUSt3R predicts flat geometry.
- On Davis, MonST3R outputs reliable camera trajectories and dynamic-scene geometry compared with CasualSAM and DUSt3R.
- MonST3R produces plausible static/dynamic masks, while DUSt3R fails because of camera-pose and depth errors.
B.1 ABLATION ON TRANING/INFERENCE WINDOW SIZE
Window-size ablations show that larger inference windows generally improve performance up to the training-window size, while stride-based sampling improves efficiency.
- Performance generally improves as inference window size increases, up to the training window size.The ablation measures camera pose with ATE and video depth with Abs Rel on Sintel.
- For training windows of 7 or 9, inference with window size 7 and stride 2 outperforms window size 4 at similar memory use.The reported memory use is 19.9 GB versus 20.1 GB.
- With training window size 9, window size 9 and stride 2 outperforms window size 7 while reducing memory consumption by 20%.
- Stride-based sampling provides a better trade-off between inference-window size and computational cost.
B.2 ABLATION ON LOSS WEIGHT SENSITIVITY
Loss-weight sensitivity is modest for the optical-flow term but more pronounced for camera-trajectory smoothness, especially when the smoothness weight is too high.
- The optical-flow loss weight w_flow does not significantly affect overall performance, although some settings slightly improve other metrics.
- The camera-trajectory smoothness weight w_smooth has more noticeable effects on camera pose and video-depth performance.
- Lower w_smooth values cause small overall differences but noticeably reduce RPE performance.
- Setting w_smooth too high degrades performance, likely because the camera trajectory becomes over-constrained.
C FULLY FEED-FORWARD RECONSTRUCTION
MonST3R enables predominantly feed-forward reconstruction by aligning every video frame to a shared anchor frame. This design improves runtime but remains sensitive to anchor selection and independent frame alignment.
- Anchor-frame alignment: All T video frames are paired with a common anchor frame, such as the first or middle frame, to share one camera coordinate system.The construction uses pairs denoted {t_anchor ← t | t ∈ 1, ..., T}.
- Real-time reconstruction: Aligning all frames to the middle frame enables real-time reconstruction from monocular video input.Figure A8 provides a qualitative real-time reconstruction result.
- Runtime: Approximately 40 FPS is achieved on a single RTX4090 GPU through the feed-forward alignment strategy.The approach may support streaming reconstruction by adaptively updating the anchor frame.
- Limitations: Reconstruction quality is sensitive to anchor-frame choice, and independent frame alignment can introduce shifting artifacts because global information is not shared.These limitations constrain the current approach despite its potential for streaming, real-time, fully feed-forward reconstruction.
- Optimization losses: Camera trajectory smoothness loss penalizes large changes in consecutive rotations and translations to encourage smoother pose transitions.For each frame, the loss is defined using rotation R_t and translation T_t, with the Frobenius and Euclidean norms specified.
- Optimization losses: Flow projection loss enforces consistency between camera-induced flow and estimated optical flow in regions identified as static.Camera-induced flow uses global camera parameters, intrinsics, and depth; the static mask is initialized from sampled pairwise masks and updated with global parameters.