Source-linked AI summary
MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos
Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, Noah Snavely
TL;DR
Casual monocular videos of dynamic scenes remain difficult for conventional and neural SfM/SLAM because of limited parallax, uncontrolled motion, and varied focal lengths. MegaSaM extends deep visual SLAM with motion maps, monocular depth priors, and uncertainty-aware optimization, achieving stronger camera and depth estimation across synthetic and real videos with competitive runtime. Its scope remains limited by extreme scene dynamics, absent trackable content, and changing focal lengths or strong radial distortion.
Problem
Conventional SfM and monocular SLAM often struggle with casual dynamic videos containing limited parallax, uncontrolled camera motion, and varied focal lengths.
Method
MegaSaM extends deep visual SLAM by integrating motion probability maps, monocular depth priors, and uncertainty-aware global bundle adjustment.
Results
MegaSaM significantly outperforms prior and concurrent baselines in camera and depth estimation accuracy while achieving competitive or superior runtime performance.
Takeaways & Limitations
The pipeline scales deep visual SLAM to in-the-wild dynamic videos with unconstrained camera paths and complex scene dynamics.
Takeaways & Limitations
The system can fail when moving objects dominate the image or nothing is trackable, and it cannot handle varying focal lengths or strong radial distortion within a video.
Abstract
from arXiv · showhide
We present a system that allows for accurate, fast, and robust estimation of camera parameters and depth maps from casual monocular videos of dynamic scenes. Most conventional structure from motion and monocular SLAM techniques assume input videos that feature predominantly static scenes with large amounts of parallax. Such methods tend to produce erroneous estimates in the absence of these conditions. Recent neural network-based approaches attempt to overcome these challenges; however, such methods are either computationally expensive or brittle when run on dynamic videos with uncontrolled camera motion or unknown field of view. We demonstrate the surprising effectiveness of a deep visual SLAM framework: with careful modifications to its training and inference schemes, this system can scale to real-world videos of complex dynamic scenes with unconstrained camera paths, including videos with little camera parallax. Extensive experiments on both synthetic and real videos demonstrate that our system is significantly more accurate and robust at camera pose and depth estimation when compared with prior and concurrent work, with faster or comparable running times. See interactive results on our project page: https://mega-sam.github.io/
1. Introduction
MegaSaM addresses the difficulty of estimating camera parameters and scene geometry from casual monocular videos containing limited parallax, broad focal lengths, and dynamic content. It extends deep visual SLAM with learned motion handling, monocular depth priors, and uncertainty-aware bundle adjustment for robust in-the-wild estimation.
- Motivation: Casual monocular videos challenge conventional SfM and SLAM because they combine limited camera parallax, varied focal lengths, and moving scene content.These methods were developed for stationary scenes with large camera baselines and often falter in uncontrolled settings.
- Approach: MegaSaM presents a full pipeline for accurate, fast, and robust camera tracking and depth estimation from in-the-wild dynamic videos.The system reexamines and extends a prior deep visual SLAM framework.
- Approach: Learned flow and uncertainty predictions support efficient camera pose estimation in challenging dynamic videos.The framework uses a differentiable bundle-adjustment layer whose intermediate predictions are learned from camera and flow supervision.
- Approach: MegaSaM integrates monocular depth priors and motion probability maps into differentiable SLAM to handle dynamic scene content.These intermediate predictions help account for object motion within the SLAM optimization.
- Robustness: Uncertainty-aware global bundle adjustment improves robustness when the input video poorly constrains camera parameters.The method also obtains consistent video depths without test-time network fine-tuning.
- Results: Extensive synthetic and real-world evaluations show significantly better camera and depth accuracy than prior and concurrent baselines, with competitive or superior runtime.The reported gains cover both estimation accuracy and runtime performance.
2. Related Work
Related work addresses difficult dynamic-video reconstruction through optimization, monocular depth modeling, and dynamic radiance fields. MegaSaM instead couples differentiable SLAM with intermediate predictions for dynamic scenes, while its outputs can support downstream reconstruction systems.
- Visual SLAM and SfM: Conventional and deep SLAM/SfM methods estimate correspondences or scene representations before optimizing camera parameters and 3D structure.Deep systems often use neural correspondences, radiance fields, or global point clouds.
- Visual SLAM and SfM: Deep visual SLAM and SfM typically assume predominantly static scenes with sufficient camera baselines, limiting performance under dynamics or low parallax.Performance can degrade significantly or fail entirely when these assumptions are violated.
- Related dynamic-video methods: Robust-CVD and CasualSAM jointly optimize camera parameters and dense depth, while Particle-SfM and LEAP-VO use moving-object masks from long-range trajectories.These approaches represent distinct strategies for addressing dynamic videos.
- MegaSaM: MegaSaM couples a differentiable SLAM system with intermediate predictions of underlying dynamic scenes.This positioning distinguishes its approach from related methods that separately optimize or infer scene components.
- Monocular depth: Single-image monocular depth models generalize well but tend to produce temporally inconsistent video depth, motivating test-time optimization and direct video-depth prediction.MegaSaM follows this line of work while targeting consistent depths within its camera-estimation pipeline.
- Dynamic scene reconstruction: Dynamic radiance-field methods often require camera parameters or video depth maps as inputs, which MegaSaM can provide for downstream reconstruction.The paper describes its outputs as complementary to most such techniques.
3. MegaSaM
MegaSaM extends deep visual SLAM for unconstrained dynamic videos by combining learned correspondence estimation with motion-aware weighting, monocular-depth priors, and uncertainty-aware bundle adjustment.
- Problem and goal: MegaSaM estimates camera poses, unknown focal length, and dense video depth without constraining camera or object motion.Its camera tracking and depth modules build on deep visual SLAM and casual structure-and-motion frameworks.
- Deep visual SLAM formulation: The system uses a differentiable learned bundle-adjustment layer that iteratively updates low-resolution disparity and camera-pose variables over overlapping frame pairs.The frame graph is built dynamically from image pairs with overlapping fields of view, while learned correspondence fields provide the motion estimates.
- Deep visual SLAM formulation: MegaSaM jointly optimizes poses, focal length, and disparity by minimizing weighted reprojection error, using Levenberg–Marquardt and a Schur-complement update for efficient differentiable BA.The weights are derived from predicted correspondence uncertainty, and the disparity block is diagonal because each pairwise term contains one disparity variable.
- Scaling to dynamic videos: For dynamic scenes, the model predicts object movement maps and combines them with pairwise flow confidence to downweight dynamic elements during bundle adjustment.The movement maps use multiframe context and are trained to identify pixels corresponding to dynamic content.
- Scaling to dynamic videos: A two-stage training scheme first learns ego-motion correspondences on static synthetic scenes, then freezes that model while fine-tuning the movement module on dynamic videos.The staged design separates scene-dynamics learning from correspondence learning, producing more stable differentiable-BA training and accurate dynamic-video camera estimation.
- Robust inference and depth: MegaSaM uses mono-depth priors selectively and uncertainty-aware global BA to address degenerate geometry when camera parallax is limited, while avoiding test-time network fine-tuning.The full configuration produces better camera and scene geometry on difficult near-rotational sequences, and the depth design differs from CasualSAM by optimizing per-video disparity and uncertainty variables.
4. Experiments
MegaSaM is evaluated on synthetic, real-world, and in-the-wild dynamic videos using camera-pose and depth metrics, qualitative comparisons, and ablations. It achieves the best camera-tracking accuracy across reported metrics and significantly outperforms depth baselines while remaining competitive in running time.
- Datasets: The evaluation spans Sintel, DyCheck, and in-the-wild videos with complex motion, handheld capture, long durations, or uncontrolled camera paths.Sintel contains animated sequences with complex object motions and camera paths; DyCheck contains handheld real-world videos; in-the-wild footage includes 100–600 frames and uncontrolled motion.
- Metrics: Camera pose is evaluated with ATE, RTE, and RRE after unit-length normalization and global Sim(3) alignment to ground truth.The reported metrics cover absolute translation, relative translation, and relative rotation error; running time is also reported.
- Quantitative comparisons: MegaSaM achieves the best camera-tracking accuracy on all reported error metrics in calibrated and uncalibrated settings while remaining competitive in running time.The comparison is reported across three benchmark tables and includes both known- and unknown-focal-length settings.
- Quantitative comparisons: MegaSaM significantly outperforms other baselines on all reported video-depth metrics.Table 4 evaluates abs-rel and log-rmse, where lower is better, and δ1.25, where higher is better.
- Ablations: Ablations evaluate vanilla DROID-SLAM, removal of mono-depth initialization or movement maps, alternative training, and other tracking configurations.The study is reported on Sintel, with configurations described in Section 4.3.
- Qualitative comparisons: Qualitatively, MegaSaM estimates camera trajectories closest to ground truth and produces more accurate, detailed, and temporally consistent depths than CasualSAM and MonST3R.On challenging DAVIS examples, CasualSAM produces distorted point clouds and MonST3R confuses rotational camera motion with translation, whereas MegaSaM yields more consistent geometry.
5. Discussion and conclusion
MegaSaM produces accurate camera parameters and consistent depths for casual monocular videos of dynamic scenes, scaling to varied in-the-wild footage. Its main limitations are extreme scene dynamics, absent trackable content, varying focal lengths, and strong radial distortion.
- Limitations: Camera tracking can fail when moving objects dominate the image or when nothing can be tracked reliably.
- Limitations: The system cannot handle videos with varying focal lengths or strong radial distortion within the video.
- MegaSaM produces accurate camera parameters and consistent depths from casual monocular videos of dynamic scenes.
- The pipeline scales to in-the-wild footage with varying durations, unconstrained camera paths, and complex scene dynamics.
A.1. System Overview
MegaSaM separates camera and scene-structure estimation into two stages. It first estimates camera poses, focal length, and low-resolution disparity using differentiable bundle adjustment, then estimates consistent video depth.
- The system follows a two-stage structure-from-motion pipeline that separates camera estimation from consistent video-depth estimation.
- In the first stage, differentiable bundle adjustment estimates camera poses, focal length, and low-resolution disparity from monocular video.
- Low-resolution disparity is initialized with monocular depth maps predicted by off-the-shelf models.
A.2. Framework and Architecture
The framework builds on DROID-SLAM components and performs iterative differentiable bundle-adjustment updates from learned flow, confidence, and motion-probability predictions. Training separates synthetic ego-motion pretraining from dynamic finetuning of the motion module.
- MegaSaM follows DROID-SLAM for feature extraction, correlation construction, and iterative bundle-adjustment updates.
- Each model input is a pair of video frames, while learned flow, confidence, and motion-probability predictions drive the updates.
- Feature and context encoders represent each input frame with two low-resolution feature maps.
- The correlation layer constructs a 4D volume whose entries are inner products between feature vectors from an image pair.
- The flow network is pretrained on synthetic videos, then frozen while a motion module is finetuned to predict additional object-motion information.
A.3. Consistent Video Depth Optimization
Consistent video depth is optimized after camera estimation, using uncertainty-weighted flow and temporal consistency together with monocular depth priors. The optimization uses multiple depth-related cost functions and fixed frame intervals.
- Consistent video depth is obtained through an additional first-order optimization over video disparity and per-frame aleatoric uncertainty maps.
- Camera parameters are fixed during depth optimization, following conventional structure-from-motion pipelines rather than jointly optimizing camera and scene structure.
- The objective contains three main cost functions: flow reprojection, temporal depth consistency, and a depth prior.
- Flow reprojection uses uncertainty-weighted l1 error between estimated optical flow and correspondences induced by camera motion and disparity.
- The temporal depth loss encourages disparity consistency across frames according to estimated 2D optical flow.
- The depth prior limits drift from monocular initialization using scale-invariant, gradient-matching, and surface-normal losses.
- The gradient loss is applied at multiple scales and emphasizes pixels where estimated disparity deviates significantly from the original monocular depth.
- Limitations: The method can lose tracks when moving objects dominate the video and can struggle when object and camera motion are colinear.
A.4. Additional Details
MegaSaM uses a staged architecture and training procedure combining flow, confidence, movement prediction, and differentiable bundle-adjustment supervision. Training uses synthetic static and dynamic data, while inference normalizes disparity and focal length within bundle adjustment.
- Training losses: Training combines pose and optical-flow losses, with flow supervision applied to adjacent frame pairs using the average L2 distance between predicted and ground-truth induced flow fields.
- Architecture: The predictor separates flow and confidence estimation in F from object movement-map prediction in Fm, training F first and then fine-tuning Fm with F fixed.The architecture uses two sequential stages: ego-motion pretraining followed by dynamic fine-tuning.
- Architecture: Feature and context encoders extract low-resolution features from the input video frames.
- Training losses: Pose and flow losses are applied at every bundle-adjustment iteration with exponentially increasing weight using γ = 0.9^k.Here, k denotes the iteration index.
- Training and inference: The two-stage training first uses 163 TartanAir scenes and 5K static Kubric videos, then fine-tunes Fm on 11K dynamic Kubric videos with 7-frame sequences.Training sequences are dynamically generated from a frame-distance matrix based on average ego-motion-induced flow magnitude.
- Training and inference: During camera tracking, video disparity is normalized so its 98th percentile equals 2, and focal length is divided by input image resolution at each bundle-adjustment stage.
B. Limitations
MegaSaM can fail in extremely challenging dynamic-video scenarios, including cases where moving objects dominate the image or camera and object motions are colinear.
- Camera tracking can fail when moving objects dominate the image or when the scene lacks anything reliable for the system to track.
- The approach struggles when camera motion and object motion are colinear.