PaperlayerView exact revision on arXiv

Source-linked AI summary

MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors

Riku Murai, Eric Dexheimer, Andrew J. Davison

arXiv:2412.12392v2cs.CVcs.RO

TL;DR

The paper addresses the lack of plug-and-play monocular SLAM that provides accurate poses and consistent dense maps without requiring a fixed camera model. It builds a real-time system around MASt3R two-view reconstruction priors with efficient matching, tracking, fusion, loop closure, and global optimisation. The resulting system handles in-the-wild video, achieves state-of-the-art performance with known calibration, and operates at 15 FPS.

  • Problem

    Minimal single-camera in-the-wild SLAM lacks a plug-and-play method that jointly provides accurate poses and consistent dense maps.

  • Method

    The system uses MASt3R two-view 3D reconstruction priors as a foundation for tracking, mapping, relocalisation, pointmap processing, loop closure, and global optimisation.

  • Results

    The system achieves globally consistent poses and dense geometry in real time at 15 FPS, with state-of-the-art performance across benchmarks when calibration is known.

  • Takeaways & Limitations

    A plug-and-play monocular SLAM system can use an off-the-shelf geometric prior to combine comparable pose estimation with consistent dense geometry.

  • Takeaways & Limitations

    Geometry predictions degrade with increasing distortion because MASt3R was trained only on pinhole images, and full-resolution decoding bottlenecks low-latency operation.

Abstract

from arXiv · show

We present a real-time monocular dense SLAM system designed bottom-up from MASt3R, a two-view 3D reconstruction and matching prior. Equipped with this strong prior, our system is robust on in-the-wild video sequences despite making no assumption on a fixed or parametric camera model beyond a unique camera centre. We introduce efficient methods for pointmap matching, camera tracking and local fusion, graph construction and loop closure, and second-order global optimisation. With known calibration, a simple modification to the system achieves state-of-the-art performance across various benchmarks. Altogether, we propose a plug-and-play monocular SLAM system capable of producing globally-consistent poses and dense geometry while operating at 15 FPS.

1. Introduction

Dense monocular SLAM remains difficult in minimal-camera, in-the-wild settings because poses, camera models, and consistent 3D geometry must be inferred together. The paper builds a real-time system around two-view 3D reconstruction priors to address these challenges without assuming a parametric camera model.

  • Motivation: Minimal single-camera SLAM currently lacks a plug-and-play method that jointly provides accurate poses and consistent dense maps in the wild.The challenge is especially important for robotics and augmented reality, where visual SLAM is foundational.
  • Motivation: Dense SLAM must reason jointly over time-varying poses, camera models, and 3D scene geometry from 2D images.Single-view geometric priors are ambiguous across views, while optical flow entangles motion and geometry.
  • Two-view priors: Two-view reconstruction priors output pointmaps in a common coordinate frame, implicitly addressing correspondence, poses, camera models, and dense geometry together.DUSt3R and MASt3R established this prior-based formulation using curated 3D datasets.
  • Contribution: The proposed SLAM framework uses MASt3R for tracking, mapping, and relocalisation while introducing efficient matching, map maintenance, and global optimisation for incremental real-time data.The system applies local pointmap filtering in the frontend and supports large-scale optimisation in the backend.
  • Contribution: The system assumes only a unique camera centre and handles generic, time-varying camera models while producing dense reconstructions in real time.This addresses the calibration assumptions common in prior SLAM systems.

3. Method

MASt3R-SLAM combines MASt3R pointmaps and features with camera-model-independent matching, robust tracking, incremental fusion, loop closure, and global optimisation. Its design uses generic central-camera rays and explicit correspondences to maintain dense, globally consistent geometry.

  • 3.1. Preliminaries: MASt3R predicts pointmaps and per-pixel features from image pairs, providing a common-coordinate-frame foundation for matching and SLAM.The system uses the network output directly through a forward pass that yields these representations.
  • 3.1. Preliminaries: Sim(3) poses optimise over independently scaled pointmap predictions, using Lie-algebra updates for pose refinement.The representation includes rotation, translation, and scale.
  • 3.1. Preliminaries: Each frame is treated as a generic central camera: pointmaps are normalised into unit rays that define the camera model without requiring parametric projection.The only camera-model assumption is that all rays pass through a unique camera centre.
  • 3.2. Pointmap Matching: Iterative projective matching minimises angular ray error to obtain pixel correspondences, then refines them by maximising feature similarity in local patches.Analytical Jacobians and Levenberg-Marquardt solve the per-point nonlinear least-squares updates, typically within 10 iterations for valid pixels.
  • 3.4. Loop Closure and 3.5. Backend Optimisation: The backend adds loop-closure edges from retrieved and MASt3R-validated keyframe matches, then applies efficient second-order optimisation for global pose and dense-geometry consistency.The system diagram connects keyframe insertion, retrieval, candidate decoding, edge construction, and global optimisation.
  • 3.2. Pointmap Matching: Custom CUDA kernels parallelise matching, reducing tracking to 2 milliseconds while keeping correspondences independent of pose estimates.The same implementation constructs newly added graph edges in a few milliseconds without initial projection estimates.
  • 3.3. Tracking and Pointmap Fusion: Tracking estimates the current-to-keyframe transformation from matched pointmaps, uses confidence-weighted robust residuals, and fuses transformed geometry incrementally.Ray-based angular errors reduce sensitivity to incorrect depths, while a small distance term prevents pure-rotation degeneracy; weighted averaging merges information across viewpoints.

4. Results

The system achieves strong monocular pose and dense-geometry results across several benchmarks, including uncalibrated settings, while maintaining real-time operation. Component analyses show that efficient matching, fusion, error formulations, and loop closure materially support the system’s performance.

  • Evaluation setup: 15 FPS operation is achieved on a desktop with an RTX 4090 while processing full-resolution MASt3R outputs resized to 512 pixels.Datasets are subsampled every two frames to simulate real-time performance.
  • 4.1. Camera Pose Estimation: State-of-the-art calibrated trajectory error is demonstrated on TUM RGB-D, while the uncalibrated system significantly outperforms DROID-SLAM* without assuming a fixed camera model.The uncalibrated results are also comparable to calibrated DPV-SLAM results.
  • 4.1. Camera Pose Estimation: The calibrated system outperforms NICER-SLAM and DROID-SLAM on 7-Scenes, while the real-time uncalibrated system outperforms offline NICER-SLAM.NICER-SLAM uses multiple depth, normal, and optical-flow priors, whereas this system uses a single 3D reconstruction prior.
  • 4.1. Camera Pose Estimation: On ETH3D-SLAM, the method achieves the best ATE and AUC because its robustness extends across more difficult trajectories, despite some methods having more precise trajectories.The evaluation uses train sequences and ATE curves because the official monocular thresholds are too strict.
  • 4.1. Camera Pose Estimation: 0.041m ATE is reported on EuRoC despite DROID-SLAM achieving lower trajectory error, with distortion handled by image undistortion in the uncalibrated case.MASt3R was not yet trained on the significant distortion present in the EuRoC camera model.
  • 4.2. Dense Geometry Evaluation: On 7-Scenes, the uncalibrated method achieves the best Accuracy and Chamfer distance, while on EuRoC both calibrated and uncalibrated variants obtain better geometry than DROID-SLAM.DROID-SLAM has higher completion on EuRoC but significantly worse accuracy, partly because it estimates many noisy points around the reference cloud.
  • 4.4. Component Analysis: Parallelised projective matching with feature refinement takes 2ms instead of 2 seconds for all-pixel MASt3R matching, making the system nearly 40x faster.Weighted fusion gives the lowest uncalibrated ATE and improves EuRoC ATE by 1.3cm; ray-error formulations also improve uncalibrated tracking and backend optimisation.
  • 4.4. Component Analysis: Loop closure improves both pose and geometry accuracy, with larger gains on longer sequences.The component analysis also evaluates canonical pointmap updates and point-versus-ray error formulations.

5. Limitations and Future Work

The system leaves all geometry unrefined in the full global optimisation and its MASt3R prior degrades under increasing image distortion. Future work targets globally consistent pointmaps and broader camera-model training.

  • The full global optimisation does not currently refine all geometry, although frontend pointmap filtering estimates accurate geometry.Future work seeks globally consistent 3D pointmaps while preserving MASt3R prediction coherence in real time.
  • MASt3R geometry predictions degrade with increasing distortion because it was trained only on pinhole images.The framework itself does not assume a parametric camera model, but the current prior remains limited by its training data.

6. Conclusion

The paper presents a real-time dense SLAM system built around an off-the-shelf MASt3R geometric prior for in-the-wild videos. It achieves comparable pose estimation to DROID-SLAM while also providing consistent dense geometry.

  • The MASt3R-based system handles in-the-wild videos and achieves state-of-the-art performance.
  • The system takes a different approach from DROID-SLAM by using an off-the-shelf geometric prior instead of an end-to-end flow-update framework.The paper reports comparable pose estimation while also providing consistent dense geometry.

8. Analytical Jacobians

The analytical Jacobians support second-order optimisation for tracking and backend point alignment. They cover point, ray, distance, and calibrated pixel errors, with pose updates transformed to the world frame.

  • Analytical Jacobians are derived for second-order optimisation in both tracking and the backend.
  • Point alignment: Point alignment defines a residual between a measured point and a transformed matched point, then differentiates it with respect to relative-pose Lie algebra perturbations.The derivation uses the skew-symmetric matrix [x]×.
  • Rays and distance: Ray residuals minimise error in normalised space, equivalent to minimising the angle between rays in the camera frame.Their Jacobians combine point normalisation with the pose transformation through the chain rule.
  • Rays and distance: A distance-based error prevents pure rotation from producing a degenerate optimisation problem.
  • Known calibration: With known calibration, the system uses pixel error because it better models pixel-level correspondence noise and is standard in bundle adjustment.The calibrated formulation uses a pinhole camera model, while future parametric camera models could supply corresponding Jacobians.
  • Runtime: The network encoder and decoder account for an average of 64% of total runtime, while datasets with more loop closures spend more time in the backend.Table 8 reports average component runtimes in milliseconds for the single-threaded system.
  • World-frame updates: Relative-pose Jacobians are converted into world-frame camera-pose updates using group-inverse and composition identities.

9. Initialisation

Initialisation reuses the last keyframe’s pointmap estimate to reduce network passes, except at startup. The system seeds startup with monocular MASt3R prediction and refines it using a running weighted average filter.

  • Tracking reuses the last keyframe’s pointmap estimate to minimise network passes, except during initialisation.
  • At startup, the same image is fed into MASt3R for monocular pointmap prediction, which is then refined with a running weighted average filter.Although monocular predictions are often inaccurate, the pointmap incorporates multiview information.

10. Runtime Breakdown

The system achieves real-time performance through efficient matching, tracking, and backend optimisation, while network encoding and decoding remain the main runtime bottleneck.

  • Tracking operates at greater than 20 FPS for every frame, while keyframing runs less frequently because it depends on motion.The runtime breakdown covers three representative datasets.
  • Network encoding and decoding consume around 64% of total runtime in both tracking and backend processing.
  • Backend runtime increases on TUM fr1/room and EuRoC MH01 because these sequences contain many detected loop closures.By contrast, 7-Scenes chess follows a more linear trajectory.
  • Efficient matching, tracking, and backend optimisation enable real-time operation, with the network limiting lower-latency SLAM.The modular prior and principled backend optimisation achieve global consistency in real time.

11. Evaluation Setup

Evaluation uses fixed system parameters across datasets and compares trajectory and dense reconstruction quality, including RMSE Chamfer to expose geometric outliers and incoherence.

  • Evaluation Parameters: All datasets use keyframe threshold ωk = 0.333, loop-closure threshold ωl = 0.1, and ωr = 0.005.Relocalisation uses a match fraction above 0.3, except ETH3D, where the threshold is 0.5.
  • Trajectory Evaluation: Trajectory evaluation runs DROID-SLAM with open-source code and dataset-specific configurations, using the TUM configuration for 7-Scenes.Remaining TUM and EuRoC entries come from reported tables.
  • Geometry Evaluation: RMSE Chamfer penalises outlying measurements and better reflects geometry quality than mean Chamfer when points are inconsistent.Figure 9 compares red ground-truth and blue estimated point clouds on 7-Scenes heads.
  • Geometry Evaluation: MASt3R-SLAM produces more coherent and accurate geometry than DROID-SLAM despite similar mean Chamfer distance, as reflected in RMSE Chamfer.
  • Qualitative Reconstruction: EuRoC reconstruction shows Spann3R failing on a non-object-centric sequence, while DROID-SLAM produces more outliers than MASt3R-SLAM.The keyframing system retains viewed scene parts, and global optimisation creates globally consistent maps in real time.

12. EuRoC Results

On EuRoC, MASt3R-SLAM improves trajectory estimation over earlier geometric-prior systems, although it does not outperform DROID-SLAM and related matching-based methods.

  • MASt3R-SLAM does not outperform DROID-SLAM or methods that leverage DROID-SLAM’s matching architecture on EuRoC.EuRoC remains challenging because of aggressive motion, large-scale trajectories, and varying exposure.
  • MASt3R-SLAM demonstrates significant trajectory-estimation improvements over DeepV2D and DeepFactors.
  • Dense geometry is more accurate and consistent for MASt3R-SLAM, including its uncalibrated system, as shown in the main-paper results and Tab. 3.

13. Comparison to Other SLAM/SfM Methods

Compared with matching-based end-to-end SLAM and sparse SfM approaches, MASt3R-SLAM uses a modular geometric prior and introduces optimisation and camera-model choices aimed at coherent geometry and generality.

  • Comparison to SLAM Methods: MASt3R-SLAM uses a modular two-view geometric prior, whereas DROID and DPV SLAM learn matching priors within end-to-end differentiable bundle-adjustment systems.The comparison identifies limitations in geometry and generality for those systems.
  • Geometry: Local fusion and scale optimisation are sufficient for consistent, coherent geometry, while DROID’s bundle adjustment can lose coherence under low parallax.Bundle adjustment lacks smoothness regularisation and constraints that guarantee coherent geometry even with accurate poses.
  • Camera Generality: Geometric priors enable continuously changing intrinsics, unlike DROID’s pinhole-fixed training model.
  • Matching: Dense projective pointmap matching achieves a 1000x speedup without compromising accuracy relative to MASt3R-SfM’s brute-force sparse correspondence approach.MASt3R-SfM subsamples 1/64 pixels because of brute-force matching.
  • Global Optimisation: The second-order optimiser uses minimal pose updates and gauge fixing to avoid degenerate solutions associated with first-order global optimisation.The uncalibrated ray formulation preserves accuracy while avoiding selection of a specific camera model.
Loading 2412.12392v2…