Source-linked AI summary
M^3: Dense Matching Meets Multi-View Foundation Models for Monocular Gaussian Splatting SLAM
Kerui Ren, Guanghao Li, Changjian Jiang, Yingxiang Xu, Tao Lu, Linning Xu, Junting Dong, Jiangmiao Pang, Mulin Yu, Bo Dai
TL;DR
Streaming reconstruction from uncalibrated monocular video needs precise poses and efficient online refinement, but feed-forward foundation models may lack correspondences suitable for geometric optimization. M3 adds dense matching to a multi-view foundation model and integrates it with monocular Gaussian Splatting SLAM, achieving state-of-the-art pose estimation and reconstruction across diverse benchmarks. Its scope remains limited by reliance on feed-forward predictions and monocular sensing.
Problem
Streaming monocular reconstruction requires precise pose estimation and efficient online refinement, while existing foundation models may lack pixel-level correspondences for rigorous geometric optimization.
Method
M3 augments Pi3X with a dedicated dense matching head and integrates it into SLAM with single-pass joint tracking and global optimization, dynamic-region suppression, and intrinsic alignment.
Results
M3 achieves state-of-the-art accuracy in pose estimation and 3D reconstruction across diverse indoor and outdoor benchmarks while maintaining competitive efficiency on long-duration monocular video.
Takeaways & Limitations
Dense correspondences, unified inference, dynamic-region suppression, and intrinsic alignment support stable tracking and high-fidelity 3DGS reconstruction in long video streams.
Takeaways & Limitations
M3 relies on correct feed-forward predictions, lacks a dedicated fallback mechanism for severely inaccurate priors, and operates only with monocular visual input.
Abstract
from arXiv · showhide
Streaming reconstruction from uncalibrated monocular video remains challenging, as it requires both high-precision pose estimation and computationally efficient online refinement in dynamic environments. While coupling 3D foundation models with SLAM frameworks is a promising paradigm, a critical bottleneck persists: most multi-view foundation models estimate poses in a feed-forward manner, yielding pixel-level correspondences that lack the requisite precision for rigorous geometric optimization. To address this, we present M^3, which augments the Multi-view foundation model with a dedicated Matching head to facilitate fine-grained dense correspondences and integrates it into a robust Monocular Gaussian Splatting SLAM. M^3 further enhances tracking stability by incorporating dynamic area suppression and cross-inference intrinsic alignment. Extensive experiments on diverse indoor and outdoor benchmarks demonstrate state-of-the-art accuracy in both pose estimation and scene reconstruction. Notably, M^3 reduces ATE RMSE by 64.3% compared to VGGT-SLAM 2.0 and outperforms ARTDECO by 2.11 dB in PSNR on the ScanNet++ dataset.
1 Introduction
M3 addresses the precision and efficiency gaps in streaming monocular reconstruction by adding dense matching to a multi-view foundation model and tightly integrating it with SLAM. It combines joint tracking, global optimization, dynamic-region suppression, and efficient inference to achieve strong pose and reconstruction accuracy.
- Motivation: Existing foundation-model approaches are often batch-oriented, while streaming methods face low resolution, cumulative drift, or redundant computation.These limitations hinder real-time feedback, scalability, and stable global refinement in open-ended environments.
- Motivation: Fine-grained pixel-to-pixel correspondences are needed to provide the epipolar constraints required for reliable Bundle Adjustment.Without precise cross-frame associations, complex sequences can produce ghosting artifacts or trajectory divergence.
- Approach: M3 adds a dedicated dense matching head that recovers pixel-level correspondences for high-frequency pose refinement.The matching head uses pixel-level descriptors to support refined cross-frame dense matching and geometric optimization.
- Approach: M3 tightly couples frontend tracking and backend global optimization through a single feed-forward inference over historical keyframes and incoming frames.This design reduces redundant model invocations while simultaneously supporting geometry updates and tracking.
- Results: M3 introduces dynamic-region identification and suppression to stabilize reconstruction in real-world environments with transient objects.Experiments across diverse indoor and outdoor benchmarks report state-of-the-art pose estimation and 3D reconstruction with competitive long-duration efficiency.
2 Related work
Related work spans classical and learned SLAM, neural and Gaussian scene representations, and streaming reconstruction. Recent systems increasingly combine foundation-model priors with online reconstruction, but efficiency, surface coherence, and geometric consistency remain active concerns.
- Visual SLAM: Learned SLAM evolved from feature-based pipelines toward systems combining recurrent pose refinement with differentiable Bundle Adjustment.DROID-SLAM is described as a benchmark integrating a GRU with differentiable BA for iterative camera-pose and disparity refinement.
- Scene Reconstruction: Scene reconstruction progressed from SfM and MVS to NeRF and 3D Gaussian Splatting, with Gaussian methods improving rendering efficiency and fidelity.Subsequent work adds 2D Gaussians, depth and normal consistency, and temporal extensions such as 4D Gaussian Splatting.
- Streaming Reconstruction: Streaming reconstruction incrementally builds 3D models from sequential sensor streams while jointly addressing online pose estimation and mapping.GS-SLAM uses adaptive Gaussian expansion and coarse-to-fine tracking, while Onthefly-NVS introduces pixel spawning and a sliding-window strategy.
- Foundation-Model Integration: Foundation-model-based systems improve streaming reconstruction but may incur repeated inference, lack explicit pixel correspondences, or compromise surface coherence.ARTDECO combines MASt3R predictions with level-of-detail representation, while PLANING uses a hybrid triangle-Gaussian representation.
3 Method
M3 couples an enhanced multi-view foundation model with a unified SLAM pipeline for streaming monocular reconstruction, using dense matching to improve geometric optimization and dynamic suppression for robustness.
- M3 Framework: M3 jointly estimates camera intrinsics, camera poses, and a static 3D scene from uncalibrated monocular video.The framework integrates these estimates in a unified frontend–backend SLAM pipeline.
- Dense Matching through Foundation Model: A dedicated matching head augments Pi3X with dense descriptors and confidence maps for pixel-level cross-frame correspondence.The matching head is trained while preserving the foundation model’s existing geometric components and metric scale.
- Dense Matching through Foundation Model: The matching objective combines a symmetric bidirectional InfoNCE loss with confidence regularization over correspondence pairs.The loss uses descriptor similarity between pixel pairs and a balancing hyperparameter for confidence regularization.
- Dense Matching through Foundation Model: Pose-guided local refinement reduces dense matching from quadratic global search to linear local refinement while preserving high matching accuracy.Correspondence candidates are restricted to a neighborhood around the pose-projected location and selected by descriptor similarity.
- M3 Framework: Dynamic-region suppression down-weights pixels with low descriptor consistency during optimization, reducing trajectory drift and reconstruction artifacts.Descriptor discrepancies identify dynamic objects or occlusions, while reference-frame motion maps maintain temporal consistency.
- M3 Framework: M3 tightly couples frontend tracking and backend optimization through multi-view inference over historical keyframes and incoming frames.A sliding window retains recent and retrieved relevant keyframes for streaming processing.
4 Experiments
M3 is evaluated across diverse indoor and outdoor benchmarks for pose accuracy, rendering quality, and efficiency. Results show strong pose estimation, high-quality reconstruction, dynamic-scene consistency, and benefits from the proposed design choices.
- Experimental Setup: The evaluation covers 38 indoor scenes and 22 outdoor sequences from ScanNet++, ScanNetV2, VR-NeRF, KITTI, Waymo, and FAST-LIVO2.Pose accuracy, rendering quality, and efficiency are measured using ATE RMSE, PSNR/SSIM/LPIPS, Gaussian count, and training time.
- Pose Estimation Results: M3 achieves the lowest ATE on most evaluated sequences against recent SLAM frameworks and the Pi3X base model.The authors attribute this to combining foundation-model geometric priors with SLAM-based geometric optimization.
- Reconstruction Results: M3 delivers strong novel-view synthesis with a compact Gaussian representation and comparable training time.Qualitative comparisons show sharper renderings, fewer artifacts, and better preservation of structural details.
- Reconstruction Results: M3 outperforms feed-forward Gaussian Splatting baselines on indoor and outdoor sets while requiring less memory and remaining stable in large-scale scenes.The comparison follows the AnySplat protocol on sampled ScanNet++ and Waymo images.
- Dynamic Environments: The Motion Map suppresses dynamic regions, excluding moving objects from static reconstruction and improving temporal and structural consistency.Without it, dynamic objects can produce blurry ghosting artifacts that degrade rendering quality and geometric accuracy.
- Ablation Study: Ablations show that fine-grained matching, intrinsic alignment, global optimization, the Pi3X backbone, and integrated inference each contribute to performance or efficiency.A sliding window of L = 8 balances temporal stability and computational cost, while decoupling frontend and backend increases redundant inference and training time.
5 Limitations
M3 remains dependent on the correctness of feed-forward foundation-model predictions and currently uses only monocular visual input.
- Scope and Robustness: M3 may fail to recover when the foundation model produces severely inaccurate correspondences or geometric priors because it lacks a dedicated fallback mechanism.The framework also does not exploit complementary sensing modalities such as LiDAR or inertial measurements.
6 Conclusion
M3 combines pixel-level dense matching with a multi-view foundation model inside a SLAM pipeline for efficient monocular streaming reconstruction. The framework supports dense pose optimization, reduced redundant inference, stable tracking, and high-fidelity 3DGS reconstruction, with strong benchmark performance and competitive efficiency.
- Framework: M3 enhances Pi3X with pixel-level dense matching and tightly integrates it into a SLAM pipeline for uncalibrated monocular video.The design targets consistent dense correspondences for pose optimization and stable long-video reconstruction.
- Framework: The framework reduces redundant model inferences while combining stable tracking with high-fidelity 3DGS reconstruction.Descriptor-based dynamic-region suppression and intrinsic alignment address real-world motion and drift.
- Conclusion: M3 achieves state-of-the-art pose estimation and reconstruction quality with competitive efficiency across diverse indoor and outdoor benchmarks.The conclusion characterizes the framework as an efficient and robust approach to streaming reconstruction.
7 Supplementary Material
The supplementary material details M3’s Pi3X-based architecture, Gaussian reconstruction, streaming management, and foundation-model ablations, with Pi3X achieving the highest ScanNet++ accuracy.
- Architecture: M3 builds on Pi3X, a permutation-equivariant feed-forward model that jointly processes uncalibrated images and predicts geometric priors.Its architecture combines a feature encoder, multi-view interaction module, and task-specific prediction heads.
- Streaming and loop closure: M3 streams video with a sliding window containing historical keyframes and incoming frames, triggering Pi3X-aided loop closure for temporally distant retrievals.The supplementary settings use a window of L = 8 frames and up to four historical keyframes.
- Neural Gaussian reconstruction: Gaussian reconstruction uses adaptive insertion, distance-aware visibility, hierarchical levels of detail, and confidence-weighted initialization to prioritize detailed regions efficiently.The method uses a Laplacian-of-Gaussian probability map, dmax-based opacity fading, and α = 0.2 · C(u, v).
- Foundation-model ablations: On ScanNet++, Pi3X and DA3 outperform VGGT, while Pi3X provides higher accuracy than DA3 because of stronger cross-inference consistency.DA3’s weaker temporal-geometric consistency yields marginally higher ATE and lower reconstruction fidelity than the Pi3X implementation.