Source-linked AI summary
MASt3R-SfM: a Fully-Integrated Solution for Unconstrained Structure-from-Motion
Bardienus Duisterhof, Lojze Zust, Philippe Weinzaepfel, Vincent Leroy, Yohann Cabon, Jerome Revaud
TL;DR
SfM remains difficult because traditional pipelines are brittle under limited overlap, insufficient motion, and challenging image collections. MASt3R-SfM builds on MASt3R to combine local 3D reconstruction, retrieval, and global alignment in a scalable pipeline, reporting strong performance across diverse conditions including small collections and purely rotational settings.
Problem
Traditional SfM decomposes a highly non-convex problem into multiple stages and remains brittle when images have insufficient overlap or motion.
Method
MASt3R-SfM uses MASt3R local reconstructions and matches, frozen-encoder image retrieval, and successive 3D-matching and 2D-reprojection optimization stages.
Results
MASt3R-SfM achieves state-of-the-art performance across diverse conditions, with nearly constant performance even for very few views and operation without camera motion.
Takeaways & Limitations
The resulting pipeline handles unconstrained image collections, including very small collections and purely rotational cases, without relying on RANSAC.
Takeaways & Limitations
The method assumes a pinhole camera model without lens distortion, although the authors state it could be extended to other camera types.
Abstract
from arXiv · showhide
Structure-from-Motion (SfM), a task aiming at jointly recovering camera poses and 3D geometry of a scene given a set of images, remains a hard problem with still many open challenges despite decades of significant progress. The traditional solution for SfM consists of a complex pipeline of minimal solvers which tends to propagate errors and fails when images do not sufficiently overlap, have too little motion, etc. Recent methods have attempted to revisit this paradigm, but we empirically show that they fall short of fixing these core issues. In this paper, we propose instead to build upon a recently released foundation model for 3D vision that can robustly produce local 3D reconstructions and accurate matches. We introduce a low-memory approach to accurately align these local reconstructions in a global coordinate system. We further show that such foundation models can serve as efficient image retrievers without any overhead, reducing the overall complexity from quadratic to linear. Overall, our novel SfM pipeline is simple, scalable, fast and truly unconstrained, i.e. it can handle any collection of images, ordered or not. Extensive experiments on multiple benchmarks show that our method provides steady performance across diverse settings, especially outperforming existing methods in small- and medium-scale settings.
1. Introduction
SfM jointly estimates scene geometry and camera parameters but remains brittle because traditional pipelines decompose a difficult non-convex problem into error-prone stages. MASt3R-SfM instead integrates robust local reconstruction, retrieval, and global optimization to handle unconstrained image collections.
- SfM estimates a scene’s 3D geometry and observing-camera parameters from images, supporting applications including navigation, multi-view stereo, and novel-view synthesis.
- Traditional SfM decomposes a highly non-convex objective into matching, relative-pose estimation, triangulation, and bundle adjustment, with outliers requiring repeated hypothesis verification.
- Traditional pipelines remain brittle when images have insufficient overlap or viewpoints provide too little translation.
- MASt3R-SfM uses MASt3R local reconstruction and matching, frozen-encoder retrieval, and two gradient-descent stages to build a quasi-linear, training-free pipeline without RANSAC.
- MASt3R-SfM handles image collections from a single view to large-scale scenes, including purely rotational settings without camera motion.
- The authors report state-of-the-art performance across diverse conditions, while competitors show failures in small-scale settings and varying-view experiments.
2. Related Works
Related work attempts to simplify SfM through learned components, test-time regressors, or faster retrieval, but important scalability and unconstrained-viewpoint limitations remain. These approaches differ in how much of the traditional pipeline they replace.
- Traditional SfM: Traditional SfM centers on matching correspondences and bundle adjustment, which jointly optimizes 3D points and camera parameters by minimizing reprojection errors.
- SfM revisited: FlowMap and Ace-Zero regress scene representations at test time, but their scaling and reliance on constrained, continuous viewpoints limit unordered collections with large viewpoint or illumination disparities.
- SfM revisited: VGGSfM, detector-free SfM, and DUSt3R retain frozen learned components, while VGGSfM still relies on handcrafted initialization and triangulation.
- Image Retrieval for SfM: Pairwise matching has quadratic complexity, motivating coarse-to-fine retrieval strategies that first identify likely image pairs before costly matching.
- Image Retrieval for SfM: This paper uses MASt3R encoder tokens as local features for ASMK retrieval instead of an external off-the-shelf retrieval module.
3. Preliminaries
MASt3R is a pairwise foundation model that jointly predicts local 3D structure and pixelwise matches. Its encoder–decoder outputs provide the geometric and correspondence information used for global reconstruction.
- Given two images, MASt3R performs joint local 3D reconstruction and pixelwise matching in a single forward pass.
- MASt3R can be represented as an encoder–decoder function whose Siamese encoder produces feature maps and twin decoders regress pointmaps and local features.
- The predicted pointmaps contain information from which camera intrinsics and metric depthmaps can be recovered.
- Fast nearest-neighbor matching on the regressed feature maps yields reciprocal sparse correspondences between image pairs.
4. Proposed Method
MASt3R-SfM reconstructs unconstrained image collections by sparsifying pair selection, producing pairwise local reconstructions, and globally aligning them under camera constraints. The pipeline combines efficient retrieval, MASt3R pairwise inference, gradient-based alignment, and anchor-based refinement.
- Scene graph: MASt3R-SfM represents an unordered image collection as a connected sparse scene graph whose edges join likely-overlapping images.The graph must remain connected so all images are linked directly or indirectly.
- Scene graph: O(N) graph edges replace naive O(N^2) pairwise processing by connecting keyframes and remaining images to their closest keyframe and k nearest neighbors.The method typically uses N_a=20 keyframes and k=10 neighbors, while treating retrieval as quasi-linear in practice.
- Local reconstruction: MASt3R computes local pointmaps and sparse pixel matches for every selected graph edge, reusing cached encoder features and running the decoder.Because MASt3R is order-dependent, both image orders are evaluated and their correspondences are united.
- Local reconstruction: Canonical pointmaps, depthmaps, and intrinsics are aggregated from multiple edge-specific estimates, assuming a pinhole camera model without lens distortion.Per-pixel confidence-weighted aggregation reduces regression imprecision before global optimization.
- Coarse alignment: Global alignment optimizes scaled rigid transformations so matching 3D points across canonical pointmaps become close, using sparse pixel correspondences and reduced memory.The constrained pointmap is constructed from camera intrinsics, poses, scale factors, and depthmaps so it follows the pinhole model by design.
- Refinement: Anchor-based refinement ties nearby pixels to shared anchor depths, reducing depth variables while preserving locally accurate relative depth offsets.With spacing δ, the optimized anchor set has W/δ × H/δ values instead of W × H pixel depths; δ=8 gives a 64-fold reduction.
5. Experimental Results
Experiments evaluate MASt3R-SfM across diverse datasets, input conditions, baselines, and design choices. The method maintains robust performance with sparse, unordered, or rotational inputs, while ablations identify retrieval and optimization choices that support its results.
- Evaluation setup: Experiments span video and unordered image collections, indoor, outdoor, object-centric, and varied scene-complexity settings.The evaluation uses Tanks&Temples, ETH3D, CO3Dv2, and other benchmark protocols with ground-truth or pseudo-ground-truth camera poses.
- Comparison with the state of the art: MASt3R-SfM provides nearly constant performance across regularly subsampled Tanks&Temples inputs, while several baselines degrade or fail to converge with fewer views.FlowMap and VGGSfM also crash on large collections because of insufficient memory despite using 80GB GPUs.
- Comparison with the state of the art: On multi-view pose estimation, MASt3R-SfM compares favorably with existing methods, particularly when the number of input images is low.The comparison uses randomly sampled images from CO3Dv2 and RealEstate10K under the PoseDiffusion protocol.
- Comparison with the state of the art: On ETH3D’s unordered photograph collections, MASt3R-SfM outperforms competing approaches by a large margin on average.COLMAP and VGGSfM also perform relatively poorly, indicating sensitivity to collections without highly overlapping images.
- Ablations: Combining short-range k-NN and long-range keyframe connections is important for reaching top performance, whereas a complete graph is about 10x slower and not scalable in general.The scene-graph ablation compares complete, local-window, random, k-NN, and keyframe-only constructions.
- Ablations: Frozen MASt3R features perform on par with FIRe for retrieval-assisted visual localization, while coarse optimization needs about 250 iterations before refinement consistently improves poses.Shared intrinsics have only a small performance impact, and MASt3R reaches competitive localization results, including a new state of the art for InLoc.
6. Conclusion
MASt3R-SfM is a simpler, fully integrated approach to unconstrained SfM that handles very small collections and even cases without motion.
- MASt3R-SfM handles very small image collections and cases without camera motion.It does not rely on RANSAC, unlike standard triangulation-based SfM.
A. Qualitative Results
The pipeline produces relatively dense reconstructions from sparse matches by inferring a 3D point for every pixel. At its 512-pixel image limit, it typically generates about 200K 3D points per image.
- The pipeline produces relatively dense outputs despite using only sparse matches.The inverse reprojection function infers 3D points beyond the pixels belonging to sparse matches.
- About 200K 3D points per image are typically produced when images are downscaled to 512 pixels in their largest dimension.
B. Other retrieval variants based on MASt3R features
The retrieval ablations compare token-feature ASMK with global representations and learned projection variants. Global average pooling with PCA whitening remains below token-feature ASMK.
- The ablations compare token-feature ASMK, global image representations, and ASMK after learning a projector on frozen MASt3R features.The global representation uses cosine similarity, while the projector follows strategies used by HOW and FIRe.
- Global average pooling performs slightly better than global max-pooling, and PCA whitening improves the global representation.
- Global representations remain less effective than ASMK applied to token features.
C. Robustness to pure rotations
Pure-rotation experiments examine settings where all cameras share an optical center, making traditional triangulation ill-defined. The observed failures arise from hard outlier matches between similar-looking structures.
- Pure-rotation experiments use perspective crops from InLoc panoramas with a shared optical center.The crops have a 60° field of view and are sampled at three pitch values every 30°.
- Failures are caused by hard false matches between similar-looking structures that can pass geometric verification.Even human observers may find it difficult to recognize that the images depict different scene parts.
D. Additional Results
Additional results show strong performance in sparse multi-view pose regression and qualitative reconstructions, while reviewed failures trace to wrong matches.
- Multi-view pose estimation: MASt3R-SfM largely outperforms competing methods on CO3Dv2 and RealEstate10K with 3, 5, and 10 input images.The comparison covers multi-view pose regression across both datasets and multiple numbers of randomly sampled frames.
- Qualitative reconstructions: The qualitative outputs on ETH-3D and Tanks&Temples are raw reconstructions produced without further refinement.The figure presents the direct outputs of the proposed pipeline rather than post-processed results.
- Failure analysis: All manually reviewed failure cases were caused by outlier matches between similar-looking parts of the same scene.Three wrong image pairs are shown for the purely rotational InLoc scene DUC1/007.
- Additional evaluations: The additional tables include pure-rotation accuracy on 20 randomly selected InLoc scenes and detailed per-scene Tanks&Temples results.The pure-rotation table reports RRA@5, while the Tanks&Temples appendix provides scene-level results.
E. Additional ablations
The sparse scene-graph ablation finds that increasing key images or nearest neighbors improves performance until gains saturate beyond moderate settings.
- Sparse scene graph: Increasing the number of key images N_a or nearest neighbors k generally improves performance, with saturation above N_a≥20 or k≥10.The study varies both hyperparameters for constructing the sparse scene graph.
F. Parametrizations of Cameras
The camera parametrization expresses poses through a kinematic tree, while tree construction affects optimization quality and balanced hierarchies perform best.
- Camera optimization: Camera optimization constructs 3D points through inverse reprojection from intrinsics, extrinsics, pixel coordinates, and depthmaps.The parametrization is motivated by the sensitivity of reconstructed points to small extrinsic changes, especially at greater depths.
- Kinematic-tree parametrization: A kinematic tree represents camera poses relative to one another, propagating all poses sequentially from a single root.The tree contains directed camera edges and conveys shared-motion structure between overlapping views.
- Tree construction: Hierarchical clustering using MASt3R correspondences performs best among tested tree-construction strategies and substantially improves over the other baselines.The comparison includes star and maximum-spanning-tree constructions.
- Tree construction: Balanced trees with approximately log2(N) levels are favored, whereas star trees have one level and maximum-spanning trees can have up to N/2 levels.The kinematic tree is distinct from the sparse scene graph despite sharing the same camera nodes.