Source-linked AI summary
BLASt3R: Bundle Adjustment of Any Image Set with Multi-View Matching and Monocular Priors
Vincent Leroy, Philippe Weinzaepfel, Lojze Zust, Yohann Cabon, Jérome Revaud
TL;DR
Dense correspondence estimation limits the scalability and usability of high-performing hybrid SfM systems, especially for online VSLAM. BLASt3R combines fast multi-view matching with monocular priors in a regularized BA framework shared by offline and online reconstruction. It reports favorable robustness–accuracy–speed trade-offs across both domains, including performance beyond calibrated VSLAM pipelines despite not using known camera intrinsics.
Problem
Dense correspondence estimation is prohibitively costly for scalable hybrid SfM, particularly under the time constraints of online VSLAM.
Method
BLASt3R uses fast multi-view matching and monocular depth priors for initialization and regularization in a unified BA that jointly optimizes cameras, tracks, and adjustable depthmaps.
Results
BLASt3R achieves a favorable robustness–accuracy–speed trade-off across diverse SfM and VSLAM benchmarks and exceeds calibrated VSLAM pipelines without known camera intrinsics.
Takeaways & Limitations
The same optimization pipeline and hyperparameters support both unordered-image reconstruction and long online sequences across heterogeneous scenarios.
Takeaways & Limitations
Traditional SfM remains brittle under low texture, wide baselines, and challenging photometric conditions, while VSLAM relies on temporal coherence and motion continuity assumptions that limit settings beyond controlled sequences.
Abstract
from arXiv · showhide
Recent hybrid Structure-from-Motion (SfM) systems combine the robustness of feed-forward 3D reconstruction with the accuracy of traditional bundle adjustment (BA) with pixel matching. They are usually the best performing methods however their scalability and usability remains limited since estimating dense correspondences between views is prohibitively costly, especially considering time constraints inherent to online applications like Visual SLAM (VSLAM). In this paper, we introduce a regularized BA framework that leverages a fast multi-view matcher and monocular priors for initialization and regularization. In contrast to existing systems, our unified approach seamlessly supports both online VSLAM and offline reconstruction from unordered image collections within the same optimization framework and sharing common hyperparameters for all tasks. Extensive experiments across both domains demonstrate improved performance and speed tradeoffs over traditional, feed-forward, and hybrid baselines. Notably for VSLAM, our uncalibrated method outperforms all previous calibrated approaches.
1. Introduction
BLASt3R addresses limitations of traditional, feed-forward, and hybrid reconstruction by combining fast multi-view matching, monocular priors, and unified bundle adjustment. The same framework supports offline unordered-image SfM and online VSLAM, with experiments showing a favorable robustness–accuracy–speed trade-off.
- Motivation: Traditional SfM is accurate and scalable on unordered image collections but can be brittle in difficult visual conditions and is not designed for real-time processing.VSLAM supports online processing through temporal coherence and motion continuity, but these assumptions limit use beyond controlled sequential settings.
- Motivation: Feed-forward 3D methods are fast and robust because they directly infer cameras and geometry, but they lack the accuracy of optimization-based SfM.
- Motivation: Hybrid systems combine learned priors with geometric optimization, but recent matching-based approaches are often dedicated to a specific reconstruction scenario.
- BLASt3R: BLASt3R couples a scalable multi-view matcher with monocular depth priors and a unified BA that jointly refines cameras, sparse tracks, and adjustable depth parameters.The depth priors provide initialization and regularization for the optimization.
- BLASt3R: BLASt3R supports offline reconstruction from unordered image collections and online VSLAM within one optimization pipeline using shared hyperparameters.
- Results: BLASt3R achieves a favorable robustness–accuracy–speed trade-off across SfM and VSLAM benchmarks, and its uncalibrated system exceeds calibrated VSLAM pipelines.
2. Related work
Related work spans classical geometric reconstruction, learned SfM and VSLAM, feed-forward pointmap regression, hybrid matching systems, and depth-regularized optimization. These approaches trade accuracy, robustness, scalability, and scenario coverage in different ways.
- Classical SfM and VSLAM: Classical SfM jointly optimizes 3D points and camera parameters through bundle adjustment after detecting, matching, verifying, registering, and triangulating features.Bundle adjustment minimizes a nonlinear multi-view reprojection objective but can get stuck when initialization is far from the optimum.
- Learned reconstruction: Learning-based methods have targeted individual SfM components, end-to-end pipelines, and direct regression of pointmaps linking image pixels to observed 3D points.
- Feed-forward and hybrid methods: Feed-forward and hybrid methods use global alignment, hybrid BA, or external SfM tools to improve scalability and accuracy over direct pointmap prediction.The cited approaches include DUSt3R, MASt3R-SfM, VGGT-Long, and VGGT.
- Feed-forward and hybrid methods: Matching-based approaches achieve strong performance but have generally been designed for one specific scenario rather than both offline and online reconstruction.
- Depth regularization: Monocular depth priors have been used to improve matching, relative pose estimation, and bundle-adjustment robustness for weak overlap or small parallax.Prior parameterizations include local constraints, splines, deformations, latent codes, covariance functions, and depthmap bases.
3. The BLASt3R Method
BLASt3R integrates streamable multi-view matching, adjustable monocular pointmaps, and unified bundle adjustment for dense reconstruction from sequential or unordered images.
- Unified reconstruction: The method estimates cameras, dense depth maps, and tracks for image sets that may come from one camera or different cameras.It uses a simple pinhole model with per-image focal length and jointly represents intrinsic and extrinsic camera parameters.
- Monocular priors and optimization: The unified pipeline combines dense tracks with monocular pointmaps that initialize and regularize bundle adjustment.The adjustable pointmap maps a small set of tunable parameters to a dense representation, whose parameters are optimized to fit observed correspondences.
- Multi-view matching: A streaming memory mechanism retains tokens from previous images, enabling the query image to attend to all earlier image patches.The coarse decoder updates query tokens using the memory bank and appends the resulting tokens for subsequent processing.
- Multi-view matching: Coarse matching exhaustively compares projected, normalized patch tokens and thresholds their similarities to select patch matches P.These selected pairs restrict the subsequent fine-matching computation.
- Multi-view matching: Fine matching converts selected patch tokens into pixel descriptors and computes block-sparse pixel similarities only within the selected patch pairs.Dense tracking is obtained from the highest-scoring pixel matches, with low-confidence matches assigned null.
4. Experiments
BLASt3R is evaluated in uncalibrated online and offline settings using shared pipelines and hyperparameters, showing strong pose and reconstruction performance with favorable scalability. Its results include state-of-the-art VSLAM accuracy, high ETH3D-MVS accuracy, and faster processing on long image sets.
- Evaluation setup: Experiments use the same hyperparameters in uncalibrated mode for online VSLAM and offline reconstruction, while allowing known calibration when available.The evaluation covers camera pose estimation and dense reconstruction across multiple benchmarks.
- Online VSLAM: BLASt3R significantly outperforms existing methods on TUM-RGBD, including calibrated systems, despite estimating poses without camera calibration.Its uncalibrated performance is reported as close to its calibrated performance.
- Online VSLAM: BLASt3R achieves state-of-the-art ETH3D-SLAM results on all reported sequences, with RMSE ATE below 2cm.
- Offline SfM: 98.5% average mAA@30 on ETH3D-MVS rises to 98.8% with COLMAP’s BA, outperforming the second-best approach.
- Offline SfM: BLASt3R outperforms existing methods across Tanks&Temples sample sizes and runs in 0.5/.75/1.2/2.1 minutes for 25/50/100/200 frames.VGGT-BA takes substantially longer and encounters out-of-memory limits beyond 200 frames; BLASt3R’s memory mechanism and linear feed-forward complexity support larger sets.
- Dense reconstruction: The method reaches state-of-the-art pointmap accuracy on 7-Scenes and outperforms π3 and VGGT on ETH3D.
- Ablations: Multiple adjustable depth components improve reconstruction accuracy by 30-40% over a single depth component, while the single component remains useful for initialization.
- Ablations: The formulation remains robust to degenerate online cases such as pure translation, static viewpoints, and pure rotations.
5. Conclusion
BLASt3R unifies online and offline SfM through feed-forward multiview matching, adjustable depth predictions, and regularized bundle adjustment. It achieves strong performance across heterogeneous scenarios with one pipeline and one hyperparameter set.
- BLASt3R jointly optimizes camera parameters and scene points using regularized BA over feed-forward multiview matches and adjustable depth predictions.
- The same processing pipeline and hyperparameters support unordered sparse views and long online sequences.
A. Additional details about the method
The unified pipeline uses retrieval and multiview matching for both online and offline inputs, then initializes and refines geometry differently according to whether frames arrive incrementally or are available together.
- The method uses the same retrieval, matching, and optimization pipelines with shared hyperparameters for online and offline scenarios.
- Retrieval: Retrieval combines top-K neighbors with farthest-point-sampled core views to balance local relevance and view diversity.The core set is intended to represent the scene while retaining nearest neighbors for SfM.
- Multi-view matching: Multiview matching proceeds causally for both settings, while offline images are processed in both directions from an empty memory.
- Optimization: Camera poses are initialized with Kabsch-Umeyama on maximum-spanning-tree edges, and online tracks are initialized from matching pixels in memory frames.
- Optimization: Online processing performs global BA for at most 10 steps every 32 frames, whereas offline processing jointly optimizes all parameters to convergence.
A.2. Implementation details
Implementation details address first-frame initialization and track growth in the shared matching pipeline. A null token resolves empty-memory attention, while hash-based pruning limits redundant tracks for efficiency.
- Matching initialization: The first image uses a learnable null token instead of empty memory, signaling sequence start and avoiding ill-defined cross-attention.
- Track subsampling: New images append additional tracks, but video and large image collections can produce millions of highly redundant tracks.
- Track subsampling: Tracks receive redundancy scores from hash-bin counts, and the most redundant tracks are pruned.
- Track subsampling: Experiments limit the total number of tracks to 100k.
A.3. Matching ground-truth
The matching ground truth is constructed at coarse patch and fine pixel levels from dense 3D geometry, reprojection, and cycle-consistency checks. The resulting training targets support dense and reliable predicted flows.
- Ground-truth construction: The matcher uses fully supervised cross-entropy losses for both coarse patch matching and fine pixel matching.The coarse and fine objectives are denoted L_coarse and L_fine.
- Fine ground-truth: Fine correspondences are derived by reprojecting 3D points between images and retaining matches that pass a 2-pixel cycle-consistency check.The procedure filters occluded and out-of-frame regions before assigning pixel matches.
- Coarse ground-truth: A coarse patch pair is valid when its patches share at least one fine correspondence.This construction supplies the fine matcher with patch pairs needed to reconstitute dense optical flow.
- Result: The trained network produces dense and reliable predicted flows from these ground-truth matches.
B. GPU–Accelerated Gauss–Newton / LM Solver
The solver implements bundle adjustment with GPU-resident block operators, Schur elimination, and preconditioned conjugate gradients rather than large dense matrices. This design supports linear scaling in 3D points and tens of millions of residuals on one GPU.
- Jacobian representation: The implementation uses a packed, block-based Jacobian representation instead of explicitly constructing the full dense Jacobian.Local residual Jacobians are placed with lightweight sparse row-block structures that remain differentiable and GPU-friendly.
- Reduced-system solve: The solver eliminates the 3D-point block with a Schur complement and solves the reduced camera system using preconditioned conjugate gradients.Point updates are recovered after solving the reduced system.
- Scalability: Block-sparse operators and implicit CUDA matrix-vector products keep Schur operations linear in the number of points.The point block is block-diagonal, enabling efficient solves without materializing dense matrices.
- Optimization loop: Levenberg–Marquardt damping stabilizes early iterations and prevents ill-conditioned steps while keeping operations sparse and GPU-resident.The outer loop recomputes residuals, adapts damping, and repeats until the loss stops decreasing.
- Performance: The implementation avoids CPU round trips and large dense factorizations, handling tens of millions of residuals on a single GPU.Its computations scale linearly with the number of 3D points and keypoints.
C.1. Extensive VSLAM benchmarks
Additional benchmarks report strong VSLAM performance and linear overall complexity, while contrasting BLASt3R’s scaling with the quadratic complexity and memory limitations of VGGT.
- Benchmark results: BLASt3R clearly outperforms existing state-of-the-art methods on the full TUM-RGBD dataset and matches VIPE on full ETH3D-SLAM.These results are reported for additional TUM-RGBD subsets and ETH3D-SLAM scenes beyond the main paper.
- Complexity: The approach has overall complexity linear in the number of images across monocular depth estimation, multi-view matching, and unified bundle adjustment.Monocular depth estimation is independent per image, while the remaining components are analyzed to establish overall linear scaling.
- Comparison to VGGT: VGGT has quadratic image complexity and can encounter out-of-memory crashes on larger collections, whereas BLASt3R is linear.A runtime comparison uses the same 80GB H100 hardware for both systems.
D. Additional Qualitative Examples
Additional visualizations show consistent reconstructions in offline and online settings, dense matching in difficult regions, and corrective depth channels that support global optimization. The examples also illustrate performance under skipped frames and varied challenging structures.
- Qualitative results: BLASt3R produces consistent 3D scenes and camera poses on unordered images, including cases where concurrent methods struggle.The delivery-area example is reported as a highly consistent reconstruction in the offline setting.
- Online reconstruction: The method handles skipped frames without temporal smoothness assumptions and uses unified bundle adjustment to refine depthmaps and global loop closure.The accompanying examples include online reconstructions across thousands of frames in varied scenarios.
- Corrective depthmaps: The monocular network outputs one main depth channel and corrective channels whose coefficients are optimized by unified bundle adjustment.The corrective channels are orthogonalized only for visualization, not as a requirement of the optimization.
- Dense matching: The matcher predicts dense optical flow from each query image to 11 retrieved memory images in a single forward pass.Memory images are retrieved using ASMK features and visualized through bilinear warping.
- Matching quality: Dense flow coverage remains visible in matchable regions containing pipes, uniform areas, and scenes with little image overlap.The network also identifies non-matchable regions, indicating 3D awareness in challenging cases.