PaperlayerView exact revision on arXiv

Source-linked AI summary

MUSt3R: Multi-view Network for Stereo 3D Reconstruction

Yohann Cabon, Lucas Stoffl, Leonid Antsfeld, Gabriela Csurka, Boris Chidlovskii, Jerome Revaud, Vincent Leroy

arXiv:2503.01661v1cs.CV

TL;DR

Pairwise DUSt3R reconstruction scales poorly because local predictions require global alignment across a quadratically growing number of image pairs. MUSt3R introduces symmetric N-view prediction with iterative memory, enabling offline and online reconstruction, and reports state-of-the-art performance across multiple 3D tasks.

  • Problem

    DUSt3R’s pairwise processing creates quadratic complexity and separate local coordinate systems, complicating robust and fast reconstruction of large image collections.

  • Method

    MUSt3R uses a symmetric N-view architecture with shared weights and an iteratively updated memory for offline and online reconstruction.

  • Results

    MUSt3R achieves state-of-the-art performance across unconstrained depth, relative pose, 3D reconstruction, and uncalibrated visual odometry tasks.

  • Takeaways & Limitations

    The same memory-enabled network can operate without architecture changes in both offline SfM and online dense VO or SLAM scenarios.

  • Takeaways & Limitations

    MUSt3R shows limitations on sequences where views drift too far from the first view.

Abstract

from arXiv · show

DUSt3R introduced a novel paradigm in geometric computer vision by proposing a model that can provide dense and unconstrained Stereo 3D Reconstruction of arbitrary image collections with no prior information about camera calibration nor viewpoint poses. Under the hood, however, DUSt3R processes image pairs, regressing local 3D reconstructions that need to be aligned in a global coordinate system. The number of pairs, growing quadratically, is an inherent limitation that becomes especially concerning for robust and fast optimization in the case of large image collections. In this paper, we propose an extension of DUSt3R from pairs to multiple views, that addresses all aforementioned concerns. Indeed, we propose a Multi-view Network for Stereo 3D Reconstruction, or MUSt3R, that modifies the DUSt3R architecture by making it symmetric and extending it to directly predict 3D structure for all views in a common coordinate frame. Second, we entail the model with a multi-layer memory mechanism which allows to reduce the computational complexity and to scale the reconstruction to large collections, inferring thousands of 3D pointmaps at high frame-rates with limited added complexity. The framework is designed to perform 3D reconstruction both offline and online, and hence can be seamlessly applied to SfM and visual SLAM scenarios showing state-of-the-art performance on various 3D downstream tasks, including uncalibrated Visual Odometry, relative camera pose, scale and focal estimation, 3D reconstruction and multi-view depth estimation.

1. Introduction

MUSt3R extends DUSt3R from pairwise to symmetric multi-view reconstruction, addressing the quadratic scaling and global alignment costs of large image collections while supporting offline and online use.

  • DUSt3R background: DUSt3R provides dense, unconstrained stereo 3D reconstruction without prior camera calibration or viewpoint poses.Its pointmap representation jointly captures 3D geometry and camera parameters for depth, pose, focal-length, and reconstruction tasks.
  • Motivation: Pairwise processing becomes a drawback for many images because local pointmaps occupy different coordinate systems and require costly global alignment.The number of pairs grows quadratically, making naive global post-processing difficult for large collections.
  • Contributions: MUSt3R makes DUSt3R symmetric, enabling N-view predictions in metric space through a shared architecture.This design directly predicts structure across multiple views in a common coordinate frame.
  • Contributions: A memory mechanism reduces computational complexity for offline and online reconstruction while supporting large image collections.The model is intended for both unordered SfM reconstruction and online dense VO or SLAM without changing the architecture.
  • Contributions: MUSt3R achieves state-of-the-art performance across unconstrained reconstruction tasks including field-of-view, camera pose, 3D reconstruction, and absolute scale.The reported contribution emphasizes maintaining inference speed while covering both offline and online scenarios.

2. Related work

Related work progresses from direct RGB-to-3D and pairwise pointmap regression toward memory-based global prediction, while MUSt3R integrates memory into an N-view architecture and supports uncalibrated VO.

  • Direct RGB-to-3D: Learning-based RGB-to-3D methods directly predict 3D and cameras, using learned priors from large datasets or differentiable SfM.DUSt3R instead formulates pairwise reconstruction as regression of pointmaps.
  • Memory-based reconstruction: Spann3R uses spatial memory to predict per-image pointmaps in global coordinates, avoiding global alignment while retaining DUSt3R’s sequential pairwise architecture.Most images pass through the model twice, with memory queried between passes.
  • Memory-based reconstruction: MUSt3R departs from pairwise processing by evolving DUSt3R to N-view prediction and making memory an essential architectural element.This distinguishes its design from approaches that retain sequential pair processing.
  • Visual Odometry: Existing visual odometry methods commonly rely on handcrafted heuristics and projective camera geometry for joint pose and scene optimization.Prior-driven approaches add monocular depth, regularization, or learned optical flow.
  • Uncalibrated regime: MUSt3R operates online or offline without calibration and estimates dense 3D, camera parameters, and scene scale at high inference speed.The related-work discussion contrasts this scope with existing VO methods.

3. Method

MUSt3R reformulates DUSt3R for arbitrary multi-view reconstruction with a shared symmetric decoder, memory, and 3D feedback. These mechanisms support scalable offline and causal online processing, while rendering enables non-causal recomputation from stored representations.

  • Simplifying the DUSt3R architecture: MUSt3R replaces DUSt3R’s paired decoders with a single weight-sharing Siamese decoder that naturally handles N views and halves decoder parameters.A learnable embedding identifies the reference image defining the common coordinate system.
  • Scaling-up to multi-view: Cross-attention lets each image exchange information with tokens from all other images at every decoder layer.The multi-view extension changes inter-view attention while retaining residual self-attention and MLP blocks.
  • Fast relative pose regression: An additional self-coordinate pointmap X_i,i enables relative pose recovery by Procrustes analysis without requiring focal length.This avoids DUSt3R’s second symmetric-pair forward for estimating the second image’s intrinsics.
  • Introducing causality in MUSt3R: An iteratively updated multi-layer memory processes arbitrary image sequences offline or online, with new frames cross-attending to saved tokens.Rendering recomputes pointmaps without appending tokens, allowing causality to be broken after future frames have entered memory.
  • Introducing causality in MUSt3R: 3D feedback injects terminal-layer information into earlier decoder layers through an extra MLP, propagating global 3D knowledge throughout the network.The mechanism uses a LayerNorm followed by a two-layer MLP; memory growth is controlled by selecting informative frames or keyframes.

4. Training

MUSt3R is trained progressively, first for metric pairwise predictions and then for multi-view reconstruction with memory updates. The training uses diverse datasets, fixed-size image tuples, and token dropout for robustness.

  • Training: MUSt3R first trains a simplified symmetric architecture for metric predictions, using log-space regression to improve convergence on distant points.The model is initialized from CroCo v2 and later fine-tuned at higher image resolution.
  • Training: The initial training uses a mixture of 14 datasets spanning indoor, outdoor, synthetic, object-centric, and driving scenes.The listed datasets include Habitat, ARKitScenes, ScanNet++, CO3D-v2, TartanAir, and an internal dataset.
  • Multi-view training: Multi-view training starts from the trained symmetric DUSt3R and uses 10-image scenes with a frozen encoder and efficient attention.The memory is initialized from two images and then updated with individual images while training losses are split across partial view sequences.
  • Regularization: Token dropout is applied independently to incoming frames while protecting first-image memory tokens, with probabilities of 0.05 at 224 resolution and 0.15 at 512 resolution.The dropout is consistent across layers and is intended to increase robustness and redundancy.

5. Experimental validation

MUSt3R is evaluated across uncalibrated visual odometry, pose, depth, and reconstruction tasks on real and standard benchmark datasets. It generally matches or exceeds competing methods while retaining high inference speed, though performance is limited when views drift far from the first view.

  • Experimental scope: MUSt3R is tested in uncalibrated visual odometry, relative pose estimation, 3D reconstruction, and multi-view depth estimation.The evaluation compares against state-of-the-art methods, especially DUSt3R and Spann3R.
  • Uncalibrated Visual Odometry: On TUM RGBD, rendered MUSt3R achieves the best average trajectory performance, while MUSt3R-C performs well across short and long sequences.Rendering reduces error but lowers speed slightly to 8.4 FPS.
  • Uncalibrated Visual Odometry: 4° average FoV error and 4.6% median scale error show strong uncalibrated estimation for MUSt3R on the reported TUM RGBD subsets.MUSt3R-C has a comparable median scale error of 5.5%.
  • Uncalibrated Visual Odometry: MUSt3R is 50% more accurate than Spann3R on average on ETH3D SLAM, with 10.0 cm average RMSE at around 10 FPS.ETH3D SLAM contains trajectories recorded in a motion-capturing system.
  • 3D Reconstruction: Across pointmap benchmarks, MUSt3R almost always outperforms Spann3R with better FPS, while achieving performance similar to DUSt3R at five times less weight and an order-of-magnitude higher speed.The comparison covers 7Scenes, Neural RGBD, and DTU using accuracy, completeness, and normal consistency.
  • Multi-view Depth Estimation: MUSt3R-224 performs better than other baselines on average for depth evaluation, while MUSt3R-512 performs similarly to DUSt3R.The evaluation reports absolute relative error and inlier ratio across KITTI, ScanNet, ETH3D, DTU, and Tanks and Temples.
  • Limitations: The method shows limitations on sequences where viewpoints drift too far from the first view.This scope boundary is reported despite strong results across multiple downstream tasks.

6. Conclusion

MUSt3R extends DUSt3R into a high-speed multi-view network for large image collections, supporting both offline and online reconstruction. Evaluations report state-of-the-art performance across several 3D downstream tasks.

  • 6. Conclusion: MUSt3R reconstructs large image collections in both offline and online scenarios at high speed.The framework covers Structure-from-Motion and online visual odometry or SLAM settings.
  • 6. Conclusion: The appendix analyzes DUSt3R simplifications, memory and loss choices, scalability to more views, qualitative examples, and full quantitative evaluations.These studies include ablations, real-scene visualizations, online reconstruction code, and additional TUM-RGBD2 and ETH3D evaluations.
  • 6. Conclusion: A symmetric decoder uses one shared decoder for all views, enabling scaling without retraining and reducing decoder trainable parameters by half.The design removes the need for N separate decoders while retaining performance comparable to DUSt3R when the learned view embedding is used.
  • 6. Conclusion: MUSt3R produces qualitative reconstructions across Cambridge Landmarks and MIP-360 scenes.The cited appendix figures provide examples from both datasets.

A.2. MUSt3R architecture and loss

The multi-view design combines sequential memory processing with global 3D feedback and a log-space loss to scale reconstruction while preserving useful predictions. Ablations identify the feedback source, loss, and memory configuration as important design choices.

  • A.2. MUSt3R architecture and loss: Directly processing all N views creates an intractable token count, motivating sequential processing with only a subset of image tokens in memory.The multi-view setting trains on 10-tuples and evaluates 100-image renderings with varying memory sizes.
  • A.2. MUSt3R architecture and loss: The ablations evaluate predictions in the first-view coordinate frame, avoiding alignment between predictions and ground truth.The reported metric is the 3D regression loss on ScanNet++ validation tuples.
  • A.2. MUSt3R architecture and loss: Global 3D feedback from the terminal memory layer outperforms no feedback and constant injection, while feedback from final layer L significantly degrades performance.The authors hypothesize that final-layer features are too constrained by Head3D, favoring feedback from layer L−1.
  • A.2. MUSt3R architecture and loss: Above 10 views in memory, the log-space loss scales significantly better than the default MASt3R metric regression loss.With n=50 views in memory, the baseline reaches 13.4cm accuracy versus 12.6 for the log-space loss.
  • A.2. MUSt3R architecture and loss: Table 10 compares MUSt3R and Spann3R across maximum memory size and the number of images used per memory update.For s=1, MUSt3R is initialized with two images to match training configuration.

A.3. More views at inference

MUSt3R generalizes beyond its training configuration at inference: it can use substantially more views in memory and update memory with multiple views at once. Performance remains stable across these changes, with a speed benefit for larger simultaneous updates.

  • A.3. More views at inference: MUSt3R predicts from n≫10 views in memory despite training with n≤10, and processes s≫1 views simultaneously despite training with s=1.This demonstrates inference-time scalability beyond the view counts used during training.
  • A.3. More views at inference: The model successfully leverages up to n=50 views in memory, with no further improvement beyond 50 views.Tabulated results also show that using all available views is better than restricting memory to 10 views in the cited comparison.
  • A.3. More views at inference: Updating memory with s=5 or s=10 views does not significantly degrade performance across DTU, NRGBD, and 7-Scenes.The comparison includes both MUSt3R-224 and MUSt3R-512.
  • A.3. More views at inference: 72.59 FPS for s=10 versus 40 for the one-view update gives MUSt3R-224 a speed increase when multiple views update memory together.The supplied passage reports the one-view value only partially, as “40”.

B. Qualitative examples

The appendix presents qualitative reconstructions across indoor, outdoor, object-centric, and heterogeneous-sensor scenes, alongside implementation material for online reconstruction. MUSt3R is presented as a plug-and-play approach for varied scenarios.

  • B. Qualitative examples: MUSt3R does not assume a single intrinsic calibration for an image collection and works with heterogeneous sensors.The paper illustrates this property using ScanNet++ scenes.
  • B. Qualitative examples: The online algorithm is provided as pseudo-code and is positioned as a simplified route toward visual odometry and RGB-SLAM.The network internally handles keypoint selection, matching, pose estimation, and 3D triangulation.

D. Full Visual Odometry results

This section provides detailed evaluation tables for the online MUSt3R model on TUM RGB-D and ETH3D.

  • The evaluation focuses on the online MUSt3R model.
  • TUM RGB-D is included among the evaluated datasets.
  • ETH3D is included among the evaluated datasets.

D.1. Full TUM RGB-D dataset

The section completes the TUM RGB-D evaluation with detailed MUSt3R-224 results for all 34 remaining sequences and comparison to GlORIE-SLAM.

  • Full TUM RGB-D dataset: 34 remaining TUM RGB-D sequences receive detailed MUSt3R-224 evaluation results.These sequences complete the evaluation beyond the 11 sequences reported earlier.
  • Full TUM RGB-D dataset: The reported metrics are RMSE APE, vertical FoV, and scale errors.
  • Full TUM RGB-D dataset: The detailed table compares MUSt3R-224 with GlORIE-SLAM.

D.2. ETH3D dataset

The ETH3D evaluation adds detailed results for 32 sequences and compares MUSt3R variants with Spann3R, with MUSt3R outperforming on most sequences but remaining challenged by large scenes.

  • ETH3D dataset: 32 ETH3D sequences receive detailed MUSt3R-224 evaluation results.These complement the 8 sequences included in the main paper.
  • ETH3D dataset: The tables report RMSE APE, vertical FoV, and scale errors for all 32 sequences.
  • ETH3D dataset: Both MUSt3R and MUSt3R-C outperform Spann3R on most sequences at 224 resolution, while large scene sizes remain challenging.The authors note room for improvement on some sequences, usually when the scene size becomes too large.
Loading 2503.01661v1…