Source-linked AI summary
DUSt3R: Geometric 3D Vision Made Easy
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, Jerome Revaud
TL;DR
Unconstrained multi-view stereo traditionally requires cumbersome camera calibration and pose estimation before triangulating scene points. DUSt3R predicts dense pointmaps directly from uncalibrated, unposed image pairs and globally aligns them for multiple views. The resulting unified system provides strong performance across depth, pose, localization, and broader 3D vision tasks.
Problem
Traditional SfM and MVS rely on sequential camera and geometry subproblems whose imperfect solutions accumulate noise and engineering complexity.
Method
DUSt3R regresses dense pointmaps from image pairs without camera calibration or poses, then globally aligns pairwise pointmaps in 3D for multi-view reconstruction.
Results
DUSt3R achieves state-of-the-art results on monocular and multi-view depth benchmarks and multi-view camera pose estimation, while also performing strongly on visual localization.
Takeaways & Limitations
The pointmap formulation unifies multiple geometric 3D vision tasks and considerably simplifies the traditional reconstruction pipeline.
Takeaways & Limitations
Pointmaps are regressed up to an unknown scale factor and are not explicitly constrained to correspond to physically plausible camera models.
Abstract
from arXiv · showhide
Multi-view stereo reconstruction (MVS) in the wild requires to first estimate the camera parameters e.g. intrinsic and extrinsic parameters. These are usually tedious and cumbersome to obtain, yet they are mandatory to triangulate corresponding pixels in 3D space, which is the core of all best performing MVS algorithms. In this work, we take an opposite stance and introduce DUSt3R, a radically novel paradigm for Dense and Unconstrained Stereo 3D Reconstruction of arbitrary image collections, i.e. operating without prior information about camera calibration nor viewpoint poses. We cast the pairwise reconstruction problem as a regression of pointmaps, relaxing the hard constraints of usual projective camera models. We show that this formulation smoothly unifies the monocular and binocular reconstruction cases. In the case where more than two images are provided, we further propose a simple yet effective global alignment strategy that expresses all pairwise pointmaps in a common reference frame. We base our network architecture on standard Transformer encoders and decoders, allowing us to leverage powerful pretrained models. Our formulation directly provides a 3D model of the scene as well as depth information, but interestingly, we can seamlessly recover from it, pixel matches, relative and absolute camera. Exhaustive experiments on all these tasks showcase that the proposed DUSt3R can unify various 3D vision tasks and set new SoTAs on monocular/multi-view depth estimation as well as relative pose estimation. In summary, DUSt3R makes many geometric 3D vision tasks easy.
1. Introduction
DUSt3R replaces the complex sequence of calibrated reconstruction subproblems with an end-to-end approach that predicts dense 3D pointmaps from uncalibrated, unposed images. It globally aligns pairwise predictions for multi-view reconstruction and achieves state-of-the-art results across several 3D vision tasks.
- Motivation: Classical SfM and MVS pipelines pass through sequential matching, geometry estimation, triangulation, camera estimation, and dense reconstruction, accumulating noise and engineering complexity.The subproblems are not solved perfectly and do not communicate with one another.
- DUSt3R: DUSt3R regresses dense pointmaps from image pairs without prior scene information, camera poses, or even camera intrinsics.The pointmaps jointly encode scene geometry, pixel-to-scene relationships, and viewpoint relationships.
- Learning strategy: The model uses supervised regression with a generic Transformer architecture, avoiding explicit geometric constraints at inference and leveraging pretrained models.Training uses annotations from synthetic data, SfM reconstructions, and dedicated sensors.
- Multi-view alignment: A global alignment procedure fuses pairwise pointmaps directly in 3D space, avoiding reprojection-error minimization for full-scale multi-view stereo.The procedure optimizes camera-pose and geometry alignment and is described as fast with excellent practical convergence.
- DUSt3R: The pointmap representation unifies monocular and binocular reconstruction while relaxing constraints imposed by conventional perspective-camera formulations.It predicts 3D shape in a canonical frame while preserving implicit pixel-to-scene relationships.
- Results: DUSt3R achieves state-of-the-art results on monocular and multi-view depth benchmarks and multi-view camera pose estimation.The all-in-one model is evaluated across a range of 3D vision tasks.
2. Related Work
Prior 3D vision methods typically decompose reconstruction into sequential or task-specific modules that rely on camera information. DUSt3R instead processes multiple viewpoints jointly and predicts pointmaps that encode dense 3D structure while implicitly handling camera pose.
- Structure-from-Motion: SfM reconstructs sparse 3D maps and camera parameters from images through feature matching, geometric estimation, and bundle adjustment.Its sequential structure remains vulnerable to noise and errors in individual components.
- Depth-based reconstruction: Depth maps provide 3D information but generally require camera intrinsics to produce pixel-aligned 3D point clouds.Prior approaches can infer intrinsics from temporal consistency when they are unavailable.
- Pointmaps: DUSt3R processes two viewpoints jointly and outputs dense pointmaps rather than depthmaps plus an explicitly estimated relative pose.Pointmaps are dense 2D fields of 3D points that implicitly handle camera poses and make regression better posed.
3. Method
DUSt3R directly regresses dense pointmaps from image pairs without requiring camera calibration or poses. Its transformer architecture jointly processes both views, while confidence prediction, downstream geometric recovery, and global alignment extend the method to multi-view reconstruction and related camera tasks.
- Pointmaps: A pointmap is a dense 2D field of 3D points that creates a one-to-one mapping between image pixels and scene points.Each pointmap has shape W × H × 3 and is associated with an RGB image.
- Network overview: The network takes two RGB images and outputs two pointmaps and confidence maps, both pointmaps expressed in the first image’s coordinate frame.This formulation avoids requiring prior camera intrinsics or poses for pairwise reconstruction.
- Network overview: A shared ViT encoder processes both images, followed by transformer decoders whose branches exchange information through cross-attention.Each decoder block performs self-attention, cross-attention, and an MLP, enabling information sharing needed for aligned pointmaps.
- Training objective: The model uses 3D regression with scale normalization and jointly predicts per-pixel confidence to handle ambiguous or difficult regions.Confidence weighting addresses ill-defined points such as sky or translucent objects and requires no explicit confidence supervision.
- Downstream applications: Pointmaps support nearest-neighbor pixel matching, intrinsic recovery, and relative or absolute pose estimation through geometric alignment and PnP-based procedures.Relative pose can be obtained by Procrustes alignment or, more robustly, RANSAC with PnP.
- Global alignment: For multiple images, global alignment optimizes pairwise poses and scales directly in 3D to place pointmaps in a common scene coordinate frame.The procedure fuses pairwise predictions and can recover camera poses, intrinsics, and depthmaps; it typically converges in a few hundred steps and takes seconds on a standard GPU.
4. Experiments with DUSt3R
DUSt3R is evaluated across visual localization, multi-view pose, monocular and multi-view depth, and full 3D reconstruction using one model and varied settings. It achieves competitive or state-of-the-art results despite operating without prior camera parameters in several tasks.
- Evaluation: The experiments benchmark the same DUSt3R model on representative classical 3D vision tasks, with task-specific routes for extracting outputs.The evaluation covers visual localization, multi-view pose estimation, monocular depth, multi-view depth, and full reconstructions.
- Qualitative results: DUSt3R produces high-quality dense 3D reconstructions in challenging situations, including non-cherrypicked pairwise and multi-view visualizations.
- Visual localization: DUSt3R obtains accuracy comparable to feature-matching and end-to-end localization methods, outperforming HLoc in some scenes without being trained for visual localization.Neither the query nor database images were seen during training.
- Multi-view pose estimation: DUSt3R with global alignment achieves the best overall multi-view pose performance on CO3Dv2 and RealEstate10K, surpassing PoseDiffusion and other learning- and structure-based methods.The PnP variant also outperforms both learning-based and structure-based existing methods.
- Monocular depth: DUSt3R adapts to outdoor and indoor monocular depth datasets, outperforming self-supervised baselines and matching state-of-the-art supervised baselines in zero-shot evaluation.The benchmarks include DDAD, KITTI, NYUv2, BONN, and TUM.
- Multi-view depth: DUSt3R achieves state-of-the-art accuracy on ETH-3D and outperforms most recent methods overall for multi-view depth, including methods using ground-truth camera poses.Predictions do not use ground-truth camera parameters, poses, or depth ranges and are valid up to scale.
- 3D reconstruction: 2.7mm average accuracy, 0.8mm completeness, and 1.7mm overall average distance are obtained on DTU without prior camera knowledge.The authors characterize this accuracy as useful in practice given the plug-and-play nature of the approach.
- Ablations: Consistent improvements across evaluated tasks show that CroCo pretraining and high image resolution substantially affect DUSt3R performance.
5. Conclusion
DUSt3R presents a paradigm for geometric 3D vision that extends from two-image reconstruction to multi-view alignment and camera recovery without requiring prior calibration or poses.
- DUSt3R is presented as a novel paradigm for solving 3D reconstruction and a broad range of geometric 3D vision tasks without prior scene or camera information.
- The qualitative results include raw depthmaps, confidence maps, colored point clouds, and recovered camera parameters from unseen scenes and image pairs.
- DUSt3R handles strong viewpoint and focal changes, including nearly opposite views, in the shown reconstruction examples.
- The method also reconstructs multi-view scenes from four frames, producing globally aligned point clouds together with recovered camera intrinsics and poses.
Appendix
The appendix supplies supplementary qualitative results, related-work discussion, and additional multi-view pose-estimation ablations for DUSt3R.
- The appendix presents qualitative pairwise predictions on challenging real-life datasets and describes the accompanying video.
- It expands the related-work review across methodological families and geometric vision tasks.
- It also reports auxiliary ablations for multi-view pose estimation.
B. Qualitative results
DUSt3R produces accurate pairwise reconstructions on unseen scenes and globally consistent multi-view pointmaps, while the appendix situates the approach among explicit and implicit 3D reconstruction methods.
- Qualitative results: On unseen, non-cherry-picked scenes, DUSt3R performs highly accurate 3D reconstruction from only two images using raw network outputs.
- Qualitative results: After global alignment, processing all image pairs yields spatially consistent pointmaps and corresponding camera parameters.
- Qualitative results: For same-camera image sequences, the method optimizes intrinsic parameters independently rather than enforcing identical intrinsics across frames.
- Related work: The related-work discussion contrasts DUSt3R’s explicit 3D reconstruction with implicit approaches and emphasizes its downstream geometric information.
D. Multi-view Pose Estimation
Additional experiments evaluate DUSt3R for multi-view pose estimation with varying numbers of frames on CO3Dv2 and RealEstate10K.
- DUSt3R consistently outperforms other methods on CO3Dv2 by a large margin, including evaluations with small numbers of frames.
- The method handles nearly opposite viewpoints and maintains relatively stable performance regardless of the number of input views.
- On RealEstate10K, the comparison reports DUSt3R after global alignment both without training on that dataset and against PoseDiffusion settings.
- A RealEstate10K reconstruction produces a consistent point cloud despite wide baseline viewpoint changes between the first and last frame pairs.
E. Visual localization
The visual-localization evaluation shows reasonable performance on 7-scenes but considerably larger errors on Cambridge-Landmarks because sparse database pointmaps prevent reliable scaling.
- On 7-scenes, the median translation error is on the order of a few centimeters.
- Cambridge-Landmarks produces considerably larger localization errors than 7-scenes.The authors attribute this difference to sparse ground-truth database pointmaps that prevent reliable reconstruction scaling.
- Further work is necessary for in-the-wild visual localization with unknown intrinsics.
F.1. Training data
The appendix describes how ground-truth pointmaps and depthmaps are constructed and reports a diverse eight-dataset training mixture totaling 8.5M image pairs.
- Ground-truth pointmaps: Ground-truth pointmaps are obtained from camera intrinsics and poses, then expressed in the reference frame of the first camera.
- Relation between depthmaps and pointmaps: Depth values at image pixels can be recovered directly from the pointmap output.
- Dataset mixture: 8.5M image pairs from eight datasets provide diverse indoor, outdoor, synthetic, real-world, and object-centric training data.The datasets are Habitat, ARKitScenes, MegaDepth, Static Scenes 3D, Blended MVS, ScanNet++, CO3Dv2, and Waymo.
F.2. Training hyperparameters
The appendix reports that DUSt3R’s detailed training hyperparameter settings are provided in Table 7.
- Detailed training hyperparameter settings are reported in Table 7.