Source-linked AI summary
MapAnything: Universal Feed-Forward Metric 3D Reconstruction
Nikhil Keetha, Norman Müller, Johannes Schönberger, Lorenzo Porzi, Yuchen Zhang, Tobias Fischer, Arno Knapitsch, Duncan Zauss, Ethan Weber, Nelson Antunes, Jonathon Luiten, Manuel Lopez-Antequera, Samuel Rota Bulò, Christian Richardt, Deva Ramanan, Sebastian Scherer, Peter Kontschieder
TL;DR
Feed-forward 3D reconstruction has lacked a unified model that uses heterogeneous geometric inputs across diverse tasks. MapAnything addresses this with a factored metric representation and transformer architecture, matching or surpassing specialist models in a single pass. Its scope remains limited by unmodeled input uncertainty, memory scalability, and the absence of dynamic-scene modeling.
Problem
Prior feed-forward methods approached diverse 3D reconstruction tasks separately or did not leverage all available input modalities.
Method
MapAnything uses a transformer with a factored representation of depth maps, local ray maps, camera poses, and metric scale to regress metric 3D geometry and cameras from flexible inputs.
Results
MapAnything outperforms or matches specialist feed-forward models across diverse 3D reconstruction tasks while enabling efficient joint training.
Takeaways & Limitations
A single feed-forward backbone supports more than 12 reconstruction configurations without task-specific tuning.
Takeaways & Limitations
The model does not explicitly account for noise or uncertainty in geometric inputs, has limited scalability for large scenes, and does not capture dynamic motion or scene flow.
Abstract
from arXiv · showhide
We introduce MapAnything, a unified transformer-based feed-forward model that ingests one or more images along with optional geometric inputs such as camera intrinsics, poses, depth, or partial reconstructions, and then directly regresses the metric 3D scene geometry and cameras. MapAnything leverages a factored representation of multi-view scene geometry, i.e., a collection of depth maps, local ray maps, camera poses, and a metric scale factor that effectively upgrades local reconstructions into a globally consistent metric frame. Standardizing the supervision and training across diverse datasets, along with flexible input augmentation, enables MapAnything to address a broad range of 3D vision tasks in a single feed-forward pass, including uncalibrated structure-from-motion, calibrated multi-view stereo, monocular depth estimation, camera localization, depth completion, and more. We provide extensive experimental analyses and model ablations demonstrating that MapAnything outperforms or matches specialist feed-forward models while offering more efficient joint training behavior, thus paving the way toward a universal 3D reconstruction backbone.
2 Carnegie Mellon University
MapAnything is presented as a flexible system for metric-scale 3D reconstruction from multiple images and optional geometric information. It covers more than 12 reconstruction tasks while matching or surpassing specialist methods.
- MapAnything predicts metric 3D reconstructions with camera information from N input images and optional poses, intrinsics, or depth maps.
- It supports over 12 tasks, including camera localization, structure-from-motion, multi-view stereo, and metric depth completion.
- MapAnything outperforms or matches the quality of specialist methods across its supported reconstruction tasks.
1. Introduction
The introduction frames MapAnything as a unified end-to-end model for diverse 3D reconstruction tasks, addressing fragmented prior approaches and heterogeneous inputs. Its factored representation supports metric reconstruction, efficient joint training, and direct prediction without costly post-processing.
- Prior feed-forward methods approached reconstruction tasks separately or failed to leverage all available input modalities.
- MapAnything unifies uncalibrated SfM, calibrated SfM, multi-view stereo, monocular depth estimation, camera localization, and metric depth completion.
- Its factored representation combines depth maps, local raymaps, camera poses, and a metric scale factor to produce globally consistent metric geometry.
- The model supports more than 12 problem configurations and uses optional intrinsics, extrinsics, depth, and metric scale inputs.
- MapAnything computes multi-view pixelwise scene geometry and cameras directly without redundancies or costly post-processing.
- It matches or surpasses expert models tailored to specific isolated tasks.
- The release includes data-processing, inference, benchmarking, training, and ablation code plus a pretrained model under Apache 2.0.
2. Related Work
Related work progresses from specialized reconstruction pipelines toward transformer-based feed-forward systems, but prior approaches retain representation, input, or efficiency constraints. MapAnything addresses these gaps with factored geometry and heterogeneous-input support.
- Traditional reconstruction decomposes the problem into specialized stages such as feature matching, calibration, pose estimation, bundle adjustment, and multi-view stereo.
- Recent transformer methods enable joint multi-view inference and large-scale training, extending earlier CNN-based efforts that did not match classical expert models.
- DUSt3R and MASt3R use coupled scene representations that require post hoc recovery or expensive post-processing for unconstrained multi-view SfM.
- MV-DUSt3R+ increases computation by parallelizing cross-attention across different reference views, while VGGT predicts multi-view pointmaps with an alternating-attention transformer.
- MV-DUSt3R+ and FASt3R cannot handle heterogeneous inputs, and FASt3R reports dense-geometry effects from pose estimation across non-visible views.
- MapAnything uses a factored representation to remove redundant predictions and directly compute scene geometry and cameras.
- Unlike prior image-only methods with simple pinhole cameras, MapAnything supports heterogeneous inputs, multiple reconstruction and calibration tasks, and a flexible camera model.
3. MapAnything
MapAnything represents multi-view geometry in factored metric form and predicts it with a unified transformer from images plus optional geometric inputs. Its design separates scale, geometry, and camera quantities to support flexible metric reconstruction and end-to-end training.
- Inputs and outputs: MapAnything maps RGB images and optional ray directions, poses, and ray depths to factored metric 3D outputs across N views.Inputs may correspond to all or only subsets of views.
- Inputs and outputs: The output contains a global metric scaling factor, local ray directions, up-to-scale ray depths, and per-view poses relative to the first image.Poses use quaternions and up-to-scale translations.
- Architecture: A multi-view transformer fuses image and geometric tokens, then DPT, pose, and scale heads decode dense geometry, cameras, and metric scale.The scale token is passed through a dedicated MLP, and the predicted scale is exponentially transformed.
- Factored representation: Factoring geometry into rays, depths, poses, and scale enables metric and up-to-scale inputs while avoiding redundant pointmap representations.The model separately encodes rotation and translation to support partial geometric priors and disentangle translation from scale.
- Training objectives: Training combines pointmap, ray, rotation, translation, depth, scale, normal, gradient, and mask losses, with log-space losses for depths, pointmaps, and scale.Normal and gradient losses are applied only to synthetic datasets because real geometry can be coarse and noisy.
4. Benchmarking & Results
MapAnything is evaluated across dense reconstruction, two-view reconstruction, calibration, and metric depth estimation, often matching or surpassing specialist and feed-forward baselines. The results also support its factored representation and probability-based training as enablers of one universal model across input configurations.
- Multi-View Dense Reconstruction: Across ETH3D, ScanNet++ v2, and TartanAirV2-WB, MapAnything delivers state-of-the-art dense multi-view reconstruction from image-only and augmented inputs.Evaluations sample up to 100 views and report depth, ray-direction, pose, and pointmap-related metrics under multiple configurations.
- Multi-View Dense Reconstruction: Additional geometric inputs further increase dense multi-view reconstruction performance by a significant factor.The model can use auxiliary inputs such as calibration, poses, and depth during feed-forward inference.
- Two-View Dense Reconstruction: MapAnything achieves state-of-the-art two-view reconstruction using images alone and significantly outperforms image-only baselines and Pow3R when auxiliary modalities are available.Pow3R is identified as the other two-view feed-forward method using scene or camera priors.
- Single-View Calibration: MapAnything achieves state-of-the-art single-view perspective calibration despite not being trained specifically for single-image inputs.The evaluation tests non-centered principal points using randomly cropped frames.
- Monocular & Multi-View Depth Estimation: For metric depth estimation, MapAnything achieves state-of-the-art or comparable performance, outperforming MASt3R-BA and MUSt3R for image-only multi-view depth.Auxiliary calibration and pose inputs improve performance, while metric scale estimation on ScanNet is reported as sub-optimal relative to MoGe-2 and MVSA.
- Insights into enabling MapAnything: The factored ray-depth-pose representation with metric scale and input probability-based training support strong reconstruction and efficient training of one universal model.The reported universal model is equivalent to various bespoke models trained for specific input configurations.
5. Limitations
MapAnything’s remaining limitations concern uncertainty handling, multimodal fusion, scalability, and support for dynamic scenes. Future work also includes extending inputs and evaluating test-time compute scaling.
- MapAnything does not explicitly account for noise or uncertainty in geometric inputs.
- The architecture does not currently support tasks where some input views lack images, such as novel view synthesis with camera-only target views.
- Multimodal features are fused before transformer input, leaving more efficient direct modality integration as an open direction.
- MapAnything’s one-to-one mapping between input pixels and output scene representation limits scalability, especially for large scenes.
- The current scene parameterization does not capture dynamic motion or scene flow.
6. Conclusion
MapAnything is presented as a universal transformer backbone that directly predicts metric 3D geometry and cameras from flexible inputs. It unifies multiple reconstruction tasks and matches or surpasses specialist feed-forward models while supporting efficient joint training.
- MapAnything directly regresses metric 3D geometry and camera poses from images and optional geometric inputs in one pass.
- Its factored representation combines depth maps, ray maps, poses, and a global scale factor to produce globally consistent metric reconstructions.
- The model supports uncalibrated structure-from-motion, calibrated multi-view stereo, monocular depth estimation, camera localization, and depth completion without task-specific tuning.
- Experiments show that MapAnything surpasses or matches specialist models while enabling efficient joint training.
B. Implementation Details
Training uses AdamW with separate learning rates for the pretrained encoder and remaining parameters, followed by warmup and cosine decay. A two-stage curriculum varies view counts and batch sizes across 420K steps.
- AdamW uses a peak learning rate of 5 · 10^-6 for the pretrained DINOv2 encoder and 10^-4 for all other parameters.
- The schedule applies 10% linear warmup, half-cycle cosine decay to a 100× lower value, and weight decay of 0.05.
- Inputs are resized and cropped to a maximum dimension of 518 pixels with randomized aspect ratios from 3:1 to 1:2.
- A two-stage 420K-step curriculum trains first for 6 days with 2–4 views, then for 4 days with 2–24 views and a 10× lower peak learning rate.
C. Additional Evaluation
Additional evaluations examine speed, memory, input flexibility, view-count robustness, checkpoint changes, and dense reconstruction across datasets and domains. Auxiliary geometric inputs and additional modalities improve reconstruction quality, while the model remains competitive with concurrent methods.
- Input Flexibility: Universal training supports 64 input combinations, and performance improves as more geometric modalities are provided.
- Speed & Memory Profiling: MapAnything has the best speed and memory usage profile among the profiled concurrent multi-view feed-forward reconstruction models.
- View-Count Robustness: The Apache and first-stage training variants maintain strong dense reconstruction across 2–100 input views and different configurations.
- Checkpoint Comparison: Checkpoint comparisons evaluate the January 2026 MapAnything release against the September 2025 MapAnything-Prev variant.
- Qualitative Results: Images-only reconstructions show high-fidelity results across indoor, landscape, art, object-centric, and off-road domains, including monocular and art images.
- Ablations: Ablations assess loss design and multi-view transformer attention choices as critical factors for reconstruction performance.
- Auxiliary Geometric Inputs: Providing camera calibration and poses improves alignment and metric-scale reconstruction compared with images-only inputs.
- Benchmarking: Dense multi-view reconstruction benchmarks compare MapAnything with concurrent models using both image-only and multimodal inputs across individual and averaged datasets.