Source-linked AI summary
Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass
Jianing Yang, Alexander Sax, Kevin J. Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, Matt Feiszli
TL;DR
Multi-view reconstruction is constrained by methods that process images pairwise and require costly alignment as view counts grow. Fast3R uses a Transformer to reconstruct N images in one forward pass, achieving scalable, competitive reconstruction and pose-estimation performance while supporting very large view sets. The method remains subject to drifting in extreme large-area scenes when view counts become very high.
Problem
Pairwise reconstruction and global alignment make multi-view processing computationally expensive, limit context, and scale poorly as the number of images grows.
Method
Fast3R predicts pointmaps for unordered, unposed image sets with a Transformer-based architecture that reconstructs N images simultaneously in one forward pass.
Results
Fast3R achieves competitive 3D reconstruction and camera pose estimation results, operates at > 250 FPS, and processes 1500 images in one forward pass.
Takeaways & Limitations
Fast3R provides a scalable alternative for efficient multi-view 3D reconstruction and can be fine-tuned for video reconstruction without changing its pointmap objective or architecture.
Takeaways & Limitations
For very large reconstruction areas with more than 300 views, some low-confidence views can exhibit drifting point maps.
Abstract
from arXiv · showhide
Multi-view 3D reconstruction remains a core challenge in computer vision, particularly in applications requiring accurate and scalable representations across diverse perspectives. Current leading methods such as DUSt3R employ a fundamentally pairwise approach, processing images in pairs and necessitating costly global alignment procedures to reconstruct from multiple views. In this work, we propose Fast 3D Reconstruction (Fast3R), a novel multi-view generalization to DUSt3R that achieves efficient and scalable 3D reconstruction by processing many views in parallel. Fast3R's Transformer-based architecture forwards N images in a single forward pass, bypassing the need for iterative alignment. Through extensive experiments on camera pose estimation and 3D reconstruction, Fast3R demonstrates state-of-the-art performance, with significant improvements in inference speed and reduced error accumulation. These results establish Fast3R as a robust alternative for multi-view applications, offering enhanced scalability without compromising reconstruction accuracy.
1. Introduction
Fast3R addresses the scalability limits of pairwise multi-view reconstruction by processing N images in parallel through a Transformer-based architecture. It removes global postprocessing and reports improved scalability, speed, and pose-estimation performance.
- Traditional SfM and MVS pipelines rely on pairwise stages for feature extraction, correspondence matching, triangulation, and global alignment, limiting scalability and speed.
- DUSt3R extends end-to-end reconstruction by predicting pointmaps, but handling more than two images requires O(N^2) pairwise computations and global alignment.This process can reach out-of-memory failure with 48 views on an A100.
- Pairwise processing restricts the model’s context and creates a bottleneck shared by DUSt3R, SfM, and MVS methods.
- Fast3R processes N images in one Transformer forward pass, allowing every frame to attend to all other frames and reducing error accumulation.The architecture is designed to eliminate sequential or pairwise processing.
- Fast3R eliminates global postprocessing and improves speed, computation overhead, and scalability while performance improves with larger view sets.It also generalizes to more views at inference than were used during training.
- 99.7% accuracy within 15-degrees on CO3Dv2 represents over a 14x error reduction compared to DUSt3R with global alignment.
2. Related Work
Fast3R builds on end-to-end pointmap regression while removing the pairwise and sequential constraints retained by prior multi-view methods. Its all-to-all attention enables joint reasoning and parallel inference over unordered image sets.
- Traditional multiview geometry sequentially extracts features, finds pairwise correspondences, triangulates 3D points, estimates relative poses, and performs global bundle alignment.
- COLMAP and OrbSLAM2 retain high accuracy in non-catastrophic cases but catastrophically fail more than 40% of the time on static ETH-3D scenes.
- End-to-end learned methods increasingly replace multiview-geometry components to improve robustness and speed, while DUSt3R replaces the pipeline through pairwise pointmap regression.
- MASt3R and MonST3R remain fundamentally pairwise, while MASt3R leaves global alignment unchanged.
- Fast3R’s architecture supports bidirectional information flow and simultaneous processing of dense input views.
- Unlike Spann3R’s incremental pairwise sliding-window reconstruction, Fast3R uses all-to-all attention to reason jointly over unordered frames and parallelize inference.Spann3R cannot revise earlier-frame reconstructions, allowing errors to accumulate.
3. Model
Fast3R is a Transformer-based model that predicts local and global pointmaps from unordered, unposed image sets, with simultaneous outputs and scalability to more than 1000 images during inference. Its architecture combines independent image encoding, all-to-all fusion, pointmap decoding, and masked-image training to support single-pass multi-view reconstruction.
- Problem definition: Fast3R predicts local and global pointmaps, along with confidence maps, from N unordered and unposed RGB images.The global pointmap uses the first camera’s coordinate frame, while the local pointmap uses each viewing camera’s frame.
- Training Objective: Fast3R trains local and global pointmap predictions with confidence-weighted normalized 3D pointwise regression losses combined into a total loss.Predictions and targets are independently normalized by their mean Euclidean distance to the origin.
- Image Encoder: The model independently encodes each image into patch features before concatenating them for fusion.Fast3R uses a feature extractor per image; CroCo ViT is the default encoder, while DINOv2 was found to work similarly.
- Image Encoder: Index embeddings identify each image’s patches and define the first-image reference needed for joint camera-pose reasoning.These embeddings let the fusion transformer distinguish patches from the same image while preserving permutation-invariant input handling.
- Fusion Transformer: A 24-layer fusion Transformer performs all-to-all attention over concatenated patches from all views, enabling simultaneous prediction rather than sequential processing.All output pointmaps and confidence maps are predicted simultaneously, and the model makes no assumptions about image ordering.
- Scaling: Uniformly sampling N images from a larger pool during training enables inference with N = 1000 images despite training with N = 20 images.The sampling strategy behaves like image masking and controls the masking ratio through the larger pool size N′.
4. Experiments
Fast3R is evaluated for camera pose estimation, 3D reconstruction, and inference scalability against DUSt3R and other baselines. It supports large view counts in one pass, improves with additional views, and remains competitive while substantially reducing inference cost.
- Inference efficiency: Fast3R processes up to 1500 views in a single pass, whereas DUSt3R runs out of memory past 32 views.Table 2 compares system performance on a single A100 using 512x384 views.
- Camera pose estimation: Fast3R estimates camera rotation and translation with RANSAC-PnP from high-confidence global pointmap points and selects focal length by minimizing outliers.Only the top 15% confidence points are used for efficient PnP processing.
- Camera pose estimation: 320× faster than DUSt3R and 1000× faster than MASt3R on CO3D while surpassing other methods on RRA and mAA.Fast3R remains competitive on RTA and achieves near-perfect RRA.
- Scaling views: Camera-pose orientation can saturate with only 3-5 views, while reconstruction quality improves with more inferred views.The reported reconstruction metrics are Accuracy and Completion, where lower is better.
- Scaling views: Fast3R’s per-view predictions improve as more views are used during inference across evaluated camera-pose and reconstruction metrics.The model uses additional multi-image context, with 50 images yielding better per-view accuracy than 20 despite training with 20.
- Evaluation setup: Training and evaluation cover CO3D, 7-Scenes, Neural RGB-D, and DTU, using comparisons with DUSt3R, MASt3R, and other task-specific methods.The training mixture includes CO3D, ScanNet++, ARKitScenes, Habitat, BlendedMVS, and MegaDepth.
5. Ablation Studies
The ablations examine how view count, index-embedding sampling, model scale, and local-versus-global pointmap heads affect Fast3R. Results consistently support broader context and the local head for reconstruction accuracy.
- 5.1. Scaling the number of views: Training on increasingly more views consistently improves camera-pose and reconstruction accuracy, even when evaluation uses fewer views than training.The improvement holds with the evaluation view count fixed.
- 5.1. Scaling the number of views: Fast3R’s per-view performance improves when inference uses more views, including 50 images outperforming 20 after training with 20.The behavior holds across all evaluated metrics in camera pose estimation and reconstruction.
- 5.1. Scaling the number of views: Increasing model size and data consistently boosts performance, indicating that the Transformer architecture scales with additional computational investment.The full experiments are reported in the appendix.
- Index positional embeddings: Without randomized image-index embeddings, regression loss spikes when testing beyond the training view range; with the strategy, four-view training remains high-quality through slots 5 to 24.The embedding strategy also performs comparably with six times the training view count.
- Local versus global pointmaps: The local pointmap head produces more accurate reconstruction pointmaps than the global head, with fewer floaters, less smearing, and less distortion.Evaluation aligns local pointmaps to the global coordinate system using ICP.
6. Conclusion
Fast3R replaces pairwise reconstruction with a Transformer that predicts 3D locations for all pixels in a common frame in one forward pass. It scales to many images while retaining competitive benchmark performance, but very large view collections can cause drifting in low-confidence views.
- Fast3R predicts 3D locations for all pixels in a common frame of reference directly in a single forward pass.
- Global attention removes the image-pair and pairwise-optimization bottlenecks that restrict context in existing reconstruction systems.
- > 250 FPS and 1500 images in one forward pass demonstrate Fast3R’s computational scalability with competitive 3D reconstruction and camera pose results.
- For scenes spanning very large reconstruction areas, views beyond roughly 300 can exhibit drifting point maps, especially when confidence is low.Dropping low-confidence frames usually does not substantially harm dense reconstruction quality; future work may improve data coverage or positional embeddings.
A. Model Scaling Effect
Scaling the Fusion Transformer from ViT-base through ViT-large to ViT-huge continually benefits camera pose estimation and 3D reconstruction.
- Larger Fusion Transformer models continually improve camera pose estimation and 3D reconstruction performance.The experiment compares ViT-base, ViT-large, and ViT-huge; the main experiments use ViT-base.
B. Data Scaling Effect
Fast3R continually benefits from increased training data, while its outputs also show potential for downstream novel view synthesis.
- Fast3R continually benefits from training on larger data scales, from 12.5% through 100%.All models use the same training settings except for the amount of accessible data.
- Fast3R outputs are qualitatively demonstrated as inputs for Gaussian Splatting in a downstream novel view synthesis pipeline.The visualization uses the InstantSplat pipeline.
D. Bundle Adjustment (via Gaussian Splatting)
Bundle adjustment through Gaussian Splatting can further refine Fast3R reconstructions by optimizing Gaussian locations and poses to reduce reprojection error. On one representative scene, it substantially reduces pose and translation errors.
- Gaussian Splatting bundle adjustment optimizes per-scene Gaussians, locations, and poses using Fast3R point-cloud initialization to minimize reprojection error.
- Over 2.5x lower translation error and 4x lower rotational error were observed after bundle adjustment on the Tanks and Temples “Family” scene.The comparison uses ground-truth trajectories from COLMAP.
- Table 6 presents bundle adjustment as an additional pose-estimation improvement using InstantSplat on the “Family” scene.
- Fast3R and DUSt3R perform on par for multi-view depth and both significantly outperform COLMAP-DENSE.
E. Multi-view Depth Evaluation
Fast3R is evaluated for multi-view depth estimation against DUSt3R and COLMAP, with additional visual evidence on indoor scenes. The visualizations show learned room regularities and loop-closure behavior.
- Fast3R is compared with DUSt3R and COLMAP on multi-view depth estimation tasks using its local pointmap prediction.
- Fast3R learns square-like indoor-room regularities and demonstrates loop closure capabilities in qualitative visualizations.
F.1. 4D Reconstruction: Qualitative Results
Fast3R produces qualitatively reasonable 4D reconstructions on dynamic scenes after finetuning a static-views checkpoint. The broader results include scalability, pose refinement, and qualitative reconstruction evidence across unseen poses and diverse scenes.
- F.1. 4D Reconstruction: Qualitative Results: Fast3R produces qualitatively reasonable reconstructions for dynamic scenes with minimal changes after finetuning on dynamic-scene data.The finetuning uses PointOdyssey and TartanAir data while freezing the ViT encoder.
- F.1. 4D Reconstruction: Qualitative Results: Unseen-pose visualizations show large regions without Gaussians when poses differ substantially from those used for reconstruction.The scene is fit from 7 CO3D images.
- F.1. 4D Reconstruction: Qualitative Results: Fast3R remains significantly faster than the pairwise dynamic-scene alternative described, supporting potential real-time applications.
- F.1. 4D Reconstruction: Qualitative Results: Global pointmaps provide location anchors, while local pointmaps aligned to them with ICP provide more accurate point locations.
- F.1. 4D Reconstruction: Qualitative Results: Fast3R’s 4D reconstruction results are obtained with one forward pass, with tracks visualized using ground-truth TAP-Vid-DAVIS annotations.