Source-linked AI summary
Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairs
Brandon Smart, Chuanxia Zheng, Iro Laina, Victor Adrian Prisacariu
TL;DR
Sparse uncalibrated stereo reconstruction and novel-view synthesis remain constrained by camera requirements, dense inputs, and per-scene optimization. Splatt3R extends MASt3R to predict Gaussian attributes from two uncalibrated images, using geometry supervision and masked rendering losses. It produces generalizable reconstructions that outperform MASt3R and feed-forward splatting baselines, including across wide baselines.
Problem
Existing reconstruction and novel-view-synthesis methods often require dense images, camera parameters, or per-scene optimization, limiting use with in-the-wild uncalibrated pairs.
Method
Splatt3R extends MASt3R with Gaussian attributes and predicts 3D Gaussians from two uncalibrated images, training geometry before masked novel-view rendering.
Results
Splatt3R outperforms MASt3R and current feed-forward splatting methods, reconstructing 3D appearance and geometry from uncalibrated stereo images and wide baselines.
Takeaways & Limitations
A single forward pass can generate Gaussian Splats for scene reconstruction and novel-view synthesis from a pair of unposed images without camera intrinsics, extrinsics, or depth.
Takeaways & Limitations
The representation assumes one unique surface intersection per pixel and does not model non-opaque structures such as glass or fog.
Abstract
from arXiv · showhide
In this paper, we introduce Splatt3R, a pose-free, feed-forward method for in-the-wild 3D reconstruction and novel view synthesis from stereo pairs. Given uncalibrated natural images, Splatt3R can predict 3D Gaussian Splats without requiring any camera parameters or depth information. For generalizability, we build Splatt3R upon a ``foundation'' 3D geometry reconstruction method, MASt3R, by extending it to deal with both 3D structure and appearance. Specifically, unlike the original MASt3R which reconstructs only 3D point clouds, we predict the additional Gaussian attributes required to construct a Gaussian primitive for each point. Hence, unlike other novel view synthesis methods, Splatt3R is first trained by optimizing the 3D point cloud's geometry loss, and then a novel view synthesis objective. By doing this, we avoid the local minima present in training 3D Gaussian Splats from stereo views. We also propose a novel loss masking strategy that we empirically find is critical for strong performance on extrapolated viewpoints. We train Splatt3R on the ScanNet++ dataset and demonstrate excellent generalisation to uncalibrated, in-the-wild images. Splatt3R can reconstruct scenes at 4FPS at 512 x 512 resolution, and the resultant splats can be rendered in real-time.
1. Introduction
Splatt3R addresses pose-free 3D reconstruction and novel view synthesis from sparse, uncalibrated image pairs using a single feed-forward pass. It extends MASt3R with Gaussian attributes and masked training to support reconstruction from extrapolated viewpoints.
- Sparse uncalibrated image pairs remain difficult because conventional methods rely on expensive per-scene optimization, dense image collections, or camera parameters.These limitations reduce accessibility and degrade reconstruction quality from only stereo pairs.
- Splatt3R takes two uncalibrated images and predicts pixel-aligned 3D Gaussian primitives without camera intrinsics, extrinsics, or depth.
- The method builds on MASt3R’s pixel-aligned geometry prediction and adds Gaussian attributes while explicitly supervising 3D point-cloud geometry.This design jointly addresses unknown camera poses and local minima in Gaussian-splat optimization.
- Loss masking uses frustum culling and covisibility testing so rendering losses apply only to regions feasibly reconstructable from the training views.The strategy targets novel-view extrapolation, where unseen or occluded points can make supervision counterproductive.
- Splatt3R generates photorealistic novel views from in-the-wild uncalibrated images and reduces dependence on dense inputs with precise camera poses.
2. Related Work
Related work progresses from per-scene radiance-field optimization toward generalizable feed-forward reconstruction from sparse images. Splatt3R builds on pose-free stereo reconstruction to avoid the camera assumptions that constrain existing Gaussian-splatting methods.
- NeRF and related representations achieve photo-realistic novel-view synthesis but traditionally require per-scene optimization on densely collected images.
- Generalizable novel-view-synthesis methods learn data-driven priors across scenes and increasingly operate from sparse images or stereo pairs.
- Existing feed-forward Gaussian-splatting methods place primitives along camera rays and therefore assume image intrinsics and extrinsics at test time.
- Splatt3R combines MASt3R with 3D Gaussians to handle larger stereo baselines without pre-processed cameras, unlike video-focused methods designed for small baselines.
- Traditional stereo pipelines estimate cameras and triangulate points through sequential matching and geometric procedures, while explicit-correspondence methods struggle with limited image overlap.
- DUSt3R and MASt3R predict point maps for uncalibrated stereo images with implicit correspondence, enabling reconstruction even when overlap is little or absent.
3. Method
Splatt3R extends MASt3R into a feed-forward model that predicts complete 3D Gaussian primitives from uncalibrated image pairs. Its training combines point-cloud supervision with masked novel-view rendering losses to handle geometry, appearance, and partially unseen target regions.
- Geometry supervision: Direct point-cloud supervision gives each pixel-aligned Gaussian a monotonic training path toward its correct 3D position, addressing unknown poses and Gaussian-splatting local minima.The method builds on MASt3R’s point-cloud prediction rather than relying on camera-ray parameterizations that require known intrinsics.
- Geometry representation: MASt3R predicts both image point maps in the first image’s coordinate frame, avoiding camera-pose transformations between the views.The representation also includes confidence maps for handling points with ill-defined depths.
- Model architecture: Splatt3R adds a Gaussian branch to MASt3R, predicting offsets, covariances, spherical harmonics, and opacities alongside pixel-aligned 3D points and confidences.Covariances are parameterized by rotations and scales, while Gaussian means are formed by adding predicted offsets to point positions.
- Training objective: During training, two context images reconstruct the scene while posed target images supervise rendered novel views with reconstruction losses.The method freezes the pretrained MASt3R parameters and trains the Gaussian prediction head for the rendering objective.
- Loss masking: Loss masking excludes target pixels that are outside context-view frustums, occluded, or inconsistent in reprojected depth, preventing destructive supervision.The mask retains pixels visible in at least one context image whose reprojected depth matches the target depth.
4. Experimental Results
Splatt3R is evaluated on held-out ScanNet++ scenes and in-the-wild images, showing stronger reconstruction quality than the compared baselines, efficient inference, and generalization beyond direct pixel correspondences. Ablations identify loss masking as critical for stable training and extrapolated-view performance.
- Training and evaluation setup: Splatt3R is trained on ScanNet++ scenes and evaluated on held-out scenes with varying stereo-view separation and overlap.The evaluation excludes bad frames and scenes without valid depth.
- Quantitative results: Splatt3R outperforms MASt3R point-cloud rendering and pixelSplat across all stereo baseline sizes, including pixelSplat evaluated with ground-truth poses.PixelSplat quality degrades when supervision includes information invisible to the input cameras.
- Qualitative comparisons: Qualitatively, loss masking prevents Splatt3R from guessing unseen regions, while pixelSplat predicts unseen content and MASt3R point-cloud rendering exhibits artifacts.The learned Gaussian representation marginally improves rendering quality over direct point-cloud rendering.
- In-the-wild generalization: Splatt3R generalizes from ScanNet++ training to mobile-phone images, including object-scale and large outdoor scenes with few or no direct pixel correspondences.The paper contrasts this setting with traditional correspondence-based multi-view stereo.
- Runtime: ~4 FPS at 512x512 resolution is achieved on an RTX2080ti, with lower total runtime than pixelSplat because Splatt3R avoids explicit pose estimation.Splatt3R directly predicts points and Gaussians in a shared coordinate space.
- Ablation studies: Without loss masking, Gaussian sizes grow unboundedly until rendering memory costs halt training; LPIPS and learned offsets improve reconstruction quality or metrics.Spherical harmonics reduce performance in the reported ablation, likely from overfitting the training scenes.
5. Conclusion
Splatt3R generates 3D Gaussian Splats from uncalibrated stereo images without camera parameters or depth, accurately reconstructing appearance and geometry from wide baselines.
- Splatt3R generates 3D Gaussian Splats from uncalibrated stereo images without camera intrinsics, extrinsics, or depth information.
- The method combines the MASt3R architecture for predicting Gaussian parameters with loss masking during training.
- Splatt3R accurately reconstructs both 3D appearance and geometry from wide baselines.
- Splatt3R outperforms MASt3R and the current state-of-the-art in feed-forward splatting.