Source-linked AI summary
MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images
Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, Jianfei Cai
TL;DR
Sparse-view feed-forward reconstruction needs efficient, accurate geometry from as few as two images, avoiding costly per-scene optimization. MVSplat uses plane-swept cost volumes to estimate depths and unproject Gaussian centers while jointly predicting other Gaussian properties. On RealEstate10K and ACID, it achieves state-of-the-art performance at 22 fps and outperforms pixelSplat in efficiency, quality, and generalization.
Problem
MVSplat addresses sparse-view 3D reconstruction and novel-view synthesis, where prior neural representations face expensive optimization, high memory cost, or slow rendering.
Method
MVSplat builds plane-swept cost volumes containing cross-view feature similarities for geometry-aware depth estimation, then unprojects depths into Gaussian centers and jointly predicts other Gaussian parameters.
Results
22 fps is MVSplat's feed-forward inference speed on RealEstate10K and ACID, with state-of-the-art performance and better appearance, geometry, and cross-dataset generalization than pixelSplat.
Takeaways & Limitations
MVSplat shows that cost-volume feature matching can support lightweight, fast feed-forward 3D Gaussian Splatting from sparse multi-view images.
Abstract
from arXiv · showhide
We introduce MVSplat, an efficient model that, given sparse multi-view images as input, predicts clean feed-forward 3D Gaussians. To accurately localize the Gaussian centers, we build a cost volume representation via plane sweeping, where the cross-view feature similarities stored in the cost volume can provide valuable geometry cues to the estimation of depth. We also learn other Gaussian primitives' parameters jointly with the Gaussian centers while only relying on photometric supervision. We demonstrate the importance of the cost volume representation in learning feed-forward Gaussians via extensive experimental evaluations. On the large-scale RealEstate10K and ACID benchmarks, MVSplat achieves state-of-the-art performance with the fastest feed-forward inference speed (22~fps). More impressively, compared to the latest state-of-the-art method pixelSplat, MVSplat uses $10\times$ fewer parameters and infers more than $2\times$ faster while providing higher appearance and geometry quality as well as better cross-dataset generalization.
1 Introduction
MVSplat targets feed-forward 3D scene reconstruction and novel-view synthesis from very sparse multi-view images. It uses cost-volume feature matching to localize Gaussian centers and achieves efficient, strong benchmark performance.
- MVSplat addresses 3D scene reconstruction and novel view synthesis from very sparse inputs, including as few as two images, in one forward pass.
- Existing neural scene representations remain impractical because they require per-scene optimization, high memory, or slow rendering.
- Cost volumes from plane-swept cross-view feature similarities provide geometry cues for localizing Gaussian centers instead of relying on direct data-driven 3D regression.
- MVSplat jointly predicts Gaussian centers and other primitives, including opacity, covariance, and color, using photometric supervision.
- 22 fps is MVSplat's feed-forward inference speed on large-scale benchmarks, while it achieves state-of-the-art performance on RealEstate10K and ACID.
- 10× fewer parameters and more than 2× faster inference than pixelSplat accompany higher appearance and geometry quality and better cross-dataset generalization.
2 Related Work
Sparse-view scene reconstruction methods seek to reduce the capture burden of dense-view approaches, spanning per-scene optimization and cross-scene feed-forward inference. MVSplat differs from prior feed-forward methods by using geometry-aware cost-volume matching for general multi-view scenes.
- Sparse-view reconstruction typically uses two or three input views, whereas original NeRF and 3DGS were designed for much denser inputs.
- Existing sparse-view methods divide broadly into per-scene optimization and cross-scene feed-forward inference approaches.
- MVSplat's pipeline extracts multi-view features, builds plane-swept cost volumes, predicts depths, unprojects them into Gaussian centers, and renders novel views.
- Unlike pixelSplat's probabilistic depth regression, MVSplat predicts depth from feature-matching information encoded in a cost volume.
3 Method
MVSplat estimates multi-view depths with 2D feature processing, plane-swept cost volumes, refinement networks, and cross-view attention. It unprojects the depths into Gaussian centers while predicting the remaining Gaussian parameters in parallel.
- The model maps posed images and camera projection matrices to pixel-aligned Gaussian position, opacity, covariance, and color parameters.
- MVSplat uses multi-view feature extraction with CNNs and Transformers, while its depth model relies only on 2D convolutions and attention.
- Plane sweeping warps features across views at sampled inverse-depth candidates and computes correlations to construct per-view cost volumes.
- Correlations across views are averaged so the model can accept an arbitrary number of input views.
- A lightweight 2D U-Net refines ambiguous cost volumes, using concatenated Transformer features and cost volumes plus cross-view attention.
- Softmax-weighted averaging over depth candidates produces per-view depth predictions, followed by lightweight depth refinement with cross-view information.
- Predicted depths are unprojected into aligned world-coordinate point clouds and combined as Gaussian centers, while opacity, covariance, and color are predicted jointly.
4 Experiments
MVSplat is evaluated on standard sparse-view benchmarks for image quality, efficiency, geometry reconstruction, cross-dataset generalization, and component importance. Across these evaluations, it delivers strong visual and geometric results while remaining fast and lightweight.
- Experimental settings: MVSplat achieves the best visual metrics and fastest running time with a lightweight model size on RealEstate10K and ACID.Table 1 averages results over thousands of test scenes, using two input views to render three novel views; running time includes encoding and rendering.
- Main results: MVSplat produces higher-quality novel views than competing models in challenging regions, including repeated patterns, partially observed objects, and distant outdoor objects.The qualitative comparisons report fewer artifacts in regions such as window frames, stair handrails, lampshades, and bridges.
- Efficiency and geometry: 10× fewer parameters and more than 2× faster inference distinguish MVSplat from pixelSplat while preserving higher appearance and geometry quality.MVSplat’s encoder runs in 0.043s versus 0.102s for pixelSplat, while rendering takes 0.0015s versus 0.0025s.
- Cross-dataset generalization: MVSplat generalizes better than pixelSplat when trained on RealEstate10K and tested zero-shot on ACID and DTU, with larger gains under larger domain gaps.Its zero-shot ACID results even slightly surpass pixelSplat trained directly on ACID.
- Ablations: Removing cost-volume components degrades performance, while U-Net refinement improves challenging-region rendering by approximately 0.7 dB PSNR.The ablation table identifies the cost-volume module as indispensable; the refinement study links U-Net refinement to improved geometry cues and detail transfer.
5 Conclusion
MVSplat is an efficient feed-forward 3D Gaussian Splatting model for sparse multi-view images, using cost volumes to exploit multi-view correspondence for geometry learning. It achieves state-of-the-art results with substantially lower computational cost than pixelSplat, but remains limited on reflective surfaces and diverse in-the-wild scenes.
- MVSplat uses a cost volume to exploit multi-view correspondence for improved geometry learning rather than relying only on data-driven design.
- MVSplat sets state-of-the-art performance on two large-scale scene-level reconstruction benchmarks.
- 10× fewer parameters and more than 2× faster inference give MVSplat a substantial efficiency advantage over pixelSplat while retaining higher appearance and geometry quality.
- Limitations and Discussions: MVSplat might produce unreliable results for reflective surfaces such as glasses and windows.
- Limitations and Discussions: Training on RealEstate10K limits robust generalization to diverse in-the-wild real-world scenarios.
A More Experimental Analysis
The supplementary experiments examine cost-volume transferability, backbone choices, Gaussian count, initialization, and ablations. They support the method’s general applicability and identify cost-volume design as fundamental, while showing efficiency–quality trade-offs and reflective-surface failures.
- Using cost volume in pixelSplat: Replacing pixelSplat’s probability-density depth branch with MVSplat’s cost-volume depth significantly boosts performance, demonstrating general applicability.
- Backbone Transformer: The Swin Transformer backbone avoids expensive epipolar sampling and is more efficient than pixelSplat’s Epipolar Transformer, with no clear rendering-quality difference.
- Backbone initialization: Random initialization still outperforms pixelSplat, but requires 450K iterations instead of 300K with UniMatch initialization.
- Gaussian numbers per pixel: Increasing the number of Gaussians per pixel improves performance but slows rendering, motivating the default choice of one Gaussian per pixel.
- Ablations: The cost volume is fundamental to the full model, while removing cross-view attention causes overfitting after certain training iterations.
- Limitations: MVSplat may be less effective on non-Lambertian and reflective surfaces, and complex real-world scenes warrant caution in safety-critical applications.
B More Visual Comparisons
Additional qualitative comparisons cover RealEstate10K and ACID scenes and extend evaluations across state-of-the-art methods. MVSplat performs best in the extended visual cases and renders challenging regions effectively.
- The extended comparisons include four RealEstate10K rows and two ACID rows, with MVSplat performing best in all cases.
C More Implementation Details
The implementation uses shallow CNN and Swin Transformer components, lightweight U-Nets, and cross-view attention for multi-view processing. Qualitative extensions compare geometry, cross-dataset generalization, and architectural details.
- Network architectures: The backbone combines a shallow six-block ResNet-like CNN with six Transformer blocks using local-window Swin attention.
- Geometry comparisons: MVSplat predicts higher-quality 3D Gaussians and depth maps than pixelSplat without extra depth-regularized fine-tuning.
- Cross-dataset generalization: Training on indoor RealEstate10K yields better generalization than pixelSplat to outdoor ACID and object-centric DTU scenes.