PaperlayerView exact revision on arXiv

Source-linked AI summary

MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images

Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, Jianfei Cai

arXiv:2403.14627v2cs.CV

TL;DR

Reconstructing 3D scenes and synthesizing novel views from very sparse images in one pass remains a target, while neural scene representations can entail costly per-scene optimization, high memory use, or slow rendering. MVSplat estimates multi-view-consistent depths using a plane-swept cost volume and predicts 3D Gaussian properties. On RealEstate10K and ACID, it achieves state-of-the-art performance and, versus pixelSplat, uses fewer parameters and runs faster while providing higher appearance and geometry quality and better cross-dataset generalization.

  • Problem

    Reconstructing 3D scenes and synthesizing novel views from as few as two images in one forward pass remains a target, while prior neural scene representations can require costly per-scene optimization, high memory, or slow rendering.

    Source: §1 · pp. 1–2

  • Method

    MVSplat plane-sweeps cross-view feature similarities across depth candidates to estimate multi-view-consistent depths and predict Gaussian centers and other properties under photometric supervision.

    Source: §1 · p. 2

  • Results

    On RealEstate10K and ACID, MVSplat achieves state-of-the-art performance at 22 fps and, versus pixelSplat, uses 10× fewer parameters, runs more than 2× faster, and provides higher appearance and geometry quality and better cross-dataset generalization.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The reported ablations and analysis underscore feature-matching cost volumes as significant to highly efficient feed-forward 3D Gaussian Splatting models.

    Source: §1 · p. 3

Abstract

from arXiv · show

We introduce MVSplat, an efficient model that, given sparse multi-view images as input, predicts clean feed-forward 3D Gaussians. To accurately localize the Gaussian centers, we build a cost volume representation via plane sweeping, where the cross-view feature similarities stored in the cost volume can provide valuable geometry cues to the estimation of depth. We also learn other Gaussian primitives' parameters jointly with the Gaussian centers while only relying on photometric supervision. We demonstrate the importance of the cost volume representation in learning feed-forward Gaussians via extensive experimental evaluations. On the large-scale RealEstate10K and ACID benchmarks, MVSplat achieves state-of-the-art performance with the fastest feed-forward inference speed (22~fps). More impressively, compared to the latest state-of-the-art method pixelSplat, MVSplat uses $10\times$ fewer parameters and infers more than $2\times$ faster while providing higher appearance and geometry quality as well as better cross-dataset generalization.

1 Introduction

MVSplat targets 3D reconstruction and novel-view synthesis from as few as two images in one forward pass, using plane-swept cost volumes to localize Gaussian centers. It predicts the remaining Gaussian properties jointly and achieves strong benchmark results with fast inference.

1 Introduction

  • The task is reconstructing 3D scenes and synthesizing novel views from very sparse inputs, as few as two images, in one forward pass.3D Gaussian Splatting offers fast rasterization-based rendering without NeRF’s expensive volumetric sampling.Source: §1 · pp. 1–2
  • Earlier neural scene methods incur per-scene optimization, memory, or rendering costs, while Splatter Image targets single-view objects and pixelSplat exhibits noisy geometry without slow depth finetuning.The paper identifies single-image scene reconstruction as ill-posed and notes that pixelSplat requires additional depth regularization for improved geometry.Source: §1 · p. 2
  • Plane-swept cost volumes store cross-view feature similarities across candidate depths, framing Gaussian-center localization as feature matching rather than direct 3D regression from image features.High similarity provides a geometry cue for likely surface points, reducing learning difficulty and supporting a lightweight, fast model.Source: §1 · p. 2
  • A 2D network estimates multi-view-consistent depths to unproject Gaussian centers, while jointly predicting covariance, opacity, and spherical harmonics coefficients under end-to-end photometric supervision.Novel views are rendered from the predicted Gaussians using differentiable splatting.Source: §1 · p. 2
  • On RealEstate10K and ACID, MVSplat achieves state-of-the-art performance at 22 fps; versus pixelSplat, it uses 10× fewer parameters, runs more than 2× faster, and provides higher quality and cross-dataset generalization.The reported quality advantages cover both appearance and geometry; ablations also underscore the importance of the feature-matching cost volume.Source: §1 · pp. 2–3

2 Related Work

Prior work on sparse-view reconstruction includes per-scene optimization and feed-forward inference, with MVSplat positioned against feed-forward NeRF, feed-forward 3DGS, and multi-view stereo approaches. It uses multi-view cost-volume matching for depth and combines unprojected per-view predictions into a 3D Gaussian representation without ground-truth geometry supervision.

Sparse View Scene Reconstruction and Synthesis

  • Sparse-view reconstruction methods use either per-scene optimization, which is slow at inference, or feed-forward inference, which uses learned priors for faster reconstruction and view synthesis.Both original NeRF and 3DGS methods were designed for much denser inputs, such as 100 views, whereas sparse-view methods use as few as two or three.Source: §2 · p. 3

Feed-Forward NeRF

  • Feed-forward NeRF methods improved with feature matching, Transformers, and 3D volumes, but incur expensive per-pixel volume sampling; MuRF also constructs a target-view volume for every view.MuRF uses a (2+1)D CNN and is expensive to train, with comparatively slow rendering.Source: §2 · p. 3

Feed-Forward 3DGS

  • Unlike Splatter Image and Flash3D, which are limited by single-image information, MVSplat aggregates multiple views; unlike GPS-Gaussian, it handles general posed views and needs no ground-truth depth.Splatter Image focuses on object-level reconstruction, while GPS-Gaussian targets humans, uses rectified stereo images, and requires ground-truth depth during training.Source: §2 · pp. 3–4
  • MVSplat predicts depth from cost-volume feature matching, unlike pixelSplat, and is reported to use 10× fewer parameters, run more than 2× faster, and produce significantly better geometry.pixelSplat predicts probabilistic depth distributions from epipolar-Transformer features, which the paper characterizes as ambiguous and unreliable for geometry reconstruction.Source: §2 · p. 4

Multi-View Stereo

  • MVSplat differs from typical multi-view stereo pipelines by forming a global 3D representation through the union of unprojected per-view depth predictions, without ground-truth geometry supervision.The paper describes the model as fully differentiable and more scalable and suitable for in-the-wild scenarios than MVS networks mostly trained with ground-truth depth.Source: §2 · p. 4

3 Method

MVSplat maps sparse multi-view images and camera parameters to pixel-aligned 3D Gaussians. It estimates their centers from multi-view depth, predicts the remaining Gaussian parameters, and trains through rendered-image supervision.

3 Method

  • Unlike pixelSplat, MVSplat unprojects predicted depths into Gaussian centers while predicting opacity, covariance, and color in parallel; its feed-forward network outputs H × W × K pixel-aligned Gaussians.The Gaussian parameters are position, opacity, covariance, and spherical-harmonic color.Source: §3 · p. 5

3.1 Multi-View Depth Estimation

  • A 2D-convolution-and-attention depth model builds per-view plane-sweep cost volumes from cross-view feature correlations, then refines volumes and depths with lightweight U-Nets and cross-view attention.The design supports arbitrary numbers of input views and avoids 3D convolutions.Source: §3.1 · pp. 5–6

3.2 Gaussian Parameters Prediction

  • Predicted depths are unprojected and aligned in world coordinates as Gaussian centers; matching confidence informs opacity, while image features, refined costs, and images inform covariance and spherical-harmonic color.Covariance is parameterized by scale and rotation, with rotation represented by quaternions.Source: §3.2 · p. 7

3.3 Training Loss

  • Training renders the predicted Gaussians at novel viewpoints and combines target-RGB ℓ2 and LPIPS losses with weights 1 and 0.05, respectively.Ground-truth target RGB images provide the supervision.Source: §3.3 · p. 7

4 Experiments

On RealEstate10K and ACID, MVSplat achieves the strongest reported visual quality and fastest inference among the compared methods. The experiments also assess its efficiency, geometry reconstruction, cross-dataset and multi-view performance, and component ablations.

4.1 Settings

  • Experiments assess RealEstate10K and ACID, evaluating three target novel viewpoints per test scene; models are also directly evaluated on 16 DTU validation scenes, with four novel views per scene.Evaluation uses 256 × 256 resolution and reports PSNR, SSIM, LPIPS, inference time, and model parameters. MVSplat uses six stacked self- and cross-attention layers and 128 depth candidates; all models are trained on one A100 GPU for 300,000 iterations with Adam.Source: §4.1 · pp. 7–8

4.2 Main Results

  • MVSplat performs best on all reported visual metrics and runs fastest among the compared methods, with a lightweight model size.The comparisons average results over thousands of scenes; each model takes two input views and renders three novel views.Source: Table 1 · p. 8, §4.2 · pp. 8–9
  • MVSplat uses 10× fewer parameters and runs more than 2× faster than pixelSplat.Its encoder takes 0.043s versus pixelSplat’s 0.102s, while rendering takes 0.0015s versus 0.0025s; MVSplat predicts one Gaussian per pixel rather than three.Source: §4.2 · p. 10
  • MVSplat produces higher-quality feed-forward 3D Gaussians than pixelSplat without extra fine-tuning, using only photometric supervision.pixelSplat requires 50,000 extra fine-tuning steps with depth regularization; its predicted geometry contains floating Gaussians, unlike MVSplat’s smoother depth and higher-quality geometry.Source: §4.2 · p. 10, Figure 4 · p. 10
  • MVSplat generalizes better than pixelSplat [1] from RealEstate10K to ACID and DTU without fine-tuning, with a larger LPIPS gain on the more distant DTU domain.Its zero-shot ACID results slightly surpass pixelSplat’s ACID-trained model; the proposed explanation is that cost-volume feature similarities are less sensitive to absolute feature scale.Source: Table 2 · p. 11, §4.2 · pp. 11–12, Figure 5 · p. 11
  • With three DTU context views, the RealEstate10K-trained MVSplat scores PSNR 14.30, SSIM 0.508, and LPIPS 0.371, versus pixelSplat’s PSNR 12.52, SSIM 0.367, and LPIPS 0.585.MVSplat improves with more input views, whereas pixelSplat performs slightly worse; the paper attributes this to pixelSplat’s sensitivity to changes in feature distribution.Source: §4.2 · p. 12

4.3 Ablations

  • Removing the cost volume from the base model decreases PSNR by more than 3dB and increases LPIPS by 0.064; removing cross-view attention drops PSNR by 1dB.The cost-volume ablation shows misaligned Gaussians, while the cross-view-attention ablation has higher error intensity and overfits, underscoring the importance of geometry cues and information flow between views.Source: Table 3 · p. 12, §4.3 · p. 13, Figure 6 · p. 13
  • Removing the cost-volume refinement U-Net lowers PSNR by about 0.7 dB, and adding depth refinement gives the best result, making “base + refine” the final model.The U-Net particularly improves regions with content in only one input view, where the initial cost volume lacks matches; depth refinement further improves depth and visual quality.Source: Table 3 · p. 12, Figure 6 · p. 13, §4.3 · p. 14

5 Conclusion

MVSplat uses a cost volume to exploit multi-view correspondence for geometry learning and achieves state-of-the-art results on two large-scale scene-level reconstruction benchmarks. Its limitations include unreliable results on reflective surfaces and insufficient training-data diversity for robust in-the-wild generalization.

5 Conclusion

  • Compared with pixelSplat, MVSplat uses 10× fewer parameters and infers more than 2× faster while providing higher appearance and geometry quality and better cross-dataset generalization.It sets new state-of-the-art on two large-scale scene-level reconstruction benchmarks.Source: §5 · p. 14
  • MVSplat constructs a cost volume to exploit multi-view correspondence for geometry learning, paired with an encoder customized for predicting 3D Gaussian primitives.The model is an efficient feed-forward 3D Gaussian Splatting approach trained using sparse multi-view images.Source: §5 · p. 14
  • MVSplat may produce unreliable results on reflective surfaces, and its RealEstate10K-only training data lacks diversity for robust in-the-wild generalization.The authors identify training on larger, more diverse datasets, potentially by mixing existing scene-level datasets, as a future direction.Source: §5 · p. 14

A More Experimental Analysis

The analysis tests MVSplat’s cost-volume generality and architectural choices, finding trade-offs in Gaussian count and initialization alongside limits on reflective surfaces and safety-critical use.

Using cost volume in pixelSplat

  • Replacing pixelSplat’s probability-density depth branch with MVSplat’s cost-volume depth prediction significantly boosts performance, demonstrating the approach’s general applicability.Other pixelSplat components remain intact.Source: A More Experimental Analysis · p. 18

Ablations on the backbone Transformer

  • The Swin Transformer avoids expensive epipolar sampling and is more efficient than the Epipolar Transformer, with no clear rendering-quality difference observed.Source: A More Experimental Analysis · p. 18

Ablations on backbone initialization

  • Random initialization still lets MVSplat outperform pixelSplat, but requires 450K iterations to reach the UniMatch-initialized model’s performance.The default UniMatch-initialized model is trained for 300K iterations.Source: Table D · p. 19, A More Experimental Analysis · p. 19

Ablations on the Gaussian numbers per pixel

Validation curves of the ablations

  • Validation curves show that the cost volume is fundamental to the full model, while removing cross-view attention leads to overfitting after some training iterations.Source: A More Experimental Analysis · p. 20

Limitation

  • MVSplat may be less effective on non-Lambertian and reflective surfaces; adding BRDF properties and more diverse training data may help address this limitation.Source: A More Experimental Analysis · p. 20

Potential negative societal impacts

  • MVSplat may produce unreliable outcomes in complex real-world scenes, so caution is warranted in safety-critical uses such as synthetic data for autonomous-vehicle training.Source: A More Experimental Analysis · p. 20

B More Visual Comparisons

This section extends qualitative comparisons of MVSplat on RealEstate10K and ACID, and compares its geometry reconstruction and cross-dataset generalization with pixelSplat.

B More Visual Comparisons

  • The section also compares MVSplat with pixelSplat on geometry reconstruction and cross-dataset generalization.These comparisons are shown in Fig. D and Fig. E, respectively.Source: B More Visual Comparisons · p. 20
  • Rendered videos and 3D Gaussian models in .ply format are available on the project page.Source: B More Visual Comparisons · p. 20
  • MVSplat performs best in all cases in extended visual comparisons on RealEstate10K and ACID.The scenes in the first four rows are from RealEstate10K, and those in the last two rows are from ACID.Source: Figure C · p. 21

C More Implementation Details

The appendix reports higher-quality MVSplat Gaussian and depth predictions than pixelSplat, stronger cross-dataset generalization, and implementation and training details for the model.

Network architectures

  • The network combines a six-block ResNet-like CNN, a six-block self- and cross-attention Transformer, and U-Net-based cost-volume and depth refinement.The CNN downsamples features by 4× overall; the Transformer uses Swin local-window attention with 2 × 2 feature splits. Cost-volume refinement uses a 128-channel U-Net with two 2× downsampling stages and attention at 4×; flattened features enable self-attention across cost volumes, inspired by. The depth-refinement U-Net uses four 2× downsampling stages and attention at 16×. The CNN has six residual blocks, and cost-volume refinement uses U-Net implementations from.Source: C More Implementation Details · pp. 21–23

More comparisons of 3D Gaussians and depth maps

  • MVSplat predicts higher-quality 3D Gaussians and depth maps than pixelSplat in the extended comparisons, without extra depth-regularized fine-tuning.Neither model receives extra depth-regularized fine-tuning in these comparisons.Source: Figure D · p. 22

More comparisons of cross-dataset generalization

  • Trained on RealEstate10K indoor scenes, MVSplat generalizes better than pixelSplat to outdoor ACID scenes and object-centric DTU scenes.The extended visual comparisons attribute this advantage to MVSplat’s cost volume-based architecture.Source: C More Implementation Details · p. 22

More training details

  • Training initializes MVSplat with UniMatch pretrained weights and uses one A100 GPU, batch size 14, and two input plus four target views per scene.Input-view frame distance increases during training, as in pixelSplat. Near/far depth planes are 1 and 100 for RealEstate10K and ACID, and 2.125 and 4.525 for DTU.Source: C More Implementation Details · p. 23

Related papers

Loading 2403.14627v2…