MVSGaussian: Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo

1 Introduction

• Existing generalizable Gaussian methods have constraints including image-pair inputs, computationally costly Transformers, human-scene focus, ground-truth depth requirements, or object-centric scope.These limitations motivate an efficient framework for unseen general scenes.Source: §1 · p. 2
• Unseen-scene Gaussian parameters differ substantially from those learned for a scene, while splatting's many-to-many Gaussian–pixel mapping complicates generalization.Further per-scene fine-tuning can improve synthesis quality, but remains lengthy for existing generalizable NeRF and 3D-GS approaches.Source: §1 · pp. 2–3
• MVS-derived geometry reasoning creates pixel-aligned Gaussian representations, while a depth-aware volume-rendering component supplements splatting to enhance generalization.The method aggregates multi-view point features and spatial information, decodes them into Gaussian parameters, and filters noisy points using depth reprojection error to initialize per-scene optimization.Source: §1 · pp. 3–4
• Compared with vanilla 3D-GS, MVSGaussian achieves better novel-view synthesis at similar 300+ FPS rendering speed and with 13.3× less training computational cost (45s).Across DTU, Real Forward-facing, NeRF Synthetic, and Tanks and Temples, it outperforms other generalizable methods; after short per-scene optimization, it is comparable to or better than methods optimized longer.Source: §1 · p. 3
• The paper presents MVSGaussian as a fast generalizable Gaussian Splatting method for novel-view synthesis in unseen scenes.Its contributions include an MVS-derived pixel-aligned representation, hybrid Gaussian rendering, and consistent point-cloud aggregation for per-scene optimization.Source: §1 · p. 4

Multi-View Stereo

• MVS methods evolved from handcrafted features to learned cost-volume pipelines that aggregate multi-view information into geometry-aware representations; MVSGaussian derives its representation from this approach.Prior work improves cost-volume methods through memory reduction, cost aggregation, feature representation, and decoding strategies.Source: §2 · p. 4

Generalizable NeRF

• Generalizable NeRF methods encode 3D-point features and decode density and radiance, but performance, optimization speed, and rendering speed remain limited.Feature approaches include appearance, aggregated multi-view, cost-volume, and correspondence-matching features.Source: §2 · p. 4

Generalizable Gaussian Splatting

• Generalizable Gaussian Splatting predicts Gaussian parameters feed-forward, with PixelSplat using an epipolar Transformer but incurring significant computational costs.3D-GS itself uses anisotropic Gaussians and differentiable rasterization for real-time rendering, but typically overfits scenes.Source: §2 · p. 4
• Prior approaches also include GPS-Gaussian, focused on human synthesis and requiring ground-truth depth, and Spatter Image, focused on object-centric reconstruction.The approaches are constrained by inefficiency, object-reconstruction focus, or dependence on image pairs or a single view.Source: §2 · p. 5

3 Preliminary

• A 3D scene is represented as a mixture of anisotropic Gaussians, each defined by a mean μ and covariance matrix Σ.Each Gaussian's value at a point X is expressed by an exponential quadratic form using the mean and inverse covariance.Source: §3 · p. 5
• For rendering, each Gaussian's covariance is transformed into image space, and pixel colors are computed by point-based alpha blending.The covariance is parameterized through scaling and rotation matrices to support optimization; the view transformation and projection Jacobian map it to 2D.Source: §3 · p. 5
• Gradient descent updates Gaussian position, scale, rotation, color, and opacity, while adaptive density control splits, clones, or prunes Gaussians.The operations respond to scale thresholds or to opacity and scale values outside the desired range.Source: §3 · p. 5

4.1 Overview

• The coarse-to-fine framework propagates depth maps and rendered views, averaging splatting and depth-aware volume-rendering outputs at each stage.A single depth-aware sample per ray supports the volume-rendering branch.Source: §4.1 · p. 6

4.2 MVS-based Gaussian Splatting Representation

• MVS estimates depth from a variance-based cost volume, and pooled multi-view features plus UNet spatial enhancement are decoded into pixel-aligned Gaussian parameters.Gaussian positions are unprojected from pixels using estimated depth; scaling, rotation, opacity, and color are decoded from features.Source: §4.2 · p. 7

4.3 Consistent Aggregation for Per-Scene Optimization

• Multi-view reprojection consistency filters unreliable depths before point clouds are concatenated as the initialization for per-scene optimization.The consistency check uses reprojection position and relative depth errors, with thresholds that increase as the number of views increases.Source: Figure 3 · p. 8, §4.3 · p. 9

4.4 Full Objective

• Generalizable training sums stage-wise MSE, SSIM, and perceptual losses, while per-scene optimization uses a weighted L1 and D-SSIM loss.The generalizable model is trained end-to-end using RGB images as supervision.Source: §4.4 · pp. 9–10

5.1 Settings

• The evaluation trains on DTU and tests on its held-out set plus three additional datasets, comparing against generalizable NeRF and Gaussian methods.For each test scene, 16 nearby views form the working set and 4 are held out for testing; Table 2 reports generalization results across three additional datasets, with starred entries evaluated at 512 × 512 resolution.Source: §5.1 · p. 10, Table 2 · p. 11, Table 3 · p. 11

5.2 Generalization Results

• MVSGaussian produces high-quality generalized views with more scene details and fewer artifacts, while achieving optimal performance at fast inference speed.The evaluation covers DTU, Real Forward-facing, NeRF Synthetic, and Tanks and Temples; Fig. 4 compares rendering quality under generalization and 3-view settings.Source: §5.2 · p. 11, Figure 4 · p. 12

5.3 Per-Scene Optimization Results

• On Real Forward-facing, MVSGaussian achieves superior performance after 45 seconds of optimization, versus 10 minutes for 3D-GS and 10 hours for NeRF.After per-scene optimization, inference speed is comparable to 3D-GS and significantly exceeds NeRF-based methods.Source: §5.3 · pp. 12–13, Figure 5 · p. 13

5.4 Ablations and Analysis

• Hybrid Gaussian rendering enhances performance over splatting or volume rendering alone, and consistency checking improves aggregation by filtering noisy points while preserving valid ones.The ablations also report that decoding RGB directly rather than spherical-harmonic coefficients avoids a generalization degradation, especially on NeRF Synthetic.Source: Table 4 · p. 12, §5.4 · pp. 13–14, Table 5 · p. 14

Approach

• MVS depth estimation establishes a pixel-aligned Gaussian representation, and depth-aware volume rendering is added to enhance generalization.The paper also reports high-quality initialization for fast scene-specific fine-tuning.Source: §6 · p. 14

Results

• Compared with generalizable NeRFs, MVSGaussian achieves real-time rendering with superior synthesis quality and, compared with 3D-GS, better view synthesis with reduced training time.The conclusion also states that experiments show state-of-the-art performance, generalizability, real-time rendering speed, and fast per-scene optimization.Source: §6 · p. 14

Limitations

• Weak textures and specular reflections can reduce MVS depth accuracy and degrade view quality.Source: §6 · p. 14

• The fragment specifies iterating i from 1 through N.Source: §12 · p. 18

Implementation Details

• Training uses 88 DTU scenes and four RTX 3090 GPUs, with 2, 3, or 4 source views selected at probabilities 0.1, 0.8, and 0.1.Evaluation uses DTU masks and the central 80% of Real Forward-facing and Tanks and Temples images; the appendix also describes COLMAP point-cloud initialization for 3D-GS.Source: A Implementation and Network Details · pp. 18–19

Additional Evaluations

• The appendix lists evaluations varying input-view count, sampled points per pixel, and density prediction.Table 6 reports Mem and FPS at 512 × 640 resolution; Table 7 varies the number of 3D points sampled along each pixel ray.Source: Table 6 · p. 19, Table 7 · p. 20, Table 8 · p. 20

Network Details

• A pooling network aggregates multi-view features by computing their mean and variance, concatenating these with each feature, and applying MLP-generated weights in a soft-argmax blend.Source: A Implementation and Network Details · p. 19

Numbers of Views; Numbers of Sampled Points; Density for Volume Rendering

• More input views improve performance with only slight added computational and memory cost; increasing sampled points improves performance but raises cost, so the model uses M = 1.With one sampled point per ray, predicting density is necessary because it indicates opacity, as supported by the ablation results.Source: B Additional Ablation Experiments · p. 20

Initialization Comparison

• ET-MVSNet surpasses SfM on Tanks and Temples, but the authors describe it as limited because it targets depth accuracy rather than view-synthesis point clouds.The paper notes that depth and view quality are not directly proportional.Source: B Additional Ablation Experiments · p. 21, Table 9 · p. 21

Depth Analysis

• MVS-based geometry reasoning yields the most accurate depth estimation among the compared generalizable NeRF methods, with visualizations across datasets.The quantitative comparison is on the DTU test set; Abs err is average absolute error, and Acc(X) is the percentage of pixels below X mm error.Source: B Additional Ablation Experiments · p. 21, Table 10 · p. 21

Point Cloud Analysis

• Consistency checking filters noisy points while retaining effective ones; concatenation creates oversized initialization clouds, while downsampling also removes effective points.The comparison uses the Real Forward-facing dataset, voxel size 2 for downsampling, and 2.5k iterations for all strategies.Source: B Additional Ablation Experiments · p. 21, Table 11 · p. 22

Inference Speed Analysis

• Discarding the neural network during per-scene optimization significantly increases speed because it is the primary inference-time overhead, while rendering adds minimal overhead.The generalizable model runs at 21.5 FPS.Source: B Additional Ablation Experiments · p. 21, Table 12 · p. 22

Qualitative Results under the Generalization Setting

• Under generalization, the method reconstructs more detail with fewer artifacts in geometrically complex scenes, around object edges, and in reflective areas.The comparison is against state-of-the-art methods.Source: C More Qualitative Results · p. 22

Qualitative Results under the Per-scene Optimization Setting

• After a short fine-tuning period, the method preserves more scene detail and exhibits fewer artifacts, which the authors attribute to its strong initialization.This is a qualitative comparison after per-scene optimization.Source: C More Qualitative Results · p. 22

Per-scene quantitative breakdown

• Per-scene results on DTU, NeRF Synthetic, Real Forward-facing, and Tanks and Temples align with the averaged main-text results.The table captions specify differing input-view and resolution conditions for PixelSplat∗ and Ours∗ versus other generalizable results.Source: D Per-scene Breakdown · p. 22, Table 13 · p. 23, Table 14 · p. 24

Depth maps

• Fig. 8 visualizes depth maps predicted by the method across different datasets.The caption names datasets [1, 28, 29].Source: Figure 8 · p. 23

Qualitative comparisons

• Fig. 9 compares generalization rendering quality with state-of-the-art methods under three-view settings.The figure caption identifies the compared methods as [6, 9, 22].Source: Figure 9 · p. 24

Point-cloud aggregation

• Fig. 10 visualizes point clouds produced by different aggregation strategies.Source: Figure 10 · p. 25

Qualitative comparisons

• Fig. 11 compares rendering quality with state-of-the-art methods after per-scene optimization.The figure caption identifies the compared methods as [19, 22].Source: Figure 11 · p. 26