MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views

1 Introduction

• Earlier differentiable-rendering methods commonly rely on per-scene optimization and hundreds or thousands of images, which can make optimization time-consuming and data collection impractical.The paper frames feed-forward synthesis as an alternative for casual users with sparse observations.Source: §1 · p. 1
• The task is to synthesize wide-sweeping or 360° views of diverse real-world scenes from very few input images, despite minimal overlap and incomplete information about the scene.The paper explores settings with fewer than five observations.Source: §1 · pp. 1–2
• Prior feed-forward work largely separated 360° synthesis of objects from scene-level synthesis at nearby viewpoints, making this a new sparse-view NVS setting.The paper describes the scene-level approaches as limited to nearby viewpoints and the 360° approaches as object-level.Source: §1 · p. 1
• MVSplat360 uses feed-forward MVSplat [10] to match and fuse views into coarse 3D geometry, then conditions Stable Video Diffusion [3] on rendered 3DGS outputs for refinement.The paper selects SVD for its temporal consistency, which aligns with the view-consistency requirement of NVS.Source: §1 · pp. 2–3
• Multichannel Gaussian feature rendering and latent-space alignment let SVD gradients bypass its frozen image encoder and improve the geometry backbone, especially on DL3DV-10K.The paper reports that this improves visual quality.Source: §1 · p. 3
• On DL3DV-10K, MVSplat360 achieves photorealistic 360° NVS with significantly better visual quality than prior scene-level feed-forward methods, and reports state-of-the-art interpolation and extrapolation performance on RealEstate10K.The DL3DV-10K benchmark uses a new training and testing split for feed-forward wide-sweeping and 360° NVS.Source: §1 · p. 3

Sparse view per-scene reconstruction and synthesis

• Per-scene reconstruction methods can use diffusion-generated pseudo-dense views, but remain slow for unseen scenes because they require per-scene optimization.The passage discusses ZeroNVS, Reconfusion, and concurrent submissions including CAT3D.Source: §2 · p. 3

Feed-forward scene reconstruction and synthesis

• Feed-forward methods such as pixelSplat [6] and MVSplat [10] synthesize from sparse views, but the passage says they fail in wide-sweeping or 360° settings; DepthSplat [61] and Long-LRM [75] require denser inputs.The cited examples of denser inputs are 12 or 32 views.Source: §2 · p. 3

Camera trajectory controllable synthesis

• Camera-controlled diffusion methods largely demonstrate 360° synthesis for single objects, while complex scenes pose challenges for precise pose control and per-scene reconstruction.The passage notes that some scene-synthesis approaches still rely on per-scene optimization.Source: §2 · p. 3

3 Methodology

• MVSplat360 targets feed-forward 360° synthesis for unseen scenes from as few as 5 sparse views, without additional per-scene training.It combines coarse multi-view geometry reconstruction with multi-frame appearance refinement.Source: §3 · p. 4

3.1 Multi-View Coarse Geometry Reconstruction

• The geometry backbone fuses cross-view features, constructs cost volumes, estimates depth, predicts 3D Gaussian parameters, and rasterizes target views.It builds cost volumes by matching features across views and depth planes.Source: §3.1 · p. 4
• Rendered Gaussian features condition SVD directly in latent space, allowing SVD loss to optimize Gaussian features while providing richer scene information.The model also selects spatially distributed inputs with farthest-point sampling and restricts cross-view attention and cost volumes to local groups.Source: §3.1 · p. 5

3.2 Multi-Frame Appearance Refinement

• The appearance module fine-tunes pre-trained SVD to refine coarse views, using CLIP embeddings for global semantics and rendered geometry features for spatial pose and visual cues.Rendered features help preserve scene texture while the multi-frame model supplies temporal consistency.Source: §3.2 · pp. 5–6
• Before pixel-aligned PSNR and SSIM evaluation, the method matches color histograms between SVD-refined views and 3DGS-rendered views.Source: §3.2 · p. 6

3.3 Training Objectives

• Training combines reconstruction, diffusion, and latent-alignment losses; SVD gradients update Gaussian features but are stopped at structural Gaussian parameters for stability.The reconstruction loss combines ℓ2 and LPIPS, and the encoder and decoder remain frozen during SVD fine-tuning.Source: §3.3 · p. 6

4.1 Experimental Details

• The DL3DV-10K benchmark tests 360° synthesis across 140 held-out scenes, using five input views and 56 sampled test views per scene.It also reports n = 300 and n = 150 trajectory-span settings; evaluation includes PSNR, SSIM, LPIPS, DISTS, and FID.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 Results on the New DL3DV-10K Benchmark

• MVSplat360 outperforms all compared methods across every metric on both DL3DV-10K trajectory-span settings, with larger gains on perceptual metrics and FID than pixel-aligned metrics.The n = 300 setting spans larger viewpoints than n = 150; Table 1 compares the methods on both settings.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 Results on the Existing RealEstate10K Benchmark

• On RealEstate10K, MVSplat360 surpasses previous state-of-the-art methods mainly on perceptual and distribution metrics and generates more plausible unobserved content than latentSplat.Qualitative comparisons also show latentSplat artifacts and failures by PixelSplat and MVSplat to render unobserved regions.Source: Table 2 · p. 8, §4.3 · p. 9, Figure 4 · p. 9

4.4 Ablations and Analysis

• Feature-rendered 3DGS conditioning lets SVD denoising gradients backpropagate through the geometry backbone, improving performance significantly over direct image conditioning.Averaging CLIP tokens from multiple context views also improves results; direct MVSplat novel-view conditioning performs slightly worse than the original MVSplat.Source: §4.4 · pp. 9–10, §4.3 · p. 9
• Although trained with five input views, MVSplat360 improves as more views are added at test time and still outperforms five-view PixelSplat and MVSplat with three views.Reducing the input-view count worsens performance.Source: §4.3 · p. 9, §4.4 · p. 10
• SfM recovers reasonable camera poses and 3D point clouds from input and rendered views, supporting the multi-view consistency and geometric correctness of MVSplat360 outputs.The figure marks input views with red borders; other views are rendered by MVSplat360.Source: §4.4 · p. 10, Figure 5 · p. 10

5 Conclusion

• MVSplat360 synthesizes 360° novel views of diverse real-world scenes from sparse input views.It is a feed-forward model.Source: §5 · p. 10
• A feed-forward 3DGS model recovers coarse geometry and appearance, then renders latent features as pose and visual cues for SVD.SVD uses these cues to generate 3D-consistent views.Source: §5 · p. 10
• On a challenging 360° NVS benchmark, MVSplat360 achieves superior visual quality to other state-of-the-art feed-forward approaches.Source: §5 · p. 10

A More Experiment Results

• 32.52 dB PSNR is achieved with 2× bilinear input upscaling, compared with 26.77 dB without upscaling.The result is reported on the test set.Source: A More Experiment Results · p. 15
• Encoding 256 × 480 inputs at the autoencoder’s pretrained resolution of 768 × 1280 helps preserve detail in the latent space.The authors upscale the input 2× before encoding.Source: A More Experiment Results · p. 15
• Fine-tuning the autoencoder could address resolution-related information loss, but risks overfitting on the roughly 2,000-scene training set.Source: A More Experiment Results · p. 15

B More Implementation Details

• The model renders 4-channel Gaussian features and resizes them to 1/4h × 1/4w to match SVD’s initial latent space.The SVD encoder downsamples inputs by a factor of 8; decoder outputs are resized back to h × w.Source: B More Implementation Details · p. 15
• Training samples 5 input views and 14 target views to align with the 14-frame SVD module.Models train for 100,000 steps with an effective batch size of 8.Source: B More Implementation Details · p. 15
• Experiments mainly use the DL3DV-10K “images_8” branch, which contains 256 × 480 images, because of resource limitations.Source: B More Implementation Details · p. 15

C Limitations and Discussions

• Oversaturated outputs limit improvements on pixel-aligned metrics such as PSNR and SSIM.The authors suggest this may reflect SVD’s training on artistic videos with vibrant colors and fine-tuning from its pretrained weights.Source: C Limitations and Discussions · pp. 15–16
• Refined views may contain hallucinated content that is absent from the input views.The figure caption attributes both oversaturation and over-hallucination to the diffusion module’s prior knowledge.Source: C Limitations and Discussions · p. 16
• Inference is slow because the diffusion process requires multiple sampling steps.Source: C Limitations and Discussions · p. 16
• The authors expect these limitations to be mitigated as better video diffusion models and pretrained weights become available.Source: C Limitations and Discussions · p. 16

• MVSplat360 can support augmented reality, immersive entertainment and media, and educational historical reconstruction.Examples include 3D videos and video games.Source: D Broader Social Impacts · p. 16
• Its generative capabilities could be misused to create fake videos.Source: D Broader Social Impacts · p. 16
• Rendered views may omit real-world details, so generated data warrants precautions in safety-critical uses such as autonomous-driving training.Source: D Broader Social Impacts · p. 16

• The appendix adds visual comparisons with existing state-of-the-art models.Source: E More Visual Comparisons · p. 16
• On DL3DV-10K complex scenes, MVSplat360 significantly outperforms all existing models in the qualitative comparisons.Source: Figure C · p. 16, Figure D · p. 17