Source-linked AI summary
VidSplat: Gaussian Splatting Reconstruction with Geometry-Guided Video Diffusion Priors
Jimin Tang, Wenyuan Zhang, Junsheng Zhou, Zian Huang, Kanle Shi, Shenkun Xu, Yu-Shen Liu, Zhizhong Han
TL;DR
Sparse-view Gaussian Splatting struggles with unseen or weakly constrained regions beyond the input coverage. VidSplat uses geometry-guided video diffusion, camera-trajectory exploration, and confidence-weighted iterative refinement to synthesize and integrate novel views, achieving superior sparse-view reconstruction performance across benchmarks.
Problem
Gaussian Splatting degrades with few views and struggles to infer unseen, occluded, or weakly constrained regions beyond the input coverage.
Method
VidSplat is a training-free framework that guides video diffusion with rendered RGB and masks, samples trajectories into unobserved regions, and iteratively fuses synthesized views into reconstruction.
Results
VidSplat achieves state-of-the-art performance in surface reconstruction and novel view synthesis across diverse real-world benchmarks.
Takeaways & Limitations
Geometry-guided video diffusion can support complete, high-fidelity 3D scene recovery from sparse inputs, including a single input view.
Abstract
from arXiv · showhide
Gaussian Splatting has achieved remarkable progress in multi-view surface reconstruction, yet it exhibits notable degradation when only few views are available. Although recent efforts alleviate this issue by enhancing multi-view consistency to produce plausible surfaces, they struggle to infer unseen, occluded, or weakly constrained regions beyond the input coverage. To address this limitation, we present VidSplat, a training-free generative reconstruction framework that leverages powerful video diffusion priors to iteratively synthesize novel views that compensate for missing input coverage, and thereby recover complete 3D scenes from sparse inputs. Specifically, we tackle two key challenges that enable the effective integration of generation and reconstruction. First, for 3D consistent generation, we elaborate a training-free, stage-wise denoising strategy that adaptively guides the denoising direction toward the underlying geometry using the rendered RGB and mask images. Second, to enhance the reconstruction, we develop an iterative mechanism that samples camera trajectories, explores unobserved regions, synthesizes novel views, and supplements training through confidence weighted refinement. VidSplat performs robustly to sparse input and even a single image. Extensive experiments on widely used benchmarks demonstrate our superior performance in sparse-view scene reconstruction.
1 Introduction
Sparse-view reconstruction degrades because limited observations make multi-view optimization under-constrained and leave unseen regions unresolved. VidSplat addresses this by using geometry-guided video diffusion and iterative view synthesis to expand coverage and refine reconstruction.
- Motivation: Sparse input views make NeRF and 3DGS optimization ill-posed because it relies heavily on multi-view consistency.The degradation is attributed to under-constrained optimization when few views are available.
- Limitations of prior work: Existing generalizable and scene-specific methods face generalization or scalability limitations and remain restricted to visible regions.They cannot infer geometry outside the input views’ field of view.
- VidSplat: VidSplat uses video diffusion priors to synthesize views along sampled camera trajectories, expanding sparse scene coverage.The generated clips are conditioned on camera trajectories and reference images.
- VidSplat: A training-free, stage-wise denoising strategy uses rendered RGB and masks to guide synthesis toward underlying geometry and improve 3D consistency.Higher-noise stages constrain RGB within masked regions to suppress dynamics and content drift.
- VidSplat: Visibility-based sampling, trajectory expansion, view selection, and confidence-weighted fusion iteratively integrate synthesized views into reconstruction.These techniques target under-covered regions while mitigating hallucinations during refinement.
- Results: VidSplat achieves state-of-the-art performance for surface reconstruction and novel view synthesis across diverse real-world datasets, including single-view generation.The evaluation covers indoor and outdoor scenarios.
2 Related Work
Prior work improves sparse-view reconstruction through learned cross-view correspondences, scene-specific priors, or generative supervision. VidSplat instead combines geometry-aware video diffusion with explicit camera control and iterative reconstruction to address missing coverage and geometric inconsistency.
- Sparse-view reconstruction: NeRF and 3DGS rely on photometric consistency across dense views, causing significant degradation under sparse inputs.This dependence motivates methods that provide additional supervision beyond the observed views.
- Sparse-view reconstruction: Generalizable methods pretrain on large datasets to capture cross-view patterns, whereas overfitting methods optimize individual scenes with monocular or multi-view-stereo priors.The two directions differ in whether priors are learned across scenes or fitted to a specific scene.
- Generative priors: Generative priors provide additional novel-view supervision, but controllable video models often produce dynamics and shakes that harm 3D geometry consistency.These limitations motivate explicit camera control and geometry-guided denoising.
- VidSplat: VidSplat guides video diffusion with rendered geometry and incorporates generated results into iterative reconstruction for consistent and complete scene recovery.Its design combines geometry-aware generation with reconstruction refinement.
3 Method
VidSplat iteratively expands sparse-view reconstruction by sampling visibility-aware camera trajectories, generating geometry-guided novel views with a video diffusion model, and incorporating them into point-cloud and Gaussian training. Its stage-wise denoising blends rendered geometry references with diffusion latents, while confidence-weighted refinement helps integrate synthesized views.
- Optimization Framework: VidSplat initializes points from sparse views, samples visibility-based trajectories, generates consistent video clips, and merges selected keyframes for denser Gaussian initialization.The generated views are merged with the original inputs and processed again with DUSt3R before Gaussian initialization.
- Optimization Framework: During training, iterative cycles generate views along new trajectories, merge them into the input set, and expand view coverage for Gaussian optimization.RGBs come from Gaussian rasterization, while masks are computed by ray tracing on periodically evaluated meshes.
- Visibility-Based Camera Pose Sampling: Visibility-based trajectories orbit the intersection of an input camera ray and scene surface on a sphere, retaining views with suitable unseen-region coverage and without near-plane occlusions.Trajectory eligibility is evaluated using keyframe depths and masks; excessive unseen coverage or wall occlusion causes rejection.
- Visibility-Based Camera Pose Sampling: The method extends each selected trajectory by 25%, discards tail frames prone to hallucination, and selects sharp keyframes with large pose variation for reconstruction.The extra frames are removed after generation because error accumulation often affects the sequence tail.
- Geometry-Guided Video Generation: Geometry-guided denoising inverses rendered RGB and mask images, blends reference and generated noisy latents, and uses a three-stage schedule to guide generation toward scene geometry.The schedule reflects global semantic formation at early stages and spatial-detail refinement at later stages.
- Loss Function: Confidence-weighted generated-view loss uses a per-pixel fusion confidence map and replaces photometric loss with Laplacian loss to mitigate high-frequency artifacts.The generated-view objective combines Laplacian, regularization, and normal-prior terms under the confidence map.
4.1 Experimental Settings
VidSplat is evaluated on indoor and outdoor benchmarks using sparse input views, with comparisons spanning reconstruction, novel view synthesis, and camera-controlled video generation.
- Datasets: Evaluation covers Tanks and Temples, Replica, and DL3DV, using 5, 10, and 6 input views per scene, respectively.TNT includes 6 scenes, Replica includes 8 scenes, and DL3DV includes 4 indoor and 4 outdoor scenes.
- Baselines: The study compares dense-view reconstruction, sparse-view reconstruction, and sparse-view novel view synthesis methods with generative priors.
- Novel View Synthesis: Novel view synthesis is evaluated on DL3DV with 6-view inputs, using numerical comparisons reported in Table 2.
- Video Generation: Video generation is compared against other camera-controlled video diffusion methods, with quality evaluation reported for images and videos.
4.2 Comparison Results
VidSplat outperforms baselines across sparse-view surface reconstruction and novel view synthesis, while its geometry-guided generation improves consistency and coverage of under-observed regions.
- Surface Reconstruction: VidSplat achieves significantly better performance than all baselines on TNT and Replica quantitative reconstruction results.Visual comparisons further show complete surfaces with high-quality geometric details under sparse-view inputs.
- Novel View Synthesis: VidSplat consistently achieves the best novel view synthesis results across indoor and outdoor DL3DV scenes.
- Novel View Synthesis: The method produces high-quality renderings in regions sparsely covered or not covered by the input views.
- Video Generation: VidSplat improves rendered-to-real video consistency across PSNR, SSIM, and LPIPS while increasing generative diversity across FID and FVD.The geometry-guided denoising produces results that adhere closely to the underlying geometry, unlike baselines that may leave regions missing or hallucinate content.
- Video Generation: Visual comparisons report more consistent results obeying ground-truth geometries than other camera-controlled video generation methods.
- Ablation Study: Ablations examine the initialization completion and training completion modules.
4.3 Application on Single-View Generation
VidSplat extends its iterative reconstruction framework to a single input view, recovering large regions invisible in that image for both an object and an indoor scene.
- Single-View Pipeline: Given one input view, VidSplat estimates metric depth, initializes Gaussian primitives from a backprojected point cloud, and expands orbiting trajectories during training.The trajectories progressively expand Gaussian primitives and training views until most of the scene is covered.
- Results: Single-view results on DTU and AIGC recover large regions invisible in the input image.The examples include an object from DTU and an indoor scene from AIGC.
4.4 Ablation Studies
Ablations show that VidSplat’s completion modules, stage-wise denoising, backbone compatibility, and exposure preprocessing each contribute to robust sparse-view reconstruction.
- Completion modules: Combining initialization and training completion recovers complete, high-quality surfaces, whereas either module alone leaves noticeable holes or lacks sufficient geometric priors.Initialization completion expands the initial point cloud, while training completion iteratively expands the Gaussian Splatting training views.
- Denoising strategy: Stage-wise denoising produces high-fidelity videos consistent with real sequences, unlike unguided, RGB-only, or single-stage alternatives.Without geometry guidance, outputs show content misalignment and 3D inconsistency; other variants fail to repair artifacts or fill missing regions.
- Video diffusion backbones: Geometry-guided generation remains strong with HunyuanVideo 1.0 and SVD, without requiring model-specific finetuning.The method maintains strong performance and outperforms existing methods even with earlier diffusion architectures such as SVD.
- Exposure consistency: Exposure-consistency preprocessing with BracketDiffusion improves photometric constraints and produces more harmonious lighting under inconsistent training exposures.The experiment addresses varying exposures that can disrupt photometric consistency in real-world sparse views.
5 Conclusion
VidSplat integrates geometry-guided video diffusion with Gaussian Splatting to reconstruct complete, high-fidelity scenes from limited inputs. Its training-free denoising and iterative scene-coverage expansion achieve strong geometry and rendering results, including promising single-image reconstruction.
- VidSplat integrates geometry-guided video diffusion priors with Gaussian Splatting to recover complete, high-fidelity 3D scenes from limited inputs.
- A training-free, stage-wise denoising strategy improves video-generation 3D consistency, while iterative optimization progressively expands scene coverage.
- Extensive real-world benchmark experiments show significant improvements over existing methods in geometry accuracy and rendering fidelity.
- VidSplat’s generalization ability supports promising applications such as single-image reconstruction.