Source-linked AI summary
InceptionGS: Generative Bootstrapping for Large-Scale Gaussian Splatting under Unstructured View Sampling
Tianheng Lu, Guangyu Wang, Ruqi Huang, Lu Fang
TL;DR
Large-scale scene digitization must render convincingly across arbitrary views, but unstructured sampling leaves regions underobserved and challenges both reconstruction- and generation-based methods. InceptionGS adapts generative priors with scene-specific geometric guidance and view-adaptive integration to refine Gaussian splatting, with experiments demonstrating significant quantitative and qualitative improvements over both alternatives.
Problem
Unstructured view sampling leaves some large-scale scene regions insufficiently observed, while reconstruction methods degrade under view variation and generation methods face generalization and structural-consistency challenges.
Method
InceptionGS adapts a generic generative prior with scene-specific geometric guidance and softly integrates it through view-adaptive refinement of an initial 3D Gaussian splatting field.
Results
InceptionGS demonstrates significant quantitative and qualitative improvements over reconstruction-based and generation-based alternatives on challenging large-scale scene benchmarks.
Takeaways & Limitations
The approach supports immersive large-scale scene digitization under unstructured imagery without requiring additional data collection.
Takeaways & Limitations
Reconstruction quality deteriorates substantially when rendering views vary widely from the training views.
Abstract
from arXiv · showhide
Achieving truly immersive large-scale scene digitization necessitates consistent and visually pleasing rendering across all possible viewing perspectives. However, collecting multi-view images covering every fine detail of a large-scale scene is prohibitive due to scene complexity, capture cost, negligence, or accessibility constraints. As a result, the sampled views tend to be highly unstructured -- the majority of the scene is well covered yet certain regions inevitably lack sufficient observations. Existing reconstruction based methods are vulnerable to view scarcity while generation based approaches suffer from generalization, controllability, and 3D consistency issues. To address this challenge, we propose InceptionGS, which bootstraps Gaussian splatting by subtly balancing reconstruction and generation. Starting from an initial Gaussian splatting, InceptionGS reasonably rethinks and repairs problematic regions caused by view scarcity while preserving the quality elsewhere, by softly incorporating scene- and view-adaptive generative priors. Extensive experiments on real-world large-scale scenes demonstrate the superiority and broad applicability of our approach in handling unstructured imagery and boosting high-fidelity Gaussian splatting. Please refer to the supplementary video for better visual demonstrations.
1 Introduction
Large-scale scenes suffer from uneven, unstructured viewpoint coverage, while reconstruction and generation approaches each have limitations under view scarcity. InceptionGS combines both paradigms by adapting and softly injecting generative priors into Gaussian splatting, improving reconstruction fidelity and visual coherence.
- Challenge: Large-scale capture produces uneven viewpoint density because complex geometry, capture cost, negligence, and accessibility constraints limit exhaustive coverage.Irregular trajectories and multiple capture distances are needed for global structure and fine details, leaving some directions insufficiently observed.
- Limitations of Existing Methods: Reconstruction-based NVS can be photorealistic for in-the-wild scenes but deteriorates drastically when rendering viewpoints far from training views.Additional data collection can repair these artifacts, but it is laborious and often impractical.
- Limitations of Existing Methods: Generation-based NVS struggles with complex large-scale scenes, causing structural distortions and appearance hallucinations while offering limited precise control and long-term 3D consistency.These issues arise alongside scarce high-quality training data and conditioning on Plücker embeddings or artifact-prone RGB renderings.
- InceptionGS: InceptionGS adapts a pretrained generative prior using scene-specific geometric guidance from an initial 3DGS reconstruction.It refines the field from virtual viewpoints selected and interpolated to indicate visual artifacts, supporting long-range 3D refinement without overfitting to one view.
- InceptionGS: Photometric blending and view-adaptive sampling softly inject the adapted prior while preserving reliable fine-detail clues and reducing reconstruction-generation conflicts.The method directly learns scene-specific appearance conditioned on geometry rather than refining artifact-prone RGB renderings.
- Results: 32% average FID reduction after bootstrapping accompanies significant quantitative and qualitative improvements over reconstruction- and generation-based alternatives.Experiments use a benchmark built by clustering and filtering real-world imagery, primarily from the challenging GigaNVS dataset.
2 Related Work
Related work spans per-scene reconstruction and generalizable novel view synthesis. Reconstruction methods encode scene-specific information but depend on viewpoint coverage, whereas generalizable methods use pretrained models to synthesize views from few inputs.
- Reconstruction-based NVS: Per-scene optimization methods represent scene-specific information with implicit neural networks, explicit feature structures, or hybrid representations.They enable photorealistic rendering when viewpoint coverage is perfect.
- Generalizable NVS: Generalizable NVS pretrains deep neural networks on diverse multiview data to synthesize novel views from few input views without costly per-scene optimization.The passage contrasts this feed-forward strategy with reconstruction quality that deteriorates under large viewpoint variation.
3 Methodology
InceptionGS initializes and refines a 3DGS field by adapting a geometry-conditioned generative prior, then bootstrapping artifact-prone viewpoints with virtual views, photometric clues, and conservative supervision.
- Stage I: generative adaptation: InceptionGS first optimizes an initial 3DGS field while adapting a generic generative prior to scene-specific geometric features.The method combines RGB, geometry-regularization, and diffusion losses during generative adaptation.
- Stage II: bootstrapping Gaussians: The method softly applies generative supervision through view-space importance sampling and uses conservative reconstructive supervision before switching to perceptual supervision.LPIPS supervision is used once the structural SSIM loss falls below a predefined threshold τ.
- Stage I: generative adaptation: Geometry buffers from planar normals and foreground opacity condition a pretrained latent diffusion model to learn scene-specific geometry-appearance statistics.ControlNet-based adaptation uses local G-buffer patches and diffusion denoising supervision.
- Stage I: generative adaptation: The resulting scene-adaptive generative model improves fine-detail appearance generation while enabling 3D consistency and controllability without temporal modeling.The paper reports these properties in comparison with other generative methods.
- Stage II: bootstrapping Gaussians: For artifact-prone target views, the method generates candidate virtual viewpoints by selecting nearby training views and interpolating camera centers and orientations.The candidate set contains |V_vrt| = LK virtual viewpoints.
- Stage II: bootstrapping Gaussians: Bootstrapping blends generated appearance with warped RGB evidence from nearby real views to improve texture while addressing photometric domain shifts.The real-view images are warped using camera parameters and depth renderings from the initial PGSR reconstruction.
4 Experiments
Experiments evaluate InceptionGS on simulated unstructured view sampling across GigaNVS and MipNeRF360, showing improved quantitative and qualitative reconstruction, robustness to scarcity, and benefits from iterative bootstrapping.
- Experimental setup: Experiments use GigaNVS and MipNeRF360, with unstructured sampling simulated by holding out 90% of one viewpoint cluster for evaluation.Clusters are formed using farthest point sampling over camera position and orientation.
- Quantitative results: 10% lower average FID and 14% lower average LPIPS than the best-performing diffusion method demonstrate InceptionGS's quantitative advantage.The comparison reports mean metrics across all test views and scenes.
- Qualitative results: InceptionGS produces scene-specific, 3D-consistent generation that remains faithful under drastic camera movements and scale variations.The method learns geometry-appearance correspondence from reconstruction, unlike generation approaches prone to structural or appearance hallucinations.
- Qualitative results: Lifted 3DGS comparisons show that InceptionGS preserves detail while diffusion baselines exhibit blurriness, structural distortions, or floating artifacts.The generative prior is incorporated softly to support visually pleasing bootstrapping.
- Ablation study: Photometric blending, view-adaptive sampling, and finetuning are all crucial components, with blending improving textures and sampling preserving well-reconstructed regions.The ablation compares the full pipeline with versions without photometric blending or view-adaptive sampling.
- Sensitivity analysis: The method remains robust when input views observing a held-out region are reduced by 50% and remains stable when the number of virtual viewpoints changes by 50%.The reported robustness is attributed to ControlNet training on continuously updated Gaussian-splatting renderings and soft generative guidance.
- Iterative bootstrapping: A second bootstrapping round improves results when targeting different underobserved regions, while repeated repair of the same region remains stable without negative impacts.The evaluation uses 122 test views on the TW-Pavilion (Day) scene.
5 Conclusion
InceptionGS addresses unstructured view sampling in real-world large-scale scenes by adapting generative priors to incrementally refine 3DGS without additional data collection. Experiments demonstrate superiority on challenging benchmarks, while computational overhead remains a limitation.
- InceptionGS adapts generic generative priors using scene-specific geometry-appearance correspondence and applies them in a view-adaptive manner to refine the 3DGS field.
- InceptionGS addresses unstructured view sampling in real-world large-scale scenes without requiring additional data collection.
- Extensive experiments on challenging benchmarks demonstrate InceptionGS's significant superiority for large-scale scene digitization.
- Stage I takes approximately 30 minutes with peak VRAM of 30GB, while stage II takes around 25 minutes with peak VRAM below 15GB.
- Future work will investigate advanced optimization schemes such as test-time training to reduce per-scene overhead and improve scalability.