Source-linked AI summary
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting
Zehao Zhu, Zhiwen Fan, Yifan Jiang, Zhangyang Wang
TL;DR
Few-shot novel view synthesis must recover accurate 3D structure from limited observations without sacrificing efficiency. FSGS uses 3D Gaussian Splatting with proximity-guided Gaussian unpooling and augmented-view monocular-depth guidance, achieving photo-realistic rendering from as few as three images at more than 200 FPS. Its scope is limited because it cannot generalize to occluded views unobserved during training.
Problem
Few-shot novel view synthesis seeks accurate rendering from limited observations, while existing NeRF-based approaches often require substantial computation for accurate 3D representation.
Method
FSGS densifies sparse SfM initialization through proximity-guided Gaussian unpooling and regularizes optimization with pseudo-views and monocular relative-depth correspondences.
Results
FSGS achieves photo-realistic novel views from as few as three images and performs inference at more than 200 FPS across few-shot settings.
Takeaways & Limitations
FSGS offers real-time few-shot rendering and supports more cost-effective capture methods from extremely sparse point clouds.
Takeaways & Limitations
FSGS cannot generalize to occluded views that are unobserved during training.
Abstract
from arXiv · showhide
Novel view synthesis from limited observations remains an important and persistent task. However, high efficiency in existing NeRF-based few-shot view synthesis is often compromised to obtain an accurate 3D representation. To address this challenge, we propose a few-shot view synthesis framework based on 3D Gaussian Splatting that enables real-time and photo-realistic view synthesis with as few as three training views. The proposed method, dubbed FSGS, handles the extremely sparse initialized SfM points with a thoughtfully designed Gaussian Unpooling process. Our method iteratively distributes new Gaussians around the most representative locations, subsequently infilling local details in vacant areas. We also integrate a large-scale pre-trained monocular depth estimator within the Gaussians optimization process, leveraging online augmented views to guide the geometric optimization towards an optimal solution. Starting from sparse points observed from limited input viewpoints, our FSGS can accurately grow into unseen regions, comprehensively covering the scene and boosting the rendering quality of novel views. Overall, FSGS achieves state-of-the-art performance in both accuracy and rendering efficiency across diverse datasets, including LLFF, Mip-NeRF360, and Blender. Project website: https://zehaozhu.github.io/FSGS/.
1 Introduction
FSGS targets few-shot novel view synthesis by using 3D Gaussian Splatting with sparse SfM initialization, proximity-guided Gaussian growth, and augmented-view depth guidance. It improves rendering quality and speed while addressing sparse-view coverage and overfitting.
- Motivation: Few-shot novel view synthesis is important for 3D vision applications, but dense support views and NeRF-based computation limit practical use.Prior methods reduce view requirements with depth regularization, pretrained-model supervision, large-scale pretraining, and frequency annealing, but often remain computationally demanding.
- Challenges: Sparse-view 3D Gaussian Splatting struggles because gradient-driven densification places new Gaussians unreliably and photometric optimization can oversmooth textures.Extremely sparse SfM points also leave scene regions uncovered and encourage overfitting to training views.
- Method: FSGS introduces Proximity-guided Gaussian Unpooling, which inserts new Gaussians between neighboring ones and increases representation density.The process initializes new Gaussians using observations from existing ones, supporting more detailed scene representation.
- Method: FSGS generates unseen viewpoints during training and uses distance correspondences with monocular depth priors to guide Gaussian optimization toward accurate, visually compelling solutions.The added constraints address overfitting in sparse-view Gaussian splatting across actual and synthesized views.
2 Related Works
Related work develops neural representations and efficiency improvements for novel view synthesis, while few-shot methods add geometric, semantic, or learned context supervision to reduce view requirements.
- Neural representations: NeRF maps 3D coordinates and view direction to color and density through an MLP and volume rendering function.Subsequent work targets efficiency, quality, and generalization to unseen scenes.
- Neural representations: MVSNeRF uses a 3D cost volume, while Mip-NeRF and Mip-NeRF 360 address aliasing and unbounded scenes with conical-frustum rendering.These NeRF-like models generally require several hours of training, motivating efficiency-oriented alternatives.
- Few-shot synthesis: Few-shot NeRF methods reduce training views using depth supervision, depth-smoothness regularization, CLIP supervision, or learned contextual features.The original NeRF requires more than one hundred input images, motivating these view-reduction strategies.
3 Method
FSGS addresses sparse-view 3D Gaussian Splatting by densifying sparse SfM initialization and guiding optimization with pseudo views and monocular depth. Its pipeline combines proximity-guided Gaussian Unpooling with photometric and geometry-aware optimization to improve scene coverage and novel-view reconstruction.
- 3.2 Proximity-guided Gaussian Unpooling: FSGS initializes Gaussians from COLMAP and applies Proximity-guided Gaussian Unpooling to increase density where sparse points provide inadequate scene coverage.The method constructs a proximity graph using nearest neighbors and inserts new Gaussians at selected graph-edge centers.
- 3.1 Preliminary and Problem Formulation: Sparse SfM initialization and conventional densification can leave 3D-GS under-covered, oversmoothed, and prone to overfitting sparse training views.The problem is especially severe when the input contains very few views and some Gaussians grow to excessively large volumes.
- 3.2 Proximity-guided Gaussian Unpooling: When a Gaussian’s proximity score exceeds t_prox, FSGS creates a new Gaussian at the center of each edge connecting it to a selected neighboring Gaussian.New scale and opacity attributes are matched to the destination Gaussian, while other attributes are initialized from existing observations.
- 3.3 Geometry Guidance for Gaussian Optimization: FSGS synthesizes pseudo views between nearby training cameras, with random camera-location perturbations, to add constraints during sparse-view optimization.Online pseudo-view synthesis supports progressively updated geometry and reduces the risk of fitting only the observed views.
- 3.3 Geometry Guidance for Gaussian Optimization: Monocular depth from a pretrained DPT is generated for training and pseudo views, and Pearson correlation aligns rendered and estimated depth structure despite scale ambiguity.The resulting geometry regularization guides Gaussian deformation and optimization toward a more plausible scene geometry.
4 Experiments
Across LLFF, Mip-NeRF360, Blender, and Shiny datasets, FSGS delivers stronger novel-view quality and faster rendering than the compared baselines. Ablations attribute these gains to Gaussian Unpooling, relative depth regularization, and pseudo-view training.
- LLFF: FSGS achieves the best LLFF results at both resolutions, surpassing SparseNeRF by 0.45dB and 0.81dB in PSNR while running 2,180× faster.It also outperforms 3D-GS by 2.88dB in PSNR and raises FPS from 385 to 458.
- Qualitative comparisons: Qualitatively, FSGS preserves sharper details and thin structures, whereas competing methods show floaters, aliasing, oversmoothing, or degraded geometry.The comparisons cover LLFF and large-scale Mip-NeRF360 scenes, including leaves, piano keys, and other fine-grained structures.
- Mip-NeRF360: FSGS improves Mip-NeRF360 rendering over NeRF-based methods by 0.85dB in PSNR and raises FPS from 0.07 to 290 at 1/8 resolution.Table 2 reports an SSIM improvement of more than 0.05 and 4,142× faster execution than NeRF-based methods.
- Blender and Shiny: On Blender and Shiny datasets, FSGS outperforms the baselines across reported metrics, improving PSNR by 0.40 over FreeNeRF on Blender and 1.80dB over 3D-GS on Shiny.The Shiny results cover challenging view-dependent materials such as CDs and glass.
- Ablation studies: Proximity-guided Gaussian Unpooling improves PSNR by 1.21dB over 3D-GS by expanding scene geometry and enriching structural details.The ablation identifies blurring from heuristic densification, especially in bush and grass regions.
- Ablation studies: Relative depth regularization produces more plausible, coherent geometry by reducing grassy-region artifacts and aligning rendered depths with scene structures.Pseudo-view synthesis further anchors Gaussians to plausible geometry and improves modeling quality when densification geometry is inaccurate.
- Ablation studies: FSGS remains robust across different pretrained monocular depth estimators.Depth-Anything achieves the most comparable results among the evaluated estimators.
5 Conclusion and Limitation
FSGS is a real-time few-shot novel-view synthesis framework that combines proximity-guided Gaussian Unpooling with pseudo-view and monocular relative-depth guidance. It produces photo-realistic views from as few as three images and over 200 FPS, but does not generalize to unobserved occluded views.
- Conclusion: FSGS combines sparse point-based Gaussian representation, Proximity-guided Gaussian Unpooling, pseudo-view generation, and monocular relative-depth correspondences.These components guide expanded scene geometry toward a better solution under insufficient view overlap.
- Conclusion: FSGS generates photo-realistic novel views from as few as three images and performs inference at more than 200FPS.The authors connect this capability with real-time rendering and more cost-effective capture methods.
- Limitation: FSGS cannot generalize to occluded views that are unobserved during training.The limitation bounds its ability to synthesize arbitrary unseen regions behind occlusions.
6 More Technical Details
FSGS initializes Gaussians from COLMAP-derived point clouds and optimizes them using RGB and depth supervision from input and synthesized views. Its training pipeline progressively refines the representation through scheduled refinement, densification, pruning, and Gaussian Unpooling.
- Initialization: FSGS calibrates input images with COLMAP, obtains a fused stereo point cloud, and initializes Gaussian positions and degree-0 spherical-harmonic coefficients from it.The remaining coefficients and Gaussian parameters are optimized during training.
- Training schedule: FSGS increases spherical-harmonic degree from 0 to 4 every 500 iterations and resets all Gaussian opacities to 0.05 at iterations 2000, 5000, and 7000.The resets are used to eliminate low-opacity floaters.
- Training-view analysis: FSGS consistently outperforms the other methods across all metrics when evaluated with training-view counts ranging from 3 to 9.The experiment isolates the effect of using different numbers of training views.
- Training objectives: The training loss combines RGB L1 error, D-SSIM, and Pearson depth correlation, using monocular depth predictions from a pretrained estimator.The pipeline also synthesizes pseudo camera poses from the input poses for additional supervision.
- Training objectives: After a threshold iteration, FSGS rasterizes RGB and depth for sampled pseudo views and adds their depth-based constraint to the optimization.The estimated depth for pseudo views is obtained before computing the additional loss term.
- Refinement: During refinement, FSGS removes Gaussians according to opacity or size, densifies where position gradients exceed a threshold, and applies Gaussian Unpooling when proximity criteria are unmet.The Gaussian parameters are then updated through the loss gradient until convergence.
7 More Experiment Results
Across sparse-view evaluations, FSGS maintains strong rendering quality and generalization, including on consumer-smartphone imagery. Its advantages are especially evident in speed, geometry reconstruction, and fine-grained depth details.
- LLFF results: FSGS outperforms competing methods across all metrics when evaluated with 3, 6, and 9 training views on LLFF datasets.Using more views generally improves photo-realistic performance by providing broader scene coverage and stronger supervision.
- Mobile-phone generalization: FSGS reveals most scene details from only three training views on self-captured iPhone images after COLMAP calibration.The evaluation tests generalization to mobile-phone imagery.
- Rendered-view comparisons: FSGS improves visual quality over 3D-GS particularly in geometry reconstruction on mobile-phone scenes.The qualitative comparison covers indoor and outdoor scenes captured with an iPhone 15 Pro.
- Rendered-depth comparisons: FSGS produces more accurate and detailed predicted depth than 3D-GS, FreeNeRF, and SparseNeRF in qualitative comparisons.FreeNeRF and SparseNeRF show distorted complex geometries, while 3D-GS loses fine details in regions distant from the camera.