Source-linked AI summary
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction
Jiaqi Lin, Zhihao Li, Xiao Tang, Jianzhuang Liu, Shiyong Liu, Jiayue Liu, Yangdi Lu, Xiaofei Wu, Songcen Xu, Youliang Yan, Wenming Yang
TL;DR
Large-scene reconstruction with 3D Gaussian Splatting is limited by memory, optimization time, and appearance variation. VastGaussian partitions scenes into visibility-aware cells for parallel optimization and merging, and applies decoupled appearance modeling during training. It reports high-quality reconstruction and real-time 1080p rendering while outperforming NeRF-based methods.
Problem
Large-scene 3D Gaussian Splatting is challenged by limited video memory, long whole-scene optimization, and appearance variations that cause floaters.
Method
VastGaussian progressively partitions scenes, selects cameras and point clouds by visibility for parallel cell optimization and seamless merging, and uses decoupled appearance modeling during optimization.
Results
VastGaussian outperforms compared methods on SSIM and LPIPS, achieves better or comparable PSNR, and provides real-time rendering at 1080p resolution.
Takeaways & Limitations
The method supports high-fidelity large-scene reconstruction with fast optimization, scalable memory use, consistent cross-view appearance, and real-time rendering after removing the appearance module.
Takeaways & Limitations
Quality comparisons for Grid-NeRF are not reproducible because critical rendered images and tuned configuration files are unavailable, so its quality metrics are copied from its paper.
Abstract
from arXiv · showhide
Existing NeRF-based methods for large scene reconstruction often have limitations in visual quality and rendering speed. While the recent 3D Gaussian Splatting works well on small-scale and object-centric scenes, scaling it up to large scenes poses challenges due to limited video memory, long optimization time, and noticeable appearance variations. To address these challenges, we present VastGaussian, the first method for high-quality reconstruction and real-time rendering on large scenes based on 3D Gaussian Splatting. We propose a progressive partitioning strategy to divide a large scene into multiple cells, where the training cameras and point cloud are properly distributed with an airspace-aware visibility criterion. These cells are merged into a complete scene after parallel optimization. We also introduce decoupled appearance modeling into the optimization process to reduce appearance variations in the rendered images. Our approach outperforms existing NeRF-based methods and achieves state-of-the-art results on multiple large scene datasets, enabling fast optimization and high-fidelity real-time rendering.
1. Introduction
Large-scene reconstruction requires high visual quality and real-time rendering, but scaling 3D Gaussian Splatting is constrained by memory, optimization cost, and appearance variation. VastGaussian addresses these issues through progressive cell partitioning and decoupled appearance modeling.
- A 32 GB GPU optimizes about 11 million 3D Gaussians, while the small Garden scene already requires approximately 5.8 million for high-fidelity reconstruction.
- Large-scene optimization is time-consuming and potentially unstable, while uneven illumination creates appearance variations that produce floater artifacts.
- VastGaussian partitions a large scene into cells, optimizes them independently, and merges them into a complete scene.Progressive visibility-based data selection adds cameras and point clouds to support seamless merging and eliminate air floaters.
- Decoupled appearance modeling uses image-level appearance embeddings and a CNN-derived transformation map during optimization to fit appearance variations while learning constant information.The module is discarded after optimization, so it does not slow real-time rendering.
- Experiments on several large-scene benchmarks report superiority over NeRF-based methods, with contributions including parallel optimization, seamless merging, and floater suppression.
2. Related Work
Large-scene reconstruction builds on SfM, MVS, NeRF, and scene-decomposition methods, while appearance modeling addresses variation from lighting and camera settings. VastGaussian combines visibility-aware image adjustment with Gaussian splatting.
- 2.1. Large Scene Reconstruction: Large-scene methods extend NeRF through geographic blocks, grid-based ray assignment, or learned mixture-of-experts scene decomposition.
- 2.2. Varying Appearance Modeling: Appearance variation arises under changing lighting and camera settings such as auto-exposure, auto-white-balance, and tonemapping.
- 2.2. Varying Appearance Modeling: VastGaussian concatenates appearance embeddings with rendered images and uses a CNN to produce transformation maps that adjust renderings to fit appearance variations.
3. Preliminaries
3D Gaussian Splatting represents scenes with optimized 3D Gaussians initialized from a sparse SfM point cloud and rendered through differentiable rasterization. Optimization compares rendered and input images and includes adaptive point densification.
- 3DGS represents geometry and appearance with 3D Gaussians defined by position, anisotropic covariance, opacity, and spherical-harmonic view-dependent colors.
- The optimization dataset contains a sparse point cloud P and training views V={(C_i,I_i)}, estimated from images using Structure-from-Motion.P initializes the 3D Gaussians, while V supports differentiable rendering and gradient-based optimization.
- For camera C_i, a differentiable rasterizer R produces the rendered image I^r_i from the Gaussian set G.
- The Gaussian properties are optimized using a loss between rendered images and corresponding input images, with adaptive point densification interleaved during training.Densification is triggered when a point's cumulative gradient reaches a threshold.
4. Method
VastGaussian partitions large scenes into cells for independent optimization, using visibility-aware data assignment and later seamless merging. Decoupled appearance modeling adjusts training images while preserving consistent rendering for real-time inference.
- Progressive data partitioning: Large scenes are partitioned into cells containing subsets of views and point clouds, reducing each optimization problem’s data size and memory demands.Cells are optimized independently and later combined into a complete scene.
- Progressive data partitioning: Cells are divided according to projected camera positions, with expanded boundaries and balanced view counts to support optimization across regions.The scene is partitioned along two ground-plane axes so cells contain approximately similar numbers of training views.
- Visibility-based selection: Cameras are added when a cell’s projected visibility exceeds a threshold, and observed points are incorporated to improve initialization and reduce floaters.Airspace-aware visibility uses the cell’s bounding-box projection rather than only projected surface points, providing supervision for visible airspace.
- Decoupled appearance modeling: Decoupled appearance modeling downsamples rendered images, concatenates per-image embeddings, and uses a CNN to generate a transformation map for appearance adjustment.The adjusted image fits appearance variations while structural loss is applied to the consistent rendering.
- Decoupled appearance modeling: The appearance module is discarded after optimization, so appearance correction does not slow real-time rendering during deployment.The optimized Gaussians retain consistent appearance and geometry across input views.
- Seamless merging: After independent optimization, Gaussians outside each cell’s original boundaries are removed before non-overlapping cells are merged, avoiding obvious border artifacts.Shared training views between adjacent cells support seamless appearance and geometry.
5. Experiments
Experiments evaluate reconstruction quality, efficiency, memory, and component effects across large-scene benchmarks. VastGaussian improves perceptual quality and reduces optimization memory while retaining real-time rendering, with visibility and appearance modules suppressing floaters and boundary artifacts.
- Reconstruction quality: VastGaussian outperforms compared methods on all SSIM and LPIPS metrics, while achieving better or comparable PSNR.The method produces cleaner detail than NeRF-based methods and avoids the floaters seen with Modified 3DGS.
- Ablation study: Visibility-based camera selection and coverage-based point selection improve visual quality by reducing floaters in cell airspace.Selecting cameras by visibility also provides more common cameras between adjacent cells, reducing boundary appearance jumping.
- Ablation study: Airspace-aware visibility supplies more supervision for cell optimization and avoids floaters produced by airspace-agnostic visibility.The comparison is reported in the ablation study and illustrated in Figure 8.
- Ablation study: More cells improve SSIM and LPIPS and shorten parallel training time, but excessive partitioning can slightly decrease PSNR.The PSNR decrease is associated with gradual brightness changes between cells that are far apart.
6. Conclusion and Limitation
VastGaussian combines progressive partitioning with decoupled appearance modeling for high-quality, real-time large-scene reconstruction. The authors note unresolved division and scalability constraints involving scene layout, storage, and rendering speed.
- Conclusion: Progressive partitioning enables independent cell optimization and seamless merging into a complete scene with sufficient 3D Gaussians.The merged scene is intended to provide consistent rendering across views after cell optimization.
- Conclusion: Decoupled appearance modeling separates training-image appearance variations from consistent geometry and colors, and can be discarded after optimization.Discarding the module preserves faster rendering after training.
- Limitations: The method does not provide an optimal spatial division solution accounting for scene layout, cell number, and training-camera distribution.This is an explicit limitation of the proposed approach.
- Limitations: Very large scenes may require substantial storage and significantly slower rendering because they contain many 3D Gaussians.The limitation becomes more relevant as scene scale increases.
Supplementary Material
The supplementary material provides additional details about decoupled appearance modeling and its implementation and analysis.
- Supplementary Material: Supplementary material expands the CNN architecture, training time and memory, complex transformations, and transformation-map visualizations.
7. Details of Decoupled Appearance Modeling
Decoupled appearance modeling uses a CNN to transform rendered images during optimization, helping VastGaussian learn consistent colors across views. The module improves quality with limited overhead and can be removed after training.
- CNN architecture: The CNN receives a 32× downsampled rendered image concatenated with a 64-dimensional appearance embedding at each pixel.The resulting feature map has shape H/32 × W/32 × 67 before convolutional processing.
- CNN architecture: Four upsampling blocks progressively restore resolution before convolutional layers produce the appearance transformation map.Each block doubles spatial resolution and halves channel depth.
- Efficiency: Less than 10% additional training time and video memory significantly improves rendering quality when decoupled appearance modeling is enabled.The overhead comes from introducing appearance embeddings and the CNN.
- Transformation choices: Simple pixel-wise multiplication is sufficient on the evaluated datasets, while affine transformation and Gamma correction have marginal effects.The extensions require CNN outputs with 6 or 4 channels, respectively.
- Optimization behavior: The model learns consistent colors between adjacent training views as transformation maps converge and adjusted images fit the training images.During optimization, appearance variations between rendered images are reduced.