Source-linked AI summary
CityGaussian: Real-time High-quality Large-Scale Scene Rendering with Gaussians
Yang Liu, He Guan, Chuanchen Luo, Lue Fan, Naiyan Wang, Junran Peng, Zhaoxiang Zhang
TL;DR
Large-scale 3DGS remains difficult to train and render efficiently across widely varying scene scales because memory and computation grow substantially with scene capacity. CityGS uses divide-and-conquer training with global-prior-guided fusion and block-wise LoD selection. It achieves state-of-the-art rendering fidelity and consistent real-time performance across drastically different scales, while its generalization is limited by a static-scene assumption.
Problem
Large-scale 3DGS faces prohibitive training memory and rendering costs, despite the need for high-fidelity real-time reconstruction across large areas and scales.
Method
CityGS partitions scenes into blocks for parallel training, uses a coarse global Gaussian prior for fusion, and selects compressed detail levels block-wise during rendering.
Results
CityGS attains state-of-the-art rendering fidelity and consistent real-time performance across drastically varying scales, with PSNR 27.46 on the MatrixCity benchmark.
Takeaways & Limitations
The combined blocking and LoD strategy enables high-fidelity large-scale reconstruction while reducing rendering time across different scales.
Takeaways & Limitations
CityGS assumes a static scene, and combining drastically different aerial and street views degrades performance.
Abstract
from arXiv · showhide
The advancement of real-time 3D scene reconstruction and novel view synthesis has been significantly propelled by 3D Gaussian Splatting (3DGS). However, effectively training large-scale 3DGS and rendering it in real-time across various scales remains challenging. This paper introduces CityGaussian (CityGS), which employs a novel divide-and-conquer training approach and Level-of-Detail (LoD) strategy for efficient large-scale 3DGS training and rendering. Specifically, the global scene prior and adaptive training data selection enables efficient training and seamless fusion. Based on fused Gaussian primitives, we generate different detail levels through compression, and realize fast rendering across various scales through the proposed block-wise detail levels selection and aggregation strategy. Extensive experimental results on large-scale scenes demonstrate that our approach attains state-of-theart rendering quality, enabling consistent real-time rendering of largescale scenes across vastly different scales. Our project page is available at https://dekuliutesla.github.io/citygs/.
1 Introduction
Large-scale 3DGS must overcome prohibitive training memory and rendering costs while preserving high fidelity across viewing scales. CityGS addresses these challenges with divide-and-conquer training and block-wise LoD rendering.
- Directly training 3DGS can exceed available GPU memory, while millions of Gaussians make depth sorting and rasterization extremely slow.A 24G RTX3090 runs out of memory above 11 million Gaussians, while city-scale scenes may require over 20 million.
- CityGS partitions scenes into spatial blocks trained in parallel, reducing each block’s Gaussian and data load while balancing workloads through normalized cubic-grid division.Training poses are retained when inside a block or when that block substantially contributes to rendering.
- A coarse global Gaussian prior guides adjacent blocks, avoiding interference and enabling seamless fusion.
- Block-wise LoD rendering feeds only necessary Gaussians to the rasterizer and substitutes compressed representations for distant regions.The strategy exploits frustum occupancy and the reduced screen-space detail of distant regions.
- CityGS combines parallel reconstruction and LoD rendering to support real-time large-scale rendering across different scales with minimal quality loss.
2 Related Works
Related work spans implicit neural representations, explicit point-based rendering, large-scale divide-and-conquer reconstruction, and Level of Detail methods. These approaches seek better quality, speed, scalability, or editability.
- Neural Rendering: NeRF represents scenes implicitly with MLPs and volumetric rendering, but intensive ray sampling causes high training and inference latency.InstantNGP and Plenoxels reduce this burden through multiresolution hash grids or sparse voxel grids.
- Point-Based Representation: 3DGS uses explicit 3D Gaussian primitives and an optimized rasterizer to provide fast, high-quality rendering while addressing discontinuities in point-based images.
- Large-Scale Reconstruction: Large-scale reconstruction methods such as Block-NeRF and Mega-NeRF divide scenes into blocks represented by smaller neural models.
- Level of Detail: Level of Detail regulates representation complexity by reducing the workload for objects that become less important, such as distant objects.
3 Method
CityGS builds a global Gaussian scene prior, divides primitives and training data into blocks for parallel refinement, then fuses the results. Its pipeline contracts unbounded space for balanced partitioning and selects block data using spatial inclusion and rendering contribution.
- Overview: CityGS first generates a global 3DGS prior, then adaptively divides Gaussian primitives and data for parallel block training before fusing the scene.The global prior initializes block finetuning and supports the subsequent division strategy.
- Global Gaussian Prior: 30,000 iterations of training COLMAP points with all observations produce a coarse global geometry prior that reduces block interference during fusion.The prior also provides cleaner rendered images and more accurate geometry distribution for later division.
- Gaussian Partitioning: Unbounded Gaussian positions are contracted into a bounded cubic space before uniform grid division, producing more balanced Gaussian partitions.Foreground positions are normalized to [−1, 1], while the outer region uses nonlinear contraction.
- Data Partitioning: A pose is assigned to a block when the block contains the pose or contributes significantly to the rendered image according to an SSIM threshold.The contribution test uses the SSIM difference between rendering with and without the block’s Gaussians.
- Parallel Finetuning: Block finetuning uses the coarse global prior under the original uncontracted space, then retains Gaussians within each block’s spatial bounds.The training loss combines weighted L1 and SSIM losses, while enlarged bounds can ensure sufficient training data for sparse blocks.
4 Experimets
CityGS is evaluated on synthetic and real large-scale scenes, where it delivers strong rendering quality and LoD maintains real-time performance across varying camera scales.
- Datasets and baselines: Experiments cover the 2.7 km^2 MatrixCity Small City scene and four real-world datasets: Residence, Sci-Art, Rubble, and Building.The evaluation also includes MatrixCity street-scene generalization and comparisons with Mega-NeRF, Switch-NeRF, GPNeRF, and 3DGS†.
- Comparison with SOTA: 27.46 PSNR is achieved on the full MatrixCity reconstruction, while CityGS captures richer details than 3DGS† and reconstructs thin structures effectively.On the other realistic scenes, CityGS reports higher SSIM and LPIPS than the compared methods.
- Comparison with SOTA: CityGS outperforms NeRF-based baselines by a large margin and achieves higher SSIM and LPIPS on the other three realistic scenes.The reported visual improvements include girder steel, window frames, grass, and rubble.
- Level of Detail: LoD combines three compressed detail levels, trading quality and speed so that the combined version is near LoD 1 in speed and second only to LoD 2 in SSIM and PSNR.LoD 2 is finest, LoD 1 intermediate, and LoD 0 coarsest; their compression rates are 50%, 34%, and 25%, respectively.
- Level of Detail: Across different camera heights, LoD achieves the highest mean FPS and consistent worst-case real-time performance, whereas single-detail rendering degrades as altitude increases.The minimum-FPS comparison uses the 25 FPS threshold to distinguish real-time from non-real-time rendering.
- Training ablation: Global-model guidance mitigates floaters caused by training with cameras from an area 1.5× larger than each block.The ablation attributes the baseline’s underperformance to overfitting outside the block.
- LoD ablation: Larger LoD distance intervals preserve richer details but reduce speed, while point-wise selection causes considerably worse real-time performance.The ablation compares block-wise and point-wise selection strategies and different intervals for LoD 2 and LoD 1.
5 Conclusions
CityGS combines blocking and LoD to achieve high-fidelity, real-time reconstruction and rendering for large-scale scenes across different scales. Its generalization is limited by the hidden static-scene assumption, and combining aerial and street views degrades performance.
- Conclusion: CityGS achieves state-of-the-art rendering fidelity while reducing rendering time across drastically different scales of the same scene.The approach is built around blocking and a Gaussian-geometry-tailored LoD strategy.
- Limitations: The hidden static-scene assumption limits CityGS generalization.The conclusion identifies this as a scope limitation of the method.
- Limitations: Combining drastically different aerial and street views degrades CityGS performance instead of improving it.The conclusion states that the underlying mechanism requires further investigation.
A.1 Additional Quantitative Comparison
CityGS outperforms other methods on SSIM and LPIPS across five datasets and achieves the highest PSNR on MatrixCity, Rubble, and Building. LoD improves efficiency, while original 3DGS trades speed for lower rendering quality on large scenes.
- CityGS outperforms other methods in SSIM and LPIPS across MatrixCity, Residence, Rubble, Building, and Sci-Art.The comparison is reported in Table S1.
- CityGS achieves the highest PSNR on MatrixCity, Rubble, and Building.
- Sci-Art and Residence have relatively weaker PSNR because of appearance variations across views.The paper leaves this issue for future work.
- LoD significantly improves rendering efficiency, especially for extremely large-scale scenes such as MatrixCity.
- 3DGS† renders faster than CityGS but has significantly lower rendering quality on large-scale scenes.The original 3DGS is constrained by the memory and iterations needed to optimize thousands of images.
A.2 Additional Ablations
The ablations examine how block partitioning, data-assignment threshold ε, and compression settings affect CityGS configuration. Performance is optimal around 3 × 3 partitions, while detail-level compression uses three rates on most datasets.
- CityGS achieves optimal results around 3 × 3 partitions in the block-number ablation.The ablation is conducted on the Rubble dataset.
- As the block number grows, the average number of training poses assigned to each block decreases.
- As ε grows, the average number of poses assigned to blocks decreases.ε is the SSIM threshold used for data assignment.
- For datasets other than MatrixCity, LoD uses three detail levels with compression rates of 60%, 50%, and 40%.
C More Visualization on MatrixCity Dataset
Additional MatrixCity visualizations compare CityGS with state-of-the-art methods and emphasize reconstruction of intricate urban details. The examples particularly highlight crowded cars and crosswalks.
- CityGS showcases high-fidelity reconstruction of intricate details, including crowded cars and crosswalks.
- Figure S1 provides a qualitative comparison between CityGS and state-of-the-art methods on MatrixCity.
- The MatrixCity comparisons report enhanced visual fidelity for CityGS relative to other methods.
D Qualitative Validation on Concatenated Fusion
Concatenated fine-tuned Gaussians remain visually continuous when views span multiple blocks, supported by a coarse global Gaussian prior. CityGS’s explicit representation also supports scene manipulation demonstrations.
- Views spanning four or more blocks show no discernible discontinuities after concatenating fine-tuned Gaussians.The paper attributes smooth boundary transitions to the coarse global Gaussian prior.
- CityGS demonstrations repaint building parts, replace buildings, and reposition cars to simulate traffic conditions.These manipulations rely on the explicit city representation.
- The scene-manipulation demonstrations indicate potential real-time and interactive applications of CityGS.
- Figure S2 visualizes block partitions, overall Gaussians, visible Gaussians, and rendered images for synthetic and real datasets.Blue points denote overall Gaussians, red points denote visible Gaussians, and grey grids show block partitions.