Source-linked AI summary
Multi-Scale 3D Gaussian Splatting for Anti-Aliased Rendering
Zhiwen Yan, Weng Fei Low, Yu Chen, Gim Hee Lee
TL;DR
3D Gaussian Splatting loses rendering quality and speed at low resolutions or from distant viewpoints because small Gaussians are undersampled and many splats compete for each pixel. The paper introduces multi-scale Gaussians selected by pixel coverage, reporting substantial quality and speed gains at reduced scales while acknowledging that all Gaussians are still splatted before filtering.
Problem
Low-resolution or distant rendering causes severe aliasing and speed degradation in 3D Gaussian Splatting because small Gaussians crowd pixels and require sequential blending.
Method
The method represents the scene with Gaussians at multiple scales, aggregates fine Gaussians into larger ones, and selects them by pixel coverage during rendering.
Results
13%-66% PSNR and 160%-2400% rendering speed improvements are achieved at 4×-128× scale rendering on Mip-NeRF360 with fewer than 5% additional Gaussians.
Takeaways & Limitations
Multi-scale representation supports anti-aliased, faster rendering at reduced scales while maintaining comparable quality and speed at 1× scale.
Takeaways & Limitations
At very low resolutions, all Gaussians must still be splatted before filtering, so rendering time does not decrease linearly as resolution decreases.
Abstract
from arXiv · showhide
3D Gaussians have recently emerged as a highly efficient representation for 3D reconstruction and rendering. Despite its high rendering quality and speed at high resolutions, they both deteriorate drastically when rendered at lower resolutions or from far away camera position. During low resolution or far away rendering, the pixel size of the image can fall below the Nyquist frequency compared to the screen size of each splatted 3D Gaussian and leads to aliasing effect. The rendering is also drastically slowed down by the sequential alpha blending of more splatted Gaussians per pixel. To address these issues, we propose a multi-scale 3D Gaussian splatting algorithm, which maintains Gaussians at different scales to represent the same scene. Higher-resolution images are rendered with more small Gaussians, and lower-resolution images are rendered with fewer larger Gaussians. With similar training time, our algorithm can achieve 13\%-66\% PSNR and 160\%-2400\% rendering speed improvement at 4$\times$-128$\times$ scale rendering on Mip-NeRF360 dataset compared to the single scale 3D Gaussian splitting. Our code and more results are available on our project website https://jokeryan.github.io/projects/ms-gs/
1. Introduction
3D Gaussian Splatting offers fast, high-quality rendering at high resolution but suffers severe aliasing and speed degradation at low resolution or from distant views. The paper addresses both issues with multi-scale Gaussians selected according to rendering resolution and pixel coverage.
- Problem: Low-resolution or distant rendering creates aliasing because detailed scene signals are sampled too coarsely relative to the pixel size.Rendering at high resolution and downscaling is not viable for large scenes because of its resource demands.
- Problem: Small Gaussians concentrated in detailed regions can crowd a pixel, allowing a front-most small Gaussian to dominate its color and create severe artifacts.Per-Gaussian low-pass filtering can further enlarge Gaussian extents at low resolutions.
- Problem: Low-resolution rendering is slower because many Gaussians remain to be rendered while more of them are concentrated in fewer pixels and cannot be blended in parallel.This differs from NeRF rendering, whose time decreases linearly with resolution.
- Approach: The method represents scenes with multi-scale Gaussians, aggregating finer Gaussians into larger, coarser ones for lower resolutions and distant views.It draws inspiration from mipmap and level-of-detail algorithms that precompute representations at different scales.
- Results: 13%-66% PSNR and 160%-2400% rendering speed improvements are achieved at 4×-128× scale rendering on Mip-NeRF360, with fewer than 5% additional Gaussians.At 1× scale, rendering quality and speed remain comparable to the original method.
- Context: 3D Gaussian Splatting has representation-specific anti-aliasing challenges, and this work presents an anti-aliasing algorithm designed for that representation.The challenges arise partly from detail-rich regions containing many small Gaussians.
3. Preliminaries
The preliminaries explain aliasing as a sampling problem and describe how EWA Splatting applies per-Gaussian low-pass filtering. At low resolutions, that filtering can worsen both artifacts and rendering cost because enlarged Gaussians overlap more and require sequential blending.
- Gaussian rendering: 3D Gaussian Splatting projects 3D Gaussians into 2D screen-space Gaussians, sorts tile-overlapping Gaussians by depth, and rasterizes pixels through sequential alpha blending.The scene representation includes each Gaussian’s variance, center, density, and color.
- Sampling and aliasing: Aliasing occurs when a signal’s highest frequency fmax exceeds half the sampling frequency fs = 1/∆x, causing sampled frequency replicas to overlap.The minimum sampling frequency needed to avoid aliasing is the Nyquist frequency fNy = 2 · fmax.
- EWA filtering: EWA Splatting applies a Gaussian low-pass filter independently to each splatted 2D signal to produce a band-limited rendered color.The filter is combined with the reconstruction Gaussian through covariance addition.
- EWA filtering: The filter covariance is determined by screen-pixel size, effectively increasing each Gaussian’s covariance during rendering.This smooths Gaussian edges when their size is not too small relative to the pixel size.
- Low-resolution failure modes: At low resolutions, covariance enlargement makes small front Gaussians dominate pixel colors and increases the number of Gaussians involved in sequential per-pixel blending.These effects produce severe artifacts and slower rendering.
4. Our Method
The method represents a scene with Gaussians at multiple detail levels and selects scale-appropriate subsets according to screen-space pixel coverage. It aggregates undersized fine Gaussians into enlarged coarse Gaussians, enabling lower-resolution rendering with fewer primitives while retaining fine detail at high resolution.
- Multi-Scale Gaussians: Multi-scale 3D Gaussians represent the scene with four levels of detail corresponding to 1×, 4×, 16×, and 64× downsampled resolutions.
- Pixel Coverage: Pixel coverage measures a splatted Gaussian’s smaller screen-space axis in pixels up to a low-opacity level set.The coverage approximates the Gaussian’s spatial extent and relates to its highest represented frequency.
- Pixel Coverage: Gaussians with pixel coverage below ST = 2px are filtered to avoid aliasing, but naive filtering creates holes because fine Gaussians encode scene content.The method therefore replaces filtered fine Gaussians with coarse Gaussians encoding low-frequency signal.
- Aggregate to Insert Large Gaussians: Fine Gaussians below the coverage threshold are grouped spatially by voxels, average-pooled, enlarged, and inserted as coarse-level Gaussians.For level lm, selected Gaussians are binned using a (400/lm)^3 voxel grid, and new scaling is enlarged by ST/Savg.
- Selective Rendering: Selective rendering includes a Gaussian when its current coverage falls within its stored minimum and maximum coverage range, with boundary rules preserving trained extreme levels.The coverage range determines which scale is rendered at a given resolution or camera distance.
- Selective Rendering: Higher resolutions render more smaller Gaussians, whereas lower resolutions render fewer larger Gaussians for less aliasing and faster speed.
5. Experiments
The experiments compare vanilla and multi-scale Gaussian splatting across datasets, resolutions, quality metrics, and rendering time. The method’s advantages become strongest at low resolutions, while ablations show that filtering and inserted large Gaussians jointly support the quality-speed improvement.
- Experimental Setup: The evaluation compares vanilla and proposed Gaussian splatting on object-centric, indoor, and outdoor datasets using matched training settings and a single RTX3090 GPU.Models are trained for 40000 iterations with other hyperparameters unchanged.
- Experimental Setup: Quality is measured with PSNR and LPIPS, while speed is measured by per-image rendering time at 1×, 4×, 16×, and 64× downsampling.The evaluation targets low-resolution and distant-camera rendering conditions.
- Quantitative Comparison: 6-10dB PSNR and 20-30× speed gains occur at the 64× resolution scale compared with original 3D Gaussian Splatting.At 1× resolution, quality and speed remain comparable, while the improvements become more pronounced as resolution decreases.
- Qualitative Comparison: At 16×-64× downsampling, the original method produces severe foreground-dominating artifacts, whereas the proposed method remains close to ground truth across scales.At 1×-8×, both methods render novel views relatively faithfully.
- Ablations: Multi-scale training alone marginally improves low-resolution quality but remains slow, while adding large Gaussians without filtering significantly reduces quality.
- Ablations: Combining large-Gaussian insertion with small-Gaussian filtering significantly improves low-resolution quality and speed without sacrificing high-resolution quality.The speed gain comes from rendering considerably fewer Gaussians, while inserted large Gaussians improve low-resolution quality.
6. Limitations
The method’s Gaussian filtering must wait until pixel coverage is computed, so all Gaussians are initially splatted even when few are ultimately rendered.
- Filtering occurs only after initial splatting because it relies on pixel coverage calculated during that process.Consequently, every Gaussian must be splatted before selection.
7. Conclusion
The paper identifies scale mismatch as the key challenge in anti-aliased Gaussian splatting and uses coverage-based filtering plus aggregation to improve rendering quality and speed.
- The key challenge is representing the scene with Gaussians of appropriate scale for the rendering resolution.
- The method calculates pixel coverage during splatting and filters Gaussians that are too large or too small for the current resolution.It also aggregates small Gaussians during training to insert larger Gaussians that preserve low-frequency details and prevent missing parts.
- Experiments on various datasets support improved rendering quality and speed at both high and low resolutions while mitigating severe aliasing.
8. Video Comparison
The paper includes a video comparison that presents the proposed method and original 3D Gaussian Splatting across multiple scenes, views, and resolutions.
- The video compares the proposed algorithm with original 3D Gaussian Splatting across multiple scenes, views, and resolutions.
9. Details of Gaussian Aggregation Algorithm
The aggregation algorithm builds coarse Gaussian representations for lower-resolution rendering by grouping, averaging, enlarging, and frequency-analyzing finer Gaussians across scene scales.
- Render at Lower Resolution: For lower-resolution levels, all relevant Gaussians are rendered first so their pixel coverage can determine which small Gaussians require aggregation.
- Gaussian aggregation: The method omits low-pass filters for individual Gaussians and replaces smaller Gaussians with fewer larger ones to reduce bandwidth and rendered primitives.
- Theoretical Anti-aliasing Analysis: The Gaussian mixture is transformed into the frequency domain to analyze the resulting 3dB bandwidth after scaling.
- Theoretical Anti-aliasing Analysis: The scaled Gaussians retain bandwidth invariant to the smaller Gaussians they replace and remain below half the sampling frequency to avoid aliasing.The paper characterizes this as equally effective as traditional low-pass filtering while being more efficient for rendering.
- Unbounded Scene Normalization: Unbounded scene coordinates are normalized before voxelization, with the exact normalization specified for the bounded center region and outer region.
- Voxelization: After normalization, Gaussians are grouped by voxel, with voxel size increasing for coarser levels that require fewer larger Gaussians.The stated empirical voxel size for level lm is (400/lm)^3.
- Average Pooling and Enlargement: The aggregation process forms an average Gaussian from grouped smaller Gaussians and averages position, rotation, spherical harmonics, opacity, and scaling parameters.
- Average Pooling and Enlargement: The averaged Gaussian is enlarged using its average pixel coverage so the resulting representation reaches the intended rendering scale.
11. Qualitative Ablation Study
The ablation study shows that filtering small Gaussians improves low-resolution speed, while inserting large Gaussians improves low-resolution quality. The full method combines these components to maintain high-quality rendering at both high and low resolutions.
- Ablation components: Filtering small Gaussians significantly accelerates low-resolution rendering but can remove scene content and introduce black-dot artifacts.The filtering is based on pixel coverage and is applied after multi-scale training.
- Ablation components: Inserting large aggregated Gaussians provides good low-resolution rendering speed and quality, but leaves high-resolution images over-smoothed.The over-smoothing results from optimizing finer Gaussians together with inserted large Gaussians at low resolutions.
- Full method: The full method overcomes the ablation weaknesses, producing high-quality rendering at fast speed across high and low resolutions.Small-Gaussian filtering improves speed, while large-Gaussian insertion improves low-resolution quality.
12. Quantitative Results on More Resolutions
The authors evaluate the method beyond the training resolutions across three datasets, comparing it with original 3D Gaussian Splatting and ablation methods. These tests assess performance and robustness at more downsampled scales.
- Evaluation scope: The experiments include resolutions not used during training to evaluate performance and robustness beyond the training conditions.The comparison covers the proposed method, original 3D Gaussian Splatting, and various ablation methods.
- Datasets: The quantitative evaluations are conducted on the MipNeRF-360, Tank and Temple, and Deep Blending datasets.Results are reported in Tables 4, 5, and 6, respectively.
- Evaluation scope: The study reports quantitative comparisons and ablation results at more downsampled resolutions.This design examines how the methods behave as rendering resolution decreases.
13. Per-Scene Quantitative Results
The paper decomposes quantitative performance by scene for the proposed method and original 3D Gaussian Splatting across multiple resolutions. The analysis spans three datasets and uses scene selections aligned with the original 3D Gaussian Splatting experiments.
- Per-scene comparison: Per-scene quantitative results compare the proposed method with original 3D Gaussian Splatting at various resolutions.The decomposition is intended to expose scene-level performance differences rather than only aggregate results.
- Datasets: The per-scene analysis covers MipNeRF-360, Tank and Temple, and Deep Blending datasets.Results are presented in Tables 7, 8, and 9, respectively.
- Experimental setup: The tested scenes follow the experiments conducted in the original 3D Gaussian Splatting paper.This keeps the scene selection consistent with the baseline study.