Source-linked AI summary
FreGS: 3D Gaussian Splatting with Progressive Frequency Regularization
Jiahui Zhang, Fangneng Zhan, Muyu Xu, Shijian Lu, Eric Xing
TL;DR
3D Gaussian splatting can over-reconstruct high-variance regions during densification, causing blur and artifacts. FreGS uses progressive Fourier-space frequency regularization for coarse-to-fine densification, and experiments show superior novel view synthesis and consistent outperformance of state-of-the-art methods.
Problem
3D Gaussian splatting can over-reconstruct high-variance regions when a few large Gaussians cover them, leading to blur and artifacts in rendered images.
Method
FreGS progressively regularizes rendered-ground-truth frequency-spectrum discrepancies from low to high frequencies, using frequency annealing for coarse-to-fine Gaussian densification.
Results
Experiments over multiple widely adopted indoor and outdoor scenes show superior novel view synthesis and consistent outperformance of state-of-the-art methods.
Takeaways & Limitations
Frequency-space regularization mitigates over-reconstruction and achieves superior Gaussian densification for high-quality novel view synthesis.
Abstract
from arXiv · showhide
3D Gaussian splatting has achieved very impressive performance in real-time novel view synthesis. However, it often suffers from over-reconstruction during Gaussian densification where high-variance image regions are covered by a few large Gaussians only, leading to blur and artifacts in the rendered images. We design a progressive frequency regularization (FreGS) technique to tackle the over-reconstruction issue within the frequency space. Specifically, FreGS performs coarse-to-fine Gaussian densification by exploiting low-to-high frequency components that can be easily extracted with low-pass and high-pass filters in the Fourier space. By minimizing the discrepancy between the frequency spectrum of the rendered image and the corresponding ground truth, it achieves high-quality Gaussian densification and alleviates the over-reconstruction of Gaussian splatting effectively. Experiments over multiple widely adopted benchmarks (e.g., Mip-NeRF360, Tanks-and-Temples and Deep Blending) show that FreGS achieves superior novel view synthesis and outperforms the state-of-the-art consistently.
1. Introduction
3D Gaussian splatting enables efficient, high-quality novel view synthesis but can over-reconstruct high-variance regions during densification. FreGS addresses this issue with progressive frequency regularization and improves Gaussian densification and novel view synthesis.
- 3D Gaussian splatting offers superb training and inference speed with competitive rendering quality through explicit anisotropic Gaussian representations and efficient splatting.
- Over-reconstruction occurs when a few large Gaussians cover high-variance image regions, producing deficiencies in learned representations.
- FreGS regularizes frequency signals in Fourier space to address over-reconstruction during Gaussian densification.
- Frequency annealing progressively applies regularization from low- to high-frequency signals, enabling coarse-to-fine Gaussian densification.Low-frequency signals generally encode easier-to-model global structure, while high-frequency signals encode harder-to-model local details.
- Experiments over multiple benchmarks show that FreGS improves Gaussian densification and novel view synthesis while outperforming 3D-GS consistently.
2. Related Work
Novel view synthesis has progressed from CNN and volumetric approaches to NeRF and efficient 3D Gaussian splatting. FreGS extends this line of work by using progressive frequency regularization to reduce over-reconstruction in Gaussian splatting.
- Early novel view synthesis methods used CNNs for blending-weight prediction and volumetric ray-marching with persistent 3D scene representations.
- NeRF models 3D scenes with MLPs from multi-view images and supports novel-view generation with strong multi-view consistency, but often requires expensive training or rendering.
- 3D Gaussian splatting provides high-quality explicit scene representations with efficient training and real-time rendering, but densification can introduce blur and artifacts through over-reconstruction.
- FreGS progressively regularizes low-to-high frequency components using frequency annealing and low-pass and dynamic high-pass filters for coarse-to-fine Gaussian densification.
- Frequency-space learning has also been explored in NeRF through varying-frequency encodings, gradual frequency increases, and wavelet frequency decomposition.
3. Proposed Method
FreGS regularizes rendered-image frequency spectra against ground truth to expose over-reconstruction and guide coarse-to-fine Gaussian densification. It combines amplitude and phase discrepancies with frequency annealing that progressively incorporates low- to high-frequency components.
- Preliminary: 3D-GS over-reconstruction occurs when large high-variance regions are covered by too few Gaussians, while densification relies on gradient signals.Small pixel gradients prevent affected Gaussians from being split, leaving regions inadequately represented.
- Preliminary: Over-reconstruction densification splits large Gaussians into smaller ones, increasing their count while preserving total volume.This operation targets large regions inadequately represented by a few Gaussians.
- Frequency Regularization: Frequency regularization increases pixel gradients in over-reconstruction regions, helping distinguish them from well-reconstructed regions more effectively than spatial pixel-wise L1 loss.The comparison uses the pre-15000-iteration interval, matching the Gaussian densification schedule of 3D-GS.
- Frequency Regularization: FreGS converts rendered and ground-truth images into Fourier representations and minimizes amplitude and phase discrepancies across frequency components.The method uses a 2D discrete Fourier transform and Euclidean discrepancies between corresponding frequency components.
- Frequency Regularization: Naive frequency regularization treats amplitude and phase discrepancies without separating low and high frequencies, limiting densification and producing undesirable artifacts.Low frequencies correspond to large-scale structures, whereas high frequencies correspond to local details.
- Frequency Annealing: Frequency annealing begins with low-frequency discrepancies and progressively introduces high-frequency components for coarse-to-fine Gaussian densification.A low-pass filter and dynamic high-pass filter extract the frequency bands used during training.
4. Experiments
Experiments evaluate FreGS across multiple datasets and compare its frequency regularization, frequency annealing, and densification behavior with baselines. FreGS achieves stronger rendering quality and progressively improves densification with fewer artifacts and finer details.
- Datasets and implementation: Experiments use 11 real scenes from Mip-NeRF360 and Tanks&Temples, with every 8th photo allocated to testing.The scenes span bounded indoor and unbounded outdoor environments.
- Datasets and implementation: FreGS begins frequency regularization with low-frequency amplitude and phase discrepancies before progressively incorporating high-frequency discrepancies.Gaussian densification stops after the 15,000th iteration, while spatial L1 and D-SSIM losses remain throughout training.
- Comparisons with the state of the art: Table 1 compares FreGS with 3D-GS and four NeRF-based methods using the same training data and similar Gaussian counts for FreGS and 3D-GS.The comparison covers Mip-NeRF360, Tanks&Temples, and Deep Blending.
- Comparisons with the state of the art: FreGS outperforms competing methods in PSNR, SSIM, and LPIPS across all real scenes, with fewer artifacts and finer details.The reported comparisons include Mip-NeRF360, INGP-Base, INGP-Big, and Plenoxels.
- Ablation studies: Frequency regularization improves Gaussian splatting over the spatial-loss baseline in PSNR, SSIM, and LPIPS.The Base+FR model outperforms Base in the ablation study.
- Ablation studies: Frequency annealing further improves novel view synthesis over frequency regularization alone, while visualizations show progressively better densification and rendered images.Gaussian counts stop changing after 15,000 iterations because densification terminates.
5. Conclusion
FreGS boosts 3D Gaussian splatting through progressive frequency regularization. Its frequency annealing supports coarse-to-fine densification, reduces over-reconstruction, and consistently improves novel view synthesis.
- FreGS applies progressive frequency regularization to improve 3D Gaussian splatting from a frequency-space perspective.
- Frequency annealing progressively leverages low-to-high frequency components for coarse-to-fine Gaussian densification.The components are extracted with low-pass and high-pass filters in Fourier space.
- Minimizing the discrepancy between rendered-image and ground-truth frequency spectra mitigates over-reconstruction and improves Gaussian densification.
- FreGS achieves superior novel view synthesis and consistently outperforms the state-of-the-art across multiple widely adopted indoor and outdoor scenes.