Source-linked AI summary
Mip-Splatting: Alias-free 3D Gaussian Splatting
Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, Andreas Geiger
TL;DR
Mip-Splatting addresses sampling-rate artifacts in 3DGS, which arise from missing 3D frequency constraints and screen-space dilation. It adds a 3D smoothing filter and a 2D Mip filter, and reports competitive matched-scale performance plus significantly better out-of-distribution rendering across sampling rates.
Problem
3DGS produces strong artifacts when rendering at sampling rates different from training because it lacks 3D frequency constraints and uses 2D dilation.
Method
Mip-Splatting combines a 3D smoothing filter based on training-view sampling limits with a 2D Mip filter approximating the physical box filter.
Results
Mip-Splatting is competitive at matched sampling rates and significantly outperforms state-of-the-art methods when testing at sampling rates different from training.
Takeaways & Limitations
The method enables alias-free rendering across various sampling rates while retaining the 3DGS representation and requiring only a few code changes.
Takeaways & Limitations
The Gaussian approximation to a box filter introduces errors for small screen-space Gaussians, and sampling-rate computation adds training overhead.
Abstract
from arXiv · showhide
Recently, 3D Gaussian Splatting has demonstrated impressive novel view synthesis results, reaching high fidelity and efficiency. However, strong artifacts can be observed when changing the sampling rate, \eg, by changing focal length or camera distance. We find that the source for this phenomenon can be attributed to the lack of 3D frequency constraints and the usage of a 2D dilation filter. To address this problem, we introduce a 3D smoothing filter which constrains the size of the 3D Gaussian primitives based on the maximal sampling frequency induced by the input views, eliminating high-frequency artifacts when zooming in. Moreover, replacing 2D dilation with a 2D Mip filter, which simulates a 2D box filter, effectively mitigates aliasing and dilation issues. Our evaluation, including scenarios such a training on single-scale images and testing on multiple scales, validates the effectiveness of our approach.
1. Introduction
3D Gaussian Splatting delivers efficient, high-quality novel view synthesis but develops artifacts when sampling rates differ from training. Mip-Splatting addresses this with frequency-aware 3D smoothing and a 2D Mip filter.
- Motivation: 3DGS represents scenes with 3D Gaussians and renders them efficiently through splatting-based rasterization.Gaussian attributes are optimized using a multi-view photometric loss.
- Method: Mip-Splatting derives per-primitive frequency bounds from training views and applies a 3D low-pass filter during optimization.The filter remains intrinsic to the scene representation after training and constrains frequencies to the Nyquist limit.
- Method: A 2D Mip filter approximates the physical imaging process’s box filter to support alias-free rendering across scales.It replaces 2D dilation and targets aliasing and dilation artifacts.
- Evaluation: Experiments evaluate single-scale training with rendering at multiple resolutions, and the method is reported to outperform existing methods in this setting.The modifications require only a few changes to the original 3DGS code.
2. Related Work
Related work establishes NVS and 3DGS as efficient rendering foundations, while emphasizing that prior 3DGS evaluation largely uses matched sampling rates. Mip-Splatting studies generalization across scales and differs from existing anti-aliasing approaches in its 3D filtering strategy.
- Novel View Synthesis: NVS generates images from viewpoints different from those of the original captures and supports applications such as virtual reality, cinematography, and robotics.NeRF and later scene representations advanced the quality and efficiency of NVS.
- Novel View Synthesis: 3DGS represents scenes explicitly with 3D Gaussians and uses rasterization for real-time high-definition rendering.Its prior evaluations focus mainly on training and testing at similar sampling rates.
- Novel View Synthesis: The paper evaluates 3DGS out-of-distribution by training at one scale and testing across multiple scales.This setting targets changes in focal length or scene distance.
- Anti-aliasing in Rendering: Anti-aliasing methods generally use super-sampling or prefiltering, including EWA splatting’s screen-space Gaussian low-pass filter.Mip-Splatting instead applies a band-limited filter in 3D space with size determined by sampling constraints.
- Anti-aliasing in Rendering: Mip-NeRF-style approaches require multi-scale supervision, whereas Mip-Splatting determines filter size from pixel size for scales unobserved during training.This distinction is made for 3DGS-based rendering.
3. Preliminaries
The preliminaries frame aliasing through the Nyquist-Shannon theorem and describe how 3DGS represents, projects, filters, and optimizes Gaussian primitives. Screen-space dilation prevents undersized projections but changes Gaussian scale without changing its maximum.
- Sampling Theorem: The Nyquist-Shannon theorem requires a band-limited signal and a sampling rate at least twice its highest frequency.Low-pass filtering removes frequencies above half the sampling rate before sampling.
- 3D Gaussian Splatting: 3DGS represents a scene as scaled 3D Gaussian primitives parameterized by opacity, center, and world-space covariance.The primitives are rendered through volume splatting.
- 3D Gaussian Splatting: A semi-definite covariance parameterization uses a scaling vector and rotation matrix to keep each covariance valid.The rotation is parameterized by a quaternion.
- 3D Gaussian Splatting: For a viewpoint, Gaussians are transformed into camera coordinates and projected into ray space using a local affine approximation.The projected 2D Gaussian is used for screen-space rendering.
- 3D Gaussian Splatting: 3DGS renders view-dependent colors by alpha blending primitives in depth order.All Gaussian parameters are jointly optimized through a multi-view photometric loss.
- Dilation: Screen-space dilation enlarges projected 2D Gaussians that are smaller than a pixel while leaving their maximum unchanged.Its scale is adjusted using a scalar dilation hyperparameter.
4. Sensitivity to Sampling Rate
3DGS has a scale ambiguity that is hidden at matched sampling rates but produces erosion, brightness, dilation, and aliasing artifacts when sampling changes. Removing dilation alone is impractical for complex scenes and does not eliminate aliasing.
- Scale Ambiguity: Joint optimization and screen-space dilation allow a degenerate Dirac-like 3D Gaussian to produce a similar image, leaving its 3D scale underconstrained.The ambiguity becomes especially problematic in representations containing millions of Gaussians.
- Scale Ambiguity: 3DGS systematically underestimates Gaussian scale during optimization.This implicit shrinkage bias does not affect rendering at similar sampling rates.
- Changing Sampling Rate: When zooming in, projected dilated Gaussians become smaller in screen space and produce erosion effects with overly thin object structures.The resulting images contain high-frequency artifacts.
- Changing Sampling Rate: When sampling rates decrease, dilation spreads radiance across pixels without attenuation, causing excess brightness and dilation artifacts.The effect strongly degrades bicycle-wheel spokes in the cited example.
- Practical Constraints: Discarding screen-space dilation creates optimization challenges for complex scenes and can exceed GPU capacity because density control creates many small Gaussians.Even successful training without dilation still suffers aliasing when sampling rates decrease.
5. Mip Gaussian Splatting
Mip-Splatting addresses scale-dependent artifacts in 3D Gaussian Splatting with 3D frequency regularization and a 2D Mip filter. The method limits Gaussian frequencies using training-view sampling bounds and approximates pixel-area integration for lower-rate rendering.
- 3D Smoothing Filter: A 3D smoothing filter limits each Gaussian’s frequency below half the maximum sampling rate determined by training images.The filter is applied in 3D before screen-space projection and is intended to eliminate high-frequency artifacts when zooming in.
- 3D Smoothing Filter: Nyquist-based bounds motivate preventing primitives smaller than twice the sampling interval, since they may produce aliasing during splatting.The sampling interval depends on image resolution, focal length, and scene distance from the camera.
- 3D Smoothing Filter: The maximal sampling rate is selected from the depth- and camera-dependent sampling rates of views where each primitive is visible.The implementation recomputes each primitive’s maximal sampling rate every m iterations, assuming Gaussian centers remain relatively stable.
- 3D Smoothing Filter: The 3D smoothing operation is efficient because convolving Gaussian primitives adds their covariance matrices.The resulting filtered Gaussian becomes an intrinsic part of the representation and remains constant after training.
- 2D Mip Filter: A 2D Mip filter replaces 3DGS’s screen-space dilation to approximate the physical box filter over a single pixel and mitigate aliasing and dilation artifacts.Its scale is chosen to cover one screen-space pixel, unlike EWA’s empirically selected bandwidth-limiting filter size.
- Evaluation: On Blender multi-scale training and testing, the approach achieves state-of-the-art performance in most metrics and significantly outperforms 3DGS and 3DGS + EWA.The reported table evaluates multi-scale rendering, while the text also reports smooth results when zooming out.
6. Experiments
Mip-Splatting is evaluated across multi-scale training and testing settings on Blender and Mip-NeRF 360. It maintains comparable performance at training scale while improving rendering quality at other scales, with limitations from Gaussian box-filter approximation and sampling-rate computation overhead.
- Multi-scale evaluation: Mip-Splatting significantly surpasses previous work at non-training scales in both single-scale multi-scale evaluations.On Blender, it outperforms prior methods at lower resolutions; on Mip-NeRF 360, it surpasses previous work at higher resolutions.
- Baseline behavior: 3DGS-based methods capture finer details at training resolution, but 3DGS exhibits dilation artifacts and EWA produces oversmoothed images at lower resolutions.These effects limit their rendering quality when the sampling rate changes.
- Training-scale performance: At the original training scale, Mip-Splatting yields comparable results to prior methods on Blender and Mip-NeRF 360.The Mip-NeRF 360 same-scale evaluation also reports performance on par with 3DGS and 3DGS + EWA.
- Zoom-in evaluation: In zoom-in tests, Mip-Splatting avoids the high-frequency artifacts observed in 3DGS + EWA and produces images closer to ground truth.The models are trained on images downsampled by a factor of eight and rendered at progressively higher resolutions.
- Limitations: The Gaussian filter approximates a box filter efficiently but introduces larger errors when zooming out, especially when the Gaussian is small in screen space.Sampling-rate computation every m = 100 iterations also adds slight training overhead.
- Limitations: The 3D smoothing filter can be fused with Gaussian primitives, eliminating additional rendering overhead.A more efficient CUDA implementation and improved sampling-rate storage remain future-work directions.
7. Conclusion
Mip-Splatting modifies 3D Gaussian Splatting with 3D smoothing and 2D Mip filters for alias-free rendering across scales. It remains competitive at matched sampling rates and improves out-of-distribution rendering when testing differs from training.
- Mip-Splatting introduces a 3D smoothing filter and a 2D Mip filter to achieve alias-free rendering at arbitrary scales.The 3D filter constrains Gaussian frequencies, while the 2D filter approximates the physical imaging process.
- At the training scale, Mip-Splatting is competitive with state-of-the-art methods without a performance decrease.
- When testing uses sampling rates different from training, Mip-Splatting significantly outperforms state-of-the-art methods.The paper reports better generalization to out-of-distribution camera poses and zoom factors.
Supplementary Material
The supplementary material contains ablation studies and additional quantitative and qualitative results.
- The supplementary document presents ablation studies in Section 8.
- It reports additional quantitative and quality results in Section 9.
8. Ablation
The ablations evaluate the two proposed filters under multi-scale testing, including zoom-in and zoom-out settings. Results show that 3D smoothing addresses high-frequency artifacts, while the 2D Mip filter addresses aliasing and dilation effects.
- 8.1. 3D smoothing filter: The ablation evaluates the 3D smoothing filter under single-scale training and multi-scale testing to simulate zoom-in effects.The experiment uses the Mip-NeRF 360 dataset.
- 8.1. 3D smoothing filter: Omitting the 3D smoothing filter produces high-frequency artifacts at higher rendering resolutions.
- 8.2. 2D Mip filter: Removing 2D dilation eliminates dilation effects but causes aliasing artifacts because anti-aliasing is absent.
- 8.3. Single-scale Training and Multi-scale Testing: Mip-Splatting significantly outperforms 3DGS and 3DGS + EWA when rendering across zoom-in and zoom-out scales.The additional Mip-NeRF 360 experiment evaluates scales from 1/4× to 4× after training on images downsampled by a factor of 4.
- 8.3. Single-scale Training and Multi-scale Testing: Removing the 2D Mip filter results in aliasing artifacts, while removing 3D smoothing results in high-frequency artifacts.
9. Additional Results
Additional experiments across Blender and Mip-NeRF 360 evaluate matched-scale and multi-scale generalization. Mip-Splatting matches prior methods at training resolution while improving lower-resolution and higher-resolution rendering, especially by eliminating high-frequency artifacts.
- 9.1. Blender Dataset: On Blender, Mip-Splatting achieves state-of-the-art performance under multi-scale training and testing.
- 9.1. Blender Dataset: On Blender, Mip-Splatting outperforms 3DGS and 3DGS + EWA, particularly at lower resolutions that simulate zoom-out.
- 9.2. Mip-NeRF 360 Dataset: On Mip-NeRF 360 at matched training and testing scales, Mip-Splatting performs on par with 3DGS and 3DGS + EWA without decreasing performance.Indoor scenes are downsampled by 2 and outdoor scenes by 4.
- 9.2. Mip-NeRF 360 Dataset: When trained on images downsampled by a factor of 8 and rendered at higher resolutions, Mip-Splatting eliminates high-frequency artifacts and produces renderings closer to ground truth.The evaluation uses 1×, 2×, 4×, and 8× rendering resolutions to simulate zoom-in effects.
- 9.2. Mip-NeRF 360 Dataset: In multi-scale testing on Mip-NeRF 360, Mip-Splatting surpasses previous methods at scales other than the training resolution.The reported scales include 1×, 2×, 4×, and 8× in the single-scale training setup.