Source-linked AI summary
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering
Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, Bo Dai
TL;DR
3D Gaussian Splatting delivers high-quality, fast rendering but can use redundant Gaussians and handle view changes, texture-less regions, and lighting effects poorly. Scaffold-GS uses anchor-based local Gaussians with view- and distance-dependent prediction plus anchor refinement, achieving compact representations with comparable or better rendering and real-time speed. Its initialization remains suboptimal in large texture-less regions with extremely sparse points.
Problem
3D Gaussian Splatting can generate redundant Gaussians that neglect scene geometry, reducing robustness to substantial view changes, texture-less areas, and lighting effects.
Method
Scaffold-GS organizes local neural Gaussians around anchor points and predicts their attributes from anchor features and viewing position, with anchor growing and pruning for scene coverage.
Results
Scaffold-GS achieves comparable or better rendering than state-of-the-art algorithms with a much more compact set of Gaussians and similar rendering speed.
Takeaways & Limitations
View-adaptive neural Gaussians accommodate challenging view-dependent observations and varying scene detail while reducing storage and preserving real-time rendering.
Takeaways & Limitations
SfM-point initialization can be suboptimal in large texture-less regions and still suffers when points are extremely sparse.
Abstract
from arXiv · showhide
Neural rendering methods have significantly advanced photo-realistic 3D scene rendering in various academic and industrial applications. The recent 3D Gaussian Splatting method has achieved the state-of-the-art rendering quality and speed combining the benefits of both primitive-based representations and volumetric representations. However, it often leads to heavily redundant Gaussians that try to fit every training view, neglecting the underlying scene geometry. Consequently, the resulting model becomes less robust to significant view changes, texture-less area and lighting effects. We introduce Scaffold-GS, which uses anchor points to distribute local 3D Gaussians, and predicts their attributes on-the-fly based on viewing direction and distance within the view frustum. Anchor growing and pruning strategies are developed based on the importance of neural Gaussians to reliably improve the scene coverage. We show that our method effectively reduces redundant Gaussians while delivering high-quality rendering. We also demonstrates an enhanced capability to accommodate scenes with varying levels-of-detail and view-dependent observations, without sacrificing the rendering speed.
1. Introduction
3D Gaussian Splatting combines high rendering quality and speed but can produce redundant, view-specific Gaussians that neglect scene structure. Scaffold-GS organizes local neural Gaussians around anchors, predicts their attributes view-dependently, and refines scene coverage through anchor operations.
- 3D Gaussian Splatting achieves state-of-the-art rendering quality and speed by combining volumetric continuity with rapid Gaussian rasterization.It initializes Gaussians from Structure-from-Motion point clouds and splats them onto 2D image planes.
- 3D Gaussian Splatting can excessively expand Gaussians to fit every training view, creating redundancy and neglecting scene structure.This limits scalability, especially for complex large-scale scenes.
- View-dependent effects baked into individual Gaussian parameters reduce robustness to substantial view changes and lighting effects.
- Scaffold-GS uses anchor points to distribute local neural Gaussians within a hierarchical, region-aware 3D scene representation.The anchors are initialized from Structure-from-Motion points, and their associated Gaussians have learnable offsets.
- Scaffold-GS predicts neural Gaussian attributes on the fly from anchor features and viewing position, while growing and pruning anchors to improve scene coverage.At inference, it predicts only for anchors within the view frustum and filters trivial neural Gaussians by opacity.
- Around 100 FPS at 1K resolution is achieved with little computational overhead, while rendering quality remains on par with or surpasses original 3D-GS.The method also reduces storage by storing anchor points and MLP predictors for each scene.
2. Related work
Related work spans neural-field representations based on MLPs, spatial grids, and point primitives. These approaches trade off rendering quality, training or inference efficiency, spatial structure, and artifact behavior.
- MLP-based Neural Fields and Rendering: MLP-based neural fields use spatial coordinates and viewing direction to predict point-wise scene attributes for novel view synthesis.Their volumetric representation and MLP inductive bias support state-of-the-art performance, while the passage notes a major challenge without completing it.
- Grid-based Neural Fields and Rendering: Grid-based neural fields store scene features in spatial voxel structures that can be queried during ray marching to accelerate radiance-field training and inference.These representations are also widely used for 3D shape and geometry modeling.
- Point-based Neural Fields and Rendering: Point-based neural fields rasterize unstructured point clouds with GPU-specialized modules for fast and flexible scene rendering.
- Point-based Neural Fields and Rendering: Point-based representations can suffer from holes and outliers that create rendering artifacts, motivating differentiable point-based and surface-splatting methods.Some methods augment points with neural features and render them using 2D convolutional networks.
3. Methods
Scaffold-GS organizes neural Gaussians around scene-structured anchor points, predicts their attributes from view conditions, and refines coverage through gradient-guided growing and opacity-based pruning.
- Anchor Point Initialization: Scaffold-GS initializes anchors from a voxelized SfM point cloud, using voxel centers with local features, scales, and learnable offsets.Duplicate voxel entries are removed to reduce redundancy and irregularity.
- Neural Gaussian Derivation: Each visible anchor spawns k neural Gaussians whose positions use learnable offsets and anchor scaling factors.The spawned Gaussian positions are computed from the anchor position, offsets, and associated scale.
- Neural Gaussian Derivation: Neural Gaussian attributes are decoded in one pass from anchor features, relative camera-anchor distance, and viewing direction.Separate MLPs predict opacity, color, quaternion, and scale for the spawned Gaussians.
- Efficient Rendering: Only frustum-visible anchors are activated, and Gaussians below the opacity threshold τα are filtered before rasterization.This pre-filtering reduces MLP and rasterization overhead while maintaining rendering speed comparable to original 3D-GS.
- Anchor Points Refinement: Anchor growing adds points in quantized voxels whose neural-Gaussian gradients exceed a threshold, while random candidate elimination limits rapid expansion.The refinement uses multi-resolution voxelization to identify significant regions and suppresses unnecessary anchor growth.
- Anchor Points Refinement: Anchor pruning removes anchors whose associated neural Gaussians fail to maintain satisfactory accumulated opacity during training.Training combines rendered-color L1 loss, SSIM, and volume regularization that encourages small, minimally overlapping Gaussians.
4. Experiments
Scaffold-GS is evaluated across diverse real-world and synthetic scenes, challenging viewing conditions, and multi-scale content. It achieves comparable or better rendering quality than 3D-GS while reducing storage and retaining real-time speed through structured anchors, adaptive neural Gaussians, and refinement strategies.
- Experimental Setup: Evaluation spans 27 scenes across synthetic, indoor, outdoor, large-scale, and multi-level-of-detail datasets, using PSNR, SSIM, and LPIPS alongside efficiency measures.The main baseline is 3D-GS, with additional comparisons to Mip-NeRF360, iNGP, and Plenoxels.
- Qualitative Results: Qualitative comparisons show advantages over 3D-GS for thin geometry, fine details, texture-less regions, light effects, insufficient observations, and varying scales and viewing distances.These improvements are reported across diverse datasets and scene conditions.
- Real-World Comparisons: Scaffold-GS achieves comparable results to state-of-the-art methods on Mip-NeRF360 and surpasses them on Tanks&Temples and DeepBlending.These datasets include challenging lighting changes, texture-less regions, and reflections.
- Multi-scale Scene Contents: Scaffold-GS handles multi-scale scenes with higher quality and less storage than 3D-GS, while reducing blurry and needle-shaped artifacts at unseen viewing distances.Refined neural Gaussian properties and compact local features support extrapolation across levels of detail.
- Ablation Studies: Opacity filtering reconstructs coarse geometry from random points, and different initial k values converge to similar activated-Gaussian counts, indicating non-redundant scene representations.Anchor addition improves details and texture-less areas, while pruning removes trivial Gaussians and maintains efficiency.
- Ablation Studies: Filtering substantially improves inference speed without notable fidelity changes, but may mask pertinent neural Gaussians and remains an acknowledged risk.The authors identify this masking risk as a target for future work.
5. Conclusion
Scaffold-GS is a compact, view-adaptive 3D neural scene representation that structures Gaussians around SfM-guided anchors and decodes their attributes from view-dependent MLPs. It achieves comparable or better results than state-of-the-art methods, especially on challenging views.
- Scaffold-GS organizes 3D Gaussians around anchor points from SfM for efficient, view-adaptive rendering.Anchor-guided structure supports a compact representation.
- View-dependent MLPs decode Gaussian attributes on the fly from anchor features and camera conditions.
- A compact set of Gaussians achieves comparable or better results than state-of-the-art algorithms, particularly in challenging cases where 3D-GS usually fails.
6. Overview
The supplementary material covers implementation details, dataset preparation, and additional experimental analyses of Scaffold-GS. It is organized into implementation, data, and results sections.
- Implementation details: The first supplementary section details anchor feature enhancement, MLP structures, and anchor point refinement strategies.
- Dataset preparation: The second supplementary section describes dataset preparation steps.
- Experimental analysis: Additional experimental results and analyses are presented based on training observations.
7. Implementation details.
Scaffold-GS uses view-dependent feature banks and MLP decoders to adapt Gaussian attributes across scene granularities, while anchor refinement and voxelization control scene coverage and resolution.
- View-dependent feature bank: A view encoding MLP predicts weights from camera-anchor distance and viewing direction to blend multi-resolution anchor features.The blended feature is formed from original, downsampled, and repeatedly sliced features.
- View-dependent feature bank: Feature-bank weight distributions indicate that finer features are more activated at center view positions, capturing scene granularity through view direction and distance.
- MLP-based neural fields: Small MLP branches predict opacity, color, scale, and quaternion attributes for neural Gaussians from anchor features and camera conditions.The decoder branches use a linear-ReLU-linear structure with hidden dimension 32.
- Anchor resolution and refinement: Voxel size sets the finest anchor resolution, while refinement adds anchors to fill gaps in texture-less or poorly observed regions.Median nearest-neighbor spacing adapts resolution to point-cloud density, whereas manual settings can miss details in texture-less areas.
8. Experiments and Results
Scaffold-GS is evaluated across synthetic, indoor, outdoor, urban, landscape, and varying-level-of-detail datasets using image-quality, perceptual, and storage metrics. Training analyses report faster convergence, higher testing PSNR, and stronger generalization than 3D-GS in challenging settings.
- Datasets and metrics: The evaluation spans Mip-NeRF360, Tanks&Temples, DeepBlending, Synthetic Blender, BungeeNeRF, and VR-NeRF datasets.The datasets include synthetic objects, indoor and outdoor environments, large-scale urban scenes, landscapes, and varying levels of detail.
- Datasets and metrics: PSNR, SSIM, LPIPS, and storage size are reported across the evaluated scenes and methods.
- Training process analysis: Scaffold-GS converges faster and achieves higher testing PSNR than 3D-GS during training-process analysis.The comparison covers both training and testing views.
- Varying levels of detail: 3D-GS obtains higher training PSNR but lower testing PSNR when trained at coarser scales and evaluated at a novel finer scale, indicating overfitting to training scales.