Source-linked AI summary
gsplat: An Open-Source Library for Gaussian Splatting
Vickie Ye, Ruilong Li, Justin Kerr, Matias Turkulainen, Brent Yi, Zhuoyang Pan, Otto Seiskari, Jianbo Ye, Jeffrey Hu, Matthew Tancik, Angjoo Kanazawa
TL;DR
Gaussian Splatting research needs efficient, user-friendly, and extensible tooling for developing reconstruction and novel-view-synthesis methods. gsplat addresses this with a modular PyTorch-facing library backed by optimized CUDA kernels, achieving the same rendering performance as the original implementation with less memory and shorter training time.
Problem
Gaussian Splatting research requires an efficient, user-friendly, and extensible library for developing models and supporting evolving research features.
Method
gsplat combines Python bindings compatible with PyTorch, optimized CUDA kernels, a native PyTorch implementation, and modular APIs for Gaussian Splatting features.
Results
gsplat achieves the same rendering performance as the original implementation while using less memory and significantly reducing training time.
Takeaways & Limitations
gsplat provides an open-source, modular foundation for integrating and extending Gaussian Splatting methods in external research projects.
Abstract
from arXiv · showhide
gsplat is an open-source library designed for training and developing Gaussian Splatting methods. It features a front-end with Python bindings compatible with the PyTorch library and a back-end with highly optimized CUDA kernels. gsplat offers numerous features that enhance the optimization of Gaussian Splatting models, which include optimization improvements for speed, memory, and convergence times. Experimental results demonstrate that gsplat achieves up to 10% less training time and 4x less memory than the original implementation. Utilized in several research projects, gsplat is actively maintained on GitHub. Source code is available at https://github.com/nerfstudio-project/gsplat under Apache License 2.0. We welcome contributions from the open-source community.
1 Introduction
gsplat is an open-source Gaussian Splatting library designed to make model development efficient, user-friendly, modular, and extensible. It builds on prior implementations while supporting research and community contributions.
- gsplat provides an efficient, user-friendly library with a simple, modifiable API for PyTorch-based Gaussian Splatting projects.
- 39 contributors and over 1.6k GitHub stars followed gsplat’s initial release in October 2023.
- gsplat is released under the Apache License 2.0 and welcomes contributions from students, researchers, and the open-source community.
- Unlike earlier open-source efforts, gsplat combines performance improvements with an easy-to-use, modular API for external extensions and modifications.
2 Design
gsplat combines optimized CUDA kernels with Python and PyTorch interfaces to support efficient, modular Gaussian Splatting development. Its examples and tests also support learning and experimentation.
- gsplat exposes optimized CUDA operations through Python bindings while retaining a native PyTorch implementation for new research ideas.
- The standalone library runs on Windows and Linux, installs from PyPI, and integrates Gaussian Splatting functionality into external projects.
- Well-documented examples, CUDA correctness tests, and online documentation make gsplat an educational resource for new researchers.
- Figure 1 demonstrates initializing one Gaussian and rendering it as an RGB image using 13 lines of gsplat code.
3 Features
gsplat provides modular Gaussian Splatting features that users can enable through simple API calls. These include interchangeable densification strategies, pose optimization, and anti-aliased rendering support.
- Densification strategies: gsplat supports ADC, Absgrad, and MCMC densification strategies, which users can switch through its modular API.
- Densification strategies: Figure 2 presents a code block for training a Gaussian model with a selected densification strategy.
- Pose optimization: Differentiable rendering enables gradients to update camera rotations and translations, supporting optimization of initial camera poses.
4 Evaluation
On MipNeRF360, gsplat is evaluated against the original 3DGS implementation using matched settings and the standard ADC strategy. It achieves the same rendering performance while reducing memory use and training time.
- Overall comparison: The evaluation compares gsplat with the original implementation on MipNeRF360 using equivalent configuration settings and standard ADC densification.
- Overall comparison: gsplat achieves the same rendering performance as the original implementation while using less memory and significantly reducing training time.The comparison averages novel-view synthesis, memory usage, and training time over 7 scenes at 7k and 30k iterations on an A100 GPU.
- Feature comparison: Table 2 analyzes the impact of gsplat features on the MipNeRF360 dataset averaged over 7 scenes.
Supplementary Material
The supplementary material documents gsplat’s implementation details, quantitative comparisons, and evolving library conventions, including support for large-scale multi-GPU training.
- Supplementary Material: The appendices provide additional feature details, quantitative comparisons, and mathematical descriptions of gsplat’s forward and backward passes.They also explain conventions used throughout the library.
- Supplementary Material: As of July 2024, gsplat supports multiple densification strategies.
A.1.1 ADC
Adaptive Density Control tracks accumulated positional gradients and uses thresholds to split, clone, or remove Gaussian primitives during training.
- A.1.1 ADC: ADC averages positional gradients for each Gaussian primitive across multiple camera-view renderings.The tracked quantity is the positional gradient of the primitive’s center.
- A.1.1 ADC: Gaussians exceeding the positional-gradient threshold are split when their largest scale exceeds 0.01 and cloned otherwise.The default positional-gradient threshold is 0.0002.
- A.1.1 ADC: ADC periodically removes Gaussian primitives whose opacity falls below 0.005 during training.
A.1.2 Absgrad
This section describes alternative gradient accumulation, Bayesian densification, pose optimization, and depth-rendering quantities supported by gsplat.
- A.1.2 Absgrad: View-space positional gradients are tracked across M camera views to set splitting and duplication criteria.
- A.1.2 Absgrad: Absolute-sum accumulation avoids cancellation between negative and positive view-space gradients that can weaken the densification heuristic.gsplat supports both the original and absolute-sum versions.
- A.1.2 Absgrad: The Absgrad feature is enabled through a simple API call.
- A.1.2 Absgrad: SGLD densification adds noise to Gaussian-center updates, controlled by λ_noise and λ_lr.
- A.1.2 Absgrad: Differentiable rendering allows gradients to optimize camera rotation and translation alongside Gaussian parameters.This supports optimization of initial camera poses for datasets with pose uncertainty.
- A.1.2 Absgrad: Accumulated depth and expected depth are defined for each pixel using depth-sorted Gaussians and their transparencies.
A.4 Anti-aliasing
gsplat supports classic and anti-alias rendering modes that modify screen-space Gaussian sizes, with a default scale parameter chosen to span one pixel.
- A.4 Anti-aliasing: Classic and anti-alias modes modify the screen-space 2D Gaussian size G2D.
- A.4 Anti-aliasing: The anti-aliasing scale parameter s defaults to 0.3 during training.
- A.4 Anti-aliasing: The default s value is intended to make a 2D Gaussian span the width of a single pixel.
Appendix B. Additional Evaluations
This appendix evaluates gsplat’s rendering, rasterization, and differentiable backward computations, including additional feature ablations and memory measurements. It describes how 3D Gaussians are projected, composited, and differentiated through the rendering pipeline.
- Additional evaluations: Additional evaluations ablate default settings, Absgrad and MCMC densification, antialiased rendering, novel-view metrics, and memory usage.The evaluations use the MipNeRF360 dataset and report results in Tables 3–6.
- Forward pass: 3D Gaussians are parameterized by means, covariances, opacity, and features, then projected into 2D and composited front to back.Covariances are represented through scaling vectors and rotation quaternions; visible projected Gaussians are sorted by depth.
- Projection of Gaussians: Camera extrinsics transform world-space Gaussians into camera coordinates, while intrinsics project them into image coordinates.The projection uses a first-order Taylor approximation with an affine transform J.
- Rasterization: The rasterizer bins 2D Gaussians into 16 × 16 tiles, sorts them by depth, and rasterizes the sorted Gaussians within each tile.Tile inclusion uses the axis-aligned bounding box of each projected Gaussian’s 99th-percentile ellipse.
- Backward pass: Backward computation propagates pixel loss gradients through colors, opacities, 2D means, and 2D covariances to the original Gaussian parameters.Because front-to-back compositing makes later Gaussians downstream, Gaussian gradients are computed from back to front.
Appendix E. Data Conventions
This appendix records conventions used by gsplat for rotations, coordinate conversion, and camera projection. The conventions specify quaternion representation and mappings among continuous image, pixel, and 3D coordinates.
- Rotation convention: gsplat represents Gaussian rotations with Hamilton-convention four-dimensional quaternions q = (w, x, y, z).The quaternion defines an SO(3) rotation matrix.
- Coordinate conversion: Continuous image coordinates are converted to discrete pixel coordinates under the assumption that each pixel center lies at the center of a unit-area box.The coordinate relation also uses the 3D camera coordinates t = (tx, ty, tz).
- Camera intrinsics: The pinhole camera intrinsics are represented by fx, fy, cx, and cy.These parameters appear in the mapping between pixel space, image space, and 3D coordinates.