PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Gaussian Splatting for Real-Time Radiance Field Rendering

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis

arXiv:2308.04079v1cs.GRcs.CV

TL;DR

High-quality radiance-field rendering remains costly, while faster methods fall short of real-time high-resolution synthesis. This paper uses optimized anisotropic 3D Gaussians and visibility-aware splatting to achieve state-of-the-art quality with competitive training and real-time rendering.

  • Problem

    Existing radiance-field methods either require costly training and rendering or sacrifice visual quality, and fast methods fall short of real-time high-resolution rendering.

  • Method

    The method optimizes anisotropic 3D Gaussians with adaptive density control and uses visibility-aware GPU rendering for fast novel-view synthesis.

  • Results

    The approach matches or exceeds prior implicit radiance-field quality, achieves competitive training speeds, and provides high-quality real-time novel-view rendering.

  • Takeaways & Limitations

    The results show that fast, high-quality radiance-field training and rendering do not require a continuous scene representation.

  • Takeaways & Limitations

    The method produces artifacts in poorly observed regions and can create elongated or splotchy Gaussians.

Abstract

from arXiv · show

Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.

1 INTRODUCTION

The paper targets high-quality, real-time novel-view synthesis for multi-view captured scenes, addressing the quality, training-time, and high-resolution rendering limitations of prior radiance-field methods. It introduces anisotropic 3D Gaussians, adaptive optimization and density control, and visibility-aware GPU rendering, achieving competitive training and state-of-the-art-quality real-time rendering.

  • Motivation: Prior fast radiance-field methods achieve 10-15 frames per second but fall short of real-time high-resolution rendering, while Mip-NeRF360 requires up to 48 hours of training.These methods trade visual quality or training speed against performance.
  • Method: The method represents radiance fields with anisotropic 3D Gaussians initialized from sparse Structure-from-Motion point clouds.This provides a flexible, expressive, high-quality, and unstructured scene representation.
  • Method: It optimizes Gaussian position, opacity, anisotropic covariance, and spherical-harmonic coefficients alongside adaptive density control that adds and removes Gaussians during optimization.The resulting representation contains 1-5 million Gaussians across the tested scenes.
  • Method: A fast differentiable GPU renderer uses visibility-aware tile-based rasterization, anisotropic splatting, fast sorting, and accelerated backpropagation.These design choices support high-quality novel-view synthesis and real-time rendering.
  • Results: Experiments on published datasets achieve equal or better quality than leading implicit radiance-field methods while providing training speeds similar to the fastest methods and high-quality real-time rendering.The paper describes this as the first real-time rendering with high quality for novel-view synthesis.

2 RELATED WORK

Prior novel-view synthesis evolved from light fields and SfM-based point clouds to deep, volumetric, and accelerated radiance-field representations. Point-based rendering offers efficient discrete geometry processing, while prior methods retain limitations from MVS, ray marching, or restricted scene settings that motivate this work.

  • Traditional reconstruction: Light fields and Structure-from-Motion established early foundations for novel-view synthesis from densely sampled or unstructured captures and photo collections.SfM estimates a sparse point cloud during camera calibration, initially used for simple 3D visualization.
  • Radiance-field methods: Deep-learning approaches used CNNs for blending or texture-space solutions, but MVS geometry and CNN rendering can introduce reconstruction drawbacks and temporal flickering.The cited limitations include dependence on MVS-based geometry and temporal instability in final rendering.
  • Radiance-field methods: Volumetric radiance fields represent geometry with continuous differentiable density fields, but ray marching is costly because it requires many volume samples.NeRFs followed earlier volumetric approaches using deep learning coupled with volumetric ray marching.
  • Radiance-field methods: Recent acceleration methods use spatial data structures, alternative encodings, and smaller MLPs, including InstantNGP’s hash and occupancy grids and Plenoxels’ sparse voxel grid.These designs can reduce MLP capacity or eliminate neural networks altogether, while both highlighted methods use Spherical Harmonics.
  • Point-based rendering: Point-based rendering efficiently handles disconnected, unstructured geometry, but differentiable variants often inherit MVS artifacts in featureless, shiny, or thin regions.Reported artifacts include over- or under-reconstruction, despite fast or real-time view synthesis in some methods.
  • Point-based rendering: Point-based alpha blending and NeRF-style volumetric rendering share an image-formation model, but points avoid NeRF’s expensive sampling through discrete, editable geometry.The paper contrasts NeRF’s implicit empty or occupied space and random sampling with points’ ability to create, destroy, and displace geometry.

3 OVERVIEW

The method takes images of a static scene with SfM-calibrated cameras and uses the resulting sparse point cloud to initialize a compact, flexible representation of 3D Gaussians.

  • Input and representation: The input comprises images of a static scene and corresponding cameras calibrated by SfM, which also produces a sparse point cloud.The sparse point cloud is produced as a side effect of camera calibration.
  • Input and representation: Each 3D Gaussian is defined by a position, covariance matrix, and opacity α, enabling a flexible optimization regime.The Gaussians are created from the sparse points produced during calibration.
  • Input and representation: The resulting representation is reasonably compact, partly because it uses highly anisotropic volumetric splats.

4 DIFFERENTIABLE 3D GAUSSIAN SPLATTING

The method represents scenes with differentiable 3D Gaussians initialized from sparse SfM points, avoiding normals while enabling fast projection to 2D splats for rendering. It optimizes anisotropic covariance through a valid scaling-and-rotation parameterization that adapts to varied scene geometry and remains compact.

  • Gaussian representation: 3D Gaussians represent scene geometry without normals and can be projected to 2D splats for fast α-blending.The representation starts from sparse SfM points and combines differentiability with explicit, unstructured primitives.
  • Gaussian projection: Projecting 3D Gaussians with the viewing transformation and Jacobian produces a 2×2 image-space variance matrix with planar-point structure.The matrix preserves the same structure and properties as representations based on planar points with normals.
  • Covariance optimization: Covariance optimization uses separate scaling and rotation parameters because unconstrained gradient descent can produce invalid non-positive-semidefinite matrices.A 3D vector controls scaling and a normalized quaternion controls rotation, which are converted into valid covariance matrices.
  • Optimization: Anisotropic covariance adapts optimized 3D Gaussians to different scene geometries while producing a fairly compact representation.The method also derives parameter gradients explicitly to reduce automatic-differentiation overhead during training.

5 OPTIMIZATION WITH ADAPTIVE DENSITY CONTROL OF 3D GAUSSIANS

The method optimizes 3D Gaussians and view-dependent color while interleaving optimization with adaptive density control. Gradient-guided cloning, splitting, and culling produce a denser, compact scene representation.

  • Optimization: Optimization jointly adjusts Gaussian positions, opacities, covariances, and spherical-harmonics color coefficients while interleaving parameter updates with density control.The color representation captures view-dependent scene appearance.
  • Optimization: Fast rasterization is the optimization bottleneck, while periodic opacity culling and removal of oversized Gaussians help control the total primitive count.The implementation uses stochastic gradient descent with GPU-accelerated frameworks and custom CUDA kernels.
  • Adaptive density control: The method starts from sparse SfM points, densifies every 100 iterations after warm-up, and removes Gaussians with opacity below ε_α.This adaptively increases Gaussian density while culling essentially transparent primitives.
  • Adaptive density control: High view-space positional gradients identify both under-reconstructed and over-reconstructed regions as candidates for Gaussian densification.The tested gradient threshold is τ_pos = 0.0002.
  • Adaptive density control: Small Gaussians in under-reconstructed regions are cloned and moved along the positional gradient to cover missing geometry.Cloning preserves the Gaussian’s size while adding a new primitive.
  • Adaptive density control: Large Gaussians in high-variance regions are replaced by two smaller Gaussians whose scales are divided by φ = 1.6.The new positions are initialized by sampling from the original Gaussian’s probability density function.

6 FAST DIFFERENTIABLE RASTERIZER FOR GAUSSIANS

The method introduces a tile-based Gaussian rasterizer that pre-sorts splats per image, supports approximate α-blending, and enables efficient backpropagation over arbitrarily many blended Gaussians. It uses tile and view-frustum culling, depth-and-tile GPU sorting, front-to-back traversal, and opacity-based stopping without limiting gradient-bearing primitives.

  • Rasterizer design: The tile-based rasterizer pre-sorts Gaussian primitives for an entire image, avoiding per-pixel sorting and enabling backpropagation over an arbitrary number of blended Gaussians.It uses low additional memory consumption and supports approximate α-blending, including for anisotropic splats.
  • Culling and sorting: The renderer divides the screen into 16×16 tiles and culls Gaussians against the view frustum and each tile.It retains Gaussians whose 99% confidence interval intersects the view frustum and uses a guard band to reject extreme positions.
  • Culling and sorting: Each Gaussian instance receives a key combining view-space depth and tile ID, followed by a single fast GPU Radix sort without additional per-pixel ordering.Blending therefore uses the initial ordering and can be approximate in some configurations, with approximations becoming negligible as splats approach individual-pixel size.
  • Rasterization and gradients: One thread block rasterizes each tile by loading Gaussian packets into shared memory and accumulating pixel color and α through front-to-back traversal.Traversal stops when the target saturation of α is reached, while the backward pass recovers the full per-pixel sequence of blended points.
  • Rasterization and gradients: The method does not limit the number of blended primitives receiving gradient updates, allowing arbitrary, varying depth complexity without scene-specific hyperparameter tuning.During backward traversal, pixels begin expensive overlap testing only for points no deeper than the last point contributing during the forward pass.

7 IMPLEMENTATION, RESULTS AND EVALUATION

The evaluation compares the method with established quality and fast-rendering baselines across real and synthetic scenes, showing competitive or better quality with substantially shorter training and real-time rendering. Ablations identify anisotropic covariance and spherical harmonics as important, while limitations include artifacts, memory demands, and convergence issues in large scenes.

  • Real-World Scenes: 35–45min training achieves quality on par with or slightly better than Mip-NeRF360, whose average training time is 48 hours and rendering time is 10s/frame.The method reaches comparable quality to InstantNGP and Plenoxels after 5–10m, while further training enables state-of-the-art quality.
  • Results: 180–300 FPS is achieved when rendering trained synthetic scenes.The reported result concerns the trained synthetic scenes shown in the evaluation.
  • Anisotropic Covariance: Anisotropic covariance significantly improves the 3D Gaussians’ ability to represent scene structure compared with using one scalar radius across all axes.The ablation removes anisotropy by optimizing a single scalar controlling the Gaussian radius on all three axes.
  • Spherical Harmonics: Spherical harmonics improve overall PSNR by compensating for view-dependent effects.The effect is reported in Table 3.
  • Limitations: The method can produce artifacts in poorly observed regions, elongated or splotchy Gaussians, popping artifacts, and over 20 GB peak GPU memory use during large-scene training.Large scenes may also require a reduced position learning rate to converge; popping is linked to guard-band rejection and visibility-order changes.

8 DISCUSSION AND CONCLUSIONS

The paper presents a real-time, high-quality radiance-field rendering approach across diverse scenes and capture styles, with training times competitive with both the fastest prior methods and existing solutions. Its 3D Gaussian representation preserves volumetric optimization properties while enabling fast splat-based rasterization, though Python-based optimization leaves substantial room for speedup.

  • Contributions: The approach delivers real-time, high-quality radiance-field rendering across diverse scenes and capture styles with competitive training times.The authors characterize it as the first approach achieving this combination.
  • Representation: 3D Gaussians preserve volumetric-rendering properties for optimization while directly enabling fast splat-based rasterization.This demonstrates that continuous representations are not strictly necessary for fast, high-quality radiance-field training.
  • Limitations and Future Speedups: ∼80% of training time is spent in Python code, while only rasterization uses optimized CUDA kernels.Porting the remaining optimization entirely to CUDA could enable significant further speedup where performance is essential.
  • Performance Design: GPU-oriented real-time rendering principles and software-rasterization architecture drive performance in both training and rendering.These design choices provide a competitive performance edge over previous volumetric ray-marching approaches.
  • Conclusion: The method combines rendering quality matching the best expensive previous methods with training times competitive with the fastest existing solutions.The conclusion reiterates real-time rendering as the central contribution.

A DETAILS OF GRADIENT COMPUTATION

The section derives gradients for Gaussian covariance parameters by applying the chain rule through world/view transformations and the covariance factorization. It then obtains rotation gradients from the quaternion-to-rotation-matrix conversion, with quaternion normalization handled directly.

  • A DETAILS OF GRADIENT COMPUTATION: The chain rule computes derivatives with respect to Gaussian scaling and rotation through world/view covariance transformations.Σ and Σ′ denote world- and view-space covariance matrices; W is the viewing transformation and J is the projective Jacobian.
  • A DETAILS OF GRADIENT COMPUTATION: Using U = JW and the symmetric upper-left 2×2 block of UΣU^T simplifies the covariance derivative expressions.Matrix elements are represented with subscripts in the simplified formulation.
  • A DETAILS OF GRADIENT COMPUTATION: Writing M = RS and Σ = MM^T provides a compact route for computing dΣ gradients.The covariance matrix and its gradient are symmetric, allowing shared terms to be simplified.
  • A DETAILS OF GRADIENT COMPUTATION: Rotation gradients are derived by converting the unit quaternion q into its rotation matrix R and differentiating its components.The quaternion consists of real part q_r and imaginary parts q_i, q_j, and q_k.
  • A DETAILS OF GRADIENT COMPUTATION: Quaternion normalization gradients are derived straightforwardly.The normalization step completes the gradient computation for quaternion-based rotation parameters.

B OPTIMIZATION AND DENSIFICATION ALGORITHM

Algorithm 1 jointly optimizes 3D Gaussian parameters from SfM points through iterative view sampling, rasterization, loss computation, and Adam updates. At refinement iterations, it prunes unsuitable Gaussians and densifies regions according to opacity, size, position-gradient, and scale criteria.

  • Initialization: The procedure initializes Gaussian positions from SfM points and initializes covariances, colors, and opacities before iterative optimization.The iteration count starts at zero and continues until convergence.
  • Optimization: Each iteration samples a training view, rasterizes the Gaussian model, computes loss against the training image, and updates parameters with Adam backpropagation.The optimized quantities include the Gaussian model, positions, covariances, colors, and opacities.
  • Densification and pruning: During refinement iterations, Gaussians are pruned when opacity is below ε or their mean and covariance are too large.The pruning condition tests α < ε or IsTooLarge(μ, Σ).
  • Densification and pruning: Densification occurs when the position-loss gradient exceeds τ_p, splitting over-reconstructed Gaussians and cloning under-reconstructed Gaussians.The branch uses scale magnitude ∥S∥ > τ_S to distinguish splitting from cloning.

C DETAILS OF THE RASTERIZER

The rasterizer sorts small splats per frame with GPU radix sort, organizes them into overlapping 16×16 tiles, and processes compact per-tile lists in parallel. Numerical safeguards skip negligible opacities and clamp large α values during forward and backward passes.

  • Sorting: Per-frame GPU radix sorting assigns splats to every overlapping 16×16 pixel tile, trading moderate Gaussian duplication for simpler control flow and high parallelism.The design targets workloads with many small splats.
  • Sorting: Parallel range detection produces compact per-tile Gaussian lists and eliminates sequential primitive-processing steps compared with Lassner and Zollhofer 2021.One thread per 64-bit sorted-array element compares neighboring tile identifiers.
  • Numerical stability: During blending, updates with α<1/255 are skipped and α is clamped to 0.99 to improve numerical stability in both forward and backward passes.These safeguards prevent problems such as division by zero when reconstructing intermediate opacities for gradients.

D PER-SCENE ERROR METRICS

Tables 4–9 report SSIM, PSNR, and LPIPS error metrics across evaluated techniques and real-world scenes. Across the full Mip-NeRF360 dataset, the reported averages are PSNR 27.58, SSIM 0.790, and LPIPS 0.240.

  • D PER-SCENE ERROR METRICS: Tables 4–9 list collected SSIM, PSNR, and LPIPS metrics for evaluated techniques across Mip-NeRF360, Tanks&Temples, and Deep Blending scenes.Tables 4–6 cover Mip-NeRF360; Tables 7–9 cover Tanks&Temples and Deep Blending.
  • D PER-SCENE ERROR METRICS: 27.58 is the full Mip-NeRF360 dataset average for PSNR.The averages combine copied Mip-NeRF360 numbers with results from the authors’ runs.
  • D PER-SCENE ERROR METRICS: 0.790 is the full Mip-NeRF360 dataset average for SSIM.The reported averages cover the full Mip-NeRF360 dataset.
  • D PER-SCENE ERROR METRICS: 0.240 is the full Mip-NeRF360 dataset average for LPIPS.The reported averages cover the full Mip-NeRF360 dataset.
Loading 2308.04079v1…