PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on 3D Gaussian Splatting

Guikun Chen, Wenguan Wang

arXiv:2401.03890v9cs.CVcs.AIcs.GRcs.MM

TL;DR

The paper addresses the need to understand 3D Gaussian splatting amid limitations of earlier reconstruction and rendering methods, especially for latency-sensitive applications. It provides the first systematic survey of GS principles, developments, applications, benchmarks, and challenges, concluding that 3D GS offers advantages in real-time rendering and editability for practical applications.

  • Problem

    Earlier NeRF-based methods offer high-fidelity rendering but are computationally intensive, while scene editing through implicit representations is challenging.

  • Method

    The paper systematically surveys 3D GS principles, literature, applications, benchmark comparisons, unresolved challenges, and future research directions.

  • Results

    The survey’s analysis and quantitative studies report 3D GS advantages over NeRF-based methods in real-time rendering and editability, particularly for latency-sensitive practical applications.

  • Takeaways & Limitations

    3D GS is presented as a transformative explicit-radiance-field technology with practical relevance for computer graphics and computer vision.

  • Takeaways & Limitations

    Approximations in GS can produce aliasing and blur while limiting illumination-estimation accuracy and the modeling of challenging objects.

Abstract

from arXiv · show

3D Gaussian splatting (GS) has emerged as a transformative technique in radiance fields. Unlike mainstream implicit neural models, 3D GS uses millions of learnable 3D Gaussians for an explicit scene representation. Paired with a differentiable rendering algorithm, this approach achieves real-time rendering and unprecedented editability, making it a potential game-changer for 3D reconstruction and representation. In the present paper, we provide the first systematic overview of the recent developments and critical contributions in 3D GS. We begin with a detailed exploration of the underlying principles and the driving forces behind the emergence of 3D GS, laying the groundwork for understanding its significance. A focal point of our discussion is the practical applicability of 3D GS. By enabling unprecedented rendering speed, 3D GS opens up a plethora of applications, ranging from virtual reality to interactive media and beyond. This is complemented by a comparative analysis of leading 3D GS models, evaluated across various benchmark tasks to highlight their performance and practical utility. The survey concludes by identifying current challenges and suggesting potential avenues for future research. Through this survey, we aim to provide a valuable resource for both newcomers and seasoned researchers, fostering further exploration and advancement in explicit radiance field.

1 INTRODUCTION

3D Gaussian splatting emerges as an explicit alternative to computationally intensive and difficult-to-edit NeRF representations, combining learnable 3D Gaussians with differentiable rendering. The survey systematizes its principles, developments, applications, benchmarks, challenges, and future directions.

  • Motivation: NeRF methods provide high-fidelity continuous volumetric rendering but require substantial computation and are difficult to edit intuitively.NeRF-based methods can require long training times and significant resources, while their weights do not map directly to scene properties.
  • 3D Gaussian Splatting: 3D GS represents scenes explicitly with millions of learnable 3D Gaussians, unlike implicit coordinate-based models.Its highly parallelized workflows support more efficient computation and rendering.
  • 3D Gaussian Splatting: 3D GS combines differentiable pipelines with point-based rendering while retaining volumetric radiance fields’ fitting capability and avoiding ray-marching overhead.The approach avoids unnecessary calculations in empty space while supporting high-quality image synthesis.
  • Applications: Real-time rendering without compromising visual quality supports applications including virtual and augmented reality, cinematic rendering, and latency-sensitive systems.The survey also associates explicit representation with flexible control over objects and scene dynamics.
  • Survey scope: The paper presents the first systematic and comprehensive survey of 3D GS, organizing principles, developments, applications, performance comparisons, and open questions.Its macro-level taxonomies and frameworks provide a roadmap for relationships among approaches, while coverage focuses on research from 2023 through 2024.
  • Survey scope: The survey is inevitably biased because 3D GS is nascent and rapidly evolving, despite efforts to provide a balanced perspective.The authors maintain repositories containing the survey organization and performance comparisons with analysis data.

2 BACKGROUND

This section introduces implicit and explicit radiance fields, then explains 3D Gaussian Splatting as an explicit representation paired with differentiable rendering. It also contrasts volumetric ray-marching with point-based rasterization, highlighting the latter’s speed advantages and remaining computational challenges.

  • Radiance Field: Implicit radiance fields use neural networks to map spatial coordinates and viewing directions to color and volume density.NeRF provides a differentiable, compact scene representation, but its rendering commonly incurs high computational load.
  • Radiance Field: Explicit radiance fields store radiance directly in discrete structures such as voxel grids or point sets, enabling faster access at the cost of higher memory usage and potentially lower resolution.Plenoxels exemplifies this approach by directly optimizing a sparse voxel grid with spherical-harmonic coefficients while converging much faster than NeRF.
  • 3D Gaussian Splatting: 3D Gaussian Splatting represents scenes with learnable 3D Gaussians and combines neural-network optimization with explicit, structured data storage.The survey describes this hybrid design as targeting real-time, high-quality rendering with less training time, particularly for complex scenes and high-resolution outputs.
  • Rendering Algorithms: Volumetric rendering computes image colors by integrating radiance along camera rays, while ray-marching approximates this integral through discrete samples.The approach becomes computationally expensive for high-resolution images because it requires systematic sampling along rays.
  • Challenges: The section frames computational efficiency, scalability, and robustness as continuing challenges for radiance-field representations and rendering methods.Reported constraints include expensive ray-marching, lighting sensitivity in color-based methods, and computational overhead in feature-based methods.
  • Rendering Algorithms: Point-based rendering rasterizes point clouds and can exploit parallel computation, whereas 3D GS uses Gaussian primitives with explicit attributes and alpha-blending.The survey notes that this rendering approach shares the image-formation model of NeRF-style volumetric rendering but demonstrates substantial speed advantages.

3 3D GAUSSIAN SPLATTING: PRINCIPLES

3D Gaussian splatting represents scenes with learnable 3D Gaussians and uses differentiable rasterization to generate images. Its tile-based rendering pipeline enables highly parallel computation and real-time, high-resolution rendering.

  • Scene Representation: 3D GS represents scenes with learnable 3D Gaussians whose center, opacity, covariance, and view-dependent color are optimized through back-propagation.Spherical harmonics allow color to vary with viewing angle and capture non-Lambertian effects.
  • Forward Process: The forward process culls out-of-view Gaussians, projects remaining ellipsoids into image-space ellipses, and computes projected covariance using an affine approximation.The affine mapping uses the Jacobian of a first-order Taylor approximation because standard projective mappings cannot directly project covariance matrices.
  • Pixel Rendering: Pixel rendering sorts overlapping Gaussians by depth and combines their learned colors with alpha blending to produce each pixel.The per-pixel formulation is difficult to parallelize because constructing sorted Gaussian lists is costly.

4 3D GAUSSIAN SPLATTING: DIRECTIONS

Recent 3D GS research extends the representation toward sparse inputs, memory efficiency, and broader scalability. These directions expose trade-offs between geometric fidelity, inference speed, storage reduction, and practical deployment constraints.

  • Research Directions: The survey organizes extensions into sparse-input, memory-efficient, photorealistic, optimization, property-augmented, hybrid, and rendering-algorithm directions.The authors note that their selected overview is inevitably biased and refer readers to a more comprehensive collection on Github.
  • 3D GS for Sparse Input: Sparse-input methods use depth or other regularization, generative augmentation, or learned feed-forward priors to reconstruct scenes from limited views.Regularization-based approaches favor geometric quality but require per-scene optimization, while learned-prior methods favor speed and cross-scene generalization.
  • 3D GS for Sparse Input: Feed-forward Gaussian models can reconstruct 3D scenes from one input image, but pixel-aligned Gaussians may be poorly distributed for representing high-frequency details and smooth regions.Generative view augmentation is computationally intensive and bounded by the capabilities of its generative model.
  • Memory-efficient 3D GS: Memory-efficient approaches reduce Gaussian counts through pruning or shared properties stored in clustered local anchors or hash grids.Other methods compress Gaussian colors and parameters with codebooks, sensitivity-aware quantization, or adaptive quantization modules.
  • Memory-efficient 3D GS: Compression methods often achieve storage reduction factors of 10-20×, while deployment on edge devices remains constrained by tight memory budgets.Training-phase memory efficiency and the trade-off between compression efficiency and visual fidelity remain open problems.

4.3 Photorealistic 3D GS

Photorealistic 3D GS research addresses artifacts and difficult appearance conditions caused by approximations in the rasterization-based pipeline. Current solutions target aliasing, blur, reflections, and transparency, but no universal solution yet covers these failure modes.

  • Aliasing: Discrete single-point pixel sampling makes 3D GS susceptible to aliasing across resolutions, producing blurred or jagged edges.Training-time methods modify sampling with multi-scale Gaussians and 2D Mip filters, while inference-time filtering can improve fidelity.
  • Blur: Real-world captures often contain motion or defocus blur, despite strong performance on carefully curated datasets.Recent approaches incorporate blur modeling during training through coarse-to-fine kernel optimization and photometric bundle adjustment.
  • Challenging Objects: Rasterization-based approximations limit accurate modeling of illumination and challenging reflective, transparent, glossy, or anisotropic objects.Extensions add normal-based shading, mirror attributes, view-dependent appearance fields, or relightable representations.
  • Open Challenges: Existing efforts generally address individual failure modes rather than providing a universal solution for aliasing, blur, reflections, and transparency.In practice, the survey describes diagnosing scene-specific issues and applying targeted optimization strategies.
  • Open Challenges: Robust reconstruction of reflective and transparent objects remains challenging, and dynamic or relightable scenes further increase system complexity.The stated goal is more general reconstruction through fundamental 3D GS improvements or new architectures.

4.4 Improved Optimization Algorithms

Improved optimization research targets reconstruction quality, convergence, Gaussian allocation, and reduced dependence on external initialization tools. The survey also discusses extending Gaussian representations with semantic, linguistic, and spatiotemporal properties.

  • Optimization Challenges: Raw optimization can overrepresent some regions and underrepresent others, producing blur and visual inconsistencies.Additional frequency and geometry regularization, improved optimization procedures, and densification strategies address these issues.
  • Constraint Relaxation: SfM initialization is error-prone in complex scenes, motivating COLMAP-free approaches that jointly optimize camera poses and Gaussian parameters.Stream-continuity methods may support learning from internet-scale unposed video datasets.
  • Future Optimization Directions: Existing methods mainly reconstruct scenes from scratch, leaving few-shot reconstruction through meta representations as a promising direction.Such representations could combine scene-specific and general knowledge through adaptive meta-learning strategies.
  • Additional Properties: Quantized language embeddings can reduce computational and memory demands while improving open-vocabulary querying by smoothing semantic features across views.The smoothing is guided by uncertainty values and is intended to mitigate semantic ambiguity.
  • Additional Properties: A 4D Gaussian representation models spatiotemporal volumes, supports arbitrary rotations in space and time, and permits end-to-end training.The survey presents spatiotemporal properties as an extension for dynamic-scene modeling.

4.6 Hybrid Representation

Hybrid representation adapts 3D Gaussian splatting to downstream tasks by adding structured information such as spatial MLPs and grids. Applications include facial avatars, dynamic scenes, and stylization, while alternative rendering approaches address rasterization limitations and enable richer physical effects.

  • Structured information such as spatial MLPs and grids complements 3D Gaussians for downstream tasks incompatible with their sparsity and disorder.
  • Facial Expression Modeling: Gaussian Head Avatar combines controllable 3D Gaussians with an MLP deformation field to capture detailed facial expressions and dynamics.
  • Spatiotemporal Modeling: Deformable and dynamic Gaussian models represent spatial-temporal changes through deformation fields or time-varying positions, orientations, and additional properties.
  • Style Transfer: GS in style couples pre-trained 3D Gaussians with a multi-resolution hash grid and a small MLP to produce cohesive stylized views without compromising rendering speed.
  • Ray-tracing alternatives support physically accurate volume rendering and eliminate popping artifacts, while opening possibilities for reflections, refraction, shadows, and global illumination.
  • These advanced capabilities currently incur additional computational costs but provide building blocks for inverse rendering, material modeling, relighting, and complex scene reconstruction.

4.8 Kernel Engineering

Kernel engineering modifies Gaussian distributions, primitive shapes, or rendering operators to better represent sharp boundaries and geometric features. These changes reduce density leakage but introduce implementation, evaluation, and resource-allocation trade-offs.

  • Standard Gaussian kernels have infinite spatial support, making sharp boundaries difficult to represent and affecting consistency and extracted-mesh quality.
  • Overlapping semi-transparent splats can create cloudy artifacts near discontinuities because of low-pass behavior and cross-boundary leakage.
  • Alternative distributions and shape modifications: Alternative kernels use bounded-support, asymmetric, deformable, or heavy-tailed formulations to control attenuation, prevent cross-boundary leakage, and compactly represent hard edges.
  • Non-Gaussian geometric primitives and operator redesigns: Non-Gaussian primitives and redesigned operators use smooth convexes, image pyramids, or sorting-free splatting to accommodate distinct geometric features.
  • Alternative parameterizations require custom renderers and primitive-management heuristics, while comparisons remain confounded by differing primitive budgets.
  • A proposed direction is adaptive allocation of heterogeneous primitives, reserving computationally complex kernels for geometrically intricate regions.

5 APPLICATION AREAS AND TASKS

3D Gaussian splatting has expanded across robotics, dynamic reconstruction, editing, avatars, medicine, large-scale scenes, physics, and other scientific disciplines. Its explicit representation supports efficient, editable applications, but scalability, dynamics, semantics, and spatial organization remain challenges.

  • Applications span robotics, dynamic scene reconstruction, generation and editing, avatars, medical systems, large-scale reconstruction, physics, and other scientific disciplines.The survey also provides a GitHub collection and representative datasets organized by its taxonomy.
  • Robotics: Explicit Gaussian representations provide direct spatial and structural scene data while balancing reconstruction fidelity with computational efficiency for robotics.
  • Robotics: GS-based manipulation can bypass auxiliary pose estimation and support rapid single-stage grasping through geometric and semantic Gaussian attributes.
  • Dynamic scene reconstruction: Dynamic-scene methods either use spatial MLPs or grids for deformation or encode temporal changes through specialized Gaussian properties and rendering.
  • Dynamic scene reconstruction: Fine-grained dynamic primitives limit scalability and robustness, making it difficult to balance efficiency and precision in large, unstructured environments.
  • Object-centric frameworks and physics-based constraints are proposed to disentangle dynamic and static entities and support responsive interactions.

6 PERFORMANCE COMPARISON

The survey compares 3D Gaussian splatting methods across localization, static and dynamic rendering, avatars, and endoscopic reconstruction. GS methods generally show strong accuracy or speed advantages, although results depend on task, implementation, and benchmark conditions.

  • Uniform comparison is impracticable across all algorithms because applications use task-specific designs, while quantitative scores vary with implementations and hyperparameters.
  • Localization: ∼50% trajectory-error improvement reduces SplaTAM’s error from 0.52cm to 0.36cm versus the previous SOTA in localization.
  • Static rendering: 3D Gaussian systems generally outperform dense SLAM competitors in static rendering, with Gaussian-SLAM establishing new SOTA by a large margin.
  • Static rendering: GSSLAM is about 578 times faster than Point-SLAM while achieving very competitive accuracy, without requiring depth-guided ray sampling for novel-view synthesis.
  • Dynamic rendering: 6.83dB PSNR improvement lets D-3DGS outperform FFDNeRF in dynamic-scene rendering, supporting structured modeling of Gaussian deformation.
  • Human avatar modeling: 1.21dB PSNR improvement lets GART outperform Instant-NVR, while GS-based avatar modeling also offers inference speed and editability.
  • Endoscopic reconstruction: Approximately 200-fold higher speed with 10% of GPU resources is reported for EndoGaussian, easing hardware demands for surgical deployment.

7 FUTURE RESEARCH DIRECTIONS

Future work spans scalable Gaussian-based world models, physics- and semantics-aware representations, learned physical priors, volumetric object modeling, autonomous-driving simulation, and broader applications. Key boundaries include sparse physics knowledge, difficulty modeling internal structures, and challenges in photorealistic, controllable simulation.

  • Recent Trends and The Big Picture: Recent models expand Gaussian scene representations toward uncalibrated-camera inference, geometric priors, temporal 4D prediction, and broader scene-level modeling.These trends support a prospective shift toward Gaussian-based world models for perception, reasoning, and action.
  • Physics- and Semantics-aware Scene Representation: Physics- and semantics-aware 3D GS could integrate scene reconstruction and understanding while supporting dynamics modeling, editing, and generation.Prior knowledge such as object shape may reduce required training viewpoints and improve geometry or surface reconstruction.
  • Modeling Internal Structures of Objects with 3D GS: Modeling object interiors remains difficult because splatting and density control produce disorganized Gaussians that align poorly with actual internal structures.Existing approaches either avoid splatting for volumetric representation or retain splatting without generating volumes.
  • 3D GS for Simulation in Autonomous Driving and beyond: Autonomous-driving simulation needs photorealistic, diverse, controllable synthetic data across lighting conditions, user-defined objects, and physics-aware scene changes.Urban and street-scene reconstruction with 3D GS is encouraging, but the survey describes these results as an early stage.
  • Empowering 3D GS with More Possibilities: Extending 3D Gaussians with linguistic, spatiotemporal, or structured information could broaden applications including registration, compression, and fluid synthesis.The survey states that the full scope of 3D GS applications remains largely untapped.

8 CONCLUSIONS

The survey presents a comprehensive overview of 3D GS and its shift from NeRF-based methods toward real-time rendering and enhanced editability. It combines analysis and quantitative studies with applications, research directions, and unresolved challenges.

  • This survey presents the first comprehensive overview of 3D GS as an explicit radiance-field technique.
  • The paper highlights a paradigm shift from traditional NeRF-based methods toward real-time rendering and enhanced editability.
  • The survey’s analysis and quantitative studies demonstrate 3D GS superiority in practical applications, particularly those sensitive to latency.
  • The paper identifies principles, prospective research directions, and unresolved challenges in 3D GS.
  • The survey is intended as a foundational resource for further exploration and progress in this rapidly evolving field.

SUPPLEMENTARY MATERIAL

The supplementary material provides figures, research-direction summaries, dataset information, and benchmark comparisons covering localization, mapping, and reconstruction.

  • Figure S1 tracks the number of published papers and GitHub stars on 3D GS.Its statistics are sourced from paper-count and GitHub-star measures.
  • Figure S2 contrasts NeRF backward mapping with 3D GS forward mapping through splatting and parallel rendering.NeRF samples along rays and queries an MLP, whereas 3D GS projects Gaussians into image space before rendering.
  • Figure S3 illustrates tile-based parallel rendering, where pixels in a tile share an ordered Gaussian list and process it once.Each pixel evaluates Gaussian contributions according to distance while the Gaussian list is traversed sequentially.
  • Table S1 summarizes major 3DGS research directions and lists representative works chronologically as a rough progress timeline.
  • Table S2 compares regularization-based and generalizability-based 3DGS methods for sparse input.
  • Table S4 compares localization methods on the static Replica dataset using absolute trajectory error in centimeters.The three best scores are marked in red, blue, and green.
Loading 2401.03890v9…