PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Gaussian as a New Era: A Survey

Ben Fei, Jingyi Xu, Rui Zhang, Qingyuan Zhou, Weidong Yang, Ying He

arXiv:2402.07181v2cs.CVcs.GR

TL;DR

3D Gaussian Splatting still faces challenges in geometric fidelity, dynamic-scene representation, and efficient real-time use. This paper organizes recent work into a unified taxonomy and surveys applications, methods, limitations, and future opportunities across the field.

  • Problem

    3D-GS faces limited geometric information in monocular and few-shot reconstruction and requires improvements in memory efficiency, rendering quality, novel-view synthesis cost, and dynamic-scene fidelity.

  • Method

    The paper presents a unified framework that categorizes existing 3D Gaussian research across six aspects and provides detailed application taxonomies.

  • Results

    The survey provides a comprehensive perspective on 3D Gaussian applications and research directions through systematic taxonomies.

  • Takeaways & Limitations

    The framework serves as a practical reference for organizing the expanding 3D-GS literature and identifying directions for further improvement.

  • Takeaways & Limitations

    Dynamic-scene reconstruction remains constrained by motion blur, while approaches balancing surface accuracy and rendering quality or speed involve trade-offs.

Abstract

from arXiv · show

3D Gaussian Splatting (3D-GS) has emerged as a significant advancement in the field of Computer Graphics, offering explicit scene representation and novel view synthesis without the reliance on neural networks, such as Neural Radiance Fields (NeRF). This technique has found diverse applications in areas such as robotics, urban mapping, autonomous navigation, and virtual reality/augmented reality, just name a few. Given the growing popularity and expanding research in 3D Gaussian Splatting, this paper presents a comprehensive survey of relevant papers from the past year. We organize the survey into taxonomies based on characteristics and applications, providing an introduction to the theoretical underpinnings of 3D Gaussian Splatting. Our goal through this survey is to acquaint new researchers with 3D Gaussian Splatting, serve as a valuable reference for seminal works in the field, and inspire future research directions, as discussed in our concluding section.

1 INTRODUCTION

This survey introduces 3D Gaussian Splatting and organizes its optimization methods and applications into a unified taxonomy. It reviews current techniques, identifies technical limitations and open questions, and outlines future research directions.

  • 3D Gaussian Splatting represents objects and surfaces with collections of Gaussians for efficient, detailed rendering.
  • The survey analyzes Gaussian generation, reconstruction, manipulation, perception, and human applications while discussing efficient data structures and real-time rendering.
  • The paper is structured around background, 3D-GS optimization, reconstruction, manipulation, perception, generation, and other applications.
  • Its unified framework categorizes existing 3D Gaussian work into six main aspects and provides detailed application taxonomies.
  • The review covers classical and cutting-edge approaches with fine-grained classifications and is presented as the first review of 3D Gaussians to the authors’ knowledge.
  • The survey highlights technical limitations and proposes future directions intended to inspire further advances in this rapidly evolving field.

2 BACKGROUND

This background reviews point-based rendering and introduces 3D-GS as an explicit Gaussian-based radiance-field representation rendered through projection, sorting, and alpha compositing.

  • Traditional Point-based Rendering: Point-based rendering generates images by rendering distinct geometric primitives, including ellipsoid-shaped splats that can occupy multiple pixels.Differentiable pipelines further combine elliptical splatting, probabilistic depth testing, and camera selection.
  • Neural Implicit Representation: Neural implicit rendering, exemplified by NeRF, models geometry with an implicit density field and predicts view-dependent color with an appearance field.Unlike predefined-geometry point rendering, this approach does not depend on fixed geometry for reconstruction.
  • Rendering Comparison: 3D-GS directly models opacity and uses rasterization without sampling points, whereas NeRF converts density to opacity and requires dense spatial sampling.Both approaches combine contributions along camera rays to produce pixel colors.
  • Gaussian Splatting Radiance Fields: 3D-GS represents a radiance field with many anisotropic 3D Gaussian distributions carrying means, covariances, opacities, and spherical-harmonics color parameters.Covariance is decomposed into rotation and scaling components, represented using quaternions and scaling factors.
  • Gaussian Splatting Radiance Fields: Rendering projects Gaussians onto the image plane, sorts patch-intersecting splats by depth, and alpha-composites covering Gaussians for each pixel.The process operates over 16 × 16 pixel patches and computes colors from spherical-harmonics parameters.
  • Datasets: The survey covers datasets used to evaluate 3D-GS across optimization, reconstruction, manipulation, generation, perception, and virtual humans.These datasets are summarized in Table 1.

3 REPRESENTATION

The survey organizes representation improvements around efficiency, photorealism, cost, and physics, while identifying memory, image quality, initialization, and dynamic-scene constraints.

  • Representation Goals: 3D-GS representation research targets memory-efficient real-time rendering, higher image quality, lower novel-view synthesis costs, and faithful dynamic-scene modeling.These four directions are illustrated in Fig. 3.
  • 3.1 Efficiencies: Millions of Gaussian parameters require substantial memory, motivating methods that reduce storage while preserving quality for real-time rendering.Scaffold-GS prunes expanded Gaussians using sparse SfM-derived anchor grids and predicts Gaussian attributes from anchor features.
  • 3.1 Efficiencies: Compression methods reduce 3D-GS storage through vector quantization, pruning, spherical-harmonics reduction, coefficient quantization, and octree-based position encoding.CompGS separately clusters color, spherical harmonics, scale, and rotation vectors, while LightGaussian combines pruning with additional compression steps.
  • 3.1 Efficiencies: Dynamic-scene compression separates time-invariant from time-varying Gaussian parameters and uses flow information to reduce ambiguity between consecutive frames.This avoids storing all Gaussian parameters independently at every time step.
  • Future Directions: The survey highlights open directions including compact dynamic representations, learned scene-specific dynamics, larger dynamic scenes, and physical constraints in optimization.Learned dynamics may improve deformation fidelity, while physical constraints are proposed for broader real-world settings.
  • 3.2 Photorealism: Rendering quality is challenged by aliasing, artifacts, and imperfect reflections; proposed remedies include multiscale Gaussians, 3D and 2D filtering, relightable points, visibility ray tracing, and light-surface modeling.These methods target finer detail, material appearance, shadows, specular features, and reflective surfaces.
  • 3.3 Costs: Sparse or inaccurate initial points constrain reconstruction, while sparse views can cause collapsed, over-fitted, or over-smoothed results.The survey identifies this as a central challenge for reducing image acquisition costs.
  • 3.4 Physics: Dynamic reconstruction remains limited by intrinsically sparse input point clouds, large motions, and the absence of physical constraints for plausible dynamics.Current methods largely address indoor object-level deformations and often require multiple camera views with precise poses.

4 RECONSTRUCTION

3D-GS reconstruction spans static and dynamic scenes, supported by fast rendering and novel-view synthesis. The survey reviews methods for mesh extraction, monocular and few-shot reconstruction, and large-scale dynamic scenes while identifying unresolved geometry, motion, and quality challenges.

  • Overview: 3D-GS supports realistic novel-view synthesis and rapid rendering, with training times on the order of minutes.These properties facilitate real-time rendering and dynamic-scene reconstruction.
  • Static Reconstruction: Static reconstruction remains challenging because the explicit distribution of 3D Gaussians complicates surface reconstruction and mesh extraction.SuGaR addresses this by aligning Gaussians with scene surfaces, then applying Poisson reconstruction to derive a mesh.
  • Static Reconstruction: 3D-GS is promising for monocular and few-shot reconstruction, but these settings lack geometric information.pixelSplat predicts Gaussian primitives from image pairs using a multiview epipolar transformer and scale-aware feature maps.
  • Dynamic Reconstruction: Dynamic reconstruction includes specific objects, daily-life scenes, and large urban or driving environments.VastGaussian uses progressive partitioning, parallel optimization, seamless merging, and decoupled appearance modeling for large scenes.
  • Challenges and Opportunities: Dynamic scenes remain difficult when motion blur, large motion, sparse input points, or physically plausible dynamics must be captured without sacrificing quality.The survey identifies a need to balance reconstruction speed with preservation of image details.

5 MANIPULATION

3D-GS enables direct scene manipulation because its Gaussians are explicit and individually editable. The survey covers text-, image-, box-, mesh-, and control-point-guided editing, while highlighting efficiency, physical realism, segmentation noise, and temporal consistency as open challenges.

  • Overview: The explicit representation makes 3D scenes editable by directly manipulating individual Gaussians under desired constraints.This property is especially useful because editing must remain time-efficient for real-time workflows.
  • Guided Manipulation: Text-guided editing methods identify a Gaussian region of interest from instructions, while image prompts and 3D boxes provide finer appearance and spatial control.GaussianEditor uses text instructions; TIP-Editor accepts text, image, and 3D bounding-box prompts.
  • Representation and Deformation: Mesh-based Gaussian Splatting uses mesh geometry to guide Gaussian splitting and combines vertex positions with deformation gradients for large-scale deformation.The method is designed to improve the quality of the learned Gaussian representation.
  • Articulated and Non-rigid Objects: Articulated Gaussian representations model hand shape, appearance, object contacts, and complex deformations such as loose clothing.MANUS-Hand, MANUS, and MANUS-Grasps target articulated hands, contact modeling, and real-world grasp data.
  • 4D Manipulation: Dynamic 4D editing must preserve spatial-temporal consistency and quality while remaining interactive.Control4D uses GaussianPlanes and text instructions, while CoGS manipulates explicit scene elements through control signals.
  • Challenges and Opportunities: Editing remains limited by noisy segmentation, overlooked physical effects, artifacts from large edits, and insufficient frame consistency.Existing methods are also mainly tested with minimal motion changes and accurate camera poses.

6 GENERATION

3D-GS enables fast or real-time generation of explicit objects and scenes, while 4D generation remains challenging and under-explored. The survey reviews generation methods and identifies compositionality, multi-face artifacts, and Gaussian-count constraints as key issues.

  • Generation overview: 3D-GS provides explicit object and scene representations that make fast or even real-time rendering possible.
  • 3D generation methods: GaussianDreamer combines 3D diffusion geometry priors with 2D diffusion optimization, using noisy point growing and color perturbation to enrich initialized Gaussians.
  • 3D generation methods: SDS-based 3D generation can suffer from over-smoothing because 2D diffusion models produce inconsistent, low-quality pseudo-GTs.
  • 4D generation: 4D methods extend Gaussian representations with temporal deformation or use static assets and video sequences to model dynamic content.
  • Challenges and opportunities: Compositional generation remains open, while fixed Gaussian counts can severely limit the creation of complex geometry.
  • Challenges and opportunities: Future directions include reducing Janus artifacts with 3D-aware or multi-view diffusion models and improving language understanding for complex prompts.

7 PERCEPTION

The survey presents 3D-GS applications across detection, segmentation, tracking, and SLAM, supported by semantic features, dynamic representations, and differentiable rendering. It also identifies difficult object appearances, deformable motion, and adverse sensing or motion conditions as limitations.

  • Perception overview: 3D-GS supports open-vocabulary detection and localization, 3D segmentation, dynamic-object tracking, and SLAM.
  • Detection and localization: Language Embedded 3D Gaussians attach compact quantized semantic features to large Gaussian scenes and smooth them across viewpoints.
  • Segmentation: Gaussian Grouping jointly reconstructs and segments open-world scenes using Identity Encoding supervised by 2D masks and 3D spatial consistency.
  • Tracking: StreetGaussian models dynamic urban streets with point clouds augmented by semantic logits and Gaussians, optimizing trackable poses and dynamic appearance.
  • SLAM: GS-SLAM uses real-time differentiable splatting and adaptive Gaussian expansion to accelerate map optimization and RGB-D re-rendering.
  • Challenges: Perception limitations include reflective or translucent objects, Gaussian-to-object ambiguity, deformable-object tracking, and sensitivity to blur, depth noise, and aggressive rotation.

8 VIRTUAL HUMANS

3D-GS virtual-human methods use explicit Gaussian avatars with body or head templates, deformation models, and rendering supervision from multi-view or monocular data. The survey covers increasingly generalizable and compositional approaches while noting initialization and facial-expression limitations.

  • Overview: 3D-GS improves virtual-human training and rendering speed while offering explicit control of body deformation compared with implicit representations.
  • Body modeling: Most body-avatar methods initialize Gaussians from an SMPL template, deform them with Linear Blend Skinning, and supervise rendering using multi-view or monocular video.
  • Body modeling: HuGS represents animatable bodies with Gaussians, skinning weights, and a latent code for non-rigid motion, learning the weights after SMPL-based initialization.
  • Deformable avatars: D3GA uses separate cages and specialized networks to deform body, face, and garment avatars compositionally.
  • Generalizable avatars: GPS-Gaussian regresses Gaussian parameters feed-forward from diverse 3D human scans to synthesize novel views of unseen performers without finetuning or optimization.
  • Head modeling: Head-avatar methods rig Gaussians to morphable face models or use expression-conditioned latent features to produce color and opacity from monocular video.
  • Challenges and opportunities: Initialization constrains SMPL- and 3DMM-based reconstruction, while 3DMM-controlled heads may fail to express subtle facial movements.

9 DISCUSSIONS AND FUTURE WORK

The survey identifies persistent challenges in 3D Gaussian Splatting across rendering, reconstruction, few-shot learning, generation, and physics integration. It outlines future directions involving adaptive representations, improved geometry and appearance, foundation-model integration, and unified simulation-rendering pipelines.

  • Rendering and Reconstruction: Floating elements from image backgrounds reduce rendered-image quality, motivating opacity-threshold strategies evaluated with PSNR and SSIM.The issue is linked specifically to background-derived artifacts in rendered space.
  • Rendering and Reconstruction: 3D-GS reconstruction faces a trade-off: direct Gaussian use is efficient but produces noncontinuous meshes, whereas signed-distance-field combinations improve smoothness at increased reconstruction time.The survey highlights balancing rendering quality, reconstruction accuracy, and processing speed.
  • Rendering Realism: Lighting decomposition remains limited in scenes with indistinct boundaries and often requires object masks because 3D-GS point clouds have particle-like properties that amplify background influence.The limitation concerns optimization for realistic rendering.
  • Rendering and Reconstruction: Real-time methods improve speed through sparse-grid anchors or octrees, but fixed grids, manual octree assignment, and poor scalability limit adaptation to large environments such as cityscapes.Adaptive octree representations are identified as a needed research direction.
  • Few-shot 3D-GS: Few-shot 3D-GS depends heavily on monocular depth accuracy and may vary across domains, with difficulties in textureless areas, complex surfaces, and regions where COLMAP encounters problems.Suggested directions include interdependent depth estimates and geometry regularization across diverse datasets.
  • Physics Integration: Physics integration should seek a unified material representation for simulation and rendering, while expanding research to large complex motions, multiple materials, and automatic material assignment.The motivation is to reduce discrepancies introduced by multi-stage geometry, simulation, and rendering pipelines.
  • Generation: Generated 3D assets still need more realistic geometry and textures; unresolved issues include BRDF decomposition and temporal consistency for 4D generation.The survey points to more precise SDF/UDF integration, improved shading, and enhanced temporal consistency.
  • Foundation Models and Integration: Embedding language and using large foundation models, including SAM, are emerging ways to enhance 3D scene understanding and segmentation, while 3D-GS can also supplement existing reconstruction and rendering methods.Examples include NeuSG for NeuS reconstruction and SpecNeRF for specular-reflection modeling.
Loading 2402.07181v2…