PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities

Yanqi Bao, Tianyu Ding, Jing Huo, Yaoli Liu, Yuxin Li, Wenbin Li, Yang Gao, Jiebo Luo

arXiv:2407.17418v2cs.CV

TL;DR

3DGS research has expanded rapidly, creating a complex landscape of related tasks, technologies, challenges, and opportunities that is difficult to organize. This survey categorizes works by motivations, classifies technical improvements, and analyzes common task-level challenges and technologies. It provides a structured framework for understanding the evolving field and identifying research directions.

  • Problem

    The rapid growth of 3DGS tasks and technologies has created complexity through shared motivations, reused techniques, and nuanced interconnections across works.

  • Method

    The survey categorizes optimization, application, and extension tasks by focus, classifies nine technical modules and their improvements, and analyzes cross-task challenges and technologies.

  • Results

    The survey provides a structured analysis connecting downstream tasks with technical improvements, common challenges, and prospective research opportunities.

  • Takeaways & Limitations

    The resulting framework helps readers understand relationships among 3DGS works and guides the adaptation of technical components to customized tasks.

Abstract

from arXiv · show

3D Gaussian Splatting (3DGS) has emerged as a prominent technique with the potential to become a mainstream method for 3D representations. It can effectively transform multi-view images into explicit 3D Gaussian through efficient training, and achieve real-time rendering of novel views. This survey aims to analyze existing 3DGS-related works from multiple intersecting perspectives, including related tasks, technologies, challenges, and opportunities. The primary objective is to provide newcomers with a rapid understanding of the field and to assist researchers in methodically organizing existing technologies and challenges. Specifically, we delve into the optimization, application, and extension of 3DGS, categorizing them based on their focuses or motivations. Additionally, we summarize and classify nine types of technical modules and corresponding improvements identified in existing works. Based on these analyses, we further examine the common challenges and technologies across various tasks, proposing potential research opportunities.

I. INTRODUCTION

3DGS enables high-quality real-time novel-view synthesis through an explicit Gaussian representation, and this survey organizes its tasks, techniques, challenges, and opportunities. It connects downstream applications with technical improvements to help readers understand and adapt the field.

  • 3DGS enables high-quality real-time novel-view synthesis at 1080p resolution with at least 30 fps.
  • It classifies technical improvements across fundamental components, including initialization, attribute configurations, and regularization.
  • The survey examines relationships between downstream tasks and enabling technologies, identifies four fundamental challenges, and proposes research directions.
  • The survey categorizes 3DGS derivative tasks by motivations or focuses, covering optimization, application, and extension.
  • 3DGS represents point-cloud points as volumetric 3D Gaussian primitives with geometric attributes, opacity, and spherical-harmonic parameters for view-dependent appearance.
  • Training initializes Gaussians from sparse Structure-from-Motion points or randomly, then adaptively increases or decreases primitives using view-space position gradients.
  • Rendering projects 3D Gaussians onto the image plane with EWA splatting and computes pixel colors through alpha blending.

1) Storage Efficiency:

Storage efficiency is a central 3DGS concern because explicit scenes require many individually stored Gaussian attributes. Existing approaches compress, prune, restructure, or accelerate Gaussian representations while sometimes improving rendering efficiency.

  • Storage Efficiency: Outdoor 3DGS reconstructions typically require several hundred megabytes to multiple gigabytes of explicit storage.
  • Storage Efficiency: Vector-quantization methods compress Gaussian attributes into codebooks and store per-Gaussian codebook indices to reduce storage overhead.
  • Storage Efficiency: Pruning methods reduce Gaussian counts using global significance scores, spherical-harmonic distillation, or learnable masks, but depend on identifying non-essential primitives.
  • Storage Efficiency: Scaffold-GS uses anchors and multiresolution voxel-grid strategies for representation growth and pruning, while HAC adds a structured hash grid to mitigate unstructured compression limits.
  • Storage Efficiency: DISTWAR targets training bottlenecks from atomic updates by exploiting intrawarp locality and reducing threads at the SM sub-core level.
  • Storage Efficiency: Storage-efficient representations can also improve rendering performance by reducing the number of Gaussian primitives.
  • Storage Efficiency: 10.7× speedup over a GPU was achieved by offline removal of unnecessary Gaussians combined with specialized hardware architecture.

D. Sparse Views Setting

Sparse-view 3DGS reconstruction relies on regularization and supplementary information such as depth or diffusion-model priors. These methods improve spatial guidance but remain constrained by 3D consistency and scene-specific training requirements.

  • Sparse-input reconstruction develops novel regularization strategies and incorporates supplementary information such as depth data or diffusion models.
  • Depth data alleviates 3DGS’s demand for spatial comprehension under sparse inputs, while diffusion models expand sparse views into refined dense information.
  • GaussianObject initializes Gaussians from a Visual Hull and uses distance statistics during optimization to eliminate floaters.
  • Body Reconstruction: Human reconstruction methods target deformable avatars from multiview or monocular videos while supporting real-time rendering.
  • Body Reconstruction: Existing high-fidelity human reconstruction methods remain limited by scene-specific training requirements, motivating generalizable reconstruction.

1) Text to 3D Objects:

Text-to-3D work uses 2D generative priors, score distillation, depth, and multimodal generation to create objects, scenes, and dynamic content. Persistent limitations include unstable 3D consistency and dependence on 3D training data.

  • Text to 3D Objects: Score Distillation Sampling reduces dependence on extensive 3D training data and has become a major paradigm for text-to-3D generation.
  • Text to 3D Objects: DreamGaussian extracts explicit meshes from 3DGS and refines textures in UV space to improve geometric consistency and rendering quality.
  • Text to 3D Objects: Video and multi-view generative models provide direct prior guidance, but their lack of guaranteed 3D consistency still destabilizes reconstruction.
  • Text to 3D Objects: L3DG uses latent diffusion for compressed 3D Gaussian representations, achieving superior visual quality and real-time rendering, but depends on available 3D data.
  • Text to 3D Objects: Scene generation incorporates depth estimators for spatial understanding or large language models for improving text prompts.
  • 4D Generation: Dynamic generation extends text-to-image score distillation with text-to-video losses or hybrid supervision from text-to-image and text-to-video models.

2) Simultaneous Localization and Mapping (SLAM):

3DGS-SLAM improves reconstruction efficiency through adaptive primitives, coarse-to-fine optimization, feature integration, and GPU acceleration, while remaining vulnerable to sensor noise. Related dynamic and surface-reconstruction work extends 3DGS across temporal and geometric settings.

  • Simultaneous Localization and Mapping (SLAM): GS-SLAM uses adaptive primitive expansion and coarse-to-fine optimization, while Photo-SLAM combines ORB features, Gaussian attributes, LM optimization, and loop closure.
  • Simultaneous Localization and Mapping (SLAM): CG-SLAM uses an uncertainty-aware Gaussian field and GPU acceleration, while RGBD GS-ICP SLAM integrates G-ICP with shared covariances and scale alignment.
  • Simultaneous Localization and Mapping (SLAM): 3DGS-SLAM methods achieve higher efficiency than NeRF-based methods, but further computational optimization remains necessary for real-world deployment.
  • Simultaneous Localization and Mapping (SLAM): Practical SLAM methods remain susceptible to sensor noise.
  • Dynamic 3DGS: Dynamic 3DGS methods represent motion through frame-by-frame temporal Gaussians, canonical reconstruction with deformation prediction, or continuous motion models.
  • Surface Reconstruction: Surface reconstruction combines 3DGS with signed distance functions, surfels, planar disks, or opacity fields to address the difficulty of extracting surfaces.

1) Manipulation by Text:

3DGS manipulation methods use text, images, points, and other conditions to control geometry, appearance, and motion, while consistency remains a central challenge. Related extensions also address relighting, semantics, simulation, and broader component improvements.

  • SDS-based methods use editing prompts to guide semantic-controlled 3DGS manipulation, with hierarchical representations, anchor losses, and 2D inpainting supporting object edits.
  • Multi-view image editing locates editable regions using multimodal, language, and segmentation models before optimizing targeted Gaussians, but cross-view consistency remains difficult.
  • Control4D combines Tensor4D representations with a super-resolution GAN to maintain spatiotemporal consistency, yet remains challenged by non-rigid movements.
  • Mixed text, reference-image, location, and point conditions support fine-grained control, including object insertion, stylization, and object-level editing.
  • Relightable 3DGS decouples texture and lighting using normals, BRDFs, and lighting attributes, but transparent materials and indirect lighting require further exploration.
  • Semantic attributes extend pretrained 2D semantic models into 3D for detection, segmentation, editing, and spatial semantic awareness.

B. Attribute Expansion

Attribute expansion modifies existing Gaussian properties or adds new attributes and supervision to adapt 3DGS to downstream tasks. These changes support semantics, dynamics, physical effects, compression, and improved representation efficiency.

  • Vanilla 3DGS stores position, scale, rotation, spherical-harmonic coefficients, and opacity; attribute expansion either improves these properties or introduces new ones.
  • Existing attributes: Flattened or spherical scale constraints and opacity binarization target surface quality, geometric structure, and artifact reduction.
  • New attributes: New semantic, weight, uncertainty, and correspondence attributes provide task-specific supervision for spatial awareness, stability, and 2D-to-3D matching.
  • New attributes: Temporal and displacement attributes represent time-varying position, scale, rotation, appearance, and changes from initial to final Gaussian states.
  • New attributes: Physical and discrete attributes support relighting, motion, and high-dimensional compression, although discrete representations can degrade performance.
  • Attribute modification broadens downstream applicability without requiring additional structural elements, while supplementary constraints can improve representation and object boundaries.

C. Splatting

3DGS research improves the splatting pipeline and applies regularization at primitive, local, global, and image levels. These constraints target projection quality, geometry, motion, physical relationships, and under-constrained reconstruction.

  • Splatting: Splatting converts 3D Gaussian data into smooth 2D images, and newer methods target projection accuracy, anti-aliasing, efficiency, and rendering performance.
  • Regularization: 3D regularization constrains individual primitives, local neighborhoods, or the entire Gaussian representation according to task objectives.
  • Regularization: Local rigidity, rotation similarity, isometry, and momentum losses constrain deformable scenes by preserving neighborhood structure and smoothing trajectories.
  • Regularization: Global physics regularization uses gravity and contact losses to constrain relationships between objects, floors, and multiple objects.
  • Regularization: Depth-normal, depth, normal, and SDF regularization impose geometric constraints that support surface reconstruction and more complete surfaces.

2) 2D Regularization:

2D regularization supplements insufficient reconstruction losses, using diffusion, optical flow, frequency, pseudo-view, and other image-space signals. Multi-stage training further stabilizes under-determined and multi-objective tasks.

  • 2D Regularization: 2D regularization addresses under-constrained situations where the original loss functions provide insufficient guidance.
  • 2D Regularization: SDS distills pretrained 2D diffusion models into 3DGS training, while 3D, multi-view, image-editing, video, and RGB-depth models provide additional guidance.
  • 2D Regularization: Interval score matching, negative prompts, and LoRA-based noise replacement target randomness, instability, and out-of-distribution effects in SDS.
  • 2D Regularization: Flow loss uses pretrained 2D optical flow as supervision by rendering Gaussian displacements, but its 2D nature makes it susceptible to noise.
  • 2D Regularization: Frequency-domain losses, pseudo-labels for hypothetical viewpoints, and primitive selection provide additional guidance for overfitting, sparse views, and large primitive sets.
  • Training Strategy: Multi-stage training uses coarse-to-fine reconstruction, representation conversion, static-to-dynamic modeling, camera refinement, and iterative optimization to improve stability.

2) End-to-End Training Strategy:

End-to-end training strategies improve 3DGS training efficiency and can enhance performance across downstream tasks. The survey organizes these strategies around progressive, block, robust, and distillation-based approaches.

  • Progressive Optimization Strategy: Progressive optimization learns global representations before local details by increasing high-frequency signals or supervision resolution.Generative tasks can also progressively select camera poses from near to far viewpoints.
  • Block Optimization Strategy: Block optimization improves efficiency and alleviates catastrophic forgetting in large-scale scene reconstruction.Its effectiveness depends on block partitioning and training-data selection, motivating strategies for balancing workloads and preserving details.
  • Robust Optimization Strategy: Robust optimization introduces perturbations to camera poses, timesteps, or images as data augmentation to prevent overfitting.Avoiding continuous training from a single viewpoint can also mitigate catastrophic forgetting.
  • Distillation-based Strategy: Distillation-based strategies use an original 3DGS teacher and a low-dimensional SH 3DGS student to compress model parameters.Additional pseudo views are introduced to improve the low-dimensional spherical-harmonics student.
  • Discussion: End-to-end training strategies can improve performance while maintaining efficiency across many 3DGS tasks.The survey identifies end-to-end training as particularly useful for jointly improving training outcomes and efficiency.

2) Pruning:

3DGS pruning removes unimportant Gaussian primitives using importance, consistency, distance, and learnable criteria. These methods support efficient representations, but pruning must be balanced against rendering fidelity and performance.

  • Pruning: Pruning removes unimportant Gaussian primitives to maintain an efficient 3DGS representation.Initial methods used opacity, while later methods incorporated scale and distractor masks.
  • Importance scores: Global importance scores combine Gaussian volume, training-view hit count, and opacity before low-scoring primitives are pruned.The resulting scores rank Gaussians according to their global significance across training views.
  • Multi-view consistency: Multi-view consistency prunes Gaussians that lack sufficient visibility across keyframes or virtual views.Examples remove newly added Gaussians unseen by three keyframes or invisible in all virtual views but visible in real views.
  • Distance Metric: Surface-aware pruning uses distance to surfaces, SDF values, or Gaussian distances to remove poorly positioned primitives.GauHuman additionally merges small-scale Gaussians with low KL divergence.
  • Discussion: Excessive densification or pruning can reduce both 3DGS efficiency and performance, requiring a balance between them.The survey presents this balance as a central consideration for adaptive control strategies.

1) Point Clouds:

3DGS can interoperate with point clouds, meshes, triplanes, grids, volumes, and implicit representations to address reconstruction and downstream-task requirements. Auxiliary priors and sensors can improve performance in ill-posed settings, although they may increase computational overhead and cost.

  • Point Clouds: Point clouds initialize 3DGS positions, fill reconstruction holes, improve details, and can be converted into voxels or BEV grids for navigation.Their use spans both reconstruction initialization and downstream spatial guidance.
  • Meshes: Meshes provide stronger geometric representation, can alleviate 3DGS artifacts or blur, and may be jointly optimized with 3DGS.3DGS-to-mesh conversion supports subsequent geometry and appearance optimization.
  • Alternative Representations: Triplanes, grids, Gaussian volumes, and implicit representations provide compact, structured, or more adjustable alternatives for representing Gaussian attributes.These representations support generalization, compression, spatiotemporal continuity, or improved surface reconstruction.
  • Discussion: Structured representations can provide useful priors for human-body reconstruction and cross-representation geometric transformations.The survey cautions that representational limitations may degrade rendering quality.
  • Challenges and Opportunities: Auxiliary information such as pretrained models, additional sensors, and task-specific priors can improve 3DGS performance in ill-posed problems.Such priors may increase computational overhead and cost while expanding representational capacity.

B. Suboptimal Data

The survey frames 3DGS progress around data, generalization, physics, realism, efficiency, and initialization challenges. It connects these challenges to technical opportunities and a framework for organizing tasks and technologies.

  • B. Suboptimal Data: Limited high-quality 3D data and multi-view images constrain accurate reconstruction, while camera-pose accuracy and image clarity directly affect performance.Frontal-only images make unseen appearance inference difficult.
  • B. Suboptimal Data: Underconstrained training motivates reducing Gaussian primitives according to uncertainty while increasing individual primitive capacity, balancing representation count against rendering performance.The survey also identifies transient occlusions and changing sky, weather, and lighting as difficult in-the-wild conditions.
  • C. Generalization: Scene-specific training remains a bottleneck for multi-target reconstruction, while generalizable feedforward methods have unsatisfactory, scene-dependent performance and often require accurate poses.The survey proposes reference-feature-based adaptive control and generalization under missing poses as research directions.
  • D. Physics Reconstruction and Rendering: Static 3DGS neglects physical motion, motivating Gaussian attributes and rendering models that represent dynamics, materials, forces, relighting, and material editing.Suggested directions include velocity, acceleration, force distribution, geometry–appearance decoupling, normal reconstruction, and illumination modeling.
  • E. Realness and Efficiency: Surface reconstruction and storage efficiency remain coupled challenges: planar Gaussians can reduce rendering performance, while vector quantization and spherical-harmonics compression affect performance.The survey proposes surface-suited primitives, multi-stage training, regularization, and more efficient representations that preserve performance.
  • F. Other Technical Challenges and Opportunities: Initialization is important because traditional Structure-from-Motion is unsuitable for sparse-view, AIGC, and low-light settings; efficient parallel splatting is another underexplored opportunity.The survey highlights robust initialization and pretrained-3DGS splatting strategies as potential improvements to rendering performance and efficiency.
Loading 2407.17418v2…