PaperlayerView exact revision on arXiv

Source-linked AI summary

StopThePop: Sorted Gaussian Splatting for View-Consistent Real-time Rendering

Lukas Radl, Michael Steiner, Mathias Parger, Alexander Weinrauch, Bernhard Kerbl, Markus Steinberger

arXiv:2402.00525v3cs.GRcs.CV

TL;DR

3DGS’s simplified global depth ordering can cause popping and view inconsistency, while accurate per-ray sorting is too costly for real-time rendering. StopThePop uses hierarchical culling, depth evaluation, and resorting to approximate per-pixel sorting efficiently. It removes popping in evaluations with 4% average overhead, and with half as many Gaussians achieves nearly identical quality while rendering 1.6× faster with half the memory.

  • Problem

    3DGS’s simplified depth ordering can invert splat order under camera rotation, while fully correct per-ray sorting is too expensive for real-time rendering.

  • Method

    The paper introduces a hierarchical renderer that interleaves culling, per-ray depth evaluation, and resorting across neighboring view rays.

  • Results

    4% overhead is incurred versus original 3DGS, while the hierarchical renderer is virtually indistinguishable from full per-pixel sorting and removes popping artifacts.

  • Takeaways & Limitations

    With approximately half as many Gaussians, the method retains nearly identical quality and view-consistency, renders 1.6× faster, and uses 50% less memory than 3DGS.

  • Takeaways & Limitations

    Resorting does not guarantee the correct blend order, and the method still ignores overlaps between Gaussians along the view ray.

Abstract

from arXiv · show

Gaussian Splatting has emerged as a prominent model for constructing 3D representations from images across diverse domains. However, the efficiency of the 3D Gaussian Splatting rendering pipeline relies on several simplifications. Notably, reducing Gaussian to 2D splats with a single view-space depth introduces popping and blending artifacts during view rotation. Addressing this issue requires accurate per-pixel depth computation, yet a full per-pixel sort proves excessively costly compared to a global sort operation. In this paper, we present a novel hierarchical rasterization approach that systematically resorts and culls splats with minimal processing overhead. Our software rasterizer effectively eliminates popping artifacts and view inconsistencies, as demonstrated through both quantitative and qualitative measurements. Simultaneously, our method mitigates the potential for cheating view-dependent effects with popping, ensuring a more authentic representation. Despite the elimination of cheating, our approach achieves comparable quantitative results for test images, while increasing the consistency for novel view synthesis in motion. Due to its design, our hierarchical approach is only 4% slower on average than the original Gaussian Splatting. Notably, enforcing consistency enables a reduction in the number of Gaussians by approximately half with nearly identical quality and view-consistency. Consequently, rendering performance is nearly doubled, making our approach 1.6x faster than the original Gaussian Splatting, with a 50% reduction in memory requirements.

1 INTRODUCTION

3D Gaussian Splatting enables compact, high-quality real-time rendering, but its global depth simplification creates view-rotation inconsistencies that require per-pixel sorting to resolve efficiently.

  • Problem: Global sorting is efficient, but view-dependent depth errors can invert splat order during camera rotation and cause popping.The simplification projects Gaussian means onto the view direction rather than computing depth independently for each ray.
  • Background: 3D Gaussian Splatting renders oriented 3D Gaussians with spherical-harmonics view-dependent colors.It supports gradient-based optimization and real-time rasterization with compact memory requirements.
  • Challenge: Fully evaluating overlapping Gaussians along each ray is impractical in real time, while per-ray sorting may involve thousands of Gaussians.The paper therefore exploits coherence among neighboring rays through hierarchical culling, depth evaluation, and resorting.
  • Approach: The proposed hierarchical renderer performs per-pixel sorting for both forward and backward passes to remove popping artifacts.It also analyzes culling, depth approximation, sorting strategies, and workload distribution.
  • Results: 4% overhead is added by the hierarchical renderer compared with original 3DGS, while a full per-pixel sorted renderer is 100× slower.The hierarchical result is virtually indistinguishable from the full per-pixel sorted renderer.

2 PRELIMINARIES AND RELATED WORK

The section reviews volume-rendering foundations and 3DGS approximations, then situates the method among neural rendering, splatting, and software-rasterization approaches.

  • 3DGS foundations: NeRF-style rendering and 3DGS use the volume-rendering equation to compute ray colors from opacity and emitted radiance.For 3DGS, the scene is represented as a mixture of 3D Gaussians.
  • 3DGS approximations: 3DGS approximates each Gaussian along a ray as a separated Dirac delta and projects its mean onto the view direction.These simplifications enable faster rendering but replace overlap-aware ray integration with approximate splatting.
  • 3DGS approximations: Exact overlap-aware evaluation would require numerical integration and strict sorting of Gaussian endpoints along each ray.The absence of an elementary indefinite Gaussian integral makes this approach computationally demanding.
  • 3DGS rasterization: 3DGS reduces ray workloads by sorting tile-associated Gaussians using keys that combine tile indices with depths.The renderer evaluates the compositing equation after this combined depth-and-tile sorting pass.
  • Related work: Related work spans faster NeRF variants, Gaussian-splatting extensions, software rasterizers, and order-independent transparency methods.These approaches address representation, rendering speed, editing, aliasing, hardware, or transparency rather than the paper’s specific popping problem.

3 REAL-TIME SORTED GAUSSIAN SPLATTING

The paper introduces per-pixel sorted Gaussian splatting with hierarchical resorting and culling to improve view consistency while retaining real-time rendering. Its approach computes per-ray depths, progressively refines sorting, and reduces processing through tile-level operations and culling.

  • 3 REAL-TIME SORTED GAUSSIAN SPLATTING: Per-pixel sorting stabilizes Gaussian blending under camera rotation, addressing popping caused by 3DGS’s global view-space depth sort.The method still approximates true 3D Gaussians by neglecting overlap between them.
  • 3 REAL-TIME SORTED GAUSSIAN SPLATTING: The optimal blend depth t_opt is computed per ray because its view-dependent function has no simple rasterizable primitive.Bounding S^-1 to 10^3 removes instabilities for very thin Gaussians with minimal impact on computed depth.
  • 3 REAL-TIME SORTED GAUSSIAN SPLATTING: Full per-ray sorting eliminates popping but slows rendering by more than 100×, motivating local resorting and hierarchical sorting.A resorting window of 16 to 24 removes most visible popping artifacts but increases runtime 2 to 6×.
  • 3 REAL-TIME SORTED GAUSSIAN SPLATTING: Tile-based culling reduces the average number of Gaussians per tile by approximately 44%, lowering the work required for sorting and rendering.The optimized depth location also reduces reported maximum and average sort errors across the views in Table 1.
  • 3 REAL-TIME SORTED GAUSSIAN SPLATTING: The hierarchical renderer refines sorting from neighboring tiles to individual rays while culling non-contributing Gaussians at each level.It changes Gaussian blend order while leaving the other parts of 3DGS, including 2D splatting, untouched.

4 EVALUATION

Across 13 real-world scenes, the method maintains image quality comparable to 3DGS while improving view and depth consistency, with optimized hierarchical rendering keeping runtime competitive.

  • The evaluation uses 13 real-world scenes from Mip-NeRF 360, Deep Blending, and Tanks & Temples.
  • Image quality is comparable to 3DGS overall, while the method loses slightly less quality under Opacity Decay.
  • MSE is ineffective for detecting popping because warping inaccuracies can outweigh the artifact, whereas LIP more reliably reflects perceived frame-to-frame differences.
  • The method outperforms 3DGS on view-consistency metrics for both short-range and long-range novel-view sequences, with and without Opacity Decay.
  • The method achieves better depth-consistency scores than 3DGS, especially for outdoor Mip-NeRF 360 scenes.
  • The fully optimized hierarchical renderer achieves competitive runtimes, while tile-based culling and load balancing reduce sorting and duplication workloads.
  • The backward render pass is 1.1× the 3DGS cost, and overall training time increases by only about 3%.

5 CONCLUSION, LIMITATIONS, AND FUTURE WORK

The paper concludes that hierarchical resorting substantially improves view consistency and efficiency, but it remains an approximation that cannot guarantee correct blending for all geometric configurations.

  • Per-ray depth computation removes popping but is 100× more costly, whereas hierarchical culling and resorting are only 1.04× slower than 3DGS on average.
  • The approach significantly reduces popping, improves view-consistency and depth estimates, and remains consistent when using half as many Gaussians.
  • Using half as many Gaussians reduces memory by 2× and render times by 1.6× compared with 3DGS.
  • Resorting does not guarantee the correct blend order for very complex geometric relationships and still ignores Gaussian overlaps along the view ray.

A DERIVING DEPTH FOR 3D GAUSSIANS ALONG A RAY

The method derives an accurate ray-wise depth by locating the point where a 3D Gaussian contributes most strongly along the viewing ray.

  • The optimal ray depth t_opt is defined as the value of t that maximizes the Gaussian contribution G(r(t)).
  • Because the inverse covariance matrix is symmetric, the derivation simplifies equivalent quadratic expressions for computing the optimum efficiently.

B ADDITIONAL IMPLEMENTATION DETAILS

The implementation finds the maximally contributing point of a projected 2D Gaussian within each axis-aligned tile using a constrained optimization procedure.

  • The hierarchical rasterizer is designed to make the implementation and optimization strategies viable for real-time rendering.
  • For a 2D Gaussian, the algorithm finds the point of maximum contribution inside an axis-aligned tile.
  • If the Gaussian mean lies inside the tile, it is the maximum; otherwise, the optimum lies on one of the two reachable tile edges.
  • The procedure represents the Gaussian with its 2D mean and inverse covariance matrix and the tile with its axis-aligned bounding-box dimensions.
  • The algorithm selects the closest tile corner, evaluates edge directions, and clamps the resulting parameters to the valid [0, 1] range.

B.2 Tighter Bounding of 2D Gaussians

This section tightens Gaussian bounds and describes a more efficient tile-wise sorting setup for hierarchical rasterization.

  • Tighter Gaussian bounds: The original tO=3 bound is too small for the renderer’s opacity threshold ϵO=1/255.The analysis upper-bounds tOmax at approximately 3.3290 and supports tighter circular bounds when α<1.
  • Partitioned sorting: Global sorting of all tile/depth keys is wasteful, so entries can instead be sorted separately within each tile.This requires contiguous per-tile entries and tile-specific ranges produced during preprocessing.

B.4 Per-stage details

The renderer’s per-stage implementation combines common Gaussian preprocessing with warp-level load balancing and hierarchical sorting and blending optimizations.

  • Preprocess and Duplication: Preprocessing computes and stores Gaussian-dependent quantities, including G2, spherical harmonics, Σ−1, and Σ−1(μ−o).The implementation packs six unique covariance coefficients with a precomputed vector for efficient loading.
  • Preprocess and Duplication: A conservative culling threshold compensates for floating-point instruction-order differences caused by fast math and rescheduling.This protects memory allocation based on the stored number of tiles contributed by each Gaussian.
  • Preprocess and Duplication: Warp ballots identify active threads, while shuffle operations broadcast register values for culling and depth evaluation without extra memory loads.Successive potential tiles are assigned according to thread rank within the warp.
  • Render: The hierarchical rasterizer interleaves pipeline stages and initializes each level with a minimal fill level before sorting and blending proceed.The main operation maintains a minimal fill level for each queue.
  • Render: Custom merge sorting uses register-held elements, binary search, role-switched arrays, and loop unrolling for small fixed-size arrays.Local presorting of four elements uses three comparisons.
  • Render: Shared-memory view directions reduce register pressure when repeatedly evaluating top-depth values across many ray directions.The same directions are reused by threads at multiple hierarchy levels.
  • Render: Queue retirement must avoid thread divergence, while batch lifetimes and register residency are managed to reduce pressure.A 32-wide batch replaces smaller live batches once loaded.

C POPPING DETECTION METRIC

The popping metric evaluates view consistency across temporal offsets, where the proposed method outperforms 3DGS and avoids the pronounced per-frame artifacts associated with popping.

  • Metric design: LIPt is evaluated at t∈{1,7} to measure short-range and long-range view consistency.The study also examines t∈{3,5,9} across three Garden camera paths.
  • Metric results: LIPt outperforms 3DGS for every tested t∈{1,3,5,7,9} on three Garden camera paths.LIPt scales almost linearly as t increases.
  • Per-frame results: 3DGS shows significant LIP1 peaks caused by popping, whereas the proposed method does not suffer from these issues.The per-frame plots come from a complete Garden video sequence.
  • Per-frame results: At t=7, 3DGS has significantly higher error rates because artifacts accumulate across several iterations.This longer offset more clearly indicates popping when averaged over the complete video sequence.
  • View rotation: Under slight camera rotations, the proposed method produces more consistent results than 3DGS.3DGS appearance changes significantly around test views, sometimes improving image metrics through popping.

D USER STUDY

The user study compares paired videos while participants judge rendering consistency, finding a significant preference for the proposed method overall.

  • Participants: The study recruited 18 participants aged 26–34 with normal or corrected vision and computer-graphics familiarity.All participants rated their familiarity at 3–5 on a five-point Likert scale.
  • Procedure: Participants viewed randomized paired videos from 13 scenes containing translation and rotation along 8–19-second camera paths.The videos used versions trained specifically for the compared approaches without Opacity Decay.
  • Results: 54.3% of judgments preferred the proposed method, 33.3% were equal, and 12.4% preferred 3DGS, yielding s_mean=0.42.The result was statistically significant with Wilcoxon Z=2276.5 and p<.0001.
  • Results: The proposed method was preferred by larger margins in scenes with large Gaussians, while participants slightly preferred 3DGS for Bicycle.Scenes with mostly small Gaussians showed less difference in voted scores.

E DETAILED PERFORMANCE TIMINGS

The section analyzes per-scene and per-stage performance, showing that StopThePop’s speed depends on scene characteristics and queue configuration rather than Gaussian count alone.

  • Our method is slightly slower than 3DGS on Mip-NeRF 360 scenes but achieves higher performance on most Tanks & Temples and Deep Blending scenes.
  • Performance improves over 3DGS when Gaussians are larger or more anisotropic because culling and load balancing reduce rendering work.
  • Similar Gaussian counts can produce substantially different timings because screen-space splat sizes differ across scenes.
  • Preprocessing is slower, but load balancing accelerates duplication and culling reduces the number of splats requiring sorting.
  • The selected queue configuration uses sizes 8 and 4 because it has the lowest memory and compute requirements.

F PER-SCENE QUALITY METRICS

This section reports per-scene image-quality metrics and ablations for the proposed rasterizer, including comparisons across datasets and queue configurations.

  • Per-scene results cover Mip-NeRF 360, Tanks & Temples, and Deep Blending, comparing the final hierarchical rasterizer with fixed-size head sorting variants.
  • The ablation varies tile-queue size, per-pixel queue size, and load-balancing thresholds without retraining the underlying model.
  • The quality tables combine PSNR, SSIM, LPIPS, and LIP for Tanks & Temples and Deep Blending.
  • Mip-NeRF 360 quality results are likewise reported using PSNR, SSIM, LPIPS, and LIP.
Loading 2402.00525v3…