PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes

Nicolas Moenne-Loccoz, Ashkan Mirzaei, Or Perel, Riccardo de Lutio, Janick Martinez Esturo, Gavriel State, Sanja Fidler, Nicholas Sharp, Zan Gojcic

arXiv:2407.07090v3cs.GRcs.CV

TL;DR

Existing particle-based radiance-field renderers rely on rasterization, limiting distorted-camera rendering, secondary effects, and stochastic ray sampling. This paper develops a GPU-accelerated differentiable ray tracer for semi-transparent particles, achieving near-rasterization quality at real-time rates while enabling these capabilities and generalized particle kernels. The method remains slower than rasterization for pinhole cameras and incurs BVH rebuild overhead during training.

  • Problem

    Rasterized particle renderers do not efficiently support distorted cameras, secondary rays, or stochastic sampling, while ray tracing semi-transparent Gaussian scenes remains difficult.

  • Method

    The paper builds a GPU-accelerated differentiable ray tracer using k-buffer hit marching, bounding-mesh proxies, ordered intersection processing, and generalized particle formulations.

  • Results

    Ray tracing nearly matches or exceeds the 3DGS rasterizer on standard multiview benchmarks while maintaining real-time rendering framerates and enabling secondary effects, distorted cameras, and stochastic rays.

  • Takeaways & Limitations

    The renderer provides a practical algorithmic ingredient for future ray-based research on particle scenes, including inverse rendering, relighting, and material decomposition.

  • Takeaways & Limitations

    Ray tracing is slower than rasterization for pinhole cameras, and regularly rebuilding the BVH adds training cost and overhead for dynamic scenes.

Abstract

from arXiv · show

Particle-based representations of radiance fields such as 3D Gaussian Splatting have found great success for reconstructing and re-rendering of complex scenes. Most existing methods render particles via rasterization, projecting them to screen space tiles for processing in a sorted order. This work instead considers ray tracing the particles, building a bounding volume hierarchy and casting a ray for each pixel using high-performance GPU ray tracing hardware. To efficiently handle large numbers of semi-transparent particles, we describe a specialized rendering algorithm which encapsulates particles with bounding meshes to leverage fast ray-triangle intersections, and shades batches of intersections in depth-order. The benefits of ray tracing are well-known in computer graphics: processing incoherent rays for secondary lighting effects such as shadows and reflections, rendering from highly-distorted cameras common in robotics, stochastically sampling rays, and more. With our renderer, this flexibility comes at little cost compared to rasterization. Experiments demonstrate the speed and accuracy of our approach, as well as several applications in computer graphics and vision. We further propose related improvements to the basic Gaussian representation, including a simple use of generalized kernel functions which significantly reduces particle hit counts.

1 INTRODUCTION

The paper replaces rasterized Gaussian-particle rendering with an efficient differentiable ray tracer, targeting capabilities that rasterization handles poorly. It reports near-rasterization quality at real-time rates while enabling ray-based effects, distorted-camera rendering, and stochastic ray training.

  • 3D Gaussian Splatting represents scenes as optimized fuzzy particles rendered by a tile-based rasterizer.
  • Rasterization struggles with highly distorted and rolling-shutter cameras, secondary rays for reflections and shadows, and stochastic ray sampling.
  • The proposed GPU-accelerated tracer uses k-buffer hit marching, bounding-mesh proxies, and a differentiable backward pass for semi-transparent Gaussian particles.
  • The method generalizes beyond Gaussian kernels to broader particle-based scene representations and reduces intersections through generalized particle formulations.
  • On multiview benchmarks, ray tracing nearly matches or exceeds the 3DGS rasterizer while maintaining real-time rendering framerates.
  • The contributions include an improved optimization pipeline and applications spanning depth of field, shadows, mirrors, distorted cameras, rolling shutter, incoherent rays, and instancing.

2 RELATED WORK

Related work establishes ray tracing as a route beyond rasterization’s limitations in novel-view synthesis and secondary effects, but semi-transparent particle scenes remain difficult to trace efficiently. The paper’s formulation addresses this gap with consistent, differentiable hit processing for large particle scenes.

  • 2.1 Novel-View Synthesis and Neural Radiance Fields: Novel-view synthesis methods span geometric proxies, neural radiance fields, and particle-based representations, with differing speed and training costs.
  • 2.1 Novel-View Synthesis and Neural Radiance Fields: Particle-based and Gaussian-splatting methods inherit rasterization limits for distorted cameras, secondary lighting, rolling shutter, and motion blur.
  • 2.2 Ray Tracing: Ray tracing hardware supports real-time rendering, but existing pipelines remain challenging for order-independent semi-transparent surfaces and particles.
  • 2.3 Differentiable Ray Tracing of Volumetric Particles: Earlier particle-tracing methods rely on full-scene traversal, isotropic particles, uniform distributions, or approximate ordering, assumptions that limit Gaussian-scene efficiency or accuracy.
  • 2.3 Differentiable Ray Tracing of Volumetric Particles: The proposed formulation processes every ray-intersecting particle with consistent hit order, preserving differentiability while avoiding earlier methods’ restrictions.
  • 2.3 Differentiable Ray Tracing of Volumetric Particles: Unlike Fuzzy Metaballs, the method targets scenes with several millions of particles rather than only small particle sets and low-resolution images.

3 BACKGROUND

This section introduces differentiable particle-based radiance fields, their volume-rendering formulation, and the hardware ray-tracing pipeline used to accelerate particle intersections. It also motivates specialized handling for semi-transparent volumes, where many hits per ray make standard ray-tracing pipelines inefficient.

  • 3.1 3D Gaussian Parameterization: 3D Gaussian scenes represent differentiable semi-transparent particles through a kernel function parameterized by position and covariance.The covariance is optimized through a rotation-scale factorization and stored using quaternion and vector representations.
  • 3.1 3D Gaussian Parameterization: Each particle combines an opacity coefficient with a view-dependent radiance function represented using spherical harmonics of order 3.The radiance function uses 48 coefficients and a sigmoid to normalize colors.
  • 3.2 Differentiable Rendering of Particle Representations: Classical volume rendering accumulates view-dependent particle colors along rays using opacity-weighted contributions and transmittance.The numerical approximation evaluates each Gaussian at the point along the ray where its response is highest.
  • 3.3 Hardware-Accelerated Ray Tracing: NVIDIA OptiX builds a hardware-accelerated bounding volume hierarchy for ray-primitive intersections, with RT cores handling traversal while SMs perform computation such as shading.The pipeline exposes ray-generation, intersection, any-hit, and closest-hit programs for programmable traversal and hit processing.
  • 3.3 Hardware-Accelerated Ray Tracing: The accelerated tracing algorithm inserts particle bounding primitives into a BVH, gathers the next k particles per ray, and accumulates their radiance in depth order.It repeats this process until all particles are evaluated or transmittance reaches a predefined threshold.
  • 3.3 Hardware-Accelerated Ray Tracing: Ray-tracing volumes is challenging because semi-transparent particles generate many hits per ray, unlike the low-hit-count workloads for which hardware pipelines are optimized.The particle size and spatial distribution therefore require specialized traversal and processing strategies.

4 METHOD

The method ray traces semi-transparent particles through a BVH, gathers sorted hit batches, and renders them consistently while supporting differentiable optimization and generalized particle kernels.

  • Core architecture: The renderer combines adaptive particle bounding primitives in a BVH with batched ray traversal scheduled for NVIDIA OptiX.Its two core components are adaptive bounding mesh primitives and a rendering algorithm that gathers intersection batches.
  • Bounding primitives: Stretched regular icosahedra tightly bound particles while exploiting hardware-optimized ray-triangle intersections.The proxy is rescaled anisotropically to capture a specified minimum response, typically αmin = 0.01, while incorporating particle opacity.
  • Rendering algorithm: Each tracing round gathers the next k closest particle hits, sorts them by depth, evaluates radiance in order, and recasts the ray for subsequent hits.Processing terminates after all intersections are handled or when transmittance reaches the predefined minimum threshold Tmin.
  • Rendering algorithm: Unlike prior approximations, the renderer processes ray intersections consistently without missing particles or approximating transmittance.This ordering supports differentiable rendering, with the backward pass recasting rays to sample the same particles in order.
  • Optimization: The renderer supports stochastic ray training, enabling randomly or importance-sampled rays instead of requiring whole images or tiles.This capability addresses a limitation of differentiable rasterization for stochastic optimization over scene pixels.
  • Particle kernels: Generalized Gaussian particles reduce intersections and improve rendering performance by a factor of 2 compared with standard Gaussians.Kernelized surface particles provide flat particles with well-defined normals and can use a two-triangle primitive.

5 EXPERIMENTS AND ABLATIONS

Experiments compare method variants and show that denser generalized-Gaussian particles reduce ray hits, yielding faster rendering with minimal quality loss.

  • Method variants: The experiments use a high-quality reference variant and a faster variant adapted with degree-2 generalized Gaussians, denser particles, and incoherent-ray training.The adapted variant changes the density learning rate to 0.09 and starts batch-size-219 incoherent-ray optimization after 15,000 iterations.
  • Method variants: The adapted variant achieves fewer ray hits and faster rendering speeds with minimal quality loss.The paper attributes this combination to denser particles and the faster fall-off of degree-2 generalized Gaussian particles.

5.1 Novel View Synthesis Benchmarks

Across standard novel-view benchmarks, the ray-traced method achieves quality comparable to or better than established methods while maintaining real-time rendering. Results vary by dataset, with particularly strong PSNR performance on NeRF Synthetic and a quality drop on Deep Blending.

  • MipNeRF360: On MipNeRF360, the method performs on par with or slightly better than state-of-the-art methods, reaching 78 FPS versus 238 FPS for rasterized 3DGS.It remains substantially faster than MipNeRF360 and Zip-NeRF, both below 1 FPS.
  • Tanks & Temples: On Tanks & Temples, quality remains on par with state-of-the-art methods while rendering at 190 FPS, approximately 1.7 times slower than 3DGS.The incoherent-ray model has better PSNR but worse LPIPS and SSIM than the reference version.
  • Deep Blending: On Deep Blending, the reference implementation outperforms all baselines across the qualitative metrics.The authors observe a larger quality drop for the incoherent-ray version, linked to unstable training on Playroom.
  • NeRF Synthetic: On NeRF Synthetic, both method variants outperform all baselines in PSNR, while the method reaches 450 FPS on simpler scenes.It is only 50% slower than 3DGS in this lower-resolution setting.
  • Distorted Cameras: For distorted-camera evaluation, the ray-traced method directly uses fisheye images, whereas 3DGS requires undistorted views and associated cropping or wasted image space.The comparison evaluates training and rendering on distorted versus undistorted views.

5.2 Ray Tracing Analysis and Ablations

The ablations analyze tracing primitives, traversal strategies, buffer size, and kernel functions. They show that performance depends strongly on hit counts and approximation choices, while generalized Gaussian kernels improve speed at a small quality cost.

  • Bounding Primitives: More complex icosahedron primitives make BVH construction close to linear, exceeding 30 ms per million particles, whereas simple AABBs have nearly constant build time.Framerate is not strictly determined by particle count.
  • Tracing Algorithms: Naive closest-hit tracing is almost twice as slow as the proposed method because it over-traverses the BVH.SLAB and MLAT improve runtime through approximate image formation, but lose accuracy.
  • Tracing Algorithms: In differentiable rendering, SLAB and MLAT approximations are unusable for scene optimization, while tile-based rendering offers a promising speedup at a small approximation cost.Stochastic depth sampling provides no immediate benefit because most computation remains in the any-hit program.
  • Kernel Functions: Generalized Gaussians with n=2 significantly increase ray-tracing speed at only a small quality cost.Their sharper extent reduces particle hits, explaining the speedup.
  • Kernel Functions: The number of ray hits, rather than the number of particles, determines performance across tracing configurations.This relationship is shown by plotting mean hits against mean FPS for Gaussian and degree-2 generalized Gaussian kernels.
  • Buffer Size: k=16 provides a good compromise between false rejected hits and false accepted hits and is used in the remaining experiments.The buffer controls how many particle hits are gathered before evaluating their response.

6 APPLICATIONS

Differentiable ray tracing extends particle-scene rendering beyond standard rasterized views and effects. The applications include optical effects, stochastic sampling, instancing, complex cameras, and autonomous-vehicle reconstruction.

  • Ray-Based Effects: Ray tracing supports reflections, refractions, inserted meshes, depth of field, and artificial shadows through interleaved mesh and particle tracing.Reflected and refracted rays continue along redirected optical paths, while shadow rays test emitter visibility.
  • Instancing: Instancing renders 1024 transformed copies of a fitted particle scene at more than 25 FPS by linking repeated objects in BVH subtrees.The linked representation avoids duplicating geometry.
  • Stochastic Sampling: Stochastic sampling improves performance by accepting importance-sampled hits, while the resulting noise can be removed by an off-the-shelf denoiser.The performance gain comes with a quality cost.
  • Complex Cameras and AV Scenes: The ray-traced formulation handles highly distorted fisheye cameras and rolling-shutter effects simultaneously, unlike separate rasterization workarounds.This supports accurate rendering for camera models used in robotics and autonomous vehicles.
  • Complex Cameras and AV Scenes: 29.99 rectified PSNR versus 29.83 for ordinary 3DGS was achieved on nine Waymo scenes using the full camera model and rolling-shutter compensation.The modest improvement corresponded to correctly reconstructing important geometries such as a signpost.

7 DISCUSSION

Ray tracing offers more general ray distributions and image-formation effects than tiled rasterization, while retaining interactive performance at a measurable speed cost. The discussion highlights benefits for lighting, complex cameras, and future inverse-rendering applications, alongside interoperability and sampling limitations.

  • Generality: Ray tracing uses a BVH to render arbitrary ray distributions from any direction, unlike rasterization’s fixed 16x16 screen tiles.
  • Primary vs. General Rays: Secondary rays enable effects including reflection, refraction, depth of field, artificial shadows, global illumination, inverse lighting, and physical BSDFs.
  • Complex Cameras: Per-pixel rays model highly distorted and high-FOV fisheye cameras and rolling-shutter distortions that are difficult or impossible with tile-based rasterization.
  • Speed: Forward ray tracing is real-time and about 2× slower than tiled rasterization for primary rays from pinhole cameras; differentiable rendering is 2×-5× slower.For Truck, ray tracing takes 100ms per optimization iteration versus 30ms for rasterization; incoherent-ray training is 5× slower.
  • Sub-Pixel Behavior: Ray tracing point-samples the rendering function without rasterization’s automatic antialiasing, potentially changing subpixel-particle appearance.Point-sampled rendering is described as well-suited to modern denoisers.
  • Interoperability: Directly switching rasterizer-trained scenes to ray tracing causes a noticeable quality drop, which fine-tuning can quickly remedy.
  • Limitations: Ray tracing remains slower than rasterization for pinhole cameras, and rebuilding the BVH during training adds overhead, including for dynamic scenes.The implementation is still reported as fast enough for training and interactive rendering.
  • Future Work: Efficient particle ray tracing supports future work on inverse rendering, relighting, and material decomposition without relying on prior approximations.

A IMPLEMENTATION AND TRAINING DETAILS

The implementation wraps NVIDIA OptiX as a PyTorch extension and trains particle representations with Adam. Training combines densification, pruning, randomized-ray supervision, and transmittance-based early stopping.

  • Training Setup: The renderer is exposed as a PyTorch extension through NVIDIA OptiX and optimized with Adam for 30,000 iterations.
  • Particle Management: Densification and pruning begin after 500 iterations and continue until iteration 15,000.
  • Particle Management: Particles are split when their maximum scale exceeds 1% of scene extent and cloned otherwise; particles below opacity 0.01 are pruned.
  • Particle Management: Visibility pruning caps scenes at 3,000,000 particles by reducing excess populations to 2,700,000 using training-view contribution.
  • Ray Termination: Ray tracing terminates rays when accumulated transmittance falls below 0.001 during training and raises the threshold to 0.03 during inference.
  • Random-Ray Training: Random-ray experiments sample 524,288 rays per batch during the final 15,000 iterations and use only L1 supervision.

A.1 Autonomous Vehicles

For autonomous-vehicle scenes, the training protocol incorporates lidar and depth supervision. Initialization uses visible lidar points with image-derived colors and camera-distance-based scales.

  • Training Protocol: Autonomous-vehicle training incorporates lidar and depth supervision.
  • Initialization: Initialization randomly samples 1 million lidar points visible in at least one training image.
  • Initialization: Sampled lidar points receive initial colors by lookup in a training image and initial scales based on distance to the closest recorded camera pose.

B ADDITIONAL EXPERIMENTS AND ABLATIONS

Additional experiments compare renderings against MIPNeRF360 and examine visibility pruning across Tanks & Temples and Deep Blending. The reported qualitative results are comparable or better, while pruning preserves quality with fewer particles.

  • Qualitative Comparisons: Figure 18 compares the method with MIPNeRF360 across MIPNeRF360 and Tanks & Temples scenes.
  • Qualitative Comparisons: Zoomed-in insets show comparable or better renderings with sharp features for both tested settings.
  • Visibility Pruning: The visibility-pruning ablation evaluates maximum particle counts on Tanks & Temples and Deep Blending.
  • Visibility Pruning: Visibility pruning maintains quality with as few as one million particles by removing particles contributing least to training-view renderings.
Loading 2407.07090v3…