PaperlayerView exact revision on arXiv

Source-linked AI summary

RT-Splatting: Joint Reflection-Transmission Modeling with Gaussian Splatting

Ji Shi, Xianghua Ying, Bowei Xing, Ruohao Guo, Wenzhen Yue

arXiv:2605.18263v1cs.CV

TL;DR

Semi-transparent specular surfaces remain difficult for 3D Gaussian Splatting because reflection and transmission coexist. RT-Splatting factorizes geometric occupancy from optical opacity and uses hybrid rendering with specular-aware gradient gating, achieving state-of-the-art performance with sharp reflections and clear transmission.

  • Problem

    3D Gaussian Splatting struggles to represent semi-transparent specular surfaces with coexisting reflection and transmission, often producing floaters that blur reflections and occlude transmitted backgrounds.

  • Method

    RT-Splatting factorizes each Gaussian’s geometric occupancy and optical opacity, combining hybrid surface-volume rendering with Specular-Aware Gradient Gating.

  • Results

    RT-Splatting achieves state-of-the-art performance on challenging scenes with strongly coupled reflection and transmission.

  • Takeaways & Limitations

    A single Gaussian representation supports high-fidelity reflections, clear transmission, real-time rendering, and flexible scene editing.

  • Takeaways & Limitations

    Existing reflection-transmission methods remain limited to simple planar surfaces and do not generalize to complex shapes.

Abstract

from arXiv · show

3D Gaussian Splatting (3DGS) enables real-time novel view synthesis with high visual quality. However, existing methods struggle with semi-transparent specular surfaces that exhibit both complex reflections and clear transmission, often producing blurry reflections or overly occluded transmission. To address this, we present RT-Splatting, a framework that disentangles each Gaussian's geometric occupancy from its optical opacity. This factorization yields a unified surface-volume scene representation with a single set of Gaussian primitives. Our hybrid renderer interprets this representation both as a surface to capture high-frequency reflections and as a volume to preserve clear transmission. To mitigate the ambiguity in jointly optimizing reflection and transmission, we introduce Specular-Aware Gradient Gating, which suppresses misleading gradients from highly specular regions into the transmission branch, effectively reducing distracting floaters. Experiments on challenging semi-transparent scenes show that RT-Splatting achieves state-of-the-art performance, delivering high-fidelity reflections and clear transmission with real-time rendering. Moreover, our factorization naturally enables flexible scene editing. The project page is available at https://sjj118.github.io/RT-Splatting.

1. Introduction

RT-Splatting addresses 3DGS failures on thin semi-transparent specular surfaces by jointly modeling reflection and transmission with a unified surface-volume Gaussian representation. It factorizes geometric occupancy from optical opacity and uses Specular-Aware Gradient Gating to reduce transmission floaters caused by misleading specular gradients.

  • Motivation: 3DGS struggles with semi-transparent specular surfaces because behind-surface floaters distort reflections and spuriously occlude transmission.These floaters fail to capture the true reflected appearance while corrupting visibility of background geometry.
  • Method: RT-Splatting jointly models reflection and transmission using a single unified surface-volume representation of Gaussian primitives.The framework targets real-world scenes containing thin semi-transparent surfaces.
  • Method: The representation factorizes each Gaussian’s contribution into geometric occupancy and optical opacity, separating surface roles from ray-wise light attenuation.This factorization enables the same Gaussian primitives to support surface and volume rendering interpretations.
  • Optimization: Specular-Aware Gradient Gating attenuates misleading gradients from complex specular regions into the transmission branch, reducing erroneous floaters.The mechanism identifies pixels dominated by complex specular patterns and suppresses misleading supervision that would otherwise degrade background clarity.

2. Related Work

Prior work improves reflective-scene reconstruction through direction-conditioned radiance, stronger directional encodings, surface-geometry recovery, and real-time Gaussian Splatting. However, semi-transparent surfaces remain difficult because reflection and transmission are view-dependent and volumetric opacity conflates geometry with transmissivity.

  • Reflective-scene reconstruction: Ref-NeRF conditions outgoing radiance on reflection direction, while later methods strengthen directional encodings or recover surface geometry to capture specular effects.These approaches target high-frequency light-surface interactions and shape-radiance ambiguity.
  • Reflective-scene reconstruction: GaussianShader, 3DGS-DR, and EnvGS extend reflective-scene rendering with real-time Gaussian Splatting using normals, deferred G-buffers, and differentiable ray tracing.EnvGS uses environment Gaussians to capture near-field reflections in real time.
  • Semi-transparent surfaces: Despite strong high-frequency specular representation, these Gaussian methods struggle with thin semi-transparent surfaces combining transmitted and reflected light.The limitation concerns appearances formed by both light passing through and light reflecting from the surface.
  • Semi-transparent surfaces: Volumetric alpha blending can simulate translucency but conflates geometric surface presence with optical transmissivity, preventing distinct geometry for physically based shading.Existing work therefore explores first-surface extraction strategies to recover transparent-object surfaces explicitly.
  • Semi-transparent surfaces: Disentangling reflection and refraction is inherently ill-posed because both are highly view-dependent and lack multi-view photometric consistency.A practical direction focuses on thin glass or plastic surfaces where negligible refraction permits straight-path transmission approximations.

3. Preliminaries

The preliminaries describe 3DGS as an anisotropic Gaussian representation rendered by depth-sorted alpha blending, alongside 2DGS surface alignment and deferred shading through G-buffers.

  • 3D Gaussian Splatting: Rendering projects Gaussians onto the image plane, sorts them by depth, and computes pixel colors using front-to-back alpha blending.The blending weight combines each Gaussian’s opacity and projected 2D kernel value with transmittance from preceding Gaussians.
  • 2D Gaussian Splatting: 2DGS models scenes as 2D Gaussian surfels embedded in 3D, providing surface normals and mitigating multi-view depth inconsistencies from projected 3D Gaussians.
  • Deferred Shading: Deferred shading separates geometry processing from lighting and material computation through geometry-pass G-buffers followed by per-pixel shading.The G-buffers store attributes including depth, normal, albedo, and roughness.

4. Method

RT-Splatting factorizes Gaussian opacity into geometric occupancy and optical opacity, enabling a unified surface-volume representation for sharp reflections and clear transmission. Its hybrid renderer combines deferred surface reflection with forward volumetric transmission, while specular-aware gating and transparent-mask supervision address optimization ambiguity and floaters.

  • Opacity Factorization: Transparent objects can have high geometric occupancy but low optical opacity, contributing to surface aggregation without excessively occluding volumetric transmission.This factorization separates geometric presence from optical properties, overcoming the conflation inherent in a single opacity parameter.
  • Hybrid Rendering: The hybrid renderer uses a deferred pass for high-frequency first-surface specular reflections and a forward pass for volumetric transmission through the effective opacity.Surface properties are aggregated into G-buffers for specular shading, while the transmitted and scattered components are combined through Csub = τCtrans + (1-τ)Cscatter.
  • Opacity Factorization: RT-Splatting factorizes each Gaussian’s opacity into geometric occupancy σ and optical opacity α, with effective opacity αeff = σα for volumetric compositing.Geometric occupancy models interaction with the Gaussian substance, while optical opacity models conditional absorption or scattering.
  • Hybrid Rendering: The final color combines specular reflection with an attenuated subsurface-transport component, suppressing transmitted background light under strong highlights.The formulation uses C = Cspec + βCsub and attenuates transmission rather than modulating reflection.
  • Specular-Aware Gradient Gating: Specular-Aware Gradient Gating uses local variance of the specular component to reduce misleading image-loss gradients into the transmission branch and prevent spurious floaters.The gate scales gradients through the transmitted background color, targeting regions with high-frequency specular details.
  • Optimization Supervision: A transparent mask loss provides explicit supervision for Gaussian optical opacity to resolve ambiguity in jointly optimizing reflection and transmission.The method leverages a transparent mask to supervise optical opacity directly.

5. Experiments

RT-Splatting is evaluated on public and self-captured semi-transparent scenes against established Gaussian Splatting baselines, consistently outperforming them while rendering in real time. Ablations explain the benefits of its factorization, joint optimization, gradient gating, and mask regularization, while decomposition enables intuitive scene editing.

  • Datasets: The evaluation covers six public scenes and two smartphone-captured scenes featuring high-frequency reflections and clear transmission on semi-transparent surfaces.The captured Van and Swab scenes each contain 220 ∼240 views.
  • Quantitative comparison: RT-Splatting is compared with 3DGS, 2DGS, GaussianShader, 3DGS-DR, Ref-GS, and EnvGS using image-wide and transparent-region metrics.The reported metrics are PSNR, SSIM, and LPIPS; baselines use publicly available codebases and configurations.
  • Quantitative comparison: RT-Splatting consistently outperforms all baselines across evaluated metrics, with particularly significant gains in transparent regions, while maintaining real-time rendering and competitive training time.Results are reported on both public benchmarks and self-captured scenes.
  • Qualitative comparison: Qualitative comparisons show that RT-Splatting simultaneously reconstructs sharp reflection details and clear transmitted light, unlike methods compromised by reflection-transmission ambiguity.Existing approaches often blur reflections or struggle to preserve transmitted-light structure during optimization.
  • Ablation study: Ablations show that removing occupancy-opacity factorization compromises transmission, separating reflection and transmission training prevents truck-interior reconstruction, and disabling gating or mask regularization causes artifacts or instability.Without factorization, high opacity needed for sharp reflections occludes the background; without gating, artifacts appear near transparent surfaces; without Lmask, surface quality degrades.
  • Scene editing: The reflection-transmission decomposition supports independent editing of roughness, transparency, specular reflections, and material tint.These operations demonstrate intuitive control over decoupled surface attributes.

6. Conclusion

RT-Splatting jointly models high-fidelity reflections and clear transmissions on thin semi-transparent surfaces using disentangled geometric occupancy and optical opacity. Its hybrid representation supports reflective-surface and transmissive-volume rendering, while its stated limitation is the lack of explicit refraction and multiple-light-bounce modeling.

  • Conclusion: RT-Splatting jointly models high-fidelity reflections and clear transmissions on semi-transparent surfaces.The framework is designed for thin semi-transparent surfaces.
  • Conclusion: Disentangling geometric occupancy from optical opacity lets one Gaussian set support reflective-surface and transmissive-volume interpretations.The hybrid renderer targets sharp specular highlights and clear background content.
  • Conclusion: RT-Splatting does not explicitly model refraction or multiple light bounces.Future extensions could target thicker refractive media and multi-bounce transport, including water or solid glass objects.

Supplementary Material

The supplementary material provides implementation details, analyzes Specular-Aware Gradient Gating sensitivity to gating strength k, and reports per-scene metrics with additional qualitative results.

  • Supplementary Material: The supplementary material includes implementation details, a gating-strength k ablation, per-scene quantitative metrics, and further qualitative results.These materials are organized in Sections A–C.

A. Implementation Details · A.1. Density Control Strategy · A.2. Specular-Aware Gradient Gating

The implementation follows established evaluation and preprocessing protocols on a single RTX 4090 GPU, while adapting density control and gradient gating to the occupancy-opacity representation. The gating uses a 3 × 3 local window, k = 4, and partial stop-gradient modulation that preserves forward values while scaling backward gradients.

  • A. Implementation Details: Experiments follow the evaluation protocols of 3DGS-DR and Ref-GS, including spherical foreground regions and consistent image downsampling.All experiments run on a single NVIDIA RTX 4090 GPU.
  • A. Implementation Details: The implementation uses consistent image downsampling factors across experiments to ensure fair comparison.The preprocessing standards are adopted from 3DGS-DR and Ref-GS.
  • A.1. Density Control Strategy: The density control strategy adapts standard 2DGS to manage primitives under the proposed occupancy-opacity factorization.Unlike standard 2DGS, the decoupled representation requires controlling both geometric occupancy and optical opacity.
  • A.2. Specular-Aware Gradient Gating: Specular-aware gradient gating computes local variance with a 3 × 3 window and sets k = 4.The value of k is selected based on sensitivity analysis in Sec. B.
  • A.2. Specular-Aware Gradient Gating: The gradient modulation is implemented through partial stop-gradient, combining stopped and unstopped transmission colors using the gate g.The gated color is defined as (1 - g)·sg(C_trans) + g·C_trans.
  • A.2. Specular-Aware Gradient Gating: The gated transmission color equals the original transmission color during the forward pass while scaling backward gradients by g.This preserves forward rendering behavior while applying gradient modulation during backpropagation.

A.3. Losses

The method combines normal consistency, perceptual appearance, and transparent-mask regularization losses into a single total objective. Appearance supervision augments L1 and D-SSIM with a perceptual term weighted by λ = 0.2 and λperc = 0.01.

  • Losses: Normal consistency loss Ln enforces geometric alignment between rendered normals and depth gradients.This follows 2DGS [14].
  • Losses: Appearance loss Limg combines L1, D-SSIM, and perceptual losses as (1 - λ)L1 + λL_D-SSIM + λpercLperc.The perceptual loss Lperc is adopted from EnvGS [41].
  • Losses: The appearance-loss weights are λ = 0.2 and λperc = 0.01.These weights apply to the D-SSIM and perceptual terms, respectively.
  • Losses: The total loss combines Limg, weighted normal consistency λnLn, and weighted transparent-mask regularization λmaskLmask.The objective is L = Limg + λnLn + λmaskLmask.

B. Additional Ablation Study

The ablation study evaluates the Specular-Aware Gradient Gating strength k across eight scenes, finding that k = 4 provides the best balance between suppressing specular artifacts and preserving transmission details.

  • Additional Ablation Study: Performance peaks at k = 4, offering the optimal tradeoff between suppressing specular artifacts and preserving transmission details.The quantitative results are averaged over transparent regions across all eight scenes.

C. Additional Results

RT-Splatting reports per-scene image and transparent-region metrics and further qualitative decomposition results. Compared with baselines, it separates surface reflections and transmission with high fidelity, whereas baselines often recover only one component.

  • Qualitative Results: Baseline methods often fail to reconstruct surface reflections and transmission simultaneously, recovering one component at the expense of the other.Additional qualitative results in Fig. 6 demonstrate the decomposition capability of RT-Splatting.
  • Qualitative Results: RT-Splatting achieves high-fidelity separation of Reflection and Transmission layers in real-world scenes.Its Normal component captures surface geometry, while Depth corresponds to volumetric accumulation; baseline diffuse components are visualized as Transmission.
Loading 2605.18263v1…