Source-linked AI summary
GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces
Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, Yuexin Ma
TL;DR
Reflective surfaces expose limitations in explicit 3D Gaussian representations because view-dependent appearance is difficult to model. GaussianShader adds simplified shading and shortest-axis-based normal estimation, reporting a balance of efficiency and quality while retaining real-time rendering.
Problem
3D Gaussian Splatting offers efficient rendering but performs poorly on reflective surfaces because it does not explicitly model view-dependent appearance.
Method
GaussianShader applies a simplified shading function to 3D Gaussians and estimates normals using shortest-axis directions with a normal-geometry consistency loss.
Results
GaussianShader achieves a balance between efficiency and robust performance on general and reflective scenes while maintaining real-time rendering.
Takeaways & Limitations
The approach supports realistic rendering of reflective and general objects with the efficiency required for interactive applications.
Takeaways & Limitations
On Tanks and Temples, improvement over Gaussian Splatting is modest because the dataset is predominantly diffuse and does not fully leverage the method’s strengths.
Abstract
from arXiv · showhide
The advent of neural 3D Gaussians has recently brought about a revolution in the field of neural rendering, facilitating the generation of high-quality renderings at real-time speeds. However, the explicit and discrete representation encounters challenges when applied to scenes featuring reflective surfaces. In this paper, we present GaussianShader, a novel method that applies a simplified shading function on 3D Gaussians to enhance the neural rendering in scenes with reflective surfaces while preserving the training and rendering efficiency. The main challenge in applying the shading function lies in the accurate normal estimation on discrete 3D Gaussians. Specifically, we proposed a novel normal estimation framework based on the shortest axis directions of 3D Gaussians with a delicately designed loss to make the consistency between the normals and the geometries of Gaussian spheres. Experiments show that GaussianShader strikes a commendable balance between efficiency and visual quality. Our method surpasses Gaussian Splatting in PSNR on specular object datasets, exhibiting an improvement of 1.57dB. When compared to prior works handling reflective surfaces, such as Ref-NeRF, our optimization time is significantly accelerated (23h vs. 0.58h). Please click on our project website to see more results.
1. Introduction
GaussianShader addresses the difficulty of rendering reflective surfaces with efficient 3D Gaussian Splatting by adding simplified shading and improved normal estimation. It aims to retain real-time efficiency while improving quality across reflective and general scenes.
- Motivation: 3D Gaussian Splatting enables high-quality modeling and real-time rendering but performs poorly on specular and reflective surfaces.Its lack of explicit appearance modeling limits its ability to capture strong view-dependent changes, especially specular highlights.
- Motivation: Reflective-surface methods based on implicit representations remain costly to optimize and render, while ENVIDR struggles with complex scenes and general objects.The paper identifies combining shading functions with Gaussian Splatting while preserving efficiency as an open problem.
- Approach: GaussianShader adds a simplified shading function to 3D Gaussians, modeling diffuse colors and direct reflections while representing advanced reflections with a residual color term.The residual term allows more complex reflective appearances than a shading function considering only direct reflections.
- Approach: The method estimates normals from each Gaussian sphere’s shortest axis and learns a normal residual, then applies a normal-geometry consistency constraint.This avoids locally continuous surface reconstruction and expensive neighborhood searches for normal computation.
- Results: GaussianShader maintains real-time rendering while accommodating reflective materials and balancing efficiency with robust performance on general and reflective scenes.The method explicitly approximates the rendering equation and provides a new normal estimation framework for realistic rendering.
2. Related Work
Prior work addresses reflective rendering through neural fields, inverse rendering, and shading-aware representations, but these approaches face capture, efficiency, or flexibility constraints. GaussianShader builds on differentiable 3D Gaussian primitives and rapid 2D splatting.
- Neural rendering: NeRF introduced high-quality photorealistic novel-view synthesis through implicit representation and volume rendering, motivating many extensions.Related work improves rendering quality, geometric reconstruction, and other aspects of neural rendering.
- Reflective object rendering: Reflective-object rendering is difficult because complex light interactions require accurate scene illumination and material-property estimation.Earlier light-field approaches required dense discrete captures, while inverse rendering targets lighting and BRDF estimation.
- GaussianShader: GaussianShader extends Gaussian Splatting with shading attributes and a differentiable environment-light map for modeling reflective and diffuse appearances.The shading attributes include diffuse color, roughness, specular tint, normal, and residual color.
- Reflective object rendering: NeRF-based methods disentangle appearance into lighting and material properties, while Ref-NeRF and SDF-based methods introduce view-dependent or reflectance-aware modeling.These approaches address unknown or varying lighting and surface reflectance through specialized parameterizations or representations.
- 3D Gaussian Splatting: Projected Gaussian spheres support rapid 2D splatting with alpha blending, while spherical harmonics represent directional appearance.Tile-based rasterization sorts Gaussians by depth before accumulating pixel colors and applying image losses.
3. Method
GaussianShader extends 3D Gaussian Splatting with a simplified shading model for reflective surfaces and a normal-estimation framework that maintains efficiency while improving geometric consistency.
- 3.1. Shading on 3D Gaussians: GaussianShader represents each Gaussian sphere with shading attributes including diffuse color, roughness, specular tint, normal, and residual color.A differentiable environment light map models direct lighting.
- 3.1. Shading on 3D Gaussians: The method approximates the rendering equation with diffuse color, direct specular lighting, and a residual color term for unexplained reflections.The residual term accounts for effects such as scattering and indirect-light reflections.
- 3.2. Specular Light: The specular-light function integrates incoming radiance with the GGX normal distribution, whose lobe varies with roughness around the reflective direction.A trainable cube map is pre-filtered into mip maps so specific direction–roughness integrals can be interpolated efficiently.
- 3.3. Normal Estimation: GaussianShader estimates normals from each Gaussian’s shortest axis, which becomes a useful approximation as optimization progressively flattens Gaussian spheres toward planar shapes.Trainable residuals refine the axis-based normals, with two residuals handling the axis orientation ambiguity relative to the viewing direction.
- 3.3. Normal Estimation: A normal-geometry consistency loss aligns rendered normals with depth-gradient normals, linking separately estimated Gaussian normals to local geometry.The depth-derived normals capture geometry formed by multiple Gaussian spheres, while the normal map contains the per-sphere estimates.
4. Experiments
Experiments evaluate GaussianShader across novel-view synthesis, reflective objects, and large-scale scenes, using comparisons, ablations, and efficiency measurements. The method generally preserves efficient rendering while improving reflective-surface quality and normal estimation.
- Evaluation setup: Experiments cover NeRF Synthetic, Shiny Blender, Glossy Synthetic, and Tanks and Temples, with comparisons against Gaussian Splatting and prior reconstruction methods.The evaluation includes quantitative, qualitative, and ablation studies.
- NeRF Synthetic: GaussianShader achieves numerically and visually comparable results to Gaussian Splatting and neural rendering methods on NeRF Synthetic general objects.ENVIDR performs poorly in shadowed regions of complex objects.
- Shiny Blender: GaussianShader outperforms original Gaussian Splatting in all Shiny Blender scenes while slightly underperforming ENVIDR and Ref-NeRF.Its qualitative results correctly render surfaces with strong specular appearances.
- Efficiency: 0.5 hour is the reported training time for GaussianShader, versus 6 hours for ENVIDR and 23 hours for Ref-NeRF on the same hardware.The method is somewhat slower than Gaussian Splatting but retains reasonable efficiency and real-time rendering speed.
- Large-scale scenes and ablations: GaussianShader outperforms Gaussian Splatting qualitatively on Tanks and Temples and produces smoother, more reasonable normals.The ablations report that direct normal regularization, residual color, shortest-axis normal formulation, and explicit environment-map lighting improve rendering quality or reflective-object rendering.
- Glossy Synthetic: GaussianShader provides high-fidelity renderings with detailed normal and lighting maps on Glossy Synthetic, capturing reflective-surface and specular-lighting details.Compared with Gaussian Splatting, the method reports an average PSNR increase of 1.1 on Glossy Synthetic.
5. Conclusions
GaussianShader extends 3D Gaussian Splatting with shading functions and a normal prediction method to render reflective and general objects with high quality.
- GaussianShader integrates shading functions with 3D Gaussians to handle view-dependent appearances on reflective surfaces.
- The method predicts high-quality normals correlated with the true geometry of 3D Gaussians.
- The reported results show high-quality renderings and improved efficient, realistic 3D object rendering.
6. Residual Color Training Details
Residual color uses third-order spherical-harmonics coefficients and is delayed during training to support material and lighting separation before later refinement.
- Residual color comprises third-order SH coefficients and is initially excluded from training.
- The residual color is introduced in later training stages for refinement after color learning is attributed to materials and lighting.
7. Additional Results
Additional results evaluate GaussianShader on relighting and quantitative comparisons with Gaussian Splatting across glossy and predominantly diffuse datasets.
- Relighting: Four diverse relighting sets show realistic scenes whose object highlights mirror surrounding light sources under varied lighting conditions.The examples include warm and cool tones and indoor and outdoor environments.
- Glossy Synthetic: 1.1 average PSNR improvement over Gaussian Splatting is reported on the Glossy Synthetic dataset.
- Tanks and Temples: GaussianShader shows a modest improvement over Gaussian Splatting on Tanks and Temples, whose predominantly diffuse objects less fully leverage the method.
- Quantitative comparisons: Table 5 presents quantitative comparisons with Gaussian Splatting on Glossy Synthetic.
- Quantitative comparisons: Table 6 presents quantitative comparisons with Gaussian Splatting on Tanks and Temples.