Source-linked AI summary
Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracing
Jian Gao, Chun Gu, Youtian Lin, Zhihao Li, Hao Zhu, Xun Cao, Li Zhang, Yao Yao
TL;DR
Existing 3D Gaussian Splatting does not provide relightable scenes or robust point-based ray tracing for realistic shadows. The paper adds per-point geometry, BRDF, and lighting properties with differentiable physically based rendering, and introduces BVH ray tracing for visibility; experiments report improved material and lighting estimation, novel-view rendering, and relighting, while the method assumes BRDF smoothness in homogeneous areas.
Problem
3D Gaussian Splatting is limited to novel-view synthesis because it cannot relight scenes under different lighting, while point-based ray tracing remains unresolved.
Method
The framework augments 3D Gaussians with normals, BRDFs, and incident lighting, decomposes lighting through physically based differentiable rendering, and uses BVH ray tracing for visibility.
Results
Experiments confirm reasonable normals, material and lighting estimation, novel-view rendering quality, and realistic relighting with accurate visibility and shadows.
Takeaways & Limitations
A discretized point-based graphics pipeline can support relighting, editing, and ray tracing within one representation.
Takeaways & Limitations
The method assumes BRDF properties change smoothly in homogeneous areas.
Abstract
from arXiv · showhide
In this paper, we present a novel differentiable point-based rendering framework to achieve photo-realistic relighting. To make the reconstructed scene relightable, we enhance vanilla 3D Gaussians by associating extra properties, including normal vectors, BRDF parameters, and incident lighting from various directions. From a collection of multi-view images, the 3D scene is optimized through 3D Gaussian Splatting while BRDF and lighting are decomposed by physically based differentiable rendering. To produce plausible shadow effects in photo-realistic relighting, we introduce an innovative point-based ray tracing with the bounding volume hierarchies for efficient visibility pre-computation. Extensive experiments demonstrate our improved BRDF estimation, novel view synthesis and relighting results compared to state-of-the-art approaches. The proposed framework showcases the potential to revolutionize the mesh-based graphics pipeline with a point-based pipeline enabling editing, tracing, and relighting.
1 Introduction
The paper extends 3D Gaussian Splatting into a relightable point-based pipeline by adding material and lighting properties, physically based rendering, and BVH ray tracing for occlusion-aware shadows.
- 3D Gaussian Splatting reconstructs scenes efficiently but cannot relight them under different lighting conditions and remains limited in point-based ray tracing.
- The pipeline supports multi-object composition, relighting, editing, and ray tracing using a discretized point representation.
- The method assigns each Gaussian normal vectors, BRDF values, and incident lighting, then uses physically based rendering before alpha compositing.
- BVH-based point ray tracing enables efficient visibility pre-computation and handles complex occlusions for realistic shadow effects.
2 Related Works
Related work spans neural radiance fields and inverse-rendering approaches that model varying illumination, visibility, materials, and indirect lighting.
- Neural Radiance Fields use an implicit MLP to map 3D positions and viewing directions to density and view-dependent colors for differentiable volume rendering.
- Prior inverse-rendering methods address varying illumination, visibility, material and lighting decomposition, indirect lighting, and direct optimization on mesh surfaces.
3 Relightable 3D Gaussians
The framework extends explicit 3D Gaussian primitives with geometry, material, and lighting attributes, then optimizes them through differentiable physically based rendering. Geometry enhancement, BRDF modeling, lighting decomposition, and regularization support relightable point-based rendering.
- 3.1 Preliminary: 3D Gaussian Splatting represents scenes with explicit Gaussian points, each parameterized by spatial mean, covariance, opacity, and view-dependent color.The points are projected to 2D and composited through front-to-back alpha blending.
- 3.1 Preliminary: The proposed pipeline applies the physically based rendering equation to each Gaussian before rasterization and alpha blending produce color, PBR color, depth, and normal maps.Ground-truth color and pseudo normals derived from depth supervise optimization.
- 3.2 Geometry Enhancement: Normals are optimized by back-propagation and supervised through rendered-depth expectations, pseudo normals, normal consistency, depth uncertainty, and optional object-mask constraints.Normal-gradient densification targets thin regions, while depth-distribution constraints encourage points toward the object surface.
- 3.3 BRDF and Light Modeling: Each Gaussian receives BRDF properties and incident-light information, while PBR colors are computed per Gaussian rather than through coordinate-based neural fields.This reduces repeated image-pixel rendering and avoids querying a global neural field at millions of points.
- 3.3 BRDF and Light Modeling: Incident lighting is decomposed into a globally shared direct component and individual per-Gaussian indirect components to constrain lighting estimation.The BRDF uses a simplified Disney model with albedo and specular properties.
- 3.4 Regularizations: Smoothness priors regularize roughness, normal, and albedo in regions expected to be homogeneous, with image gradients modulating the roughness constraint.The rendered roughness map is formed by weighted per-Gaussian roughness values.
4 Point-based Ray Tracing
The paper introduces BVH-based ray tracing for semi-transparent 3D Gaussians, enabling efficient visibility queries and realistic shadow effects. Rays traverse bounding volumes, identify contributing Gaussians, approximate intersections at peak contribution points, and accumulate transmittance.
- Ray tracing structure: BVH-based ray tracing enables efficient visibility querying for 3D Gaussians.A binary radix tree uses Gaussian bounding boxes at leaves and child-enclosing boxes at internal nodes.
- Ray tracing structure: Ray traversal recursively tests child bounding volumes until reaching leaf nodes containing intersected Gaussians.The traversal begins at the BVH root and proceeds through intersected child nodes.
- Equivalent intersections: Because Gaussian influence spans a ray segment, the method represents the intersection by the point of peak Gaussian contribution.The equivalent intersection point is computed using Eq. 13 and the ray parameter defined in Eq. 14.
- Transmittance: At each equivalent intersection, a Gaussian contributes opacity αj, while ray transmittance is progressively attenuated by accumulated Gaussian opacities.The transmittance update is given by T_i = (1 - α_{i-1})T_{i-1}, with T_1 = 1.
- Transmittance: Early termination accelerates tracing when a ray’s transmittance falls below Tmin.This reduces computation after sufficient attenuation has accumulated.
- Visibility and relighting: For static scenes, per-Gaussian visibility is pre-computed across the hemisphere and then used while optimizing material and lighting parameters.Optimization first estimates geometry and normals, then locks Gaussian geometry for material and lighting optimization.
- Visibility and relighting: The resulting point-based pipeline combines scene editing with realistic relighting, including occlusion handling in multi-object compositions.The method updates visibility under inter-object occlusions in novel lighting environments.
5 Experiments
Experiments evaluate novel view synthesis, relighting, material decomposition, real-world relighting, and ablations of the method’s principal components. Results show strong rendering and relighting quality, while component studies identify improvements from normal densification, full lighting, and depth-distribution constraints.
- Evaluation scope: The evaluation covers novel view synthesis, decomposed materials, relighting, and real-world scenes across NeRF synthetic, Synthetic4Relight, and Mip-NeRF 360 datasets.The experiments combine quantitative comparisons with qualitative visualizations and ablations.
- Novel View Synthesis: R3DG slightly trails vanilla 3DGS in novel view synthesis but outperforms most point-based methods and significantly surpasses other relightable approaches in NVS quality.The comparison reports PSNR, SSIM, and LPIPS averages across NeRF synthetic scenes.
- Synthetic4Relight: The method outperforms existing approaches in Synthetic4Relight novel view synthesis and relighting precision, with superior albedo accuracy in SSIM and LPIPS.Qualitative results also show visually pleasing material decomposition and realistic relighting.
- Real-world Relighting: On real-world Mip-NeRF 360 scenes, the method produces pleasing shadow effects in a kitchen under the second environment map.The real-world evaluation is presented as qualitative relighting.
- Ablation Study: Normal gradient densification enhances thin-region normal-map detail, while the full lighting model and depth-distribution constraint improve relighting, ambient occlusion, and depth certainty.The ablations compare NGD, full lighting against a global environment map, and the constraint Lu against corresponding simplified variants.
6 Conclusion
The paper concludes that Relightable 3D Gaussians combine inverse rendering with point-based ray tracing to support editing, visibility computation, and photo-realistic relighting. The current pipeline targets static objects, and large scenes remain challenging because high point density can slow optimization during ray sampling.
- Conclusion: The proposed pipeline enriches 3D Gaussians with normals, BRDFs, and indirect lighting, enabling differentiable reconstruction, ray tracing, editing, and photo-realistic relighting.Experiments support reasonable normal and material reconstruction together with accurate novel view synthesis and scene relighting.
- Limitations: The current pipeline targets static objects, and design choices make maintaining performance on large-scale scenes challenging because high point density can slow optimization during ray sampling.This is identified as a limitation and future-work direction.