GS-IR: 3D Gaussian Splatting for Inverse Rendering

Zhihao Liang; Qi Zhang; Ying Feng; Ying Shan; Kui Jia

Source-linked AI summary

GS-IR: 3D Gaussian Splatting for Inverse Rendering

Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, Kui Jia

arXiv:2311.16473v3Nov 26, 2023cs.CV

TL;DR

GS-IR addresses inverse rendering from multiview images under unknown lighting, where 3D Gaussian Splatting lacks reliable normals and direct occlusion computation. It introduces regularized normal estimation and baked occlusion, achieving state-of-the-art inverse-rendering results with improved reconstruction quality and efficiency over previous neural methods.

Problem
Inverse rendering seeks to infer scene geometry, materials, and lighting from multiview images, while 3D Gaussian Splatting lacks reliable normals and direct occlusion computation.
Method
GS-IR uses 3D Gaussians with depth-derivation-based normal regularization and baking-based occlusion to estimate geometry, materials, and illumination.
Results
GS-IR achieves state-of-the-art inverse-rendering results, surpassing previous neural methods in reconstruction quality and efficiency across challenging synthetic and real scenes.
Takeaways & Limitations
3D Gaussian Splatting provides a fast, compact representation for physically based inverse rendering, including novel-view synthesis and relighting.
Takeaways & Limitations
Forward-mapping 3D Gaussian Splatting cannot use ray marching to calculate occlusions, requiring an alternative occlusion-modeling strategy.

Abstract

from arXiv · show

We propose GS-IR, a novel inverse rendering approach based on 3D Gaussian Splatting (GS) that leverages forward mapping volume rendering to achieve photorealistic novel view synthesis and relighting results. Unlike previous works that use implicit neural representations and volume rendering (e.g. NeRF), which suffer from low expressive power and high computational complexity, we extend GS, a top-performance representation for novel view synthesis, to estimate scene geometry, surface material, and environment illumination from multi-view images captured under unknown lighting conditions. There are two main problems when introducing GS to inverse rendering: 1) GS does not support producing plausible normal natively; 2) forward mapping (e.g. rasterization and splatting) cannot trace the occlusion like backward mapping (e.g. ray tracing). To address these challenges, our GS-IR proposes an efficient optimization scheme that incorporates a depth-derivation-based regularization for normal estimation and a baking-based occlusion to model indirect lighting. The flexible and expressive GS representation allows us to achieve fast and compact geometry reconstruction, photorealistic novel view synthesis, and effective physically-based rendering. We demonstrate the superiority of our method over baseline methods through qualitative and quantitative evaluations on various challenging scenes.

1. Introduction

GS-IR introduces a 3D Gaussian Splatting framework for inverse rendering from multiview images under unknown illumination. It addresses normal estimation and occlusion modeling while targeting geometry, material, and illumination recovery.

Introduction: Inverse rendering deduces scene geometry, materials, and lighting from multiview images but is ill-posed under unknown illumination.Implicit neural approaches face low expressive capacity and high computational demands.
Introduction: Adaptive Gaussian density control can produce loose geometry, making accurate normal estimation difficult and requiring dedicated regularization.GS-IR concentrates depth gradients around 3DGS to produce reliable normals.
Introduction: GS-IR introduces baking-based occlusion handling to model indirect illumination under complex effects such as soft shadows and interreflections.This addresses the limitation that forward mapping cannot directly trace occlusion like NeRF’s ray tracing.
Introduction: GS-IR is presented as a 3D Gaussian-based inverse rendering framework that simultaneously estimates scene geometry, materials, and illumination from multiview images.The framework leverages forward mapping splatting for physically-based rendering and decomposition of complex scenes.
Introduction: The method is evaluated qualitatively and quantitatively against baseline methods on challenging TensoIR synthesis and Mip-NeRF 360 real datasets.The evaluation is described as demonstrating GS-IR’s superiority over baseline methods.

2. Related Works

Neural rendering methods such as NeRF have enabled numerous task-specific neural representations, but their implicit formulations require massive repeated queries. GS-IR instead uses a 3DGS-based pipeline to recover geometry, material, and lighting for objects and unbounded scenes.

Neural Representation: Neural rendering techniques, exemplified by NeRF, have achieved impressive success in visual computing and motivated numerous task-specific neural representations.The cited passage lists neural representations tailored for different tasks.
Neural Representation: Vanilla NeRF implicitly models a continuous radiance field in MLPs, requiring massive repeated queries.This describes the computational characteristic highlighted for the vanilla NeRF formulation.
3DGS-based Inverse Rendering: GS-IR proposes a 3DGS-based pipeline that recovers geometry, material, and lighting for both objects and unbounded scenes.The pipeline is explicitly described as applicable to objects and unbounded scenes.

3. Preliminary

This section introduces 3D Gaussian Splatting as an explicit point-cloud scene representation and establishes the rendering and material models used by GS-IR. It defines Gaussian parameterization and projection, volume-rendered color, the rendering equation, and Cook-Torrance BRDF properties.

3D Gaussian Splatting: 3DGS represents a scene explicitly as a point cloud whose points are Gaussian functions approximating bell-curve shapes.Each Gaussian is defined by a mean vector and anisotropic covariance matrix.
3D Gaussian Splatting: Each Gaussian uses mean µ = (µx, µy, µz) and covariance Σ = RSS⊤R⊤, with scaling S and rotation R constructed from a unit quaternion q.The scaling matrix is diag(sx, sy, sz), while R is a rotation matrix.
Projection and Rendering: Under camera transformation, Gaussian means and covariances are projected as µ′ = KT [µ, 1]⊤ and Σ′ = JT ΣT ⊤J ⊤ using the projection Jacobian J.T and K are the extrinsic and intrinsic camera matrices, respectively.
Projection and Rendering: 3DGS renders view-dependent color using spherical-harmonic coefficients multiplied by opacity α, with accumulated transmittance Ti quantifying each Gaussian’s pixel probability density.The rendered pixel color is denoted ˆC.
The Rendering Equation: GS-IR formulates outgoing surface radiance with the classic rendering equation over incident directions in the upper hemisphere.Li(x, l) is received radiance, while l and v denote incident and view directions.
The Rendering Equation: Its Cook-Torrance BRDF uses albedo a, metallic m, and roughness ρ, with microfacet distribution D, Fresnel reflection F, and geometric shadowing G determined by ρ.GS-IR stores these material properties in 3D Gaussians.

4. Method

GS-IR extends 3D Gaussian Splatting into a three-stage inverse-rendering framework that reconstructs geometry, materials, and unknown illumination from calibrated multi-view images. It addresses forward-mapping limitations through depth-based normal estimation, baked occlusion and indirect illumination, and differentiable PBR rendering.

Geometry reconstruction: GS-IR optimizes 3D Gaussians as a geometric proxy for surface points and normals during its initial geometry-reconstruction stage.The framework starts from calibrated multi-view RGB images captured under static, unknown illumination and uses the optimized Gaussian representation for inverse rendering.
Geometry reconstruction: Depth is constrained between the minimum and maximum Gaussian distances and modeled as a linear interpolation to avoid peak-selection aliasing and floating-depth problems.Forward-mapping splatting can place accumulated depth in front of the Gaussians, unlike backward-mapping volume rendering.
Normal derivation: GS-IR derives normals from the Gaussian proxy rather than directly from noisy, view-inconsistent depth gradients, then applies total-variation regularization for smoothness.The Gaussian stores normals that are accumulated volumetrically, while the TV term addresses their roughness.
Occlusion and indirect illumination: Because forward mapping cannot ray-march occlusions, GS-IR bakes occlusion into spherical-harmonic volumes using six cubemap rendering passes and a distance-thresholded binary cubemap.The method also maintains illumination volumes to cache indirect illumination.
Intrinsic decomposition: In its final stage, GS-IR combines differentiable splatting with a physically based rendering pipeline, using image-based lighting and split-sum approximation for diffuse and specular components.The baking-based model calculates occlusion and illumination online, preserving real-time rendering performance during intrinsic decomposition.

5. Experiments

GS-IR is evaluated on synthetic objects and real unbounded scenes using metrics for normals, novel-view synthesis, albedo, relighting, and efficiency. Experiments also analyze depth-based normal regularization, indirect illumination, and relighting under varied lighting conditions.

Dataset & Metrics: Experiments use TensoIR Synthetic and Mip-NeRF 360, evaluating normal quality, novel-view synthesis, albedo fidelity, relighting, and efficiency.Normal quality uses mean angular error, while synthesis, albedo, and relighting use PSNR, SSIM, and LPIPS.
Quantitative Comparisons: GS-IR outperforms baseline methods in novel-view synthesis and albedo quality, while its normal reconstruction is slightly inferior to TensoIR and relighting ranks second behind TensoIR.These results are reported on the TensoIR Synthetic dataset and are attributed to effective material decomposition and physically based rendering.
Quantitative Comparisons: On complex real unbounded scenes, GS-IR renders high-fidelity appearance and recovers fine geometric details, including details on leaves and bicycle axles.The method’s efficient, compact, and expressive representation supports these results on Mip-NeRF 360.
Normal Regularization: Depth-derivation-based normal regularization is evaluated through experiments comparing different depth acquisition strategies and their effects on normal quality and inverse-rendering results.The analysis addresses the reliability of normal estimation for inverse rendering, with linear interpolation reported to overcome floating and disc-aliasing problems.
Indirect Illumination: Occlusion-volume modeling and indirect illumination capture intricate shadowing and occlusion details and enhance decomposition capabilities for synthetic and real scenes.Table 4 analyzes the effect of occlusion and indirect illumination on inverse rendering.
Relighting: Relighting experiments under different lighting conditions and directions produce photorealistic renderings from recovered geometry, material, and illumination.The experiments cover both synthetic and real scenes.

6. Conclusion

GS-IR presents a 3D Gaussian Splatting-based inverse rendering approach using forward-mapping volume rendering for photorealistic novel-view synthesis and relighting. Its optimization scheme estimates normals through depth-derivation regularization and models indirect lighting through baking-based occlusion for material and illumination decomposition.

Method: GS-IR applies forward-mapping volume rendering to 3D Gaussian Splatting for photorealistic novel-view synthesis and relighting.The approach is presented as a novel inverse rendering method based on 3DGS.
Method: Depth-derivation-based regularization enables normal estimation within the GS-IR optimization scheme.This regularization addresses normal estimation as one of the method’s core optimization components.
Method: Baking-based occlusion models indirect lighting, enabling GS-IR to decompose material and illumination.The regularization and occlusion components are ultimately used for material and illumination decomposition.

Supplementary Material · 7. Implementation Details

GS-IR is implemented in PyTorch with CUDA extensions and a customized baking-based method. Its training losses use image-conditioned total-variation smoothing for normals and materials, a distinct smoothing term for illumination, and fixed loss weights.

7. Implementation Details: GS-IR is implemented in the PyTorch framework with CUDA extensions and a customized baking-based method.
7. Implementation Details: The vanilla GS representation uses learnable geometric parameters T = {p, s, q} and volumetric appearance parameters A = {α, fc}.The passage identifies p as position, s as scaling, q as unit-quaternion rotation, α as opacity, and fc as spherical-harmonics coefficients.
7. Implementation Details: The normal-loss smoothing term TV normal is conditioned on the predicted normal map and reference image.Its formulation weights normal differences using image differences.
7. Implementation Details: The decomposition-stage supervision loss Ld comprises Lshade, Lmaterial, and Llight.
7. Implementation Details: The material smoothing term TV mat uses a total-variation loss similar to TV normal.The predicted material map M̂ appears in its image-conditioned formulation.
7. Implementation Details: Unlike TV normal and TV mat, TV light is defined without image conditioning for the predicted illumination map.Its formulation uses squared differences between neighboring illumination values.
7. Implementation Details: During training, λn-T V, λM, and λE are set to 5.0, 1.0, and 0.01, respectively.The efficacy of these smoothing terms is studied in Section 11.

8. Occlusion Caching and Recovery

GS-IR caches occlusion in spherical-harmonic coefficients during baking and recovers ambient occlusion through normal-aware masked-trilinear interpolation. Indirect illumination is recovered from illumination volumes using vanilla trilinear interpolation.

Occlusion caching: During baking, GS-IR caches occlusion volumes using spherical-harmonic architectures and stores occlusion components as SH coefficients.Six render passes obtain depth cubemaps before storing the occlusion representation.
Occlusion recovery: During decomposition, GS-IR recovers each surface point’s ambient occlusion from the cached occlusion volumes.The method first obtains the occlusion coefficients associated with the surface point.
Occlusion recovery: Normal-aware masked-trilinear interpolation selects contributions from neighboring volumes whose positions lie in the normal-facing hemisphere.The method finds the eight nearest volumes and masks weights according to (pk −x) · n.
Indirect illumination recovery: GS-IR recovers indirect illumination from illumination volumes using vanilla trilinear interpolation.This recovered quantity is the indirect illumination term Iindir d in Eq. (13).

10. Results on Mip-NeRF 360

On the real-world Mip-NeRF 360 dataset, GS-IR is evaluated for novel view synthesis against NeRF variants and for normal estimation, synthesis, and relighting across seven scenes.

Dataset and evaluation: Mip-NeRF 360 is a real-world captured dataset used to evaluate GS-IR.The evaluation covers novel view synthesis and additional scene-level rendering tasks.
Novel view synthesis: Novel view synthesis results are reported using PSNR, SSIM, and LPIPS for GS-IR and NeRF variants.The NeRF variants are identified as [43], with results listed in Tabs. 6 to 8.
Scene-level results: Figure 12 presents normal estimation, novel view synthesis, and relighting results for all seven publicly available scenes.These results extend the evaluation beyond the tabulated novel view synthesis comparisons.

11. Ablation on Loss

GS-IR’s loss combines fixed-weight contrast terms with tunable smoothing terms, whose effectiveness is evaluated through ablation on the TensoIR dataset.

Loss-term design: The contrast terms assign weight 1 to color reconstruction, normal penalty, and shade losses.These fixed weights are described as intuitive.
Loss-term design: The ablation varies the smoothing-term weights for TV normal, TV material, and TV light.The adjusted weights are λn-T V, λE, and λM.
Ablation protocol: Table 9 analyzes the impact of different loss terms on the TensoIR dataset, with checkmarks indicating valid smoothing terms.The table reports the resulting ablation comparisons.