Source-linked AI summary
Modeling Indirect Illumination for Inverse Rendering
Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, Xiaowei Zhou
TL;DR
Inverse rendering under unknown illumination rarely models indirect light because recursive path tracing is computationally intractable. This paper derives spatially varying indirect illumination from a neural outgoing radiance field and combines it with visibility-aware direct illumination. On synthetic and real data, the pipeline recovers high-quality albedo and roughness and supports realistic relighting.
Problem
Indirect illumination is rarely modeled in inverse rendering because expensive recursive path tracing makes optimization computationally intractable.
Method
The method learns geometry and an outgoing radiance field, derives indirect illumination with an MLP, and fixes it while optimizing SVBRDF and environmental light.
Results
The full pipeline estimates high-quality albedo and roughness from natural-illumination multi-view images and supports realistic relighting on synthetic and real data.
Takeaways & Limitations
The approach can recover shadow- and interreflection-free albedo and enables realistic free-viewpoint relighting.
Takeaways & Limitations
The pipeline strongly relies on fine geometry and cannot handle cases where the geometry reconstruction fails.
Abstract
from arXiv · showhide
Recent advances in implicit neural representations and differentiable rendering make it possible to simultaneously recover the geometry and materials of an object from multi-view RGB images captured under unknown static illumination. Despite the promising results achieved, indirect illumination is rarely modeled in previous methods, as it requires expensive recursive path tracing which makes the inverse rendering computationally intractable. In this paper, we propose a novel approach to efficiently recovering spatially-varying indirect illumination. The key insight is that indirect illumination can be conveniently derived from the neural radiance field learned from input images instead of being estimated jointly with direct illumination and materials. By properly modeling the indirect illumination and visibility of direct illumination, interreflection- and shadow-free albedo can be recovered. The experiments on both synthetic and real data demonstrate the superior performance of our approach compared to previous work and its capability to synthesize realistic renderings under novel viewpoints and illumination. Our code and data are available at https://zju3dv.github.io/invrender/.
1. Introduction
Inverse rendering from casually captured multi-view images remains difficult because natural illumination produces shadows and interreflections that are expensive to simulate. The paper derives indirect illumination from a learned outgoing radiance field, enabling more efficient SVBRDF estimation and realistic relighting.
- Motivation: Inverse rendering seeks to recover geometry, materials, and lighting from images, but conventional controlled capture systems limit user-friendliness.The paper motivates more flexible capture under natural illumination for VR and AR content creation.
- Problem: Natural illumination introduces soft shadows and interreflections whose recursive path tracing makes joint SVBRDF and lighting optimization intractable.Ignoring these effects can cause them to be absorbed into recovered reflectance and lighting.
- Approach: The method models indirect illumination with an MLP derived from the outgoing radiance field learned from multi-view images, rather than jointly learning it with SVBRDF and environmental light.This separates indirect-illumination reconstruction from the remaining inverse-rendering optimization.
- Approach: The pipeline first learns geometry and outgoing radiance, trains the indirect-illumination MLP from reachable surface points, then fixes it while optimizing SVBRDF and environmental light.Direct querying avoids recursive path tracing and makes the inverse-rendering problem better constrained and more efficient to solve.
- Results: The approach outperforms baseline methods on synthetic and real data while recovering shadow- and interreflection-free albedo, high-quality roughness, and realistic free-viewpoint relighting.These are the reported outcomes across both evaluation settings.
2. Background
Inverse rendering is ill-posed because image appearance must be decomposed into geometry, materials, and lighting. Neural representations support more flexible capture, but prior methods largely model only direct illumination, while the rendering equation integrates reflected incoming light over directions.
- Inverse rendering: Inverse rendering decomposes image appearance into intrinsic geometry, material, and lighting properties, and requires priors or regularization because the general problem is severely ill-posed.The paper frames this decomposition as a longstanding computer-vision and graphics problem.
- Prior approaches: Single-image methods use strong planar-image priors to infer plausible materials and normals but usually cannot recover spatially varying 3D representations.Their CNN-friendly planar maps support inference while limiting fully spatially varying 3D recovery.
- Prior approaches: Fully factorized 3D recovery methods commonly require constrained capture, such as rotating objects or handheld flash video, because varied or known illumination supplies information about geometry and materials.These settings improve observability but reduce capture flexibility.
- Implicit neural representation: Implicit neural representations enable differentiable inverse rendering from image collections, with NeRF modeling volumetric radiance fields and IDR or NeuS modeling surfaces with SDFs.These representations support novel-view synthesis or surface reconstruction from posed images.
- Prior approaches: Recent inverse-rendering methods use implicit representations under complex or unknown illumination, but generally consider only direct light from sources and ignore indirect illumination.This omission motivates explicitly modeling indirect effects.
- Rendering equation: The rendering equation computes outgoing radiance by integrating incoming radiance, BRDF response, and the cosine-weighted incident direction over the hemisphere.Lin(ˆx, ωi) denotes incoming radiance, while fr describes reflection toward ωo.
3. Method
The method decomposes rendering into geometry, visibility, direct and indirect illumination, and SVBRDF components, using neural fields and spherical Gaussians for efficient optimization. It derives indirect illumination from a learned outgoing radiance field and represents it with an MLP, while visibility and material priors improve rendering and material recovery.
- Overview: The system represents geometry with an SDF and queries surface normals, visibility, indirect illumination, diffuse albedo, and roughness during differentiable rendering.The SDF surface is intersected using sphere tracing, and its gradient provides the surface normal.
- Visibility for Direct Illumination: Direct environment lighting is modeled with M=128 spherical Gaussians, whose visibility is approximated by an MLP mapping surface location and direction to a visibility value.The approximation preserves each lobe’s center and integrated value; visibility ratios use S=32 sampled directions.
- Indirect Illumination: The method derives indirect illumination from a learned outgoing radiance field instead of exhaustive recursive ray tracing, then caches it as a spherical-Gaussian mixture in an MLP.The outgoing field captures cumulative multiple-bounce radiance, while the indirect illumination MLP supports hemispherical integration without Monte Carlo sampling.
- Indirect Illumination: Indirect incoming radiance is represented as G(ωi; I(x)), where I(x) outputs spherical-Gaussian parameters at any 3D location.The representation allows indirect illumination to be queried during optimization without repeatedly tracing from one surface intersection to another.
- BRDF: The SVBRDF network maps surface points to sparse latent codes and decodes them into diffuse albedo and roughness, with KL sparsity and decoder smoothness constraints.The sparsity target is ρ=0.05, and decoder smoothness compares outputs under a small Gaussian perturbation with variance 0.01.
- BRDF: The renderer approximates diffuse and specular direct illumination using spherical-Gaussian inner products, while optimizing SVBRDF and environmental light against observed-image reconstruction error.The BRDF uses diffuse albedo and roughness and assumes dielectric materials with fixed F0=0.02.
4. Experiments
Experiments compare the method with baselines on synthetic data, evaluate component ablations, and assess real-capture inverse rendering and relighting. The full model performs best for SVBRDF recovery and relighting, while visibility sampling slightly reduces view-synthesis quality.
- Experimental setup: The evaluation uses four synthetic CAD scenes with natural environment maps, training renders, test images, albedo and roughness maps, and separate relighting evaluations.The experiments also include four real objects captured with a mobile phone while moving around the upper hemisphere.
- Baseline comparisons: Quantitative comparisons use PSNR, SSIM, and LPIPS against NeRFactor and an adapted PhySG baseline under the same fixed-unknown-illumination setting.PhySG is modified to output spatially varying roughness with an MLP rather than a global variable.
- Baseline comparisons: The method recovers finer environment lighting and SVBRDF details than NeRFactor, whose optimization can bake albedo into the environment map and impair relighting.Figure 4 compares normals, diffuse albedo, roughness, environment maps, and renderings under novel views and lighting.
- Ablation studies: Without visibility and indirect illumination, PhySG performs badly because geometry optimization becomes highly ill-posed in regions with shadows and interreflection.The ablation without both components performs worst for inverse rendering and relighting.
- Real captures and overall results: The full model achieves the best SVBRDF recovery and relighting, but slightly worse view synthesis than baselines, likely because visibility sampling introduces rendering noise.On real captures, the approach infers plausible SVBRDF and supports realistic relighting despite nonideal indoor illumination and moving-person shadows.
- Ablation studies: Removing indirect illumination bakes indirect-lighting effects into albedo, while removing the latent space produces noisy roughness from independently optimized surface points.The ablations evaluate combinations of three components affecting inverse-rendering quality.
5. Conclusion
The paper presents an efficient strategy for modeling indirect illumination by deriving it from a neural outgoing radiance field rather than simulating recursive light transport. This supports high-quality material recovery and realistic relighting, while relying on accurately reconstructed geometry and dielectric-material assumptions.
- 5. Conclusion: Indirect illumination is derived from the neural outgoing radiance field to avoid expensive recursive path tracing.The approach models indirect illumination efficiently within inverse rendering.
- 5. Conclusion: Real captures demonstrate relighting of objects with multiple materials under four novel real-world illuminations.The decomposed outputs include normal, albedo, and roughness alongside estimated shape and SVBRDF.
- 5. Conclusion: The full pipeline estimates high-quality albedo and roughness and supports realistic relighting from images captured under natural illumination.These results combine indirect-illumination modeling with BRDF priors and SG-based visibility estimation.
- 5. Conclusion: The pipeline strongly relies on fine geometry and cannot handle cases where the geometry reconstruction fails.The rendering model could be migrated to other surface-based geometric representations.
- 5. Conclusion: The BRDF uses fixed F0 = 0.02, assuming recovered materials are dielectric.Making F0 learnable would exacerbate inverse-problem ambiguity; learning-based priors or extra observations are suggested as future directions.