Source-linked AI summary
Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting
Zian Wang, Jonah Philion, Sanja Fidler, Jan Kautz
TL;DR
Indoor inverse rendering from one image is challenging because reflectance, geometry, and complex 3D lighting are ambiguous, while common 2D lighting representations omit spatial variation. The framework jointly estimates scene properties and 3D spatially varying lighting using voxel-based Volumetric Spherical Gaussians and a differentiable renderer with a re-rendering constraint. The method outperforms prior work and produces realistic virtual-object insertions, including highly specular objects.
Problem
Indoor inverse rendering from one image is challenging because reflectance, geometry, and complex 3D lighting are ambiguous, while common 2D lighting representations omit spatial variation.
Source: §1 · p. 1Method
The framework jointly estimates scene properties and 3D spatially varying lighting using voxel-based Volumetric Spherical Gaussians and a differentiable renderer with a re-rendering constraint.
Source: §1 · pp. 1–2, §3 · p. 3Results
Takeaways & Limitations
The predicted HDR lighting supports realistic cast shadows and angular high-frequency details in virtual-object insertion.
Source: §1 · pp. 1–2The cited depth and normal estimation methods use specially designed architectures and other data sources [15] [19].
Source: §5.2 · p. 7
Abstract
from arXiv · showhide
In this work, we address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image. Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene. However, indoor scenes contain complex 3D light transport where a 2D representation is insufficient. In this paper, we propose a unified, learning-based inverse rendering framework that formulates 3D spatially-varying lighting. Inspired by classic volume rendering techniques, we propose a novel Volumetric Spherical Gaussian representation for lighting, which parameterizes the exitant radiance of the 3D scene surfaces on a voxel grid. We design a physics based differentiable renderer that utilizes our 3D lighting representation, and formulates the energy-conserving image formation process that enables joint training of all intrinsic properties with the re-rendering constraint. Our model ensures physically correct predictions and avoids the need for ground-truth HDR lighting which is not easily accessible. Experiments show that our method outperforms prior works both quantitatively and qualitatively, and is capable of producing photorealistic results for AR applications such as virtual object insertion even for highly specular objects.
1. Introduction
Indoor inverse rendering is difficult because reflectance, geometry, and complex lighting are ambiguous, while indoor scenes involve complex 3D light transport. This work jointly estimates scene properties and 3D spatially varying HDR lighting, reporting improved benchmark performance and realistic virtual-object insertion.
1. Introduction
- Inverse rendering estimates reflectance, shape, and lighting from one image, supporting applications such as virtual-object insertion and semantic scene analysis [5].The task is challenging because observed pixels do not uniquely determine those intrinsic properties and their interactions with lighting.Source: §1 · p. 1
- Existing learning methods often use 2D image-to-image representations that omit spatially varying lighting, limiting depth and angular high-frequency effects.Indoor scenes are especially challenging because of complex 3D light transport.Source: §1 · p. 1
- The framework jointly estimates reflectance, shape, and 3D spatially varying lighting through an end-to-end trainable rendering process.Its Volumetric Spherical Gaussian representation and differentiable renderer support re-rendering-based joint training without ground-truth HDR lighting.Source: §1 · pp. 1–2
- The method outperforms prior inverse-rendering and lighting-estimation methods and produces more realistic virtual-object insertions, including highly specular objects.Its insertion results include realistic shadows and angular high-frequency details.Source: §1 · p. 2
2. Related Work
Earlier inverse-rendering methods used hand-crafted priors or learned 2D mappings, while lighting-estimation approaches vary in their handling of spatial variation, HDR, and high-frequency detail. The paper situates its voxel-based lighting representation among neural scene representations.
2. Related Work
- Classic inverse-rendering methods rely on hand-crafted priors, while SIRFS jointly estimates intrinsic properties; assumptions and test-time optimization can limit real-image performance and offline use.Earlier work also addressed subtasks such as intrinsic-image decomposition and shape from shading.Source: §2 · p. 2
- Related learning-based methods use varied annotations and synthetic data, while NIR and Li et al. model general indoor scenes with 2D image-to-image approaches.NIR uses environment maps and a non-interpretable neural renderer; Li et al. predict per-pixel spherical-Gaussian lighting but lack a depth degree of freedom and sacrifice angular frequency [33] [22].Source: §2 · p. 2
- Indoor lighting methods use environment maps, per-pixel spherical lobes, or light-source parameters, but may omit spatial variation or high-frequency detail.Neural Illumination predicts environment maps at queried 3D locations but suffers spatial instability; Lighthouse does not address HDR or guarantee physically correct inpainted lighting.Source: §2 · p. 2
- Neural scene representations include voxels, point clouds, and implicit functions; the proposed Volumetric Spherical Gaussian augments voxel RGBα features with view-dependent effects for directional lighting.Prior voxel and multi-plane methods can predict 3D properties with 2D supervision [27] [35] [46].Source: §2 · p. 2
3. Lighting Representation
The lighting representation models exitant radiance throughout the scene so that illumination can be rendered at different spatial locations and viewing angles. Its voxel-wise spherical Gaussians add directional and sharpness controls beyond RGBα volumes.
3. Lighting Representation
- Volumetric Spherical Gaussian represents scene-surface exitant radiance, including surfaces outside the field of view, enabling volume-rendered illumination at arbitrary locations and viewing angles.This addresses the difficulty of directly extending visible-surface incident-radiance representations to 3D [11] [22].Source: §3 · p. 3
- Each voxel stores opacity and spherical-Gaussian parameters that define radiance as a function of viewing direction.The voxel grid represents the scene’s spatially varying lighting.Source: §3 · p. 3
- Per voxel, c encodes HDR RGB intensity, µ the lobe axis, and 1/σ² sharpness; the lighting volume is an 8-channel tensor.This parameterization treats each voxel as a light source.Source: §3 · p. 3
- The renderer samples points along a ray and alpha-composites their contributions to calculate incident radiance; replacing spherical Gaussians with voxel depth values also renders depth.The ray samples index into the lighting volume for their Gaussian and opacity values.Source: §3 · p. 3
- Unlike RGBα volumes, VSG controls emission direction and sharpness to capture view-dependent effects and strong directional lighting.For σ ≫ 1, VSG reduces to an RGBα representation.Source: §3 · p. 3
4. Method
The model jointly estimates albedo, normals, depth, and volumetric lighting from one image through prediction, differentiable re-rendering, and refinement. Training combines synthetic supervision with rendering-based consistency losses.
4. Method
- A single image drives initial intrinsic predictions, 3D lighting-volume prediction, raytraced re-rendering, and joint refinement.The refined outputs include albedo, normals, and depth.Source: §4 · p. 3, Figure 2 · p. 3
4.1. Model Design
- Global scene features and unprojected visible-field-of-view features are fused and processed by a 3D network to produce volumetric lighting.The global feature is decoded into a scene feature volume, while local features come from the visible image, albedo, normals, and depth.Source: §4.1 · pp. 3–4, Figure 3 · p. 4
- The differentiable renderer queries the lighting volume along sampled directions and uses Lambertian reflectance with energy-conserving HDR-to-LDR clipping.The renderer uses predicted depth, albedo, and normals to reconstruct the input image.Source: §4.1 · p. 4
- A fully convolutional re-prediction network uses the image, current intrinsic estimates, re-rendering error, shading, and shading’s normal Jacobian to refine albedo, normals, and depth.The shading signal indicates how albedo changes affect the re-rendered image, while its normal Jacobian indicates how normal changes affect shading.Source: §4.1 · p. 4
4.2. Training
- Synthetic training supplies ground-truth albedo, normals, depth, novel-view LDR panoramas, and camera poses, avoiding densely rendered spherical-lobe lighting labels.The volumetric representation also improves angular frequency relative to those prior lighting labels.Source: §4.2 · p. 4
- Training combines direct supervision, novel-view and visible-field-of-view consistency, adversarial loss, and re-rendering loss; staged training first pretrains intrinsic branches, then adds lighting and refinement.The re-rendering loss encourages physically correct lighting and HDR recovery even when supervision uses LDR images.Source: §4.2 · pp. 4–5
5. Experiments
Experiments evaluate intrinsic-property prediction, re-rendering, and lighting on synthetic and real-world data. The authors report improved results over baselines and realistic object insertion, including for highly specular objects.
5.2. Evaluation of Albedo and Shape
- On InteriorNet, the method outperforms NIR for albedo, normals, and depth; Joint Re-prediction and its lighting-related properties improve intrinsic-property performance.Figure 4 qualitatively shows crisper, more accurate predictions with Joint Re-prediction.Source: §5.2 · p. 6, Table 1 · p. 6, Figure 4 · p. 7
- On real-world IIW and NYUv2 data, the method outperforms baselines for albedo, normals, and depth, while not targeting specialized state-of-the-art depth or normal estimation.Tables 3 and 4 report the albedo and normals/depth evaluations, respectively.Source: §5.2 · pp. 6–7, Table 3 · p. 6, Table 4 · p. 6
- The method achieves lower re-rendering error than NIR with and without its residual appearance renderer while explicitly modeling 3D spatially-varying lighting.The authors report that NIR leaves complex lighting effects to its renderer, which may introduce artifacts affecting other properties.Source: Table 5 · p. 6, §5.2 · p. 7
- Ablations identify re-rendering loss as critical for physically correct predictions, while the spherical Gaussian volume improves re-rendering over RGBα volume.Training with 120 real-world LDR panoramas further improves performance on NYUv2.Source: Table 5 · p. 6, §5.2 · p. 7
5.3. Evaluation of Lighting
- On InteriorNet, lighting evaluation reports that the method outperforms NIR and Lighthouse, although Lighthouse uses stereo input while the proposed method uses monocular input.The lighting comparison is summarized in Table 2.Source: Table 2 · p. 6, §5.3 · p. 7
- On InteriorNet, qualitative comparisons show realistic HDR cast shadows and angular details for highly specular insertions, which competing methods fail to reproduce together.The authors attribute physically correct outputs to re-rendering loss and energy constraints.Source: §5.3 · p. 7, Figure 6 · p. 8
- On real-world images, the method produces spatially coherent lighting and realistic shading and shadows for specular and diffuse objects, including objects inserted at sampled 3D locations.The real-world examples include bunny, kettle, cart, and armchair insertions.Source: §5.3 · p. 8, Figure 7 · p. 8, Figure 8 · p. 8
6. Conclusion
The framework jointly estimates albedo, normals, depth, and HDR light field from a single image. It outperforms prior work on standard benchmarks and renders virtual objects with realistic shadows and high-frequency details, including highly specular objects.
6. Conclusion
- The method outperforms prior work on standard benchmarks and realistically renders virtual objects with shadows and high-frequency details, even for highly specular objects.The authors describe these results as promising for AR applications.Source: §6 · p. 8
- The framework jointly estimates albedo, normals, depth, and HDR light field in a monocular inverse-rendering setup.Source: §6 · p. 8
- Joint training with a re-rendering constraint supports physically correct HDR-lighting predictions despite training only on LDR images.The physically based differentiable renderer learns and reproduces complex indoor lighting effects.Source: §6 · p. 8