Source-linked AI summary
Neural Inverse Rendering of an Indoor Scene from a Single Image
Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W. Jacobs, Jan Kautz
TL;DR
Single-image indoor inverse rendering remains difficult because prior work often handles single objects or individual scene attributes, while real-image labels for reflectance and lighting are scarce. The paper jointly estimates albedo, normals, and lighting using synthetic pretraining, self-supervised real-image reconstruction, and RAR, and reports better performance than prior single-attribute and related methods. Its RAR is intended to support reconstruction-based learning rather than realistic illumination for object insertion.
Problem
Single-image indoor inverse rendering is challenging because the task is ill-posed, scene appearance is complex, and large-scale real-image labels for reflectance and lighting are difficult to obtain.
Method
The method jointly predicts albedo, surface normals, and environment-map lighting, using synthetic pretraining and self-supervised reconstruction on real images with a Direct Renderer and trainable RAR.
Results
The approach outperforms prior methods estimating one or more scene attributes and generalizes better across datasets.
Takeaways & Limitations
RAR enables reconstruction-based self-supervision by synthesizing complex appearance effects, supporting joint inverse rendering of indoor scenes from single images.
Takeaways & Limitations
RAR is designed to reconstruct images for self-supervised learning and is not intended to provide realistic illumination for object insertion.
Abstract
from arXiv · showhide
Inverse rendering aims to estimate physical attributes of a scene, e.g., reflectance, geometry, and lighting, from image(s). Inverse rendering has been studied primarily for single objects or with methods that solve for only one of the scene attributes. We propose the first learning-based approach that jointly estimates albedo, normals, and lighting of an indoor scene from a single image. Our key contribution is the Residual Appearance Renderer (RAR), which can be trained to synthesize complex appearance effects (e.g., inter-reflection, cast shadows, near-field illumination, and realistic shading), which would be neglected otherwise. This enables us to perform self-supervised learning on real data using a reconstruction loss, based on re-synthesizing the input image from the estimated components. We finetune with real data after pretraining with synthetic data. To this end, we use physically-based rendering to create a large-scale synthetic dataset, which is a significant improvement over prior datasets. Experimental results show that our approach outperforms state-of-the-art methods that estimate one or more scene attributes.
1. Introduction
The paper addresses single-image indoor inverse rendering by jointly estimating albedo, surface normals, and lighting despite ill-posedness, complex appearance effects, and scarce real-image labels. It introduces self-supervised reconstruction with RAR and reports stronger performance and cross-dataset generalization than single-attribute methods.
- Motivation: Single-image indoor inverse rendering is ill-posed and previous single-image methods largely focus on individual objects or attributes.Scene appearance is further complicated by inter-reflection, cast shadows, and near-field illumination.
- Approach: The method jointly decomposes an indoor image into albedo, surface normals, and environment-map lighting.This targets complete scene-level inverse rendering from one image rather than one isolated scene attribute.
- Approach: Residual Appearance Renderer (RAR) learns complex appearance effects, enabling self-supervised photometric reconstruction on unlabeled real images.The input image is re-synthesized from estimated components, while a physically based synthetic dataset supports pretraining.
- Contribution: The work presents a data-driven solution for single-image inverse rendering of an indoor scene.The authors position it as the first such learning-based approach to their knowledge.
- Results: The approach outperforms single-attribute methods and generalizes better across datasets in quantitative and qualitative evaluations.Comparisons include methods estimating albedo, lighting, or normals separately, as well as SIRFS.
2. Related Work
Prior work studies inverse rendering through optimization, specialized learning-based estimators, differentiable renderers, and synthetic or intrinsic-image datasets. The paper differs by combining complete scene decomposition with a renderer designed to model complex appearance effects on real images.
- Optimization-based approaches: Optimization-based inverse rendering often relies on strong statistical priors for illumination or reflectance and includes subproblems such as intrinsic decomposition and BRDF estimation.SIRFS factorizes an image into surface normals, albedo, and spherical-harmonics lighting.
- Learning-based approaches: Learning-based methods estimate selected properties such as depth, normals, BRDFs, lighting, or intrinsic components from images.These approaches generally address individual inverse-rendering problems rather than the paper’s complete indoor-scene decomposition.
- Differentiable Renderer: The proposed RAR shares the differentiable-rendering spirit but targets complex appearance synthesis for inverse rendering on real images.Its purpose differs from differentiable Monte Carlo and neural mesh renderers that optimize rendering parameters or visibility and rasterization.
- Datasets for inverse rendering: Existing indoor datasets trade off scale, photorealism, noise, or lighting diversity, motivating improved physically based synthetic data.SUNCG uses diffuse materials and point lighting, while PBRS is more photorealistic but noisy and limited to one lighting condition.
- Intrinsic image decomposition: Intrinsic image decomposition combines geometry and illumination into shading, whereas complete inverse rendering explicitly recovers geometry and illumination.The broader decomposition supports applications such as AR/VR.
3. Our Approach
The approach jointly estimates albedo, normals, and lighting, then reconstructs images with a direct renderer plus RAR to support self-supervised learning on real images. It transfers from synthetic to real data through supervised pretraining, learned lighting targets, reconstruction, and sparse-label supervision.
- IRN estimates surface normals, albedo, and environment-map lighting from a single input image.
- RAR: Self-supervised Training on Real Images: The self-supervised reconstruction loss compares the input image with the sum of direct-rendered and RAR-predicted images.
- Synthetic pretraining uses supervised losses for albedo, normals, and lighting, while real-image adaptation uses reconstruction and sparse weak labels.
- RAR: Self-supervised Training on Real Images: RAR learns residual appearance effects that direct rendering cannot model, including inter-reflections, cast shadows, and near-field lighting.
- RAR: Self-supervised Training on Real Images: RAR is trained on synthetic images with ground-truth normals and albedo, then fixed during real-image training to model residual appearance rather than IRN estimation errors.
- The CG-PBR dataset contains 235,893 images with diffuse and specular reflectance plus ground-truth scene attributes for training.
- RAR: Self-supervised Training on Real Images: RAR improves albedo and normal estimation, but its stated purpose is image reconstruction for self-supervision rather than realistic illumination during object insertion.
4. The CG-PBR Dataset
CG-PBR is introduced as a large-scale physically based indoor-scene dataset designed to improve realism, noise levels, and lighting diversity over prior datasets. It includes paired diffuse and specular renderings and denoised ray-traced images.
- CG-PBR renders scenes under multiple outdoor environment maps instead of a single lighting condition.
- The dataset provides paired images with Lambertian and default materials, supporting learning involving diffuse and specular appearance.
- Deep denoising enables high-quality ray-traced images from limited samples per pixel.
- Compared with SUNCG and PBRS, CG-PBR provides more photorealistic, less noisy images with specular highlights under multiple lighting conditions.
5. Experimental Results
The method is evaluated against prior inverse-rendering and intrinsic-decomposition approaches for albedo, normals, and lighting. It generally outperforms these methods and generalizes better across datasets, with qualitative advantages in reflectance–shading separation, texture preservation, and environment estimation.
- The method is compared with SIRFS for joint single-image estimation of surface normals, albedo, and spherical-harmonics lighting.
- It outperforms intrinsic image decomposition methods on IIW albedo evaluation without using additional intrinsic-specific datasets or post-processing.
- The method outperforms Li et al. on both CG-PBR and CGI datasets, preserving more texture and producing fewer albedo artifacts.
- It significantly outperforms PBRS on Scannet and CG-PBR, slightly improves on 7-Scenes, and performs slightly worse on NYUv2, indicating better cross-dataset generalization.
- On real indoor scenes, the estimated environment map significantly outperforms Gardner et al. when evaluated by rendering a diffuse ball against ground-truth imagery.
6. Ablation Study
The ablations show that RAR and weak supervision improve the quality and consistency of estimated scene attributes. RAR enables reconstruction losses to model complex appearance effects, while weak supervision improves albedo and normal estimation.
- Role of RAR: RAR improves albedo and normal estimates when IRN is trained on real data with weak supervision.
- Role of RAR: Without RAR, reconstruction loss cannot capture complex appearance effects and produces worse estimates of scene attributes.
- Role of weak supervision: Weak supervision significantly reduces median angular error on NYUv2 and WHDR on IIW.
- Role of weak supervision: Weak supervision makes albedo predictions more consistent across large objects such as walls, floors, and ceilings.
7. Conclusion
The paper presents a learning-based approach for single-image indoor inverse rendering that jointly estimates albedo, normals, and lighting. Its RAR supports self-supervised learning on real images, while CG-PBR provides a large-scale physically based synthetic dataset.
- The approach jointly estimates albedo, surface normals, and lighting from a single indoor RGB image.
- RAR synthesizes complex appearance effects and enables self-supervised reconstruction-based learning on real images.
- CG-PBR is a large-scale, high-quality synthetic dataset created with physically based rendering for the research community.
8. Appendix
The appendix provides architectural, loss-function, and additional qualitative-evaluation details for reproducibility and further assessment.
- The appendix details the IRN and RAR architectures, real-data training losses, and additional qualitative evaluations.
8.1. Network Architectures
IRN jointly predicts albedo and surface normals at image resolution and an environment-map lighting representation. RAR uses normals and albedo to reconstruct appearance effects through a U-Net-based renderer.
- IRN takes a 240×320 image and outputs same-resolution albedo and normal maps plus an 18×36 environment map.
- The network uses an encoder, separate residual-block streams for normals and albedo, decoders, and a lighting-estimation module.
- The lighting-estimation network predicts an environment map from image features and estimated or provided intrinsic components.
- RAR is a U-Net with normals and albedo as inputs, using convolutional encoding, bilinear-upsampling decoding, and skip connections.
8.2. Training Details
Training combines pseudo-supervision, photometric reconstruction with RAR, and reflectance or normal supervision. The real-data objectives use different weighted losses for IIW-style reflectance judgments and NYUv2 Kinect normals.
- For IIW-style training, the pair-wise reflectance loss weights predictions according to human confidence and penalizes incorrect darker, brighter, or similar relations.
- Real-data training combines pseudo-supervision for intrinsic components, RAR-based photometric reconstruction, and pair-wise or Kinect-normal supervision.
- The real-data objective weights are 0.5 for albedo, 0.5 for normals, 0.1 for lighting, 1 for reconstruction, and 30 for weak supervision.
- The NYUv2 objective uses albedo and lighting pseudo-supervision, RAR reconstruction, and Kinect-normal supervision with weights 0.2, 0.05, 1, and 20.
8.3. Our CG-PBR Dataset
The CG-PBR dataset uses physically based rendering to provide more photo-realistic and less noisy indoor-scene images with specular highlights than PBRS under the same illumination.
- CG-PBR provides more photo-realistic and less noisy images with specular highlights than PBRS under the same illumination condition.Both datasets are rendered with Mitsuba.
8.4. More Experimental Results
Additional experiments compare IRN with optimization-based and single-attribute methods, evaluate lighting through rendered probes, and analyze the roles of real-data training, RAR, and weak supervision. The results favor IRN while showing that its components address distinct reconstruction and generalization issues.
- Comparison with SIRFS: Compared with SIRFS, IRN produces more accurate surface normals and better separates reflectance from shading.
- Lighting estimation: Both IRN lighting variants outperform Gardner et al. by producing more realistic environment maps, while Ours+he(·) improves over Ours.
- Ablation study: Training only on synthetic data fails to generalize well to real images, while removing RAR or weak supervision produces distinct albedo artifacts.
- Ablation study: Without RAR, albedo becomes piece-wise smooth and low contrast; without weak supervision, consistency across large surfaces often fails.
- Ablation study: Without RAR and weak supervision, direct-renderer reconstruction places cast shadows, inter-reflections, and highlights into the albedo.
- CG-PBR dataset: The CG-PBR examples and comparisons show a dataset with specular highlights and reduced noise relative to PBRS.
- Comparison with Li et al.: Compared with Li et al., IRN performs better reflectance–shading disambiguation and preserves texture in the albedo on IIW and CG-PBR.