Source-linked AI summary
Deep Parametric Indoor Lighting Estimation
Marc-André Gardner, Yannick Hold-Geoffroy, Kalyan Sunkavalli, Christian Gagné, Jean-François Lalonde
TL;DR
Single-image indoor lighting estimation is difficult because global environment maps do not capture localized illumination. The paper learns discrete 3D light parameters from one image and trains them through a differentiable environment-map projection, achieving more accurate lighting estimates and realistic spatially varying object insertion.
Problem
Global environment maps assume distant lighting and therefore inadequately represent localized, spatially varying illumination in indoor scenes.
Method
A deep network predicts discrete 3D area-light positions, areas, intensities, and colors from a single image, using depth-annotated HDR data and a differentiable environment-map loss.
Results
The method significantly outperforms previous work in lighting accuracy and enables realistic virtual object insertion with lighting that varies by insertion point.
Takeaways & Limitations
Parametric 3D lights provide a representation that can render incident illumination at different scene locations for indoor relighting and object compositing.
Takeaways & Limitations
Using the 3D lighting representation requires scene reconstruction for location specification and visibility, and the model assumes diffuse area-light sources rather than directional or focused lights.
Abstract
from arXiv · showhide
We present a method to estimate lighting from a single image of an indoor scene. Previous work has used an environment map representation that does not account for the localized nature of indoor lighting. Instead, we represent lighting as a set of discrete 3D lights with geometric and photometric parameters. We train a deep neural network to regress these parameters from a single image, on a dataset of environment maps annotated with depth. We propose a differentiable layer to convert these parameters to an environment map to compute our loss; this bypasses the challenge of establishing correspondences between estimated and ground truth lights. We demonstrate, via quantitative and qualitative evaluations, that our representation and training scheme lead to more accurate results compared to previous work, while allowing for more realistic 3D object compositing with spatially-varying lighting.
1. Introduction
Single-image indoor lighting recovery is under-constrained, while global environment maps fail to represent localized, spatially varying illumination. The paper addresses this with learned parametric 3D lights and a differentiable environment-map loss.
- Single-image lighting recovery is ill-posed because lighting is conflated with reflectance, geometry, and camera response, especially in LDR images with limited field of view.
- Environment maps assume distant lighting, so a single map can produce inconsistent shadows and shading when indoor sources are localized.
- Indoor parametric lighting is difficult because scenes contain varying numbers of nearby light sources with different geometric and photometric properties.
- The method predicts 3D area-light positions, areas, intensities, and colors from one 2D LDR image for relighting objects at arbitrary scene locations.
- Training uses indoor HDR environment maps with manually added depth and parametric-light annotations, decoding a fixed number of light sources from image features.
- A differentiable layer converts predicted light parameters into an environment map, allowing direct loss computation without fragile light correspondences.
2. Related Work
Prior indoor-lighting methods rely on geometry, user input, heuristics, or environment-map representations, whereas this work learns discrete parametric 3D lights end to end. The comparison emphasizes the challenge of modeling spatially varying illumination for relighting at different scene locations.
- Object-based illumination methods do not easily scale to large indoor scenes, while outdoor methods use simpler low-dimensional analytical lighting models.
- Large-scale indoor lighting methods often assume known geometry, require multiview reconstruction, or depend on substantial user input for single-image estimation.
- Karsch et al. estimate indoor parametric lighting through geometry and reflectance recovery, light detection, and rendering optimization, with errors potentially propagating across heuristic stages.
- Gardner et al. regress a full environment map from one image, whereas this work predicts discrete parametric 3D lights with position, area, color, and intensity.
- The parametric representation can be projected at different scene locations to account for spatial lighting variation during relighting.
3. Method
The method predicts a fixed set of 3D area-light parameters from a single indoor image and trains them through differentiable projection into an environment map. It uses depth-annotated real HDR panoramas and a global environment-map loss to avoid unstable light correspondences.
- Parametric lighting representation: The model represents illumination with N lights and an ambient RGB term, predicting each light’s direction, distance, angular size, and color.A fixed N can represent scenes with fewer lights by setting a light’s color magnitude to zero.
- Dataset and annotations: The training dataset contains 2,100 real indoor HDR panoramas with manually annotated per-pixel depth and extracted parametric light sources.Light sources are initialized from bright panorama regions, parameterized using depth and image statistics, and intensity-tuned with rendering-based optimization.
- Network architecture: A DenseNet-121 encoder maps the input image to a latent vector, which separate decoders use to predict the light parameters and ambient term.The parametric decoders require fewer parameters and make the network faster than full environment-map decoders.
- Training challenge: Directly matching predicted and annotated lights is unstable because assignments depend on changing predicted positions and can produce ambiguous correspondences.The paper reports that this assignment-based training leads to substantially worse RMSE and si-RMSE than the proposed two-step procedure.
- Differentiable training: The first training step converts predicted lights into an environment map and compares it with the ground-truth map using a weighted L2 loss.The differentiable projection represents each light as a spherical Gaussian, enabling gradients from environment reconstruction to reach the predicted 3D parameters.
- Benefits: Environment-map optimization avoids light correspondences, supports global convergence, handles mismatched light counts, and permits any preset number of predicted lights.The representation also provides individual 3D lights for spatially varying relighting and runs at roughly 51 ms per image on a CPU.
4. Evaluation
The evaluation combines quantitative render comparisons, user judgments, qualitative relighting examples, and tests on out-of-dataset images. Across these evaluations, the method improves lighting accuracy and produces plausible, location-aware object relighting.
- Quantitative evaluation: The evaluation compares rendered diffuse objects against ground truth at center and ±1 meter insertion positions using RMSE and scale-invariant RMSE.The spiky sphere captures both shading and shadowing, while EnvyDepth geometry warps the ground-truth environment map.
- Quantitative evaluation: Our method significantly outperforms Gardner et al. in RMSE, with performance improving as the estimated light count increases from 2 to 3 and 5 lights.The authors attribute the improvement to additional degrees of freedom for fitting the overall energy distribution.
- Quantitative evaluation: A 3-light configuration achieves a 40% increase in median si-RMSE performance over Gardner et al., while 5 lights slightly reduce performance because their positions are harder to optimize.The si-RMSE measure factors out scale differences and emphasizes shading and shadows driven mainly by light position.
- Quantitative evaluation: A 49-participant user study across 19 scenes confirms improved realism over Gardner et al. and Karsch et al. in composited-object comparisons.Participants compared images rendered with ground-truth illumination against images using randomly selected method predictions.
- Qualitative evaluation: The method generalizes to smartphone images, outdoor daylight, generic stock photos, and artistic processing while producing realistic relighting results.Rendered and real bunny comparisons also demonstrate recovery of light positions and intensity, although the two bunnies differ slightly in appearance.
- Qualitative evaluation: Qualitative comparisons show that the compact parametric representation produces renders similar to or better matching ground truth than Gardner et al., including diffuse and glossy objects.The comparison estimates lighting at the scene center to highlight differences between parametric and non-parametric representations.
5. Conclusion
The paper presents a parametric 3D lighting model that estimates discrete lights from a single indoor image and renders illumination at arbitrary scene locations. It reports improved lighting accuracy and realistic insertion with location-dependent lighting, while identifying reconstruction, source-model, and scene-property boundaries.
- Conclusion: The method estimates discrete 3D lights with positions, areas, colors, and intensities from a single indoor image.Its representation can render incident illumination at any scene location, addressing localized indoor lighting.
- Conclusion: The method uses an end-to-end differentiable parametric loss based on an environment-map representation.This training design supports estimating the parametric lighting model without directly matching individual light sources.
- Conclusion: The authors report robustness, significantly better lighting accuracy than previous work, and realistic virtual object insertion with lighting that varies by insertion point.The conclusion ties these outcomes to the method’s ability to represent localized illumination.
- Conclusion: Using the representation requires 3D scene reconstruction for specifying lighting locations and computing light visibility.This is identified as a direction for future exploration rather than a reported failure of the evaluation.
- Conclusion: The model assumes diffuse area-like sources and cannot model directional lights or focused beams, while estimating illumination alone omits geometry and materials.The authors suggest more general sources and joint reasoning about scene appearance as future extensions.