Source-linked AI summary
Learning to Predict Indoor Illumination from a Single Image
Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, Jean-François Lalonde
TL;DR
The paper addresses the ill-posed task of recovering indoor HDR illumination from a single limited-field-of-view LDR image. It trains a deep network in stages using annotated and warped LDR panoramas, then HDR fine-tuning, and reports significantly better quantitative and perceptual performance than prior methods.
Problem
Single-image indoor illumination is ill-posed because observed pixels mix geometry, materials, illumination, camera properties, and post-processing, while limited views omit relevant light sources.
Method
The method learns light locations from detected and warped LDR panoramas, then fine-tunes the network on HDR environment maps to predict illumination intensities.
Results
Quantitative and perceptual evaluations show that the method significantly outperforms prior state-of-the-art illumination-estimation methods.
Takeaways & Limitations
The estimates support photo-realistic applications such as 3D object insertion, with a perceptual study finding inserted objects almost indistinguishable from those lit by ground-truth illumination.
Takeaways & Limitations
Absolute illumination intensity remains ill-posed from uncalibrated LDR images, although relative illumination ratios can be recovered accurately.
Abstract
from arXiv · showhide
We propose an automatic method to infer high dynamic range illumination from a single, limited field-of-view, low dynamic range photograph of an indoor scene. In contrast to previous work that relies on specialized image capture, user input, and/or simple scene models, we train an end-to-end deep neural network that directly regresses a limited field-of-view photo to HDR illumination, without strong assumptions on scene geometry, material properties, or lighting. We show that this can be accomplished in a three step process: 1) we train a robust lighting classifier to automatically annotate the location of light sources in a large dataset of LDR environment maps, 2) we use these annotations to train a deep neural network that predicts the location of lights in a scene from a single limited field-of-view photo, and 3) we fine-tune this network using a small dataset of HDR environment maps to predict light intensities. This allows us to automatically recover high-quality HDR illumination estimates that significantly outperform previous state-of-the-art methods. Consequently, using our illumination estimates for applications like 3D object insertion, we can achieve results that are photo-realistic, which is validated via a perceptual user study.
1 INTRODUCTION
The paper frames single-image indoor illumination recovery as an ill-posed problem and proposes end-to-end learning to predict HDR illumination without strong scene assumptions. It combines automatically generated LDR training data, HDR fine-tuning, and perceptual evaluation.
- Single limited-field-of-view images confound geometry, materials, illumination, camera properties, and post-processing, making illumination recovery ill-posed.
- The proposed method predicts HDR illumination from a single LDR indoor photograph using an end-to-end deep neural network.
- The method significantly outperforms prior state-of-the-art approaches, with photo-realistic object insertion supported by a perceptual user study.
- A robust light-source detector and panorama-warping method generate training data from a large LDR panorama database.
- A new HDR environment-map dataset enables fine-tuning for illumination intensities after learning light locations from LDR data.
2 RELATED WORK
Prior illumination methods use specialized capture, user input, image matching, or low-dimensional lighting models. This paper instead learns a direct mapping from image appearance to indoor illumination using non-parametric HDR representations.
- Earlier methods recover illumination from mirrored or diffuse spheres, RGBD scans, known geometry, or user-annotated scene properties.
- Outdoor illumination methods can exploit analytical low-dimensional sun-sky models, whereas indoor lighting has more varied localized sources.
- Image-matching approaches use similar panoramas and pre-classified lights, but matched panoramas may have lighting arbitrarily different from the input scene.
- The paper directly learns the mapping between image appearance and scene illumination rather than relying on image matching or inverse-rendering assumptions.
- Its indoor-scene setting includes substantially more complex shape and reflectance variation than object-focused reflectance-map methods.
3 METHOD OVERVIEW
Because LDR panoramas do not directly encode indoor lighting and their projection centers may not match the cropped scene, the method separates light-location learning from intensity learning. It uses detected and warped LDR panoramas for pretraining, then fine-tunes on HDR environment maps.
- The method extracts rectified crops from existing panoramas at varied orientations and focal lengths to learn relationships between photographs and panoramas.
- LDR panoramas cannot directly provide indoor lighting targets because their dynamic range is limited and indoor sources are spatially localized.
- Light-source detection and panorama warping adapt LDR panoramas to train prediction of light locations from photographs.
- A new dataset of 2,100 HDR environment maps is used to fine-tune the pretrained network for light-intensity prediction.
4 LDR PANORAMA LIGHT SOURCE DETECTION
The method detects indoor light sources in LDR panoramas using specialized classifiers, post-processing, and a comparison against prior and intensity-based detectors.
- Light classification: The proposed detector identifies bright regions in LDR panoramas to provide annotations for CNN training.It is designed for light-source detection before single-photo illumination learning.
- Training details and evaluation: The pipeline combines multiscale panorama features with thresholding, dense-CRF refinement, and morphological opening and closing.Features include HOG, elevation, and intensity statistics computed after compensating for equirectangular distortion.
- Light classification: Separate classifiers for spotlights and lamps versus windows and reflections improve detection because these source types differ substantially.The classifiers are applied independently before their scores are merged.
- Light classification: The classifier is trained on 400 manually annotated SUN360 panoramas covering spotlights, lamps, windows, and reflections.The split uses 80% for training and 20% for testing, with hard-negative mining.
- Training details and evaluation: Our detector performs significantly better than Karsch et al. [2014] at every recall level, while a pixel-intensity baseline performs poorly on LDR data.The comparison uses precision-recall curves on the test set.
- Training details and evaluation: Example SUN360 results show detection across varied lighting arrangements, including large light patches and spotlights.Detected regions are overlaid in cyan on the input panoramas.
5 PANORAMA RECENTERING WARP
The panorama recentering warp addresses the mismatch between a panorama’s viewpoint and the cropped photo by approximating lighting from a virtual camera near the insertion location.
- 5.1 Warping operator: Using the original panorama directly can produce incorrect relighting because its center of projection may be far from the cropped scene.In the illustrated example, the unwarped panorama makes a virtual bunny appear backlit.
- 5.1 Warping operator: The operator assumes scene points lie on a sphere and that image warping sufficiently models camera movement without important occlusions.These assumptions are considered more suitable for light sources near ceilings and walls than for all scene points.
- 5.1 Warping operator: The warp models a virtual camera whose nadir is displaced by angle β, with arbitrary directions handled by rotating the environment map.For z-axis translation, the virtual camera center is set to {0, 0, sin β}.
- 5.1 Warping operator: β is selected from surface normals by locating the lowest point of the largest horizontal surface and back-projecting it onto the panorama.The resulting point defines the desired nadir from which β is recovered.
- 5.2 Impact on lighting estimation: Relighting with the proposed warped panoramas closely approximates ground-truth and EnvyDepth results without EnvyDepth’s lengthy manual annotation.The geometric-warp labeling process takes around 10 minutes per panorama.
- 5.2 Impact on lighting estimation: The warp does not model occluded lights that illuminate scene points but are invisible from the original camera.The authors report these cases as rare and state that the network filters them as outliers.
6 LEARNING FROM LDR PANORAMAS
The learning pipeline generates recentered photo–panorama pairs from SUN360 and trains a multi-head CNN to predict both light masks and RGB illumination.
- Data generation: Each SUN360 panorama yields eight random-elevation rectilinear crops, paired with warped panoramas and light masks for training.The warped data are rotated so the crop region is centered in azimuth.
- Network architecture: The CNN encodes the photo through convolutional and residual layers, then splits into light-mask and RGB-panorama decoder heads.The input is 256 × 192, while panorama and mask outputs are 256 × 128.
- Loss function: The RGB panorama loss uses a solid-angle-weighted L2 distance between predicted and ground-truth pixels.N equals width × height × 3, and s_i is the solid angle for pixel i.
- Loss function: The light-mask target is progressively blurred before L2 loss so small localization errors are penalized less severely.The differentiable cosine filter moves from coarse low frequencies toward sharper masks during training.
- Loss function: The global loss combines RGB reconstruction and cosine-filtered mask losses as L(y, t,e) = w1LL2(yRGB, tRGB) + w2Lcos(ymask, tmask,e).The experiments use w1 = 100, w2 = 1, and α = 3.
- Loss function: The cosine-filtered loss has a rendering interpretation: it evaluates illumination through appearances ranging from Lambertian to increasingly glossy spheres.Lambertian reflection suppresses high frequencies, whereas specular reflection preserves more of them.
7 LEARNING HIGH DYNAMIC RANGE ILLUMINATION
The network is fine-tuned on a new HDR indoor-panorama dataset to predict light-source intensities after learning their positions from LDR data. Its modified loss and HDR training setup support joint illumination estimation.
- HDR dataset: 2,100 high-resolution HDR indoor panoramas were captured across diverse environments for training and testing illumination estimation.Each panorama has resolution 7768 × 3884 and covers a 22 f-stop HDR 360° view.
- HDR adaptation: HDR fine-tuning changes the target from a binary light mask to log-scaled light-source intensities while retaining the pretrained network representation.Only decoder weights are updated, and intensities below the training-set median are clamped to zero.
- HDR adaptation: The modified loss combines the existing light-mask term with cosine and L2 losses on predicted and target intensities.The intensity L2 term reduces deconvolution artifacts; the weights are w1 = 10, w2 = 1, and w3 = 0.1.
- Training setup: The HDR dataset was split into 85% training and 15% testing data, with eight crops per panorama yielding 14,000 input-output pairs.The crops were tone-mapped for network input.
- Evaluation: Relit examples indicate that the LDR network generalizes to a wide range of indoor scenes and illumination conditions.The figure visualizes both virtual-object relighting and predicted lighting probabilities over panoramas.
8 EXPERIMENTS
The experiments evaluate light localization, HDR intensity prediction, relighting quality, robustness, and comparisons with prior methods through quantitative, qualitative, and perceptual tests.
- LDR network evaluation: The LDR network localizes light sources from small-field-of-view images despite few obvious illumination cues.Visualization combines a thresholded light mask with the RGB panorama using λmask = 500 and λRGB = 1.
- LDR network evaluation: Occluded-light cases are rare, yet the network generalizes beyond erroneous warped training data and predicts lights consistent with observed shading and shadows.In one example, its prediction is more consistent with the cropped image than the ground-truth annotation.
- HDR network evaluation: On the HDR test set, the network finds dominant lights, including those outside the input view, but struggles with small concentrated sources and very large area lights.The test set contains 2,100 images, and an example with L = 0.02 corresponds to about 4.7% global relative error under the reported L2 loss.
- Virtual object relighting: HDR predictions can be combined into environment maps for relighting, producing convincing results close to ground-truth lighting in virtual-object insertions.The HDR network outputs log-intensity, and the RGB prediction and light-source color are postprocessed using the input image’s mean RGB value.
- Global intensity scaling: Absolute illumination remains ambiguous in uncalibrated LDR images, so relighting may require a single global intensity scale that can be manually tuned.The network is reported to recover correct relative illumination even when absolute exposure is too dark or too bright.
- User study: In a study of 105 participants and 1,080 comparisons per method, HDR-network renderings were judged at least as realistic as ground truth in 41.85% of responses.This exceeded Khan et al. [2006] at 27.78% and Karsch et al. [2014] at 16.76%; the LDR network achieved 27.32%.
9 DISCUSSION
The method frames indoor illumination estimation as end-to-end learning, supported by automatically generated training data and HDR fine-tuning. It outperforms prior methods but remains limited by light-shape errors, ambiguous scenes, exposure sensitivity, and simplified color or spatial modeling.
- 9 DISCUSSION: The direct image-to-lighting mapping avoids explicit scene-property estimation and is enabled by light detection and panorama warping for automatic labeled-data creation.The network is trained on cropped panorama data and then fine-tuned with HDR environment maps.
- 9 DISCUSSION: Quantitative and perceptual evaluations show the method is significantly better than the state of the art.
- 9 DISCUSSION: Typical failures involve incorrect light-source extent or orientation, blurred sharp sources, complex geometry, and strong local illumination variations.The lower half of one example is illuminated while the upper half is not.
- 9 DISCUSSION: The network generally recovers light locations better than intensities because location pre-training uses more data, while intensity prediction is more sensitive to input exposure.
- 9 DISCUSSION: The method predicts intensity with a simple color-assignment scheme and one lighting solution, leaving robust color and spatially varying illumination as future goals.These limitations matter because indoor illumination can vary within an image.
- 9 DISCUSSION: Joint inference of lighting, geometry, and intrinsic images could benefit each individual scene-inference task.