Source-linked AI summary
Deep Outdoor Illumination Estimation
Yannick Hold-Geoffroy, Kalyan Sunkavalli, Sunil Hadap, Emiliano Gambaretto, Jean-François Lalonde
TL;DR
The paper tackles full outdoor illumination recovery from a single image despite the lack of paired HDR outdoor training data. It fits a physically based sky model to large-scale panoramas, trains a CNN to predict its parameters, and reports improved lighting estimation with realistic virtual-object insertion. Limitations remain for overcast skies and specular materials because of model bias and sky-only environment maps.
Problem
Recovering outdoor illumination from one image is ill-posed, and large datasets pairing outdoor images with true HDR lighting conditions do not exist.
Method
The method fits a physically based sky model to panorama skies, then trains a CNN on panorama-derived images and fitted HDR lighting parameters.
Results
The method significantly outperforms previous work and produces lighting estimates that support realistic virtual-object insertion into photos.
Takeaways & Limitations
A single outdoor LDR image can yield a full HDR lighting model usable for image-based lighting and photorealistic rendering.
Takeaways & Limitations
Accuracy degrades for increasingly overcast skies, and the environment map models only the sky hemisphere, limiting realism for specular materials.
Abstract
from arXiv · showhide
We present a CNN-based technique to estimate high-dynamic range outdoor illumination from a single low dynamic range image. To train the CNN, we leverage a large dataset of outdoor panoramas. We fit a low-dimensional physically-based outdoor illumination model to the skies in these panoramas giving us a compact set of parameters (including sun position, atmospheric conditions, and camera parameters). We extract limited field-of-view images from the panoramas, and train a CNN with this large set of input image--output lighting parameter pairs. Given a test image, this network can be used to infer illumination parameters that can, in turn, be used to reconstruct an outdoor illumination environment map. We demonstrate that our approach allows the recovery of plausible illumination conditions and enables photorealistic virtual object insertion from a single image. An extensive evaluation on both the panorama dataset and captured HDR environment maps shows that our technique significantly outperforms previous solutions to this problem.
1. Introduction
The paper addresses the ill-posed problem of recovering outdoor illumination from a single image without strong assumptions about scene properties. It trains a CNN on panorama-derived image–lighting pairs to estimate full HDR outdoor lighting for practical image-based rendering.
- Outdoor illumination recovery is difficult because image formation conflates lighting with scene geometry and material properties.
- Existing methods use shadows, shading, geometry estimates, or hand-tuned priors, but often perform poorly on real-world or large-scale outdoor images.
- A large dataset of outdoor images with corresponding HDR lighting is unavailable, while capturing light probes is costly and time-consuming.
- The method fits a physically based sky model to panorama skies, recovering HDR information and compressing illumination into physically meaningful parameters.
- The CNN estimates full HDR lighting from an arbitrary single outdoor image, and the resulting environment map supports image-based lighting and photorealistic virtual-object rendering.
2. Related work
Prior outdoor illumination methods model skies, use handcrafted cues, retrieve panoramas, or exploit image collections and scene priors. The paper positions deep learning as a route to full HDR outdoor lighting estimation from one image.
- Outdoor illumination models: Outdoor sky models progressed from five-parameter all-weather distributions to simplified turbidity-based models and combined sun models.
- Outdoor lighting estimation: Lalonde et al. combine shadows, surface shading, and sky appearance to estimate sun direction and sky illumination, whereas this work trains a CNN to learn illumination directly.
- Outdoor lighting estimation: Other single-image methods rely on known geometry or strong reflectance, geometry, and illumination priors that typically do not generalize to large-scale outdoor scenes.
- Outdoor lighting estimation: Multi-view collections have been used to recover geometry, sun direction, capture conditions, or outdoor sun–sky colorimetric variation.
- Inverse graphics/vision problems in deep learning: Deep learning has addressed depth, normals, materials, intrinsic images, reflectance maps, and lighting from specular objects, but not previously full HDR outdoor lighting from one image.
3. Overview
The system uses SUN360 panoramas to create training images and targets despite their low dynamic range. A fitted physically based sky model supplies compact illumination parameters for CNN prediction.
- Full spherical panoramas capture scene appearance and provide direct views of the sun and sky, the most important outdoor light sources.
- Figure 2 varies turbidity from low to high, showing corresponding latitude–longitude environment maps and bunny renders on a ground plane.
- The pipeline fits a physically based sky model to SUN360 panoramas, then trains a CNN to predict the fitted illumination parameters from sampled input images.
- Here, “photo” denotes a standard limited-field-of-view camera image, while “panorama” denotes a 360-degree full-view panoramic image.
4. Dataset preparation
The method fits the Hošek-Wilkie physically based sky model to panorama skies, then estimates lighting parameters through a staged optimization robust to varied, uncalibrated imagery. The resulting model represents sky appearance using sun position, exposure, turbidity, and camera-related outputs, with validation showing both measurable sun-position accuracy and successful scene-consistency judgments.
- Sky lighting model: The Hošek-Wilkie model represents sky radiance using atmospheric turbidity, ground albedo, and sun position, with ground albedo fixed at 0.3.Its solar radiance extension is also used.
- Sky lighting model: The model converts spectral radiance into RGB sky colors and renders a discretized sky environment map, replacing pixels within 0.25° of the sun with the solar model.A scale factor estimates the varying exposure of each panorama.
- Sky lighting model: The fitted lighting representation retains sun position, exposure, and turbidity, with turbidity controlling the relative sun color and intensity against the sky.Low turbidity corresponds to clearer skies and brighter suns, while high turbidity approaches overcast conditions.
- Optimization procedure: Parameters are estimated by minimizing visible-sky reconstruction error over panorama pixels using a fixed gamma curve of γ = 2.2.The fitting procedure addresses unknown camera response, white balance, photographer processing, and stitching artifacts.
- Optimization procedure: The optimization first estimates sun position from the largest bright connected sky component, then searches turbidity initializations from 1 through 10 while solving exposure in closed form.The lowest-error parameter set is retained, and sky masks are refined using segmentation followed by a CRF.
- Validation of the optimization procedure: On 543 tonemapped HDR sky images, median sun-position error was 4.59°, while 65.6% of 1,236 SUN360 images received successful sky-parameter fit judgments.The user evaluation rendered a bunny under the synthesized sky and compared its lighting with the panorama scene.
5. Learning to predict outdoor lighting
The method trains a two-headed CNN on rendered views from fitted outdoor panoramas to predict sun position and sky-camera lighting parameters. It uses a distributional sun-position target and a combined loss, then evaluates accuracy against test-set and prior-method baselines.
- Training data: 38,814 panoramas yield seven randomly sampled 320 × 240 training photos each, varying camera elevation, azimuth, and field of view.The pinhole-camera views use elevations from −20° to 20°, azimuths from −180° to 180°, and vertical fields of view from 35° to 68°.
- CNN architecture: The feed-forward CNN has seven convolutional layers, a fully connected layer, and separate heads for sun position versus sky and camera parameters.The architecture explicitly separates directional-light estimation from the remaining lighting and camera quantities.
- Prediction targets: The sun-position head discretizes the sky hemisphere into 160 bins, while the parameters head directly regresses a four-dimensional parameter vector.The 160 bins combine five elevation divisions with 32 azimuth divisions.
- Prediction targets: The target sun-position distribution is a von Mises-Fisher distribution centered on the ground-truth position, with concentration κ = 80.This produces a confident target distribution around the correct sun position rather than a single hard label.
- Optimization: Training minimizes the sum of a KL-divergence loss for sun position and an MSE loss for the regressed parameters.The KL divergence supports learning a distribution over plausible sun positions, while MSE supervises the parameter vector.
- Evaluation: Approximately 10% higher quadrant precision than Lalonde et al. is achieved on their dataset, while the CNN remains robust on challenging SUN360 images.The methods have similar octant precision below 22.5°, but Lalonde et al.'s approach fails on the challenging 176-image SUN360 subset.
6. Evaluation
The CNN is evaluated on parameter prediction, relighting quality, virtual object insertion, and captured HDR panoramas. It produces robust sun-direction estimates and plausible relighting, while showing specific errors for high exposure and slightly overestimated exposure in HDR validation.
- 6.1. Illumination parameters on SUN360: 80% of SUN360 test images have sun-position errors below 45°.
- 6.1. Illumination parameters on SUN360: The CNN remains robust across scenes, fields of view, and layouts when predicting sun direction.
- 6.1. Illumination parameters on SUN360: CNN-based sun-azimuth precision within 45° improves by approximately 10% over on its 239-image dataset.
- 6.1. Illumination parameters on SUN360: The CNN favors low turbidity and has higher errors at high exposure.
- 6.2. Relighting on SUN360: Even the 75th error-percentile relit examples look qualitatively good, despite slight sun-direction and overall-color differences.
- 6.2. Relighting on SUN360: Estimated HDR illumination supports plausible virtual object insertion across sunny, overcast, noon, and dawn/dusk conditions without additional steps.
- 6.3. Validation with HDR panoramas: On captured HDR panoramas, the estimated exposure is slightly overestimated, making renders brighter than ground truth, while the relit bunny remains realistic.
7. Discussion
The method significantly outperforms previous work and supports realistic virtual object insertion, but its sky-model and environment-map scope impose important limitations.
- The method significantly outperforms previous work and can realistically insert virtual objects into photos.
- The Hošek-Wilkie model is accurate for clear skies but becomes less accurate with increasing cloud cover.Overcast panoramas can lead to underestimated turbidity and a low-turbidity bias in the CNN.
- Sun-position estimation fails when illumination cues mix with complex geometry, are absent, or appear on mirror-like surfaces.
- The reconstructed environment map models only the sky hemisphere, which is adequate for diffuse objects but problematic for specular materials.Specular surfaces cannot reflect scene texture from outside the modeled sky hemisphere.