Deep Sky Modeling for Single Image Outdoor Lighting Estimation

1. Introduction

• Outdoor lighting varies with weather and time of day, while common analytical sky models degrade for partially cloudy and overcast skies.The paper frames estimating this wide range of conditions from outdoor images as challenging.Source: §1 · p. 1
• The proposed nonanalytical, learned HDR sky model can be estimated directly from a single outdoor image and models more lighting conditions than previous approaches.The authors illustrate the model's broader range with skies from sunny to overcast.Source: §1 · pp. 1–2
• Training combines a sky autoencoder on Laval HDR skies with HDR-converted SUN360 panoramas mapped to sky labels, then trains an image encoder to predict those labels from a crop.The process uses complementary datasets to learn the sky representation and its mapping from image content.Source: §1 · p. 2
• The paper reports a dataset of 206 radiometrically calibrated outdoor HDR 360° panoramas and qualitative and quantitative outperformance of the state of the art in experiments and a user study.It also proposes bridging its implicit representation and low-dimensional parametric sky models.Source: §1 · p. 2

Outdoor lighting modeling

• Sky modeling has been studied across atmospheric science, physics, and graphics, with models including Perez's five-parameter All-Weather model and the Preetham and Hošek-Wilkie models.The supplied passage describes Perez's model as an improvement over the CIE Standard Clear Sky model and Preetham's as using atmospheric turbidity.Source: §2 · p. 2
• Figure 2 compares the captured HDR panoramas, learned-sky reconstructions, and fitted Hošek-Wilkie models, with renders illustrating their lighting and shadow characteristics.The caption describes examples spanning sunny to overcast conditions.Source: Figure 2 · p. 2

Outdoor lighting estimation

• Earlier single-image estimation approaches combine image cues, transfer lighting from matched panoramas, or use known geometry and priors on geometry or reflectance.The panorama-matching approach can yield lighting that is inconsistent.Source: §2 · p. 2

Deep learning for lighting estimation

• Deep-learning approaches include estimating lighting and reflectance from known-geometry objects and regressing Hošek-Wilkie parameters from outdoor images; this paper instead learns a data-driven model for generic scenes.The related work also contrasts face-based multi-step nonlinear optimization and low-frequency spherical harmonics with the paper's approach.Source: §2 · p. 3

3. Overview

• The method's goal is to estimate illumination conditions from an outdoor image.Source: §3 · p. 3
• Direct supervised training is described as currently impossible because no large-scale dataset pairs images with their illumination conditions.Source: §3 · p. 3
• The paper therefore proposes a three-step approach, also illustrated in Figure 3.Source: §3 · p. 3

1. Train the SkyNet autoencoder on HDR skies

• SkyNet is trained on 33,420 hemispherical sky images from the Laval HDR sky database.Source: §1 · p. 3
• The autoencoder compresses an HDR sky image into a 64-dimensional latent vector z and reconstructs it at the original resolution.Source: §1 · p. 3
• Training enforces robustness to white-balance and exposure changes and occlusions.Source: §1 · p. 3

2. Label LDR panoramas with SkyNet

• SUN360 panoramas are converted to HDR, sky-segmented, and passed through the SkyNet encoder to obtain compact latent labels for outdoor illumination.The resulting SUN360-HDR panoramas and their sky masks provide data-driven representations for the panoramas.Source: §2 · p. 3

3. Train image encoders to predict illumination

• An image encoder predicts the SkyNet latent vector z for each limited-field-of-view crop, and the SkyNet decoder recovers the corresponding HDR sky image.The encoder is trained on crops from SUN360 using the methodology of [12].Source: §3 · pp. 3–4

4.1. Deep autoencoder

• SkyNet compresses each HDR sky into 64 latent parameters and reconstructs it using an autoencoder with residual layers.The training scheme includes occlusions and radiometric distortions, supporting end-to-end learning.Source: §4.1 · p. 4
• Training uses Laval HDR panoramas augmented with SUN360-HDR, masking non-sky regions and randomly occluding Laval skies so SkyNet learns to fill masked areas.The sky is represented as a 32 × 128 RGB equirectangular image with the sun centered horizontally.Source: §4.1 · p. 4

4.2. Training losses

• The training losses align latent codes for radiometrically distorted and original inputs, then reconstruct both inputs against the original panorama using L1 loss.Reconstruction is evaluated only where ground-truth sky is known in SUN360-HDR, while masked Laval regions are supervised against the original sky.Source: §4.2 · pp. 4–5
• SkyNet reconstructions model sky energy distributions and reproduce shadow contrast and sharpness, while the Hošek-Wilkie model does not generalize to non-clear skies.The paper notes that LDR content such as clouds is lost in the reconstructions.Source: §4.2 · p. 5

4.3. Implementation details

• The sky model has approximately 1 million parameters, converges after 127 epochs, and takes roughly 10 ms for inference on a Titan Xp GPU.Training took roughly 4 hours on the same machine.Source: §4.3 · p. 5

5.1. Image lighting estimation

• The proposed model shows a slight improvement over [12] in sun-position estimation on both the SUN360 test set and the captured HDR panoramas.The comparison uses cumulative sun-position estimation error; the passage attributes the slight improvement to using DenseNet-161 [15].Source: Figure 5 · p. 5
• A pretrained DenseNet-161 with a 64-output layer estimates sky parameters from a limited-field-of-view image using L2 loss, supplemented by an L1 loss on reconstructed sky panoramas.The added panorama loss addresses average predictions that missed sky-energy details and strong sun intensities; it is weighted by solid angle.Source: §5.1 · p. 5

5.2. Sun azimuth estimation

• Because SkyNet centers the sun in its estimated panorama, a separate DenseNet-161 predicts sun azimuth as a distribution over 32 bins.Joint training of sky parameters and azimuth failed to generalize well for both tasks simultaneously.Source: §5.2 · p. 5

5.3. Implementation details

• Training uses SUN360-HDR augmented with 100 images from 15 captured HDR panoramas, weighting errors on captured panoramas by a factor of 4.The weighting counters the severe imbalance between the two data sources.Source: §5.3 · pp. 5–6

5.1. Image lighting estimation

• The estimated environment map yields almost 10× improvement in RMSE over [12] and enhances si-RMSE by over 60% over the previous state-of-the-art.The reported relighting comparison uses the bunny scene; the lighting is rotated so the sun has the same azimuth for both methods.Source: Figure 6 · p. 6

6.1. A dataset of outdoor HDR panoramas

• The study captured a new dataset of 206 panoramas, selected 191 for evaluation, and extracted 7 crops per panorama, totaling 1,337 evaluation images.The 191 evaluation panoramas did not overlap in location or illumination conditions with the 15 used to train the image-to-sky encoder.Source: §6.1 · pp. 6–7

6.3. Lighting evaluation on HDR panoramas

• On HDR panoramas, the method reproduces sharp, smooth, and absent cast shadows, which the parametric sky model of [12] cannot reproduce.SUN360 has no ground-truth illumination for quantitative relighting evaluation; its available panorama is a saturated LDR image.Source: Figure 7 · p. 6, Figure 8 · p. 7

6.4. User study on SUN360 LDR panoramas

• More than 68% of user-study votes favored the method, which was preferred on 16 of 20 SUN360 image pairs.The study showed 39 participants paired images with virtual objects lit by the method or [12].Source: Table 2 · p. 7, §6.4 · p. 7

6.2. Quantitative sun position evaluation

• Sun-position estimation systematically improves over [12] on both SUN360 and the new panorama dataset.Ground truth is the brightest-region center; the method estimates azimuth explicitly and takes elevation from the brightest pixel of its reconstructed panorama.Source: §6.2 · p. 7

6.3. Lighting evaluation on HDR panoramas

• Scale-invariant RMSE improves by an additional 60%, alongside significant improvement in RMSE, over [12] on 1,337 images with ground-truth illumination.The authors attribute most of the RMSE improvement to [12]'s exposure estimate being biased toward bright skies.Source: §6.3 · p. 7
• Virtual object insertions have overall lighting intensity closer to ground truth than [12] while retaining plausible shading.The comparisons use the Cycles renderer and backgrounds from the evaluation dataset.Source: §6.3 · p. 7, Figure 9 · p. 8

• Interpolating between implicit sky parameters z can cause abrupt transitions and artifacts such as multiple suns.Unlike low-dimensional analytical models, z cannot be directly hand-tuned.Source: §7 · p. 8
• The editing method iteratively uses sky-decoder gradients to morph a known sky toward a target sun-elevation or sun-intensity change.For elevation edits, it shifts a 5×5 region around the sun; for intensity edits, it scales that region toward the desired maximum.Source: §7 · p. 8
• After up to 300 iterations, re-projecting edits back into the sky model removes manually induced artifacts and yields visually smooth transitions.The example edits change sun position on cloudy and clear days and vary sun intensity from fully visible to mostly occluded.Source: §7 · p. 8, Figure 10 · p. 8

• The authors report that their method significantly outperforms prior state of the art on lighting reconstruction and estimation.They describe the learned sky model as, to their knowledge, the first trained end-to-end.Source: §8 · p. 8
• The model captures sky energy accurately, but its texture recovery quality remains limited.Source: §8 · p. 8
• Some users preferred [12]'s stronger lighting and sharper shadows in the study.Source: §8 · p. 8