PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Illumination: Lighting Prediction for Indoor Environments

Shuran Song, Thomas Funkhouser

arXiv:1906.07370v1cs.CV

TL;DR

Estimating full HDR illumination at a selected indoor location from one LDR image is difficult because geometry, unseen regions, and dynamic range must all be inferred. Neural Illumination decomposes this into geometry-aware warping, LDR completion, and LDR-to-HDR prediction; the paper reports better quantitative and qualitative results than prior work.

  • Problem

    Estimating HDR illumination at a selected indoor location from one RGB image requires inferring scene geometry, unseen lighting, and high dynamic range from partial observations.

    Source: §1 · p. 1

  • Method

    Neural Illumination estimates scene geometry to warp observations around the target locale, completes unobserved LDR regions, and maps the completed representation to HDR.

    Source: §1 · pp. 1–2

  • Results

    The three-subtask architecture achieves significantly better quantitative and qualitative results than prior work on the new benchmark.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The predicted illumination map can light virtual objects at the selected locale to blend them into real imagery and can assist estimation of scene properties such as materials.

    Source: §1 · p. 1

    The generated illumination maps are not always perfect, especially for highly specular surfaces.

    Source: §4 · p. 4

Abstract

from arXiv · show

This paper addresses the task of estimating the light arriving from all directions to a 3D point observed at a selected pixel in an RGB image. This task is challenging because it requires predicting a mapping from a partial scene observation by a camera to a complete illumination map for a selected position, which depends on the 3D location of the selection, the distribution of unobserved light sources, the occlusions caused by scene geometry, etc. Previous methods attempt to learn this complex mapping directly using a single black-box neural network, which often fails to estimate high-frequency lighting details for scenes with complicated 3D geometry. Instead, we propose "Neural Illumination" a new approach that decomposes illumination prediction into several simpler differentiable sub-tasks: 1) geometry estimation, 2) scene completion, and 3) LDR-to-HDR estimation. The advantage of this approach is that the sub-tasks are relatively easy to learn and can be trained with direct supervision, while the whole pipeline is fully differentiable and can be fine-tuned with end-to-end supervision. Experiments show that our approach performs significantly better quantitatively and qualitatively than prior work.

1. Introduction

The paper estimates the HDR illumination arriving at a selected indoor location from a single RGB image, a task requiring reasoning about scene geometry, unseen lighting, and dynamic range. Neural Illumination decomposes the task into supervised, differentiable sub-tasks and evaluates the approach on a benchmark of paired images and illumination maps.

1. Introduction

  • The task is to infer an HDR map of light arriving from every direction at the 3D location associated with a selected image pixel.The image provides only a partial scene observation, while the prediction depends on geometry, unobserved light sources, occlusions, and LDR-to-HDR inference.Source: §1 · p. 1
  • Neural Illumination estimates scene geometry, warps observed pixels to a spherical projection around the target locale, completes unseen LDR regions, and maps the result to HDR.Its differentiable sub-modules are directly supervised individually and then fine-tuned end-to-end.Source: §1 · p. 2, Figure 2 · p. 2
  • Geometry-based warping aligns input and output representations pixel by pixel, while task decomposition lets each sub-module focus on an easier, directly supervised task.The paper identifies both properties as central to making end-to-end training practical.Source: §1 · p. 2
  • The benchmark contains 90,280 ground-truth illumination maps paired with 129,600 images, curated from dense HDR RGB-D panoramas in real-world scenes.The dataset supports training and evaluation across a diverse set of locales.Source: §1 · p. 2
  • The three-subtask architecture achieves significantly better quantitative and qualitative results than prior work on the new benchmark.The architecture combines intermediate and end-to-end supervision with an intermediate representation that shares pixel-wise spatial correspondence with the output.Source: §1 · p. 2

2. Related Work

Prior illumination-estimation approaches capture light with physical probes, optimize scene parameters, retrieve similar images, or learn mappings from images. Neural Illumination contrasts with these methods by using geometry-aware warping and dense RGB-D/HDR training data.

Capture-based Methods

  • Physical probes can capture illumination accurately, but collecting high-quality maps across many locations and environments is expensive and difficult to scale.Probe-based approaches use objects such as mirrored or hybrid spheres, known 3D objects, materials, or faces as light proxies.Source: §2 · p. 2

Optimization-based Methods

  • Jointly optimizing geometry, reflectance, and lighting is under-constrained, so prior methods use user annotations, depth assumptions, or low-dimensional lighting models.Lombardi and Nishino's low-dimensional approximation was later found sub-optimal for indoor scenes because of object reflectance and geometry.Source: §2 · p. 3
  • Image-matching and image-flipping methods use similar images or observed HDR images to estimate illumination, whereas this system learns illumination priors without user inputs or manual annotations.The cited approaches include image matching to retrieve annotated panoramas and flipping observed HDR images.Source: §2 · p. 3

Learning-based Methods

  • Gardner et al.'s direct image-to-map network [7] lacks pixel-wise correspondence and target-pixel conditioning; Neural Illumination warps using predicted geometry and trains on accurate geometry and illumination data.The prior training data used spherical warping and binary light masks as approximations where depth and HDR information were unavailable.Source: §2 · p. 3

3. Problem formulation

The paper formulates illumination estimation as pixel-wise regression from an image and a selected locale to an HDR illumination map. The output map is a spherical panorama representing incoming light directions.

3. Problem formulation

  • The regression function maps an input LDR scene image and selected locale to the locale's HDR illumination map.The paper denotes this mapping as f(I|ℓ) = Hℓ.Source: §3 · p. 3
  • The output is a spherical panorama with 180° vertical and 360° horizontal field of view, whose pixels encode incoming RGB light by direction.Each panorama pixel h(φ, θ) stores the RGB intensity of light arriving from direction (φ, θ).Source: §3 · p. 3
  • The training pairs of images, locales, and illumination maps are generated from Matterport3D.Source: §3 · p. 3

4. Generating a Dataset of Illumination Maps

The dataset pipeline uses dense HDR RGB-D panoramas to generate illumination maps for arbitrary indoor locales. It samples supported locations, reconstructs maps through depth-aware warping and blending, and provides varied locale-specific training examples, though maps can be imperfect on highly specular surfaces.

4. Generating a Dataset of Illumination Maps

  • Physical probes are costly and time-consuming to scale, while existing panorama datasets provide maps only at camera locations.The paper instead uses Matterport3D HDR RGB-D imagery with geometric warping to generate training data for arbitrary locales.Source: §4 · pp. 3–4
  • Matterport3D's 194,400 registered HDR RGB-D images across 10,800 panoramas and 90 indoor scenes support reconstruction of maps at arbitrary scene points.The panoramas are densely spaced through each scene, about 2.5 m apart, and include depth for image-based rendering.Source: §4 · p. 4
  • Locales are sampled 10 cm above horizontal floor or furniture surfaces, subject to clearance and spacing criteria, then paired with unoccluded views.The support surface must have sufficient clearance for an object of radius 10 cm, and locales are kept at least 50 cm apart.Source: §4 · p. 4
  • The reconstruction estimates nearest-surface distance by direction, fills depth holes, and blends reverse-mapped HDR samples with weights proportional to 1/d^4.This produces 90,280 locales and 360,432 image-locale/map pairs.Source: §4 · p. 4
  • The warped maps offer diverse lighting and unique illumination for each locale, but are not always perfect, especially for highly specular surfaces.The RGB-D warping accounts for complex geometry and occlusions more accurately than spherical warping.Source: §4 · p. 4

5. Network Architecture

The architecture chains geometry estimation, differential warping, panorama completion, and LDR-to-HDR prediction, with modules pretrained under direct supervision and then fine-tuned end-to-end. Its stages map image pixels to a locale-centered panorama, complete unseen illumination, and recover HDR intensity.

5. Network Architecture

  • Four sequential modules—geometry estimation, differential warping, out-of-view illumination estimation, and LDR-to-HDR estimation—are pretrained with ground-truth input-output pairs and then fine-tuned together.Intermediate and end-to-end supervision are described as crucial for efficient learning.Source: §5 · pp. 4–5

5.1. Geometry Estimation

  • The geometry module predicts per-pixel surface normals and plane offsets, converts them to 3D locations, and uses cosine and ℓ1 losses for supervision.The plane-equation representation regularizes large planar surfaces while retaining flexibility for other surfaces.Source: §5.1 · p. 5

5.2. Geometry-aware Warping

  • Forward projection maps observed pixels onto an upright locale-centered sphere, setting unprojected regions to -1 and aligning the panorama pixelwise with the output illumination map.For data generation, 3D scene geometry yields geometrically accurate ground-truth illumination compared with spherical warping from 2D panoramas [7].Source: §5.2 · p. 5, Figure 5 · p. 5

5.3. LDR Panorama Completion

  • The completion module predicts a full LDR panorama from mapped observations, including unseen pixels; adversarial training addresses multiple plausible completions and yields sharper, richer details.The module uses a generative network and discriminator, with distortion-aware convolutional filters for spherical panoramas.Source: §5.3 · p. 5

5.4. LDR-to-HDR Estimation

  • The LDR-to-HDR module predicts HDR intensity for every direction from LDR illumination, using pixelwise and diffuse-convolution losses to encourage both low- and high-frequency signals.The pixelwise loss measures specular relighting error, while diffuse convolution measures diffuse relighting error; the combined loss uses λ1 = 0.1 and λ2 = 0.05.Source: §5.4 · p. 6

6. Evaluation

Evaluation compares Neural Illumination with prior methods and tests oracle and loss variants; the proposed method produces more accurate illumination maps with richer high-frequency detail.

Evaluation metrics

  • Evaluation includes pixel-wise ℓ2 and log-intensity ℓ2 errors, plus diffuse-convolution error that approximates rendering differences on a perfectly diffuse surface.The log-intensity error approximates visual differences on a perfectly specular surface.Source: §6 · p. 6

Comparisons to state-of-the-art

  • The method predicts spatially varying lighting for any selected image locale, whereas Gardner et al. produces one lighting solution per image.Different locales in one image can have quite different illumination.Source: §6 · p. 6
  • Neural Illumination is more accurate than the Im2Im network and Gardner et al.; qualitative comparisons show richer, more realistic high-frequency detail.Table 1 compares quantitative performance against Gardner et al. and nearest-neighbor retrieval.Source: §6 · p. 6, Table 1 · p. 7, Figure 7 · p. 7
  • Relighting comparisons show richer, more realistic high-frequency detail than Gardner et al., though the method can add plausible but incorrect ceiling lights.Figure 8 compares ground-truth, proposed, and Gardner et al. illumination maps alongside composites rendered with two surface materials.Source: Figure 8 · p. 8

Comparisons to variants with oracles

  • Ground-truth depth improves performance marginally (∆ℓ2 = 0.011), while ground-truth HDR improves it more (∆ℓ2 = 0.043).The oracle variants use ground-truth inputs and are not fair comparisons; the authors place their method about halfway between the best oracle and baselines.Source: §6 · p. 8, Table 3 · p. 8

Effects of different losses

  • Adding adversarial loss yields more realistic high-frequency signals and spotlights but adds noise; diffuse convolution further improves accuracy, especially for high-intensity areas.With only ℓ2 loss, estimates tend to be blurry and close to the input images’ mean intensity.Source: §6 · p. 8

7. Conclusion and Future Work

Neural Illumination decomposes indoor illumination estimation into subtasks and predicts more detailed, accurate maps than alternatives. Its estimates can remain plausible rather than accurate when the input shows no lights.

Conclusion

  • Neural Illumination estimates HDR illumination for a selected pixel by inferring geometry, warping observations, estimating unobserved illumination, and mapping LDR to HDR.The framework is end-to-end.Source: §7 · p. 8
  • Experiments show that the decomposition predicts illumination maps with better detail and accuracy than alternative methods.Source: §7 · p. 8

Future Work

  • When no lights are directly observed, predictions can be plausible rather than accurate; future work may model materials and reflections or explore other geometric representations.The proposed geometric representations are intended to facilitate out-of-view illumination estimation through whole-scene understanding.Source: §7 · p. 8

A. Appendix

The appendix provides additional algorithm details, evaluation information, and results.

A. Appendix

  • Supplementary material provides further details about the algorithm and evaluation, along with additional results.Source: A. Appendix · p. 10

B. Network Architecture Details

The architecture comprises geometry estimation, LDR illumination completion, and LDR-to-HDR estimation modules.

Geometry estimation module

  • A two-stream U-Net takes one RGB image and predicts dense visible geometry through surface normals and plane offsets.A PN-layer converts these predictions into pixel-wise 3D locations; the normal stream outputs three channels and the plane-distance stream one.Source: B. Network Architecture Details · p. 10

LDR completion module

  • The LDR completion module maps warped observations to dense illumination predictions for observed and unobserved pixels using a generative network.The generator is a fully convolutional ResNet50 with distortion-aware convolutional filters; a discriminator is also specified.Source: B. Network Architecture Details · p. 10

LDR-HDR Estimation module

  • The LDR-HDR module maps predicted LDR illumination to dense HDR intensities representing incident radiance in every direction at the target locale.It uses a U-Net architecture.Source: B. Network Architecture Details · p. 10

C. Training details.

The modules are pretrained separately and then fine-tuned end to end, using supervision and inputs tailored to each stage. Training data include illumination maps assembled from warped observations at sampled locales.

C. Training details.

  • Each module is pretrained for five epochs, then the whole network is fine-tuned for 10,000 iterations.Pretraining uses batch size 4 and learning rate 0.001; fine-tuning uses batch size 1 and learning rate 0.0001.Source: C. Training details · p. 10
  • Geometry estimation uses the color image as input and surface normal and plane distance computed from Matterport3D depth images as supervision.Source: C. Training details · p. 10
  • The LDR completion module is pretrained on partial maps warped with ground-truth geometry, then fine-tuned and tested on maps warped with estimated geometry.Its input is the partial illumination map called the “observation panorama,” and its output is a completed LDR illumination map.Source: C. Training details · p. 10
  • The LDR-to-HDR module is pretrained on dataset LDR–HDR pairs and uses the completion module’s estimated LDR output during fine-tuning and testing.Source: C. Training details · p. 10
  • Training data comprise locale illumination maps generated by combining warped partial observations from different viewpoints.The sampled locale positions and examples of observations and warped partial maps are shown in Figures 9 and 10.Source: C. Training details · p. 10

D. Additional Results

Additional results illustrate location-conditioned illumination, object relighting, and intermediate-to-final predictions. The examples show both how locale maps are assembled and how the estimated geometry and illumination stages compare visually with ground truth.

D. Additional Results

  • Different selected pixels in one image yield ground-truth and estimated illumination maps conditioned on their respective locales.Figure 11 illustrates spatially varying illumination using 3D geometry to generate ground truth for any target locale.Source: D. Additional Results · p. 10, Figure 11 · p. 12
  • Figure 10 combines warped partial illumination observations from different viewpoints to form a locale’s ground-truth illumination map.The sampled locale positions are shown as green dots, with floor and furniture locales visualized separately in Figure 9.Source: Figure 10 · p. 11, Figure 9 · p. 11
  • Figures 12 and 13 compare Mitsuba relighting rendered with ground-truth, proposed-method, and Gardner et al. illumination maps across three surface materials.Source: Figure 12 · p. 13, Figure 13 · p. 14
  • Figures 14–16 show qualitative examples spanning input and selected locale, ground-truth and estimated geometry, warped observation, completed LDR, and final HDR illumination.Figure 14 specifies these stages, with the final HDR visualized using gamma correction γ=3.3.Source: D. Additional Results · p. 11, Figure 14 · p. 15, Figure 15 · p. 16
Loading 1906.07370v1…