Source-linked AI summary
DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality
Chloe LeGendre, Wan-Chun Ma, Graham Fyffe, John Flynn, Laurent Charbonnel, Jay Busch, Paul Debevec
TL;DR
Unconvincing lighting can make virtual objects look fake, while image cues may support plausible lighting estimates. DeepLight learns omnidirectional HDR illumination from LDR backgrounds using reflective-sphere supervision and image-based relighting. It improves rendered-object realism over state-of-the-art indoor and outdoor methods and runs interactively on mobile devices.
Problem
Lighting that is too bright, too dark, or inconsistent with the scene can make composited virtual objects look unconvincing, though background images may provide cues for plausible estimates.
Source: §1 · pp. 1–2Method
A CNN predicts omnidirectional HDR lighting from LDR backgrounds, trained by matching reflective-sphere images to differentiable image-based relighting renderings.
Source: §1 · p. 2Results
The method more accurately renders synthetic objects than state-of-the-art methods in both indoor and outdoor scenes and runs at interactive mobile frame rates.
Source: §1 · p. 2Takeaways & Limitations
Combined with real-time rendering, interactive mobile inference enables more realistic mobile mixed-reality composites.
Source: §1 · p. 2The reference spheres capture illumination from a point 60 cm in front of the camera and do not reveal spatially varying lighting cues.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We present a learning-based method to infer plausible high dynamic range (HDR), omnidirectional illumination given an unconstrained, low dynamic range (LDR) image from a mobile phone camera with a limited field of view (FOV). For training data, we collect videos of various reflective spheres placed within the camera's FOV, leaving most of the background unoccluded, leveraging that materials with diverse reflectance functions reveal different lighting cues in a single exposure. We train a deep neural network to regress from the LDR background image to HDR lighting by matching the LDR ground truth sphere images to those rendered with the predicted illumination using image-based relighting, which is differentiable. Our inference runs at interactive frame rates on a mobile device, enabling realistic rendering of virtual objects into real scenes for mobile mixed reality. Training on automatically exposed and white-balanced videos, we improve the realism of rendered objects compared to the state-of-the art methods for both indoor and outdoor scenes.
1. Introduction
Realistic mobile compositing requires plausible illumination, but estimating it from a single LDR image with limited field of view is under-constrained. DeepLight learns omnidirectional HDR lighting from unconstrained images using reflective-sphere supervision, and improves rendered-object realism indoors and outdoors.
1. Introduction
- A single LDR image with limited field of view misses much of the illumination arriving from surrounding directions, while bright sources can saturate in a single exposure.The camera observes only 6% of the panoramic scene in typical mobile phone video.Source: §1 · p. 1
- Differentiable image-based relighting trains the network by matching LDR sphere images to renderings under predicted HDR lighting, with an adversarial loss for plausible high-frequency illumination.The training examples use a single exposure, allowing videos of real scenes to provide data and a prior on automatic exposure and white balance.Source: §1 · p. 2
- On a random subset, inferred lighting more accurately rendered synthetic objects than state-of-the-art methods indoors and outdoors, while running at interactive mobile frame rates.Combined with real-time rendering, the network enables more realistic mobile mixed-reality composites.Source: §1 · p. 2
- The 200k-example dataset pairs background images with reference spheres whose diverse materials reveal different lighting cues in a single LDR exposure.The spheres occupy the bottom portion of the camera’s field of view, leaving the rest of each image available as input.Source: §1 · p. 2
- A CNN predicts plausible omnidirectional HDR illumination from a single unconstrained image and generalizes to both indoor and outdoor scenes.The authors describe it as the first approach, to their knowledge, to generalize across both scene types.Source: §1 · p. 2
2. Related work
Earlier lighting-estimation work often relied on HDR panoramas, scene-specific inputs, or constrained geometry and reflectance assumptions. DeepLight instead estimates lighting from unconstrained images, using a rendering-based loss and photographed reflectance bases.
2. Related work
- Prior inverse-rendering studies inferred lighting from constrained objects, segmented images, known material classes, or known geometry [34] [29] [53] [40] [15] [54], unlike this work’s unconstrained scenes.The target setting has unknown geometry and arbitrarily complex scenes.Source: §2 · p. 2
- Khan et al. inferred 360° lighting by projecting a limited-field-of-view HDR image onto a hemisphere and flipping it.Source: §2 · p. 2
- Gardner et al. [14] refined light-source intensities using 2,100 newly captured HDR panoramas, whereas the proposed rendering-based loss learns incident-illumination colors and intensities relative to the input without HDR imagery.Source: §2 · p. 3
- Face-lighting approaches used simple or low-frequency shading models, whereas this work uses image-based relighting with photographed reflectance bases and multiple BRDFs.Xu et al. jointly learned a low-dimensional basis and renderer; this method applies image-based relighting as a fixed function.Source: §2 · p. 3
3. Method
The method trains on mobile LDR backgrounds paired with images of three reflective spheres, using their varied BRDFs to supervise HDR illumination through image-based relighting. A CNN predicts log-space omnidirectional lighting, with reconstruction and adversarial losses.
3.1. Training Data Acquisition and Processing
- LDR videos pair each background with three differently reflective spheres, whose BRDFs reveal distinct lighting cues while leaving most of the background visible.Automatic exposure and white balance provide a prior for disambiguating color, reflectance, and illumination.Source: §3.1 · p. 3
3.2. Network Architecture
- A MobileNetV2-based encoder-decoder maps a mobile LDR image to a 32×32 log-space HDR environment in mirror-ball mapping.The output mapping represents equal solid angular portions of the sphere for lighting directions.Source: §3.2 · pp. 3–4
3.3. Reflectance Field Acquisition
- Reflectance fields for diffuse and matte silver spheres are photographed with a controllable LED sphere, while the aliased mirror BRDF uses a synthesized one-hot mirror-ball basis.The bases are converted to a common geometric and relative radiometric space.Source: §3.3 · p. 4
3.4. Loss Function
- The relighting reconstruction loss compares rendered, clipped, gamma-encoded sphere images with LDR ground-truth images using masked L1 error across the BRDFs.Rendered images are soft-clipped and gamma-encoded to match the target images.Source: §3.4 · pp. 4–5
- An adversarial loss on clipped mirror-ball composites supplements reconstruction loss, which alone produces blurred, low-frequency illumination.The discriminator receives local color cues from a hallucinated clean plate around the sphere.Source: §3.4 · p. 5
3.5. Implementation Details
- Training used 4.06 million examples from 37.6 video hours; the network trained for 16 epochs, and evaluation sets cover seen and unseen indoor and outdoor scenes.The test videos were recorded with the same camera; the four sets comprise 28 UI, 27 UO, 27 SI, and 34 SO videos.Source: §3.5 · p. 5
4. Evaluation
The evaluation tests sphere renderings, virtual-object composites, ablations, and mobile inference speed. It compares the method with prior indoor and outdoor estimators and examines temporal consistency.
4.1. Quantitative Results
- Ablations show that single-BRDF training raises losses for other materials, mirror-only training misses full lighting dynamic range, and adversarial loss yields higher-frequency illumination.Matte-silver-only training still gives low diffuse-sphere loss, but matte-silver-only or diffuse-only training fails to produce a realistic mirror sphere.Source: §4.1 · pp. 5–6, Figure 5 · p. 6
4.2. Qualitative Results
- Figure 6 compares ground-truth spheres with renderings using image-based relighting (IBRL) and HDR lighting inference, showing examples at the 25th, 50th, and 75th percentiles for Lrec.Source: Figure 6 · p. 6, §4.2 · p. 6
4.3. Comparisons with Previous Work
- The benchmark uses training and test data from the same fixed-FOV camera, unlike prior approaches with varied, unknown FOVs.Source: §4.3 · pp. 6–7
- Across 450 test images, the method significantly improves sphere-rendering comparisons over [14] and [18], using one model that generalizes to indoor and outdoor scenes.The comparisons report average L1 loss by BRDF and diffuse-sphere RGB angular error.Source: §4.3 · p. 6, Table 2 · p. 7
- Temporal consistency is not explicitly optimized; adjacent training-video frames provide indirect regularization, and the sequential-frame comparison shows qualitative improvement over [14].Source: §4.3 · p. 7, Figure 10 · p. 8
4.4. Performance and Demonstration
- Inference runs at 12–20 fps on mobile phone CPUs, and a demo application renders virtual objects using real-time pre-computed radiance transfer.Source: §4.4 · pp. 7–8
5. Limitations and Future Work
The method does not capture spatially varying illumination, may not generalize across camera FOVs, and can be challenged by certain scene content. The authors identify data, geometry, and temporal modeling as directions for improvement.
Spatially-varying illumination
- Reference spheres capture illumination from one point 60 cm in front of the camera, so they do not reveal spatially varying lighting cues.The authors suggest replacing the estimate’s bottom directions with pixels sampled from the surface beneath each object to represent differently colored bounce light.Source: §5 · p. 8
Using a different camera
- The same camera was used for training and testing, and the network may not generalize to other camera FOVs.Results from an iPhone 6 qualitatively show white-balance differences.Source: §5 · p. 8, Figure 11 · p. 8
Challenging image content
- Scenes with little surface-normal or albedo variation, and scenes dominated by strongly hued material, can challenge lighting inference.Figure 12 shows examples of these challenging scenes.Source: §5 · p. 8, Figure 12 · p. 8
Future work
- Future work could use geometric reasoning and inertial measurements, explicitly optimize temporal stability, and simulate camera models from raw video training data.The authors note that temporal structure is already present in the training data.Source: §5 · p. 8
6. Conclusion
The paper presents an HDR lighting inference method for mobile mixed reality, trained only on LDR imagery and using differently reflective reference spheres. It reports generalization across indoor and outdoor scenes and improved lighting estimation compared with prior work designed for one lighting class.
6. Conclusion
- The method trains on LDR imagery, using reference spheres with different materials to reveal distinct lighting cues in one exposure.The inferred HDR lighting is intended for mobile mixed reality.Source: §6 · p. 8
- The authors describe this as the first CNN-based approach to generalize to both indoor and outdoor scenes from a single input image.Source: §6 · p. 8
- The method improves mobile mixed-reality lighting estimation over previous work developed to handle only a single class of lighting.Source: §6 · p. 8