Source-linked AI summary
Fast Spatially-Varying Indoor Lighting Estimation
Mathieu Garon, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, Jean-François Lalonde
TL;DR
Estimating indoor lighting from a single image is difficult because lighting is spatially varying and entangled with scene properties, while prior practical methods often provide only global estimates or require extra geometry information. The paper predicts local fifth-order spherical-harmonic lighting from an image and 2D location in real time, with local estimates supporting AR relighting and reported advantages over prior work. Its scope is limited by occasional hue shifts and the current single-image formulation.
Problem
Indoor lighting is spatially varying and difficult to recover from a single image because lighting is conflated with geometry, reflectance, and camera properties.
Method
A CNN takes one RGB image and a 2D image location and outputs fifth-order spherical-harmonic coefficients for local lighting without explicit geometry estimates.
Results
The method estimates local lighting in less than 20ms on a mobile GPU and significantly outperforms previous work across synthetic and real-data evaluations.
Takeaways & Limitations
Local spherical-harmonic estimates can be used directly for real-time augmented-reality relighting that adapts to lighting changes across an indoor scene.
Takeaways & Limitations
The method sometimes produces incorrect hue shifts, and it is currently applied to single images rather than trained with temporal video information.
Abstract
from arXiv · showhide
We propose a real-time method to estimate spatiallyvarying indoor lighting from a single RGB image. Given an image and a 2D location in that image, our CNN estimates a 5th order spherical harmonic representation of the lighting at the given location in less than 20ms on a laptop mobile graphics card. While existing approaches estimate a single, global lighting representation or require depth as input, our method reasons about local lighting without requiring any geometry information. We demonstrate, through quantitative experiments including a user study, that our results achieve lower lighting estimation errors and are preferred by users over the state-of-the-art. Our approach can be used directly for augmented reality applications, where a virtual object is relit realistically at any position in the scene in real-time.
1. Introduction
Indoor lighting varies across a scene, but prior single-image methods estimate global lighting and practical alternatives often require cumbersome additional information. The paper introduces a real-time method that predicts spatially varying lighting from one RGB image and 2D location.
- Motivation: Indoor illumination is difficult to recover because image appearance conflates lighting with geometry, reflectance, and camera properties.This makes lighting inversion severely underconstrained, while probe-based solutions require inserting objects with known properties.
- Limitations of prior work: Prior approaches use depth, scene scans, or user input, whereas recent single-image learning predicts one global lighting estimate rather than local illumination.The global estimate is unsuitable for scenes where nearby light sources create different conditions through occlusions and non-uniform distributions.
- Proposed approach: The proposed method estimates spatially varying spherical-harmonic lighting from a single RGB image in real time.It takes an image and a 2D image location as input and outputs fifth-order spherical-harmonic coefficients for lighting at that location.
- Advantages: The method predicts fifth-order spherical-harmonic coefficients in less than 20ms on a mobile GPU and directly supports real-time relighting with off-the-shelf shaders.Local estimates also embed visibility effects without explicit geometry estimation.
- Evaluation resource: The paper presents a public dataset containing 79 ground-truth HDR light probes sampled across varied indoor scenes for quantitative evaluation.The dataset is intended to evaluate spatially varying lighting methods quantitatively.
2. Related work
Related work estimates illumination using known objects, reconstructed geometry, user input, multiple images, or single-image learning. The paper differs by targeting spatially varying lighting from a single indoor image without requiring explicit scene reconstruction.
- Object-based estimation: Many illumination methods rely on inserted or known-geometry objects to disentangle lighting from reflectance and shape.Examples include reflective spheres, generic 3D objects, faces, and learned reflectance maps.
- Geometry- and scan-based methods: Real-time augmented-reality methods model scene-wide radiance transfer or spatially varying illumination but require scene reconstruction, scans, or user-annotated light positions.These requirements make them cumbersome for practical single-image use.
- Single-image methods: Single-image approaches approximate unseen illumination, use user guidance, or learn from panoramas, but directional lighting can remain unreliable and user interaction may be required.Panorama-based neural methods also provide a single global lighting estimate.
3. Dataset
Because existing panorama datasets lack local occlusions and collecting sufficient real training data is resource-intensive, the paper constructs synthetic local-lighting supervision. It renders images, location-specific light probes, and depth probes, then fits fifth-order spherical-harmonic ground truth.
- Dataset motivation: Learning local lighting requires images paired with illumination measurements at multiple scene locations, but panorama datasets do not capture local occlusions.The authors therefore use realistic synthetic data for training and reserve a small real-photo dataset for evaluation.
- Rendering images: The synthetic scenes use SUNCG renders with randomly scaled source intensities and randomly rotated HDR outdoor panoramas for window illumination.Outdoor panoramas were found to improve realism and lighting distributions in scenes containing windows.
- Rendering images: The dataset contains 26,800 images rendered at 640×480 resolution, with training and validation separated by house.Rendering uses Mitsuba's Metropolis Light Transport with 512 samples.
- Local probe sampling: Four image locations are sampled per image, and each probe camera is placed 10cm from the first visible surface intersection along its normal.Probe-camera axes remain aligned with the scene camera, with only translation applied.
- Ground-truth probes: Each probe is rendered as six cube-map faces with corresponding depth, filtered for erroneous or invalid cases, and converted to fifth-order spherical-harmonic coefficients.Bidirectional path tracing uses 1,024 samples per face at 64×64 resolution; rendering all six faces takes about five minutes on average.
4. Learning to estimate local indoor lighting
The method combines global image context with a local patch centered on the queried location to estimate spatially varying lighting and related scene properties. Joint auxiliary tasks and domain adaptation are integrated into the network to improve lighting estimation and transfer from synthetic training data to real photographs.
- Main architecture for lighting estimation: The network combines full-image and location-centered patch features before predicting 5th-order RGB lighting SH coefficients.The global path receives the full 341 × 256 image, while the local path receives a 150 × 150 patch centered at the queried coordinate.
- Main architecture for lighting estimation: A coordinate mask identifies the queried region in the global stream, preserving probe-position information alongside scene-wide context.The mask is concatenated with the global feature map before encoding.
- Learning additional subtasks: The model jointly predicts probe-depth SH coefficients to capture local visibility effects that influence illumination.The depth branch adds a 36-dimensional output after the shared latent vector and uses an MSE loss on depth SH coefficients.
- Learning additional subtasks: A low-resolution decoder decomposes the local patch into albedo and shading to help disambiguate reflectance from illumination.The decoder reconstructs 7 × 7 color albedo and grayscale shading images from an intermediate feature representation.
- Training: Domain adaptation connects a discriminator to the latent vector through gradient reversal, adapting synthetic-image training to real photographs.The discriminator is trained with cross-entropy while the overall objective combines lighting, depth, reflectance/shading, and domain-adaptation losses.
- Training: Using both local information and the full image reduces synthetic-test lighting error, while auxiliary tasks improve complementary SH components.Depth supervision improves directional SH degrees, whereas albedo/shading supervision improves ambient-light estimation; all tasks achieve better results across degrees.
5. Experimental validation
The evaluation shows that combining global and local information, auxiliary subtasks, and domain adaptation improves spatially varying lighting estimation, with strong results on synthetic and real indoor data. The method outperforms prior approaches overall, although incorrect hue shifts remain a failure case.
- 5.1. Validation on synthetic data: Combining global and local paths lowers SH-coefficient MAE more than either global or local information alone.The local-only path outperforms the global image baseline, while the combined model performs best.
- 5.1. Validation on synthetic data: Combining depth and albedo/shading subtasks improves both directional and ambient SH predictions.Depth supervision mainly improves directional components, whereas albedo/shading decomposition improves degree-0 ambient lighting.
- 5.2. A dataset of real images and local light probes: The real-image benchmark contains 20 indoor scenes and 79 HDR light probes, supporting quantitative and perceptual evaluation.The dataset uses spatially varying probes captured at multiple locations in each scene.
- 5.3. Quantitative comparison on real photographs: Our method has lower relighting error than both versions of Gardner et al. across center and off-center probes, while global-[8] is slightly better on center-probe RMSE.Table 4 reports RMSE and si-RMSE with 95% confidence intervals; the method remains constant across probe types.
- 5.3. Quantitative comparison on real photographs: Users preferred our method overall, with 35.8% confusion versus 28% for local-[8] and 31% for global-[8], although global-[8] was slightly better on center probes.On off-center probes, our method maintained 34.5% confusion while global-[8] fell to 27.1%.
- 5.3. Quantitative comparison on real photographs: Incorrect hue shifts, typically purple or green, occur in some scenes and may reflect differences between synthetic training and real-test response distributions.The authors hypothesize that regularization or dataset enrichment could remedy this issue.
6. Real-time lighting estimation for AR
The system supports real-time augmented-reality relighting by estimating lighting independently at every frame. Demonstrations show relighting while either the virtual object or a scene light source moves.
- 6. Real-time lighting estimation for AR: The network estimates lighting in 20ms per image on an Nvidia 970M mobile GPU, enabling real-time relighting demonstrations.A virtual sphere is illuminated using the estimated SH lighting at every frame.
- 6. Real-time lighting estimation for AR: Relighting is performed independently at every frame without enforcing temporal consistency.The demonstrations therefore show framewise lighting adaptation rather than temporally stabilized predictions.
- 6. Real-time lighting estimation for AR: The demonstrations relight a virtual object as it is dragged through the scene or as a light source moves around a static object.The two scenarios illustrate spatially varying lighting changes in interactive use.
7. Conclusion and Future Work
The method provides real-time local-lighting prediction for indoor augmented reality and outperforms previous work on synthetic and real data. Future work targets temporal information and richer lighting representations for sharper shadows and reflection maps.
- The method predicts local indoor lighting in real time and is particularly suitable for augmented reality applications.
- Evaluations on synthetic and real data show that the method significantly outperforms previous work.
- Future work could use temporal image information and cross-frame scene aggregation to improve prediction accuracy.
- Alternative lighting representations may improve angular-frequency detail, enabling crisper shadows and eventually reflection maps.