PaperlayerView exact revision on arXiv

Source-linked AI summary

RoomLight: A 2.5D Illumination Prior for Indoor Environments

Andreea Ardelean, Bernhard Egger

arXiv:2609.28300v1cs.CV

TL;DR

Indoor illumination recovery from images is ill-posed, and environment maps cannot represent the spatial variation of real interiors. RoomLight learns a prior over paired HDR radiance and depth, decoding it into a differentiable 2.5D area light; the authors report more faithful recovery than environment-map priors fitted through the same renderer.

  • Problem

    Environment-map priors represent illumination as direction-only, while indoor lighting varies spatially with emitter distance and visibility.

    Source: §1 · pp. 1–2

  • Method

    A VAE decodes a latent code into paired HDR radiance and depth, which form a 2.5D area light for optimization through a differentiable renderer.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    RoomLight recovers indoor illumination more faithfully than environment-map priors fitted through the same renderer across diverse scenes and materials.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The finite emitter extent models position-dependent illumination, while the learned decoder keeps latent optimization on the manifold of plausible rooms.

    Source: §5 · p. 8

    Depth discontinuities are bridged by mesh triangles, emitters among objects cannot be properly expressed, and fitting requires scene points for a repulsion-based clearance term.

    Source: §5 · p. 8

Abstract

from arXiv · show

Ill-posed inverse problems require priors to constrain the solution space toward plausible outcomes. In inverse rendering, learned priors modeling the distribution of natural illumination improve the recovery of scene properties. However, existing models rely on the distant-illumination assumption, representing lighting as a far-field environment map. This limits their applicability to indoor scenes, where illumination is highly spatially varying due to finite-distance emitters, visibility changes, and parallax, all of which are poorly approximated by a single environment map. To address this, we introduce a spatially-aware illumination prior trained on real-world indoor panoramas and their estimated depth. Our variational autoencoder model learns a compact, optimizable latent space that decodes into HDR radiance and depth, parameterizing an area light emitter for direct integration into standard differentiable rendering pipelines. This design bridges the plausibility guarantees of a learned prior with the gradient flow required for downstream optimization. Crucially, by jointly modeling radiance and depth, our prior captures the spatial structure of indoor illumination, instead of treating the light sources as infinitely distant. We demonstrate that this formulation enables spatially-varying illumination modeling and achieves higher-fidelity recovery of indoor lighting compared to existing approaches. Project page: https://andreead-a.github.io/RoomLight

1. Introduction

Indoor illumination recovery is ill-posed, and the usual environment-map representation cannot capture the spatial variation of real interiors. RoomLight represents lighting as a depth-based 2.5D area light and learns a prior that supports plausible, differentiable recovery.

1. Introduction

  • Recovering illumination from limited-field-of-view images is ill-posed because geometry, reflectance, and lighting are coupled, while incoming light spans many directions and HDR intensities.Source: §1 · p. 1
  • A single environment map cannot match indoor lighting that varies with position because nearby emitters create parallax, inverse-square falloff, and visibility changes.Environment maps encode incident radiance by direction at one point, an assumption more appropriate for outdoor daylight scenes.Source: §1 · p. 1
  • Parallax-corrected cubemaps use proxy geometry to account for finite distance, but such proxies cannot represent real interior layouts.Source: §1 · p. 1
  • A depth-based mesh textured with an HDR panorama forms a 2.5D area light that can reproduce nearby objects’ differing appearances.The representation treats the emitter as a surface in space, so illumination varies with position and direction.Source: Figure 1 · p. 1, §1 · p. 2
  • Existing learned illumination priors constrain optimization to plausible latent samples but decode direction-only radiance, leaving emitter distance unrepresented.Directly recovering a 2.5D area light from one image is even more underconstrained because most of its radiance and geometry are unobserved.Source: §1 · p. 2
  • RoomLight trains a VAE on paired HDR radiance and depth panoramas so one latent code decodes a differentiable 2.5D light for renderer-based recovery.The authors report more accurate spatially varying illumination recovery than environment-map priors fitted through the same renderer, with transfer to real photographs.Source: §1 · p. 2

2. Related work

Prior work parameterizes illumination with environment maps, learned priors, image-space predictions, or localized and volumetric representations. RoomLight’s overview places a learned, spatially varying light representation within differentiable inverse-rendering optimization.

2. Related work

  • Inverse rendering recovers intrinsic scene properties from images, including geometry, material reflectance, and incident illumination.Source: §2 · p. 2
  • Environment-map methods range from compact SH and SG bases to expressive pixel grids, while RoomLight targets an expressive, low-dimensional representation for differentiable optimization.Compact bases have limited angular bandwidth for sharp specular reflections; unconstrained pixel grids are high-dimensional and underconstrained.Source: §2 · p. 2
  • Learned illumination priors include statistical lighting models, autoencoders, neural-field decoders, large-scale VAEs, and diffusion models, but produce a single global estimate.Source: §2 · p. 2
  • Spatially varying approaches predict image-space lighting, localized 3D sources, voxel-grid volumetric SH, or illumination from probes and video, with differing structural and detail constraints.Image-space methods do not enforce coherent 3D lighting; localized sources may lack reflection detail.Source: §2 · pp. 2–3
  • The method overview shows paired HDR radiance-depth panoramas encoded as a global latent, then decoded into a 2.5D light for differentiable rendering.For recovery, the encoder is discarded and the frozen decoder’s output is optimized through photometric loss to update the latent code and exposure scale.Source: Figure 2 · p. 3
  • Panorama outpainting followed by depth-based emissive-mesh conversion can estimate illumination, but does not provide a learned prior for optimization-based inverse rendering.The cited discussion notes that this limits joint recovery of other scene properties such as materials.Source: §2 · p. 3

3. Method

The method learns a compact indoor-lighting prior that decodes HDR radiance and depth into a spatially varying emitter, then optimizes its latent representation through differentiable rendering.

3. Method

  • A VAE jointly decodes panorama radiance and depth from a global latent code, enabling optimization of plausible indoor illumination through a differentiable renderer.The decoded pair is lifted into a 2.5D area light used as an emitter.Source: §3 · p. 3

3.1. Emitter construction

  • Equirectangular radiance and aligned depth define a textured 3D mesh whose differentiable construction lets rendered-image gradients reach both depth-driven positions and radiance.The image coordinates map to spherical angles and corresponding ray directions, which are scaled by depth to obtain mesh vertices.Source: §3.1 · p. 3

3.2. Panoramic convolutional VAE

  • A convolutional VAE compresses joint radiance-depth panoramas into a global 512-dimensional latent and adapts decoding to spherical topology with wraparound padding and latitude channels.Adaptive Group Normalization modulates decoder features across scales and improves gradient flow during latent optimization.Source: §3.2 · pp. 3–4

3.3. Training objective

  • Training combines exposure-calibrated absolute reconstruction, multiscale Scharr-gradient losses, a brightest-0.5%-pixel log-space loss, and KL regularization.The Laval Indoor HDR panoramas are inpainted at the camera blind spot and paired with estimated depth for training.Source: §3.3 · pp. 4–5, Figure 4 · p. 5

3.4. Prior-guided illumination recovery

  • With the decoder fixed, recovery optimizes latent code and exposure against a censored LDR render, while a repulsion penalty discourages emitter-scene collisions.The setup assumes known object geometry, materials, and poses under a calibrated camera; saturated pixels constrain illumination from below.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate the prior as a generative model, in synthetic inverse rendering, and on real photographs. Across these settings, results show the prior’s reconstruction performance and position-dependent illumination modeling.

4. Experiments

  • The illumination prior is trained on 2,233 Laval Indoor HDR panoramas augmented with pseudoground-truth depth from a monocular estimator, and compared with direct optimization and single-image illumination estimators.Synthetic inverse-rendering evaluation covers 25 indoor panoramas, three materials, and probe-sphere PSNR, RMSE, and angular-error metrics. Real-image fitting minimizes photometric loss over uniform object materials, exposure scale s, and prior latent code z.Source: §4 · p. 5, §4.2 · p. 7, §4.3 · p. 8

4.1. Illumination prior evaluation

  • In direct panorama evaluation, the model outperforms VENI in both autoencoder reconstruction and decoder-only latent optimization, while generated samples depict coherent rooms with diverse layouts and light sources.The evaluation uses 25 real indoor HDR panoramas with pseudo-ground-truth depth; Smoothness scores reflect imposed smoothing rather than representation quality.Source: §4.1 · pp. 6–7, Table 1 · p. 7

4.2. Inverse rendering on synthetic scenes

  • The 2.5D area light performs best on probe-sphere evaluation overall, while the prior leads on almost every case and metric except angular error for diffuse and mirror probes.The evaluation averages results across three materials and 25 synthetic scenes; angular error is scale-free, while RMSE and PSNR reflect intensity differences.Source: §4.2 · p. 7, Table 2 · p. 8
  • On synthetic scenes, the prior produces more realistic radiance details than direct optimization, and its decoded depth resembles the reference layout unlike the smoothness prior’s low-frequency field.The comparison is qualitative; direct optimization reduces diffuse-case radiance to a few bright blobs.Source: §4.2 · pp. 7–8
  • A fair comparison with DepthLight and 4D Lighting is not possible in the synthetic setting because they require a representative background masked during optimization.Those optimization-based methods would also have an unfair advantage with ground-truth object geometry and materials.Source: §4.2 · p. 8

4.3. Application on real images

  • On real photographs, the 2.5D area light produces position-dependent probe appearances that reflect the room layout, unlike environment-map methods whose probes see the same radiance.The setting also shows the prior remains usable when geometry and materials are estimated, though it is a simplified real-world evaluation.Source: §4.3 · p. 8

5. Discussion

The paper introduces a learned prior that models indoor illumination as a 2.5D area light and reports more faithful recovery than environment-map priors. It also identifies representational and fitting limitations.

5. Discussion

  • A single radiance and distance per direction can bridge depth discontinuities and cannot properly express emitters located among scene objects.The fitting also relies on a repulsion term and scene points to keep the emitter from intersecting scene content.Source: §5 · p. 8
  • The learned prior recovers indoor illumination more faithfully than environment-map priors fitted through the same renderer across diverse scenes and materials.A latent code decodes paired HDR radiance and radial depth into an emissive mesh for differentiable rendering.Source: §5 · p. 8

Acknowledgement

  • The authors acknowledge feedback from Paul Himmler and Timotei Ardelean and support from NHR@FAU for HPC resources.NHR funding is provided by federal and Bavarian state authorities.Source: §5 · p. 9

A. Representation motivation

The section compares 2.5D area lights with environment maps, showing the benefits of finite-distance illumination and the approximation limits of a single 2.5D emitter.

Representation motivation

  • Across three panorama capture locations—the centers of both objects and their midpoint—the environment map correctly relights only its capture-location object, while the 2.5D area light reproduces both objects at every capture point.Source: A. Representation motivation · p. 12

Capacity of a single 2.5D area light

  • 2× to 5× lower RMSE than the environment map for every material, though the error remains above the renderer noise floor.The comparison uses three probe spheres and three capture viewpoints; the remaining gap bounds what a single 2.5D area light can recover.Source: A. Representation motivation · p. 12, Table 3 · p. 12
  • A single 2.5D area light captures one depth per direction, omitting hidden surfaces and bridging depth discontinuities with invalid emitting triangles.The passage identifies this representational mismatch as a limit on approximation fidelity.Source: A. Representation motivation · p. 12

B.1. Training data preprocessing

The appendix describes panorama preprocessing and a synthetic-scene evaluation setup designed to compare recovered illumination against an exact solution while retaining real illumination diversity.

B.1. Training data preprocessing

  • Nadir regions are inpainted by tone-mapping HDR to LDR, completing a gnomonic view with LaMa, then restoring HDR values.The completed nadir region is backprojected through the inverse gnomonic mapping; the rest of the panorama remains unchanged.Source: §B.1 · p. 12
  • MoGe-2 depth is estimated from 12 overlapping perspective views, projected back to the equirectangular panorama, and fused with global consistency and spherical pole constraints.The authors use MoGe-2 instead of DA360, whose predictions can assign extreme depths to transparent window regions.Source: §B.1 · p. 12

B.2. Synthetic scene design

  • Each of 25 evaluation panoramas and its estimated depth are resampled and lifted into a 2.5D area light used to illuminate fitted and evaluation views.Every method is scored against renders produced with that emitter.Source: §B.2 · p. 12
  • The synthetic scenes place three objects at stratified depths in non-overlapping screen-space boxes, viewed by a perspective camera at the panorama capture point.The bunny and armadillo models come from the Stanford 3D Scanning Repository.Source: §B.2 · p. 12
  • Synthetic ground truth uses 2.5D area lights because diverse measured 3D indoor emitters are unavailable, and this setup provides an exact solution for scoring.The protocol also extends prior panorama-lit object evaluations by assigning the panorama finite distance.Source: §B.2 · pp. 12–13

B.3. Training details

The appendix specifies model training, fitting procedures, and baseline details, including the conditions used for synthetic and real-image evaluation.

B.3. Training details

  • The encoder produces a 512-dimensional posterior, and the decoder mirrors its convolutional structure with latent injection through AdaGN.The encoder uses four downsampling stages and projects an 8 × 16 feature map to 32 channels before the posterior layer.Source: §B.3 · p. 13
  • The VAE trains for 300 epochs on 2,233 panoramas at 128 × 256 resolution, using AdamW and azimuthal-roll, mirror, and camera-tilt augmentations.The learning rate warms up to 2 × 10−4 and then follows a cosine schedule to 10−5.Source: §B.3 · p. 13

B.4. Fitting details

  • For direct evaluation, latent fitting minimizes squared error over four normalized channels; depth-only fitting reaches AbsRel 0.019 versus 0.045 for joint fitting and 0.049 for the encoder.Depth receives one quarter of the joint objective because it is one channel among four.Source: §B.4 · p. 13
  • The 2.5D area light closely reproduces object relightings across panorama viewpoints, whereas environment-map lighting cannot reproduce both objects simultaneously.Figure 8 compares the relightings for different panorama viewpoints.Source: Figure 8 · p. 14
  • Real-image fitting uses MoGe-2 geometry, optimizes object material properties, and constrains the emitter using unmasked room observations and repulsion points.Optimization uses a Gaussian latent prior and initializes from a sampled latent with larger geometry to avoid small rooms decoded from zero.Source: §B.4 · pp. 14–15

B.5. Baselines details

  • DPI’s black nadir cap reflects its training data and confounds its performance comparison; masking the affected mirror-probe pixels reduces, but does not eliminate, its deficit relative to 2.5D methods.The masking analysis is reported only for the mirror probe because the affected region is well defined there.Source: §B.5 · p. 15

C.1. Fitting metrics

Environment-map methods fail to fit the observations even on the views they were optimized against. The supplementary comparisons also evaluate DPI’s unrepresentable probe region and show single-image estimators under a different input and loss setup.

C.1. Fitting metrics

  • Environment-map methods fail to fit the fitted-view observations, not just generalize poorly to held-out probes.The fitted-view metrics are computed on the bunny and armadillo used for optimization.Source: §C.1 · p. 15
  • Table 4 compares mirror-probe fits on the full probe with fits masked to exclude DPI’s unrepresentable nadir region.Excluding those pixels favors DPI; the red outline in Figure 10 marks the mask used for the masked evaluation.Source: Table 4 · p. 16, Figure 10 · p. 16

C.2. Single-image illumination estimators

  • DepthLight and 4D Lighting are evaluated on the same synthetic scenes, but their scores are not directly comparable to prior optimization.They predict from the photograph, including its background, while optimization-based methods render known geometry and materials with an object-masked loss; both estimators are run unmodified.Source: §C.2 · p. 16
  • Figure 11 shows the evaluation object re-rendered using DepthLight and 4D Lighting predictions from the fitted view, panorama, and depth where applicable.The figure complements Figure 5.Source: Figure 11 · p. 16

C.3. Ablations

The ablations show that individual objective and decoder components contribute differently to latent fitting and inverse rendering, while their removal accumulates. Replacing the prior produces a larger gap than removing any single component, and the 2.5D light increases fitting cost relative to an environment map.

C.3. Ablations

  • Removing AdaGN costs most in latent fitting, while removing latitude channels leaves latent fitting unchanged but degrades all three inverse-rendering metrics.The ablations evaluate direct reconstruction and downstream optimization separately; AdaGN routes the latent code to every decoder scale.Source: §C.3 · p. 16
  • The four ablations address distinct failure modes of a plain panorama-trained autoencoder: clipped highlights, blurred structure, weak global conditioning, and latitude-agnostic convolutions.The ablation set varies training objective and decoder conditioning while holding architecture fixed.Source: §C.3 · p. 17
  • VENI+D’s gap to the full model is larger than any single component ablation, isolating the prior as the difference while emitter, fitting procedure, and data are held fixed.The comparison is reported in Tables 1 and 2.Source: §C.3 · p. 17

C.4. Computational cost

  • Fitting through the 2.5D area-light prior takes significantly longer than through VENI because its gradients cover emitter geometry as well as texture.Path tracing, rather than decoder evaluation, dominates each inverse-rendering iteration.Source: §C.4 · p. 17

C.3. Ablations

  • The fully ablated model is worse than the full model on all eight reported metrics, with component effects accumulating almost additively.The ablations remove the peak loss, Scharr gradient terms, decoder AdaGN, and explicit latitude channels individually and retrain under otherwise identical configurations.Source: §C.3 · p. 16, Table 8 · p. 18

Related papers

Loading 2609.28300v1…