PaperlayerView exact revision on arXiv

Source-linked AI summary

Shedding Light: A Benchmark for Evaluating Lighting Understanding in Generative Image Models

Justine Giroux, Jack Oliver Hilliard, Yannick Hold-Geoffroy, Javier Vazquez-Corral, Jean-François Lalonde

arXiv:2609.10787v1cs.CV

TL;DR

Generative models’ physical understanding of illumination remains insufficiently characterized despite advances in visual realism. This paper introduces a light-probe benchmark that inpaints objects into real scenes and compares their estimated lighting with ground truth, finding persistent physical inaccuracies alongside perceptually convincing results.

  • Problem

    There is no standardized quantitative framework for determining whether generative models reproduce illumination physically correctly, even though lighting is central to realistic image synthesis and scene understanding.

  • Method

    The benchmark masks real light probes, prompts models to inpaint diffuse spheres, and compares estimated direction, colour, and radiance distributions with ground-truth probes.

  • Results

    Lighting accuracy has shown little improvement over time; closed models lack a consistent advantage, editing-specific models do not consistently outperform general inpainting models, and scale alone is unreliable.

  • Takeaways & Limitations

    Most models reproduce spatially varying lighting beyond local harmonisation, while their errors are often below human perceptual detection thresholds.

  • Takeaways & Limitations

    The benchmark is limited to indoor environments with a single point light source, and the probe analysis has reliability limitations for front/back lighting coefficients.

Abstract

from arXiv · show

Accurate modelling of illumination is central to realistic image synthesis and scene understanding. Yet, there is little exploration into whether image generative models are good at this task or whether physical plausibility remains a key challenge for them. Clearly, significant progress has been made in realistic image synthesis, but do models truly understand lighting in a physically accurate manner? To answer this question, this work proposes a benchmark to assess the lighting understanding and harmonisation capabilities of generative models. Our key insight is that evaluating lighting understanding for such models only requires testing how well they insert novel objects into real photographs whilst maintaining consistent illumination. To do so, we use a multi-illumination dataset with images containing simple objects serving as ``light probes'', and prompt models to inpaint the same object onto the original image, then compare the generated results against the ground-truth light probes. We then estimate the lighting direction, colour and radiance distribution from the inpainted probes, providing a quantitative measure of illumination accuracy and photometric realism. Our work establishes a scalable evaluation protocol to systematically assess how well generative models capture and reproduce real-world lighting, offering a foundation for benchmarking the photometric accuracy of any future models. All code and data are available at https://lvsn.github.io/SheddingLight/ .

1 Introduction

The paper addresses the lack of standardized quantitative evaluation for whether generative image models understand illumination physically. It introduces a benchmark that tests lighting consistency by inserting light probes into real scenes and comparing them with ground-truth probes.

  • Generative models produce realistic imagery, but their ability to reason about illumination remains unclear despite lighting’s importance for geometry, materials, and scene realism.
  • Illumination evaluation is difficult because lighting is entangled with scene geometry, material reflectance, exposure, and semantics.
  • Existing evaluations are fragmented across VLM judgments, environment-map supervision, and manual or domain-specific assessments, leaving no standardized quantitative framework for illumination consistency.
  • The benchmark prompts models to insert simple objects into real images, using the generated objects to infer and quantitatively assess scene illumination while isolating it from geometry, reflectance, and semantics.
  • Across evaluated models, lighting accuracy has not improved over time; models favor front-facing light, while most still reproduce spatially varying illumination beyond local harmonization.

2 Related work

Prior work evaluates image quality, physical understanding, relighting, lighting estimation, and image editing with metrics or judgments that often do not directly measure physically correct illumination. This paper builds on light-probe evaluation while targeting scalable, quantitative benchmarking for generative image editing.

  • Standard full-reference image metrics require pixel-aligned generated and reference images, while distribution metrics compare sets of images and do not directly establish lighting correctness.
  • Image-editing research progressed from classical structure and texture propagation to learned generative priors and modern generative editing methods.
  • Most image-generation benchmarks emphasize text-image alignment or broad image quality, with relatively few targeting physical accuracy and lighting specifically.
  • Existing lighting evaluations include VLM scoring, environment-map supervision, and manual inspection, but respectively sacrifice direct physical measurement, require unavailable supervision, or limit scalability.
  • Single-image lighting estimation commonly compares rendered spheres against reference spheres using image-quality measures, motivating this paper’s use of an inpainted diffuse probe compared with a ground-truth probe.
  • Unlike dedicated relighting methods that explicitly control illumination, this work tests whether general-purpose editing models handle lighting correctly without task-specific training.

3 Methodology

The methodology replaces real light probes with masked regions, asks generative models to inpaint diffuse spheres, extracts the generated probes, and estimates lighting parameters for comparison with the originals. The benchmark uses calibrated multi-illumination scenes and evaluates direction, colour, and radiance-related consistency.

  • The controlled proxy fixes scene geometry and reflectance while testing whether models can infer and reproduce the scene’s illumination through object inpainting.
  • The pipeline uses diffuse grey spheres as light probes, because their Lambertian shading reveals incoming light directions across the visible hemisphere.
  • Models receive probe-masked images and a prompt to insert two matte spheres; generated probes are detected, extracted, and compared with real probes using inverse rendering and several measures.
  • The Multi-Illumination dataset contains 23 023 filtered images across 1006 indoor scenes and 24 lighting directions, with diffuse probes used as lighting measurements.
  • The method cannot reliably disentangle light intensity from sphere albedo, so it focuses on dominant light direction and colour, with colour measured by averaging probe pixels in CIE Lab space.
  • Dominant light direction is estimated by fitting a Cook-Torrance sphere under directional and ambient lighting, minimizing pixelwise L1 distance with heuristically initialized parameters.
  • The evaluation computes angular error between ground-truth and generated light directions and ΔE_ab colour difference between the corresponding probes.

4 Benchmarking generative models for lighting accuracy

Across evaluated models, lighting accuracy shows no clear improvement with release date, model size, openness, or editing task. Directional and spatial analyses reveal systematic biases, while spherical-harmonic statistics broadly align with ground truth but expose representation limits.

  • 4.2.1 Light direction angular error distribution.: 11.38° and 11.92° median angular errors make FLUX.1 Fill [dev] and Z-Image Turbo the most accurate models, while SD 3.5 reaches 34.84°.Hunyuan 2.1 and HiDream also have higher errors at 28.17° and 24.77°, respectively.
  • 4.2.1 Light direction angular error distribution.: Lighting accuracy shows no clear improvement over time, and later generations within the same model family do not consistently outperform earlier ones.FLUX.1 Fill [dev], released in November 2024, remains the most accurate model for lighting direction.
  • 4.2.1 Light direction angular error distribution.: Closed models do not outperform open models, while inpainting and editing have similar median angular errors of 16.51° and 15.74°, respectively.The reported task difference is only 0.77°, indicating no clear correlation between task type and light-direction accuracy.
  • 4.2.2 Angular error distribution per ground-truth light direction.: Models struggle most with top-back lighting, whose average median error is 41.94°, compared with 11.51° for front and 11.90° for top lighting.Left-right directions are handled relatively symmetrically, with average medians of 22.70° for left and 19.96° for right.
  • 4.2.3 Azimuth & elevation.: Regression slopes above one—1.79 for azimuth and 1.55 for elevation—indicate frontal and lower-elevation prediction biases.Some models exhibit an S-shaped azimuth response, whereas top-performing models better span wider azimuth ranges.
  • 4.2.4 Spatially varying lighting.: Spatial-lighting discrepancies are at or below 5° for 12/16 models, but SD 3.5 exaggerates variation by 13.3° whereas SD 2.0 Inpaint underestimates it by −7.3°.The authors hypothesise that these opposite failures reflect differences in how the architectures propagate spatial context.
  • 4.2.5 Spherical harmonics coefficients.: Relative spherical-harmonic medians generally align with ground truth, but C0/C1 ambiguity and hemispherical crosstalk limit interpretation of individual coefficients.The C3 azimuth component stays near zero, while C2 tends positive; second-order agreement does not necessarily coincide with accurate dominant-light direction.

5 Discussion

The discussion distinguishes local harmonisation from broader lighting understanding, while identifying measurement and survivorship biases and a restricted evaluation scope.

  • Lighting understanding: Models reproduce spatially varying light differences across image locations, which requires consistent scene-level lighting beyond local colour matching.The analysis tests this at two scene locations.
  • Measurement error: The inverse-rendering pipeline achieves 2.55° median error on synthetic PBR data and 9.35° against real-probe specular-highlight detection, treated as an upper bound on measurement error.Unknown probe materials and real-image artefacts complicate exact error quantification.
  • Evaluation bias: Lower usable-probe rates tend to accompany higher median angular errors, biasing evaluation against models that generate fewer valid probes.The authors associate this pattern with difficulty in inpainting probes successfully.
  • Scope: The benchmark assumes a single dominant indoor light source and therefore does not establish performance for outdoor scenes or multiple lighting sources.Spatially varying lighting is evaluated at only two locations.
  • Scope: The simple grey-sphere calibration object and model-specific prompt differences limit conclusions about complex materials, geometry, and universally optimal prompting.Per-model tuning could improve results, but the study uses a simple sphere as a proxy.

6 Conclusion

The paper presents a physically grounded benchmark that measures lighting accuracy through calibrated probes and complementary photometric metrics. Its findings indicate persistent gaps between perceptual realism and physical correctness, motivating diagnostic evaluation for future models.

  • Benchmark: The benchmark isolates lighting inference from geometry, material properties, and scene semantics using calibrated light probes in indoor scenes.It uses generative inpainting to assess photometric consistency quantitatively.
  • Metrics: The benchmark measures light-direction angular error, ΔEab colour error, and radiance-distribution divergence.These metrics provide complementary views of lighting accuracy.
  • Findings: Lighting accuracy shows little improvement over time; closed models lack a consistent advantage, editing models do not outperform general inpainting, and scale alone is unreliable.The conclusion reports these patterns across the evaluated models.
  • Perceptual versus physical accuracy: Most predicted errors fall below typical human perceptual thresholds, suggesting models can produce convincing lighting without physically accurate light estimation.The authors contrast perceptual plausibility with physical correctness.
  • Implications: Physical inaccuracies may matter for applications such as autonomous-vehicle training data and physically grounded simulations.The authors identify these as settings where perceptual realism may be insufficient.
  • Future use: The benchmark is intended as a diagnostic tool for studying how generative models represent and reproduce illumination.The authors propose it as a foundation for further work on physically accurate lighting.

A.1 Dataset: filtering and processing

The dataset-processing pipeline filters unsuitable light-probe images, normalises probe geometry and masks, and evaluates model outputs for usability and robustness across configurations.

  • Probe detection: Generated probes are detected with SAM3, and sphere masks are smoothed by fitting an ellipse.The pipeline uses separate prompts for matte and mirror-reflective spheres.
  • Dataset filtering: The dataset removes images with multiple illumination sources, overexposure, noise, or other probe-quality problems.The filtering removed 2052 images.
  • Probe processing: Ellipse fitting and perspective warping rectify spheres to circular form while retaining their surface normals.This preprocessing supports subsequent lighting estimation.
  • Experimental configuration: The evaluation uses default ComfyUI workflows and selected seeds, with no significant seed differences reported for the tested open models.Closed-model APIs were not tested across multiple seeds.
  • Configuration sensitivity: Classifier-free guidance tuning is mostly consistent with recommended model settings, while marginal alternatives can reduce usable-probe yield.FLUX.1 Fill performs marginally better at CFG 3, whereas Z-Image Turbo trades lower error for lower yield.
  • Compute cost: The full dataset-generation pipeline requires approximately 2395.8 H100-equivalent GPU hours, or about 100 H100-equivalent days.The estimate excludes some API-model inpainting usage and may vary with hardware and optimisation.
  • Usable-probe yield: Qwen Edit produces high-quality probes when successful but often fails to place them at the requested position, while several other model families have low success rates.Usable-probe yield is reported as a percentage of inferred images.
  • Failure cases: Models frequently generate unusable probes, including noisy outputs, multiple probes, misplaced probes, scene-embedded probes, or probes blended into the surrounding image.These failure modes vary across Hunyuan 2.1, Qwen Edit, HiDream, and SD 3.5.

B.1 Light parameter optimisation evaluation

The light-parameter optimiser is evaluated on synthetic spheres with known lighting and on real Multi-Illumination probes, using angular-error comparisons against reference labels.

  • Synthetic evaluation: The synthetic PBR dataset varies sphere materials and single-light directions, providing ground-truth labels for light-direction estimation.Evaluation covers both HDR and LDR inputs to match the benchmark and model outputs.
  • Real-probe evaluation: The optimiser is also evaluated on Multi-Illumination probes against specular-highlight and human-annotated light-direction labels.The accompanying table reports median and standard-deviation angular errors.

B.2 Probe generation evaluation

The benchmark evaluates whether models can place usable light probes while reproducing scene illumination. Prompt wording generally does not change light-direction accuracy, although probe placement reliability varies across models.

  • Probe usability: Qwen Edit generated high-quality probes ranking sixth in median angular error but often misplaced them or placed them partly outside the image.Its difficulty adhering to the requested probe position reduced usable probe yield.
  • Prompt influence: The prompt study compares simple, medium, complex, and task-specific instructions across FLUX.1 Fill [dev], Qwen Edit, Z-Image Turbo, and Kolors on 55 scenes.The prompts range from naming two grey spheres to specifying geometry, materials, visibility, placement, and lighting consistency.
  • Evaluation setup: Probe-generation analysis uses a PBR test set spanning probe materials, light directions, and intensities to estimate inverse-rendering uncertainty.Generated examples include diffuse and chrome probes overlaid on reference images across models.
  • Probe usability: The evaluation reports usable probes as a percentage of inferred images and compares model-level probe generation outcomes.The accompanying visual summarizes generated images, generated probes, and usable probes for each model.
  • Prompt influence: No statistically significant differences in light-direction accuracy appeared across prompts, although usable probe yield varied for some models.The authors therefore retained their robust prompt, which includes probe-location information needed by editing models.

B.3 Model size analysis

Model scale and release date do not predict light-direction accuracy. The best and worst models have similar parameter counts, and performance shows no discernible temporal improvement.

  • Parameter count: No clear correlation exists between parameter count and median angular light-direction error; FLUX.1 Fill [dev] performs best while SD 3.5 performs worst despite similar sizes.The comparison uses model-level median angular error against parameter count.
  • Release date: No discernible trend links model release date to median light-direction error, so newer models and larger parameter counts do not predict accuracy.The analysis covers quality-filtered probes and compares error with release date.

B.4 Spatially varying lighting analysis

Models differ in how much spatial light-direction variation they reproduce between probes. This statistic reveals distributional tendencies but does not directly establish scene-level directional accuracy.

  • Spatial variation: The spatial angle-difference measure compares probe-direction distributions across models with the ground-truth distribution.Models are organized by release date and colored by editing type and availability in the figure.
  • Spatial variation: Some models predict too little within-image variation, while others vary probe light directions more than expected.SD 2.0 Inpaint and FLUX.2 [klein] 9B tend toward similar directions, whereas SD 3.5 varies more than expected.
  • Interpretation: Matching the ground-truth angle difference does not guarantee the correct dominant light direction for a specific scene.The measure captures statistical spatial trends rather than direct scene-level accuracy.
  • Probe scale: The analysis also examines whether probe size within the image affects angular-error distributions and artifacts from lower-resolution inpainting.Ground-truth probe distances vary across scenes, motivating comparisons by relative probe size.

B.6 Angular error distribution

Angular-error distributions vary by ground-truth light direction. The top-back direction is consistently harder, while front and top directions are similar for most models without a significant left-right bias.

  • Directional clusters: The top-back light-direction cluster is considerably harder for all models than the other evaluated directions.This directional difficulty matches the global trends seen in the aggregated analysis.
  • Directional clusters: Front and top direction clusters have similar performance for most models, with no significant bias toward either the left or right side.The per-model distributions preserve the same broad directional patterns as the aggregated results.

B.7 Colour metrics

The benchmark supplements chroma-based colour error with ΔE CIE2000 and hue angular error, while accounting for their respective sensitivities. It also examines whether quality filtering creates survivorship bias in colour accuracy.

  • B.7 Colour metrics: FLUX.1 Fill [dev] and SD 1.0 Inpaint remain among the best methods under ΔE CIE2000, while HiDream and Nano Banana 1 are more strongly penalised.ΔE CIE2000 includes luminance, which can reflect albedo differences because the prompt specifies a grey sphere.
  • B.7 Colour metrics: ΔE CIE2000 captures luminance and chrominance differences, but luminance errors may arise from albedo rather than illumination colour alone.
  • B.7 Colour metrics: FLUX.1 Fill [dev] achieves the best performance on hue angular error, highlighting robustness across multiple colour measures.Hue angular error is independent of luminance and saturation but unstable near the achromatic axis.
  • B.7 Colour metrics: Hue angular error can be large despite negligible perceptual differences when illumination colour is weakly pronounced against the sphere’s original grey colour.
  • B.7 Colour metrics: Models with fewer valid probes do not have better colour accuracy than models with more valid probes, according to the quality-filtering analysis.
Loading 2609.10787v1…