LightIt: Illumination Modeling and Control for Diffusion Models

• Diffusion methods lack consistent, explicitly controllable lighting, leaving artists reliant on uncertain prompt engineering.Source: §1 · p. 1
• Lighting datasets are difficult to obtain because scene appearance must be decomposed into lighting and material properties.Source: §1 · pp. 1–2
• LightIt predicts a 3D density field from one image, traces light rays for cast shadows, and combines them with normals to estimate single-bounce shading for arbitrary directions.Source: §1 · p. 2
• The resulting image-shading dataset conditions diffusion generation on shading and normals, while an image-and-target-shading module supports relighting.Source: §1 · p. 2

• ControlNet and T2I-Adapter [24] control several image modalities, but no approach explicitly controls generated lighting.Source: §2 · p. 2
• Relighting research includes image-based rendering, shape from shading, style transfer, image translation, and specialized methods using captured or paired illumination data.Source: §2 · p. 2
• Scene relighting methods use geometric and shading priors across multi-view, single-view, and outdoor NeRF settings.Source: §2 · p. 2
• LightIt combines volumetric scene representation and explicit shadow ray-marching with diffusion-based image generation.Source: §2 · p. 2

3. Method

• LightIt generates paired real images and shading maps, then trains a diffusion control module and reuses the dataset for relighting.Source: §3 · p. 2

3.1. Shading estimation

• Direct shading estimation predicts a density field from image features, ray-marches cast shadows, combines shadows with N-dot-L shading, and refines the result.Source: §3.1 · pp. 2–3, Figure 2 · p. 3
• The estimator is trained on synthetic image-shading pairs with L2 losses for shading and density-field depth, and can use N-dot-L shading when only depth is available.Source: §3.1 · p. 3

3.2. Lighting-conditioned diffusion

• Lighting-conditioned diffusion concatenates direct shading with normals and trains a control module alongside a reconstruction-supervised Residual Control Encoder and Decoder while keeping Stable Diffusion fixed.Source: §3.2 · p. 3, Figure 3 · p. 3

3.3. Relighting Application

• Relighting conditions diffusion on an input image and target shading, using predicted relit images as inputs while targeting real images to exploit Stable Diffusion’s image prior.Source: §3.3 · p. 3

3.4. Dataset

• The dataset uses Outdoor Laval panoramas, randomized crops and lighting conditions, estimated depth-derived normals, prompts, and 51250 image samples.Source: §3.4 · p. 4
• Depth-derived normals were more robust than directly estimated normals, and swapping in MiDaS produced similar generation results when plausible depth was available.Source: §3.4 · p. 4

4.1. Image Synthesis

• The synthesis pipeline conditions generation on shading maps and optionally normal maps, using a dataset generated from cropped panoramas with automatically predicted normals, shading, and captions.The relighting dataset is extended with OutCast-generated relit images.Source: §4.1 · p. 4, Figure 4 · p. 4, §4.2 · p. 5

4.1.1 Lighting Consistency.

• A 42-participant user study finds LightIt better follows desired lighting and is preferred over SD [34] for image quality and textural alignment.The comparison uses both test-set and in-the-wild images with predicted or manually prepared prompts.Source: §4.1.1 · pp. 4–5, Table 1 · p. 5

4.1.2 Lighting Controllability

• LightIt synthesizes images whose shading follows user-defined target lighting across test-set examples, diverse text prompts, styles, and lighting conditions.In-the-wild inputs and fully generated images are also evaluated, using automatically estimated normal and shading maps; identity preservation is not enforced.Source: §4.1.1 · p. 4, §4.1.2 · p. 5, Figure 6 · p. 6

4.2. Image Relighting

• The relighting model achieves consistent relighting with more realistic image quality than OutCast, while OutCast produces stronger shadows and slightly higher lighting perceptual scores.Evaluation covers PSNR, FID, I-PQ, and L-PQ, with a 17-person user study for perceptual quality.Source: Table 2 · p. 5, §4.2 · pp. 5–6

4.3.1 Image Synthesis

• Without cast shadows, N-dot-L conditioning captures overall lighting but produces less realistic shadows and weaker fine-grained shadow control.The direct-shading representation supplies cast-shadow information that the simpler conditioning lacks.Source: §4.3.1 · p. 6, Figure 10 · p. 8
• Direct shading achieves the best lighting quality, while adding normals improves normal consistency at minimal lighting-quality cost.Table 3 compares shading and normal control using PSNR and angular error.Source: Table 3 · p. 6
• Without normals, shadow regions lack geometric signal, causing random or flattened details; normal conditioning restores structures such as doors and windows.Normal conditioning improves consistency specifically in shadowed regions.Source: §4.3.1 · p. 6, Figure 11 · p. 8

4.3.2 Image Relighting

• The proposed Control Decoder and full residual architecture improve identity consistency over ControlNet, which can ignore control signals and alter colors or shadows.The full residual design achieves high consistency in the relighting examples.Source: §4.3.2 · pp. 6, 8, Figure 9 · p. 8
• The full relighting method produces more natural images and the best FID, although OutCast yields slightly more perceptually consistent shadows.Users rated the proposed method significantly more realistic overall, while OutCast generally produced stronger shadows.Source: §4.3.2 · p. 8, Figure 8 · p. 8

4.4. Limitations and Future Work

• The method assumes directional lighting, which suits outdoor scenes, while point and other light sources remain future extensions.The shading estimator can trace rays in arbitrary directions, but adapting the system to other source types is left for future work.Source: §4.4 · p. 8

5. Conclusion

• LightIt explicitly controls diffusion-model illumination using direct shading that contains both shading and shadow information.The shading map can be computed from an existing picture or a generated image.Source: §5 · p. 8
• The method achieves high-quality image generation while maintaining user-defined lighting.Source: §5 · p. 8
• The approach is presented as a way to increase the editability of diffusion-based generative imaging.Source: §5 · p. 8

A. Method Details

• The paper states that additional method details follow in subsequent sections.Source: A. Method Details · p. 12

A.1. Dataset Generation

• The dataset starts from 205 HDR panorama images in the Outdoor Laval dataset, whose full panoramas provide dominant-light-direction information.Source: §A.1 · p. 12
• Each panorama produces 250 virtual-camera images with varied intrinsic and extrinsic parameters.Source: §A.1 · p. 12
• Rendered images are normalized to mean intensity 0.5 and converted to LDR sRGB using gamma correction γ = 2.2 with clipping to [0, 1).Source: §A.1 · p. 12

A.2. Lighting Control

• The full control module combines a Residual Control Encoder, Lighting Control module, and training-only Residual Control Decoder.The modules contain approximately 2.9M, 363M, and 1.3M parameters, respectively.Source: §A.2 · p. 12
• The Residual Control Encoder maps 512 × 512 × 3 inputs to 64 × 64 × 320 features through seven residual blocks.It uses zero convolution at its input, following the cited diffusion-control architecture.Source: §A.2 · p. 12
• During training, the Residual Control Decoder reconstructs the control signal, improving control consistency by preserving information in the latent representation.The decoder mirrors the encoder with seven residual blocks in transposed order.Source: §A.2 · p. 12
• The Lighting Control module converts encoded control signals into intermediate and encoded diffusion features that guide the diffusion process.It follows the architecture of the diffusion UNet encoder.Source: §A.2 · p. 12
• Images are cropped from the referenced dataset using parameters specified in degrees.The supplied table identifies the cropping-parameter scope but does not provide the parameter values.Source: Table 4 · p. 12

B. Experiment Details

• A user study perceptually evaluates the real-world predictions produced for lighting consistency.Samples and study results are provided in the supplementary generation_results.html.Source: B. Experiment Details · p. 12
• Lighting controllability is tested on both in-the-wild images and SDXL-generated images across multiple lighting conditions with a fixed seed.The evaluation also shows original images, estimated normals, and text prompts, while SDXL samples use the same prompts.Source: B. Experiment Details · p. 12
• A second user study perceptually evaluates the real-world predictions of the relighting application.Samples and study results are provided in the supplementary relighting_results.html.Source: B. Experiment Details · p. 12

C. Additional Results

• Direct shading includes cast-shadow information, which provides valuable guidance because Stable Diffusion does not model light transport.Additional samples for this lighting-representation ablation are shown in Figure 12.Source: C. Additional Results · p. 12, Figure 12 · p. 13
• Out-of-domain synthesis results pair each original input image with estimated normals and the text prompts used for generation.These details correspond to the results shown in Figure 7.Source: Figure 13 · p. 13
• Control robustness is evaluated on generated samples by reconstructing the original control while progressively blurring the input.Figure 14 reports the test-set average and shows increasingly blurred input in the top-left insets.Source: Figure 14 · p. 13