PaperlayerView exact revision on arXiv

Source-linked AI summary

DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation

Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, Xin Tong

arXiv:2402.11929v2cs.CVcs.GR

TL;DR

Text-driven diffusion models can generate images under varied lighting, but lighting is entangled with content and prompts provide limited fine-grained control. DiLightNet guides generation with coarse-shape radiance hints in a three-stage pipeline. The paper demonstrates the approach across prompts and lighting conditions, including point and environment lighting.

  • Problem

    Diffusion models exhibit lighting bias, and existing methods do not precisely control lighting independently of image content while text prompts struggle to express detailed lighting.

    Source: §1 · pp. 1–2

  • Method

    DiLightNet first starts from an uncontrolled provisional image, then resynthesizes the foreground using radiance hints from a coarse shape estimate and an encoded provisional image, before inpainting the background.

    Source: §1 · p. 2

  • Results

    The paper demonstrates lighting-controlled generation across varied prompts and lighting conditions, including point lights and environment lighting.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Coarse radiance hints can guide diffusion toward target lighting without requiring exact geometry for the hints.

    Source: §1 · p. 2

    Text prompts provide limited control over scene materials, so material-light interactions may not match the prompt engineer’s intention.

    Source: §8 · p. 11

Abstract

from arXiv · show

This paper presents a novel method for exerting fine-grained lighting control during text-driven diffusion-based image generation. While existing diffusion models already have the ability to generate images under any lighting condition, without additional guidance these models tend to correlate image content and lighting. Moreover, text prompts lack the necessary expressional power to describe detailed lighting setups. To provide the content creator with fine-grained control over the lighting during image generation, we augment the text-prompt with detailed lighting information in the form of radiance hints, i.e., visualizations of the scene geometry with a homogeneous canonical material under the target lighting. However, the scene geometry needed to produce the radiance hints is unknown. Our key observation is that we only need to guide the diffusion process, hence exact radiance hints are not necessary; we only need to point the diffusion model in the right direction. Based on this observation, we introduce a three stage method for controlling the lighting during image generation. In the first stage, we leverage a standard pretrained diffusion model to generate a provisional image under uncontrolled lighting. Next, in the second stage, we resynthesize and refine the foreground object in the generated image by passing the target lighting to a refined diffusion model, named DiLightNet, using radiance hints computed on a coarse shape of the foreground object inferred from the provisional image. To retain the texture details, we multiply the radiance hints with a neural encoding of the provisional synthesized image before passing it to DiLightNet. Finally, in the third stage, we resynthesize the background to be consistent with the lighting on the foreground object. We demonstrate and validate our lighting controlled diffusion model on a variety of text prompts and lighting conditions.

1. Introduction

Text-driven diffusion can generate detailed images, but its lighting is biased and entangled with image content, while text prompts provide only coarse lighting descriptions. The paper proposes radiance-hint guidance and a three-stage pipeline for fine-grained lighting control.

1. Introduction

  • Diffusion models exhibit lighting bias, and current methods do not precisely control lighting independently of image content.The paper notes that prompts struggle to express fine-grained lighting, and direct lighting conditioning does not resolve the entanglement.Source: §1 · pp. 1–2
  • Radiance hints visualize the target scene under target lighting with a homogeneous canonical material, but exact geometry is unavailable for text-driven generation.The method relies on coarse approximations because the diffusion model can fill in details while being guided toward the target lighting.Source: §1 · p. 2
  • The method first generates an uncontrolled provisional image, then resynthesizes the foreground with DiLightNet using radiance hints and an encoded provisional image, and finally inpaints the background.The image encoding is multiplied with the hints to retain texture information during resynthesis.Source: §1 · p. 2
  • The paper demonstrates lighting-controlled generation across varied text prompts and lighting conditions and reports an ablation study of DiLightNet components.The stated lighting range includes point lights and environment lighting.Source: §1 · p. 2

2. Related Work

Prior work provides semantic, spatial, material, and relighting controls, but the paper positions DiLightNet as controlling incident lighting directly during diffusion-based generation. It distinguishes its method from face-focused relighting and language-based lighting controls.

2. Related Work

  • Semantic-control methods span edits, subject identity, and material properties, but most offer mid- or high-level control; Alchemist also targets low-level shading properties.DiLightNet is presented as complementary to Alchemist: it controls incident lighting, while Alchemist controls relative material properties such as translucency and gloss.Source: §2 · p. 3
  • Spatial-control methods condition on sketches, depth, strokes, identity, or photo collections, but the paper states that they do not control incident lighting.DiLightNet injects radiance hints modulated by a neural-encoded image through ControlNet.Source: §2 · p. 3
  • Methods that generate 3D reflectance properties still require rendering to visualize appearance, while this method controls lighting during diffusion sampling.The comparison includes methods for viewpoint changes, 3D models, reflectance estimation, and intrinsic decomposition.Source: §2 · p. 3
  • Unlike single-image relighting methods that target particular scene types and captured photographs, this method targets general objects and generated images without explicit scene decomposition.It conditions diffusion on diffuse and specular radiance hints.Source: §2 · p. 3
  • Prior diffusion relighting methods described here focus on faces and require captured photographs, whereas DiLightNet targets general scenes and operates on text-prompt-generated images.The paper identifies multiple radiance hints and general shape and material variation as distinctions from those methods.Source: §2 · pp. 3–4
  • Lasagna controls lighting with language tokens and supports 12 predefined directional lights, while DiLightNet uses radiance hints and handles point and environmental lighting.The paper characterizes Lasagna's lighting control as less fine-grained.Source: §2 · p. 4

3. Overview

The method takes a text prompt, target lighting, and separate content and appearance seeds, assuming an isolated foreground object and a physically based image style. It generates a provisional image, relights the foreground with radiance hints, and inpaints the background.

3. Overview

  • The method assumes an isolated foreground object, with background content implicitly described by the target lighting, and an image style depicting physically based light-matter interactions.It makes no assumption on target lighting and supports arbitrary lighting conditions.Source: §3 · p. 4
  • The content-seed controls shape and texture variation, while the appearance-seed controls variation in light-material interactions.The inputs also include a text prompt describing image content and the target lighting.Source: §3 · p. 4
  • Stage one generates an uncontrolled-lighting provisional image; stage two synthesizes the foreground using radiance hints and DiLightNet; stage three inpaints the background for consistency with target lighting.The provisional image and content-seed are used in the first stage, while the appearance-seed is used in the second.Source: §3 · p. 4

4. Synthesis with Radiance Hints

The method guides a ControlNet-based diffusion model with radiance hints rendered from an estimated foreground shape, accepting coarse geometry because exact shape is unavailable. It encodes the provisional image and multiplies its features with the hints, and trains on synthetic objects with varied shapes, materials, and lighting.

4. Synthesis with Radiance Hints

  • Radiance hints encode each pixel’s outgoing radiance under target lighting because ControlNet requires a per-pixel control signal.A hint visualizes the target shape under target illumination with a homogeneous proxy material; the paper argues that a coarse shape estimate suffices.Source: §4 · p. 4, §4.1 · p. 4

4.1. Radiance Hint Generation

  • The foreground shape estimate comes from segmentation and monocular depth, then is triangulated and rendered with proxy materials; Laplace smoothing reduces high-frequency shading discontinuities.The paper reports that high-frequency shading errors can cause artifacts, whereas ControlNet is less sensitive to low-frequency shading errors.Source: §4.1 · p. 4
  • Four hints use one diffuse and three specular Disney BRDF materials, with roughness values 0.34, 0.13, and 0.05 for the specular materials.The paper uses separate hints to encode lighting effects at different frequencies.Source: §4.1 · p. 4

4.2. Lighting Conditioned ControlNet

  • The provisional image is encoded with its alpha channel set to the segmentation mask, and its 12-channel features are multiplied channel-wise with the four 3-channel hints before ControlNet.This encoding is intended to retain relevant shape and texture information while disentangling it from the provisional image’s unknown lighting.Source: §4.2 · p. 5, Figure 4 · p. 5

4.3. Training

  • Training uses synthetic objects because lighting, geometry, and material distributions can be precisely controlled, with dataset augmentation addressing limited spatially varying material diversity.The dataset includes additional objects assigned spatially varying materials from INRIA-Highres SVBRDF.Source: §4.3 · p. 5
  • The training set varies five lighting categories and uses 12 lighting conditions per object, while radiance hints approximate local shading to limit the training–evaluation domain gap.Training randomly uses geometric normals or normals from smoothed depth estimates; input images omit colored environment lighting, while output images may include it.Source: §4.3 · p. 6

5. Background Inpainting

Background handling depends on lighting specification: environment maps provide a rendered background, while other conditions use diffusion-based inpainting.

5. Background Inpainting

  • For environment-map lighting, the background is rendered with the radiance-hint camera configuration and composited with the foreground using a smoothed segmentation mask.The mask is filtered with a 3 × 3 average filter to smooth its edges.Source: §5 · p. 6
  • For other lighting conditions, a pretrained diffusion inpainting model completes the background from the synthesized foreground, inverse mask, and original prompt.The model is stable-diffusion-2-inpainting.Source: §5 · p. 6
  • Figure 5 places the provisional reference first, followed by outputs under point lighting in columns 2–3 and environment lighting in columns 4–6.The provisional images for the last two examples use DALL-E3 to handle more complex prompts.Source: Figure 5 · p. 7

6. Results

User ratings place lighting similarity and appearance consistency near positive baselines, while qualitative results show plausible lighting across varied prompts and materials. Appearance-seed and prompt edits provide additional control over generated appearance.

6. Results

  • The authors train the provisional-image encoder and ControlNet jointly for 150K iterations with batch size 64, using stable diffusion v2.1 as the base model.Training took approximately 30 hours on 8 NVidia V100 GPUs.Source: §6 · p. 8
  • DiLightNet produces plausible results with consistent appearance under the same target lighting across prompts spanning varied materials and geometry.The examples include high-specular materials, rich geometric detail, multiple homogeneous materials, nonrealistic geometry, and spatially varying materials.Source: §6 · p. 8
  • Changing the second-stage appearance-seed yields alternative interpretations of the provisional image, while reusing a seed preserves appearance across controlled lighting conditions.The paper demonstrates the latter consistency in Figure 5.Source: §6 · p. 8
  • Specializing the prompt between stages with material descriptions keeps variants consistent under the same lighting while constraining their material appearance.The variants range from diffuse without a highlight through mixed diffuse/specular to metallic surfaces with different roughness.Source: §6 · p. 8
  • 19.61/19.85/12.25 average ratings across generated-versus-target, positive-baseline, and negative-baseline pairs place DiLightNet near the positive reference.The first study averaged ratings from 20 non-expert participants.Source: §6 · p. 8
  • 25.75/25.05/11.35 average ratings for same-prompt-and-seed, positive-baseline, and negative-baseline pairs put appearance consistency on par with the positive baseline.The second study used image pairs generated with rotated environment lighting.Source: §6 · p. 8

7. Ablation Study

The ablations examine how radiance-hint conditioning, hint count, foreground masking, training augmentations, and user controls affect DiLightNet. Four radiance hints perform best, while encoding and multiplying the provisional image with the hints is important for satisfactory results.

Additional User Control

  • Changing the appearance-seed can yield alternative interpretations of material properties, while specializing the second-stage prompt adds material guidance under fixed lighting.The seed example changes the interpretation of leather-glove glossiness and smoothness; the prompt example specializes “toy robot” with material descriptions.Source: Figure 6 · p. 9, Figure 7 · p. 9

Evaluation Setup

  • Quantitative evaluation uses 50 high-quality synthetic objects, three viewpoints, six lighting conditions, and the lowest-LPIPS result among four appearance-seeds for each scene/view/lighting combination.Objects with multiple objects or low-quality scanned textures with baked-in lighting were excluded.Source: §7 · pp. 8–9

Provisional Image Encoding

  • Directly passing the provisional image or multiplying it with radiance hints without encoding produces unsatisfactory results; DiLightNet’s encoded multiplication generates more plausible results.The comparison is described qualitatively and is further confirmed quantitatively in Table 1 rows 1–3.Source: §7 · p. 9, Figure 8 · p. 10

Impact of Number of Radiance Hints

  • Four radiance hints perform best; three can be insufficient for complex shapes, while too many can hurt complex materials given smoothed depth-estimated normals.All tested variants include a diffuse hint; the four-hint variant adds a specular hint at roughness 0.05, and the five-hint variant adds one at 0.02.Source: §7 · p. 10

Foreground Masking

  • Removing the foreground mask increases errors, especially for cases with complex shapes or materials.The no-mask variant is trained with black backgrounds and evaluated over foreground pixels only.Source: Table 1 · p. 9, §7 · p. 10

Training Augmentation

  • All three training augmentations improve robustness, with normal augmentation having the largest impact by bridging the gap to smoothed depth-estimated normals.Color augmentation also improves quality, while material augmentation is most noticeable on smooth-shaped objects.Source: §7 · pp. 10–11

8. Discussion

The discussion describes approximate single-image relighting and identifies limits in material control, estimated geometry, and temporal consistency. A reference depth map can bypass depth and mask estimation and improve results.

Relation to Single Image Relighting

  • Photograph relighting is approximate and may be less ideal without a text prompt; plausible results require reasonably accurate depth and mask estimates.Unlike methods trained on narrower scene classes, DiLightNet targets plausible outputs for varied synthesized images, including scenes without real references under novel lighting.Source: §8 · p. 11

Limitations

  • Text prompts provide limited control over scene materials, and some depth-map errors, such as bass-relief ambiguity, can produce unsatisfactory results.The method relies on off-the-shelf depth-map and foreground-mask estimation, although it is robust to some depth errors.Source: §8 · p. 11

Alternative Guidance and Temporal Stability

  • A reference depth map bypasses depth and mask estimation and further increases result quality, while independently generated lighting variations can cause minor structural changes.Cross-frame consistency is identified as a future direction for improving temporal stability.Source: §8 · p. 11

9. Conclusion

The conclusion presents a three-stage method for controlling lighting in text-to-image diffusion. DiLightNet resynthesizes the foreground using encoded provisional-image information multiplied by radiance hints, and the method generates images matching the text prompt and target lighting.

Conclusion

  • The method generates a provisional image, resynthesizes its foreground with DiLightNet conditioned on radiance hints, then inpaints a background consistent with target lighting.DiLightNet multiplies radiance hints by an encoded provisional image to retain shape and texture information.Source: §9 · p. 11
  • The method generates images that match both the text prompt and target lighting.Source: §9 · p. 11

Comparison to Concurrent Work

DiLightNet is compared with Lasagna on a synthetic test set under top and front point lighting. DiLightNet consistently outperforms Lasagna across the reported metrics, with a qualitative comparison also provided.

Comparison to Concurrent Work

  • DiLightNet consistently outperforms Lasagna across PSNR, SSIM, and LIPIPS on the synthetic test set.The comparison uses top or front point lighting, corresponding to Lasagna’s ID-0 and ID-6 settings.Source: Comparison to Concurrent Work · p. 14
  • The comparison is limited to top and front point lighting because ambiguities in Lasagna’s public model’s lighting specification restrict the available comparison.Lasagna supports a predefined set of 12 directional lights.Source: Comparison to Concurrent Work · p. 14
  • The qualitative comparison reports that DiLightNet better matches point-light shading and cast shadows than Lasagna.Lasagna’s result has incorrect shadows and shading effects, including on the barrel.Source: Comparison to Concurrent Work · p. 14

Additional Ablation Study

The additional ablations show how mask input and radiance-hint count affect specular highlights. Additional examples demonstrate lighting-controlled text-to-image and synthetic results, with residual synthetic differences linked mainly to approximate depth-based shape estimates.

Additional Ablation Study

  • Without a mask, DiLightNet more often produces incorrect specular highlights because it cannot distinguish dark foreground pixels from background.The mask ablation is shown in Figure 13.Source: Additional Ablation Study · p. 14, Figure 13 · p. 16
  • Three radiance hints can miss or blur highlights, while too many can misplace sharp specular highlights because depth-estimated normals are limited in accuracy.The implementation uses four hints, which the figure caption describes as producing more plausible results.Source: Additional Ablation Study · p. 14, Figure 14 · p. 16

Additional Results

  • The synthetic test examples span shapes with varied shape and material complexity, and the radiance-hint example visualizes control for a leather glove.Figures 12 and 15 provide representative examples and hint visualizations.Source: Additional Results · p. 14, Figure 12 · p. 15, Figure 15 · p. 16
  • Additional text-to-image results in Figures 16–22 show the impact of changing the content-seed while using the same text prompt; all examples are shown under 3 different lighting conditions.Source: Additional Results · p. 14
  • Synthetic results plausibly match target-lit references, with minor shadow and specular-highlight differences mostly attributed to approximate depth-derived shape.Figure 23 compares the provisional image, target-lit reference, and DiLightNet result; the selected result is the lowest-LPIPS of four candidate seeds.Source: Additional Results · p. 14, Figure 23 · p. 23

Related papers

Loading 2402.11929v2…