LightsOut: Diffusion-based Outpainting for Enhanced Lens Flare Removal

1. Introduction

• Lens flare degrades image quality and harms tasks including object detection and autonomous driving.Nighttime SIFR also remains challenging because real-world paired training data are limited.Source: §1 · p. 1
• Incomplete off-frame light-source views cause existing SIFR methods to leave more flare artifacts and reduce realism.Earlier SIFR methods have advanced through new datasets and architectures, but still degrade in this setting.Source: §1 · p. 1
• LightsOut uses multitask regression and a LoRA-fine-tuned, light-source-conditioned diffusion model to complete missing off-frame sources.The paper describes the generated content as aligned with real-world flare and illumination distributions.Source: §1 · p. 2
• LightsOut acts as plug-and-play preprocessing and consistently improves flare-removal realism and effectiveness across state-of-the-art methods.The framework is presented as enhancing existing SIFR models without additional retraining.Source: §1 · p. 2

Lens Flare Removal

• Earlier flare-removal work spans detection and removal, glare and reflective-flare methods, and learning approaches, while current models still degrade with incomplete off-frame sources.The cited work includes dataset development and synthetic-data extensions as well as unpaired-data methods and newer architectures.Source: §2 · p. 2

Adapting Diffusion Models

• Diffusion-model adaptation includes fine-tuning pretrained models and text embeddings, but existing models lack physics-based optical modeling and struggle beyond image boundaries.LoRA is described as an efficient fine-tuning approach.Source: §2 · p. 2

Image Completion and Outpainting

• Outpainting methods span GAN, semantic, edge-guided, spiral, recurrent, transformer, and diffusion approaches, with physical light-source and flare extrapolation remaining challenging.The passage notes that diffusion approaches lack explicit light-source and optical-effect modeling.Source: §2 · p. 2

Image Conditioned Diffusion Models

• Image-conditioned diffusion methods use image features, sketches, keypoints, or instructions, and LightsOut adds light-source constraints as a conditioning factor.The overview describes predicting source parameters, generating a conditioned outpainting, and using it to boost SIFR methods.Source: §2 · p. 2, Figure 3 · p. 3

3. Method

• LightsOut expands an incomplete-light-source image, predicts a light-source mask, outpaints the scene, applies a SIFR model, and crops the flare-free result to the original region.The pipeline first prepares a larger masked image and ultimately extracts the original image region from the SIFR output.Source: §3 · p. 3

3.1. Preliminaries

• Diffusion training progressively adds Gaussian noise, while LoRA adapts a frozen denoising network by optimizing only low-rank added weights.The noisy sample at each step depends on a variance-scheduling parameter.Source: §3.1 · p. 3

3.2. Image Outpainting for Light Source

• Stable Diffusion v2 is LoRA-fine-tuned for light-source outpainting, with RGB-space composition preserving existing regions and noise reinjection helping align masked and unmasked regions.Noise reinjection reintroduces noise at intermediate steps so the model can re-denoise the regions.Source: §3.2 · pp. 3–4

3.3. Light Source Prediction and Conditioning

• A parameterized regressor predicts N light sources' positions, radii, and existence confidence; thresholded predictions form a spatial mask that conditions physically plausible light placement.Bipartite matching supports permutation-invariant prediction, while position loss is applied only to existing ground-truth sources.Source: §3.3 · pp. 4–5

3.4. Flare Removal Methods

• LightsOut is a model-agnostic preprocessing step that reconstructs incomplete or off-frame light sources and integrates into existing SIFR pipelines without architectural modifications.Source: §3.4 · p. 5

4.1. Experimental Setup

• Evaluation uses Flare7K test images in no-light-source and incomplete-source scenarios, comparing SIFR methods and diffusion-based inpainting or outpainting baselines.The test set contains 100 real and 100 synthetic images; incomplete sources are simulated by shifting the boundary outward by 15 pixels.Source: §4.1 · pp. 5–6

4.2. Quantitative Evaluations

• PSNR rises from 26.29 dB to 28.41 dB with Flare7K++ on real images without light sources, and LightsOut outperforms diffusion-based methods on real and synthetic datasets.Difflare was excluded because public implementations and pretrained models were unavailable.Source: §4.2 · p. 6, Table 1 · p. 6
• The multitask regression module achieves mIoU scores of 0.6310 on real and 0.6619 on synthetic data, exceeding the UNet baseline’s 0.6216 and 0.6563, respectively.Source: §4.2 · p. 6, Table 2 · p. 6

4.3. Qualitative Evaluations

• In no-source and incomplete-source cases, adding LightsOut improves the compared SIFR methods’ flare removal, producing results closer to ground truth.Source: §4.3 · p. 6, Figure 5 · p. 7
• Compared with SD-Inpainting, SDXL-Inpainting, and PowerPaint, LightsOut produces more realistic outpainting with accurate flare artifacts and scene-aligned illumination.The reported explanation is that standard diffusion methods lack explicit conditioning on light sources and flare distributions.Source: Figure 6 · p. 7, §4.3 · p. 7

4.4. Ablation Study

• Noise reinjection improves coherence and flare-removal performance, RGB blending gives smoother transitions than latent blending, and conditioning improves off-frame source prediction.Source: Table 4 · p. 7, §4.4 · p. 8, Figure 7 · p. 8
• Multitask regression outperforms differentiable rendering and standard regression for light-source prediction, while a retrained off-frame SIFR baseline still lags behind LightsOut.Source: §4.4 · p. 7, Table 3 · p. 7, Table 6 · p. 8
• LoRA fine-tuning and the light-source conditioning module each improve performance, and combining them yields the best overall results.Their respective gains are PSNR increases of 0.30/0.24 dB and LPIPS drops of 0.0030/0.0027.Source: §4.4 · p. 8, Table 7 · p. 8

• LightsOut reconstructs off-frame illumination with diffusion-based outpainting and conditioning modules to enhance existing SIFR methods, but added outpainting stages introduce computational overhead.The authors identify end-to-end optimization as a possible way to reduce that overhead.Source: §5 · p. 8

• Flare7K images are preprocessed with YCbCr luminance masks to find the largest rectangular region excluding the light source, which is cropped and masked for outpainting.Source: §A.1 · p. 12
• The framework’s three modules are optimized independently; the multitask regression module uses learning rate 1 × 10−4, batch size 32, and 100 epochs, with four predicted light sources.The light-source condition module uses learning rate 1×10−5 and batch size 8 for 20,000 steps; the Stable Diffusion inpainting network is LoRA-fine-tuned at 1 × 10−4 with batch size 8 for 25,000 steps.Source: §A.1 · p. 12
• Inference uses 50 sampling steps, guidance scale 7.0, and four noise reinjections; BLIP-2 generates captions automatically to minimize human bias.Source: §A.1 · p. 12
• Flare removal is evaluated with PSNR, SSIM, and LPIPS, while light-source prediction is assessed with mIoU.Source: §A.1 · p. 12

A.2. Downstream Tasks

• The evaluation compares images processed directly by SIFR models with images first enhanced by outpainting before SIFR.Source: §A.2 · p. 13
• Outpainting before SIFR improves detection accuracy, particularly for objects in regions previously compromised by flare artifacts.The comparison uses a pre-trained YOLOv11 detector.Source: §A.2 · p. 13
• The reported accuracy improvement is especially apparent for objects in areas previously compromised by flare artifacts.Source: §A.2 · p. 13

A.3. In-the-Wild Images.

• On challenging self-collected in-the-wild images, LightsOut outpaints off-frame regions and improves existing SIFR models.Figure 10 compares flare removal against Zhou et al., Flare7K++ [13], and MFDNet [28]; Figure 11 shows additional outpainting results.Source: §A.3 · p. 13

A.4. Failure Cases

• High overall image brightness or a flare covering a relatively large proportion of the image makes precise flare-region delineation difficult, even with LightsOut.Source: §A.4 · p. 13

A.5. Qualitative Comparisons of Light Source Mask Prediction.

• The multitask regression module predicts light-source positions and radii more accurately than U-Net in both single- and multiple-light-source scenarios.Source: §A.5 · p. 13
• In-the-wild flare-removal comparisons indicate that LightsOut addresses scenes with off-frame light sources that existing SIFR models fail to handle.Source: Figure 10 · p. 13

A.6. Additional Qualitative Comparisons

• Additional flare-removal results cover diverse imaging conditions, while comparisons include baseline and state-of-the-art diffusion-based inpainting and outpainting methods.The comparisons are presented in Figures 13–16.Source: §A.6 · p. 13