PaperlayerView exact revision on arXiv

Source-linked AI summary

Lens Flare Removal and Reconstruction

Tarun Yenamandra, Jonathon Luiten, Daniel Cremers, Nathan Matsuda

arXiv:2609.39527v1cs.CVcs.GR

TL;DR

Large reflective flares challenge removal methods and are difficult to represent consistently across views because they are not 3D-consistent scene content. The paper introduces a flare-removal model and a multi-view flare representation, and reports that both outperform compared baselines on two benchmarks.

  • Problem

    Existing flare-removal datasets lack full-frame reflective flares, while 3DGS may bake flares into the scene or ignore them because flares are 3D-inconsistent.

    Source: §1.1 · p. 2, §1.2 · p. 2

  • Method

    The paper compiles a large-reflective-flare dataset, fine-tunes a diffusion model for removal, and represents flare components along the image-plane line through the principal point and projected light location.

    Source: §1.1 · p. 2, §3.2.2 · p. 5

  • Results

    Both proposed models outperform compared baselines on Flare7k++ and the VFX benchmark.

    Source: §7 · p. 17

  • Takeaways & Limitations

    The flare representation reconstructs highly variable reflective flares consistently across views, which vanilla 3DGS ignores.

    Source: §5.3 · p. 10

    Scene and flare metrics use a model-derived reference rather than ground truth, so shared reference errors modestly favor variants supervised with it.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

The presence of lens flares in images can significantly reduce the quality of downstream application results for tasks such as 3D scene reconstruction. This is because lens flares are a property of the camera imaging system, and not a part of the underlying scene being modeled. There are previous methods that tackle the removal of small flares focused around a light source. However, existing methods struggle with large flares, such as those that fill the entire image. In this work, we compile a novel dataset for large-flare removal, combining publicly available real-world data with a procedural generation pipeline. We fine-tune a diffusion-based model on our dataset to remove complex, large lens flares. On the other hand, lens flares remain effective artistic tools, widely used in the media. While there are ways to simulate 2D flares, representing and reconstructing lens flares consistently across multiple views has not yet been explored. To achieve this, we introduce a flare representation model that leverages the symmetry of lens flares about the camera's principal point. We propose a computational pipeline to jointly optimize this flare model and a Gaussian splatting model (3DGS). This enables the decomposition of a 3D scene into lens flares and the scene itself, using our flare-removal model. Because the reconstructed flare is explicit and re-renderable, it can be edited and transferred to novel images and new 3D scenes. We evaluate removal on an established benchmark and a new one for large reflective flares, quantify the flare/scene decomposition directly, and show that the pipeline is robust to errors in automatic light-source localization.

1 Introduction

The paper addresses large-flare removal and the difficulty of reconstructing camera-dependent flares consistently across views. It introduces a removal dataset and model, plus a flare representation and joint reconstruction pipeline for separating flares from scene content.

1.1 Lens flare removal

  • On the Flare7K++ scattering benchmark and a new large-reflective-flare benchmark, one of the two proposed models outperforms all prior methods on each benchmark.The work also evaluates flare removal across multiple views and reports better 3D reconstructions.Source: §1.1 · p. 2

1.2 Lens flare representation and reconstruction

  • Existing flare simulators can produce plausible effects, but the paper targets reconstruction of capture-specific patterns that stock, procedural, and physically based approaches do not provide as fitted capture reconstructions.The comparison notes that physically based methods require a matching lens prescription for viewpoint-correct appearance, while the stock flare is unrelated to the target lens and light.Source: §1.2 · p. 2, Table 1 · p. 3
  • Because flares change with camera and light motion rather than behaving as 3D-consistent scene content, existing 3DGS methods may bake them into the scene or ignore them.The paper identifies the same difficulty for deformable 3DGS methods.Source: §1.2 · p. 2, Figure 4 · p. 2
  • The proposed flare representation anchors Gaussians to a camera near-plane and constrains them to the projected line between each light source and the principal point.A learned deformation field adjusts the flares based on light and camera locations; the plane is rasterized with the scene in one pass.Source: §1.2 · p. 2
  • A nine-scene, multi-view reflective-flare dataset supports jointly training the flare model with 3DGS, with the removal model supervising decomposition into scene and flare.The resulting representation is compatible with 3DGS rasterizers and intended for real-time rendering.Source: §1.2 · p. 3
  • The paper presents a Gaussian-splatting flare representation, a diffusion-based removal model, a multi-view scene-and-flare reconstruction pipeline, and editing and transfer applications.The representation is designed to support transferring captured flares onto novel images and 3D scenes.Source: §1.2 · p. 3

2 Related Work

Related work spans learned flare removal, 3D reconstruction strategies that suppress flare-affected regions, and representations that model flare optics or geometry. The paper positions its approach against methods that struggle with large flares or reflective flares across views.

2.1 Flare Removal

  • Earlier removal work trains UNet models on synthetic flare composites [32] [23], while Flare7K++ [5] adds real scattering data and a benchmark; later methods target distinct flare types and removal settings.The discussion also covers UFormer [29], BracketFlare [6], DiffFlare [35], ACL-FR, FlareX, LightsOut [27], and methods not quantitatively compared because code or models are unavailable.Source: §2.1 · p. 3

2.1.1 Flare removal in 3D Reconstruction.

  • GN-FR masks small scattering flares during NeRF supervision, but large scattering or reflective flares can cause most image information to be lost.The paper says its own model removes flares across multi-view images without losing scene information and introduces a benchmark for 3D reconstruction and novel-view synthesis with challenging scattering flares.Source: §2.1.1 · pp. 3–4
  • The depicted removal model fine-tunes Difix3D+ [31] with LoRA [9] for one-step flare removal, using text conditioning rather than Difix3D+'s reference-view conditioning.The two training targets differ in whether lens glare is retained; the paper describes the no-glare target as more realistic, though the glare version may be subjectively preferred.Source: Figure 5 · p. 4

2.2.1 Physically-based Lens Flare Representations.

  • Physically based flare methods model specific lens ensembles, while prior analysis places ghost positions on the line between the light projection and optical center under lens rotational symmetry.The paper uses this geometric property as a constraint in its differentiable 3DGS pipeline; existing 3DGS methods often ignore reflective flares, which vary across views.Source: §2.2.1 · p. 4, §2.2.2 · p. 4

3 Method

The method combines diffusion-based flare removal with a camera-conditioned Gaussian representation for reconstructing flares alongside a 3D scene. It trains removal on synthetic flare composites and jointly renders the scene and flare Gaussians for decomposable reconstruction.

3.1.1 Image formation model.

  • Flare-free paired captures are impractical, so training composites blend flare images with clean Flickr24K images in linear space under an additive composition assumption.The additive model is used for synthesis, not for the 3D flare representation.Source: §3.1.1 · p. 4, §3.1.3 · p. 5

3.1.2 Diffusion-based removal model.

  • A fine-tuned latent diffusion model targets scattering and reflective flares, including full-image and small flares, using the Difix3D+ model without reference-view conditioning.The authors fine-tune the VAE encoder and decoder as well as the diffusion UNet.Source: §3.1.2 · pp. 4–5

3.1.3 Training data.

  • Training composites use a scalar γ in [1.8, 2.2], and the supervision image includes the expected light source for Flare7K++ scattering flares, but not the VFX benchmark.The authors report that adding the light source improves diffusion-model performance.Source: §3.1.3 · p. 5

3.2 Lens Flare Representation and Reconstruction

  • The flare representation uses 3D Gaussians on a camera-near plane and places them along the image-plane line between the principal point and projected light location.The flare model is jointly optimized with a scene-representing Gaussian splatting model.Source: §3.2 · p. 5, Figure 6 · p. 5

3.2.2 Flare Representation.

  • Each flare component uses canonical 1D Gaussians whose scalar means encode radial position; a camera- and light-conditioned deformation model maps them into rendered 3DGS primitives.The construction uses the corrected light projection and near-plane depth, and reuses the scene rasterizer.Source: §3.2.2 · pp. 5–6
  • Additional 2D deformations allow asymmetries and off-axis flares, while a camera-conditioned correction adjusts the light location for pixel-level inconsistencies.The authors state that finer deformations help represent flare asymmetries.Source: §3.2.2 · p. 6

3.2.3 Light Positions.

  • The model needs light-source locations, not intensities, and can use projected locations for off-screen or occluded sources whenever they have positive camera-frame depth.The authors use open-vocabulary detection and report graceful degradation with missing detections and robustness to imperfect detections.Source: §3.2.3 · p. 6, §3.2.2 · p. 6

3.2.4 Reconstruction.

  • Reconstruction separately supervises a scene-only render and a jointly rasterized scene-plus-flare image, with the latter compared against the input and the former against flare-removed imagery.The reconstruction loss uses λ = 0.9.Source: §3.2.4 · p. 7

4 Datasets

The datasets target large reflective flares and multi-view scenes with flares, using held-out clips and views for evaluation. Reconstruction evaluation is limited to nearby-view interpolation and a model-derived reference because real captures do not provide separable flare-free ground truth.

4.1 Flare removal

  • 1866 training images from 90 clips and 18 held-out clips form the VFX data; benchmark flares are composited onto 128 captured images.The model is also trained with Flare7k++ and Flickr24K.Source: §4.1 · p. 7
  • 78.0% median frame coverage in the VFX flares versus 0.39% in Flare7K++ shows that large full-frame reflective flares are absent from existing data.The coverage IQRs do not overlap; Flare7K++ reflective flares are synthetic, while the VFX flares are real captures.Source: §4.1 · p. 7
  • To cover almost ring-shaped full-image flares missing from existing datasets, the authors procedurally generate circular flares with opacity increasing radially from a transparent center.The circle center, radius, color, and opacity are randomly sampled.Source: §4.1 · p. 7

4.1.1 Scenes with Flares.

  • 5 diverse indoor and outdoor scenes contain 50–110 images each, with 10% flare-free test views held out for model-independent scene-only evaluation.Test views place the light source outside the camera frustum or sufficiently occluded.Source: §4.1.2 · p. 7

4.2 Flare Reconstruction

  • 9 scenes provide 300–400 training frames and 40 test images each, with held-out views interleaved to test nearby-view interpolation.Distant held-out views were impractical because sequences ended when the flare vanished.Source: §4.2 · p. 7
  • Real-capture decomposition lacks pixelwise flare-free ground truth because scene and flare radiance are entangled, so evaluation uses a model-derived reference.The reported held-out evaluation tests interpolation within the captured viewpoint range; extrapolation is shown through transfer.Source: §4.2 · p. 8

5 Experiments

The experiments evaluate flare removal, multi-view reconstruction, flare decomposition and transfer, and robustness to light-location errors.

5.1.1 Scattering Flares.

  • On scattering-focused Flare7K++, our model better recovers the scene behind flares and leaves less residual flare than the baselines.The benchmark mainly evaluates scattering flare removal.Source: §5.1.1 · p. 8

5.1.2 Reflective Flares.

  • On the 128-image VFX benchmark, Ours beats all baselines on every metric, while Ours-wGlare leads on PSNR and LPIPS.The benchmark evaluates the proposed flare-removal dataset for reflective flares.Source: §5.1.2 · p. 8

5.2 3D Reconstruction with Flare Corruption

  • 3DGS trained on either of our models’ flare-removed images outperforms existing methods on average; inconsistent removal across views produces reconstruction artifacts.Both models are evaluated against captured images that were not processed by a flare-removal model.Source: §5.2 · p. 8, Table 3 · p. 10

5.1 Removing Lens Flares

  • Across Flare7K++ and VFX, one of Ours or Ours-wGlare beats all baselines on every metric; on VFX, Ours leads every metric.Table 2 reports the benchmark comparisons and ablations.Source: §5.1.2 · p. 8, §5.1.1 · p. 8, Table 2 · p. 10

5.3 Flare Reconstruction

  • Ours gives the best scene/flare separation while vanilla 3DGS has no flare component; component scores use model-derived references, unlike composite scores.The reference-based scene and flare scores measure agreement with that reference rather than absolute quality; only the composite render is compared directly to the captured image.Source: §5.3 · pp. 10–11, §5.3.1 · p. 12

5.3.1 Ablations.

  • Ours unsup achieves the highest Scene+Flare PSNR but poor scene decomposition, while Ours gives better separation; removing δf prevents modeling asymmetric flares.Ours 2D has flare PSNR of 27.77 dB versus 33.27 dB for Ours; Ours wo δl performs on par with Ours because finer deformations absorb light-position error.Source: §5.3.1 · p. 12, Table 4 · p. 14

5.4 Application: Flare transfer

  • Reconstructed flares from three source scenes transfer to a novel target scene and remain editable by uniformly scaling their Gaussians 2×.The transfer keeps the learned flare shape and color in-distribution while anchoring placement to the target scene.Source: §5.4 · p. 15, Figure 14 · p. 15

5.5 Light-Location Robustness Analysis

  • With both corrections removed, composite PSNR falls from 32.72 to 28.24 dB and flare PSNR from 32.69 to 27.27 dB; either correction alone preserves robustness.With at least one correction, the flare retains its structure through moderate noise; with neither, the starburst is lost from σ=0.1.Source: §5.5 · p. 16, Figure 16 · p. 16

6 Future Work

The paper identifies open problems in representing ring-shaped flares and explicitly handling flare occlusion by scene geometry. It also notes limits in removal-model training data and the gap between composited training images and real captures.

6 Future Work

  • A single flare Gaussian cannot represent ring-shaped flares because its opacity peaks at the mean and decreases radially; their 3D representation is left for future work.The removal model handles ring-shaped flares, but the 3D representation does not.Source: §6 · p. 16
  • Occlusion by scene geometry is handled implicitly rather than with an explicit occlusion-aware mechanism, which the authors leave for future work.Out-of-frame light sources are handled based on projected depth.Source: §6 · p. 16
  • Removal is harder for flare types far outside the training distribution, and training on composited images leaves a synthetic-to-real gap that may require optical flare isolation to close.The paper notes that the same real scene cannot be captured simultaneously with and without the same flare.Source: §6 · p. 16

7 Conclusion

The paper presents a lens-flare removal model and a lens-flare representation reconstructed from captured multi-view images. Its removal models outperform compared baselines on both reported benchmarks, and the removal model also performs well in 3D scenes.

7 Conclusion

  • On both Flare7k++ and the proposed VFX benchmark, each of the two flare-removal models outperforms all compared baselines.The paper also reports state-of-the-art performance on its proposed benchmark for consistent flare removal across multiple views.Source: §7 · p. 17
  • The paper introduces a lens-flare removal model and, to the authors’ knowledge, the first lens-flare representation reconstructed from captured multi-view images.The representation supports decomposition of a scene with flares into scene Gaussians and flare Gaussians.Source: §7 · p. 17
  • The 2D flare-removal model outperforms baselines even in 3D scenes, with state-of-the-art performance on the proposed multi-view consistency benchmark.Source: §7 · p. 17
  • The paper suggests the models may be useful for downstream tasks and end-user applications.Source: §7 · p. 17

A Appendix

The appendix provides implementation details, visualizes the multi-view flare-removal benchmark, and adds qualitative results and novel-view visualizations of reconstructed flare models.

A Appendix

  • The supplementary material begins with implementation details for the models.Source: A Appendix · p. 19
  • It includes a visualization of the multi-view flare-removal benchmark.Source: A Appendix · p. 19
  • The appendix adds qualitative experiment results and novel views of reconstructed flare models in a supplementary video.Source: A Appendix · p. 19

A.1 Implementation

The flare-removal model fine-tunes a pretrained diffusion model using LoRA and flare datasets added to clean Flickr24K images. Training updates about 2% of the model parameters and takes approximately 20 hours.

A.1 Implementation

  • The model fine-tunes a pretrained latent UNet diffusion model and VAE with LoRA, updating approximately 19.6M parameters against 950M frozen weights.The trainable parameters include rank-16 UNet adapters, VAE adapters, and the VAE’s skip-connection and output convolutions.Source: §A.1.1 · p. 19
  • Training uses four flare sources: Flare7K++ real scattering flares, Flare7K++ reflective flares, the proposed VFX dataset, and procedurally generated reflective flares.Source: §A.1.1 · p. 19
  • The flare datasets are added to Flickr24K images, and the model is conditioned on a prompt to remove flares while retaining the original image and colors.Source: §A.1.1 · p. 19
  • Training runs for 200k iterations with batch size 1 and learning rate 1e-4 on an Nvidia A100, taking approximately 20 hours.Source: §A.1.1 · p. 19
  • The three baselines are trained on the same datasets using their respective strategies and hyperparameters, with the best-performing Uformer selected for each.The paper attributes the model’s faster convergence relative to baselines trained from scratch to training only about 2% of its parameters.Source: §A.1.1 · p. 19

A.1.1 Flare removal model.

The flare-removal model uses an instant-NGP-style deformation network and camera-frame flare Gaussians, trained with specified initialization, optimization, and densification settings.

A.1.1 Flare removal model

  • The deformation network encodes light position and camera position with multiresolution hash grids, then decodes them through a three-layer, width-128 ReLU MLP.The canonical flare position uses a three-band frequency encoding.Source: §A.1.2 · p. 19
  • Training initializes 200 flare Gaussians along a line and clones or prunes them every 500 iterations from 1k to 12k iterations.Cloning is allowed above opacity 0.005; pruning occurs below 0.002.Source: §A.1.2 · p. 19
  • The model trains for 30,000 iterations with Adam at a 10^-4 learning rate for the network and encodings, plus separate rates for Gaussian parameters.Canonical Gaussian position learning rate decays exponentially from 1.6 × 10^-4 to 1.6 × 10^-6.Source: §A.1.2 · p. 19

A.1.2 Flare representation.

The appendix provides example train and test images for the dog sequence and specifies an initialization used in an ablation.

A.1.2 Flare representation

  • Figure 18 shows given training images, flare-removed training images, and example test images from the dog sequence.The figure is associated with the flare-corrupted 3D reconstruction experiment in Section 5.2.Source: Figure 18 · p. 19
  • Figure 19 plots pixel-level light-source detection precision–recall across GroundingDINO box-confidence thresholds on 1,128 synthetic images.The images contain 1–5 sources each; the plot labels thresholds and marks the best-F1 operating point.Source: Figure 19 · p. 19
  • In the wo G-SAM2 ablation, each light is initialized at the scene’s COLMAP point-cloud centroid plus uniform noise in [0, 1) scene units per axis.Source: §A.1.2 · p. 19

A.2 3D Reconstruction with Flares

This subsection points to example images from a scene in the flare-corrupted 3D reconstruction dataset.

A.2 3D Reconstruction with Flares

  • Figure 18 presents example images from one scene in the flare-corrupted 3D reconstruction dataset described in Section 5.2.Source: §A.2 · p. 19

A.3 Reliability of Light-Source Detection

On the synthetic benchmark, the refined light-detection pipeline improves precision over raw Grounded-SAM2, with a small recall cost at low thresholds. Captured-scene analysis also shows that multi-view aggregation addresses intermittent detections.

A.3 Reliability of Light-Source Detection

  • On captured images, raw Grounded-SAM2 detections include non-emissive reflective or high-contrast regions, while refinement removes false positives and merges duplicate boxes.The comparison uses threshold 0.20; the refined pipeline applies an exposure gate and merges overlapping boxes onto one source per light.Source: Figure 20 · p. 20, §A.3 · p. 20
  • The synthetic benchmark composites real Flare7K++ light sources onto flare-free Flickr24K and captured-image backgrounds, with 1–5 sources per image and exact masks.Predicted SAM2-mask unions are compared against ground-truth masks using pixel-level precision, recall, F1, and IoU.Source: §A.3 · p. 20
  • F1 reaches 0.66 (precision 0.51, recall 0.93, IoU 0.49) at threshold 0.20, versus raw-pipeline peak F1 0.31 at threshold 0.50.The refined pipeline substantially improves precision over raw output, at a small recall cost at the lowest thresholds.Source: §A.3 · p. 20
  • On captured scenes, overlap resolution removes boxes that standard NMS would retain: 93.6% of overlap-removed boxes fall below the IoU threshold.The analysis covers nine scenes and approximately 10.8K frames without ground truth.Source: §A.3 · p. 20
  • Captured-scene detections are intermittent, with 2,216 frames lacking detections and 319 fully filtered; multi-view aggregation with TRASE consolidates them into per-scene 3D light sources.At threshold 0.35, the pipeline retains 9,034 of 10,526 proposed boxes after filtering and overlap resolution.Source: §A.3 · p. 20
Loading 2609.39527v1…