Source-linked AI summary
Auto-Exposure Fusion for Single-Image Shadow Removal
Lan Fu, Changqing Zhou, Qing Guo, Felix Juefei-Xu, Hongkai Yu, Wei Feng, Yang Liu, Song Wang
TL;DR
Shadow removal remains difficult because shadow degradation is background-dependent and spatially variant, making traceless recovery challenging. The paper formulates the task as exposure fusion, combining multiple over-exposure images with shadow-aware weighting and boundary refinement, and reports stronger shadow-region performance with comparable non-shadow performance on ISTD, ISTD+, and SRD.
Problem
Shadow removal is difficult because spatially varying, background-dependent degradation and inconsistent boundaries hinder traceless background recovery.
Method
The method estimates multiple over-exposure images, fuses them with shadow-aware FusionNet, and refines residual traces using boundary-aware RefineNet.
Results
The method achieves better performance in shadow regions and comparable performance in non-shadow regions than state-of-the-art methods on ISTD, ISTD+, and SRD.
Takeaways & Limitations
Multiple exposure levels and adaptive per-pixel fusion compensate position-specific degradation while boundary refinement addresses residual penumbra traces.
Takeaways & Limitations
The method focuses on paired training data, relying on paired shadow and shadow-free datasets.
Abstract
from arXiv · showhide
Shadow removal is still a challenging task due to its inherent background-dependent and spatial-variant properties, leading to unknown and diverse shadow patterns. Even powerful state-of-the-art deep neural networks could hardly recover traceless shadow-removed background. This paper proposes a new solution for this task by formulating it as an exposure fusion problem to address the challenges. Intuitively, we can first estimate multiple over-exposure images w.r.t. the input image to let the shadow regions in these images have the same color with shadow-free areas in the input image. Then, we fuse the original input with the over-exposure images to generate the final shadow-free counterpart. Nevertheless, the spatial-variant property of the shadow requires the fusion to be sufficiently `smart', that is, it should automatically select proper over-exposure pixels from different images to make the final output natural. To address this challenge, we propose the shadow-aware FusionNet that takes the shadow image as input to generate fusion weight maps across all the over-exposure images. Moreover, we propose the boundary-aware RefineNet to eliminate the remaining shadow trace further. We conduct extensive experiments on the ISTD, ISTD+, and SRD datasets to validate our method's effectiveness and show better performance in shadow regions and comparable performance in non-shadow regions over the state-of-the-art methods. We release the model and code in https://github.com/tsingqguo/exposure-fusion-shadow-removal.
1. Introduction
Shadow removal is difficult because shadows distort color and illumination in background-dependent, spatially varying ways, while boundaries and interiors can retain visible traces. The paper addresses these issues with auto-exposure fusion, shadow-aware weighting, and boundary refinement for paired-image training.
- Spatially varying shadow distortions hinder vision tasks including detection, tracking, recognition, and semantic segmentation.
- GAN-based methods can produce artifacts and blur, while supervised approaches require paired shadow and shadow-free images.
- Multiple over-exposure images provide flexible compensation for background-dependent color and illumination degradation across shadow regions.
- The auto-exposure fusion network estimates multiple exposure levels and uses shadow-aware FusionNet to select position-specific pixels during fusion.
- Boundary-aware RefineNet removes remaining shadow traces, and experiments report better shadow-region performance with comparable non-shadow performance than state-of-the-art methods.
2. Related Work
The paper reframes paired-image shadow removal as auto-exposure fusion, generating multiple over-exposed shadow images and fusing them pixel-wise to recover shadow-free results. This direction builds on exposure-fusion weighting while adapting it to spatially varying shadow degradation.
- The paper models shadow removal as auto-exposure fusion on paired shadow and shadow-free images.
- Multiple over-exposure shadow images compensate color and illumination degradation before being smartly fused into a shadow-free image.
- Multi-exposure fusion combines images with different exposures using fusion weight maps and weighted summation.
- For shadow removal, the method applies pixel-wise fusion to over-exposure images together with the original shadow image.
3. Methodology
The method reframes shadow removal as exposure fusion: it generates multiple over-exposure images, fuses them with the input using spatially adaptive weights, and refines boundary artifacts.
- Exposure fusion: Multiple over-exposure images compensate shadow regions so their color and illumination approach those of non-shadow regions.The images are generated from the shadow input using channelwise exposure parameters and intensity shifts.
- Shadow-aware FusionNet: Spatially varying shadow degradation makes a single over-exposure insufficient, so FusionNet selects suitable pixels from multiple exposure levels.FusionNet predicts pixel-wise fusion weights using the shadow image and mask as guidance.
- Shadow-aware FusionNet: The fusion combines the input and over-exposure images, while pixel-wise kernels additionally integrate neighboring pixels to improve smoothness and avoid noisy results.The kernel formulation lets each pixel use context from its local neighborhood rather than fusing positions independently.
- Boundary-aware RefineNet: The refinement loss combines pixel-wise reconstruction with a boundary-gradient objective weighted by λ = 0.1.The boundary term preserves non-shadow gradients while reducing gradient differences from the ground truth in shadow regions.
- Boundary-aware RefineNet: Boundary-aware RefineNet removes residual traces caused by penumbra and inconsistent shadow patterns along boundaries and inside shadow regions.It uses a penumbra mask and pixel-wise refine kernels over local neighborhoods.
4. Experiments
Experiments on ISTD, ISTD+, and SRD evaluate the method with LAB-space RMSE against paired shadow-free images. The method performs especially strongly in shadow and penumbra regions while remaining comparable in non-shadow areas.
- Datasets and evaluation: The experiments use paired shadow and shadow-free images from ISTD, ISTD+, and SRD, with RMSE in LAB color space as the evaluation metric.ISTD and ISTD+ also provide shadow masks, while SRD does not provide ground-truth masks.
- ISTD results: On ISTD, the method achieves the lowest RMSE across shadow, non-shadow, and whole-image regions.It reduces shadow-region RMSE by 20.3% versus DSC and from 8.14 to 7.77 versus DHAN.
- ISTD+ results: On ISTD+, the method obtains the best shadow-region performance, with 17.7% lower RMSE than SP+M-Net and comparable non-shadow and whole-image performance.It also reduces RMSE by 59.1% versus DeshadowNet and 51.4% versus ST-CGAN.
- SRD results: On SRD, the method obtains the lowest shadow-region RMSE, reducing it from 8.94 to 8.56 compared with DHAN.Non-shadow RMSE values are close across methods, whereas shadow-region errors vary more substantially.
- Ablation study: Ablations show that neighboring-pixel kernel fusion, multiple over-exposures, RefineNet, and boundary loss each improve shadow-removal performance.The full model reaches RMSE values of 6.5, 3.8, and 4.2 in shadow, non-shadow, and whole-image regions, respectively.
- Ablation study: Multiple over-exposures are most beneficial when ground-truth exposure varies substantially across spatial coordinates, while the method reduces residual penumbra traces versus SP+M-Net.The penumbra-region RMSE decreases by 15.5% compared with SP+M-Net.
5. Conclusion
The method uses multiple over-exposure images and adaptive fusion to address spatially varying shadow degradation, then refines shadow boundaries for traceless background recovery.
- Multiple over-exposure images compensate each pixel at different exposure levels, addressing spatially varying color and illumination degradation.
- Shadow-aware FusionNet adaptively fuses over-exposure shadow images with the input using per-pixel kernel weight maps.
- Boundary-aware RefineNet removes residual traces in penumbra regions along shadow boundaries.
- With boundary loss, the method preserves non-shadow regions, recovers shadow-free areas, and achieves state-of-the-art performance on ISTD, ISTD+, and SRD.