Source-linked AI summary
Shadow Feature Refinement Network: Progressive Feature Refinement based on Knowledge Distillation for Effective Shadow Removal
Donghyun Han, Byoung-Dai Lee
TL;DR
Shadow removal remains challenging because shadows alter image colors and textures, and datasets can limit generalization. SFR-Net combines supervised learning, image inpainting, and feature-level knowledge distillation, with post-processing for color restoration. The paper reports performance similar to state-of-the-art methods and notes limitations in generalization, paired-data dependence, and joint optimization of post-processing.
Problem
Shadow removal is difficult because shadows change regional lighting, texture, and color, while dataset limitations and color differences complicate reconstruction and generalization.
Source: Introduction · p. 1, Related Works · p. 3Method
SFR-Net combines supervised shadow removal with image inpainting, feature-level knowledge distillation, and color-restoration post-processing.
Source: Introduction · p. 2, Conclusion · p. 14Results
SFR-Net achieved performance similar to state-of-the-art methods, with competitive quantitative and qualitative results reported on ISTD+.
Source: Introduction · p. 2Takeaways & Limitations
The paper reports that feature distillation and linear-regression-based color-restoration post-processing can support effective shadow-image reconstruction.
Source: Conclusion · p. 14The model was trained on ISTD+ and SRD, may struggle to generalize to significantly different environments, and still depends on paired data.
Source: Conclusion · p. 14
Abstract
from arXiv · showhide
In the field of deep learning, has seen significant advancements; however, shadow removal remains a persistent challenge owing to the variable sizes and colors of shadows influenced by lighting conditions. This study proposes a novel shadow feature refinement network (SFR-Net), which leverages supervised learning, feature refinement loss, and knowledge distillation to enhance shadow removal performance. A dedicated post-processing algorithm is further introduced to restore natural color consistency in the generated shadow-free images. We evaluated our method on two public datasets: the adjusted image shadow triplet dataset (ISTD+) and the shadow removal dataset (SRD), which demonstrate strong generalization capabilities under diverse conditions. On ISTD+, our model achieved a root mean square error (RMSE) of 3.4627 and structural similarity index measure (SSIM) of 0.9382 across the entire image. On SRD, it recorded an RMSE of 4.3781 and an SSIM of 0.9341. These comprehensive results show that our approach performs competitively across both shadow and non-shadow regions while setting a promising direction for robust and perceptually natural shadow removal. Code is available at https://github.com/DongHyun99/SFRNet.
Introduction
Shadow removal is difficult because shadows alter regional lighting, texture, and color, while varied boundaries and limited training data constrain accurate reconstruction and generalization. SFR-Net combines image inpainting with feature-level knowledge distillation, and the paper reports performance similar to state-of-the-art methods.
Introduction
- Shadows alter regional lighting and image texture and color, while variable penumbra boundaries and overlapping shadows make accurate shadow detection and removal difficult.The penumbral region varies with light intensity and distance, complicating boundary definition.Source: Introduction · p. 1
- Limited diversity in shadow datasets and pixel differences between paired images can hinder learning and generalization to new objects, environments, and shadow conditions.The paper notes that constructing diverse, accurate shadow datasets is labor-intensive and time-consuming.Source: Introduction · p. 1
- SFR-Net combines supervised shadow removal with image inpainting and knowledge distillation using guiding and guided image-generation networks.The guided network takes a shadow image and mask; the guiding network processes the ground-truth image and an empty mask, with corresponding layer features used for supervision.Source: Introduction · p. 2
- The proposed post-processing uses correlations between non-shadow pixels in paired images to correct pixel values across the generated shadow-free image and restore its overall colors.The paper presents this operation as a way to make generated colors more closely match the original images.Source: Introduction · p. 2
- SFR-Net achieved performance similar to state-of-the-art methods, with competitive quantitative and qualitative results reported on ISTD+.The network uses a guiding network to supervise a guided shadow-removal network.Source: Introduction · p. 2
Related Works
Prior shadow-removal work spans physics-based methods, generative models, transformers, diffusion, and inpainting. The paper identifies color differences between shadowed and ground-truth images as a limitation of prior methods and positions SFR-Net's feature-level distillation and pretrained inpainting model as its response.
Related Works
- Prior approaches include physics-based methods, GANs, transformers, and diffusion models; Inpaint4Shadow adapts inpainting specifically for shadow removal.The cited work reports competitive results in shadow elimination and texture consistency.Source: Related Works · pp. 2–3
- Prior methods often struggled to address color differences between shadowed and ground-truth images, which can leave unclear boundaries or color loss in generated images.The paper notes that this incongruity may be more noticeable in images with simple colors and textures.Source: Related Works · p. 3
- SFR-Net applies feature-level knowledge distillation and a pretrained image-inpainting model to preserve shadow-region information and minimize changes to non-shadow regions.The approach trains shadow-removal features to resemble features extracted from the ground-truth image.Source: Related Works · p. 3
Methodology
SFR-Net removes shadows with a guided network trained against a pretrained inpainting network, combining feature distillation with image and adversarial losses. A per-image post-processing step estimates color changes in non-shadow regions and applies them across the output.
Methodology
- SFR-Net takes a shadow image and mask as input, then generates a shadow-removed image for color-restoring post-processing.ISTD+ ground-truth masks are used during training and testing; SRD uses predicted masks.Source: Methodology · p. 3
Shadow Feature Refinement Networks
- A guided network removes shadows while matching intermediate features from a pretrained guiding network that reconstructs the ground-truth image.The guided network uses an FFC-based architecture; LaMa is used for both networks.Source: Shadow Feature Refinement Networks · p. 3
Loss Function
- The objective combines adversarial, high-receptive-field perceptual, discriminator feature-matching, mix reconstruction, and feature-refinement losses.Mix loss combines L1 and MS-SSIM; the perceptual loss uses a ResNet-based model.Source: Loss Function · pp. 4–6
- The patch-based discriminator supervises shadow-region patches to change and non-shadow patches to remain consistent with the target image.The shadow mask determines the region-specific supervision for each patch.Source: Loss Function · pp. 4–5
- Feature refinement loss uses L2 distance between guided- and guiding-network features to distill ground-truth image representations into the shadow-removal network.The networks receive paired image-and-mask inputs, and the selected target layers define where features are compared.Source: Loss Function · p. 6
Post-processing for Color Restoration
- Per-image post-processing learns color differences between the input and shadow-removed output in non-shadow regions, then applies the regression across the output.The method assumes those regions have no color or pixel differences between the shadowed input and ground truth; regression is applied separately to RGB channels.Source: Post-processing for Color Restoration · pp. 6–7
Experiments
SFR-Net was evaluated on ISTD+ and SRD, where it achieved competitive results against state-of-the-art methods and preserved non-shadow regions well. Ablations favored feature refinement in deeper stages, combining all three losses, and BGR linear-regression post-processing for color restoration.
Experimental Setup
- ISTD+ has 1,870 triplets; SRD has 3,088 pairs without masks, so DHAN generated masks and SRD images were resized to 640×480 for comparison.Source: Experimental Setup · p. 7
Comparison with State-of-the-Art Models
- Figures 4 and 5 compare methods against input and ground truth; SFR-Net most closely resembles ground truth on simple scenes.Source: Figure 4 · p. 8, Figure 5 · p. 9, Comparison with State-of-the-Art Models · p. 9
- 4.9750 shadow-region RMSE versus HomoFormer’s 5.0098 on ISTD+; SFR-Net’s entire-image RMSE on SRD was 4.3781, with overall competitive performance.Source: Comparison with State-of-the-Art Models · pp. 7–8, Table 1 · p. 9, Table 2 · p. 10
- On SRD non-shadow regions, SFR-Net achieved best RMSE (2.3307), PSNR (32.8520), and SSIM (0.9859); on ISTD+, it had highest SSIM (0.9941).Source: Comparison with State-of-the-Art Models · p. 8, Table 3 · p. 10
Ablation Study
- Feature refinement loss improved performance over no application, with deeper-stage application performing better regardless of how many stages used it.Source: Table 4 · p. 10, Ablation Study · pp. 10–11
- Combining LLaMa, Lmix, and Lfr in Config. #3 achieved the best performance across all regions and categories.LLaMa + Lmix reduced non-shadow color differences, while LLaMa + Lfr showed stronger PSNR performance in the shadow region.Source: Ablation Study · p. 11
- Across both datasets, post-processing improved quality and BGR generally beat LAB; BGR linear regression took 0.046 s/image versus 2.517 s for gradient boosting.Linear regression also produced the most natural results, while other methods sometimes introduced color artifacts or oversaturation.Source: Ablation Study · p. 12
Conclusion
SFR-Net distills feature maps from a pretrained inpainting network and combines feature refinement loss with MS-SSIM, L1 loss, and linear-regression color restoration. The authors identify limits in generalization, paired-data dependence, and separately optimized postprocessing.
Conclusion
- SFR-Net compares shadow-removal features with features from a pretrained inpainting network and uses MS-SSIM, L1 loss, feature refinement loss, and linear-regression postprocessing for color restoration.The authors report that experiments indicated SFR-Net can effectively reconstruct images for shadow removal.Source: Conclusion · p. 14
- Training on ISTD+ and SRD may limit generalization to unseen settings, and the supervised method still depends on paired data.The authors specifically note different lighting, object types, noisy backgrounds, and complex overlapping shadows as potential challenges.Source: Conclusion · p. 14
- Color postprocessing remains separate from the core model and cannot be optimized jointly; future work proposes integrating it end-to-end and evaluating more diverse conditions.The authors also propose self- or weakly supervised learning to reduce dependence on paired datasets.Source: Conclusion · p. 14
- Interpretability is preliminary through feature and residual visualizations; the authors propose exploring attribution methods to improve decision-process transparency.Source: Conclusion · p. 14