Source-linked AI summary
From Shadow Generation to Shadow Removal
Zhihao Liu, Hui Yin, Xinyi Wu, Zhenyao Wu, Yang Mi, Song Wang
TL;DR
Shadow removal commonly requires difficult-to-obtain shadow-free training images, while patch-based alternatives remain laborious. G2R-ShadowNet uses shadow images and masks to generate paired data for jointly trained generation, removal, and refinement, achieving competitive results and outperforming the patch-based method. Its performance remains sensitive to predicted shadow-mask quality.
Problem
Paired shadow and shadow-free images are difficult to capture, and existing shadow-only patch construction is laborious and computationally heavy.
Method
G2R-ShadowNet uses shadow images and masks to generate pseudo-shadows on non-shadow regions, then jointly trains shadow generation, removal, and refinement sub-networks.
Results
G2R-ShadowNet achieves competitive performance against state-of-the-art methods and outperforms the best weakly supervised patch-based method on the adjusted ISTD and Video Shadow Removal datasets.
Takeaways & Limitations
Shadow generation enables weakly supervised shadow removal without using shadow-free images for training while retaining competitive performance against methods using more training data.
Takeaways & Limitations
Performance is highly affected by predicted shadow masks: missed shadows remain, while falsely detected non-shadow regions may become brighter after removal.
Abstract
from arXiv · showhide
Shadow removal is a computer-vision task that aims to restore the image content in shadow regions. While almost all recent shadow-removal methods require shadow-free images for training, in ECCV 2020 Le and Samaras introduces an innovative approach without this requirement by cropping patches with and without shadows from shadow images as training samples. However, it is still laborious and time-consuming to construct a large amount of such unpaired patches. In this paper, we propose a new G2R-ShadowNet which leverages shadow generation for weakly-supervised shadow removal by only using a set of shadow images and their corresponding shadow masks for training. The proposed G2R-ShadowNet consists of three sub-networks for shadow generation, shadow removal and refinement, respectively and they are jointly trained in an end-to-end fashion. In particular, the shadow generation sub-net stylises non-shadow regions to be shadow ones, leading to paired data for training the shadow-removal sub-net. Extensive experiments on the ISTD dataset and the Video Shadow Removal dataset show that the proposed G2R-ShadowNet achieves competitive performances against the current state of the arts and outperforms Le and Samaras' patch-based shadow-removal method.
1. Introduction
Shadow removal is important but usually depends on difficult-to-acquire paired shadow and shadow-free images. G2R-ShadowNet instead constructs paired training data from shadow images and masks through shadow generation, removal, and refinement.
- 1. Introduction: Paired shadow-free training images are difficult to collect in uncontrolled settings and may limit generalisation when captured in controlled environments.Illumination changes complicate paired capture, while controlled laboratory data cover limited scenarios.
- 1. Introduction: The approach avoids the patch-based alternative’s careful crop design, physics-based constraints, and heavy computational load.Le and Samaras’ method uses patches from shadow images but requires laborious construction procedures.
- 1. Introduction: Shadow generation stylises non-shadow regions as pseudo-shadow regions, which are paired with their original regions to train shadow removal.The proposed network includes shadow generation, shadow removal, and refinement sub-networks trained end to end.
- 1. Introduction: G2R-ShadowNet addresses this data problem using only shadow images and corresponding shadow masks to construct paired shadow and non-shadow data.This preserves the paper’s weakly supervised setting without requiring shadow-free training images.
- 1. Introduction: Experiments on ISTD and Video Shadow Removal show competitive performance against state-of-the-art methods and improvement over the patch-based method.The contribution statement reports these comparisons without providing specific metric values.
2. Related Work
Prior shadow-generation and shadow-removal methods commonly rely on shadow-free images or patch-based construction. G2R-ShadowNet uses same-image regions and mask-guided shadow generation to reduce domain differences and avoid repetitive cropping.
- 2. Related Work: Fully supervised shadow-generation methods require shadow images, shadow-free images, and shadow masks, while unpaired approaches still require shadow-free images.These acquisition requirements can create a substantial source–target domain gap.
- 2. Related Work: Sampling shadow and non-shadow regions from the same image reduces their distribution difference compared with methods trained across separate domains.The paper contrasts this design with Mask-ShadowGAN and LG-ShadowNet, which use adversarial training in both generation and removal.
- 2. Related Work: The patch-based method reduces domain gap but incurs heavy computation from repetitive small-step cropping and uses physics-based constraints that limit handled shadow types.G2R-ShadowNet constructs paired data from the same image and mask without repetitive cropping.
- 2. Related Work: G2R-ShadowNet’s architecture contains shadow generation, shadow removal, and refinement modules jointly trained end to end.During testing, the architecture does not perform shadow generation.
3. Methodology
G2R-ShadowNet jointly trains shadow generation, removal, and refinement to learn from shadow images and masks without paired shadow-free data. Its generator creates pseudo-shadow pairs, the removal network reconstructs shadow-free regions, and refinement uses whole-image context for consistency.
- Overall architecture: The three modules are jointly trained end-to-end: shadow generation constructs training pairs, shadow removal restores content, and refinement improves the final image.The combined objective balances loss terms for the generator, removal sub-net, and refinement sub-net.
- Shadow generation sub-net: The shadow generator applies randomly selected masks to non-shadow regions, then adversarially generates pseudo shadows whose distribution matches sampled real shadows.The region-area constraint uses tolerance α=0.2, except when the original shadow covers more than half the image.
- Shadow removal sub-net: The shadow-removal sub-net learns from pseudo-shadow and non-shadow pairs, mapping generated shadows to shadow-free results that preserve the same content.Its gradient also propagates through the generated pseudo shadows, functioning as a cycle loss for training both networks.
- Refinement sub-net: The refinement sub-net embeds the removal output and mask into the original image, then uses whole-image context to correct color inconsistencies.It receives a four-channel tensor containing three image channels and one mask channel, and outputs a refined image.
- Loss function: Context-aware and adjacent-region losses train refinement while propagating gradients back through the removal and generation modules.A dilation-guided mask emphasizes pixels near the processed non-shadow region.
4. Experiments
Experiments evaluate G2R-ShadowNet on ISTD and video shadow-removal data using standard image-quality metrics, ablations, state-of-the-art comparisons, and qualitative results. The method benefits from joint training and refinement, performs competitively with less training data, and is sensitive to shadow-mask quality.
- 4.3. Ablation study: The refinement sub-network improves shadow-removal quality across RMSE, PSNR, and SSIM, and supplying the shadow mask improves all three metrics.These comparisons are reported on the ISTD test set for shadow-region and whole-image evaluation.
- 4.3. Ablation study: Jointly training all three sub-networks produces the highest PSNR and SSIM among the tested training variants.Detaching all three sub-networks reduces shadow-region performance across all metrics, while connecting two provides intermediate gains.
- 4.3. Ablation study: A dilation kernel size τ of 50 gives the best performance for the Larea term.The study compares τ values of 0, 5, 15, 50, and 100.
- 4.4. Comparison with the state-of-the-arts: G2R-ShadowNet achieves competitive performance using less training data, outperforms unpaired-data methods, and surpasses Le and Samaras on most metrics.The paired-data supervised variant performs best in the shadow region and whole image, while the weakly supervised method uses no shadow-free images.
- 4.4. Comparison with the state-of-the-arts: Qualitative ISTD comparisons show fewer artefacts and more consistent shadow-region colour than competing methods.The patch-based method tends to produce overlightened colour in non-shadow regions.
- 4.5. Generalisation ability: On the video dataset, the method produces lighter shadow regions with fewer artefacts and gains about 14% RMSE improvement after one epoch of per-video fine-tuning.The reported RMSE changes from 21.8 to 18.7 after fine-tuning.
5. Conclusion
G2R-ShadowNet performs weakly-supervised shadow removal without shadow-free training images by jointly using shadow generation, removal, and refinement. Experiments show it outperforms the best weakly-supervised method while remaining competitive with methods trained using more data.
- G2R-ShadowNet outperforms the best weakly-supervised method on the adjusted ISTD and Video Shadow Removal datasets.
- The network jointly trains shadow generation, shadow removal, and refinement in an end-to-end fashion.
- Shadow generation stylises non-shadow regions into shadow regions to construct paired training data for shadow removal.
- The method remains competitive with state-of-the-art approaches trained using paired or unpaired shadow-free images.