Source-linked AI summary
Mask-ShadowGAN: Learning to Remove Shadows from Unpaired Data
Xiaowei Hu, Yitong Jiang, Chi-Wing Fu, Pheng-Ann Heng
TL;DR
Shadow removal traditionally depends on tedious paired captures, while unpaired adversarial and cycle-consistent translation cannot represent the multiple shadows possible for one background. Mask-ShadowGAN learns shadow masks to guide shadow generation within an unpaired framework, constructs the USR dataset, and achieves comparable benchmark performance while outperforming other methods on more general unpaired test images. Its evaluation on USR excludes methods requiring paired training data or unavailable code.
Problem
Paired shadow-removal data is tedious and limited, while direct adversarial and cycle-consistent translation cannot model the non-one-to-one relationship between shadow and shadow-free images.
Method
Mask-ShadowGAN jointly learns shadow removal and shadow masks from unpaired domains, using the masks to guide shadow generation and cycle-consistency training.
Results
Mask-ShadowGAN achieves comparable performance on existing benchmarks and outperforms other methods for more general shadow images without paired ground truth.
Takeaways & Limitations
The framework enables data-driven shadow removal from unpaired data and supports more diverse shadow and shadow-free training images.
Takeaways & Limitations
Evaluation on USR excludes DSC because it requires paired images, and excludes ST-CGAN and DeshadowNet because their code is unavailable.
Abstract
from arXiv · showhide
This paper presents a new method for shadow removal using unpaired data, enabling us to avoid tedious annotations and obtain more diverse training samples. However, directly employing adversarial learning and cycle-consistency constraints is insufficient to learn the underlying relationship between the shadow and shadow-free domains, since the mapping between shadow and shadow-free images is not simply one-to-one. To address the problem, we formulate Mask-ShadowGAN, a new deep framework that automatically learns to produce a shadow mask from the input shadow image and then takes the mask to guide the shadow generation via re-formulated cycle-consistency constraints. Particularly, the framework simultaneously learns to produce shadow masks and learns to remove shadows, to maximize the overall performance. Also, we prepared an unpaired dataset for shadow removal and demonstrated the effectiveness of Mask-ShadowGAN on various experiments, even it was trained on unpaired data.
1. Introduction
Shadow removal requires both eliminating shadows and restoring occluded backgrounds, but paired training data is tedious, limited in scene diversity, and vulnerable to capture inconsistencies. Mask-ShadowGAN learns shadow removal from unpaired shadow and shadow-free domains by using learned masks to guide shadow generation and cycle consistency.
- Shadow removal must remove shadows while restoring the background behind them across varied shapes and backgrounds.
- Paired training data is tedious to collect because each scene requires fixed-camera captures with objects added and removed.
- Paired datasets restrict scene diversity because shadow-free images are difficult to capture for shadows cast by large objects such as trees and buildings.
- Capture changes can make training pairs inconsistent in color, luminosity, camera view, exposure, pose, and environmental lighting.
- Mask-ShadowGAN learns the relationship between unpaired shadow and shadow-free domains and trains Gf to map shadow images toward the shadow-free domain.
- Cycle consistency alone is insufficient because Gs produces a single shadow output for a given shadow-free input, despite multiple possible shadows sharing a background.
- Mask-ShadowGAN learns an input shadow mask that guides Gs, enabling different shadow outputs and shadows that better match the corresponding inputs.
- Mask-ShadowGAN reports comparable performance with existing works on benchmarks and better performance on more general shadow images without paired ground truth.
2. Related Work
Earlier shadow-removal methods used physical models, color transfer, handcrafted features, or paired CNN training, while unsupervised domain translation introduced cycle consistency for unpaired data. Mask-ShadowGAN addresses the remaining one-to-one mapping limitation by learning shadow masks and modeling shadow-region differences.
- Early methods modeled images as shadow and shadow-free layers or transferred colors from non-shadow to shadow regions.
- Physical shadow models often fail on complex real scenes because their underlying assumptions are not physically correct.
- Statistical-learning methods used intensity, color, texture, and gradient features, but handcrafted features lacked high-level semantic understanding.
- Recent CNN methods learn shadow removal end-to-end from paired shadow and shadow-free images, including architectures using multiple views or conditional GANs.
- Unsupervised learning includes representation learning through reconstruction, adversarial transfer of synthetic images, and unpaired domain mapping.
- CycleGAN learns arbitrary one-to-one mappings between domains, so a trained network produces only one output for the same input.
- Mask-ShadowGAN uses shadow masks to guide shadow generation and models the relationship through differences between shadow and shadow-free images in shadow regions.
3. Methodology
Mask-ShadowGAN learns shadow removal and shadow generation from unpaired images by using learned shadow masks to guide cycle-consistent mappings. Its objective combines adversarial, cycle-consistency, and identity losses while maintaining a mask set for diverse shadow synthesis.
- Framework overview: The framework has separate components for learning from real shadow images and real shadow-free images.These two parts jointly train the shadow-removal and shadow-generation mappings.
- Learning from shadow images: Gf transforms a shadow image into a shadow-free image, while Df distinguishes generated outputs from real shadow-free images.Adversarial optimization alone can permit artifacts, so the framework also reconstructs the original shadow image through Gs.
- Learning from shadow-free images: Gs concatenates a shadow mask with a generated shadow-free image, enabling multiple shadow images with different shapes and locations from one background.Masks can be randomly selected from a learned list to increase the number and diversity of generated shadow images.
- Learning from shadow images: The shadow cycle uses a binary mask derived from the difference between the input shadow image and generated shadow-free image to reconstruct the original shadow image.Zeros denote non-shadow regions and ones denote shadow regions; the reconstruction is optimized with an L1 cycle-consistency loss.
- Learning from shadow-free images: The shadow-free cycle applies Gf to generated shadow images, while the identity loss encourages shadow removal without changing colors in non-shadow regions.The shadow-free identity constraint preserves the content of real shadow-free inputs.
- Loss function: The final objective is a weighted sum of adversarial, cycle-consistency, and identity losses optimized jointly for Gs, Gf, Ds, and Df.The reported weights are ω1=1, ω2=10, and ω3=5.
- Shadow-mask generation: Shadow masks are obtained by differencing and binarizing the real shadow image and generated shadow-free image, then thresholding with Otsu’s algorithm.A queue stores multiple masks, replacing the least recently added mask as newly generated masks become available.
- Implementation: The generators use a Johnson et al. architecture with convolutional layers, residual blocks, upsampling deconvolutions, and instance normalization.Training uses Gaussian initialization and Adam with a learning-rate schedule spanning 200 epochs.
4. Unpaired Shadow Removal Dataset - USR
USR is an unpaired shadow-removal dataset designed to address color inconsistencies and limited scene coverage in paired data. It contains thousands of shadow and shadow-free images spanning over a thousand scenes.
- Dataset motivation: Paired datasets can have inconsistent colors and luminosity because exposure, lighting, and camera conditions vary between captures.They also cover only limited scenes, which affects model generality and practicality.
- Dataset composition: USR contains 2,445 shadow images and 1,770 shadow-free images for unpaired shadow-removal training.
- Scene diversity: The dataset includes shadows cast by diverse objects, including trees, buildings, traffic signs, persons, umbrellas, and railings.
- Scene diversity: USR covers over a thousand different scenes, whereas existing datasets typically cover only hundreds of scenes despite having thousands of samples.
- Dataset splits: Shadow images are split into 1,956 training and 489 testing images, while all 1,770 shadow-free images are used for training.Shadow-free images are excluded from shadow-removal testing.
5. Experimental Results
Mask-ShadowGAN is evaluated on unpaired USR data through user studies and against established methods on paired SRD and ISTD data using RMSE and visual comparisons. Across these experiments, it performs strongly despite unpaired training, while mask guidance supports diverse shadow generation and realistic shadow removal.
- USR comparison: Mask-ShadowGAN received the highest user-study ratings on the unpaired USR test set, with differences from other methods statistically significant at p < 0.001.The study used 10 participants and 300 ratings per method.
- USR comparison: On USR, Mask-ShadowGAN more effectively removed shadows and recovered backgrounds, whereas other methods sometimes blurred images or left shadow regions.
- USR comparison: When trained on the SRD or ISTD training sets, Mask-ShadowGAN still outperformed DSC on the USR test set.The comparison used separately trained Ours-S and Ours-I models against DSC-S and DSC-I.
- SRD and ISTD comparison: On SRD and ISTD, Mask-ShadowGAN achieved RMSE values comparable to deep networks trained on paired images and outperformed methods based on hand-crafted features.RMSE evaluates differences between predicted and ground-truth shadow-free images in LAB color space, with lower values indicating better performance.
- SRD and ISTD comparison: Although its RMSE was higher than paired-data deep networks on these datasets, Mask-ShadowGAN produced more realistic images and better preserved texture details occluded by shadows.The comparison includes challenging cases such as large shadow regions and shadows crossing complex textures.
- CycleGAN comparison: Mask-ShadowGAN outperformed CycleGAN on both SRD and ISTD, while mask guidance enabled multiple realistic shadows for one shadow-free input instead of one fixed output.CycleGAN tended to produce artifacts in regions occluded by shadows.
6. Conclusion
Mask-ShadowGAN enables shadow removal from unpaired shadow and shadow-free images while learning masks automatically. The paper introduces an unpaired shadow-removal dataset, evaluates across datasets, and reports quantitative and visual comparisons with state-of-the-art methods.
- Mask-ShadowGAN performs shadow removal using unpaired shadow and shadow-free images.
- The framework learns shadow masks automatically and transforms uncertain shadow-free-to-shadow translation into deterministic translation guided by those masks.
- The authors construct the first unpaired shadow removal dataset and evaluate Mask-ShadowGAN across various datasets against state-of-the-art methods quantitatively and visually.
- The method assumes a small domain difference between unpaired shadow and shadow-free images and generally needs a larger training set for better results.