Source-linked AI summary
ARGAN: Attentive Recurrent Generative Adversarial Network for Shadow Detection and Removal
Bin Ding, Chengjiang Long, Ling Zhang, Chunxia Xiao
TL;DR
Shadow detection and removal remain difficult in complex scenes, while robust deep models also need to exploit limited labeled data. ARGAN uses recurrent coarse-to-fine attention, adversarial generation, and semi-supervised training, and experiments report effective detection and realistic removal across public datasets.
Problem
Effective shadow detection and removal is needed for complex shadows and downstream visual understanding, but existing approaches face complex-environment and supervised-data limitations.
Method
ARGAN progressively detects shadow regions with attention maps, recovers shadow-lighter or shadow-free images through removal encoders, and supports semi-supervised adversarial training.
Results
On ISTD, N = 3 gives BER 2.01 for shadow detection and RMSE 6.68 for shadow removal, offering a trade-off between performance and complexity.
Takeaways & Limitations
ARGAN handles shadows with complex scenes and incorporates sufficient unsupervised shadow images to train a powerful model.
Takeaways & Limitations
The reported semi-supervised loss uses an unsupervised image and a supervised training set of input-output pairs, with λ as a weighting hyperparameter.
Abstract
from arXiv · showhide
In this paper we propose an attentive recurrent generative adversarial network (ARGAN) to detect and remove shadows in an image. The generator consists of multiple progressive steps. At each step a shadow attention detector is firstly exploited to generate an attention map which specifies shadow regions in the input image.Given the attention map, a negative residual by a shadow remover encoder will recover a shadow-lighter or even a shadow-free image. A discriminator is designed to classify whether the output image in the last progressive step is real or fake. Moreover, ARGAN is suitable to be trained with a semi-supervised strategy to make full use of sufficient unsupervised data. The experiments on four public datasets have demonstrated that our ARGAN is robust to detect both simple and complex shadows and to produce more realistic shadow removal results. It outperforms the state-of-the-art methods, especially in detail of recovering shadow areas.
1. Introduction
Shadow detection and removal matter for scene understanding and downstream vision tasks, but existing approaches struggle with complex environments or limited labeled data. ARGAN addresses these challenges with recurrent coarse-to-fine processing, adversarial training, and semi-supervised learning.
- Shadow detection provides clues for scene geometry, camera location, object relighting, and scene illumination inference.
- Traditional methods rely on priors that may fail in complex environments, while deep learning methods depend heavily on supervised data.
- ARGAN uses multiple progressive steps to detect shadows and remove them in a coarse-to-fine fashion.
- At each step, an attention detector produces a shadow map and a removal encoder uses it with the previous result to recover a shadow-lighter or shadow-free image.
- Adversarial training makes generated shadow-removal images increasingly difficult to distinguish from real shadow-free images.
- Semi-supervised learning incorporates unlabeled shadow images and is reported to improve ARGAN's generalization ability and robustness.
- Experiments on four public datasets evaluate ARGAN for both shadow detection and shadow removal, with results demonstrating the model's efficacy.
2. Related Work
Prior work spans traditional, deep learning, GAN, and attention-based approaches to shadow detection and removal. ARGAN combines recurrent attention and adversarial generation within a framework containing progressive detectors and removal encoders.
- Related work includes traditional methods using user interactions or hand-crafted features and deep learning methods for automatic shadow detection.
- Shadow removal methods use gradient manipulation, illumination, color transfer, shadow mattes, depth information, and deep contextual features.
- GANs have been applied to image translation tasks including shadow detection and removal, while ARGAN progressively detects and removes shadows.
- Attention mechanisms select important information across natural language, speech, computer vision, image captioning, and visual question answering.
- ARGAN's framework contains a generator with N shadow attention detectors and N shadow image encoders, plus a discriminator for real-or-fake classification.
3. Approach
ARGAN uses a recurrent generator that progressively detects shadow regions and removes them through attention-guided negative residuals, while a discriminator judges the final output. Its attention maps become more accurate across iterations, and outputs become progressively lighter until nearly shadow-free.
- Generative network: ARGAN’s generator contains N progressive steps, each pairing a shadow attention detector with a shadow removal encoder.The discriminator evaluates the generator’s final shadow-free output under a semi-supervised framework.
- Shadow attention detector: At each step, the attention detector recurrently estimates a soft shadow map, where larger values indicate regions more likely to be shadow.The detector uses convolutional layers, an LSTM unit, and a convolutional layer to combine information across steps.
- Shadow attention detector: Attention progressively shifts toward missed shadow regions, making detected regions more accurate over successive recurrent iterations.Figure 4 compares the input with attention maps A1, A2, and A3 from the first three progressive steps.
- Shadow removal encoder: The removal encoder multiplies its sigmoid output by the detected attention map to produce a negative residual that recovers a lighter or shadow-free image.Its architecture uses eight convolutional and eight deconvolutional layers with skip connections, followed by additional convolutional and sigmoid layers.
- Shadow removal encoder: As progressive steps increase, output shadows become lighter, and O3 is reported as almost shadow-free.Figure 6 presents the input and outputs O1, O2, and O3.
- Discriminative network: The discriminator is a binary classifier trained to distinguish the final generated image from a real shadow-free image, including unlabeled shadow images through semi-supervised training.Unsupervised inputs are passed through the generator, and the discriminator judges whether the generated result is real.
4. Experiments
Experiments evaluate ARGAN for shadow detection and removal across public datasets, compare it with prior methods and variants, and analyze recurrent-step choices and qualitative behavior. ARGAN generally achieves stronger detection and removal results, while semi-supervised training and recurrent refinement improve performance, including in complex scenes.
- Experimental setup: ARGAN and its semi-supervised variant are evaluated against traditional and deep-learning methods on SBU, UCF, SRD, and ISTD datasets.Detection uses BER, while shadow removal uses RMSE in LAB color space; the datasets support different evaluation tasks.
- Shadow detection: ARGAN achieves the best BER among competing methods on all three shadow-detection datasets, while ARGAN+SS further improves performance.AGAN, which removes the LSTM layers, performs much worse than ARGAN.
- Qualitative detection: ARGAN detects slender shadows more accurately than traditional methods and produces regions closer to human observation than the compared deep-learning methods.The authors also report the strongest detection performance on examples with complex scenes, attributing this to recurrent coarse-to-fine updates.
- Shadow removal: ARGAN achieves the best RMSE in shadow regions and whole images on both shadow-removal datasets, and ARGAN+SS consistently improves results across shadow, non-shadow, and whole-image regions.The authors attribute the qualitative difference between ARGAN and AGAN to LSTM effects on the attention map and recovered image quality.
- Qualitative removal: ARGAN preserves surrounding illumination consistency and texture details in complex scenes, while competing methods may incorrectly treat dark non-shadow areas as shadows.The authors associate ARGAN’s more natural removal results with using color information across the whole image.
- Progressive-step analysis: On ISTD, N = 3 provides a reported trade-off between performance and complexity: BER is 2.01 and whole-image RMSE is 6.68.For N = 1, 2, 3, and 4, BER is 2.22, 2.08, 2.01, and 2.01; RMSE is 7.35, 6.97, 6.68, and 6.67, respectively.
5. Conclusions
The paper concludes that ARGAN performs shadow detection and removal through multiple coarse-to-fine progressive steps. It is reported as effective on complex scenes and flexible enough to incorporate unsupervised shadow images, with video extension identified as future work.
- ARGAN is proposed as an attentive recurrent generative adversarial network for shadow detection and removal.
- The generator uses multiple progressive steps in a coarse-to-fine fashion to generate shadow attention maps and recover shadow-removal images.
- The model is reported to handle complex scenes and flexibly incorporate sufficient unsupervised shadow images, while extending it to video is left for future work.