Source-linked AI summary
Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal
Jifeng Wang, Xiang Li, Le Hui, Jian Yang
TL;DR
Shadow detection and shadow removal have largely been studied separately, leaving joint learning underexplored despite their shared information. The paper introduces ST-CGAN, a stacked two-stage conditional GAN framework that jointly performs both tasks end to end. It reports consistent improvements on detection and removal, while releasing a large-scale triplet dataset for evaluation.
Problem
Existing shadow research generally treats detection and removal separately, and available datasets are limited for evaluating both tasks jointly.
Method
ST-CGAN stacks two conditional GANs so detection outputs feed shadow removal, with densely connected task outputs and adversarial components.
Results
ST-CGAN consistently improves overall performance on both shadow detection and shadow removal across evaluated datasets.
Takeaways & Limitations
The stacked joint-learning design offers a distinct multi-task perspective and supports shadow understanding through mutual task information and hierarchical global scene characteristics.
Takeaways & Limitations
The study excludes the large-scale SRD dataset because it was unavailable to the authors for personal reasons.
Abstract
from arXiv · showhide
Understanding shadows from a single image spontaneously derives into two types of task in previous studies, containing shadow detection and shadow removal. In this paper, we present a multi-task perspective, which is not embraced by any existing work, to jointly learn both detection and removal in an end-to-end fashion that aims at enjoying the mutually improved benefits from each other. Our framework is based on a novel STacked Conditional Generative Adversarial Network (ST-CGAN), which is composed of two stacked CGANs, each with a generator and a discriminator. Specifically, a shadow image is fed into the first generator which produces a shadow detection mask. That shadow image, concatenated with its predicted mask, goes through the second generator in order to recover its shadow-free image consequently. In addition, the two corresponding discriminators are very likely to model higher level relationships and global scene characteristics for the detected shadow region and reconstruction via removing shadows, respectively. More importantly, for multi-task learning, our design of stacked paradigm provides a novel view which is notably different from the commonly used one as the multi-branch version. To fully evaluate the performance of our proposed framework, we construct the first large-scale benchmark with 1870 image triplets (shadow image, shadow mask image, and shadow-free image) under 135 scenes. Extensive experimental results consistently show the advantages of ST-CGAN over several representative state-of-the-art methods on two large-scale publicly available datasets and our newly released one.
1. Introduction
Shadow detection and removal are correlated but have generally been studied separately, despite complementary information between the tasks. ST-CGAN addresses this gap with stacked joint learning, supported by a new triplet dataset and favorable results on both tasks.
- Prior shadow-understanding pipelines: Shadow detection identifies shadow regions, while shadow removal reconstructs shadow-free images; prior work commonly treated them as detection-only, removal-only, or two-stage pipelines.Detection-only methods predict masks, removal-only methods estimate illumination attenuation, and two-stage methods localize shadows before reconstruction.
- Motivation: Shadow masks segment images into two regions, while removal must model their semantic relationship, creating strong correlations and possible mutual benefits between the tasks.Detection also provides clues about illumination conditions, object shapes, and geometry, while removal supports downstream vision tasks.
- Proposed framework: ST-CGAN jointly learns shadow detection and removal end to end using stacked conditional GAN components, with outputs of preceding tasks supplied to subsequent components.The framework uses two generators and two discriminators; the second stage receives prior task outputs as part of its input.
- Proposed framework: The stacked paradigm differs from multi-branch multi-task learning by focusing on tasks in stages and sharing mutual improvements through forward and backward information flows.Multi-branch designs instead learn a shared embedding by aggregating supervision from separate task branches.
- Dataset: 1870 image triplets across 135 scenarios form the ISTD dataset, with 1330 training and 540 testing triplets.Each triplet contains a shadow image, shadow mask, and shadow-free image.
- Results: Extensive experiments show favorable ST-CGAN performance on both shadow detection and removal across two public benchmarks and the newly released dataset.The paper also reports empirical advantages over the widely used multi-branch version.
2. Related Work
Prior work developed data-driven and deep-learning approaches for shadow detection, conditional or stacked GANs for image generation, and shared multi-branch architectures for multi-task learning.
- Shadow Detection: Shadow detection research progressed from data-driven methods to deep CNNs that learn features for shadow regions and boundaries.A multikernel model also classified shadow regions using least-squares SVM optimization.
- Generative Adversarial Networks: Conditional GANs use adversarial loss with an informative conditioning variable to encourage generated images that resemble real images.Prior stacked GAN variants progressively generated higher-resolution images or lower-level representations conditioned on higher-level ones.
- Multi-task Learning: Multi-task learning commonly uses a shared component with parallel branches, each assigned to an individual task.Mask R-CNN and MultiNet are cited as examples of this formulation.
3. A new Dataset with Image Shadow Triplets – ISTD
ISTD is introduced as a large-scale benchmark designed for simultaneous evaluation of shadow detection and removal using image triplets across diverse scenarios. Its construction addresses the lack of datasets supporting both tasks and controls illumination while varying shadow shapes and scene materials.
- Motivation: Existing public datasets target either shadow detection or shadow removal, limiting evaluation in multi-task settings.SBU and UCF support detection, whereas SRD, UIUC, and LRSS support removal.
- Dataset scale and content: ISTD contains 1870 triplets of shadow, shadow mask, and shadow-free images under 135 different scenarios.The triplet format provides corresponding inputs and targets for simultaneous task evaluation.
- Benchmark role: ISTD is presented as the first large-scale benchmark for simultaneous evaluations of shadow detection and shadow removal.The paper compares ISTD with other popular shadow-related datasets in Table 1.
- Dataset properties: Shadow-image and shadow-free-image pairs minimize illumination differences through fixed-exposure capture before and after occluder removal.The shadow is cast by an object, which is then removed to obtain the corresponding shadow-free image.
- Dataset properties: ISTD varies shadow shapes using objects such as umbrellas, boards, persons, and twigs, and covers 135 ground-material types for diverse backgrounds and reflectances.These design choices broaden the range of shadow geometries and scene appearances represented in the benchmark.
4. Proposed Method
ST-CGAN stacks two conditional GANs so shadow detection feeds shadow removal, while adversarial components model relationships among images, masks, and reconstructions. Its stacked design supports task-specific embeddings with forward/backward information flow and is evaluated with dataset-specific training setups.
- Architecture: ST-CGAN uses two conditional GANs: the first predicts a shadow mask from the RGB shadow image, and the second uses both to reconstruct the shadow-free image.The second generator receives x and G1(x), while its discriminator evaluates the concatenation of outputs from G1 and G2 conditioned on x.
- Optimization: The joint objective trains two generators against two discriminators in a two-player zero-sum game, encouraging generated outputs to approach their corresponding ground truths.Training alternates discriminator gradient ascent with generator gradient descent using Adam.
- Architecture: The first discriminator receives the shadow image and mask pair, while the second accepts an additional reconstruction-related input, increasing its input channels from 4 to 7.The generators follow a U-Net-inspired contracting and expanding architecture for context capture and precise localization.
- Multi-task design: Unlike multi-branch learning, the stacked scheme lets tasks retain individual feature embeddings while enhancing one another through forward/backward information flow.The paper contrasts this with learning a shared embedding and aggregating task supervisions in parallel branches.
- Evaluation: Detection evaluation uses BER on three datasets, with ST-CGAN trained on ISTD image triplets; a separate setup forms triplets from SBU pairs and roughly generated shadow-free images.The tables specify that smaller BER is better and mark the best and second-best results by color.
5. Experiments
Experiments evaluate ST-CGAN for shadow detection and removal across multiple datasets using BER and RMSE, finding strong performance and benefits from stacked joint learning.
- Datasets: ISTD contains 1870 shadow, mask, and shadow-free image triplets across 135 scenarios, with 1330 for training and 540 for testing.
- Evaluation metrics: Detection is evaluated with BER and per-class pixel errors, while removal uses LAB-space RMSE over whole, shadow, and non-shadow regions.
- Detection results: 14.4% BER error reduction on SBU and 18.1% on ISTD are achieved compared with scGAN.
- Detection results: ST-CGAN detects shadows despite low scene brightness, bright-area shadows, and fine-grained details such as leaf shadows.
- Removal results: ST-CGAN achieves the best removal performance among compared methods, with non-shadow-region error close to the original difference.
- Removal results: The detection block supplies clearer shadow and shadow-free area clues, supporting the removal advantage of joint learning.
- Stacked joint versus multi-branch learning: ST-CGAN’s stacked joint learning consistently outperforms the multi-branch version across every reported detection and removal metric on ISTD.
6. Conclusion
The paper concludes that ST-CGAN jointly learns shadow detection and removal end to end through a stacked multi-task design. It reports consistent improvements on both tasks and releases a large-scale triplet dataset.
- Conclusion: ST-CGAN is presented as the first end-to-end approach to tackle shadow detection and shadow removal simultaneously.
- Conclusion: Its stacked mode densely connects tasks for multi-task learning and is reported to outperform the common multi-branch design.
- Conclusion: Stacked adversarial components preserve global scene characteristics hierarchically, supporting fine-grained and natural shadow-free recovery.
- Conclusion: ST-CGAN consistently improves overall performance for both shadow detection and shadow removal.
- Conclusion: The authors publicly release the first large-scale dataset containing shadow, shadow-mask, and shadow-free image triplets.