Source-linked AI summary
Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN
Xiaodong Cun, Chi-Man Pun, Cheng Shi
TL;DR
The paper addresses shadow-removal ghosts, including color inconsistencies in shadow regions and artifacts at shadow boundaries, amid limited paired training data. It proposes DHAN for border-free removal and SMGAN for dataset augmentation, achieving ghost-free images and outperforming state-of-the-art methods, especially in visual quality.
Problem
Shadow-removal methods face color inconsistencies and boundary artifacts, while paired datasets cover relatively few scenes and shadow types.
Method
The paper combines Dual Hierarchical Aggregation Network for multi-context feature and attention aggregation with Shadow Matting GAN for synthesizing shadow images from shadow-free images and masks.
Results
The proposed methods generate ghost-free shadow-free images and outperform other state-of-the-art methods, especially in visual quality.
Takeaways & Limitations
Synthesized data improves novel-scene shadow removal, reducing SRD RMSE from 11.9 to 9.82 when training with ISTD+S instead of ISTD alone.
Abstract
from arXiv · showhide
Shadow removal is an essential task for scene understanding. Many studies consider only matching the image contents, which often causes two types of ghosts: color in-consistencies in shadow regions or artifacts on shadow boundaries. In this paper, we tackle these issues in two ways. First, to carefully learn the border artifacts-free image, we propose a novel network structure named the dual hierarchically aggregation network~(DHAN). It contains a series of growth dilated convolutions as the backbone without any down-samplings, and we hierarchically aggregate multi-context features for attention and prediction, respectively. Second, we argue that training on a limited dataset restricts the textural understanding of the network, which leads to the shadow region color in-consistencies. Currently, the largest dataset contains 2k+ shadow/shadow-free image pairs. However, it has only 0.1k+ unique scenes since many samples share exactly the same background with different shadow positions. Thus, we design a shadow matting generative adversarial network~(SMGAN) to synthesize realistic shadow mattings from a given shadow mask and shadow-free image. With the help of novel masks or scenes, we enhance the current datasets using synthesized shadow images. Experiments show that our DHAN can erase the shadows and produce high-quality ghost-free images. After training on the synthesized and real datasets, our network outperforms other state-of-the-art methods by a large margin. The code is available: http://github.com/vinthony/ghost-free-shadow-removal/
Introduction
The paper targets ghost artifacts in shadow removal by addressing border quality through DHAN and limited scene diversity through shadow synthesis and dataset augmentation.
- Shadow removal matters for scene understanding because shadows can degrade object detection and tracking.
- Existing methods can leave artifacts at shadow borders, motivating a network structure designed specifically for shadow removal.The paper identifies border quality as vital to visual quality.
- 0.1k+ unique scenes are available in the largest dataset, limiting scene coverage and potentially affecting color consistency between shadow and shadow-free regions.
- DHAN hierarchically aggregates dilated multi-context features and attentions for joint shadow removal and detection.Its dilated convolutions preserve low-level details while aggregation promotes feature reuse.
- SMGAN synthesizes paired shadow samples from novel masks or scenes to augment existing datasets.The augmentation pipeline obtains novel paired samples from a trained network and uses them for shadow-related tasks.
- Experiments on real and synthetic data indicate that the algorithm outperforms other state-of-the-art methods, especially in visual quality.
Related Work
Shadow-removal research progressed from interactive or hand-crafted methods to deep networks using contextual features, attention, joint learning, and unpaired data.
- Early shadow-removal methods relied on user interaction or hand-crafted features.
- Later neural methods used global-and-local contexts, direction-aware attention, or joint shadow removal and detection.
- Other work used a stacked GAN or unpaired data for learning shadow removal and detection-related tasks.
Proposed Methods
The paper proposes DHAN for shadow removal and SMGAN for synthesizing shadow images to augment limited paired datasets. DHAN aggregates multi-context features and attention, while SMGAN combines shadow-free images with masks to generate new training examples.
- Dual Hierarchical Aggregation Network: The DHAN backbone uses multi-context features encoded with dilated convolutions and preserves low-level image details for shadow-specific processing.The network is based on context aggregation and uses hyper-column features from a pretrained VGG19 together with the RGB image.
- Dual Hierarchical Aggregation Network: DHAN hierarchically aggregates dilated multi-context features and attentions for joint shadow removal and detection.Its dual aggregations target spatial attention and mixed-layer features.
- Dual Hierarchical Aggregation Network: Aggregation nodes reweight feature channels with squeeze-and-excitation, reduce channels using 3 × 3 convolutions, and remix contexts with spatial pooling.Sigmoid activation is applied only at the end of attention aggregation nodes.
- Training Objectives: DHAN training combines perceptual, mask, and adversarial losses to match shadow-free images, predict shadow masks, and improve border realism.The perceptual loss compares generated and natural shadow-free images across VGG16 feature layers; BCE supervises the predicted mask.
- Shadow Matting GAN: The augmentation strategy addresses datasets with limited scene diversity by generating examples from novel backgrounds and shadow masks.The paper describes current paired datasets as containing only about 100+ different scenes and 10+ unique shadow types.
- Shadow Matting GAN: SMGAN learns shadow mattings from shadow-free images and masks, then synthesizes novel shadow images for dataset augmentation.The generator is trained with supervised similarity and adversarial realism objectives, and can combine USR shadow-free images with randomly selected ISTD masks.
Experiments
Experiments evaluate DHAN and shadow synthesis on standard benchmarks, ablations, shadow synthesis, novel-domain transfer, and comparisons with prior methods. Results indicate that hierarchical aggregation and synthesized training data improve shadow removal, detection, and generalization.
- Experimental setup: Experiments use the ISTD and SRD benchmarks, jointly learning shadow removal and attention with DHAN, then retraining with real and synthesized data.SRD provides 2k+ training and 408 testing images; ISTD provides 1.3k training and 540 testing images.
- State-of-the-art comparison: Compared with prior shadow-removal methods, DHAN produces better visual quality on both standard datasets, where competing methods retain boundary ghosts or color inconsistencies.The comparison includes ST-CGAN, DSC, DeShadowNet, and traditional methods.
- Network ablation: DHAN's feature-layer aggregation improves shadow-region quality over the CAN baseline, while dual aggregation with shadow-mask supervision achieves far better results.The ablation attributes the improvement to hierarchical mixed features and attention aggregation.
- Attention comparison: DHAN's attention captures whole large shadow regions more effectively than local gated convolution, which relies on sibling-convolution information.The comparison is supported qualitatively and quantitatively in the reported attention experiment.
- Shadow synthesis: The shadow synthesis network outperforms Mask-ShadowGAN and ShadowGAN on SSIM, PSNR, and LPIPS when trained and tested using ISTD data.The evaluation covers both reconstruction and perceptual quality of generated shadows.
- Novel-domain transfer: Training with ISTD+S lowers SRD RMSE from 11.9 to 9.82 and improves removal on novel domains without retraining.The synthesized dataset helps the network handle novel scenes with fewer ghosts and color inconsistencies.
Conclusion
The paper combines DHAN for hierarchical feature and attention aggregation with a shadow-matting GAN for dataset augmentation. Experiments report that both components help generate ghost-free shadow-free images.
- Conclusion: The proposed method addresses ghost-free shadow removal through hierarchical aggregation of features and attentions plus GAN-based synthesis from shadow-free images and masks.The synthesized shadows augment existing datasets to enlarge scene coverage and reduce color inconsistencies.
- Conclusion: Experiments show that both the network structure and shadow synthesis method help generate ghost-free shadow-free images.