Source-linked AI summary
Direction-aware Spatial Context Features for Shadow Detection
Xiaowei Hu, Lei Zhu, Chi-Wing Fu, Jing Qin, Pheng-Ann Heng
TL;DR
Shadow detection requires global image semantics despite varied backgrounds and remains vulnerable to confusing black objects with shadows or missing unobvious shadows. The paper addresses this with direction-aware spatial context learned by attention in spatial RNNs, embedded across CNN levels and trained with weighted cross entropy. On two benchmark datasets, the network outperforms previous methods, reporting 97% accuracy and a 38% reduction in balance error rate.
Problem
Shadow detection requires global image semantics, while existing methods can misrecognize black objects as shadows and miss unobvious shadows.
Method
The network learns direction-aware spatial context with attention-weighted spatial RNNs, embeds DSC modules across CNN levels, and uses weighted cross entropy training.
Results
97% accuracy and 38% reduction in balance error rate are reported, with the network outperforming previous methods on the evaluated benchmark datasets.
Takeaways & Limitations
Direction-aware spatial context provides the paper's basis for improving single-image shadow detection across multiple feature levels and benchmark datasets.
Takeaways & Limitations
The paper identifies detecting time-varying shadows in videos as future work.
Abstract
from arXiv · showhide
Shadow detection is a fundamental and challenging task, since it requires an understanding of global image semantics and there are various backgrounds around shadows. This paper presents a novel network for shadow detection by analyzing image context in a direction-aware manner. To achieve this, we first formulate the direction-aware attention mechanism in a spatial recurrent neural network (RNN) by introducing attention weights when aggregating spatial context features in the RNN. By learning these weights through training, we can recover direction-aware spatial context (DSC) for detecting shadows. This design is developed into the DSC module and embedded in a CNN to learn DSC features at different levels. Moreover, a weighted cross entropy loss is designed to make the training more effective. We employ two common shadow detection benchmark datasets and perform various experiments to evaluate our network. Experimental results show that our network outperforms state-of-the-art methods and achieves 97% accuracy and 38% reduction on balance error rate.
1. Introduction
Shadow detection benefits from global, direction-aware spatial context because different regions provide unequal evidence for identifying shadows. The paper introduces a multi-level network with direction-aware attention and weighted training to improve detection.
- 1. Introduction: Existing methods can misclassify black objects as shadows and miss unobvious shadows despite 87% to 90% accuracy on two benchmark datasets.BER exposes these errors by equally considering shadow and non-shadow regions.
- 1. Introduction: Different spatial directions contribute unequally when contextual regions are compared to determine whether a target region is shadow.In the motivating example, region B provides stronger evidence for region A being a shadow than region C.
- 1. Introduction: The DSC module uses a spatial RNN to aggregate context in four principal directions while learning direction-specific attention weights.This mechanism produces direction-aware spatial context features for shadow detection.
- 1. Introduction: Multiple DSC modules are embedded at different CNN layers, combining DSC and convolutional features to produce and fuse multi-scale shadow score maps.The network is trained end-to-end with a weighted cross entropy loss designed to balance shadow and non-shadow detection.
- 1. Introduction: 97% accuracy and 38% reduction in balance error rate are reported against previous methods across the evaluated benchmark setting.The evaluation compares the network with several state-of-the-art methods on shadow detection, saliency detection, and semantic image segmentation.
2. Related Work
Prior single-image shadow detectors use physical models, hand-crafted features, or CNN-based learned representations. Although benchmark accuracy has improved, methods can still confuse black objects with shadows and miss unobvious shadows, motivating global semantic reasoning.
- 2. Related Work: Physical-model approaches exploit illumination and color but tend to work satisfactorily only for wide dynamic range images.
- 2. Related Work: Hand-crafted approaches describe image regions with color, texture, edge, and T-junction features before classifying shadow and non-shadow regions.
- 2. Related Work: CNN-based methods learn shadow features using superpixels, object boundaries, structured labels, noisy annotations, patch-level inputs, or shadow prior maps.
- 2. Related Work: Existing methods may still misrecognize black objects as shadows and miss unobvious shadows, while recent work emphasizes reasoning about global semantics.
3. Methodology
The network learns direction-aware spatial context at multiple CNN levels, combines it with convolutional features, and predicts shadow maps with deep supervision and weighted cross entropy. Its DSC module uses spatial RNN propagation in four directions with learned attention weights to selectively aggregate context.
- Overall network: The network extracts hierarchical CNN features, applies DSC modules at multiple layers, and produces an end-to-end final shadow detection map.DSC features are combined with convolutional features before score-map prediction and fusion.
- Spatial context features: The spatial RNN aggregates local context through independent translations in the left, right, up, and down directions.Two rounds of recurrent translations further propagate the aggregated context across the spatial domain.
- Direction-aware spatial context: The DSC attention mechanism learns separate attention-weight maps for four directions and multiplies them element-wise with directional context features.The maps are estimated from input feature maps by successive convolutional layers, then used to selectively leverage spatial context.
- Direction-aware spatial context: The DSC module concatenates attention-weighted context features, reduces their dimensions with a 1×1 convolution, and reuses the same attention weights in the second recurrent round.The module is applied across CNN layers except the first layer because of its large memory footprint.
- Training objective: Weighted cross entropy balances shadow and non-shadow detection and emphasizes the class with fewer correctly classified pixels.The loss is applied to each predicted score map, with the overall loss summing layer, MLIF, and fusion losses.
4. Experimental Results
Experiments on two benchmark datasets show that DSC outperforms shadow-detection, saliency-detection, and semantic-segmentation baselines, while component and architecture analyses support direction-aware spatial context.
- Datasets and Evaluation Metrics: The SBU and UCF benchmark datasets contain 4089/638 and 145/76 training/testing images, respectively, and the network is trained on SBU.SBU spans diverse scenes and image types, while UCF covers outdoor scenes with varied backgrounds.
- Datasets and Evaluation Metrics: Accuracy and BER evaluate shadow detection, with BER weighting shadow and non-shadow regions equally because shadow pixels are usually fewer.Lower BER indicates better detection.
- Comparison with the State-of-the-art Shadow Detection Methods: DSC outperforms four recent shadow-detection methods on both accuracy and BER across both benchmark datasets, including UCF despite training on SBU.The comparison includes scGAN, stacked-CNN, patched-CNN, and Unary-Pairwise.
- Comparison with the State-of-the-art Shadow Detection Methods: Visual comparisons include light and dark neighboring shadows, complex backgrounds, and black objects, where DSC locates shadows while avoiding false positives.Other methods may misclassify black objects as shadows or miss unobvious shadows.
- Comparison with Saliency Detection and Semantic Segmentation Methods: DSC outperforms saliency-detection and semantic-segmentation models on both benchmark datasets using accuracy and BER comparisons.The compared models are SRM, Amulet, and PSPNet, retrained on the SBU training set.
- Evaluation on the DSC Module: Component analysis shows that adding spatial context and DSC features improves a basic multi-scale network trained with weighted cross entropy.The baselines remove DSC modules or retain spatial context without direction-aware attention weights.
- Evaluation on the DSC Module: Two recurrent-translation rounds with shared attention weights produce the best detection result in DSC architecture analysis.One round may limit global context propagation, whereas three rounds or separate weights increase parameters and training difficulty.
- More Shadow Detection Results: DSC handles several difficult shadow patterns but fails in scenes with many small shadows, large black regions, or soft shadows.These failures reflect lost detail, insufficient semantic or surrounding context, and small shadow–non-shadow differences.
5. Conclusion
The paper presents a direction-aware spatial-context network for single-image shadow detection, using attention in spatial RNNs and multi-layer DSC modules. It reports superior accuracy and BER performance, while identifying future work on other applications and time-varying shadows.
- The network harvests multi-level spatial context directionally by learning attention weights in a spatial RNN.This produces direction-aware spatial context features and forms the DSC module.
- Multiple DSC modules are embedded in a multi-layer CNN to predict score maps at different scales, with weighted cross entropy used for training.
- The network outperforms state-of-the-art methods on two benchmark datasets in accuracy and balance error rate metrics.
- Future work will explore saliency detection, semantic segmentation, and detection of time-varying shadows in videos.