PaperlayerView exact revision on arXiv

Source-linked AI summary

Direction-aware Spatial Context Features for Shadow Detection and Removal

Xiaowei Hu, Chi-Wing Fu, Lei Zhu, Jing Qin, Pheng-Ann Heng

arXiv:1805.04635v2cs.CV

TL;DR

Shadow detection and removal need global image semantics, but complex scenes and inconsistent training pairs remain challenging. The paper learns direction-aware spatial context with DSC modules in a CNN, using task-specific losses and color compensation. Across two detection and two removal benchmarks, the method performs favorably against state-of-the-art approaches for both tasks.

  • Problem

    Shadow detection and removal remain challenging because complex scenes require global semantics, while training pairs can have inconsistent colors and luminosity.

  • Method

    The method learns direction-aware spatial context through attention-weighted spatial RNNs, embeds DSC modules across CNN layers, and uses weighted cross entropy, Euclidean loss, and color transfer.

  • Results

    The network performs favorably against state-of-the-art methods for shadow detection and shadow removal across benchmark datasets.

  • Takeaways & Limitations

    Direction-aware spatial context provides a unified network design for evaluating and addressing shadow detection and removal.

  • Takeaways & Limitations

    The method can fail in extremely complex scenes involving many small shadows, large black regions, or soft shadows.

Abstract

from arXiv · show

Shadow detection and shadow removal are fundamental and challenging tasks, requiring an understanding of the global image semantics. This paper presents a novel deep neural network design for shadow detection and removal by analyzing the spatial image context in a direction-aware manner. To achieve this, we first formulate the direction-aware attention mechanism in a spatial recurrent neural network (RNN) by introducing attention weights when aggregating spatial context features in the RNN. By learning these weights through training, we can recover direction-aware spatial context (DSC) for detecting and removing shadows. This design is developed into the DSC module and embedded in a convolutional neural network (CNN) to learn the DSC features at different levels. Moreover, we design a weighted cross entropy loss to make effective the training for shadow detection and further adopt the network for shadow removal by using a Euclidean loss function and formulating a color transfer function to address the color and luminosity inconsistencies in the training pairs. We employed two shadow detection benchmark datasets and two shadow removal benchmark datasets, and performed various experiments to evaluate our method. Experimental results show that our method performs favorably against the state-of-the-art methods for both shadow detection and shadow removal.

1 INTRODUCTION

Shadow detection and removal require global image semantics because shadows provide useful scene cues but can degrade other vision tasks. The paper addresses this by learning direction-aware spatial context with DSC modules embedded across CNN layers.

  • Shadows convey depth, geometry, lighting direction, and camera information, yet can impair object detection and tracking.
  • Existing physical-model approaches rely on assumptions that may fail in complex scenes, motivating data-driven deep neural networks.
  • Directional context matters because comparing a candidate region with different surrounding regions provides unequal evidence that it is shadowed.
  • The DSC module uses a spatial RNN with learned attention weights to aggregate global context differently across directions.
  • Multiple DSC modules learn context at different CNN layers, combine it with convolutional features, and predict shadow masks with weighted cross entropy.
  • For shadow removal, the network uses Euclidean loss and color-compensated shadow-free targets produced by a transfer function.
  • Experiments on benchmark datasets show favorable performance against state-of-the-art methods for both shadow detection and shadow removal.

2 RELATED WORK

Prior shadow methods progressed from physical and hand-crafted models to CNN-based approaches, but complex scenes still challenge detection and removal. This work extends direction-aware context modeling with color compensation and broader evaluation.

  • Shadow detection: Physical shadow models and hand-crafted features often struggle with complex real scenes and lack high-level semantic knowledge.
  • Shadow detection: CNN-based shadow detectors improve benchmark accuracy but may still confuse black objects with shadows or miss unobvious shadows.
  • Shadow removal: Shadow removal methods require a global image view for consistent predictions, yet may alter colors in non-shadow regions.
  • Shadow removal: The proposed method analyzes global spatial context directionally and compensates training-pair color and luminosity inconsistencies using non-shadow regions.
  • This work: The network combines multi-scale CNN features with DSC modules and integrates layerwise predictions for shadow detection.
  • This work: The paper extends earlier work through redesigned shadow-removal outputs and losses, color compensation, additional benchmark experiments, and time-performance measurements.

3 METHODOLOGY

The methodology uses a fully convolutional shadow-detection network that combines hierarchical features with direction-aware spatial context. The same design is adapted for shadow removal.

  • Network overview: The detection network takes the whole image as input and produces a shadow mask end-to-end using multiple DSC modules across feature scales.Shallow features preserve shadow boundaries, while deep features capture global semantics.
  • Network overview: DSC modules harvest direction-aware spatial context at each CNN layer and combine it with convolutional features to predict layer-specific shadow masks.Predictions from different layers are fused into the final detection result.
  • DSC module: The DSC module aggregates context in four principal directions through two rounds of recurrent translations and attention-weighted directional fusion.The same attention weights are used in both recurrent rounds.
  • Shadow removal: For shadow removal, the detection network is further adopted with shadow-free images as supervision and a Euclidean loss replacing the detection objective.The removal-specific color compensation mechanism is described in Section 3.3.

3.1 Direction-aware Spatial Context

The DSC module extends spatial RNNs to aggregate image context across four directions and selectively combine it with learned attention weights. Two recurrent rounds propagate this context to support shadow recognition and removal.

  • Spatial RNN: A spatial RNN processes 2D feature maps by aggregating local context independently along the left, right, up, and down directions.A 1×1 convolution simulates input-to-hidden translation, followed by directional translations and fusion.
  • Spatial RNN: After two rounds of recurrent translations, each pixel obtains relevant direction-aware global spatial context rather than only local context.The second round further propagates the context aggregated during the first round.
  • Spatial RNN: The recurrent translation uses learned directional weights initialized as identity matrices, with ReLU applied during propagation.For the right direction, the operation is repeated across the feature-map width.
  • Direction-aware attention: The attention mechanism generates four directional weight maps from the input feature maps and selectively weights the corresponding recurrent spatial-context features.The attention estimator uses successive 3×3 convolutions, ReLU, and a 1×1 convolution to produce four channels.
  • DSC module: The DSC module concatenates attention-weighted contexts, reduces their dimensions with a 1×1 convolution, then repeats recurrent translation using the same attention weights.This completes the direction-aware spatial-context construction.

3.2 Our Shadow Detection Network

The shadow-detection network applies DSC modules across VGG-based feature layers and optimizes multiscale shadow masks with a class- and difficulty-aware weighted cross-entropy loss. During testing, layer predictions are averaged and refined with a CRF.

  • Architecture: The network is built on VGG with one DSC module per layer except the first, preserving directional relationships through its fully convolutional architecture.The first layer is excluded because of its large memory footprint.
  • Training objective: Weighted cross-entropy addresses the imbalance caused by shadows occupying smaller image regions than non-shadow areas.Without weighting, overall accuracy would favor the much larger non-shadow regions.
  • Training objective: The loss combines class-distribution weighting with per-class accuracy weighting to emphasize underrepresented and difficult-to-classify shadow or non-shadow pixels.Misclassified shadow pixels receive greater penalty when shadows occupy less area, while difficult classes receive larger weights.
  • Training objective: The overall training loss sums weighted losses from predicted shadow masks at multiple scales, including the MLIF and fusion layers.The layer, MLIF, and fusion weights are empirically set to one.
  • Testing: At test time, the network averages the MLIF- and fusion-layer masks and applies a fully connected CRF to improve spatial coherence among neighboring pixels.Each layer produces a shadow mask with its own supervision signal.

3.3 Our Shadow Removal Network

The shadow-removal network adapts the detection architecture to predict shadow-free images, while color compensation addresses inconsistent training pairs caused by exposure and environmental-lighting variation.

  • Network adaptation: Shadow removal replaces shadow-mask supervision with adjusted shadow-free images and replaces weighted cross-entropy with a Euclidean loss.These modifications train the network to predict shadow-free outputs.
  • Color compensation: A color transfer function is learned for each training pair to adjust the shadow-free ground truth toward the input shadow image’s colors.The function is formulated as a linear transformation and estimated from corresponding non-shadow regions.
  • Color compensation: Training pairs can have inconsistent colors and luminosity because shadow and shadow-free photographs may differ in camera exposure or environmental lighting.Such inconsistencies are observed in the SRD and ISTD datasets.
  • Color compensation: The color-transfer parameters are solved by least squares on non-shadow pixel pairs, then applied to the whole shadow-free image as new supervision.The adjusted image is denoted Tf(In) in the training procedure.
  • Training objective: The Euclidean removal loss is computed over the whole image in LAB color space and applied at each network layer.The overall loss sums contributions from layer, MLIF, and fusion outputs.
  • Testing: At test time, the final shadow-free image is the mean of outputs from the MLIF and fusion layers.Each of these outputs is produced with a corresponding supervision signal.

4 EXPERIMENTS ON SHADOW DETECTION

Experiments evaluate the shadow-detection network on two benchmark datasets, against recent methods, and through network-design analyses. The method performs favorably across datasets and metrics, while analysis identifies benefits from DSC components, two recurrent-translation rounds, shared attention weights, and CRF post-processing, alongside challenging failure cases.

  • Comparison with the State-of-the-art: Our method performs favorably against four recent shadow-detection methods on both datasets for both accuracy and BER.The comparison includes scGAN, stacked-CNN, patched-CNN, and Unary-Pairwise.
  • Comparison with the State-of-the-art: The method generalizes from SBU training to UCF testing and handles challenging cases including adjacent light and dark shadows, complex backgrounds, and black objects.Visual comparisons report fewer false positives for black objects that other methods misrecognize as shadows.
  • Component Analysis: Component analysis finds that multi-scale features with weighted cross entropy improve results, while adding spatial context and DSC features yields further improvement.The baselines remove DSC modules or retain spatial context without direction-aware attention weights.
  • DSC Architecture Analysis: Two recurrent-translation rounds with shared attention weights produce the best DSC architecture result; one round propagates global context insufficiently.The comparison varies recurrent-translation rounds and whether attention weights are shared.
  • Network Design and Additional Results: CRF post-processing reduces BER from 5.68 before CRF to 5.59 after CRF, while inference takes about 0.16 seconds plus 0.5 seconds for CRF.The deeper ResNet-101 has BER 5.73 versus 5.59 for VGG, reflecting stronger semantics but lost detail from smaller feature maps under GPU-memory limits.
  • Additional Results: The method struggles with extremely complex scenes containing many small shadows, large black regions, or soft shadows.These cases respectively involve lost deep-layer detail, insufficient surrounding context, or small differences between shadow and non-shadow regions.

5 EXPERIMENTS ON SHADOW REMOVAL

The shadow removal experiments evaluate DSC and DSC+ on SRD and ISTD using RMSE, visual comparisons, color-space analysis, and failure cases. DSC+ addresses color and luminosity inconsistencies in training pairs and improves agreement with adjusted ground truth, especially on ISTD.

  • Datasets and metrics: The evaluation uses SRD and ISTD benchmark datasets and RMSE in LAB color space, where lower values indicate better shadow removal.SRD contains 2680 training pairs and 408 testing pairs; ISTD covers varied shadow shapes and ground materials.
  • Network variants: DSC is trained with original shadow-free images, whereas DSC+ uses color-adjusted shadow-free images to support fair comparison and compensate for training-pair inconsistencies.The adjustment targets inconsistent colors and luminosity caused by camera exposure and environmental lighting.
  • Visual results: Visual results show that DSC and DSC+ remove shadows while maintaining non-shadow contents, including challenging dark regions and multiple backgrounds.Additional examples include varied background colors, irregular small shadows, and complex backgrounds.
  • Color compensation analysis: 6.66 vs. 6.12 and 8.54 vs. 4.90: DSC+ improves RMSE against adjusted ground truth on SRD and ISTD, respectively, with the clearest improvement on ISTD.Against the original ground truth, the corresponding comparisons are 6.21 vs. 6.66 and 6.67 vs. 8.54.
  • Color compensation analysis: 7.07 and 3.36: DSC and DSC+ obtain these RMSE values on ISTD non-shadow regions, further revealing the effectiveness of DSC+.The comparison evaluates preservation of non-shadow regions using the available ISTD shadow masks.
  • Color space analysis: Both LAB- and RGB-trained networks perform similarly, while the LAB version is slightly better overall and both outperform the state-of-the-art methods in Table 4.The authors therefore choose LAB for their method.
  • Failure cases: The method can over-remove fragmented black floor tiles or fail to recover a handbag’s bright color when surrounding context is misleading or information is lacking.The authors suggest that more training data may help overcome these cases.
  • Time performance: Testing takes around 0.16 seconds per 400×400 image, while training takes around 22 hours on each of SRD and ISTD.Both training and testing use an NVIDIA GeForce TITAN Xp GPU.

6 CONCLUSION

The paper concludes with a direction-aware spatial-context network for single-image shadow detection and removal. Experiments on four benchmark datasets show superiority over state-of-the-art methods, while future work targets broader applications, improved completion, and time-varying shadows.

  • 6 CONCLUSION: The network learns direction-aware spatial context through attention in a spatial RNN and uses multi-layer DSC modules for shadow detection and removal.Detection uses weighted cross entropy, while removal uses Euclidean loss and a color-luminosity compensation mechanism.
  • 6 CONCLUSION: The method is evaluated on two shadow detection and two shadow removal benchmark datasets and shows superiority over state-of-the-art methods for both tasks.The conclusion reports this result across the paper’s detection and removal evaluations.
  • 6 CONCLUSION: Future work includes saliency detection, semantic segmentation, image-completion strategies for shadow removal, and time-varying shadows in videos.These directions are presented as planned extensions of the network and removal approach.
Loading 1805.04635v2…