PaperlayerView exact revision on arXiv

Source-linked AI summary

CANet: A Context-Aware Network for Shadow Removal

Zipei Chen, Chengjiang Long, Ling Zhang, Chunxia Xiao

arXiv:2108.09894v1cs.CV

TL;DR

Shadow removal remains challenging because shadows degrade computer-vision tasks and geometry and illumination interact. CANet matches shadow and non-shadow patches, transfers contextual features between them, and refines the result in a second stage. The authors report effective, realistic results and superior performance to state-of-the-art methods.

  • Problem

    Shadow removal is challenging because geometry and illumination interact, while shadows can degrade the accuracy of other computer-vision tasks.

    Source: §1 · p. 1

  • Method

    CANet uses contextual patch matching to identify shadow and non-shadow relationships, transfers multiscale features from non-shadow to shadow regions, and refines the result with an encoder-decoder.

    Source: §1 · p. 1, §3.1 · p. 4

  • Results

    The authors report effective, realistic shadow-removal results and superior performance to state-of-the-art methods.

    Source: §1 · p. 2, §4.3 · p. 8

  • Takeaways & Limitations

    CANet removes shadows in complex real-world scenes, although its video-level frame results do not ensure continuity.

    Source: §4.3 · p. 8

    CANet may fail to recover illumination consistent with the surroundings when shadow and non-shadow regions lack strong contextual correlation.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

In this paper, we propose a novel two-stage context-aware network named CANet for shadow removal, in which the contextual information from non-shadow regions is transferred to shadow regions at the embedded feature spaces. At Stage-I, we propose a contextual patch matching (CPM) module to generate a set of potential matching pairs of shadow and non-shadow patches. Combined with the potential contextual relationships between shadow and non-shadow regions, our well-designed contextual feature transfer (CFT) mechanism can transfer contextual information from non-shadow to shadow regions at different scales. With the reconstructed feature maps, we remove shadows at L and A/B channels separately. At Stage-II, we use an encoder-decoder to refine current results and generate the final shadow removal results. We evaluate our proposed CANet on two benchmark datasets and some real-world shadow images with complex scenes. Extensive experimental results strongly demonstrate the efficacy of our proposed CANet and exhibit superior performance to state-of-the-arts.

1. Introduction

Shadow removal remains challenging because geometry and illumination interact, while learning-based methods may overlook contextual matching between shadow and non-shadow regions. CANet addresses this with a two-stage network that matches patches, transfers contextual features, and refines the result; the authors report effective, realistic shadow removal.

1. Introduction

  • Shadow removal remains challenging because geometry and illumination interact, and shadows can degrade the accuracy of other computer-vision tasks.Source: §1 · p. 1
  • Existing learning-based methods may overlook contextual matching information when they focus mainly on increasing receptive fields.Source: §1 · p. 1
  • CFT transfers multiscale features from non-shadow to shadow patches using CPM matches; Stage-I recovers L and A/B channels, and Stage-II refines them into the final image.Source: §1 · pp. 1–2
  • CANet is an end-to-end, two-stage network that transfers contextual information from non-shadow to shadow regions at embedded feature spaces.Source: §1 · pp. 1–2
  • CPM predicts potential contextual relationships between shadow and non-shadow patches, while its pair-type classification filters irrelevant matches.Source: §1 · pp. 1–2
  • CANet’s quantitative and qualitative experiments demonstrate effectiveness and superior performance in generating realistic shadow-removal images.Source: §1 · p. 2

2. Related work

Prior shadow-removal work includes physical models, information transfer, and learned contextual features, but these approaches face limitations in artifacts or matching context. CANet instead combines patch matching and contextual feature transfer in a two-stage pipeline.

2. Related work

  • Physical-based methods use prior knowledge to recover shadow-region illumination, but illumination changes can leave visible shadow-boundary artifacts.Source: §2 · p. 2
  • Information-transfer methods can produce pleasant results, but their effectiveness depends on accurate texture matching.Source: §2 · p. 2
  • CANet combines contextual patch matching and feature transfer at Stage-I, then uses a second stage to generate the final shadow-removal result.Source: Figure 1 · p. 3
  • Learning-based methods use contextual features, yet the cited approaches may focus on receptive fields while ignoring paired matching information in images.Source: §2 · pp. 2–3

3. Method

CANet transfers context from non-shadow to shadow regions through patch matching and feature-space transfer, then refines the recovered channels with an encoder-decoder. Its stages combine learned patch relationships, multiscale transfer, and a final image-consistency refinement.

3. Method

  • CANet first recovers L and A/B channels using transferred non-shadow context, then feeds those results with the shadow image to a DenseUNet refinement stage.Stage-II uses an encoder-decoder to predict the final shadow-removal image.Source: §3 · p. 3, §3.3 · p. 5

3.1. Contextual Patch Matching Module

  • CPM uses a shared learned patch descriptor with correlation-regression and pair-type classification heads to identify relevant shadow/non-shadow patch pairs.It uses 32 × 32 patches from the input and a lightness-averaged, mean-filtered shadow-unaware image; the type head distinguishes same-region pairs from ordered cross-region pairs.Source: §3.1 · p. 4

3.2. Contextual Feature Transfer Mechanism

  • CFT samples contextual features from matched non-shadow patches with Gaussian sampling, then integrates the top-k results according to pair correlation.The authors note that Gaussian sampling incorporates surrounding information through a larger receptive field.Source: §3.2 · p. 5

3.3. Shadow Removal with Two-stage Strategy

  • Stage-I uses a pretrained DenseNet to extract multiscale context and removes shadows separately in the L and A/B channels.The separation is intended to avoid over-processing A/B and under-processing L; Stage-II uses the recovered channels as guidance because inaccurate matches may leave undesirable areas.Source: §3.3 · p. 5
  • CANet training combines removal, perceptual, and gradient losses to enforce visual consistency, preserve image structure, and encourage smooth results.The removal loss compares both stage outputs with the ground truth.Source: §3.3 · pp. 5–6

4. Experiments

CANet is evaluated against recent methods on two benchmark datasets, through component ablations, and on real-world images and video. It outperforms state-of-the-art methods across the reported dataset regions, while discussion identifies limitations in illumination recovery, color consistency, and video continuity.

4. Experiments

  • Evaluation uses the ISTD and SRD benchmark datasets and compares learning-based methods using the same training data and 400 × 400 inputs.ISTD has 1,330 training triples and 540 test triplets; SRD has 2,680 training pairs and 408 test pairs.Source: §4 · p. 6, §4.1 · p. 6

4.1. Comparison with State-of-the-arts

  • CANet outperforms state-of-the-art methods in shadow, non-shadow, and whole-image RMSE on both ISTD and SRD.The comparison uses Table 1; the supplied passage does not give the individual RMSE values.Source: Table 1 · p. 6, §4.1 · p. 6
  • CANet produces more realistic shadow-removal results than the compared methods, whose outputs include artifacts, boundary illumination errors, color distortion, or excessive shadow removal.Figure 5 shows input images, methods’ outputs, CANet outputs, and ground truth; the prose describes the qualitative differences.Source: §4.1 · pp. 6–7, Figure 5 · p. 7

4.2. Ablation study

  • Ablations indicate that contextual mapping improves accuracy, CPM supports matching quality, and the proposed CFT yields CANet’s best performance.Visual variants show artifacts from incorrect mapping, removing CFT, or directly replacing contextual features without Gaussian sampling.Source: §4.2 · p. 7, Figure 6 · p. 7

4.3. Discussion

  • On collected real-world shadow images with complex scenes, CANet’s results look more realistic than those of the compared algorithms.Figure 7 presents the input and results from ST-CGAN, DSC, ARGAN, RIS-GAN, and CANet.Source: §4.3 · pp. 7–8, Figure 7 · p. 8
  • Processing video frames separately removes shadows at frame level, but does not ensure continuity across frames.The authors identify video continuity as future work; Figure 9 samples one result every 0.5 seconds.Source: §4.3 · p. 8, Figure 9 · p. 8
  • CANet may fail to recover surrounding-consistent illumination without strong shadow/non-shadow context, and inconsistent training colors or luminosity may produce color inconsistency.Figure 8 illustrates an input region without a matching non-shadow position, making its light difficult to recover.Source: §4.3 · p. 8, Figure 8 · p. 8

5. Conclusion

CANet uses CPM and CFT in Stage-I, then an encoder-decoder refinement in Stage-II to produce shadow-removal results. Experiments confirm the method’s effectiveness and superiority, and the framework may be extended to highlight removal as future work.

5. Conclusion

  • CANet pairs Stage-I contextual patch matching (CPM) with contextual feature transfer (CFT), then refines those results with a Stage-II encoder-decoder.CPM searches for potential match pairs for CFT.Source: §5 · p. 8
  • Extensive experiments confirm the effectiveness and superiority of CANet for shadow removal.Source: §5 · p. 8
  • The framework may be extended to highlight removal, which the authors identify as future work.Source: §5 · p. 8

A. Details about Network Architectures

The CPM architecture has a feature extractor, pair-type classifier, and correlation-degree regressor; the paper also illustrates CANet’s network architecture.

A. Details about Network Architectures

B.1. Effectiveness of Light-unaware Images

The supplementary light-unaware image reduces differences between shadow and non-shadow regions. Its contrast with the shadow image also indicates patch type for CPM’s classifier.

B.1. Effectiveness of Light-unaware Images

  • The supplementary light-unaware image largely reduces the difference between shadow and non-shadow regions, avoiding matching errors caused by shadows.Source: §B.1 · p. 11
  • The shadow-image/light-unaware-image difference is larger in shadow regions and smaller in non-shadow regions, indicating patch type for the classifier.Source: §B.1 · p. 11

B.2. Large-scale Training Dataset for CPM

The CPM training collection uses patch pairs from ISTD and SRD, constructed through spatial matches or similarity-based matches within shadow images.

B.2. Large-scale Training Dataset for CPM

  • The CPM collection contains more than 360,000 ISTD patch pairs and more than 600,000 SRD patch pairs, with each dataset split evenly between matches and non-matches.Source: §B.2 · p. 11
  • Pairs are collected either by matching a shadow patch to the same-position patch in a shadow-free image or by selecting patches within a shadow image.Source: §B.2 · p. 11
  • For within-image pairs, cosine similarity above 0.95 marks matches and similarity below 0.6 marks non-matches in the corresponding shadow-free image.Source: §B.2 · p. 11
  • Because SRD lacks shadow-mask ground truth, the authors use DSD results and manually select correct detections as approximate ground truth.Source: §B.2 · p. 11

C. More Visual Shadow Removal Results

The section presents visual shadow-removal comparisons of CANet with six state-of-the-art methods, alongside figures illustrating the network and its patch-matching approach.

C. More Visual Shadow Removal Results

  • Figure 15 compares CANet’s shadow-removal results with Guo, Zhang, ST-CGAN [35], ARGAN [5], DSC [15], and RIS-GAN [42].The figure places input images first, the six methods’ results next, and corresponding shadow-free ground truths last.Source: C. More Visual Shadow Removal Results · p. 11, Figure 15 · p. 13
  • Figures 10–14 illustrate CANet’s architecture, light-unaware images, and two approaches for collecting matched or non-matched patch pairs.The light-unaware image is described as eliminating the shadow-caused difference between regions A and B.Source: Figure 10 · p. 12, Figure 11 · p. 12, Figure 12 · p. 12
  • Figure 15 includes corresponding shadow-free ground-truth images alongside the method outputs.This gives a visual reference for comparing each method’s removal results with the target images.Source: Figure 15 · p. 13

Related papers

Loading 2108.09894v1…