PaperlayerView exact revision on arXiv

Source-linked AI summary

Shadow Removal by a Lightness-Guided Network with Training on Unpaired Data

Zhihao Liu, Hui Yin, Yang Mi, Mengyang Pu, Song Wang

arXiv:2006.15617v1cs.CV

TL;DR

Paired shadow and shadow-free images are difficult to collect, and unpaired methods have lagged paired-data CNNs on benchmarks. The paper introduces a two-stage lightness-guided network with a colour-prior loss, and reports that it outperforms state-of-the-art methods trained on unpaired data.

  • Problem

    Paired image collection is difficult and time-consuming, while unpaired methods have performed worse than paired-data CNNs on benchmark datasets.

    Source: §I · p. 1

  • Method

    LG-ShadowNet first learns lightness features, then uses multiplicative connections to guide a second module that considers all Lab channels, with an added colour-prior loss.

    Source: §I · p. 2

  • Results

    The method outperforms state-of-the-art methods trained on unpaired data in experiments on ISTD, adjusted ISTD, and USR.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors conclude that the lightness-guided architecture is effective for shadow removal using unpaired data.

    Source: §V · p. 8

Abstract

from arXiv · show

Shadow removal can significantly improve the image visual quality and has many applications in computer vision. Deep learning methods based on CNNs have become the most effective approach for shadow removal by training on either paired data, where both the shadow and underlying shadow-free versions of an image are known, or unpaired data, where shadow and shadow-free training images are totally different with no correspondence. In practice, CNN training on unpaired data is more preferred given the easiness of training data collection. In this paper, we present a new Lightness-Guided Shadow Removal Network (LG-ShadowNet) for shadow removal by training on unpaired data. In this method, we first train a CNN module to compensate for the lightness and then train a second CNN module with the guidance of lightness information from the first CNN module for final shadow removal. We also introduce a loss function to further utilise the colour prior of existing data. Extensive experiments on widely used ISTD, adjusted ISTD and USR datasets demonstrate that the proposed method outperforms the state-of-the-art methods with training on unpaired data.

I. INTRODUCTION

The introduction motivates unpaired shadow removal as easier to train for than paired-data methods, while noting its performance gap. It presents a two-stage, lightness-guided network and reports improved results over prior unpaired methods.

  • The introduction identifies shadow-darkness confusion as a challenge because some non-shadow regions are also dark.This can compromise methods attempting to recover illumination in shadow regions.Source: §I · p. 1
  • Paired shadow and shadow-free images are difficult and time-consuming to collect, while unpaired training remains inferior to paired CNNs on benchmarks.Controlled paired-data collection can lack scene diversity; unpaired models face the challenge of learning from image-level rather than pixel-level annotations.Source: §I · p. 1
  • The method first learns lightness information, then uses it to guide shadow removal while incorporating other colour information.Lightness is learned at feature level to distinguish shadows from dark albedo regions that can also have low lightness.Source: §I · pp. 1–2
  • LG-ShadowNet combines lightness and colour features through multiplicative connections and adds a colour-prior loss to improve shadow removal.The loss encourages colour consistency between generated data and input data.Source: §I · p. 2
  • The proposed method outperforms state-of-the-art methods trained on unpaired data across ISTD, adjusted ISTD, and USR.Source: §I · p. 2

II. RELATED WORK

The paper situates LG-ShadowNet against paired-data CNNs and unpaired Mask-ShadowGAN, then uses a lightness-first module to guide shadow removal across all Lab channels. Its two modules are connected as a teacher–student pair rather than as teammates with the same goal.

A. Shadow removal

  • Earlier CNN shadow-removal methods mostly use paired data, while Mask-ShadowGAN learns mappings between unpaired shadow and shadow-free domains; LG-ShadowNet separates lightness learning from other colour information.The related-work discussion says the proposed unpaired method can achieve comparable performance to state-of-the-art paired-data CNN methods.Source: A. Shadow removal · p. 2

B. Two-stream architecture

  • Unlike the teammate-style streams in, LG-ShadowNet uses Module I as a teacher guiding Module II as a student for lightness guidance and shadow removal.The architecture is structurally similar to the two-stream network in, but the papers address different tasks.Source: B. Two-stream architecture · pp. 2–3

A. Proposed network

  • Module I learns lightness compensation on the Lab L channel, and its learned features guide Module II as it removes shadows using all Lab channels.Module I is trained first and fixed before Module II is trained; Module II's generators are initialized from Module I except for the input and output layers.Source: A. Proposed network · p. 3

B. Loss function

LG-ShadowNet combines identity, cycle-consistency, adversarial, and colour losses. The colour loss compares per-pixel Lab colour directions using cosine similarity.

B. Loss function

  • The training objective combines identity, cycle-consistency, adversarial, and colour losses.The loss weights ω1, ω2, ω3, and ω4 are set to 5, 10, 1, and 10, respectively.Source: B. Loss function · p. 4
  • Identity loss encourages unchanged Lab images, while cycle-consistency loss encourages reconstructed Lab images to match their inputs.The identity and cycle-consistency terms are described as L1 losses.Source: B. Loss function · p. 4
  • The colour loss encourages generated and input Lab colours to align per pixel, using cosine similarity between their colour vectors.The authors chose cosine rather than angle-based colour loss because the latter caused gradient explosion and training failure in their experiments.Source: B. Loss function · p. 4

C. Network details

Both modules use a Mask-ShadowGAN-based backbone, with channel counts reduced to balance performance and efficiency.

C. Network details

  • Each generator has three input/down-sampling convolution layers, nine residual blocks, and three up-sampling/output convolution layers.The residual blocks use stride-two convolutions.Source: C. Network details · p. 4
  • The backbone uses two generators and two discriminators per module, with PatchGAN-based discriminators and instance normalization after each convolution.The general structure follows Mask-ShadowGAN, but the number of parameters differs.Source: C. Network details · p. 4
  • The backbone halves Mask-ShadowGAN's channels following the SqueezeNet principle to balance performance and efficiency.Source: C. Network details · p. 4

D. Multiplicative connections

Multiplicative connections pass Module I feature maps into Module II's residual blocks while preserving Module II's identity mapping.

D. Multiplicative connections

E. Convergence analysis

The convergence analysis argues that the two-module network’s outcome depends on whether Module I supplies useful lightness guidance to Module II. Its reasoning assumes suitable generator and discriminator capacity and optimal components.

E. Convergence analysis

  • The analysis assumes the first-stage generator and discriminator have enough capacity for generated shadow-free data to match the real-data distribution.Under that capacity assumption, it invokes GAN convergence for Module I.Source: E. Convergence analysis · p. 5
  • For Module II, lightness guidance changes feature maps while its generator remains trained to match the real shadow-free data distribution.The argument assumes optimal generator and discriminator components and states that applying the lightness-conditioned generator yields data with the real shadow-free distribution.Source: E. Convergence analysis · p. 5
  • If Module I provides appropriate lightness information, LG-ShadowNet converges better; ineffective guidance leads to worse convergence.The analysis frames this as the practical condition for the second module’s guidance to help.Source: E. Convergence analysis · p. 5

A. Datasets and metrics

The experiments use ISTD and its colour-adjusted version AISTD as paired datasets, alongside unpaired USR. They evaluate paired-data results with Lab-space RMSE and USR visual quality through participant votes.

A. Datasets and metrics

  • ISTD contains 1,870 triplets, split into 1,330 training and 540 testing examples, each comprising a shadow image, mask, and shadow-free image.The dataset varies in illumination, shadow shape, and scene.Source: A. Datasets and metrics · p. 5
  • AISTD uses linearly colour-adjusted ISTD shadow-free images to mitigate colour inconsistency while retaining ISTD images, masks, and splits.The passage treats AISTD as a separate dataset because the adjusted images influence experimental results.Source: A. Datasets and metrics · p. 5
  • USR contains 2,445 shadow images and 1,770 unpaired shadow-free training images, with no corresponding shadow-free image for each shadow image.The dataset has 1,956 shadow images for training and 489 for testing; it spans a thousand scenes.Source: A. Datasets and metrics · p. 5
  • Paired ISTD and AISTD results are evaluated by Lab-space RMSE, while USR is evaluated through five participants’ votes over 150 image comparisons per method pair.USR has no ground truth for RMSE; the higher vote proportion indicates better judged shadow-removal quality.Source: A. Datasets and metrics · p. 5

B. Implementation details

Implementation uses dataset-specific training schedules and a single-GPU PyTorch setup. The connection variants test where and how Module I’s features are combined with Module II.

B. Implementation details

  • Training uses Adam with batch size 1, 400×400 random crops from 448×448 resized images, and dataset-specific epoch schedules.Module I runs 200, 200, and 100 epochs on ISTD, AISTD, and USR; Module II runs 100 epochs on each. Training takes about 68 hours on ISTD using one GTX 1080 GPU.Source: B. Implementation details · p. 5

C. Variants of two-module connections

  • The default multiplicative connections are placed at the first three shallow residual blocks; connections in middle or deeper blocks were omitted because prior work found they hurt performance.The variants also compare corresponding-layer and non-corresponding-layer connections, plus an additive alternative.Source: C. Variants of two-module connections · pp. 5–6
  • LG-ShadowNet-3, -4, and -9 place corresponding-layer connections at intervals of three, two, and one residual blocks, respectively.LG-ShadowNet-N and -P connect the next and previous Module I residual blocks, respectively, to the current Module II block.Source: C. Variants of two-module connections · p. 6
  • LG-ShadowNet-A replaces corresponding-layer element-wise multiplication with addition.Figure 4 depicts nine residual blocks and marks multiplicative connections with red arrows.Source: C. Variants of two-module connections · p. 6, Figure 4 · p. 6

D. Ablation study

Ablations test the lightness-guided architecture, connection choices, and colour loss. On AISTD, adding L-channel guidance lowers RMSE relative to Lab-only Module II, while the preferred connection variant differs across datasets.

D. Ablation study

  • Using L-channel data for Module I gives better AISTD results than V-channel or Lab input and supports the benefit of learned lightness guidance.The comparison reports an 8.3% RMSE reduction from 5.64 to 5.17 for LG-ShadowNet⋆ versus Lab-trained Module II⋆.Source: D. Ablation study · p. 6
  • Training on L + Lab data reduced RMSE by 8.3%, from 5.64 to 5.17, demonstrating the effectiveness of the lightness-guided architecture.Training Modules I and II on paired data in a fully supervised manner can further improve performance.Source: D. Ablation study · p. 6
  • LG-ShadowNet⋆ produces better visual results than either individual module, including shadow removal and lightness restoration in the second-row sample’s top right.Figure 5 compares samples from ISTD, AISTD, and USR; its first two columns show input shadow images and their L channels.Source: D. Ablation study · pp. 6–7, Figure 5 · p. 7
  • LG-ShadowNet-N⋆ performs better on ISTD, suggesting deeper Module I features connected to shallower Module II layers may be more effective there.The variants are evaluated on ISTD and AISTD with RMSE and on USR through user-study votes against Module II⋆.Source: D. Ablation study · p. 7, Table II · p. 7
  • On AISTD, LG-ShadowNet⋆ surpasses the other connection variants, supporting the sufficiency of low-level guidance; embedding more high-level features worsens results, especially outside shadows.On USR, LG-ShadowNet-9⋆ achieves the best result, while additive connections perform worse than multiplicative ones; the default is selected for greater robustness across all three datasets.Source: D. Ablation study · p. 7
  • The colour loss has little effect on overall ISTD RMSE but improves non-shadow-region RMSE, while AISTD and USR show conspicuous improvement.The authors describe the loss as restricting colour direction to remain the same and conclude it is an effective constraint for shadow removal.Source: D. Ablation study · p. 7, Table III · p. 7

E. Comparison with the state-of-the-arts

LG-ShadowNet is compared with prior methods on ISTD, AISTD, and USR, with results favoring it over image-prior and unpaired-data methods. The conclusion reiterates its lightness-guided, unpaired-data approach and reports state-of-the-art performance among methods trained on unpaired data.

E. Comparison with the state-of-the-arts

  • In the USR user study, LG-ShadowNet receives 80.7% of votes against RGB-trained Mask-ShadowGAN and 72.7% against its Lab-trained version.The authors note that Lab input improves Mask-ShadowGAN on USR, but LG-ShadowNet still receives more votes.Source: E. Comparison with the state-of-the-arts · p. 8, Figure 6 · p. 8
  • LG-ShadowNet outperforms image-prior and unpaired-data methods on both ISTD and AISTD, with quantitative results reported in Table IV.The table reports RMSE; ARGAN’s starred results use a different calculation method.Source: E. Comparison with the state-of-the-arts · p. 8, Table IV · p. 8
  • On ISTD, LG-ShadowNet achieves results comparable to paired-data method DSC, while Module II⋆ on Lab data performs better than Mask-ShadowGAN on AISTD with fewer parameters.The authors attribute the latter comparison to the parameter-reduction strategy used for SqueezeNet.Source: E. Comparison with the state-of-the-arts · p. 8
  • Visual comparisons show better shadow-region lightness restoration than Mask-ShadowGAN and better handling of shadow edges than SP+M-Net on AISTD samples.The authors relate the edge comparison to LG-ShadowNet’s continuous lightness guidance versus SP+M-Net’s binary masks.Source: E. Comparison with the state-of-the-arts · p. 8

V. CONCLUSION

  • The conclusion describes a two-module unpaired-data method that learns lightness first, uses it to guide shadow removal with colour information, and adds a colour loss.It reports that LG-ShadowNet outperforms state-of-the-art methods trained on unpaired data.Source: §V · p. 8

Related papers

Loading 2006.15617v1…