PaperlayerView exact revision on arXiv

Source-linked AI summary

Consistency as Regularization for Unsupervised Shadow Removal

Anh-Kiet Duong, Petra Gomez-Krämer, Jean-Michel Carozza

arXiv:2609.01806v1cs.CV

TL;DR

Shadow removal is hindered by the cost of paired data and the continued reliance of unsupervised methods on masks or shadow-free references. ShadowCLR learns directly from shadow images by enforcing consistency across observations, and experiments report competitive or sometimes superior unsupervised results, with a limitation in preserving fine details and small shadow structures.

  • Problem

    Existing supervised methods require paired shadow and shadow-free images, while unsupervised methods often still require shadow masks or shadow-free references that are difficult to obtain.

  • Method

    ShadowCLR combines pairwise reconstruction with global and patch-wise consistency to preserve scene-consistent information and suppress shadow-specific variations without masks or shadow-free images.

  • Results

    ShadowCLR achieves competitive and in some cases superior results compared with unsupervised state-of-the-art methods across multiple benchmarks.

  • Takeaways & Limitations

    Consistency learning provides a practical and effective direction for unsupervised shadow removal without shadow masks or shadow-free references.

  • Takeaways & Limitations

    The method has difficulty preserving very fine details and small shadow structures, with subtle changes such as moving grass causing slight detail degradation.

Abstract

from arXiv · show

Shadow removal is an important preprocessing step for many vision tasks, yet existing supervised methods require paired shadow and shadow-free images, while unsupervised approaches often still rely on shadow masks or shadow-free references. We propose ShadowCLR, an unsupervised framework that learns shadow removal directly from shadow images. Our key observation is that shadows vary across observations while the underlying scene content remains largely consistent. We therefore use consistency across shadow observations as regularization, encouraging the model to recover scene-consistent appearance while suppressing shadow-specific variations. Global and local consistency further enable us to explore visually related images, learn from imperfectly aligned observations, and focus the representation on shared scene information. Experiments on multiple benchmarks show that ShadowCLR achieves competitive and often superior performance over state-of-the-art unsupervised methods, demonstrating that consistency can provide regularization for shadow removal without shadow masks or shadow-free images.

1. Introduction

Shadow removal matters because shadows can damage downstream vision tasks, but existing supervised and unsupervised methods depend on difficult-to-scale paired data, masks, or shadow-free references. ShadowCLR instead learns from consistency across varying shadow observations while preserving scene content.

  • Shadows can cause false detections and reconstruction errors by disrupting illumination, texture, and color statistics.The affected applications include change detection, surveillance, 3D reconstruction, and scene understanding.
  • Supervised shadow removal achieves strong performance but requires costly paired shadow and shadow-free images with careful alignment.These data requirements make scaling to unconstrained scenarios difficult.
  • Unsupervised methods reduce paired-data requirements but typically still depend on shadow masks or unpaired shadow-free images.Such resources remain difficult to obtain at scale.
  • ShadowCLR exploits consistency across observations because shadow patterns vary while underlying scene content remains consistent.The framework learns to retain scene-consistent information and suppress shadow-dependent variations.
  • The framework is designed to learn without target shadow masks or shadow-free images and reports competitive or sometimes superior results against unsupervised state-of-the-art methods.It also performs unsupervised shadow segmentation as a byproduct of shadow removal.

2. Related work

Prior shadow-removal research is divided mainly into supervised and unsupervised approaches, with unsupervised methods commonly retaining dependence on shadow-free references or masks. These dependencies remain important practical barriers, motivating mask-free unsupervised removal and detection for natural images.

  • Shadow-removal methods are mainly classified as supervised or unsupervised according to whether paired shadow and shadow-free supervision is required.The paper focuses on general natural-image shadow removal rather than specialized document or portrait settings.
  • Early and later supervised methods use architectural, adversarial, physical-prior, style-guided, and diffusion-based strategies but still depend on costly paired datasets.The paired-data requirement limits use in less controlled environments.
  • Unsupervised methods either exploit unpaired shadow-free images or avoid them while still requiring shadow masks.Representative methods use adversarial, frequency-domain, mask-guided consistency, or diffusion-based learning.
  • Reliable shadow-free references and accurate masks are difficult to obtain in practice and are often supplied by datasets or supervised detectors.This dependence can limit applicability where annotations are unavailable.
  • Deep unsupervised shadow detection has received relatively limited attention, while mask dependence continues to constrain unsupervised removal.The paper identifies mask-free unsupervised shadow detection or removal as a crucial development direction.

3. Proposed method

ShadowCLR trains shadow removal from consistency among grouped or generated shadow observations, combining pairwise reconstruction with global and patch-wise representation constraints. Self-refining grouping and patch matching address scarce related images and imperfect alignment, while a shadow generator removes the need for available similar observations.

  • Core framework: ShadowCLR uses consistency among different shadow observations as the central principle for unsupervised shadow removal.The framework overview combines pairwise reconstruction and patch-wise correspondence within image groups.
  • Pairwise image loss: Pairwise reconstruction uses one image or a generated shadow variant as a pseudo-target, encouraging outputs toward a group-consistent appearance.Multiple pairs encourage the network to remove shadows that are not consistently present across observations.
  • Pairwise image loss: A sigmoid-gated one-way transformation biases the reconstruction toward brightening shadowed regions rather than merely propagating shadows.The formulation blends the network output with the input and permits small decreases for smooth transitions.
  • Pairwise image loss: The reconstruction loss also addresses regions shadowed in every observation, while its consistency-only form can produce trivial constant outputs.A separate reconstruction consistency term reinforces brighter corrections across paired reconstructions.
  • Pairwise image loss: A shadow generator creates polygonal darkened regions, allowing training when similar observations are scarce or absent.The method can use xi = ˆxi in the one-image-per-group setting.
  • Self-refining image grouping: Self-refining grouping initializes pseudo-labels without known group counts and repeatedly updates them from UNet encoder feature similarities.The shared encoder supplies positive and negative pairs for global contrastive learning, progressively refining groups and representations.
  • Patch-wise correspondence loss: Patch-wise correspondence searches for the most similar local patch rather than enforcing identical spatial locations, improving robustness to camera shifts and moving objects.Negative comparisons use each other image’s mean feature, reducing false-negative constraints from similar objects across groups.
  • Overall framework: The overall objective combines reconstruction, grouping, and patch-wise correspondence terms to preserve scene information while suppressing shadow variations.Global and patch-wise objectives also relax the need for pixel-level alignment.

4. Experiments

ShadowCLR is evaluated across multiple shadow-removal datasets using standard restoration metrics and comparisons with supervised and unsupervised methods. Results show competitive or superior performance, while grouping analysis motivates pairwise consistency over fixed pseudo-targets.

  • Experimental settings: Experiments cover AISTD, INS, WSRD+, LRSS, SRD, and a video shadow-removal dataset, with images resized to 256 × 256.AISTD has 1,330 training and 540 testing images; SRD has 2,680 training and 408 testing images; INS has 30,000 training and 2,000 testing images.
  • Experimental settings: Evaluation uses RMSE over Lab channels and PSNR and SSIM over RGB to measure reconstruction error, visual fidelity, and structural consistency.Lower RMSE and higher PSNR and SSIM indicate better restoration quality.
  • Comparison with the state-of-the-art: ShadowCLR is compared with classic, supervised, and unsupervised shadow-removal methods, using reported results and official implementations where available.The comparison includes Guo et al., seven supervised methods, and eight unsupervised methods.
  • Comparison with the state-of-the-art: On AISTD, ShadowCLR achieves competitive results against state-of-the-art unsupervised methods and surpasses early supervised approaches including DHAN and AEF.The AISTD comparison is reported in Table 1.
  • Comparison with the state-of-the-art: Across SRD, LRSS, WSRD+, and INS, ShadowCLR achieves higher performance across all metrics and datasets than the evaluated unsupervised baselines.INS represents scenarios where similar images are scarce or absent, while several baselines require generated shadow masks for evaluation.
  • Qualitative results: Qualitative results show cleaner shadow removal with better color consistency, scene-detail preservation, surface-texture restoration, and illumination transitions than prior unsupervised methods.These improvements are especially noted in challenging regions partially overlapping dark objects.
  • Grouping quality: ShadowCLR improves grouping quality to an ARI of 0.918, but imperfect groups and singleton scenes make fixed representative images unreliable as pseudo-targets.The ground-truth grouping has 89 groups, while the trained method produces 119 groups; pairwise consistency is therefore used instead of fixed group representatives.

4.5. Ablation studies

Ablations on AISTD show that each major component contributes to ShadowCLR, with feature-level consistency and grouping improving restoration and detection. Parameter choices trade off preservation of non-shadow regions against shadow restoration.

  • Component ablations: Each evaluated component contributes to performance on the AISTD ablation benchmark.The study reports ARI gain before and after training across component and parameter variants.
  • Component ablations: Replacing patch-wise loss with LPIPS or fine-tuning DINOv3 instead of directly optimizing the UNet reduces performance.The comparison supports jointly improving the restoration model through contrastive learning.
  • Grouping and consistency: DINOv3 grouping and one-image-per-group outperform using only Lr with the shadow generator, especially for shadow detection and restoration.This suggests feature-level consistency learns more general appearance representations across different ground-truth groups.
  • Parameter settings: s = 64 better preserves non-shadow regions, whereas s = 256 better restores shadows.The gating parameter therefore controls a trade-off between non-shadow fidelity and shadow removal.
  • Unsupervised shadow segmentation: On SBU, more complex shadows across diverse surfaces lead to lower performance than on AISTD for both supervised and unsupervised methods.The segmentation evaluation uses the gating map g as a proxy soft shadow matte.

4.7. Video shadow removal

ShadowCLR is evaluated on video sequences with static backgrounds and moving shadows, using moving-shadow regions for quantitative assessment. The method's documented failure cases concern fine details and small shadow structures, which may appear blurred.

  • Dataset and protocol: The video dataset contains static scenes with consistent backgrounds and provides a pseudo shadow-free reference frame from maximum pixel intensities.ShadowCLR is applied directly after training on AISTD, with an additional one-epoch fine-tuned variant.
  • Evaluation protocol: Quantitative metrics are computed only within moving-shadow regions, where pixels alternate between shadowed and nonshadowed states.The official moving-shadow mask uses threshold 80; RMSE* additionally uses threshold 40.
  • Results: The qualitative video results are reported in Figure 7.The figure is associated with the video shadow removal evaluation.
  • Failure cases: Fine details and small shadow structures may appear slightly blurred in failure cases.The limitation is attributed to the learning strategy, although the method still achieves higher PSNR than G2R in the cited comparison.

5. Conclusion

The paper concludes that ShadowCLR enables unsupervised shadow removal without shadow masks or shadow-free references by enforcing consistency at multiple levels. Experiments indicate competitive and sometimes superior performance against unsupervised state-of-the-art methods.

  • Conclusion: ShadowCLR learns shadow removal without relying on shadow masks or shadow-free references.The framework uses pixel-, global-, and patch-wise consistency to restore illumination and remove shadows self-supervisedly.
  • Conclusion: Consistency learning is presented as a practical and effective direction for unsupervised shadow removal.The conclusion bases this assessment on extensive benchmark experiments against unsupervised state-of-the-art approaches.

6. Iterative blending strategy

The iterative blending extension treats shadows as structured noise and progressively refines illumination over multiple inference steps. It selects visually consistent intermediate outputs and reports higher restoration quality at substantially lower computational cost than the cited diffusion-based method.

  • Diffusion formulation: The reverse diffusion formulation iteratively recovers a clean image from Gaussian noise through parameterized Gaussian transitions.The transition mean and variance define the reverse step, with the mean reparameterized using predicted noise.
  • Iterative refinement: The extension applies a deterministic, diffusion-inspired refinement process that progressively removes structured shadow noise over T inference steps.Each iteration partially updates the image using the pretrained shadow removal network.
  • Output selection: DINOv3 similarity selects the most visually consistent intermediate result during reverse refinement.The method uses T = 100 inference steps and includes DINOv3 in the reported computational estimate.
  • Efficiency and performance: 12 TFLOPs versus 61 TFLOPs: the iterative extension requires lower computational cost than the cited diffusion-based unsupervised method.The table reports higher performance on most metrics for the proposed extension.
  • Qualitative comparison: The diffusion-based extension yields cleaner illumination recovery and fewer residual artifacts than the base model in qualitative comparisons.The cited comparison includes DC-ShadowNet as an additional visual reference because the existing diffusion-based method does not release outputs.

7. Contrastive loss weight analysis

The stop-gradient confidence weighting scales contrastive-learning updates according to pair reliability. It suppresses uncertain global and local correspondences while strengthening supervision for consistent pairs, supporting self-refining grouping.

  • Global confidence weighting: The detached confidence weight scales the contrastive gradient by exactly 1 + c_i.Because w_i is detached, it acts as a constant during backpropagation.
  • Global confidence weighting: Low-similarity pairs receive smaller updates, whereas confident positive pairs contribute more strongly to representation learning.
  • Global confidence weighting: As c_i approaches −1, an unreliable pair’s contribution approaches zero; as c_i approaches 1, it approaches twice the standard objective.
  • Self-refining grouping: The weighting supports self-refining grouping by reducing the influence of uncertain pseudo-group assignments as representations improve.More consistent positive features receive larger weights during later learning.
  • Local correspondence weighting: The same confidence mechanism applies to patch-wise correspondence loss, conservatively updating uncertain local matches and strengthening reliable ones.Local positives are corresponding patches, while negatives use mean features from images in other groups.

8. Additional results

Additional experiments examine qualitative behavior, parameter sensitivity, clustering, reconstruction-loss terms, and shadow-generation utility across multiple datasets. The results show stable parameter behavior, data-dependent grouping and loss effects, and strong qualitative shadow removal across diverse settings.

  • Parameter sensitivity: PSNR and RMSE remain stable across a wide range of s, indicating limited sensitivity to the sigmoid-gating parameter after calibration.Very small or large s can cause under-correction or over-enhancement.
  • Clustering sensitivity: Using 10–20% of the training data for Affinity Propagation yields performance close to full-data clustering, while smaller subsets degrade performance.
  • Reconstruction loss: The reconstruction-loss terms have dataset-dependent effects: group consistency helps more on AISTD, whereas self-reconstruction is relatively more important on SRD.These differences reflect stronger within-scene consistency on AISTD and greater illumination and appearance variation on SRD.
  • Shadow generation: Without similar observations, shadow generation raises one-image-per-group performance from 26.94 PSNR and 0.915 SSIM to 30.62 PSNR and 0.951 SSIM.When similar images are available, the generator has limited effect because observed shadows are more realistic than synthetic ones.
  • Dataset conditions: The INS benchmark is more challenging than SRD because it has fewer images per scene, larger interimage variation, and often small self-shadows.SRD groups mostly contain related real images with overlapping scene regions.
  • Qualitative results: ShadowCLR removes shadows while preserving fine reflections and illumination cues, sometimes producing reflection consistency superior to the ground truth.
  • Qualitative results: Qualitative examples across all datasets cover both shadow removal and segmentation, with frequent visual improvements over supervised approaches.Some outputs are reported as more visually consistent than the ground truth.
Loading 2609.01806v1…