Source-linked AI summary
ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal
Lanqing Guo, Chong Wang, Wenhan Yang, Siyu Huang, Yufei Wang, Hanspeter Pfister, Bihan Wen
TL;DR
Shadow-removal methods can leave boundary artifacts because degradation priors and image-distribution modeling are not sufficiently integrated. ShadowDiffusion combines a spatially variant degradation model, diffusion-based generative prior, iterative mask refinement, and data-consistent unrolling, with reported superiority on ISTD, ISTD+, and SRD. Its performance can degrade on unusual cases that are scarce in the training set.
Problem
Shadow-removal methods leave boundary artifacts and remaining shadow patterns when degradation priors and natural-image distribution modeling are insufficiently used.
Method
ShadowDiffusion integrates a spatially variant shadow degradation model, diffusion generative prior, dynamic mask-aware refinement, and unrolling-inspired data consistency.
Results
ShadowDiffusion consistently outperforms state-of-the-art methods on ISTD, ISTD+, and SRD datasets.
Takeaways & Limitations
The combined degradation and generative priors support natural shadow-free images without boundary artifacts and shadow patterns within the reported setting.
Takeaways & Limitations
Performance degrades on unusual images, such as stronger-contrast images with face patterns scarce in the training set.
Abstract
from arXiv · showhide
Recent deep learning methods have achieved promising results in image shadow removal. However, their restored images still suffer from unsatisfactory boundary artifacts, due to the lack of degradation prior embedding and the deficiency in modeling capacity. Our work addresses these issues by proposing a unified diffusion framework that integrates both the image and degradation priors for highly effective shadow removal. In detail, we first propose a shadow degradation model, which inspires us to build a novel unrolling diffusion model, dubbed ShandowDiffusion. It remarkably improves the model's capacity in shadow removal via progressively refining the desired output with both degradation prior and diffusive generative prior, which by nature can serve as a new strong baseline for image restoration. Furthermore, ShadowDiffusion progressively refines the estimated shadow mask as an auxiliary task of the diffusion generator, which leads to more accurate and robust shadow-free image generation. We conduct extensive experiments on three popular public datasets, including ISTD, ISTD+, and SRD, to validate our method's effectiveness. Compared to the state-of-the-art methods, our model achieves a significant improvement in terms of PSNR, increasing from 31.69dB to 34.73dB over SRD dataset.
1. Introduction
ShadowDiffusion addresses boundary artifacts and remaining shadow patterns by combining a spatially variant degradation prior with diffusion-based image modeling and iterative shadow-mask refinement. Experiments on ISTD, ISTD+, and SRD report substantial improvements over existing methods.
- Motivation: Existing shadow-removal methods often produce boundary artifacts and remaining shadow patterns because image-distribution modeling and degradation priors are insufficiently exploited.Adversarial approaches may require careful training adjustment, overfit visual features, or hallucinate content and artifacts.
- Proposed approach: ShadowDiffusion introduces a spatially variant shadow model that decomposes degradation into a shadow mask and shadow intensities.The model is designed for complicated, non-uniform lighting conditions rather than a restrictive linear and uniform degradation assumption.
- Proposed approach: ShadowDiffusion jointly restores the shadow-free image and refines the shadow mask through an auxiliary task of the diffusion generator.The mask is progressively refined alongside image restoration, supporting robust removal even with inaccurate mask estimates.
- Proposed approach: An unrolling-inspired diffusive sampling strategy explicitly integrates the shadow degradation prior into the iterative diffusion process.The framework combines degradation and generative priors during restoration.
- Experiments: ShadowDiffusion outperforms state-of-the-art shadow-removal methods on the ISTD, ISTD+, and SRD datasets.The introduction describes the improvement as occurring by large margins and reports generalization to other image-enhancement tasks.
2. Related Work
Prior shadow-removal methods rely on handcrafted or physical assumptions, while diffusion-based restoration is explored here for real-world shadows with limited training pairs. ShadowDiffusion combines iteratively refined mask-aware diffusion with degradation-model consistency.
- Shadow removal: Classic shadow-removal methods use handcrafted priors such as gradients, illumination, and regions, but ideal assumptions can cause boundary artifacts in real-world cases.Deep learning methods instead learn mappings from training pairs, while some retain global physical degradation models.
- Diffusion models: Existing diffusion applications commonly target synthetic degradations with abundant simulated training pairs, unlike this paper’s real-world shadow-removal setting with limited pairs.The paper proposes stricter, iteratively refined conditions for this setting.
- Deep unrolling methods: ShadowDiffusion uses a dynamic mask-aware diffusion model together with an unrolling-inspired process that incorporates the shadow degradation model.The sampling framework alternates diffusion-model sampling with data-consistency operations.
3. ShadowDiffusion
ShadowDiffusion combines a spatially variant shadow degradation prior with a diffusion generator to jointly restore shadow-free images and refine shadow masks. Its unrolling-inspired sampling alternates generative updates with data-consistency steps, progressively integrating both priors.
- Shadow Degradation Model: The proposed shadow degradation model represents illumination as a pixel-wise map decomposed into shadow locations and illumination weights.This spatially variant formulation is richer than a uniform degradation model and explicitly incorporates shadow-mask information.
- Dynamic Mask-Aware Diffusion Model: ShadowDiffusion jointly predicts the shadow-free image and progressively refines the shadow mask, making restoration robust to inaccurate initial masks.Mask refinement is implemented as an auxiliary task of the diffusion generator and is trained with shadow-mask references when available.
- Dynamic Mask-Aware Diffusion Model: The dynamic mask-aware diffusion model learns a conditional reverse process from shadow images, intermediate noisy images, and initial masks to predict noise and refined masks.Training samples triplets consisting of shadow-free images, shadow images, and corresponding initial masks.
- Unrolling-Inspired Diffusive Sampling: Unrolling-inspired sampling alternates diffusion-based image-mask updates with closed-form data-consistency updates derived from the degradation model.The data-consistency steps share information between the observed shadow image and reconstructed variables, while DMDM supplies the generative updates.
- Unrolling-Inspired Diffusive Sampling: The framework uses DDIM-style diffusive sampling to accelerate inference while embedding degradation priors into the diffusion process with almost no additional inference time.The authors describe progressive generation as a natural way to solve the unrolling optimization problem.
4. Experiments
Experiments compare ShadowDiffusion with prior methods across shadow-removal datasets, visual examples, ablations, and related enhancement tasks. Results show strong quantitative performance, improved robustness to mask quality, benefits from unrolling, and extension beyond shadow removal.
- Comparison with State-of-the-Art: 34.73dB PSNR on SRD, versus 31.69dB for BMNet, with shadow-region PSNR increasing from 35.05dB to 38.72dB.The comparison uses 256 × 256 images and reports improvements over the most recent competing method.
- Comparison with State-of-the-Art: ShadowDiffusion outperforms competing methods across the shadow, non-shadow, and whole-image regions on ISTD, SRD, and ISTD+.Tables 1 and 2 report quantitative comparisons across the three datasets.
- Comparison with State-of-the-Art: On complicated SRD examples, prior methods leave illumination inconsistencies, boundary artifacts, and damaged structures, whereas the proposed results better preserve these regions.The cited qualitative discussion highlights colorful and textured cases such as the blue poster and wall examples.
- Ablation Study: Iterative mask refinement reduces the performance decline caused by inaccurate initial masks, improving robustness across masks generated by different detectors.The effect is evaluated on SRD using detected masks as initialization.
- Ablation Study: The complete unrolling model performs better than DMDM without unrolling across all testing sets and produces more accurate exposure and artifact-free results.The comparison includes ISTD, ISTD+, and SRD, with qualitative examples in Figure 6.
- Extension to Other Image Enhancement Tasks: ShadowDiffusion also achieves new state-of-the-art performance for low-light enhancement and exposure correction after removing mask refinement and mask conditions.These tasks treat their degradation as a special case of the proposed degradation model.
5. Conclusion
The paper concludes by combining a spatially variant shadow degradation model with an unrolling diffusion framework that integrates degradation and generative priors. Comprehensive experiments show ShadowDiffusion significantly outperforms state-of-the-art methods on ISTD, ISTD+, and SRD.
- ShadowDiffusion decomposes the shadow degradation map into the shadow mask and shadow intensity.
- The unrolling diffusion framework explicitly integrates degradation prior and diffusive generative prior.
- Mask refinement is treated as an auxiliary task of the diffusion generator to progressively refine the shadow mask.
- Comprehensive experiments demonstrate significant improvement over state-of-the-art methods on ISTD, ISTD+, and SRD datasets.