Source-linked AI summary
Shadow Removal via Shadow Image Decomposition
Hieu Le, Dimitris Samaras
TL;DR
Shadow removal is difficult because shadows can resemble material changes, while end-to-end deep networks may ignore physical shadow properties and produce blurry or artifact-filled images. The paper combines a simplified linear illumination model with SP-Net and M-Net to estimate shadow parameters and matte layers, then evaluates the system and synthetic-data augmentation on ISTD. Compared with the state-of-the-art method, shadow-area RMSE falls from 13.3 to 7.9, and augmented training lowers it further to 7.4.
Problem
Shadow removal must address confusing shadow boundaries and dark materials, while existing physical and end-to-end approaches have practical or image-quality limitations.
Method
The method combines a simplified linear illumination model with SP-Net for shadow parameters, M-Net for the shadow matte, and model-based synthetic shadow augmentation.
Results
40% error reduction in shadow-area RMSE, from 13.3 to 7.9, is reported against the state-of-the-art method, with augmented training further lowering RMSE to 7.4.
Takeaways & Limitations
The framework achieves state-of-the-art shadow removal on ISTD while using a physical illumination model and generating additional training images through parameter modification.
Takeaways & Limitations
The ISTD testing set requires adjustment because its shadow and shadow-free image pairs have inconsistent colors.
Abstract
from arXiv · showhide
We propose a novel deep learning method for shadow removal. Inspired by physical models of shadow formation, we use a linear illumination transformation to model the shadow effects in the image that allows the shadow image to be expressed as a combination of the shadow-free image, the shadow parameters, and a matte layer. We use two deep networks, namely SP-Net and M-Net, to predict the shadow parameters and the shadow matte respectively. This system allows us to remove the shadow effects on the images. We train and test our framework on the most challenging shadow removal dataset (ISTD). Compared to the state-of-the-art method, our model achieves a 40% error reduction in terms of root mean square error (RMSE) for the shadow area, reducing RMSE from 13.3 to 7.9. Moreover, we create an augmented ISTD dataset based on an image decomposition system by modifying the shadow parameters to generate new synthetic shadow images. Training our model on this new augmented ISTD dataset further lowers the RMSE on the shadow area to 7.4.
1. Introduction
The paper combines a simplified physical illumination model with deep networks to remove and synthesize shadows, achieving improved shadow-area RMSE on ISTD.
- Shadows can confuse segmentation, tracking, and recognition because shadow edges resemble material boundaries and dark albedo regions may resemble shadows.
- The method combines a simplified physical illumination model with deep learning to define mappings between shadowed and shadow-free pixels.The model uses a linear transformation with per-channel scaling factors and additive constants for the umbra area.
- SP-Net estimates shadow-model parameters, while the image decomposition formulation combines the shadow image, model parameters, and shadow matte to reconstruct the shadow-free image.The shadow matte handles penumbra effects, where illumination changes gradually across the shadow boundary.
- 29% lower shadow-area RMSE, from 13.3 to 9.5, is achieved by SP-Net alone compared with the state-of-the-art method on ISTD.The full system further improves the overall results by another 17%, yielding 7.9 RMSE.
- 6% lower shadow-area RMSE is obtained by training on ISTD plus synthesized images compared with training on the original ISTD dataset.The augmentation modifies estimated shadow parameters to generate additional synthetic shadow images.
- The paper contributes a deep-learning shadow-removal approach, a model-based shadow-image augmentation method, and state-of-the-art results on ISTD.
2. Related Works
Prior shadow-removal work spans physical illumination models, shadow matting, and end-to-end deep networks; this framework combines a physically inspired model with automated deep-network estimation.
- Shadow Illumination Models: Physical approaches estimate illumination-model parameters to reverse shadow effects, but related work also studies matting for soft shadows.Matting methods address soft shadows, including video-based matte estimation and shadow-removal frameworks using shadow mattes.
- Deep-Learning Based Shadow Removal: Deep-learning methods use large-scale datasets to map shadow images to shadow-free images, including systems that predict shadow mattes or use conditional generative networks.One prior matte differs from this paper’s matte because it includes both shadow density and color offset.
- Deep-Learning Based Shadow Removal: This framework combines a physically inspired shadow illumination model and a shadow matte while estimating both through two deep networks automatically.The approach follows early physical-modeling work but replaces manual parameter and matte estimation with learned prediction.
3. Shadow and Image Decomposition Model
The model represents shadow removal as linear relighting plus image decomposition: global color-channel parameters produce a relit image, while a per-pixel matte blends it with the shadow image.
- 3.1. Shadow Illumination Model: The method models each color channel’s lit intensity as a linear transformation of its shadowed value using scaling factors w and additive constants b.The parameters are assumed constant across all pixels in the umbra, enabling estimation by linear regression from paired shadow and shadow-free images.
- 3.2. Shadow Image Decomposition System: The decomposition expresses the shadow-free image as a combination of the shadow image, relit image, and shadow matte α.The matte provides per-pixel coefficients for combining the relit and input images.
- 3.1. Shadow Illumination Model: The illumination transformation maps shadowed pixels to their shadow-free values, producing the relit image Irelit.The transformation is based on the estimated shadow parameters.
- 3.2. Shadow Image Decomposition System: The matte is ideally 1 outside shadows, 0 in the umbra, and gradually varying across the penumbra.This spatially varying coefficient models the soft transition near shadow boundaries.
- 3.2. Shadow Image Decomposition System: The framework predicts the unknown shadow parameters and matte with SP-Net and M-Net, then reconstructs the shadow-free image through the decomposition.The supplied framework description specifies SP-Net’s parameter prediction and M-Net’s matte prediction before final reconstruction.
4. Shadow Removal Framework
The framework uses SP-Net to estimate illumination parameters and M-Net to estimate a soft shadow matte, addressing penumbra artifacts that binary masks cannot represent.
- 4.1. Shadow Parameter Estimation Network: SP-Net takes the shadow image and mask to predict parameters (w, b), which generate the relit image used downstream.The network directly predicts the linear-model parameters from the input shadow image.
- 4.1. Shadow Parameter Estimation Network: SP-Net training parameters are obtained by least-squares regression after eroding the shadow mask to exclude penumbra pixels.The erosion defines a region intended to contain fully shadowed pixels for parameter estimation.
- 4.1. Shadow Parameter Estimation Network: At test time, the framework estimates shadow masks with Zhu et al.’s shadow-detection network instead of using ground-truth masks.Ground-truth masks are used during SP-Net training, while estimated masks are used during testing.
- 4.2. Shadow Matte Prediction Network: Binary masks cannot model gradual penumbra illumination changes, so using them in decomposition produces visible boundary artifacts.The linear model handles fully shadowed umbra pixels, whereas partially shadowed penumbra pixels require matting.
- 4.2. Shadow Matte Prediction Network: M-Net receives the shadow image, relit image, and mask, and is trained with reconstruction loss to predict a matte that avoids shadow-boundary artifacts.The matte acts as a relaxed shadow mask whose values represent shadow-effect strength.
5. Experiments
Experiments evaluate the method on an adjusted ISTD benchmark, compare its components with prior approaches, and test synthetic shadow augmentation. The full system improves shadow-area RMSE, while remaining failure cases include over-lighting and incorrect colors.
- Dataset and evaluation: ISTD contains 1870 training triplets from 135 scenes and 540 testing triplets from 45 scenes.Each triplet includes a shadow image, shadow mask, and shadow-free image.
- Dataset and evaluation: The ISTD test pairs require color correction because shadow and shadow-free images were captured at different times, producing inconsistent environment lighting.The non-shadow RMSE between paired images is 12.9 in one example and 6.83 across the testing set before correction.
- Dataset and evaluation: RMSE in LAB space is measured separately on shadow, non-shadow, and whole-image regions after resizing outputs to 256 × 256.The method itself accepts input images of any size, but evaluation follows the benchmark resolution.
- Shadow removal evaluation: 29% lower shadow-area RMSE, from 13.3 to 9.5, is achieved by SP-Net alone versus the state of the art, while SP-Net plus M-Net reaches 7.9.The full system provides another 17% improvement over SP-Net alone and removes boundary artifacts through matting.
- Shadow removal evaluation: The method better estimates overall illumination changes and avoids color inconsistencies in relit areas, whereas ST-CGAN produces blur and random artifacts.The comparison is qualitative and includes several prior shadow-removal methods.
- Dataset augmentation via shadow editing: Varying shadow parameters generates synthetic images: four scaling factors expand the ISTD training set by 5320 images, and augmented training lowers shadow-area RMSE from 7.9 to 7.4.Lower scaling factors create lighter shadows, while higher factors increase shadow effects.
6. Conclusions
The framework uses deep networks to estimate illumination-model parameters for single-image shadow removal, achieving state-of-the-art results on ISTD. Its current physical model can be extended toward greater realism and shadow editing.
- Deep networks estimate illumination-model parameters automatically from a single image, rather than producing per-pixel output values.This combines physical illumination modeling with learned parameter estimation.
- The framework produces high-quality shadow removal results without artifacts and achieves state-of-the-art performance on the ISTD dataset.
- A more physically plausible illumination model could help the framework output more realistic images.
- The approach could be extended to shadow editing through a deep-learning framework based on a physical illumination model.