PaperlayerView exact revision on arXiv

Source-linked AI summary

After a Decade: Bringing Shadow Removal into the Real World with Agentic Training Data

Shilin Hu, Jingyi Xu, Dimitris Samaras, Hieu Le

arXiv:2609.38607v1cs.CVcs.AI

TL;DR

Diverse paired shadow-removal data are difficult to capture because a valid target must preserve the scene and capture conditions while removing the shadow. The paper uses a physics-motivated agentic workflow to build AgenticShadow from real image-mask pairs. The workflow reduces CDD by 50.5%, and training existing models on the dataset reduces cross-domain macro LAB RMSE by 19.7–37.5%.

  • Problem

    Shadow removal remains brittle beyond established benchmarks, while collecting diverse paired data requires removing shadows without changing the camera, scene, or illumination conditions.

    Source: §1 · p. 1

  • Method

    A physics-motivated agentic workflow generates, evaluates, retries, filters, and selects edits from real image-mask pairs, then applies deterministic regional correction to construct AgenticShadow.

    Source: §1 · pp. 2–3

  • Results

    The workflow reduces CDD by 50.5% against the strongest previous method, and AgenticShadow training reduces cross-domain macro LAB RMSE by 19.7–37.5% across three existing models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AgenticShadow provides 17,138 triplets across general scenes, faces, and remote sensing, and its supervision improves representative architectures across domains.

    Source: §1 · p. 3, §6 · p. 10

    AgenticShadow targets are controlled approximations, not physical ground truth, and do not guarantee a unique physical solution.

    Source: §1 · p. 2

Abstract

from arXiv · show

Shadow removal looks nearly solved on established benchmarks, yet remains brittle in the real world. Models have advanced; the paired training data they rely on have barely changed in nearly a decade. The reason is simple: obtaining a shadow-free target requires removing the occluder while keeping the scene, camera, and illumination otherwise unchanged, making diverse paired data difficult to capture. Meanwhile, large shadow detection datasets already contain diverse real-world images and masks, but no shadow-free targets. To turn this abundant but incomplete data into paired supervision, we propose an offline agentic workflow combining physics-motivated generation, failure detection, feedback-driven retry, candidate selection, and deterministic correction. Using this workflow, we construct AgenticShadow, a dataset of 17,138 image-mask-target triplets spanning general scenes, faces, and remote sensing. Our construction workflow reduces Color Distribution Difference by 50.5% over previous shadow removal work, while training existing shadow removal models on AgenticShadow reduces cross-domain LAB RMSE by 19.7-37.5%.

1 INTRODUCTION

Shadow removal performs well on established paired benchmarks but remains brittle across diverse real-world scenes, while collecting varied paired targets is difficult. AgenticShadow addresses this gap with an agentic construction workflow and reports improved target fidelity and cross-domain training results.

1 INTRODUCTION

  • Although recent methods appear close on ISTD+, their performance does not reliably transfer to diverse real-world images with large or distributed shadows, mixed illumination, faces, and aerial scenes.Reported failure modes include incomplete removal, color shifts, and structural artifacts.Source: §1 · p. 1
  • Valid paired capture must remove the shadow while holding camera, scene geometry, materials, exposure, and other illumination conditions unchanged, constraining scale and domain coverage.The paper treats its targets as controlled approximations rather than physical ground truth, since exact shadow-free counterparts may be unobtainable in unconstrained scenes.Source: §1 · pp. 1–2
  • AgenticShadow contains 17,138 image-mask-target triplets spanning general scenes, faces, and remote sensing, with broader real-world coverage than constrained paired datasets.Figure 1 contrasts constrained acquisition in ISTD and SRD and controlled close-range WSRD with AgenticShadow’s broader scenes, viewpoints, shadow appearances, and domains.Source: §1 · p. 2, Figure 1 · p. 2
  • The workflow turns real shadow image-mask pairs into targets through physics-motivated generation, failure detection, feedback-guided retries, candidate filtering and selection, and deterministic regional correction.These controls target scene-content changes, retained shadow structure, structural changes, and residual brightness or color errors.Source: §1 · pp. 2–3
  • AgenticShadow training reduces cross-domain macro LAB RMSE by 19.7–37.5% across ShadowDiffusion, HomoFormer, and PhaSR, while the workflow reduces CDD by 50.5% against the strongest previous method.The workflow also achieves LAB RMSE of 3.85 against color-corrected ISTD+ ground truth without training on ISTD+ or using its references during construction.Source: §1 · p. 2

2 RELATED WORK

Prior shadow-removal methods and datasets have advanced, but generalization and paired supervision across varied real-world domains remain limited. This work combines broad real-world image sources with generated shadow-free targets.

2 RELATED WORK

  • Shadow-removal methods have progressed from illumination invariance, matting, and texture consistency to transformers, inpainting, diffusion, and semantic or geometric priors, but cross-scene generalization remains underexplored.Examples include early illumination-invariance methods and recent diffusion approaches.Source: §2 · p. 3
  • Paired SRD and ISTD datasets remain modest and limited in scene, viewpoint, and shadow diversity, while WSRD uses controlled close-range capture; synthetic data add scale but retain a domain gap with real images.Real-world detection collections cover varied domains but typically provide only shadow images and masks, not shadow-free counterparts.Source: §2 · p. 3
  • Instruction-guided editors can make localized edits but do not explicitly model shadow formation and may retain shadows, alter scene content, or introduce physically inconsistent changes.The paper adapts generation-plus-validation approaches through physics-motivated candidate evaluation, failure-driven retries, selection, and regional correction.Source: §2 · p. 3

3 AGENTIC DATASET CONSTRUCTION

The construction workflow generates shadow-free proposals from image-mask pairs, evaluates and selects candidates, then applies regional color correction. Its checks combine shadow-formation grounding, structural diagnostics, and visual judgment.

3 AGENTIC DATASET CONSTRUCTION

  • GPT Image 2 generates candidate edits, while GPT-5 mini evaluates major failures, supplies retry feedback, and selects among valid candidates.The workflow aims to preserve scene geometry, surface detail, and non-shadow appearance.Source: §3 · p. 3
  • Shadow-formation grounding treats a shadow as reduced illumination rather than a change in material or scene content, guiding both generation and evaluation.The evaluator rejects edits that reinterpret shadows as objects, material boundaries, or missing content.Source: §3 · p. 4
  • A mask-guided probe is checked for retained shadows and hallucinated content; passing probes yield two additional candidates, while failures trigger three feedback-guided retries.Failed probes are discarded, and retries use the original image and mask.Source: §3 · p. 4
  • Candidates must pass major-failure evaluation, then a VLM selector compares removal quality, scene fidelity, detail preservation, and artifacts with Retinex-LoG diagnostics as support.If none pass, a fallback selector returns the least damaging candidate.Source: §3 · p. 4
  • Regional correction estimates median YCbCr offsets from reliable pixels outside a dilated shadow mask and applies local or global adjustments to improve low-frequency color agreement.Because it adjusts brightness and chrominance only, it does not explicitly copy textures, edges, or shadow structure from the input.Source: §3 · p. 4

4 AGENTICSHADOW BENCHMARK

AgenticShadow combines four real-shadow image sources into a 17,138-triplet benchmark spanning general scenes, faces, and remote sensing. Compared with ISTD+, it covers larger shadows and broader correction requirements.

Dataset construction

  • A VLM evaluates mask-guided edits, triggers retries, filters failures, and selects candidates before segmentation-guided regional correction produces the final targets.Candidate selection uses visual quality and Retinex-LoG diagnostics.Source: Figure 2 · p. 5

Dataset composition

  • 17,138 image-mask-target triplets form AgenticShadow, with 14,117 training and 3,021 test examples across general scenes, facial shadows, and remote-sensing imagery.SBU and CUHK-Shadow contribute general scenes; ASFW and S-EO extend coverage to faces and remote sensing.Source: §4 · p. 4, Table 1 · p. 5

Benchmark diversity

  • 28.6% vs 16.1% median shadow coverage, 0.724 vs 0.849 color contrast, 1.83 vs 1.51 median log-luminance correction, and 0.395 vs 0.141 correction heterogeneity distinguish AgenticShadow from ISTD+.The comparison uses complete benchmarks and reports median statistics.Source: Figure 3 · p. 5, §4 · p. 6
  • Benchmark diversity is characterized by mask coverage and location, shadow/non-shadow color contrast, and log-luminance correction magnitude and within-shadow heterogeneity.Correction magnitude is summarized by the median; heterogeneity is measured by median absolute deviation.Source: §4 · pp. 5–6

Evaluation protocol

  • Evaluation reports PSNR, SSIM, MAE, and LAB RMSE separately for four domain test sets and as an equal-weight macro average, with ISTD+ reported separately.The four test domains are SBU, CUHK-Shadow, ASFW, and S-EO.Source: §4 · p. 6

5 EXPERIMENTS AND RESULTS

The experiments assess target-construction quality, prior fusion, and transfer across AgenticShadow’s four domains. They find that AgenticShadow supervision improves cross-domain results, while PF achieves the strongest overall performance across those domains.

5.1 AGENTIC TARGET-CONSTRUCTION QUALITY

  • Mean CDD is 0.006960 for the final pipeline, a 50.5% reduction relative to the strongest previous method, SRR.The pipeline also has the lowest standard deviation and maximum CDD, indicating fewer severe construction failures.Source: §5.1 · p. 6, Table 2 · p. 7

5.2 PRIOR-FUSION BASELINE

  • Under ISTD+-only training, prior fusion improves in-domain results but not AgenticShadow results; combined supervision enables gains, with semantic priors contributing more and depth adding complementary context.The matched component analysis finds that both priors independently improve the backbone, while their combination performs best overall.Source: Table 3 · p. 7, §5.2 · p. 8

5.3 CROSS-DOMAIN BENCHMARK EVALUATION

  • Official ISTD+ checkpoints score macro LAB RMSE of 8.43–9.33, versus 5.83–6.80 after AgenticShadow supervision; PF performs best across every AgenticShadow domain and all macro metrics.PF remains competitive on ISTD+; qualitative comparisons show cleaner outputs from AgenticShadow-trained models on general, facial, and remote-sensing scenes.Source: Table 4 · p. 8, §5.3 · p. 8, Figure 5 · p. 9

5.4 MASK-FREE AND GENERALIZATION

  • PF-MF closely matches PF, trailing by 0.14 dB PSNR and 0.10 LAB RMSE; it also outperforms SP+M+I-Net on mask-free SBU-TimeLapse video frames.On ViSha, PF produces cleaner removal than pretrained HomoFormer when using the provided masks.Source: §5.4 · p. 9, Figure 6 · p. 9

6 CONCLUSION

AgenticShadow contains 17,138 image-mask-target triplets built from diverse real shadow images. The paper reports improved cross-domain performance from its supervision and identifies PF as the strongest overall model across the four domains.

6 CONCLUSION

  • AgenticShadow contains 17,138 triplets, and evaluation shows greater cross-domain diversity than existing paired benchmarks and consistent improvement across representative shadow-removal architectures.PF achieves the strongest overall performance across the four domains, while its mask-free variant remains competitive and extends removal to settings without annotated masks.Source: §6 · p. 10
  • The workflow uses generative AI for image normalization, candidate-target generation, failure evaluation, retry guidance, and candidate selection; authors reviewed the outputs and verified reported results.The authors also state that AI assisted with implementation, analysis, supporting materials, and manuscript editing, and that they take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A.1 SOURCE PROCESSING, CLEANING, AND SPLITS

  • The four source datasets are SBU, CUHK-Shadow, ASFW, and S-EO, with S-EO crops selected after visual review for image and shadow-mask quality.The S-EO curation retained 1,026 of 1,119 visually inspected candidate crops after rejecting poor-quality or overlapping selections.Source: §A.1 · p. 13
  • From 17,151 usable image-mask pairs, candidate-generation and regional-correction exclusions yield 17,138 final triplets.S-EO uses AOI-disjoint training and test splits, while duplicate review precedes split finalization to prevent train-test overlap.Source: §A.1 · p. 13

A.2 CONSTRUCTION AND DIVERSITY STATISTICS

The workflow approximates captured ISTD+ references without seeing them during construction, while AgenticShadow has greater measured semantic diversity than ISTD+.

A.2 CONSTRUCTION AND DIVERSITY STATISTICS

  • 97.93% of initial probes pass major-failure evaluation, 99.52% of cases have at least two valid candidates, and CUHK-Shadow generation averages about $0.079 per target.The workflow uses a feedback-guided retry for rejected probes; segmentation, correction, and auditing are local.Source: §A.2 · p. 14
  • 0.9413 ± 0.0020 versus 0.7487 ± 0.0032 mean pairwise DINOv2 cosine distance indicates 25.7% greater semantic diversity for AgenticShadow than ISTD+.The comparison uses 1,000-image resamples repeated 100 times; Figure 7 reports the size-controlled comparison.Source: §A.2 · p. 14, Figure 7 · p. 15

B.1 PROMPT DESIGN AND PIPELINE SETTINGS

  • Physics-motivated prompts distinguish cast from attached shadows throughout generation, retry, evaluation, and selection, while keeping the evaluator schema and agentic control flow fixed.The grounding specifies shadow formation as light occlusion and gives type-specific guidance for preserving surfaces or objects.Source: §B.1 · pp. 15–16, Figure 8 · p. 15

B.2 VALIDATION AGAINST CAPTURED ISTD+ REFERENCES

  • 25.11 PSNR, 0.7380 SSIM, 10.95 MAE, and 3.85 LAB RMSE were achieved against color-corrected ISTD+ references without using them during construction.The workflow received only the shadow images and masks; higher-error examples still showed local color and fine-appearance differences.Source: §B.2 · p. 16

B.3 IMAGE NORMALIZATION AND API USAGE

  • The SRR workflow normalizes images to 512 × 512 before editing; a complete 399-image run takes about 14 hours and costs about $29, so it is intended for offline construction.Compared methods receive the same normalized images and aligned masks; specialized learned models operate on the order of one second per image.Source: §B.3 · pp. 16–17, Figure 10 · p. 17

B.4 ADDITIONAL QUALITATIVE EXAMPLES

  • Feedback-guided retry rejects probes with residual-shadow structure or object-like hallucinations, while physics grounding produces stronger candidate pools for visual selection.The examples show generic prompts can leave residual shadows, create boundary artifacts, or reinterpret dark regions as scene content.Source: Figure 11 · p. 17, §B.4 · pp. 17–18, Figure 12 · p. 18
  • Retinex-LoG structural evidence helps reject candidates with residual shadows and preserve underlying texture when visual-only selection favors incomplete removal.The diagnostics expose residual, lost, and added local structure after suppressing low-frequency illumination.Source: Figure 13 · p. 18, §B.4 · p. 18
  • Segmentation-guided regional correction reduces brightness and color shifts while preserving structure, though examples still show texture loss, residual shadows, and incomplete attached-shadow removal.Regions are class-agnostic, and correction estimates offsets from reliable pixels outside the dilated shadow mask.Source: §B.4 · pp. 18–19, Figure 14 · p. 19, Figure 15 · p. 19

B.5 FAILURE CASES AND LIMITATIONS

Construction can smooth fine texture, remove valid details, or leave residual shadows; pixel-level changes outside the target region also remain a limitation.

B.5 FAILURE CASES AND LIMITATIONS

  • Generative editing can smooth fine texture or remove valid detail; residual shadows beneath vehicles recur, and attached shadows may be only partly removed.These failure modes remain despite evaluation, selection, and correction.Source: §B.5 · p. 19
  • 23.13 mean non-shadow RGB RMSE for the final workflow exceeds the evaluated methods’ 3.44–18.39 range, reflecting misalignment and unintended changes beyond the target region.Regional correction reduces Retinex-LoG-selected outputs from 25.48 to 23.13, but does not eliminate color shifts, texture smoothing, or subtle structural alterations.Source: §B.5 · p. 19

C.1 BASELINE CHECKPOINTS AND EVALUATION PROTOCOL

The evaluation uses separate domain-level metrics and specified combined-data training protocols; PF and PF-MF run at approximately one second per image.

C.1 BASELINE CHECKPOINTS AND EVALUATION PROTOCOL

  • Evaluation covers 540 ISTD+ and 3,021 AgenticShadow test images, with PSNR, SSIM, MAE, and LAB RMSE computed after predictions are restored to target resolution.The AgenticShadow macro average is the unweighted mean of four domain-level results; ISTD+ is reported separately.Source: §C.1 · pp. 19–20, Table 8 · p. 20

C.2 PF AND PF-MF IMPLEMENTATION AND TRAINING

  • Combined-data training concatenates 1,330 ISTD+ and 14,117 AgenticShadow examples without source reweighting, using synchronized crops and flips or rotations.PF and PF-MF train for 100 epochs after checkpoint initialization; their backbones receive 200 cumulative epochs of combined-data training.Source: §C.2 · p. 20

C.3 MODEL COMPLEXITY AND RUNTIME

  • PF and PF-MF each require approximately one second per image; their larger online parameter counts and memory usage arise primarily from running both frozen prior extractors, while trainable parameter counts remain close to those of HomoFormer and PhaSR.Source: §C.3 · p. 20

D.1 FULL PER-DOMAIN EVALUATION

AgenticShadow supervision improves results across the four evaluation domains, with PF and PF-MF ranking near the top on several metrics.

D.1 FULL PER-DOMAIN EVALUATION

  • AgenticShadow supervision improves all four metrics for each retrained architecture in every domain, spanning general, facial, and remote-sensing scenes.Table 10 covers SBU and CUHK-Shadow; the cited per-domain results also report ASFW and S-EO.Source: §D.1 · pp. 20–21, Table 10 · p. 21
  • PF and PF-MF rank first or second in PSNR, MAE, and LAB RMSE across all four domains; PhaSR† has the highest SSIM on SBU, CUHK-Shadow, and S-EO.PF-MF remains close to PF despite requiring no input mask.Source: §D.1 · p. 21

D.2 CROSS-DATASET AND PRIOR ANALYSIS

Cross-dataset analyses find that synthetic checkpoints underperform ISTD+ checkpoints, while AgenticShadow and controlled paired data complement one another. Semantic and depth priors improve AgenticShadow results, especially when combined.

D.2 CROSS-DATASET AND PRIOR ANALYSIS

  • INS checkpoints of StableSR and PhaSR underperform their ISTD+ counterparts across all metrics on ISTD+ and AgenticShadow.INS comprises over 30,000 rendered shadow and shadow-free pairs, yet the comparison reveals a persistent synthetic-to-real gap.Source: §D.2 · p. 21, Table 12 · p. 22
  • 8.43 to 5.94 macro LAB RMSE, while ISTD+ LAB RMSE rises from 1.92 to 2.74 when training only on AgenticShadow after ISTD+ initialization.Combined ISTD+ and AgenticShadow training performs best on both evaluations.Source: §D.2 · p. 21, Table 13 · p. 22
  • Semantic and depth priors independently improve every AgenticShadow domain, and combining them performs best across all four domains and the macro average.ISTD+ performance remains nearly unchanged across the matched variants; semantic information provides the larger gain.Source: §D.2 · p. 22

D.3 QUALITATIVE RESULTS AND VIDEOS

Qualitative comparisons show more complete shadow removal with PF and AgenticShadow-trained models than with the compared checkpoints. PF and PF-MF also produce cleaner results across video frames, although the methods are applied independently without temporal modeling.

D.3 QUALITATIVE RESULTS AND VIDEOS

  • Applied independently to frames without temporal modeling, PF and PF-MF produce cleaner, more complete removal than the compared methods across the evaluated videos.PF is compared with pretrained HomoFormer on ViSha using provided masks; PF-MF is compared with SP+M+I-Net on SBU-TimeLapse without input masks.Source: §D.3 · pp. 22–23
  • PF produces more complete shadow removal and outputs closer to the constructed targets than official ISTD+ checkpoints, which often leave residual shadows or introduce color shifts.The comparison covers SID, ShadowFormer, Inpaint4Shadow, and ShadowDiffusion across four AgenticShadow domains.Source: Figure 16 · p. 24
  • INS-trained StableSR and PhaSR still exhibit residual shadows and appearance shifts on real-world AgenticShadow scenes.The comparison illustrates that large-scale synthetic supervision does not eliminate the synthetic-to-real gap.Source: Figure 17 · p. 25
  • AgenticShadow-trained HomoFormer and PhaSR produce cleaner, more complete removal, while PF-MF achieves strong results without input masks.Colored insets encode mean absolute RGB error on a shared scale.Source: Figure 18 · p. 26

Related papers

Loading 2609.38607v1…