PaperlayerView exact revision on arXiv

Source-linked AI summary

ShadowCast: On-the-Fly Context-Aware Shadow Augmentation for Remote Sensing Image Segmentation

Dongho Yoon, Yechan Kim, Jongmin Joo, JongHyun Park, Moongu Jeon

arXiv:2610.23440v2cs.CV

TL;DR

Cast shadows change remote-sensing surface appearance and obscure object boundaries, degrading segmentation. ShadowCast uses segmentation labels and local brightness to synthesize context-aware shadows during training, and consistently outperforms standard training and existing shadow augmentation on three benchmarks.

  • Problem

    Cast shadows alter surface appearance and obscure object boundaries, contributing to degraded remote-sensing segmentation predictions in shadowed regions.

    Source: §I · p. 1

  • Method

    ShadowCast extracts elevated-object masks from segmentation labels, estimates cast-shadow direction from local brightness, and projects synthetic shadows without additional learned models or acquisition metadata.

    Source: §I · p. 1, 1) Elevated Object Region (or Mask)… · p. 2, 3) Synthetic Shadow Projection · p. 3

  • Results

    ShadowCast consistently outperforms standard training and existing shadow augmentations across Vaihingen, Potsdam, and LoveDA and diverse segmentation architectures.

    Source: §I · p. 1

  • Takeaways & Limitations

    ShadowCast provides a complementary source of training variation, with gains also reported when combined with existing augmentation and regularization strategies.

    Source: §I · p. 1

    ShadowCast assumes surrounding brightness indicates dominant shadow direction and shifted object masks approximate shadows on locally flat ground; typical failures include incorrect shadow shape or length and unclear or misleading direction cues.

    Source: §VII · pp. 13–14

Abstract

from arXiv · show

Shadows cast by elevated objects alter the spectral appearance and obscure the boundaries of objects in remote sensing images, thereby degrading semantic segmentation performance. Most existing approaches handle shadows as degradations to be detected and removed, requiring additional costs from dedicated networks and shadow-specific annotations. We propose ShadowCast, which takes the opposite approach: it synthesizes shadows and injects them during training for remote sensing segmentation. It extracts masks of elevated objects, such as buildings and trees, directly from the segmentation labels and estimates an image-specific cast-shadow direction cue by probing the brightness around them. It then projects synthetic shadows along the inferred cast-shadow direction, requiring no learned network and no acquisition metadata. Experiments on three widely used remote sensing benchmarks, Vaihingen, Potsdam, and LoveDA, show that ShadowCast consistently outperforms the baseline across diverse state-of-the-art segmentation architectures. Moreover, it brings additional performance gains when combined with standard augmentation and regularization techniques, indicating that stochastic shadow injection serves as a simple and complementary augmentation strategy for remote sensing segmentation.

I. INTRODUCTION

Cast shadows alter surface appearance and obscure object boundaries, while common shadow-processing approaches add dedicated models or annotations. ShadowCast instead uses label and image context to synthesize shadows during training, and improves results across three benchmarks and diverse architectures.

I. INTRODUCTION

  • Cast shadows increase intra-class appearance variation and inter-class spectral confusion, degrading segmentation predictions in shadowed regions.Reduced illumination changes surface appearance and can obscure object boundaries.Source: §I · p. 1

II. RELATED WORK

  • Shadow detection and removal commonly require dedicated models and annotations; generic shadow augmentation darkens regions without modeling plausible shadow-casting objects.These pipelines add computational and annotation costs, while detection errors can propagate to later stages.Source: §I · p. 1

A. Motivation

  • ShadowCast combines label-derived elevated-object masks with image-specific brightness cues to project synthetic shadows without extra networks, shadow annotations, or acquisition metadata.The method uses semantic context to identify potential casters and photometric context to estimate cast-shadow direction.Source: §I · p. 1, A. Motivation · p. 2

I. INTRODUCTION

  • ShadowCast consistently outperforms standard training and existing shadow augmentations on Vaihingen, Potsdam, and LoveDA across diverse segmentation architectures.Its gains also persist when combined with existing augmentation and regularization strategies.Source: §I · p. 1

B. Method

ShadowCast derives candidate shadow casters from segmentation labels, estimates a cast-shadow direction from surrounding image brightness, and projects randomized synthetic shadows while leaving labels unchanged.

B. Method

  • ShadowCast projects label-derived elevated-object masks along an image-specific cast-shadow direction, modifying surface illumination while preserving semantic labels.The procedure operates on an image–label pair and summarizes the method's main stages.Source: B. Method · p. 2

1) Elevated Object Region (or Mask) Extraction:

  • The binary mask marks pixels whose semantic labels belong to elevated-object classes such as buildings and trees.It is extracted directly from existing segmentation masks, requiring no additional object detector, shadow annotation, or geometric metadata.Source: 1) Elevated Object Region (or Mask)… · p. 2

3) Synthetic Shadow Projection:

  • Randomly sampled direction, shift, and attenuation vary synthetic shadow orientation, size, and darkness across training iterations.Probe regions with too few valid pixels are excluded from candidate directions.Source: 2) Image-Specific Cast-Shadow… · pp. 2–3

2) Image-Specific Cast-Shadow Direction Estimation:

  • Shadow direction is estimated by converting the image to grayscale and comparing brightness in regions probed around the elevated-object mask.The direction with the lowest average brightness is used as a lightweight cue, though intrinsically dark surfaces can bias it.Source: 2) Image-Specific Cast-Shadow… · pp. 2–3

3) Synthetic Shadow Projection:

ShadowCast projects label-derived elevated-object masks into neighboring regions to create synthetic shadows. It samples shadow direction, displacement, and attenuation to vary the shadows during online augmentation.

3) Synthetic Shadow Projection:

  • After estimating the cast-shadow direction, ShadowCast samples a projection distance and uses it to translate the mask.The displacement range is specified in pixels.Source: 3) Synthetic Shadow Projection · p. 3
  • The elevated-object mask is shifted along the estimated direction, with the original object pixels excluded so attenuation affects neighboring surfaces.If the target region is empty, the image is returned unchanged.Source: 3) Synthetic Shadow Projection · p. 3
  • Shadow strength is sampled as a multiplicative attenuation factor δ between δmin and δmax, with 0 < δmin ≤ δmax < 1.Random sampling of direction, displacement, and attenuation varies shadow orientation, size, and darkness across training iterations.Source: 3) Synthetic Shadow Projection · p. 3
  • ShadowCast operates online using the image and its existing semantic label; a 512×512 Vaihingen crop takes an average of 16.55 ms on an Intel(R) Xeon(R) Gold 6330 CPU.The timing is averaged over 1,000 runs.Source: 3) Synthetic Shadow Projection · p. 3

A. Experimental Setup

Experiments use three remote-sensing benchmarks and report mIoU; the section also selects a shared ShadowCast configuration and examines cast-shadow direction choices. The no-shadow-augmentation results provide a baseline for the listed methods.

A. Experimental Setup

  • No shadow augmentation records 77.53±0.30 mIoU on Vaihingen, 82.95±0.10 on Potsdam, and 43.29±0.77 on LoveDA.Results are mean ± standard deviation over three independent runs; the clutter class is excluded for Vaihingen and Potsdam.Source: A. Experimental Setup · pp. 3–4

1) Parameter Sensitivity:

  • The shared best-performing configuration is (dmin, dmax) = (5, 30), (δmin, δmax) = (0.5, 0.8), η = 15, τ = 50, and r = 8, with p = 0.5.The sensitivity analysis reports stable performance across tested settings on all three benchmarks.Source: 1) Parameter Sensitivity · p. 4

2) Effect of Cast-Shadow Direction Selection:

  • Combining the estimated cast-shadow direction with angular jittering performs best on all three datasets among the direction-selection choices evaluated.The comparison also finds that both random and estimated directions improve over the baseline.Source: 2) Effect of Cast-Shadow Direction… · p. 4

C. Comparison across Segmentation Architectures

ShadowCast improves results across segmentation architectures and existing shadow-augmentation comparisons, and it adds gains alongside other training strategies. Evaluations also report gains in both shadowed and non-shadowed regions, while the conclusion notes limits in its shadow geometry assumptions.

C. Comparison across Segmentation Architectures

  • Across ten architectures and three benchmarks, ShadowCast generally improves performance over the corresponding baselines, with larger gains on Vaihingen and LoveDA.OCRNet on Vaihingen and LOGCAN++ on LoveDA improve by 2.49 and 2.26 mIoU, respectively.Source: C. Comparison across Segmentation… · p. 4

D. Comparison with Existing Shadow Augmentation Methods

  • ShadowCast achieves the highest mIoU on all three benchmarks among the six existing shadow-augmentation methods compared.The existing methods show less consistent improvements.Source: D. Comparison with Existing Shadow… · p. 4

E. Compatibility with Existing Training Strategies

  • Adding ShadowCast improves performance across all tested combinations of augmentation and regularization strategies and datasets.The tested strategies include conventional geometric and photometric augmentations, label smoothing, and NSegment.Source: E. Compatibility with Existing… · p. 4

F. Effects across Shadow and Non-Shadow Regions

  • ShadowCast improves shadowed-region mIoU by 0.30, 0.42, and 0.31 on Vaihingen, Potsdam, and LoveDA, respectively, and non-shadowed-region mIoU by 0.49, 1.35, and 0.55 points, respectively.Because the benchmarks lack shadow annotations, the evaluation manually annotated pixel-level shadow masks for ten images per dataset.Source: F. Effects across Shadow and… · pp. 4–5

V. CONCLUSIONS

  • ShadowCast relies on brightness-based direction estimation and simplified mask translation, which may produce inaccurate shadows in complex scenes or terrain with significant elevation variations.The conclusion also reports that random-direction augmentation improves over the baseline on all three benchmarks.Source: §V · p. 5

Appendix

The appendix reports parameter sensitivity and details the evaluation of real shadowed regions. It also illustrates and groups the compared shadow-augmentation approaches, including methods that synthesize shadows without scene context.

I. PARAMETER SENSITIVITY ANALYSIS

  • Parameter sensitivity is evaluated on Vaihingen, Potsdam, and LoveDA with DeepLabV3+ and p = 0.5; results remain stable across tested ranges, with LoveDA more sensitive than Potsdam.All Potsdam variants stay within 0.14 mIoU of default; LoveDA variants stay within 0.74 except τ = 0, which drops 1.19 mIoU.Source: §I · pp. 6–7

II. DETAILED SETTINGS FOR EVALUATION ON SHADOWED AND NON-SHADOWED REGIONS

  • Real-shadow evaluation uses manually annotated pixel-level masks from ten shadow-containing test images per dataset, dividing pixels into shadowed and non-shadowed regions.Region-specific class-wise IoU is averaged over evaluated classes, excluding clutter for Vaihingen and Potsdam.Source: §II · p. 7, Figure S1 · p. 7

III. COMPARISON WITH EXISTING SHADOW AUGMENTATION METHODS

  • The appendix provides qualitative examples and implementation details for the existing shadow-augmentation methods compared in the main manuscript.Fig. S2 shows methods on the same Vaihingen training image and omits CLAHE because it enhances rather than synthesizes shadows.Source: §III · p. 8, Figure S2 · p. 8

A. Compared Methods

  • The compared methods are grouped according to the scene context used for shadow synthesis.The supplied passage introduces the grouping but does not enumerate its categories.Source: A. Compared Methods · p. 8

1) Object-Agnostic Shadow Synthesis:

  • Object-agnostic methods include random polygon darkening, image-wide plasma attenuation, randomly sampled triangular shadows, cloud-shadow masks, and 3D-rendered shadow mattes placed independently of scene content.The compared implementations include method-specific modifications, such as randomly sampling Zhong et al.'s triangle vertices and using only SatelliteCloudGenerator's shadow component.Source: 1) Object-Agnostic Shadow Synthesis · p. 8

2) Object-Anchored Shadow Synthesis:

This section contrasts shadow augmentation methods and gives their implementation and training settings. The qualitative comparison shows how object anchoring and direction choice affect whether synthetic shadows align with the scene.

2) Object-Anchored Shadow Synthesis:

  • The adapted SDAA translates label-derived elevated-object masks along a uniformly sampled direction, enabling comparison with ShadowCast’s image-specific direction cue.The methods use different darkening models: SDAA uses gamma transformation and ShadowCast uses linear attenuation.Source: 2) Object-Anchored Shadow Synthesis · p. 8

B. Qualitative Comparison

  • Object-agnostic methods darken regions unrelated to scene structure, while SynShadow’s soft matte partly covers a rooftop despite its physically modeled attenuation.RandomShadow and Zhong et al. create shapes across rooftops, roads, vehicles, or canopies; PlasmaShadow and SatelliteCloudGenerator apply image-wide patterns.Source: B. Qualitative Comparison · pp. 8–9
  • ShadowCast projects shadows from the same elevated objects toward the observed leftward direction, unlike SDAA’s random direction that falls on sunlit sides and conflicts with real shadows.SDAA also uses gamma darkening, whereas ShadowCast uses linear attenuation.Source: 2) Object-Anchored Shadow Synthesis · p. 8, B. Qualitative Comparison · p. 9

IV. IMPLEMENTATION DETAILS AND PARAMETER SETTINGS

  • Experiments use MMSegmentation, with CLAHE, RandomShadow, and PlasmaShadow from Albumentations and several standard transforms from MMSegmentation.Other methods are implemented as MMSegmentation transforms or, for label smoothing, as a loss function.Source: §IV · p. 9

A. Training Configuration

  • Models train for 80k iterations with polynomial learning-rate decay; the baseline excludes the methods listed in the augmentation and regularization settings.Table S4 lists each model’s backbone and optimizer.Source: A. Training Configuration · p. 9, Table S4 · p. 9

B. Augmentation and Regularization Methods

  • The augmentation and regularization methods combined with ShadowCast are listed in Table S5, with custom implementation descriptions provided separately.The image and label inside a sampled rectangle are replaced with the corresponding region from another training sample for the described mixing operation.Source: B. Augmentation and Regularization… · pp. 9–10, Table S5 · p. 9

2) Random Erasing:

  • Random Erasing fills a sampled rectangle’s channels from the erased region and sets its labels to ignore index 255.The rectangle’s area ratio and aspect ratio are sampled from the stated ranges.Source: 2) Random Erasing · p. 10

3) Label Smoothing:

This section specifies label smoothing, label-only deformation, and the shadow augmentation methods used in the experiments. The methods vary in how they alter labels, place shadow regions, and darken pixels.

3) Label Smoothing:

  • The cross-entropy target assigns 1 −ϵ to the ground-truth class and ϵ/(C −1) to each other class, where C is the number of classes.Pixels with the ignore index are excluded from the loss.Source: 3) Label Smoothing · p. 10

4) NSegment:

  • NSegment deforms only the label map using smoothed per-pixel displacements and nearest-neighbor resampling.The displacement range is controlled by α, the Gaussian smoothing by σ, and each sample draws from 15 parameter combinations.Source: 4) NSegment · p. 10

C. Shadow Augmentation Methods

  • Shadow augmentation methods modify the image while keeping its label unchanged, and are applied probabilistically when a shadow region is produced.Zhong et al. and SynShadow resample shapes up to a set trial limit to keep their effective application rate near the target probability.Source: C. Shadow Augmentation Methods · p. 10, Table S6 · p. 10

1) ShadowCast:

  • ShadowCast shares one sampled direction, displacement, and attenuation across elevated objects in an image, adopting the darkest valid direction without a confidence check.Candidate directions are spaced by 45 degrees, and a probe region must contain more than 50 pixels to be valid.Source: 1) ShadowCast · p. 10

2) SynShadow:

  • SynShadow composites a rendered soft matte using affine darkening of the image intensities.The matte is resized, cropped, and horizontally flipped; it is redrawn if its mean is below 0.005.Source: 2) SynShadow · p. 10

3) SDAA:

  • The adapted SDAA translates elevated-object masks along a uniformly sampled direction and applies a gamma transform to 8-bit intensities.It omits the original shadow-removal step because shadow annotations are unavailable, and darkens only translated pixels outside the elevated-object mask.Source: 3) SDAA · p. 10

4) Zhong et al.:

  • Zhong et al.’s implementation draws a triangle across the image and darkens its CIELAB lightness channel, unlike the original traffic-sign-restricted version.A 3-pixel contour band is replaced with its Gaussian-blurred version.Source: 4) Zhong et al · p. 10

5) SatelliteCloudGenerator:

This section reports class-group effects and tests two design modifications to ShadowCast. Gains extend to elevated and non-elevated classes, while the tested smoothing variant retains less of the default formulation’s benefit.

V. EFFECTS ACROSS ELEVATED AND NON-ELEVATED CLASSES OF ShadowCast

  • ShadowCast improves every elevated and non-elevated class on all three datasets; average gains are larger for non-elevated classes on Vaihingen and LoveDA and comparable across groups on Potsdam.The average gains are +1.11 versus +0.68 on Vaihingen, +1.90 versus +0.86 on LoveDA, and +0.37 versus +0.38 on Potsdam.Source: Table S7 · p. 11, §V · p. 11

A. Non-Elevated Classes

  • The default projected mask excludes elevated-object classes, so synthetic shadows usually vary the brightness of non-elevated classes while preserving their labels.This provides a straightforward explanation for the non-elevated-class improvement and is consistent with gains in real-shadowed regions.Source: A. Non-Elevated Classes · p. 11

B. Elevated Classes

  • Elevated classes improve despite usually not being directly darkened; proposed explanations include brightness-robust features and better discrimination near elevated-object boundaries.The authors present these as possible explanations, not established mechanisms.Source: B. Elevated Classes · p. 11

VI. DESIGN CHOICES AND ADDITIONAL ABLATION STUDIES OF ShadowCast

  • ShadowCast uses rigid mask translation and uniform multiplicative attenuation, without explicitly modeling penumbra, complex light transport, or material-dependent reflectance.The method is designed for stochastic training augmentation using label-derived casters and an image-specific direction cue.Source: §VI · p. 11
  • Neither smoothing the boundary nor rejecting weak brightness cues outperforms the default configuration in the tested settings.The authors retain hard masks based on the reported segmentation results while noting that the comparison changes both boundary sharpness and attenuation distribution.Source: §VI · p. 11, A. Hard versus Soft Shadow Boundaries · p. 12

A. Hard versus Soft Shadow Boundaries

  • The soft-boundary variant gains +0.26, +0.03, and +0.07 mIoU on Vaihingen, Potsdam, and LoveDA, versus +0.93, +0.37, and +1.60 for the default hard mask.The reported comparison does not establish that soft shadows are generally less useful: smoothing also redistributes attenuation around the boundary, and a realism-improving formulation that preserves the benefit remains future work.Source: A. Hard versus Soft Shadow Boundaries · pp. 11–12, Table S8 · p. 12

B. Confidence-Gated Direction Estimation

The confidence-gated variant tests brightness-based checks before applying ShadowCast, but performs below the default in the reported settings. The section also identifies assumptions behind the direction estimate and projected-shadow geometry, alongside representative failure types.

B. Confidence-Gated Direction Estimation

  • The gated variant remains above baseline but falls below default by 0.28, 0.10, and 0.99 mIoU points on Vaihingen, Potsdam, and LoveDA.The checks require a directional brightness gap and a brighter opposite side; rejecting images may reduce augmentation frequency, so the authors leave separating reliability from augmentation frequency for future work.Source: Table S9 · p. 13, B. Confidence-Gated Direction Estimation · p. 13

VII. FAILURE CASES AND LIMITATIONS

  • ShadowCast assumes local brightness indicates dominant shadow direction and that translated object masks approximate shadows on locally flat ground.Representative failures include incorrect length or shape, unclear direction in already shaded areas, and misleading neighboring-object shadows; the listed extensions remain unevaluated.Source: §VII · pp. 13–14, Figure S4 · p. 13, Table S10 · p. 14

A. Observed Failure Cases

ShadowCast can misestimate shadow geometry and direction when object height, local shading, or neighboring shadows confound its image-based cues. Its evaluation mainly covers relatively flat terrain, leaving its performance on complex terrain uncertain despite gains when precise shadow geometry is ignored.

1) Incorrect Shadow Length and Shape:

  • Random mask shifts ignore object height, producing shadows that can be too short or fail to reproduce displaced, structurally shaped shadows.A single shift cannot model, for example, a tree shadow separated from its canopy; height information could improve lengths and shapes but adds data and computation costs.Source: 1) Incorrect Shadow Length and Shape · p. 14

2) Unclear Direction in Already Shaded Areas:

  • When an area is already largely shaded, uniformly dark surroundings provide no clear direction cue, so local texture can determine the estimated direction.The proposed remedy is to identify and down-weight ambiguous regions affected by neighboring shadows.Source: 2) Unclear Direction in Already Shaded… · p. 14

3) Misleading Shadows from Neighboring Objects:

  • In dense rows of high-rise buildings, neighboring shadows and visible side walls can bias each building’s brightness cue and reinforce an incorrect global direction estimate.The proposed remedy is to down-weight unreliable object-level measurements rather than treating all objects equally.Source: 3) Misleading Shadows from Neighboring… · p. 14

B. Terrain with Large Height Variation

  • The benchmarks mainly cover buildings and trees on relatively flat ground, so they do not establish ShadowCast’s effectiveness in hilly or mountainous areas.The method assumes flat ground around objects and that only labeled elevated objects cast shadows.Source: B. Terrain with Large Height Variation · p. 14
  • On sloped ground, shadow length and shape vary with slope orientation and can bend where the slope changes, which fixed-distance mask shifts cannot reproduce.Shadows lengthen on slopes facing away from the sun and shorten on slopes facing toward it.Source: B. Terrain with Large Height Variation · p. 14
  • Terrain can cast shadows that are not tied to labeled objects, while slope-dependent brightness can make a dark probe region misleading for direction estimation.Slopes facing away from the sun can be dark even without nearby objects, and their brightness may be mistaken for cast-shadow evidence.Source: B. Terrain with Large Height Variation · p. 14
  • Without relief-displacement correction, tall objects may lean away from their ground positions, misaligning object masks with shadow origins.Physical shadow computation on actual terrain would require height maps and sun position, inputs ShadowCast does not use; effectiveness on substantial elevation variation remains uncertain.Source: B. Terrain with Large Height Variation · p. 14
  • ShadowCast gains on the evaluated datasets despite ignoring object height, but whether that benefit extends to complex terrain remains untested.The results suggest precise shadow geometry is not always necessary for improving segmentation on the evaluated datasets.Source: B. Terrain with Large Height Variation · p. 14

Related papers

Loading 2610.23440v2…