PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Did the Light Come From? Diagnosing Cast-Shadow Reasoning in VLMs and Borrowing the Missing Shadow Prior from Image Generation Models

Ryugo Morita, Shunsuke Kitada, Stanislav Frolov, Brian Bernhard Moser, Ko Watanabe, Riku Takahashi, Jinjia Zhou, Andreas Dengel

arXiv:2610.30740v1cs.CV

TL;DR

VLMs can read the direction of a visible cast shadow, but it remains unclear whether they can infer that direction when the shadow is absent or judge whether it fits the scene. ShadowBench tests these abilities using matched real-photo states, and finds a sharp gap between reading and comprehension; routing VLMs through image-generation models improves inference and validity.

  • Problem

    Existing physical-reasoning benchmarks do not isolate cast-shadow direction or distinguish reading a visible shadow from inferring or validating it, leaving a basic test of shadow–illumination understanding unanswered.

    Source: §1 · pp. 1–2

  • Method

    ShadowBench uses matched real-photo composites with correct, removed, and wrong-direction shadows to test VLMs on direction reading, inference, and physical validity.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    VLMs read visible directions well above chance but infer absent directions weakly and reach chance on physical validity except for Gemini-3.1-pro, while humans score 85–98%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Routing a VLM through an IGM improves inference and validity, suggesting generation or reconstruction objectives as a route toward illumination-aware VLMs.

    Source: §1 · p. 2

    ShadowBench measures only screen-space cast-shadow direction, not illumination understanding across shading, specular or interreflections, soft shadows, multiple lights, or material effects.

    Source: §5 · p. 8

Abstract

from arXiv · show

Shadows are observable consequences of latent physical causes, arising from the interaction between illumination, object geometry, and scene structure. A model that understands shadows should be able not only to read a shadow's direction, but also to infer where a shadow should fall when it is absent, and to judge whether a shadow is physically consistent with the scene. We introduce ShadowBench, built from real photographs, which poses these three questions to vision-language models (VLMs). From each scene, we build a correct-shadow, a shadow-removed, and a human-annotated wrong-direction state. Evaluating 19 open VLMs (2B to 78B) and frontier proprietary models, we find that VLMs read a visible shadow's direction well above chance and improve with scale, yet infer the absent direction only weakly and fall to chance at judging physical consistency even under a bias-free two-alternative test that removes the yes-bias of single-image naturalness questions. Only Gemini-3.1-pro exceeds chance on validity, and even then only modestly and far below humans. Chain-of-Thought (CoT) does not help. The missing ingredient is not reasoning but knowledge that image generation models (IGMs) already hold. Routing a VLM through an IGM lifts its inference well above its direct attempt and helps it flag the wrong-direction shadows it otherwise accepts. Cast-shadow direction is a single facet of illumination understanding, but the sharp gap it reveals between VLMs and IGMs suggests image-generation or reconstruction objectives as a route toward illumination-aware VLMs.

1. Introduction

ShadowBench isolates cast-shadow direction as a probe of whether VLMs understand the illumination relation or merely register a visible shadow. VLMs read visible directions well above chance but infer absent directions weakly and generally fail to judge physical validity; the paper links this gap to differences in training objectives and tests borrowing image-generation priors.

1. Introduction

  • VLMs read visible shadow directions well above chance, but infer absent directions weakly and generally cannot distinguish wrong-direction from correct shadows, even with Chain-of-Thought; humans score 85–98%.Gemini-3.1-pro is the sole exception on validity, exceeding chance only modestly.Source: §1 · p. 2
  • VLMs are trained to produce image-conditioned text, whereas IGMs reconstruct plausible images under illumination and geometry constraints; routing a VLM through an IGM improves inference and validity.The paper proposes generation or reconstruction objectives as a route toward internalizing these priors.Source: §1 · p. 2

2.3. Benchmarks for Vision-Language Models

  • ShadowBench uses matched real-photo composites—a correct shadow, a removed shadow, and a wrong-direction shadow—to separate direction reading, inference, and physical-validity judgments.The matched states change shadow direction relative to the scene while supporting three distinct tasks.Source: §1 · p. 2, Figure 2 · p. 3, §2.3 · p. 3

2.2. Shadow Modeling and Analysis

  • Prior shadow work targets detection, removal, generation, and composition, while illumination-control models render light into images and inverse-rendering methods recover illumination rather than testing general-purpose VLM knowledge.ShadowBench asks whether VLMs hold directional knowledge that generative models use to render illumination.Source: §2.2 · p. 2, §2.1 · p. 2

2.3. Benchmarks for Vision-Language Models

  • Existing VLM benchmarks cover spatial relations, 3D structure, occlusion, plausibility, and causal probes, but do not isolate shadow direction or separate reading from inference and validation.The matched interventions address that benchmark gap.Source: §2.3 · pp. 2–3

3. ShadowBench

ShadowBench extracts cast-shadow layers from real photographs and recomposes matched states so that shadow direction can be tested against a shared scene background. Its six-part design covers layer construction, curation, and three tasks that progress from reading a visible shadow to inferring and validating its direction.

3. ShadowBench

  • ShadowBench starts with real-image scenes and masks, isolates the cast shadow as a soft alpha layer, and recomposes a correct-shadow, shadow-removed, and wrong-direction state on a shared background.The foreground mirror changes the shadow direction while leaving the background lighting cues fixed.Source: §3 · p. 3

3.1. Curation and ground truth

  • Three annotators filter defective scenes and mark object-shadow directions; curation retains 458 of 2,715 scenes with reliable direction ground truth for reading and inference.The validity task shifts mirrored foregrounds only when their position is physically implausible.Source: §3.1 · pp. 3–4

3.2. Three tasks on a single physical variable

  • All three multiple-choice tasks probe the same shadow-direction relation, but reading locates a visible cue while inference and validity relate it to the rest of the scene.The remaining scene cues include other cast shadows, directional shading, and visible light sources.Source: §3.2 · p. 4

3.2.1. Task 1: Direction Reading

  • Direction Reading asks for the extension direction of a visible shadow at two-, three-, or five-way angular resolution, with chance levels of 50%, 33%, and 20%.Upward options are omitted because they do not occur in the captured ground-cast-shadow scenes.Source: §3.2.1 · p. 4

3.2.2. Task 2: Direction Inference

  • Direction Inference removes the shadow and asks for a three-way left/down/right prediction from the remaining scene, with 33% chance accuracy.Its answer options and ground-truth direction match Task 1’s three-way form, isolating the cost of removing the visible shadow.Source: §3.2.2 · p. 4

3.2.3. Task 3: Physical Validity

  • Physical Validity compares correct and wrong-direction shadows in single-image Yes/No and direction-2AFC formats, with the latter balancing correct-image position to remove fixed-option strategies.A separate presence 2AFC contrasts the correct-shadow image with the shadow-removed image to distinguish direction sensitivity from shadow-presence sensitivity.Source: §3.2.3 · pp. 4–5, Table 1 · p. 4

4. Experiment

ShadowBench separates reading a visible shadow from inferring an absent direction and judging physical validity. VLMs read well but struggle with inference and validity; borrowing an image-generation model’s shadow prior improves both comprehension tasks.

4.1. Experiment Setup

  • The benchmark evaluates 458 curated scenes with balanced accuracy and human baselines across reading, inference, and two forms of validity testing.It evaluates 19 open VLMs from 2B to 78B and two proprietary models.Source: §4.1 · p. 5

4.2.1. T1: Reading a visible shadow

  • Gemini-3.1-pro reaches 92.4% on two-way reading and 88.6% on three-way reading, while human accuracy reaches 97.6% and 93.2%, respectively.Reading improves with model size overall, though scores are not determined by size alone; finer angular resolution lowers scores.Source: §4.2.1 · p. 5

4.2. Main benchmark results

  • Reading is well above chance, inference stays near chance, and validity is at chance for all models except Gemini-3.1-pro, at 63.0% on 2AFC.Humans substantially outperform models on these tasks.Source: Table 2 · p. 5, §4.2.1 · p. 5, §4.2.2 · p. 5

4.2.2. T2: Remove the shadow and inference

  • With the shadow removed, inference falls near the 33.3% three-way chance level; Gemini-3.1-pro leads at 50.8%, still 35 points below humans.Fourteen of 19 open models predict one direction on at least 80% of scenes, producing near-chance balanced accuracy.Source: §4.2.2 · pp. 5–6, Table 3 · p. 6

4.2.3. T3: Judging physical validity

  • In validity 2AFC, Gemini-3.1-pro reaches 63.0%, while other models remain near chance; models can detect a shadow’s presence but not reliably its direction.The presence comparison can be solved partly by an image-completeness heuristic, whereas the direction comparison includes shadows in both images.Source: §4.2.3 · p. 6

4.3. CoT does not supply the missing prior

  • CoT and a contextual scaffold do not improve comprehension: mean inference shifts are +0.4 points, and validity shifts remain within 3 points.Even on reading, CoT is a wash; the authors conclude the bottleneck is not reasoning over a physical variable the model possesses.Source: §4.3 · p. 7, Table 4 · p. 7

4.4. The missing knowledge already lives in image-generation models

  • With generation, inference rises from 35.6% to 58.9% and validity 2AFC rises from 51.4% to 73.5%.All 19 models improve on inference; controls indicate that the gain comes from the IGM supplying the direction rather than the pipeline alone.Source: Table 5 · p. 7, §4.4 · pp. 7–8
  • The VLM prompts an image-generation model to add a shadow without specifying its direction, then reads the synthesized shadow’s direction.The editor must infer direction from scene cues such as background shadows, the sky, and object shading.Source: §4.4 · p. 7

5. Limitations & Future works

ShadowBench tests only cast-shadow direction in screen space, not illumination understanding broadly. The authors present borrowing an IGM prior as an existence proof, while acquiring that prior in VLM weights and extending reconstruction objectives remain open.

Scope

  • The benchmark measures screen-space cast-shadow direction, not recovery of physical light sources or physics more generally.Shading, specular and interreflections, soft shadows, multiple light sources, and material effects are untested.Source: §5 · p. 8

Future work

  • The single-editor pipeline is an existence proof rather than a tuned system, so its gains are a lower bound.Acquiring the prior in VLM weights through generation or reconstruction objectives is future work; whether reconstruction generalizes to physical understanding remains open.Source: §5 · p. 8

Continuous-angle evaluation

  • Reading coarse screen-space direction does not recover a continuous angle, and a regression formulation awaits a reliable continuous-angle interface.The discrete two-, three-, and five-way bins are described as the easiest version of the task, at which models already fail.Source: §5 · p. 8

6. Conclusion

ShadowBench reveals a gap between reading visible cast shadows and inferring or validating their direction. Routing VLMs through an IGM improves inference, pointing to a cast-shadow prior already present in image generation models.

Conclusion

  • VLMs read visible shadow direction well, but inference approaches chance and validity is at chance for all but one model; CoT closes neither gap.The best reader reaches 92% versus humans’ 98%; inference’s single frontier exception reaches 51% versus humans’ 86%, and validity reaches 63% versus humans’ 91%.Source: §6 · p. 8
  • Routing an editor’s output back through VLM reading lifts inference by +23 points on average.The paper reports that the missing cast-shadow direction prior already lives in IGMs.Source: §6 · p. 8
  • The authors frame generation-grounded recovery as a first step toward illumination-aware multimodal models.ShadowBench isolates one cast-shadow direction question in three forms.Source: §6 · p. 8

7. Reading is the same on the capture and the composite

Direction reading is nearly unchanged when evaluated on the real capture or the recomposited image. The reported comparison uses the same 458 scenes and 19 open models.

Reading is the same on the capture and the composite

  • Mean balanced accuracy is 60.1% on Iall and 61.2% on Isdw, a +1.1-point shift; every model stays within 3 points.The authors conclude that the reading ceiling is unaffected by the choice of source.Source: §7 · p. 11

8. Shadow perception scales with opacity

The opacity sweep shows that shadow presence is perceptually registered, while direction performance varies modestly with shadow length and remains weaker for direction than presence. The reported per-model comparison localizes the gap to direction rather than shadow presence.

8. Shadow perception scales with opacity

  • Shadow-presence detection rises smoothly with opacity, with Qwen2.5-VL-7B increasing from 31.9% to 93.9%, InternVL3-8B from 60.3% to 96.3%, and Qwen3-VL-8B from 75.3% to 100%.The authors conclude that shadow presence is reliably available and comprehension failures are not attributable to a perceptual deficit.Source: §8 · p. 11, Figure 3 · p. 12

9. Which lighting conditions are hardest?

  • Inference accuracy is 36.5% on short-shadow scenes and 41.9% on long-shadow scenes; reading rises from 61.0% to 64.4% across those halves.The median shadow reach/object height split is 1.15; short shadows provide little geometry to anchor direction, and the effect is small.Source: §9 · p. 11, Table 7 · p. 12

10. Presence vs. direction validity (per model)

  • Every open model above the 2–3B tier is far above chance on shadow presence but at chance on direction, localizing the gap to direction rather than presence.Presence 2AFC compares correct-shadow with shadow-removed images; direction 2AFC compares correct-shadow with wrong-direction images.Source: §10 · p. 11

11. Per-class breakdown, full panel

Across the full model panel, visible-shadow reading improves with scale, but inference and validity remain near chance for most models; CoT does not reliably improve comprehension. Routing VLMs through an IGM yields strong inference scores, with FLUX.1-Kontext-dev the best open, reproducible choice.

11. Per-class breakdown, full panel

  • 14 of 19 open models assign at least 80% of inference predictions to one direction, while most models’ balanced comprehension scores remain near chance.Only Gemini-3.1-pro spreads predictions near the label priors and clears chance on the comprehension tasks; presence judgments are far above chance even when direction validity is not.Source: §11 · p. 12, Table 8 · p. 12, Table 9 · p. 13

12. Per-model CoT breakdown

  • CoT deltas are inconsistent in sign and centered near zero, with no model lifted from chance to competence on a comprehension task.The per-model deltas are sizable for some models, but the aggregate pattern is added noise rather than added competence.Source: §12 · p. 12, Table 10 · p. 13

13. Which IGM to route through the pipeline

  • All four tested IGMs drive the fixed VLM reader to 61–80% three-way direction accuracy, well above the 33.3% chance level.FLUX.1-Kontext-dev scores 79.7%, edging Gemini-3-pro-image at 79.0%; it is selected as the best open, reproducible choice.Source: §13 · p. 14, Table 11 · p. 14
  • The IGM comparison uses VLM-authored prompts that omit the shadow direction and a fixed Gemma-4-31B reader, isolating differences among editors.Each editor is evaluated on the same 458 scenes, and the full read-back loop is used for scoring.Source: §13 · pp. 12, 14

14. The shadow-only rotation control

Shadow-only rotations are detectable through non-lighting artifacts, so the authors progressively remove those cues and compare the result with an editor-naturalized construction. Detectability moves toward chance under both controls, while the leak-free mirror is at chance.

14. The shadow-only rotation control

  • 64.0% panel accuracy on uncurated rotations falls to 57.1% after shadow-mass matching and clean-compositing screens, approaching the mirror’s chance-level result.Only 112 of 1,374 rotations survive both human curation screens; silhouette mismatch remains an unscreenable cue.Source: §14 · pp. 14–15, Table 12 · p. 15
  • Humans score 94.4% on the curated shadow-only pairs, indicating the pairs remain decidable even as model accuracy declines with shortcut removal.The authors interpret the model decline as loss of non-lighting shortcuts, not loss of the lighting signal.Source: §14 · pp. 14–15

An editor-naturalized variant converges to the same

  • 52.3% panel accuracy on editor-refined negatives is close to the mirror’s 51.4%, after annotators retain 98 of 1,374 edits as artifact-free.The editor redraws the rotated shadow with the scene hidden; any remaining artifact would make the negative easier to detect, so this is an upper bound on manipulation leakage.Source: An editor-naturalized variant… · p. 15, Table 12 · p. 15

15. Option-order robustness

Shuffling direction-option order leaves reading and inference unchanged. Direction-source controls show that the pipeline transmits the direction supplied by the generator, and human ratings independently confirm that the generator can synthesize and repair shadows in the scene-consistent direction.

15. Option-order robustness

  • Reading changes only from 61.2% to 59.2% under option-order shuffling, while inference stays at 35.6%.The validity 2AFC already presents both pair orders, canceling a fixed-side preference.Source: §15 · p. 15

16. Direction-source controls for the pipeline

  • 65.0% accuracy for reading an IGM-synthesized shadow is at the 67.9% ground-truth-composite level, while randomly rotated shadows are at chance against scene truth.The control indicates the pipeline transmits the injected direction: models read the random shadow well when scored against the direction actually injected.Source: Table 14 · p. 16, §16 · p. 16

17. Human measurement of the generator’s direction

  • Human observers match FLUX.1-Kontext-dev’s synthesized shadow direction to scene ground truth on 85.9% of scenes and rate its wrong-shadow repairs as correct on 83.2%.Both rates are well above the 33.3% three-way chance level.Source: §17 · p. 16, Table 15 · p. 16

18. Curation statistics and record counts

Filtering and annotator review reduce 2,715 captured scenes to 458 with reliable direction ground truth, which form the validity set. Representative failures show models defaulting to a direction and accepting mirrored shadows rejected by humans.

18. Curation statistics and record counts

  • Curation reduces 2,715 captured scenes to 1,098 after automated filtering and then to 458 after annotator review; all 458 form the validity set.The validity set contains 916 Yes/No records and 916 counterbalanced 2AFC records.Source: §18 · pp. 16–17
  • The full open-model evaluation uses roughly 2,800 GPU-hours, and each IGM edit takes 5–25 seconds per image.Open-model evaluations run on single A100/H100-class nodes with one to four GPUs.Source: §18 · p. 17

19. Failure cases

  • On inference examples, GPT-5 answers “right” on every scene, while both frontier models accept mirrored shadows that human annotators reject.The figure presents representative failures on inference and single-image validity tasks.Source: §19 · p. 17, Figure 5 · p. 17
Loading 2610.30740v1…