PaperlayerView exact revision on arXiv

Source-linked AI summary

OccludeBench: Can VLMs Find What's Missing?

Jasin Cekinmez, Addison J. Wu, Ryo Mitsuhashi, Linrong Cai, Xingyu Fu, Zhuang Liu

arXiv:2610.13790v1cs.CVcs.CL

TL;DR

The paper asks how well VLMs reason about objects missing from images, a capability difficult to isolate with full, unconstrained-image benchmarks. It introduces OccludeBench and OccludeTrace to test and diagnose this reasoning; VLMs show a direct-evidence accuracy gap that humans do not show significantly.

  • Problem

    Because most benchmarks use full, unconstrained images, it is difficult to determine which visual cues VLMs use when reasoning about what is missing from an image.

    Source: §1 · p. 2

  • Method

    OccludeBench contrasts fully occluded objects recoverable from shadows or reflections with those inferred from broader context, and OccludeTrace decomposes the recovery process into four diagnostic stages.

    Source: §1 · p. 2

  • Results

    Humans score 63.4% on direct images and 68.5% on inferential images with no statistically significant gap, while VLM averages are 42.7% and 62.7%, respectively.

    Source: §3.1 · p. 4

  • Takeaways & Limitations

    Occlusion provides a tractable, controlled diagnostic of reasoning about missing visual content, and reliable use of task-relevant evidence remains an open challenge for current VLMs.

    Source: §5 · p. 9

Abstract

from arXiv · show

How well can vision-language models (VLMs) reason about what is missing from an image? We study this question with OccludeBench, a manually curated benchmark for recovering fully occluded objects from the visual evidence that remains. OccludeBench distinguishes direct cases, where the hidden object must be recovered from a specific physical cue such as a shadow or reflection, from inferential cases, where it can be inferred from broader scene context. Across five VLMs, accuracy averages 42.7% on direct cases versus 62.7% on inferential cases, exhibiting a 20.0-point gap. Humans attain higher accuracies of 63.4% versus 68.5% for direct and inferential images, respectively, with no significant gap. To locate this failure, we introduce OccludeTrace, a four-stage diagnostic that isolates where the visual reasoning process breaks down. The bottleneck emerges when models must recover the relevant object from the projection within the full image: they succeed on only 50% of failed examples, compared with 91% of examples they answer correctly. Extended reasoning, tool use, and few-shot prompting yield inconsistent improvements and do not reliably eliminate the gap. In contrast, an inpaint-then-confront intervention, which prompts models to first generate a candidate hidden object and then evaluate that hypothesis against the visible physical cue, reduces the gap to a statistically non-significant difference for both models with image-generation capabilities. Together, our results establish a tractable, controlled diagnostic for isolating how VLMs reason about visual content that is missing from an image.

1 INTRODUCTION

The paper asks whether VLMs can recover fully occluded objects from the visual evidence that remains, a question that ordinary full-image benchmarks do not isolate. It introduces a controlled benchmark and diagnostic to identify where this reasoning fails.

1 INTRODUCTION

  • Because most benchmarks use full, unconstrained images, it is difficult to determine which visual cues models use or whether they reason from evidence rather than prior expectations.Recovering out-of-view objects can require reasoning from a remaining shadow, reflection, or surrounding context.Source: §1 · p. 2
  • OccludeBench contrasts direct cases, requiring a shadow or reflection, with inferential cases, recoverable from broader scene context, using the same task and masking procedure.This setup isolates reasoning from specific physical evidence versus contextual inference.Source: §1 · p. 2
  • VLMs struggle not to perceive a shadow or reflection, but to isolate which object within the projection identifies what is hidden in the full image.OccludeTrace decomposes recovery into detecting the projection, recognizing it in isolation, mapping it to its source, and recovering the object from the cue in the full image.Source: §1 · p. 2
  • Inpaint-then-confront reduces the direct–inferential gap to a statistically non-significant difference for models with image-generation capabilities.Other tested approaches, including in-context learning, tool use, extended reasoning, and targeted prompting, yield inconsistent gains and leave the gap persistent.Source: §1 · p. 2
  • Occlusion offers a tractable, controlled way to test reasoning about missing visual content, and OccludeTrace decomposes the test to locate reasoning failures.The authors identify evidence selection as a key missing capability in current VLMs.Source: §1 · pp. 2–3

2 OCCLUDEBENCH

OccludeBench uses curated images with manually occluded targets to compare direct physical-cue cases with inferential context cases. The paper evaluates open-ended VLM responses and collects a free-form human baseline.

2.1 DATASET CREATION

  • 378 images comprise 197 direct and 181 inferential cases; direct cues include specular reflections (47.7%), diffuse reflections (28.4%), and shadows (23.9%).Inferential images span technology (24.9%), household scenes (24.3%), culture (20.4%), activities (17.1%), and nature (13.3%).Source: §2.1 · p. 3
  • Images come from online sources, Visual Genome, and self-collection; targets are manually occluded, and inclusion requires unanimous human agreement on category fit.Direct examples contain a shadow or reflection not reasonably inferable from context; inferential examples are selected for contextual inferability.Source: §2.1 · p. 3

2.2 EVALUATION

  • Five VLMs give open-ended object predictions; Claude Opus 4.6 and GPT-5.4 judge correctness, with either judge’s approval counting as correct.The two-judge framework agrees with author ground-truth annotations in 98% of cases.Source: §2.2 · pp. 3–4

2.3 HUMAN BASELINE

  • The human baseline recruits 473 English-speaking participants from five countries, each answering three direct and three inferential questions in free-form, zero-shot trials.Questions are sampled to be approximately equally represented across participants and presented one at a time in randomized order.Source: §2.3 · p. 4

3 RESULTS

VLMs show a substantial direct-versus-inferential accuracy gap that humans do not; OccludeTrace locates the model bottleneck in recovering the object from a projection within the full image. Common prompting and tool interventions do not reliably close the gap, while inpaint-then-confront removes the significant difference for two image-generation-capable models.

3.1 MAIN RESULTS

  • 42.7% on direct images versus 62.7% on inferential images: VLMs average a 20.0 p.p. deficit on cue-based cases.The gap ranges from 10.4 p.p. for Llama 4 Maverick to 25.5 p.p. for Qwen 3 VL 8B.Source: §3.1 · p. 4
  • Humans reach 65.9% overall and show no significant direct–inferential gap (63.4% versus 68.5%), unlike VLMs.Human accuracy exceeds four of the five evaluated VLMs.Source: §3.1 · p. 4

3.2 OCCLUDETRACE: WHY DO VLMS FAIL?

  • Cue detection is 99% versus 100%, isolated recognition 79% versus 88%, and source mapping 85% versus 82% on failed versus successful examples.These first three stages test projection detection, reading it in isolation, and linking it to its source.Source: §3.2 · p. 5, Figure 5 · p. 5
  • 91% of successful examples but only 50% of failed examples yield object recovery at the highlighted-region stage, the only stage separating the groups.The diagnosis locates the bottleneck in visual extraction of the projection within the complete image, rather than noticing, reading, or source-linking the cue.Source: §3.2 · p. 6, Table 2 · p. 6

3.3 CAN WE CLOSE THE GAP?

  • Extended reasoning raises both GPT-5.4 categories but lowers both Claude Opus 4.6 categories; both models still do worse on direct images.Image-tool use also fails to close the gap: GPT-5.4 falls from 51.8% to 46.7%, while Claude Opus 4.6 rises from 45% to 52%.Source: §3.3 · pp. 6–7, Figure 7 · p. 7
  • Steering prompts improve several settings without closing the gap; few-shot prompting leaves gaps across all five models and can coincide with lower inferential accuracy.For Qwen 3 VL 8B, the smaller gap accompanies inferential accuracy decreasing from 46.4% to 37.8%.Source: §3.3 · p. 7, Figure 8 · p. 7
  • Inpaint-then-confront eliminates the significant direct–inferential gap for GPT-5.4 and Gemini 3.1 Pro, the two tested models with image-generation capabilities.The model first generates a candidate hidden object, then evaluates that hypothesis against the visible physical cue.Source: §3.3 · pp. 7–8, Table 3 · p. 8

4 RELATED WORK

Related work spans VLM evaluation, amodal completion, and commonsense inference, but reasoning about completely hidden objects from indirect visual cues remains underexplored.

VLMs and Evaluations

  • VLMs perform strongly across perception, multimodal evaluation, and complex reasoning benchmarks, yet vision-centric reasoning remains unsolved and hidden-object reasoning is underexplored.The cited work includes general visual understanding and reasoning benchmarks (Lu et al., 2024; Yue et al., 2024; Yu et al., 2024).Source: §4 · p. 8

Occlusion and Indirect Visual Evidence

  • Total occlusion remains underexplored in benchmarks, and prior work often treats shadows and reflections as noise rather than primary evidence for hidden objects.This paper instead uses those cues to reason about fully hidden objects.Source: §4 · p. 8

Inference and Evaluation Gaps

  • Commonsense and knowledge-based VQA address world knowledge, while benchmark analyses find tasks solvable without vision and task-relevant encoder information unused downstream.Human scene grammar and early perception of partly occluded objects are also established lines of work, but whether VLMs use such cues remains unclear.Source: §4 · p. 8

5 CONCLUSION AND DISCUSSION

OccludeBench tests VLM recovery of fully hidden objects under controlled evidence, and OccludeTrace locates a key reasoning bottleneck. Inpaint-then-confront reduces the direct–inferential gap for image-generation models, while reliable use of task-relevant evidence remains an open challenge.

5 CONCLUSION AND DISCUSSION

  • 52.3% accuracy for VLMs versus 65.9% for humans, with a 20.0-percentage-point drop on direct evidence cases and no corresponding human gap.OccludeBench controls what is hidden and what evidence remains, enabling comparison of direct visual cues with contextual inference.Source: §5 · p. 8
  • A roadside-mirror example shows Gemini rejecting an empty-road infill because the scene implies the mirror should reflect the car, then recovering that reflection.The model had missed the car’s reflection at baseline.Source: Figure 9 · p. 9
  • OccludeTrace finds that models detect, recognize, and map a projection, but struggle to identify the relevant object within it in the full image.Inpaint-then-confront asks models to generate a candidate hidden object and check it against the visible physical cue; it reduces the gap to a statistically non-significant difference for both tested image-generation models.Source: §5 · p. 9
  • Occlusion offers a controlled diagnostic, and the 378-image benchmark highlights reliable use of task-relevant evidence as an open challenge for current VLMs.The authors warn that evaluations can overestimate visual reasoning when models succeed through partial cues or learned associations rather than decisive evidence.Source: §5 · p. 9

A PROMPTS

The appendix documents the evaluation prompts, judging, and intervention procedures used in the benchmark, including open-ended responses and fixed prompt settings.

A PROMPTS

  • All evaluation prompts use temperature 0 and free-form responses, without multiple-choice options.The appendix provides the full prompt text and describes each prompt’s pipeline role.Source: A PROMPTS · p. 13

A.1 EVALUATION

A.2 JUDGING

  • Two independent judges compare free-form answers with the gold label, and their ensemble agrees with author labels on 98% of 378 items.The ensemble agreement exceeds either single judge’s agreement rate.Source: §A.2 · p. 13

Extended Reasoning Query (Same Task Prompt)

  • Few-shot runs prepend one direct and one inferential assistant trace, exclude those examples from evaluation, and score 376 items.Each trace rules out plausible contextual alternatives before justifying its answer with the visible cue.Source: Extended Reasoning Query (Same Task… · p. 15

A.6 TOOL USE

  • Tool-use runs allow at most five rounds, require exactly one tool call per round, and terminate when the model submits a final answer.The wrapper supplies image dimensions so models can issue pixel coordinates directly.Source: §A.6 · p. 16, Tool-Augmented Wrapper Instruction · p. 16

A.7 STEERING PROMPTS

  • Steering prompts direct attention to shadows and reflections; the longer version also constrains answer format and gives an inspection order.The long prompt is tested only in thinking and tool-use settings; the two prompt strengths separate attention direction from additional answer constraints.Source: §A.7 · p. 18, Long steering prompt · p. 18

A.8 INPAINTING PROMPTS

  • Inpainting first generates a candidate for the masked region, then a fresh model call judges its physical consistency beside the original before re-answering.GPT-5.4 uses its native image-generation tool, while Gemini 3.1 Pro uses an image-generation variant of its model family.Source: §A.8 · p. 18

A.3 ROBUSTNESS EXPERIMENTS

An unmasked-box control tests whether localization explains failures on occluded-object recognition; its reported accuracy suggests it does not.

A.3 ROBUSTNESS EXPERIMENTS

  • 89% to 93% overall accuracy on unmasked box contents, with near-equal performance for direct and inferential images, indicates localization is not the primary bottleneck.Direct accuracy is 86% to 94%, while inferential accuracy is 90% to 93%; the authors note that transparent-box accuracy is higher than occluded recognition.Source: §A.3 · p. 14
  • Table 4 compares model accuracy on occluded versus unmasked images, where unmasked images show the bounding box with its contents visible.The control uses the box query to test identification within the target region while avoiding a prompt that leaks the answer’s modality.Source: Table 4 · p. 14, §A.3 · p. 14

B PERFORMANCE ACROSS OCCLUSION CATEGORIES

Accuracy is roughly uniform across cue subcategories within each image type, so the direct–inferential gap is not driven by one or two unusually difficult subcategories. The findings instead point to differing evidence requirements.

B PERFORMANCE ACROSS OCCLUSION CATEGORIES

  • The direct–inferential gap is not driven by one or two difficult cue subcategories; accuracy is roughly uniform within each image type.The per-model omnibus tests compare correctness across cue subcategories separately for direct and inferential images.Source: B PERFORMANCE ACROSS OCCLUSION… · p. 19, Figure 10 · p. 19

C DATASET BREAKDOWN AND SENSITIVITY

The source breakdown is dominated by Internet images, while the smaller Visual Genome and author-photo subsets are intended as a diversity check, not balanced comparison groups. The authors also report annotation checks and robustness to bounding-box color changes.

C DATASET BREAKDOWN AND SENSITIVITY

  • The source-wise accuracies are noisy because Visual Genome and author-taken images form small subsets, not equally sized comparison groups.Of 378 images, 353 (93.4%) are from the Internet, 19 (5.0%) from Visual Genome, and 6 (1.6%) from the authors.Source: C DATASET BREAKDOWN AND SENSITIVITY · p. 20
  • Annotators discarded 41 images after disagreement about cue sufficiency, a 9.8% rejection rate; they did not disagree on direct versus inferential classification.Classification depended on whether a shadow or reflection was the primary cue.Source: C DATASET BREAKDOWN AND SENSITIVITY · p. 20
  • Annotators agreed that masks covered only the target object, did not fully occlude another object, and introduced no artifacts.Source: C DATASET BREAKDOWN AND SENSITIVITY · p. 20
  • Accuracy was not sensitive to changes in polygon background and border colors.The authors report this robustness check in Table 7.Source: C DATASET BREAKDOWN AND SENSITIVITY · p. 20, Table 7 · p. 20

D DIFFICULTY ACROSS CATEGORIES

Human accuracy does not differ significantly between direct and inferential questions, and the categories span similar difficulty ranges. Human responses also show that nearly all images are solvable, while lexical analysis finds no concreteness difference between categories.

D DIFFICULTY ACROSS CATEGORIES

  • Both categories span the full per-image difficulty range, with substantial mass in the middle rather than concentration at either extreme.The distributions cover images from nearly none to nearly all participants answering correctly.Source: D DIFFICULTY ACROSS CATEGORIES · p. 20, Figure 11 · p. 21
  • 63.4% direct and 68.5% inferential human accuracy did not differ significantly (U = 15977.5, p = 0.079).When images are ranked by difficulty, the two categories are interleaved rather than clustered at opposite ends.Source: D DIFFICULTY ACROSS CATEGORIES · pp. 20–21, Figure 12 · p. 21
  • At least one participant correctly identified the occluded object in 94.2% of images (356/378), including both categories.The category-specific rates were 93.9% for direct and 94.5% for inferential examples.Source: D DIFFICULTY ACROSS CATEGORIES · p. 21
  • Direct and inferential questions did not differ in concreteness, supporting the conclusion that model accuracy gaps reflect task type rather than systematically different question difficulty.The lexical comparison is reported in Table 8.Source: D DIFFICULTY ACROSS CATEGORIES · pp. 21–22, Table 8 · p. 21

E TOOL USE ANALYSIS

Tool use differs between the two models, but more calls do not correspond to better answers; whether a call targets the relevant projection matters more. The human-study interface presents six open-ended questions per participant, split across the two image types.

E TOOL USE ANALYSIS

  • GPT-5.4 leans heavily on crop and zoom, while Claude Opus 4.6 spreads calls across tools, rarely crops, and alternates geometric transforms.Source: E TOOL USE ANALYSIS · p. 22
  • Correct-answer groups used fewer tools on average, and call volume is a poor proxy for visual grounding effort.The target of a call—whether it lands on the correct projection region—matters more than the count; additional calls did not close the direct–inferential gap.Source: E TOOL USE ANALYSIS · p. 22

F.1 HUMAN STUDY INTERFACE

  • Each human-study participant answers six questions—three direct and three inferential—shown one at a time as an image with a free-form text box.Questions are sampled approximately uniformly across the dataset, with item order randomized per participant.Source: §F.1 · p. 22

F.2 LEARNING EFFECTS

Human accuracy stayed between 62% and 70% across question positions, with no significant trend; response times were also flat, so answering multiple survey questions did not improve performance.

F.2 LEARNING EFFECTS

  • 62%–70% accuracy across question positions showed no significant increasing or decreasing trend.Cochran-Armitage p = 0.108.Source: §F.2 · p. 23, Table 9 · p. 23
  • Response times were flat across question positions.Source: §F.2 · p. 23
  • Answering multiple survey questions did not improve human performance.Source: §F.2 · p. 23
Loading 2610.13790v1…