Source-linked AI summary
Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?
Yue Zhang, Zun Wang, Han Lin, Yonatan Bitton, Idan Szpektor, Mohit Bansal
TL;DR
VLM spatial benchmarks often assume visual evidence is sufficient, despite occlusion and perspective ambiguity making observations incomplete or misleading. The paper introduces SPATIALUNCERTAIN to test abstention and informative-view selection, finding systematic overconfidence and weak viewpoint identification across eight VLMs. The study concludes that observational awareness requires evaluation beyond answer correctness, while its synthetic, single-step setting limits direct coverage of dynamic real-world environments.
Problem
Existing spatial benchmarks generally assume visual observations are sufficient and reliable, although occlusion and perspective can make spatial evidence incomplete or misleading.
Method
SPATIALUNCERTAIN uses controlled 3D scenes with occlusion and perspective ambiguity, designs questions with condition-dependent answerability, and evaluates abstention and informative-view selection.
Results
Across eight VLMs, models are systematically overconfident under incomplete or misleading evidence and struggle to identify informative viewpoints, while fine-tuning on diverse ambiguity conditions improves observational awareness.
Takeaways & Limitations
Evaluating whether VLMs assess observation reliability, abstain when needed, and seek additional evidence is necessary alongside measuring spatial-answer correctness.
Takeaways & Limitations
The study uses controlled synthetic 3D environments, focuses on occlusion and ambiguous viewpoints, and evaluates single-step reasoning rather than long-horizon interactive exploration.
Abstract
from arXiv · showhide
Spatial reasoning is a fundamental capability for vision-language models (VLMs) deployed in real-world environments. However, visual observations are inherently limited representations of a 3D world: occlusion can render objects invisible, and perspective can make geometric properties misleading. Despite this, existing spatial reasoning benchmarks typically assume that observations are sufficient and reliable, focusing on whether models produce correct answers rather than whether they recognize when a question cannot be answered and what additional observations would be needed. In this work, we challenge this assumption by constructing a controlled evaluation framework, SpatialUncertain, and introducing two types of observation challenges: (1) occlusion, which hides target information, and (2) perspective ambiguity, which produces misleading visual cues. For each configuration, we design spatial questions that are answerable under clean observations but require abstention under the introduced challenges. We further evaluate whether models can identify which additional viewpoints would resolve perspective ambiguity. Our results across a diverse set of frontier open- and closed-source VLMs reveal two consistent failure modes. First, models are prone to overconfident answering, attempting to solve spatial reasoning tasks even when visual evidence is incomplete or misleading, with average accuracy around 30\% under occlusion and below 10\% under perspective ambiguity. Second, even when additional views are available, some models perform near random chance in identifying which would provide reliable evidence. Together, our findings call for moving beyond answer correctness toward evaluating whether models know when to abstain and how to seek reliable evidence.
1 Introduction
Existing spatial benchmarks generally assume visual observations are sufficient and reliable, but real-world 2D projections can hide or distort the evidence needed for spatial reasoning. SPATIALUNCERTAIN evaluates whether VLMs recognize these failures, abstain appropriately, and seek informative additional views.
- Spatial reasoning benchmarks typically measure answers to questions about object relations, distance, or size from visual observations assumed to be sufficient and reliable.
- Occlusion can hide critical objects, while perspective can distort geometric properties, making spatial evidence incomplete or misleading.
- SPATIALUNCERTAIN starts from answerable clean 3D scenes and introduces occlusion and perspective-induced ambiguity as controlled observation perturbations.
- Across eight open- and closed-source VLMs, models confidently answer questions despite incomplete or misleading observations and struggle to identify reliable additional viewpoints.
- Structured prompting partially improves abstention but reduces answerable accuracy, whereas fine-tuning improves abstention when training includes diverse visual ambiguities.
2 Related Work
Prior work has expanded spatial-reasoning benchmarks from basic relations to broader, controlled, and multi-task evaluations. Separate uncertainty and abstention research motivates assessing whether models should withhold predictions when evidence is unreliable.
- Early spatial benchmarks evaluate relative relations, depth ordering, and size comparison using image- and video-based question answering datasets.
- Recent benchmarks broaden evaluation through large-scale and multi-task settings covering object relations and complex scene understanding.
- Uncertainty estimation and abstention research shows that neural networks can be overconfident and should sometimes abstain when predictions are uncertain.
3 SPATIALUNCERTAIN: Controlled Evaluation Framework
SPATIALUNCERTAIN constructs validated synthetic indoor scenes with controlled occlusion and perspective manipulations, then varies question answerability systematically across observation conditions. It also tests whether models can select informative viewpoints under perspective ambiguity.
- Framework and task design: The framework uses 3D simulated indoor scenes, controlled occlusion and perspective challenges, systematically varying question answerability with observation conditions.
- 3.2 Occlusion Configurations: Occlusion configurations place an occluder between the camera and target, with geometric constraints and human labels distinguishing partial from full occlusion.
- 3.3 Perspective Configurations: Perspective configurations compare an equidistant reference view with a laterally shifted view that brings one same-category object closer and changes its apparent geometry.
- 3.4 Task Design: The evaluation covers Visibility, Relative position, Depth ordering, and Size/Shape questions, with clean views answerable and some occluded or ambiguous cases unanswerable.
- Viewpoint assessment: ViewSel tests selecting one informative reference among candidate views, while AbstainViewSel first requires recognizing an ambiguous view as uninformative.
- Dataset construction: The dataset contains 240 scenes, 1,222 occlusion configurations, 701 perspective object pairs, and 10,322 generated question-answer pairs.
4 Experimental Results
Across eight VLMs, unreliable observations expose failures in abstention, viewpoint selection, and uncertainty assessment. Visual input and structured prompting help selectively, while fine-tuning is investigated as another improvement strategy.
- Evaluation setup: Eight VLMs are evaluated under occlusion and perspective ambiguity using answerability, unanswerability, and viewpoint-selection tasks.Table 1 distinguishes Ans. from Unans. and ViewS. from AbsViewS., which adds an abstention stage.
- Observational uncertainty: Models often answer confidently when observations are incomplete or misleading, with answerable performance exceeding random but inconsistent unanswerable performance.The results show an answer–abstention trade-off across models rather than a unified uncertainty-awareness capability.
- Viewpoint selection: Viewpoint selection falls sharply when models must first abstain on an ambiguous view: GPT-5.4 drops from 70.9 to 22.6, GPT-5-mini from 53.7 to 18.0, and Gemini-3.0-Flash from 50.3 to 2.4.ViewSel tests selection alone, whereas AbstainViewSel additionally requires recognizing the current view as uninformative.
- Effect of visual input: Visual input improves answerable performance and can improve occlusion unanswerable performance, but reduces unanswerable performance under perspective ambiguity.Under occlusion, GPT-5.4 gains +6.4 and Gemini-3.0-Flash +29.8 on Occ-Unans; under perspective, they drop -21.7 and -35.8 on Pers-Unans.
- Interventions: Structured prompting improves unanswerable performance unevenly but reduces answerable accuracy, so prompting alone does not reliably resolve observational uncertainty.GPT-5-mini’s Occ-Unans rises 7.8→30.4 while answerable accuracy falls 64.7→54.7; Gemini-2.5-Flash changes 45.0→48.7 and 56.1→50.4.
- Interventions: Fine-tuning experiments compare base, occlusion-only, perspective-only, and mixed LoRA variants to test generalizable abstention under observation uncertainty.The supplied passage introduces the four variants and the held-out validation and test splits.
5 Conclusion
SPATIALUNCERTAIN diagnoses whether VLMs recognize unreliable observations and seek informative evidence under occlusion and perspective ambiguity. Across eight models, it finds systematic overconfidence and weak viewpoint identification, while diverse-condition fine-tuning improves observational awareness.
- Conclusion: SPATIALUNCERTAIN evaluates observational awareness under occlusion and perspective ambiguity across eight vision-language models.The framework targets whether models assess observation reliability and seek additional evidence.
- Conclusion: Models are systematically overconfident with incomplete or misleading evidence and perform near random chance when identifying informative viewpoints.These are the two consistent failure modes reported by the evaluation.
- Conclusion: Prompting alone cannot resolve these failures, whereas fine-tuning on diverse ambiguity conditions substantially improves observational awareness.The conclusion contrasts prompting with fine-tuning across diverse ambiguity conditions.
A.1 SPATIALUNCERTAIN Construction Details
The construction procedure selects occlusion targets and perspective-sensitive object pairs so that controlled viewpoint changes create informative spatial ambiguities.
- Occlusion configuration: Occlusion targets are selected by visibility, combining angular centrality and apparent size while penalizing partial occlusion.The procedure retains the top-k objects with k=3 and requires scene-level uniqueness.
- Perspective configuration: Perspective ambiguity uses same-category object pairs with similar physical size that can appear substantially different under viewpoint distortion.The selected pairs are physically comparable but visually sensitive to viewpoint changes.
A.1.1 Human Annotation
Human validation filters generated occlusion and perspective configurations according to visibility and answerability criteria before inclusion.
- Validation process: Seven annotators independently reviewed assigned configurations during validation.The annotation interface is shown in Fig. 6.
- Occlusion annotation: Occlusion configurations are labeled no, partial, or full occlusion using paired clean and occluded views, with no-occlusion cases discarded.Target and occluder objects are identified by name, and approximately one-third of generated configurations were discarded.
- Perspective annotation: Perspective configurations are retained only when the reference view supports answering and the perspective view creates visible geometric ambiguity.Configurations failing either criterion are discarded; approximately one-third were removed, primarily for insufficient ambiguity.
A.2.1 Prompt Templates
The evaluation uses a standard prompt and a structured reasoning prompt. The structured version explicitly checks visibility and viewpoint reliability before allowing a specific answer.
- The standard prompt asks models to choose the best answer from visible evidence and permits “Cannot determine” without actively encouraging abstention.
- The structured reasoning prompt adds explicit checks for target visibility and viewpoint reliability before answer selection.
- If either reliability check fails, the structured prompt instructs the model to select “Cannot determine.”
A.2.2 Evaluation Metrics
The evaluation reports separate accuracies for answerable and unanswerable questions, overall accuracy, and viewpoint-selection metrics. It also describes fine-tuning details and identifies synthetic-scene and single-step evaluation boundaries.
- The evaluation reports Answerable Accuracy, Unanswerable Accuracy, and Overall Accuracy for answer selection.
- ViewSel measures the fraction of correctly selected views, while AbstainViewSel measures correct abstain-and-select cases.
- Qwen2.5-VL-7B-Instruct is fine-tuned with separate LoRA adapters on occlusion and perspective training splits to study cross-condition abstention transfer.
- The framework uses controlled synthetic 3D environments and focuses on occlusion, ambiguous viewpoints, and single-step reasoning rather than full interactive exploration.