Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models

1 Introduction

• Prior spatial-grounding work typically provides the viewpoint, makes the relevant scene directly available, or localizes an already observed target, leaving implicit contextual reasoning unclear.Everyday communication instead relies on shared environmental knowledge, activity context, and assumptions about viewpoint.Source: §1 · pp. 1–2
• Contextual observer grounding requires inferring the speaker’s viewpoint from activity and environmental cues, then interpreting the situated spatial relation without directly observing the target scene.The inferred observer-relative scene is assembled from previous environment observations.Source: §1 · p. 2
• POVBench uses Inferred, Stated, and Given conditions to test observer grounding with varying explicitness while keeping query construction fixed.Inferred versus Stated tests recovery of grounding from context; Stated versus Given tests relation reconstruction without the observer view.Source: §1 · p. 2
• Across multiple VLMs, performance remains low in all three grounding conditions, and the modest Inferred–Stated difference indicates that resolving observer-relative relations is also challenging.The evaluation includes proprietary and open-weight models and dedicated pointing architectures.Source: §1 · p. 2
• Explicit step-by-step reasoning over observer-relative spatial relations improves target-object localization.The authors describe this as a potential direction for embodied spatial reasoning.Source: §1 · p. 2

2.1 3D Spatial Reasoning

• Existing 3D spatial-reasoning work includes metric-space reasoning, language-guided localization, and embodied question answering over explored environments.These lines of work address different spatial and embodied tasks.Source: §2.1 · pp. 2–3

2.2 Perspective-Taking and Observer Grounding

• Related situated-reasoning benchmarks cover specified agent perspectives, egocentric-video questions, spatial mental models, or multi-view reasoning with explicit viewpoints, but do not require natural implicit descriptions to predict both perspective and an unobserved target location.The paper positions POVBench as combining those requirements.Source: §2.2 · p. 3
• POVBench couples inferring an implicit observer location from contextual cues with reconstructing that perspective from multi-view exploration, without a direct observer-view image.The authors frame this as a more realistic household-agent scenario.Source: §2.2 · p. 3

3.1 Task Formulation

• Given a sentence and first-person images from robot exploration, the model predicts an image index and pixel coordinate for the target.The target is typically absent from the observations, while its anchor object may be visible; localization therefore requires inference rather than direct detection.Source: §3.1 · p. 3

3.2 Taxonomy

• The task separates operational reasoning about the observer’s position, observer grounding to reconstruct their perspective, and relational reasoning to interpret language relative to an anchor.The spatial relation is grounded in the observer’s egocentric frame, not an absolute frame.Source: §3.2 · pp. 3–4, §3.1 · p. 3

3.3 Forms of Observer Grounding

• Inferred, Stated, and Given form a graded difficulty axis, with progressively more explicit viewpoint information removing one reasoning step at each level.Inferred supplies the least grounding and is the paper’s central focus.Source: §3.3 · p. 4

4.1 Environment Setup

• POVBench is built in ProcTHOR-generated houses, with multi-view exploration images captured along a simulated robot trajectory.The environments span household rooms and use controllable layouts and scene geometry for query generation and geometric validation.Source: Figure 2 · p. 4, §4.1 · p. 4

4.2 Query Generation

• Each query describes a target last seen relative to an anchor while the observer engages with an activity object, using right, left, front, or up relations.Inferred, Stated, and Given sentences differ in how much observer-grounding information they convey.Source: §4.2 · pp. 4–5

4.3 Ground-Truth Construction

• Target coordinates are computed for physically plausible placements and filtered against the floor plan to exclude points inside walls or outside scene boundaries.The remaining 3D coordinates are projected into image space using a pinhole camera model.Source: §4.3 · p. 5

4.4 Dataset Statistics

• The benchmark contains 47 houses, 288 observer-centric scenarios, and 813 evaluation instances per condition; front placements occur in 97% of pairs.Each scenario has up to four directions and averages 2.8 valid placement directions per pair.Source: §4.4 · p. 5
• In the Given condition, 92.9% of ground-truth targets fall within the observer-view image; Inferred and Stated in-frame rates depend on the model’s selected exploration frame.Source: §4.4 · p. 5

5.1 Experimental Setup

• In Inferred and Stated, models receive up to 30 indexed exploration frames and a sentence, select a frame, and predict target-center coordinates; Given uses one observer-view image and direct coordinates.Source: §5.1 · pp. 5–6

5.2 Main Results

• Inferred–Stated normalized L2 error differs by at most 0.012; Given is generally lower but remains substantial, with MolmoPoint-8B best on Inferred/Stated and GPT-5.4 on Given.Source: Table 1 · p. 6, §5.2 · p. 6

5.3 Failure Analysis

• Models sometimes choose frames without the relevant lamp, confuse a door frame with a refrigerator, or misinfer facing direction even after selecting the correct frame.These patterns point to difficulty integrating activity cues, visual perception, and contextual reasoning.Source: §5.3 · pp. 6–7
• UP localization error can penalize valid placements because scoring uses the anchor center despite acceptable horizontal variation; out-of-range coordinates also affect error, including 21.0% of Qwen3-VL-32B Inferred instances.Source: §5.3 · p. 7

5.4 Controlled Interventions

• Marking only the observer increases error over baseline, while adding the anchor marker restores and slightly improves performance.Source: §5.4 · p. 7
• Spatial-CoT gives the largest localization improvement by guiding directional-frame definition, anchor localization, scene inventory, and target estimation, but CoT methods sometimes fail to return parseable answers.Source: §5.4 · p. 7

5.5 Real-World Scene Evaluation

• Basic-prompt success stays at or below 50% in every real-world condition; S-CoT improves Qwen3-VL-8B accuracy in Inferred and Stated, though the evaluation is small-scale.Source: Table 4 · p. 8, §5.5 · pp. 8–9

• POVBench evaluates contextual observer grounding: reasoning about situated spatial relations from contextual and remembered environmental information.Source: §6 · p. 9
• Across ten VLMs, performance remains limited in all grounding conditions, and explicitly identifying the observer provides only modest benefit.Source: §6 · p. 9
• Gains from structured observer-relative reasoning suggest it may be a key capability for embodied AI; future work includes real-world settings and embodied models.Source: §6 · p. 9

• POVBench is primarily built in ProcTHOR, and may not capture the full complexity of real-world embodied perception and interaction.Source: Limitations · p. 9
• The benchmark covers indoor household settings, four directional relations, and limited activity contexts; it evaluates image-space localization rather than classification, multi-hop inference, or navigation.Source: Limitations · p. 9
• Semi-automatic target placement may leave residual ground-truth noise, while a single canonical location can penalize alternative valid placements under point-based L2 evaluation.Source: Limitations · p. 9
• The study does not decompose errors into reasoning steps; explicit intermediate observer predictions alter target predictions, and Spatial-CoT gains are not attributed between reasoning and prompt format.Source: Limitations · p. 9
• Evaluation is limited to passive inference from exploration observations, not interactive clarification or embodied exploration policies.Source: Limitations · p. 9

• Figure 6 shows the distribution of room counts across the benchmark houses.Source: §B.1 · p. 11

B.2 Observer / Anchor Object Categories

• POVBench uses 31 candidate object categories for observer and anchor objects.Source: §B.2 · p. 11

B.3 Target Object Selection Policy

• Target objects are sampled from pools matched to their support surface, with floor-compatible and receptacle-specific pools illustrated by the bed examples.Source: §B.3 · p. 11

B.4 Ground-Truth Placement Rules

• Relations are defined from the observer’s perspective facing the anchor; placement uses the anchor’s bounding box, relation direction, support-surface height, and specified offsets.For flat-topped anchors, LEFT/RIGHT placement on the anchor surface may be laterally offset from its center; the 0.2 m offset reflects a 0.4 m target cube.Source: §B.4 · pp. 11–12

B.5 Evaluation Prompts

• The CoT prompt asks models to select a scene, infer observer position and anchor, then estimate the target; Spatial-CoT structures spatial reasoning around the directional frame, anchor, nearby objects, and target.Source: §B.5 · p. 13

B.6 Model Configurations

• RoboPoint and Llama-3.2-Vision are evaluated only under Given because they do not produce valid responses in multi-image settings; Qwen3-VL-8B has two implementations and InternVL3-38B uses 8-bit quantization.Source: Table 8 · p. 14

B.7 Response Rates

• A response counts as a parseable coordinate prediction; unparseable outputs and API failures count as non-responses.Source: §B.7 · p. 14, Table 9 · p. 14