PaperlayerView exact revision on arXiv

Source-linked AI summary

Long Time No See: Benchmarking VLMs for Out-of-Sight Spatiotemporal Reasoning in Egocentric Videos

Fangzhou Ma, Ivo Alexander Ban, Eren Homburg, Gabriele Goletto, Rémi Pautrat, Mahdi Rad, Chiara Plizzari, Marc Pollefeys

arXiv:2609.34630v1cs.CV

TL;DR

Current benchmarks do not directly test whether VLMs retain an object’s updated spatial state after it is relocated and leaves view. BEYOND3D builds geometry-aware visibility tracks and diagnostic questions for that setting; the evaluation finds that current models remain far from reliable.

  • Problem

    Existing benchmarks cover related memory, temporal reasoning, and 3D scene understanding, but do not directly test retention of a relocated object’s updated spatial state after it leaves view.

    Source: §1 · p. 2

  • Method

    BEYOND3D uses geometry-aware visibility tracks to build questions about relocated objects after they are no longer observable, spanning visual and temporal grounding, scene localization, and 3D spatial perception.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Current VLMs remain far from reliable, with errors compounding across retrieval of past events, state maintenance, and reasoning from that state after the object is no longer visible.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The paper concludes that future work should move beyond current-view perception toward models that explicitly update and preserve persistent latent representations of dynamic scenes over time.

    Source: §6 · p. 10

    The static digital twin does not track movable fixture parts in real time, so its default closed-door geometry can falsely suggest that an object inside an open cabinet is occluded.

    Source: §10.2 · p. 18

Abstract

from arXiv · show

Real-world AI systems must reason about objects that are no longer visible: an AR assistant guiding a user back to an object used earlier, a household robot retrieving an item someone put away. This requires not just recalling where an object was last seen, but updating its state when it is moved and retaining that update once it leaves view. We refer to this as out-of-sight spatiotemporal reasoning. We introduce Beyond3D, the first VQA benchmark to isolate this ability in dynamic egocentric video: every query targets an object that has been relocated and has since left the field of view. We create our questions from HD-EPIC annotations, building a visibility track for each dynamic object from its 3D position, the camera pose, and the scene geometry to understand at each moment whether it is visible, occluded, or out of view. Beyond3D comprises 9,000 questions in eight types over 135 videos from nine participants, organized as one reasoning chain: visual grounding (is the target observable now), temporal grounding (when it was last visible and last placed), scene localization (which fixture anchors that location), and 3D spatial perception (where it lies relative to the current viewpoint or another object in the scene). We benchmark nine general-purpose and spatially specialized VLMs. The best model reaches 42.2% against 29.7% chance and text-only baselines reaching 31.9%, with the largest failures in recovering when an object was last visible, showing that tracking object movement out of sight remains far from solved for current VLMs.

1. Introduction

The paper frames out-of-sight spatiotemporal reasoning as maintaining object state after relocation and loss of visibility, and introduces BEYOND3D to test this ability in egocentric video.

1. Introduction

  • Existing benchmarks cover memory, temporal reasoning, or 3D scene understanding, but do not directly test whether models retain a relocated object’s updated spatial state after it leaves view.The paper identifies this as an open evaluation gap for vision-language models.Source: §1 · p. 2
  • BEYOND3D queries relocated objects only after they become unobservable, using visibility tracks built from object location, camera pose, field of view, and scene geometry.The tracks distinguish visible, occluded, and out-of-view states over time.Source: §1 · p. 2
  • Its 9,000 questions cover visual and temporal grounding, scene localization, and 3D spatial perception, tracing from event retrieval to spatial relations after the object leaves view.The paper describes this as an eight-question reasoning chain.Source: §1 · p. 2
  • The evaluation of nine VLMs identifies temporal retrieval and spatial-state maintenance as major bottlenecks, with substantial 3D reasoning errors persisting when upstream uncertainty is controlled.The paper also describes persistent dynamic world models as needing to retain unseen objects and update their states after interactions change them.Source: §1 · p. 2, §2 · p. 2

2. Related Work

  • The benchmark comparison distinguishes BEYOND3D by requiring relocation and loss of visibility in unscripted real-world video, with visibility verified geometrically.Related memory benchmarks do not ground memory in 3D, while dynamic spatial-state benchmarks may permit current-frame shortcuts.Source: §2 · p. 3

3. Out-of-Sight Spatiotemporal Reasoning

The task follows relocated objects through visibility changes and tests whether models can recover their state once they are no longer visible. Its questions diagnose successive stages from visibility to 3D spatial reasoning.

3.1. Problem Formulation

  • The task tracks active objects that the wearer manipulates and relocates, updating their status during movement and recalling them after they leave view.Each stationary location is associated with a semantic fixture that supports or contains the object.Source: §3.1 · p. 3
  • A query anchor is valid only when the target was relocated before query time and is stationary but not visible at the query.The out-of-sight horizon measures time since the target was last visible; reference-object questions use a stationary, visible object as a scene-relative reference.Source: §3.1 · p. 3

3.2. BEYOND3D Q&A Formulation

  • Eight diagnostic question types span visual grounding, temporal grounding, scene localization, and 3D spatial perception, probing successive stages from current observability to remembered location reasoning.The visual-grounding questions test whether the model recognizes the target and determines its visibility at query time.Source: §3.2 · p. 3

1 Visibility Check: Is o visible at Tq?

The question set checks temporal retrieval, scene localization, and spatial relations for an object that has left view; the construction figure shows how visibility checks lead to out-of-sight questions.

1 Visibility Check: Is o visible at Tq?

  • Temporal grounding asks when the object was last visible and when it was last placed before the query.Source: §1 · p. 3
  • Scene localization asks which fixture is closest to the object’s last known location.Source: §1 · p. 3
  • 3D spatial questions ask for the target’s direction and distance relative to the camera or a visible reference object.Source: §1 · p. 3
  • The construction figure links visibility tracks inferred through view, occlusion, and detection checks to valid out-of-sight anchors converted into 9,000 questions.Source: Figure 2 · p. 4

4. Benchmark Construction

BEYOND3D is constructed from HD-EPIC by tracking relocated objects’ visibility and selecting out-of-sight query anchors for eight types of questions. The resulting evaluation set spans varied query times, out-of-sight horizons, and object relocations.

4.1. Visibility Tracks

  • Visibility tracks combine 3D locations, camera poses, and scene geometry to label objects as visible, out of view, occluded, visually unconfirmed, or in motion.The track construction assumes an object remains at its last annotated location until its next movement and marks movement intervals as in motion.Source: §4.1 · p. 4
  • 83.5% accuracy was achieved across 4,102 object marks from 344 frames in 30 videos, with most errors arising from visually unconfirmed states.The tracks were checked in an independent human annotation pass.Source: §4.1 · pp. 4–5

4.2. Q&A Generation and Statistics

  • Anchors require a relocated target explicitly labeled out of view or occluded at query time; visually unconfirmed states are excluded, and questions use fixed templates with answers and task-specific distractors.Temporal distractors come from increasing distance bins; scene-localization distractors use plausible alternatives, with counter placements distinguished by nearby landmarks.Source: §4.2 · p. 5
  • The 1,000 out-of-sight anchors are approximately balanced across three query-time groups and three horizon groups; 70% involve one or two target moves, with a long tail to 11.The set spans 135 videos, nine participants, and nine kitchens; 900 targets are out of view and 100 are geometrically occluded at query time.Source: §4.2 · p. 5
  • 9,000 questions cover 135 videos, nine participants, and 581 target instances, using 1,000 out-of-sight anchors and 1,000 visible controls.The out-of-sight anchors expand into eight question types; visible controls provide positive examples for visibility questions.Source: §4.2 · p. 5

5. Experiments

Across nine VLMs, video improves performance over text-only input, but the strongest model reaches only 42.2% macro accuracy. Errors increase with longer out-of-sight horizons, and analyses of temporal cues and reasoning traces point to temporal retrieval, state retention, and 3D reasoning as continuing challenges.

5.1. Experimental Setup

  • Models were evaluated without task-specific fine-tuning, using 1 fps video prefixes and macro accuracy that weights each question type equally.The nine models include general-purpose and spatially specialized VLMs; shorter-context models received uniformly subsampled frames.Source: §5.1 · p. 6

5.2. Main Results

  • 42.2% macro accuracy is the best overall result, achieved by Qwen-3.6-27B; video improves every model’s macro accuracy by 1.6–10.5% over text-only input.Video yields the largest average gains on Visibility Check (+9.4%) and Nearest Fixture (+14.4%), while temporal grounding and spatial tasks improve less consistently.Source: §5.2 · p. 7

5.3. Diagnosing Failure Modes

  • Macro accuracy falls from 40.4% at short out-of-sight horizons to 31.9% at long horizons, while accuracy across query times is non-monotonic.Most models remain close to random on progressively widened timestamp bins, suggesting weak preference for the correct temporal region; visibility and spatial predictions also show model-specific biases.Source: §5.3 · pp. 7–8, Figure 5 · p. 7, Figure 6 · p. 7

5.4. Ablation Study

  • Temporal cues improve all downstream question types, while adding explicit invisibility information improves Nearest Fixture accuracy by 8% but 3D spatial tasks by only 0–2.4%.The results support temporal evidence retrieval as an upstream bottleneck, while visibility awareness helps scene grounding but leaves most geometric errors unresolved.Source: §5.4 · pp. 8–9, Figure 9 · p. 8

5.5. Qualitative Analysis

  • Qualitative traces show failures from missed relocations, target-identity confusion, and incorrect camera-relative mapping despite recovering an object’s location.For the food processing lid, the model tracked its movements and recovered its counter location but predicted front-right instead of back-right.Source: §5.5 · p. 9, Figure 10 · p. 9

6. Conclusions

The conclusion presents BEYOND3D as a benchmark for out-of-sight spatiotemporal reasoning in dynamic egocentric video. It finds that current VLMs remain unreliable across event retrieval, state maintenance, and spatial reasoning, and identifies persistent representations of dynamic scenes as a future-work direction.

6. Conclusions

  • BEYOND3D queries relocated objects only after they become unobservable, testing whether VLMs update and retain spatial state and reason about it in space and time.The benchmark decomposes the task into eight questions and four capabilities.Source: §6 · p. 9
  • Recent VLMs remain far from reliable, with errors compounding across past-event retrieval, object-state maintenance, and reasoning after the object leaves view.The conclusion identifies explicit updating and preservation of persistent latent scene representations as a future-work direction.Source: §6 · p. 10

7. Supplementary

The supplementary material provides additional benchmark-construction details, implementation information, visibility-track analyses, and human validation.

7. Supplementary

  • Section 8 expands on question templates, answer and distractor generation, and dataset distributions.Source: §7 · p. 13
  • Section 9 provides model-inference implementation details, including visual preprocessing and prompts for baseline and oracle interventions.Source: §7 · p. 13
  • Sections 10 and 11 detail visibility-track construction and report human validation results.Section 10 includes additional illustrations of object-location inference and cross-view projection.Source: §7 · p. 13

8. Benchmark and Statistics

The benchmark uses multiple-choice questions about relocated objects that are stationary and out of sight at query time, deriving answers from visibility tracks, movement annotations, object positions, and camera poses. Distractors and answer distributions are designed to test temporal, fixture, and 3D reasoning while limiting answer-frequency shortcuts.

8.1. Question Templates and Answer Choices

  • Questions target stationary, out-of-sight objects that were moved earlier, using templates with placeholders for the target, visible reference object, and query timestamp.The question set spans visual and temporal grounding, scene localization, and 3D spatial perception.Source: §8.1 · p. 13, Table 3 · p. 14, §8.3 · p. 14

Ground-truth answer derivation

  • Temporal answers come from visibility transitions or placement events; fixture answers use the closest annotated fixture, with counter instances manually given distinct human-readable descriptions.Among 16,213 placements with known fixtures, 63.2% were on counters; collapsing them into one label would introduce answer bias and discard spatial detail.Source: Ground-truth answer derivation · p. 13
  • 3D answers express the target’s last known position in camera coordinates or relative to a visible reference object, then map directions and distances to predefined categories.Object-relative coordinates shift the origin to the reference object while retaining the camera’s orientation.Source: Ground-truth answer derivation · pp. 13–14

Option construction

  • Temporal distractors use four increasingly distant timestamp bins, while fixture distractors use alternative counter areas in the same kitchen or other fixture categories.Nearest Fixture has variable option counts and a question-weighted chance level of 22.7%; the other question types have fixed option counts.Source: Option construction · p. 14

8.3. Answer Distribution

  • Temporal correct answers are balanced across five chronological ranks, and spatial answers are balanced across semantic classes; Nearest Fixture retains naturally occurring label frequencies.The spatial classes are exactly balanced for Object–Camera Direction and balanced 334/333/333 for the other three spatial questions.Source: §8.3 · p. 14, Figure 11 · p. 15

9. Technical Details for Model Inference

Models receive sampled egocentric video up to query time and multiple-choice prompts. Preprocessing marks visible reference objects when needed, while cue interventions add movement intervals or state explicitly that the target is no longer visible.

9.1. Visual Input Preprocessing

  • Videos are sampled at 1 fps, resized to 448×448, masked outside the circular fisheye view, and annotated with timestamps outside the visible camera field.For models with shorter context limits, frames are uniformly subsampled, while the query-time frame is always retained.Source: §9.1 · p. 15, Figure 13 · p. 17
  • Object–Object questions mark the visible reference object with an 8 × 8 red marker in the query-time frame only, identifying it without revealing its trajectory.The marker is selected to remain visible after resizing while minimally occluding surrounding content.Source: §9.1 · p. 15

9.2. Inference Prompt

  • The shared inference prompt asks models to select a multiple-choice answer; cue experiments prepend target movement intervals or state that the target is no longer visible.The visibility cue is added to downstream questions without repeating the template’s original assumption clause.Source: §9.2 · pp. 15–16

10. Technical Details for Visibility Track Construction

Visibility tracks combine inferred object locations with field-of-view projection, fixture-aware occlusion, and detector confirmation. They represent uncertain visibility separately and account for limits in the static kitchen geometry and object-name annotations.

10. Technical Details for Visibility Track Construction

  • Tracks are sampled at 1 fps and record both inferred object location and visibility state.HD-EPIC provides movement endpoints rather than intermediate trajectories, so stationary locations are inferred between annotated events.Source: §10 · p. 17, §10.1 · p. 17, Figure 14 · p. 17

10.1. Inferring Object Locations from Movement Annotations

  • Before an object’s first movement its first annotated start location is used; during movement it is in motion, and afterward it remains at its latest completed endpoint.Objects in motion are treated as visible but have no stable location and cannot serve as query anchors.Source: §10.1 · p. 17, Figure 14 · p. 17

10.2. Determining Visibility

  • Visibility is determined by checking field-of-view projection, fixture occlusion, and then detector confirmation, with consecutive matching geometric outcomes merged before detection scores intervals.The projection uses nine points from an object bounding box; at least five must be valid for the object to be in view, and at least half of rays must be blocked for geometric occlusion.Source: §10.2 · pp. 17–18
  • Because the static digital twin omits real-time states of movable fixtures, possible false occlusions by an object’s containing fixture are marked fixture ambiguous and deferred to detector confirmation.Names that cannot be mapped to a single visually identifiable object are excluded from the visibility track and recall calculations.Source: §10.2 · p. 18

11. Human Validation of the Visibility Tracks

A human audit compared track states with visibility judgments on sampled frames across 30 videos and all nine participants. The audit reports inter-rater agreement, pipeline metrics, error patterns, and participant-level variation.

Audit protocol

  • The audit sampled frames round-robin across participants, favoring frames with at least three in-view objects, and matched judgments to track states within a one-second tolerance.Frames with dense object populations were capped at 15 randomly selected markers, and annotators could flag uncertain marks.Source: §11 · p. 19

Inter-annotator agreement

  • Krippendorff’s α was 0.76 across 634 multiply rated marks, with unanimous judgments on 82.0% and pairwise Cohen’s κ values of 0.86, 0.75, and 0.68.Annotators’ visible rates ranged from 26.1% to 34.4%, indicating a threshold effect in some disagreements.Source: §11 · p. 19

Overall metrics

  • Scoring covered 4,102 marks against the majority human vote, and every metric improved when restricted to unanimous multiply rated marks.The authors characterize headline metrics as conservative because of label ambiguity; the table reports pipeline-versus-majority-vote metrics with video-cluster bootstrap intervals.Source: §11 · p. 19, Table 5 · p. 20

Where the errors are

  • The audit found 488 false negatives and 189 false positives; 372 false negatives were visually unconfirmed cases where OWLv2 failed to confirm human-visible objects.Balanced accuracy varied from 66.8% for P08 to 85.2% for P09, with precision and recall also varying across kitchens.Source: §11 · pp. 19–20, Table 6 · p. 20

Related papers

Loading 2609.34630v1…