Source-linked AI summary
SAW-Bench: Learning Situated Awareness in the Real World
Chuhan Li, Rilyn Han, Joy Hsu, Yongyuan Liang, Rajiv Dhawan, Jiajun Wu, Ming-Hsuan Yang, Xin Eric Wang
TL;DR
Existing multimodal benchmarks largely overlook observer-centric spatial reasoning grounded in an agent’s viewpoint, pose, and motion. SAW-Bench evaluates this ability using annotated egocentric real-world videos, finding that Gemini 3 Flash reaches 53.89% accuracy versus 91.55% for humans.
Problem
Existing multimodal benchmarks largely emphasize environment-centric object relations while overlooking observer-centric spatial relationships grounded in an agent’s viewpoint, pose, and motion.
Method
SAW-Bench evaluates observer-centric spatial understanding across six awareness tasks using 2,071 human-annotated questions over 786 egocentric videos.
Results
53.89% accuracy was achieved by Gemini 3 Flash versus 91.55% for humans, revealing a substantial human–model performance gap.
Takeaways & Limitations
Situated awareness remains a significant challenge for current multimodal foundation models, and high-level semantic summaries alone are insufficient for solving it.
Takeaways & Limitations
The benchmark acknowledges potential downstream misuse of models that perform well on situated-awareness tasks.
Abstract
from arXiv · showhide
A core aspect of human perception is situated awareness, the ability to relate ourselves to the surrounding physical environment and reason over possible actions in context. However, most existing benchmarks for multimodal foundation models (MFMs) emphasize environment-centric spatial relations (relations among objects in a scene), while largely overlooking observer-centric relationships that require reasoning relative to agent's viewpoint, pose, and motion. To bridge this gap, we introduce SAW-Bench (Situated Awareness in the Real World), a novel benchmark for evaluating egocentric situated awareness using real-world videos. SAW-Bench comprises 786 self-recorded videos captured with Ray-Ban Meta (Gen 2) smart glasses spanning diverse indoor and outdoor environments, and over 2,071 human-annotated question-answer pairs. It probes a model's observer-centric understanding with six different awareness tasks. Our comprehensive evaluation reveals a human-model performance gap of 37.66%, even with the best-performing MFM, Gemini 3 Flash. Beyond this gap, our in-depth analysis uncovers several notable findings; for example, while models can exploit partial geometric cues in egocentric videos, they often fail to infer a coherent camera geometry, leading to systematic spatial reasoning errors. We position SAW-Bench as a benchmark for situated spatial intelligence, moving beyond passive observation to understanding physically grounded, observer-centric dynamics.
1. Introduction
SAW-Bench addresses the underrepresentation of observer-centric situated awareness in multimodal foundation-model benchmarks by evaluating embodied, viewpoint-dependent understanding in real-world egocentric videos. Its 2,071 human-annotated question-answer pairs across 786 videos and six awareness tasks reveal a substantial gap between model and human performance, alongside systematic spatial-reasoning failures.
- Motivation: Situated awareness requires relating an agent’s location, orientation, and movement to its own body and perspective while acting in the physical world.This capability supports grasping and navigation in robotics and perspective synchronization in AR/VR.
- Motivation: Existing multimodal benchmarks largely evaluate detached, observer-independent spatial reasoning rather than viewpoint-, motion-, and position-dependent understanding.They emphasize object-object interaction, discrete mental simulation, and distance estimation, treating models as passive spectators.
- Benchmark: SAW-BENCH contains 786 egocentric indoor and outdoor videos and 2,071 human-annotated multiple-choice question-answer pairs spanning six awareness tasks.The tasks are localization, relative direction, route shape, reverse route plan, spatial memory, and spatial affordance.
- Results: 53.89%: Gemini 3 Flash’s SAW-BENCH performance remains below 91.55% human performance across the benchmark evaluation.The evaluation covers 17 open-source and 6 proprietary multimodal foundation models.
- Results: Models often conflate egocentric camera rotation with translational movement, and their accuracy degrades significantly as trajectory complexity increases.These findings are reported as systematic patterns in model behavior.
2. Related Work
Prior benchmarks evaluate temporal video understanding, reconstructed 3D spatial intelligence, or observer-independent visual-spatial reasoning. SAW-Bench addresses limitations in situated evaluation by focusing on dynamically emerging, observer-centric context in real-world egocentric videos.
- General video understanding benchmarks: Video understanding benchmarks increasingly test long-form and visual-temporal reasoning over multimodal information across time.These benchmarks evaluate models’ abilities to perceive, reason, and integrate visual and linguistic information over videos.
- 3D spatial intelligence: 3D spatial intelligence benchmarks predominantly reason over explicit reconstructed representations, including point clouds and meshes.Examples include ReferIt3D, ScanRefer, and related datasets evaluating spatial reasoning from reconstructed geometry.
- 3D spatial intelligence: SQA3D introduces questions anchored to a position and orientation, but its egocentric videos support reconstruction and scene coverage rather than natural movement.Its situation is specified as a static pose with a fixed position and facing direction, unlike context emerging from an ongoing trajectory.
- Visual-spatial intelligence benchmarks: Visual-spatial benchmarks often overlook fine-grained spatial intelligence and frame reasoning through observer-independent tasks.This framing misses the observer-centric nature of embodied spatial tasks.
3. Situated Awareness Benchmark
SAW-Bench evaluates situated awareness through six observer-centric tasks using 2,071 human-annotated question-answer pairs from 786 egocentric videos. Its design preserves viewpoint-dependent spatial perception across diverse real-world environments while applying protocol-based annotation and quality control.
- Task Taxonomy: SAW-Bench defines six tasks covering self-localization, relative direction, route shape, reverse route planning, spatial memory, and spatial affordance.Together, these tasks require reasoning about the relationship between the observer and the environment.
- Task Taxonomy: The task distribution is led by Relative Direction (40.27%), followed by Route Shape (26.36%), Reverse Route Plan (11.06%), Self-Localization (9.66%), Spatial Affordance (7.82%), and Spatial Memory (4.83%).Each percentage is the reported share for its corresponding task.
- Data Collection: Egocentric video captures object positions relative to the wearer, viewpoint changes from head and body movement, and visibility changes over time.This preserves observer-centric spatial structure during real-world interaction.
- Annotation and Quality Control: QA pairs follow predefined recording protocols and known trajectories, while videos and annotations undergo human review, filtering, re-filming, independent annotation, and disagreement resolution.These procedures target consistent coverage, video quality, and annotation accuracy.
4. Experiments
Experiments evaluate diverse open-source and proprietary multimodal foundation models alongside chance, blind-LLM, and Socratic baselines using accuracy on SAW-BENCH. Results show substantial difficulty with situated awareness, with proprietary models generally outperforming open-source systems and visual access proving necessary.
- Experimental Setup: The evaluation tests six open-source and two proprietary MFMs in a zero-shot setting.Open-source models include Qwen2.5-VL, Qwen3-VL, InternVL 2/3, LLaVA-NeXT-Video, and LLaVA-OneVision; proprietary models include GPT and Gemini.
- Baselines: Baselines include random and frequent-answer chance levels, a GPT-5.2 blind LLM, and Socratic models using holistic video captions.The blind LLM receives no visual information, whereas Socratic models access video indirectly through captioning.
- Evaluation Protocol: Accuracy is the primary metric for evaluating model performance on SAW-BENCH.Predicted answers are extracted with a regular-expression parser, with GPT-4o-mini used when parsing fails.
- Overall Results: 53.89% overall accuracy for Gemini 3 Flash leaves a 37.66% gap behind humans at 91.55%.The result indicates that situated awareness remains a fundamental challenge for current multimodal foundation models.
- Model Comparisons: Proprietary MFMs generally outperform open-source MFMs, with the largest gap on Reverse Route Plan.Reverse Route Plan requires sustained reasoning over egocentric trajectories and explicit tracking of intermediate movements; open-source models often rely on first- and last-frame shortcuts.
- Baseline Comparisons: 31.34% overall accuracy for both the blind LLM and Socratic model shows that visual access is necessary and captioning does not significantly improve performance.Static language-based video representations discard observer-centric cues including viewpoint changes, orientation, and temporal structure.
5. Analysis
The analysis identifies systematic observer-centric failures in trajectory interpretation and object persistence, while showing that environment openness alone does not predict spatial reasoning difficulty.
- Trajectory reasoning: Camera rotation causes systematic Route Shape errors when head orientation changes are mistaken for physical zigzag motion.With identical translational motion, models misclassify frequent head-rotation cases as zigzag trajectories, revealing weak observer-centric coordinate systems.
- Trajectory reasoning: 60.0% of cases are misclassified by Gemini 3 Flash, versus 53.3% by Qwen3-VL 235B, under frequent head rotations.Both models confuse camera orientation shifts with physical body displacement.
- Trajectory reasoning: Model accuracy degrades significantly as trajectory complexity increases, particularly with multiple orientation changes.Relative Direction performance is stratified across Straight, Single Turn, and Two Turns trajectories.
- Spatial memory: Persistent tracking of objects across frames remains an open challenge across models.Models often treat objects leaving the field of view as absent or changed, relying on view-dependent evidence rather than persistent world-state representations.
- Environment effects: Environment openness alone is an insufficient proxy for spatial reasoning difficulty.Outdoor performance is often comparable to or higher than indoor performance, and indoor scenes may be harder because of greater object clutter and relational ambiguity.
6. Conclusion
SAW-BENCH addresses the limited evaluation of situated awareness in multimodal models by framing spatial cognition around embodied agents’ viewpoints, motion, and position. It introduces a benchmark with 2,071 human-annotated question–answer pairs.
- Conclusion: Situated awareness links an agent to its physical environment and supports reasoning over possible actions in context.The passage describes it as a foundational component of human spatial cognition.
- Conclusion: Current multimodal evaluation frameworks have largely overlooked situated awareness by treating models as passive observers rather than embodied agents.The overlooked observer-centric factors are viewpoint, motion, and position.
- Conclusion: SAW-BENCH bridges this gap with a benchmark containing 2,071 human-annotated question–answer pairs.
Impact Statement · A. Experiment Setup · A.1. Model Configuration
SAW-BENCH evaluates observer-centric situated awareness in multimodal foundation models using egocentric videos, addressing spatial understanding that existing frameworks underrepresent. The paper also discusses potential benefits, misuse risks, and evaluation model configurations.
- Impact Statement: SAW-BENCH evaluates observer-centric situated awareness in multimodal foundation models through egocentric videos.It serves as a diagnostic tool for spatial understanding capabilities underrepresented in existing evaluation frameworks.
- Impact Statement: The benchmark targets spatial relationships from an embodied agent’s or human wearer’s perspective.This perspective is identified as critical for safe and effective operation in robotics, AR/VR, and assistive technologies.
- Impact Statement: Potential applications include robotics, augmented and virtual reality, and assistive technologies.These domains may benefit from improved reliability when systems must understand spatial relationships from an embodied perspective.
- Impact Statement: The benchmark does not introduce direct pathways to harm, but downstream misuse remains possible.Models performing well on these tasks could be integrated into applications that may be deployed in practice.
- A.1. Model Configuration: Model evaluations use configurations based on official model repositories where available.Unset values indicate that the models’ default values are used.
- A.1. Model Configuration: The configuration table documents temperature, model-specific parameter support, and Gemini 3 Pro’s thinking-level setting.It notes special handling for GPT-5 mini, GPT-5.2, and Gemini 3 Pro configurations.
A.2. Implementation Details for Model Inference … C.2.3. EXAMPLE INPUTS FOR THE SOCRATIC MODEL BASELINE
The evaluation uses zero-shot multimodal inference with model-specific reasoning settings and structured prompts. Baselines isolate language-only reasoning or decouple visual captioning from textual question answering, using constrained video inputs and outputs.
- A.2. Implementation Details for Model Inference: Models are evaluated zero-shot across all tasks, using official preprocessing code when available and official API services for proprietary and Qwen-series models.Responses are requested in a JSON-like format containing the selected answer and reasoning trace.
- A.2. Implementation Details for Model Inference: Thinking levels use default dynamic mode for Gemini 3 Flash, Gemini 2.5 Pro, and Gemini 2.5 Flash, low for Gemini 3 Pro, and medium for GPT-5.2.Gemini 3 Pro uses low thinking because dynamic reasoning traces exceed the output context window and can produce incomplete or null responses.
- B. Prompts: The system prompt defines the observer as the person wearing the AR glasses and treats camera movement as the observer’s head and body movement.Reasoning is restricted to what is visible or inferable from the egocentric video.
- B. Prompts: The evaluation prompt presents chronologically ordered, uniformly sampled video frames and requires answering multiple-choice questions using only those frames and provided options.The prompt explicitly excludes external knowledge and commonsense reasoning.
- C.1. Blind LLM: The Blind LLM baseline receives only the textual task components, isolating linguistic priors and textual reasoning without visual input.Its purpose is to diagnose language-only shortcuts and estimate performance without perceptual information.
- C.2. Socratic Model: The Socratic baseline decouples perception and reasoning: a video captioning model converts visual input into text, then a language model answers from the caption, question, and options.This baseline uses GPT-5.2 for both stages, samples videos at 2 fps, and stitches frames into 4×4 grids containing 16 frames.
- C.2.1. CAPTION GENERATION PROMPT: Caption generation asks for scene context, motion, spatial relationships, object details, temporal changes, additional observations, and a summary without speculation.The output is paragraph-formatted with the listed fields and limited to 200 words.
- C.2.2. SOCRATIC MODEL EVALUATION PROMPT: Socratic evaluation requires answering from the generated caption alone, using only the provided multiple-choice options and no external knowledge, assumptions, or commonsense.Figure 8 illustrates the baseline input format: 4×4 grids of 16 frames sampled at 2 fps with frame indices shown.
C.3. Human Evaluation … D.2. Meta Information Annotation
The benchmark uses controlled observer trajectories and deterministic or protocol-based annotations to evaluate situated spatial reasoning, alongside a human upper-bound evaluation. Its filming and labeling procedures cover viewpoint orientation, movement paths, memory, affordance, and route-based tasks.
- C.3. Human Evaluation: Two graduate students independently completed all benchmark tasks with unlimited time, full video access, and no external tools or discussion.This establishes a human performance upper bound and is documented through the evaluation interface and performance breakdown.
- D. Video Filming Protocol and Meta Information Annotation: The recording protocol pairs each video with a predefined movement pattern, spatial queries, and deterministic ground-truth answers derived from the recording plan.The protocol targets observer-centric spatial reasoning primitives while keeping trajectory complexity controllable.
- D.1. Video Filming Protocol: Four trajectory categories comprise in-place orientation, Manhattan-style piecewise linear paths, simple geometric trajectories, and extra video collections.These categories organize the benchmark’s controlled motion and supplementary task-specific recordings.
- D.1.1. IN-PLACE ORIENTATION: In-place orientation keeps the camera wearer fixed at a corner, side, or center while changing viewpoint, with one Self-Localization and one Route Shape question per video.Coverage includes different spatial locations whenever possible and excludes translational motion.
- D.1.2. MANHATTAN-STYLE PIECEWISE LINEAR: Manhattan-style trajectories follow predefined paths with two turns and include two Relative Direction, one Reverse Route Plan, and one Route Shape question per video.Egocentric action arrows visualize the observers’ movement trajectories.
- D.1.3. SIMPLE SHAPE TRAJECTORIES: Other simple trajectories use canonical geometric paths and ask two Relative Direction questions plus one additional trajectory question per video.The supplied protocol description truncates the final question type.
- D.1.4. EXTRA VIDEO COLLECTIONS: Extra recordings support Spatial Memory and Spatial Affordance by tracking displaced objects and assessing whether queried actions are physically feasible.These videos extend the predefined trajectory collections with scene-specific observations and physical constraints.
- D.2. Meta Information Annotation: Meta-information labels are deterministic for Self-Localization, Route Shape, Spatial Memory, and Spatial Affordance, while Relative Direction and Reverse Route Plan use independent human annotation.Ground truth derives from predefined viewpoints, trajectories, controlled object changes, or unambiguous physical layouts; agreement is reported for human-annotated labels.
E. Data Analysis … E.3. Key Statistics
The data-analysis section presents SAW-BENCH’s video-duration and question-scene distributions, alongside a summary of its key statistics. These analyses are provided through Figures 10–11 and Table 10.
- E.1. Video Duration Distribution: SAW-BENCH’s data analysis includes a video-duration distribution presented in Figure 10.The passage identifies Figure 10 as the source for this distribution.
- E.1. Video Duration Distribution: Figure 10 is labeled as the video-duration distribution of SAW-BENCH.The figure caption directly names the distribution and benchmark.
- E.2. Question Scene Distribution: SAW-BENCH’s question-scene distribution is presented in Figure 11.The passage identifies Figure 11 as the source for this distribution.
- E.2. Question Scene Distribution: Figure 11 is labeled as the question-scene distribution of SAW-BENCH.The figure caption directly names the distribution and benchmark.
- E.3. Key Statistics: The data analysis summarizes SAW-BENCH’s key statistics in Table 10.The passage identifies Table 10 as the location of the key-statistics summary.
- E.3. Key Statistics: Table 10 is labeled as the key-statistics summary of SAW-BENCH.The table caption directly identifies its contents and benchmark.
F. Sensitivity Analysis … G.1. Self-Localization
The sensitivity analysis shows that temporal context helps longer-horizon spatial integration more than short-horizon tasks, while increasing frame sampling generally provides little benefit. Spatial Memory can even decline at FPS = 3, suggesting redundant or temporally condensed visual information may hinder performance.
- F.1. Sensitivity to Number of Input Frames: F.1. Increasing input frames generally benefits tasks requiring longer-horizon spatial integration, including Spatial Memory and Route Shape.Figure 12 evaluates 8, 16, and 32 input frames across reasoning types.
- F.1. Sensitivity to Number of Input Frames: F.1. Short-horizon tasks such as Self-Localization and Relative D show less benefit from additional temporal context.The figure contrasts these tasks with Spatial Memory and Route Shape.
- F.1. Sensitivity to Number of Input Frames: F.1. Performance varies with temporal context length across open-source multimodal foundation models and reasoning types.Figure 12 plots accuracy against input-frame counts for different models.
- F.2. Sensitivity to Frame Sampling Rate (FPS): F.2. The frame-sampling analysis evaluates Gemini 3 Flash, the top-performing model, under otherwise fixed experimental configurations.The comparison spans multiple reasoning tasks and sampling rates.
- F.2. Sensitivity to Frame Sampling Rate (FPS): F.2. Increasing frame sampling rate across FPS ∈{1, 2, 3, 4, 6} does not substantially improve situated spatial reasoning.The evaluation varies temporal sampling while holding other configurations constant.
- F.2. Sensitivity to Frame Sampling Rate (FPS): F.2. Spatial Memory performance decreases at FPS = 3, potentially because the visual information becomes redundant or temporally condensed.This result is reported while evaluating Gemini 3 Flash across reasoning tasks.
G.1.1. EXAMPLE 36
In this lecture-hall example, the observer is positioned along the side rather than in the center or corner. Model reasoning relies on the aisle, wall, seats, and exit doors visible throughout the video.
- Question: The question asks whether the observer is near the lecture hall’s side, center, or corner.The answer choices are Side, Center, and Corner.
- Model responses: Gemini 3 Flash identifies the position as along the side of the lecture hall.Its reasoning describes movement near the right-side wall and a side perspective across the room, without moving toward the center.
- Model responses: Gemini 3 Pro also concludes that the observer is on the side, based on an aisle beside a wall with rows of seats on the opposite side.The camera passes an exit sign and remains next to the wall rather than entering the center or a corner.
- Model responses: Gemini 2.5 Pro’s trace describes standing in an aisle with seats on one side and a solid wall on the other.The observer pans along the wall toward double doors marked with an EXIT sign, supporting a side position.
G.1.2. EXAMPLE 59 · G.2. Relative Direction
Example 59 asks whether the observer is near the corner, along the side, or near the center of a squash court. Gemini 3 Flash and Gemini 3 Pro select the corner, while Gemini 2.5 Pro selects the center based on differing interpretations of the camera sweep.
- G.1.2. EXAMPLE 59: The question distinguishes among center, side, and corner positions in the lecture hall.The answer choices are A. Center, B. Side, and C. Corner.
- G.1.2. EXAMPLE 59: Gemini 3 Flash selects C, interpreting the visible intersection of two walls and the floor as evidence of a corner position.Its reasoning also notes that the camera remains focused on the court corner during the pan.
- G.1.2. EXAMPLE 59: The example evaluates observer-centric location by requiring inference from viewpoint geometry and camera motion rather than from a directly stated position.The evidence used by the models includes wall intersections, floor perspective, and the extent of the panoramic sweep.
- G.1.2. EXAMPLE 59: Gemini 3 Pro also selects C, citing the front-left wall intersection and the camera’s repeated pivot around that corner.The model treats the convergence of two perpendicular walls as direct evidence of the observer’s location.
- G.1.2. EXAMPLE 59: Gemini 2.5 Pro selects A, reasoning that a panoramic sweep across all four walls indicates a central court position.It argues that a corner or side position would produce a more skewed perspective during the pan.
G.2.1. EXAMPLE 10 … G.3.1. EXAMPLE 151
The examples test observer-centric reasoning about relative start/end locations and movement trajectories in egocentric videos. Model responses generally identify directional displacement and route shape from observed turns and motion segments, but can disagree on the inferred geometry.
- G.2.1. EXAMPLE 10: The first example asks where the observer ends relative to the initial viewing point, with answer choices including front right, front left, front, and same location.The described scene begins on a lawn facing residential buildings and ends near a sports field after forward movement and right turns.
- G.2.1. EXAMPLE 10: Gemini 2.5 Pro and the other detailed responses infer that the final location is front right of the starting point.Gemini 2.5 Pro describes ending on the right side of the lawn after moving from a central starting position; Qwen3-VL 235B and 30B also select front right.
- G.2.2. EXAMPLE 413: The second example asks where the observer was at the beginning relative to the final viewing point, with front right, back right, right, and same location as options.The trajectory proceeds from a covered walkway through a plaza and back along the walkway toward a street.
- G.2.2. EXAMPLE 413: Responses disagree on the second example’s geometry: Gemini 3 Flash selects front right, whereas Gemini 3 Pro and several Qwen models describe the start as behind or back right.The disagreement follows different interpretations of the final orientation and the path’s right turn.
- G.3.1. EXAMPLE 151: The route-shape example asks whether the motion trajectory is U-shaped, triangular, L-shaped, or straight.The video contains forward motion along a sidewalk, a right turn across the parking lot, and another turn toward a building.
G.3.2. EXAMPLE 225 … G.6.2. EXAMPLE 147
The examples test situated awareness through trajectory classification, reverse-route planning, spatial memory, and reachability judgments in egocentric videos. Models often identify piecewise motion and object changes, but route reversal and affordance reasoning can produce conflicting answers.
- G.3.2. EXAMPLE 225: The moving trajectory is classified as Manhattan-style piecewise linear, consisting of straight segments connected by sharp directional turns.Gemini 3 Flash describes repeated forward movements followed by near-right-angle turns in an outdoor plaza.
- G.4.1. EXAMPLE 168: In the reverse-route example, the annotated answer is D: turn around, go straight, turn left, go straight, then turn left and continue straight.The task requires reversing the observed path from the video’s endpoint to its starting point.
- G.4. Reverse Route Plan: The reverse-route tasks require counteracting the original turns in reverse order, linking observer viewpoint, remembered motion, and spatial route planning.Model traces explicitly reconstruct the forward path before applying the inverse sequence of turns and straight movements.
- G.4.2. EXAMPLE 196: For the kitchen reverse-route example, the annotated answer is C: turn around, go straight, turn left, go straight, then turn right and continue straight.The video begins near a stove and ends near a door, requiring route reconstruction across the kitchen and dining area.
- G.5.1. EXAMPLE 53: In the first spatial-memory example, the changing object is the scooter, which is visible beside the recycling station earlier but absent when the camera returns later.The annotated options list scooter, cart, trash bin, and round table.
- G.5.2. EXAMPLE 65: In the second spatial-memory example, the annotated answer is the outdoor coffee table, although several model traces instead identify the throw pillow.The source answer states A, while Gemini 3 Pro, Gemini 3 Flash, and Qwen3-VL 30B select D.
- G.6.1. EXAMPLE 105: The affordance example asks whether the vending-machine buttons can be reached using only the arm without moving the feet, and model traces select B.The described endpoint shows the person stationary with an outstretched hand near the control panel.
- G.6.2. EXAMPLE 147: The sanitizer example asks whether the dispenser can be reached after one small leftward step without leaning, and Gemini 3 Flash selects A.The available trace begins by placing the person a short distance from the wall-mounted dispenser.