SAW-Bench: Learning Situated Awareness in the Real World

• Situated awareness is foundational to spatial intelligence because agents must continuously track their location, orientation, and movement relative to their bodies while interacting with the physical world.In robotics and AR/VR, failures to maintain this observer-relative translation can decouple system understanding from reality and cause cascading spatial and temporal drift.Source: §1 · p. 2
• Existing multimodal benchmarks predominantly assess observer-independent object relations, mental simulation, and distance estimation from detached third-person perspectives.They largely treat models as passive spectators rather than embodied agents with viewpoint, motion, and position, leaving observer-centric awareness over time unexplored.Source: §1 · pp. 2–3
• SAW-BENCH contains 2,071 human-annotated multiple-choice questions across six awareness tasks and 786 self-recorded egocentric videos from indoor and outdoor environments.The videos were captured using Ray-Ban Meta (Gen 2) glasses, and the questions require reasoning about both the observer and the environment.Source: §1 · p. 3
• 53.89% performance by Gemini 3 Flash on SAW-BENCH trails human performance at 91.55%, with models showing four systematic situated-reasoning failures.The failures involve conflating camera rotation with translation, degrading accuracy as trajectories become more complex, mismatching view-level and persistent world-state memory, and treating environment openness as a sufficient difficulty proxy.Source: §1 · p. 3

General video understanding benchmarks

• General video benchmarks cover long-form, temporal, domain-specific, and comprehensive reasoning, yet remain largely allocentric and leave observer-centric spatial reasoning unexamined.The cited benchmark families include Chandrasegaran et al. (2024), Wu et al. (2024), Chen et al. (2025), Wang et al. (2025), Liu et al. (2024), Shangguan et al. (2025), Xue et al. (2025), He et al. (2025), Zhao et al. (2025), Deng et al. (2025), Ning et al. (2026), Li et al. (2024), and Fu et al. (2025).Source: §2 · p. 3

3D spatial intelligence

• SAW-BENCH defines situations dynamically through observer trajectories and evaluates navigation, spatial memory, and affordance from raw egocentric video.This avoids the costly, difficult-to-scale requirements of reconstructed 3D scenes, 3D capture, and externally mounted cameras.Source: §2 · p. 4
• Unlike SQA3D, which uses static position and orientation in reconstruction-oriented ScanNet videos, SAW-BENCH models situations as ongoing trajectories.SQA3D remains the closest prior work but does not capture context emerging from natural human movement.Source: §2 · pp. 3–4

Visual-spatial intelligence benchmarks

• Visual-spatial benchmarks such as VSI-Bench and related multi-view datasets typically evaluate observer-independent relations or assume the observer’s state is given or static.These benchmarks emphasize object-object interaction, counting, distance estimation, or discrete multi-view inference rather than continuous updates from the observer’s pose and perspective.Source: §2 · p. 4

3.1. Situated Awareness Tasks in SAW-BENCH

• SAW-Bench decomposes situated awareness into six tasks: Self-Localization, Relative Direction, Route Shape, Reverse Route Plan, Spatial Memory, and Spatial Affordance.These tasks cover observer position, viewpoint changes, movement geometry, return planning, temporal spatial changes, and action feasibility.Source: §3.1 · p. 4
• The task distribution is Relative Direction (40.27%), Route Shape (26.36%), Reverse Route Plan (11.06%), Self-Localization (9.66%), Spatial Affordance (7.82%), and Spatial Memory (4.83%).Each task requires reasoning about the relationship between the observer and the surrounding environment.Source: §3.1 · p. 4

3.2. Video Collection

• Egocentric video preserves observer-centric spatial structure by encoding object positions, evolving field of view, and changing environmental visibility during head and body movement.All videos use Ray-Ban Meta (Gen 2) smart glasses, with audio excluded so reasoning relies solely on visual information.Source: §3.2 · p. 4
• The collection spans 10 outdoor and 5 indoor scenes, combining dense coverage for localization and route tasks with broader environmental diversity for memory and affordance evaluation.Participants followed lightweight protocols with predefined reference locations and coarse trajectory shapes while retaining flexibility in execution.Source: §3.2 · pp. 4–5

3.3. Question-Answer Annotation and Quality Check

• QA pairs are tied to predefined recording configurations and known trajectories, while video review, independent double annotation, and final adjudication control recording and labeling quality.This design targets well-defined agent-centric awareness questions while minimizing ambiguity in ground-truth answers.Source: §3.3 · p. 5

4.1. Evaluation Protocol

• The evaluation tests diverse open-source and proprietary MFMs in a zero-shot setting against random, frequent-answer, blind-LLM, Socratic, and human baselines.Accuracy is the primary metric, with answers extracted using a regular-expression parser and GPT-4o-mini fallback when parsing fails.Source: §4.1 · pp. 5–6

4.2. Main Results

• 53.89% overall accuracy for Gemini 3 Flash leaves a 37.66% gap behind human accuracy of 91.55%, demonstrating that situated awareness remains a major challenge.Humans perform especially well on Self-Localization and Route Shape, while the human-model gap is smaller for Spatial Memory and Spatial Affordance.Source: Table 2 · p. 6, §4.2 · p. 6
• Proprietary MFMs generally outperform open-source models, especially on Reverse Route Plan, where sustained egocentric trajectory reasoning is required.Open-source models often rely on first and last frames, whereas proprietary models more consistently maintain an observer-centric representation across the video.Source: §4.2 · p. 6, Figure 4 · p. 7
• 31.34% overall accuracy for both blind and Socratic models shows that captions provide no overall gain over blind language reasoning and remain insufficient for situated awareness.Static captions discard viewpoint changes, orientation, and temporal structure, although they slightly improve Route Shape performance.Source: §4.2 · pp. 6–7

Camera rotation as a source of trajectory errors

• 60.0% of Gemini 3 Flash instances and 53.3% of Qwen3-VL 235B instances misclassify head-rotation paths as zigzag trajectories.Models attribute camera-orientation shifts to physical displacement, revealing difficulty maintaining an observer-centric coordinate system.Source: Figure 5 · p. 7, §5 · p. 7

Trajectory complexity

• Relative Direction accuracy degrades significantly as trajectories become more complex, especially with multiple turns, while human performance remains largely stable.Compounding errors reflect difficulty integrating successive egocentric orientation changes over time.Source: Table 3 · p. 8, §5 · p. 8

Failure to maintain persistent object memory

• Models treat objects outside the camera’s field of view as absent, failing to maintain persistent object identities across egocentric motion.This view-dependent reasoning confuses non-visibility with non-existence and remains an open challenge across models.Source: Figure 6 · p. 8, §5 · p. 8

Effect of openness on situated awareness

• Outdoor performance is often comparable to or higher than indoor performance, showing that environment openness alone is insufficient for predicting spatial reasoning difficulty.Indoor scenes may be harder because higher object density and more intricate layouts increase relational ambiguity.Source: Figure 7 · p. 8, §5 · pp. 8–9

Conclusion

• 53.89% accuracy for Gemini 3 Flash versus 91.55% for humans exposes a substantial human–model performance gap across 23 multimodal foundation models.The benchmark evaluates six facets of observer-centric spatial understanding.Source: §6 · p. 9
• Models conflate egocentric camera rotation with translation, degrade as trajectories become more complex, and fail to preserve object representations beyond the current view.These failures indicate unreliable accumulation and updating of observer-centric spatial state over time.Source: §6 · p. 9
• SAW-BENCH provides a diagnostic testbed for developing systems that reason about their location, motion, and possible actions in dynamic physical environments.The benchmark moves evaluation beyond treating multimodal models as passive observers of visible content.Source: §6 · p. 9

Impact Statement

• SAW-BENCH evaluates observer-centric situated awareness in multimodal foundation models using egocentric videos.It provides a diagnostic tool for measuring spatial understanding capabilities underrepresented in existing evaluation frameworks.Source: Impact Statement · p. 9
• The benchmark targets spatial reasoning needed for reliable robotics, augmented and virtual reality, and assistive technologies.These applications require understanding spatial relationships from an embodied agent’s or human wearer’s perspective for safe and effective operation.Source: Impact Statement · p. 9
• The authors acknowledge that strong benchmark performance could enable downstream misuse in harmful applications.They encourage responsible deployment practices for systems built on or evaluated with SAW-BENCH.Source: Impact Statement · p. 9

A.1. Model Configuration

• The evaluation uses configurations based on official model repositories where available, with unspecified values retaining model defaults.Table 4 documents the evaluated model settings and associated configuration conventions.Source: Table 4 · p. 15
• Gemini 3 Pro uses low thinking level because dynamic reasoning traces exceeded the output context window and produced incomplete or Null responses.This configuration choice was made to avoid failures during evaluation.Source: Table 4 · p. 15
• Sampling differs across models because some do not support fps-based sampling and process 32 frames, while compute-limited models use 8 frames per video.These frame-sampling conditions are explicitly recorded in the evaluation configuration.Source: Table 4 · p. 15

A.2. Implementation Details for Model Inference

• MFMs are evaluated zero-shot using official preprocessing or APIs, structured JSON-like answers with reasoning traces, and GPT-4o-mini fallback extraction when regular parsing fails.Thinking configurations vary by model: Gemini 3 Flash, Gemini 2.5 Pro, and Gemini 2.5 Flash use dynamic mode; Gemini 3 Pro uses low, GPT-5.2 medium, and Qwen3-VL thinking variants medium.Source: §A.2 · p. 15

System Prompt

• The system prompt frames the model as the AR-glasses wearer, treating camera motion as head and body motion and restricting reasoning to visible or inferable egocentric evidence.This establishes the observer-centered perspective and prohibits external knowledge beyond the video.Source: System Prompt · p. 16

B.2. Evaluation Prompt

• The evaluation prompt presents uniformly sampled chronological frames, questions, and answer options, requiring answers based only on the provided frames rather than external knowledge or commonsense.Responses include the selected option and a thinking trace in a JSON-like format, following the evaluation procedure used in TOMATO.Source: Evaluation Prompt · p. 16, §B.2 · p. 16

B.3. Answer Extraction Prompt

• Answer extraction converts model responses into a single multiple-choice letter using the options and index-to-answer mapping, randomly selecting a letter if no option matches.The extraction prompt follows TOMATO and outputs no additional text.Source: Answer Extraction Prompt · p. 17, §B.3 · p. 17

C.1. Blind LLM

• The Blind LLM baseline receives only textual questions, instructions, and options without visual input, isolating linguistic priors and text-based reasoning from perceptual information.It estimates how many MCQs can be answered without watching videos, including through distractor patterns, general knowledge, or straightforward question reasoning, following HourVideo (Chandrasegaran et al., 2024).Source: §C.1 · p. 18

Caption Generation Prompt

• In baseline evaluation, videos are sampled at 2 fps and arranged into 4 × 4 image grids containing 16 frames each; multiple grids are provided to generate textual descriptions.The captioning and evaluation prompts follow those used in HourVideo (Chandrasegaran et al., 2024).Source: §C.2 · p. 19

Socratic Model Evaluation Prompt

• Socratic evaluation supplies the generated caption with the question and options, requiring answers and reasoning based solely on explicitly stated caption content.An example input shows 4×4 grids containing 16 frames sampled at 2 fps with frame indices, as illustrated in Figure 8.Source: Socratic Model Evaluation Prompt · p. 21, §C.2.3 · p. 22, Figure 8 · p. 22

C.3. Human Evaluation

• Human performance is estimated by two graduate annotators completing every task independently with unlimited time, video access, and no external tools or discussion.Individual annotator performance is reported in Table 5, and the evaluation interface is shown in Figure 9.Source: §C.3 · p. 22, Table 5 · p. 22, Figure 9 · p. 23

D.1. Video Filming Protocol

• The filming protocol assigns each video a predefined movement pattern, spatial queries, and deterministic ground-truth answers to control trajectory complexity and coverage.The protocol uses four trajectory categories: in-place orientation, Manhattan-style piecewise linear motion, simple geometric trajectories, and extra collections for memory and affordance tasks.Source: §D.1 · p. 24

D.1.1. IN-PLACE ORIENTATION

• In-place orientation keeps the camera wearer fixed at a corner, side, or center while changing viewpoint orientation, pairing each video with Self-Localization and Route Shape questions.The protocol seeks coverage across positional contexts without translational motion.Source: Table 6 · p. 24

D.1.2. MANHATTAN-STYLE PIECEWISE LINEAR

• Manhattan-style trajectories use predefined paths with two turns and evaluate Relative Direction, Reverse Route Plan, and Route Shape through egocentric action trajectories.Each video contains two Relative Direction questions, one Reverse Route Plan question, and one Route Shape question.Source: Table 7 · p. 25

D.1.3. SIMPLE SHAPE TRAJECTORIES

• Simple geometric trajectories evaluate observer motion along canonical paths using two Relative Direction questions and one additional trajectory-based query.The supplied protocol table identifies this category as other simple trajectories.Source: Table 8 · p. 26

D.1.4. EXTRA VIDEO COLLECTIONS

• Extra scene recordings support Spatial Memory by displacing objects and Spatial Affordance by testing action feasibility under physical and spatial constraints.These recordings extend the predefined observer trajectories for the two tasks.Source: §D.1.4 · p. 26

D.2. Meta Information Annotation

• Ground truth is derived from filming plans and viewpoint sequences: Self-Localization and Route Shape are deterministic, while Relative Direction and Reverse Route Plan receive independent dual annotation with disagreement resolution and Cohen’s κ reporting.Spatial Memory and Spatial Affordance labels are controlled during filming and require no additional human annotation.Source: §D.2 · p. 27

E.1. Video Duration Distribution

• The accompanying description identifies Figure 10 as the source for SAW-Bench’s video-duration analysis.Source: §E.1 · p. 28, Figure 10 · p. 28
• SAW-Bench’s video-duration distribution is presented in Figure 10.Source: Figure 10 · p. 28, §E.1 · p. 28

E.2. Question Scene Distribution

• The accompanying description identifies Figure 11 as the source for SAW-Bench’s question-scene analysis.Source: §E.2 · p. 29, Figure 11 · p. 29
• SAW-Bench’s question-scene distribution is presented in Figure 11.Source: Figure 11 · p. 29, §E.2 · p. 29

E.3. Key Statistics

• The accompanying description directs readers to Table 10 for the benchmark’s key-statistics summary.Source: §E.3 · p. 29, Table 10 · p. 29
• Key statistics for SAW-Bench are summarized in Table 10.Source: Table 10 · p. 29, §E.3 · p. 29

F.1. Sensitivity to Number of Input Frames

• Long-horizon tasks such as Spatial Memory and Route Shape generally benefit from more input frames, whereas Self-Localization and Relative Direction saturate earlier.Across all tasks, additional frames produce only marginal gains beyond these trends.Source: Figure 12 · p. 30

F.2. Sensitivity to Frame Sampling Rate (FPS)

• Increasing the frame sampling rate does not substantially improve situated spatial reasoning for Gemini 3 Flash.The evaluation varies sampling rates from 1 to 6 FPS while holding other configurations constant.Source: §F.2 · p. 31, Table 11 · p. 31
• Spatial Memory performance decreases at FPS = 3, potentially because of redundant or temporally condensed visual information.This task-specific decline contrasts with the generally limited gains from higher sampling rates.Source: §F.2 · p. 31
• Reverse Route Plan performs worst at FPS = 1, suggesting that this reasoning type relies more heavily on temporal visual context.Source: §F.2 · p. 31

G.1.1. EXAMPLE 36

• The observer is positioned along the side aisle of the lecture hall, with rows of seats on one side and a wall or exit doors on the other.Gemini 3 Flash, Gemini 3 Pro, and Gemini 2.5 Pro all support the side-position answer from the camera’s movement and surrounding geometry.Source: §G.1.1 · pp. 32–33

G.1.2. EXAMPLE 59

• The question distinguishes three observer positions: center, side, and corner.The provided choices are A. Center, B. Side, and C. Corner.Source: §G.1.2 · p. 34
• Gemini 3 Flash identifies the squash-court setting through the court walls, floor markings, glass walls, and people visible outside.It treats the intersection of two walls and the floor as the primary positional cue.Source: §G.1.2 · p. 34
• Gemini 3 Flash also answers C, using the persistent view of two walls meeting and the floor to infer a corner position.The response contrasts this geometry with being in the center or merely along a side.Source: §G.1.2 · p. 34
• Gemini 3 Pro answers C, concluding that the camera remains anchored near the squash court’s front-left corner throughout the video.Its reasoning emphasizes repeated views of two intersecting walls, proximity to the walls, and a return to the same corner.Source: §G.1.2 · p. 34
• Gemini 3 Pro’s trajectory analysis describes an initial corner view, a rightward pan, a close approach to a wall, and a final return toward the corner.This sequence supports its conclusion that the perspective is anchored near the intersection of the front and side walls.Source: §G.1.2 · p. 34
• Gemini 2.5 Pro instead answers A, interpreting the panoramic sweep across all four walls as evidence that the observer stands near the court’s center.It argues that a corner or side position would produce a more skewed perspective during the pan.Source: §G.1.2 · p. 35

G.2.1. EXAMPLE 10

• In Example 10, Gemini 3 Flash and Qwen3-VL 235B answer that the final location is front right of the starting point.Source: §G.2.1 · pp. 36–37

G.3. Route Shape

• Across these examples, model explanations use landmarks, turns, and walking directions to reconstruct observer-relative position rather than relying only on the final visual scene.The traces reference buildings, plazas, sports fields, walkways, parking lots, and vehicle or fence landmarks to infer movement.Source: §G.2.1 · p. 36, §G.2.2 · p. 38, §G.3.1 · p. 40
• The examples expose ambiguity in situated reasoning when camera orientation, head panning, and physical translation must be disentangled to determine location or path shape.Some responses explicitly separate head turning from movement, while others infer displacement or unchanged position from changing views.Source: §G.2.1 · p. 36, §G.2.2 · p. 39

G.2.2. EXAMPLE 413

• In Example 413, Gemini 3 Flash identifies the starting location as front right relative to the final viewpoint, whereas Gemini 3 Pro and Gemini 2.5 Pro select back right.The disagreement reflects different interpretations of the walk through the plaza, right turn, and continued movement along the covered walkway.Source: §G.2.2 · pp. 38–39
• Qwen3-VL 235B and Qwen3-VL 30B also answer back right in Example 413, while Qwen2.5-VL 72B incorrectly treats the starting point as unchanged.The responses differ in whether the final viewpoint is interpreted as substantially displaced from the initial covered area.Source: §G.2.2 · p. 39

G.3.1. EXAMPLE 151

• In Example 151, Gemini 3 Flash and Gemini 2.5 Pro infer a U-shaped trajectory from three straight segments connected by two turns.Both analyses describe movement along one side, across the parking lot, and along a parallel return segment.Source: §G.3.1 · pp. 40–41
• Gemini 3 Pro and Qwen3-VL 30B instead classify Example 151 as L-shaped, based on one right turn followed by a perpendicular straight segment.Their reasoning emphasizes two primary path segments rather than the full sequence of turns and segments.Source: §G.3.1 · pp. 40–41

G.3.2. EXAMPLE 225

• G.3.2 identifies the trajectory as Manhattan-style piecewise linear, with straight segments connected by sharp turns.Gemini 3 Flash and Gemini 3 Pro support option A, whereas Gemini 2.5 Pro selects in-place orientation.Source: §G.3.2 · p. 42

G.4.1. EXAMPLE 168

• G.4.1’s annotated return route is option D: turn around, go straight, turn left, go straight, then turn left and continue straight.Model traces disagree, selecting A, B, or D while reversing the observed soccer-field path.Source: §G.4.1 · pp. 44–45

G.4.2. EXAMPLE 196

• G.4.2’s annotated return route is option C: turn around, go straight, turn left, go straight, then turn right and continue straight.Several model traces instead select D, while others select A or B, indicating disagreement over the kitchen-to-door reversal.Source: §G.4.2 · pp. 47–48

G.5.1. EXAMPLE 53

• G.5.1 identifies the scooter as the object that changes, disappearing between earlier and later views of the recycling station.The blue carts, bins, and round tables remain consistent with the room’s layout.Source: §G.5.1 · p. 49
• In EXAMPLE 53, the scooter is the object that changes, disappearing after being visible beside the recycling bin while the other objects remain fixed.The scene description identifies the scooter as present mid-video and absent at the end, while the recycling bin, carts, trash bin, and round tables retain their positions.Source: §G.5.1 · p. 50

G.5.2. EXAMPLE 65

• In EXAMPLE 65, the annotated answer is the outdoor coffee table, but multiple model analyses instead identify the throw pillow as disappearing between earlier and later views.The supplied model traces state that the coffee table remains unchanged while the patterned throw pillow is later absent from the sofa.Source: §G.5.2 · pp. 51–52

G.6.1. EXAMPLE 105

• In EXAMPLE 105, the answer is that the vending-machine buttons can be reached using only the arm after walking to the machine and stopping.The model analyses describe the person standing directly in front of the machine with an outstretched hand near the control panel, without taking another step.Source: §G.6.1 · p. 53

G.6.2. EXAMPLE 147

• In EXAMPLE 147, the answer is yes: one small step leftward places the hand sanitizer dispenser within arm’s reach without leaning or using support.The analyses describe the dispenser as close to the person and the later hand position as consistent with reaching it after the small lateral movement.Source: §G.6.2 · pp. 55–56