Source-linked AI summary
GST-Bench: Can VLMs Develop Global Spatial Awareness from Video?
Qifeng Zhang, Kaixiang Huang, Heng Dong, Huang Fang, Junting Chen, Junjie Zhu, Yonghang Chen, Zhiyu Zhang, Wei Li
TL;DR
Existing benchmarks under-test global spatial awareness over long-horizon video. GST-Bench evaluates this ability, finding a substantial model–human gap: Gemini-3-Pro scores 42.68 versus humans’ 79.08.
Problem
Existing benchmarks emphasize local spatial perception from limited viewpoints, leaving global spatial awareness over continuous, long-horizon visual streams underexplored.
Method
GST-Bench evaluates global spatial reasoning through long-horizon egocentric videos, novel viewpoints, precise spatial questions, and top-down scene representations.
Results
Gemini-3-Pro scores 42.68 versus humans’ 79.08, while controlled local variants show that proprietary-model failures largely involve cross-frame spatial integration.
Takeaways & Limitations
The benchmark establishes global spatial reasoning as a major limitation for current VLMs, while targeted GST-Train supervision can partially narrow the gap.
Takeaways & Limitations
GST-Bench assumes the task cannot be solved from local perception alone.
Abstract
from arXiv · showhide
Spatial intelligence is fundamental to embodied agents, yet existing benchmarks focus on local spatial perception from single or few viewpoints, overlooking global spatial awareness over continuous, long-horizon visual streams. To address this limitation, we introduce the Global-Spatial-Temporal Benchmark (GST-Bench), a VQA benchmark for global spatial intelligence in video understanding, comprising human-verified questions derived from 6,790 minutes of synthetically generated video. It requires models to perform accurate spatial inference from novel viewpoints unseen in the input video and to map egocentric observations onto global top-down images. A comprehensive evaluation of 22 state-of-the-art VLMs exposes a striking gap between models and humans: the strongest zero-shot model attains only 42.68, far below the human score of 79.08. To probe the cause of this gap, we construct GST-Bench-Local and find that models, despite strong local spatial understanding under the same task formulation, still fail to consolidate long-horizon observations into a globally consistent scene representation. We further provide GST-Train, a dataset for global spatial reasoning, as a complementary resource to facilitate future research on this challenge.
1 Introduction
GST-Bench targets global spatial awareness in VLMs, requiring long-horizon integration of egocentric observations, novel-viewpoint reasoning, and correspondence with top-down scene representations. Across 22 VLMs, models substantially trail humans, while local variants diagnose cross-frame integration as a key bottleneck and GST-Train improves performance through targeted supervision.
- Motivation: Global spatial intelligence is necessary for embodied agents to track egocentric position, remember unseen objects, and maintain a globally consistent scene representation over time.The motivating household-robot scenario requires incrementally building a scene representation from exploration video and inferring relative object locations beyond the current field of view.
- Benchmark design: GST-Bench addresses benchmark gaps by excluding single-frame-solvable questions, hiding target objects from query viewpoints, using precise numerical metrics, and evaluating global-local correspondence through top-down images.Existing benchmarks typically use one or a few views and emphasize local properties such as object size or relative depth.
- Benchmark construction: The benchmark organizes global embodied spatial intelligence into self localization, object localization, and scene structure understanding across 12 subtasks, with 2,762 human-verified questions and 6,790 minutes of video input.Its automatic construction pipeline uses OmniGibson with BEHAVIOR-1K across 50 indoor scenes and records object and viewpoint coordinates for top-down projections.
- Diagnostic analysis: Controlled local variants show that proprietary models gain up to 39 points when targets are visible, indicating failures in cross-frame spatial integration, whereas open-source models struggle with both local perception and global integration.The local variants make the target visible from the query viewpoint so models can reason from a single image rather than the full video.
- Mitigation: A 27.63-point improvement raises Qwen3-VL-8B from 25.89 to 53.52 after fine-tuning on GST-Train, surpassing all proprietary models evaluated zero-shot.GST-Train is built for global spatial reasoning from BEHAVIOR-1K, ArtVIP, HyperSim, and internal simulation scenes.
- Evaluation: 42.68 versus 79.08: Gemini-3-Pro, the strongest zero-shot model, remains far below humans on GST-Bench, while most open-source models score around 20–30 points and often near random guessing.The evaluation covers 22 state-of-the-art VLMs, including open-source, proprietary, and embodied-understanding models.
2 Related Work
Prior benchmarks progress from single-view categorical spatial judgments to multi-view and video-based scene reasoning, but existing evaluations still inadequately isolate spatial understanding that requires integrating observations across frames.
- Spatial Understanding from Single or Few Views: Spatial benchmarks evolved from preposition and relation judgments to 2D relation/counting, 3D depth and distance, and more 3D-aware reasoning [10] [16] [25].
- Spatial Understanding from Single or Few Views: Multi-view benchmarks add complementary images to evaluate spatial relationships, metric estimation, 3D grounding, relative depth, and cross-view reasoning [4] [31] [6].
- Spatial Understanding from Video: Video benchmarks extend spatial reasoning to room-scale and cross-room structure by requiring models to integrate observations over time, including configurational, metric, spatiotemporal, recall, and continual-counting tasks [28].
- Spatial Understanding from Video: A key limitation is that many evaluations do not distinguish questions answerable from one informative frame from those requiring cross-frame integration, weakening targeted assessment of temporal spatial reasoning.Examples include object-size estimation in VSI-Bench and existence queries in OST-Bench, which may be solvable from a single video frame.
3 GST-Bench
GST-Bench evaluates whether VLMs can construct globally consistent spatial representations from long-horizon egocentric video by integrating observations across time, novel viewpoints, and top-down scene representations. It organizes twelve tasks around self localization, object localization, and scene structure understanding, with inputs designed to prevent solutions based on local perception alone.
- Benchmark scope: GST-Bench tests global spatial awareness by requiring models to integrate long-horizon egocentric observations, reason from unseen viewpoints, and align them with top-down scene representations.The benchmark targets globally consistent spatial representations rather than perception from single images or a few views.
- Visual inputs: The benchmark combines exploration videos, object-annotated videos, short trajectory clips, current views, and top-down images at easy, medium, and hard abstraction levels.Top-down representations range from photo-realistic renderings to occupancy-style maps and bare floor plans, controlling the available global visual information.
- Object localization: Object-localization tasks require targets to be absent from the novel current view, ensuring they cannot be solved through local perception alone.Targets are specified either semantically by category name or visually through bounding-box annotations in an object-annotated video.
- Task organization: Twelve task types cover self localization, object localization, and scene structure understanding.Self localization aligns a novel egocentric view with a global map; object localization infers unseen target locations; scene structure understanding organizes observations into a global environment representation.
- Data generation: The automatic pipeline enforces global reasoning by sampling query viewpoints outside exploration trajectories and uses simulation geometry for precise, scalable answer generation.GST-Bench scenes are held out from GST-Train, while multiple perturbed exploration trajectories broaden scene coverage and reduce reliance on a canonical route.
4 Evaluation on GST
GST-Bench evaluates 22 VLMs across 12 subtasks measuring global spatial competencies, revealing a substantial gap between current models and humans. Targeted GST-Train fine-tuning substantially improves performance, but does not close the human gap.
- Evaluation Protocol: GST-Bench averages the four task-format metrics across 12 subtasks: distance uses MRA, angle and point prediction use thresholded accuracy, and multiple choice uses standard accuracy.The benchmark covers distance prediction, angle prediction, point prediction, and multiple-choice questions.
- GST-Train Fine-Tuning: 53.52 average score elevates fine-tuned Qwen3-VL-8B from 25.89 and surpasses every evaluated zero-shot model, including proprietary systems.The improvement demonstrates that explicit global-spatial supervision can enhance long-horizon spatial memory and cross-view alignment, while the remaining human gap persists.
- Zero-Shot Results: 79.08 human performance versus Gemini-3-Pro’s 42.68 leaves a 36.4-point gap, especially on orientation and global-position estimation.Orientation scores are 21.52 versus 85.00, while global-position scores are 42.23 versus 93.00; egocentric distance is a notable exception with a low human baseline.
- Zero-Shot Results: Gemini-3-Pro (42.68), Gemini-2.5-Pro (40.95), and GPT-5 (40.85) form the top tier, while proprietary models achieve the best score on every subtask.These models roughly double random guessing but remain below 55% of human performance.
- Zero-Shot Results: Open-source models remain near random: Qwen3-VL-32B (30.43) and InternVL3.5-38B (30.71) lead their tier, while nine of seventeen models are within three points of random.Four models score below random, indicating little meaningful global spatial competence beyond coarse scene identification.
- Zero-Shot Results: Embodied-tuned models do not improve global reasoning: RoboBrain2.5-8B (24.61) and Cosmos-Reason2-8B (21.64) trail Qwen3-VL-8B (25.89), while Robix-32B (29.26) is comparable to Qwen3-VL-32B (30.43).The results suggest current embodied post-training emphasizes local affordances and relations rather than long-horizon memory and cross-view alignment.
5 Conclusion
GST-Bench introduces a video-based benchmark for evaluating global spatial awareness in VLMs through long-horizon egocentric videos, novel off-trajectory viewpoints, and explicit top-down scene representations. Evaluation of 22 state-of-the-art VLMs reveals a substantial gap from humans, showing that global spatial reasoning remains a major limitation.
- GST-Bench introduces a video-based benchmark for evaluating global spatial awareness in VLMs.
- The benchmark requires reasoning across long-horizon egocentric videos, novel off-trajectory viewpoints, and explicit top-down scene representations.These requirements target spatial abilities essential for embodied agents but underexplored in existing benchmarks.
- Evaluation of 22 state-of-the-art VLMs reveals a substantial gap between current models and humans, identifying global spatial reasoning as a major limitation.The limitation persists even for the strongest proprietary and embodied-understanding models.