SITE: towards Spatial Intelligence Thorough Evaluation

1. Introduction

• SITE unifies spatial reasoning evaluation across natural and synthetic images, multiple views, and videos, addressing gaps in existing benchmarks.Existing efforts often cover only limited modalities or spatial factors, leaving viewpoint transformation and dynamic reasoning underrepresented.Source: §1 · pp. 1–2
• The benchmark draws on three cognitive-science classification systems to cover spatial scales, view-taking, intrinsic and extrinsic structure, and static and dynamic scenes.The taxonomy guides both coverage analysis and task design.Source: §1 · pp. 1–2
• SITE contains 8,068 standardized tasks assembled from 30 existing datasets and one newly annotated dataset.The bottom-up path contributes 6,943 tasks, while the top-down path adds novel view-taking and dynamic-scene tasks.Source: §1 · p. 2
• Evaluation shows that leading VLMs struggle especially with spatial orientation and remain substantially below human performance.The benchmark therefore exposes weaknesses not captured by conventional visual reasoning and question-answering evaluations.Source: §1 · p. 2
• VLM spatial reasoning performance correlates positively with robot manipulation performance, with a reported correlation coefficient of 0.902.This connects SITE scores to an embodied-AI capability within the reported evaluation scope.Source: §1 · p. 2

2. Related Work

• SITE differs by combining natural and synthetic images, multiple views, and videos within a structured spatial-intelligence taxonomy.Its scope is designed to cover multiple spatial aspects and diverse visual contexts rather than one narrow task family.Source: Table 1 · p. 3, §2 · p. 3
• Existing VQA datasets commonly test simple spatial relationships between objects, while 3D and multi-view benchmarks increase complexity but remain largely static.This leaves dynamic spatial intelligence insufficiently covered.Source: §2 · p. 3
• General VLM benchmarks distribute spatial questions among broader perception, comprehension, and reasoning evaluations rather than isolating spatial intelligence systematically.Image and video benchmarks include spatial tasks, but spatial intelligence is not usually their core focus.Source: §2 · p. 3
• Several specialized benchmarks remain constrained by modality or setting: CVBench lacks viewpoint transformation, 3DSRBench is single-image, and VSI-Bench focuses on indoor videos.These constraints limit comprehensive evaluation across viewpoints, environments, and temporal contexts.Source: §2 · p. 3

What Makes Spatial Intelligence (SI)?

• Spatial intelligence lacks a consensus categorization, so SITE incorporates all three major classification systems identified in cognitive science.Using multiple systems supports complementary benchmark design and coverage analysis.Source: §3 · pp. 3–4

Data collection and filtering

• The bottom-up collection pipeline selects image and video benchmarks, filters irrelevant samples with GPT-4o, creates six coarse categories, and uses stratified sampling for balanced coverage.The pipeline operationalizes the taxonomy in dataset construction.Source: Figure 2 · p. 4

Figural, vista, and environmental SI

• The scale taxonomy distinguishes figural, vista, and environmental spatial intelligence according to the size and navigational demands of the scene.Figural objects are small and single-view apprehensible, vista spaces include rooms or town squares, and environmental spaces include cities or neighborhoods.Source: §3 · p. 4

Spatial visualization & orientation

• Spatial visualization concerns mentally manipulating objects independently of viewpoint, whereas spatial orientation concerns imagining appearances from different observer viewpoints.These are treated as fundamental factors in the benchmark’s view-taking coverage.Source: §3 · p. 4

2x2 classification

• The 2x2 taxonomy separates intrinsic from extrinsic information and static from dynamic tasks.Intrinsic information concerns an object’s parts and defining relations; extrinsic information concerns relations among objects or within a broader framework.Source: §3 · p. 4

4. Data

• SITE uses a two-stage construction process that unifies existing spatial-intelligence tasks and adds evaluations for underrepresented aspects.The pipeline first compiles and filters existing benchmarks, then introduces novel tasks based on empirical analysis and cognitive-science classifications.Source: §4 · p. 4

4.1. Data Collection

• The collection starts from 22 image and 8 video datasets, retaining validation and test samples relevant to spatial reasoning.Filtering uses existing category labels when available and language-model classification otherwise.Source: §4.1 · p. 5
• GPT-4o classifies relevant samples into six categories: counting and existence, spatial relationships, localization, 3D information, multiview reasoning, and movement prediction/navigation.The process addresses inconsistent labels across source datasets and supports systematic task organization.Source: §4.1 · p. 5
• The filtered pool contains 223,083 spatial-intelligence examples, but relationship and counting tasks dominate while multiview reasoning is substantially underrepresented.All tasks are standardized into multiple-choice QA, including reformulation of open-ended questions.Source: §4.1 · p. 5

4.2. Novel Proposed Tasks

• Manual analysis finds perspective transformations lacking in existing evaluations, motivating two tasks for extrinsic-static and extrinsic-dynamic reasoning.These tasks target viewpoint changes relevant to scenarios such as navigation and route planning.Source: §4.2 · pp. 5–6
• View Association matches egocentric and exocentric images of the same scene using sampled key-step frames and human-selected distractors.The task works in both directions: exocentric-to-egocentric and egocentric-to-exocentric matching.Source: §4.2 · p. 6
• Frames Reordering requires recovering temporal order across viewpoints by reasoning over motion events between reference frames, with reverse-direction evaluation and ambiguity filtering.The procedure samples four exocentric frames within an egocentric segment, shuffles them, and asks models to restore their sequence.Source: §4.2 · p. 6

4.3. SITE

• The final SITE benchmark contains 8,068 QA pairs—4,260 image-based and 3,808 video-based—covering 30 existing datasets plus Ego-Exo4D.Stratified sampling balances spatial categories while preserving data diversity.Source: §4.3 · p. 6

5.2. Results on SITE

• Humans achieve 100% on view association, whereas GPT-4o and Gemini-1.5-pro average 28.20% and 7.03%, and LLaVA-OneVision-7B and Qwen2.5-VL-7B achieve 15.95% and 0.65%.The result indicates that strong conventional image/video question-answering performance does not transfer to viewpoint-based spatial reasoning.Source: §5.2 · p. 7, Figure 3 · p. 7
• Humans reach 98% and 96% on ego2exo and exo2ego frame reordering, while Qwen2.5-VL-7B achieves the best model average at 5.93%.The authors suggest that video-grounding training may help Qwen2.5-VL-7B model temporal order.Source: §5.2 · pp. 7–8
• GPT-4o underperforms human performance by ∼32% on the full SITE benchmark, while InternVL-2.5-8B and Qwen2.5-VL-7B achieve 32.8% and 31.4% overall accuracy, respectively.The full benchmark comparison is reported in Table 4; open-source models perform competitively with GPT-4o and Gemini-1.5-Pro despite fewer parameters.Source: Table 4 · p. 7, §5.2 · p. 8
• All VLMs obtain less than 10% CAA on the full multi-view split, and most models score below 15% on 3D understanding.The reported failures concern reasoning across viewpoints, depth relationships, and three-dimensional spatial transformations.Source: §5.2 · p. 8
• Larger VLMs generally perform better at spatial reasoning, but 3D understanding remains consistently low across model scales.The authors attribute the persistent 3D challenge to depth reasoning and transformations such as folding a 2D grid into a cube.Source: §5.2 · p. 8

5.3. Spatial Intelligence on Downstream Tasks

• The negated mean L2 distance and SITE CAA have a positive Pearson correlation coefficient of 0.902 on LIBERO-Spatial robotics manipulation.Qwen2.5-VL-7B achieves a 38% success rate while the other evaluated models fail completely in the reported toy experiment.Source: Table 5 · p. 8, §5.3 · p. 8

• SITE combines tasks from 30 existing datasets with two novel task types for view-taking and dynamic scenarios.Source: §6 · p. 8
• Evaluation on SITE demonstrates a huge gap between humans and state-of-the-art vision-language models.Source: §6 · p. 8
• SITE performance shows a positive correlation with performance on a robot manipulation task.Source: §6 · p. 8

• The survey collects 31 computer-vision datasets: 22 image-based, 8 video-based, and 1 newly annotated dataset.Source: §I · p. 13
• Image-based sources include benchmarks for visual reasoning, spatial understanding, multi-image comprehension, 3D reasoning, and multimodal evaluation.Source: §I · p. 13
• Video-based sources cover activity reasoning, long-video understanding, multimodal video comprehension, embodied question answering, temporal reasoning, and indoor spatial reasoning.Source: §I · p. 13
• Ego-Exo4D contributes synchronized egocentric and exocentric video of skilled activities with annotations supporting cross-view translation and related benchmarks.Source: §I · pp. 13–14

• GPT-4o and manual inspection assign six coarse spatial-intelligence categories, while few-shot prompts classify data lacking category labels.Source: §II · p. 14
• The six categories cover counting and existence, spatial relationships, multi-view reasoning, 3D information, object localization, and movement prediction or navigation.Source: §II · p. 14
• Category distributions are shown before balancing and in the final balanced benchmark.Source: Figure 4 · p. 14

Correlation Analysis

• SITE exhibits one of the strongest positive correlations with LIBERO-Spatial robotic-manipulation performance among the compared benchmarks.The analysis uses Pearson correlation coefficients from four vision-language models and compares SITE with general VQA, OCR, scientific-knowledge, and object-probing benchmarks.Source: §III · p. 14, Table 6 · p. 16

• The statistics section provides model-performance comparisons, spatial-category distributions, and examples across the benchmark.Source: Figure 5 · p. 15, Figure 6 · p. 16, §III · p. 17
• The benchmark includes an Ego-Exo frame-reordering task that reconstructs temporal order across egocentric and exocentric views.Source: Figure 8 · p. 17
• Few-shot prompts and dataset-composition visualizations document how spatial categories are classified and distributed.Source: Figure 9 · p. 18, Figure 10 · p. 19, Figure 11 · p. 20