PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces

Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, Saining Xie

arXiv:2412.14171v2cs.CV

TL;DR

The paper investigates whether MLLMs can think spatially from videos, addressing limited evidence about their visual-spatial intelligence. It introduces and evaluates VSI-Bench, then probes linguistic explanations and cognitive maps. MLLMs show competitive but subhuman performance, with spatial reasoning as a primary bottleneck; linguistic prompting fails to help, while cognitive maps improve spatial distance reasoning.

  • Problem

    The paper asks whether MLLMs trained on video can remember and reason about spaces with visual-spatial intelligence comparable to humans.

  • Method

    The paper introduces VSI-Bench, a benchmark of over 5,000 question-answer pairs from 288 real indoor-scene videos, and probes models with self-explanations and cognitive maps.

  • Results

    MLLMs achieve competitive but subhuman visual-spatial performance, with spatial reasoning as the primary bottleneck; linguistic reasoning techniques degrade performance, while cognitive maps improve spatial distance reasoning.

  • Takeaways & Limitations

    MLLMs develop local spatial awareness, but explicit visual cognitive maps are more useful than prevailing linguistic reasoning prompts for spatial distance tasks.

  • Takeaways & Limitations

    Forming accurate global space representations from discrete video frames remains difficult for MLLMs.

Abstract

from arXiv · show

Humans possess the visual-spatial intelligence to remember spaces from sequential visual observations. However, can Multimodal Large Language Models (MLLMs) trained on million-scale video datasets also ``think in space'' from videos? We present a novel video-based visual-spatial intelligence benchmark (VSI-Bench) of over 5,000 question-answer pairs, and find that MLLMs exhibit competitive - though subhuman - visual-spatial intelligence. We probe models to express how they think in space both linguistically and visually and find that while spatial reasoning capabilities remain the primary bottleneck for MLLMs to reach higher benchmark performance, local world models and spatial awareness do emerge within these models. Notably, prevailing linguistic reasoning techniques (e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve performance, whereas explicitly generating cognitive maps during question-answering enhances MLLMs' spatial distance ability.

1. Introduction

The paper asks whether MLLMs can develop visual-spatial intelligence from video, a capability humans use to remember and mentally reconstruct spaces. It introduces VSI-Bench and probes linguistic and visual expressions of spatial reasoning to identify both bottlenecks and emerging abilities.

  • Visual-spatial intelligence involves perceiving and mentally manipulating spatial relationships, including relational reasoning and perspective transformation.
  • VSI-Bench evaluates MLLMs on visual-spatial intelligence using more than 5,000 question-answer pairs from nearly 290 real indoor-scene videos.
  • The analysis prompts selected models to produce linguistic self-explanations and visual cognitive maps, treating these modalities as distinct yet complementary.
  • Spatial reasoning is identified as the main factor behind weak benchmark performance, while MLLMs show strong local spatial models but weak global ones.
  • Standard linguistic reasoning techniques do not improve benchmark performance, whereas explicitly generating cognitive maps improves spatial distance question-answering.

2. Visual-Spatial Intelligence

The paper defines visual-spatial intelligence for video-based real-world environments and organizes it around perceptual, linguistic, temporal, and spatial capabilities. It emphasizes relational reasoning and transforming egocentric observations into allocentric spatial representations.

  • The paper scopes visual-spatial intelligence to video-based understanding of real-world environments such as homes, offices, and factories.
  • VSI-Bench requires visual perception, linguistic intelligence, temporal processing, and varying degrees of spatial reasoning across its tasks.
  • Relational reasoning identifies object relationships through distance and direction, using visuospatial common sense about object sizes.
  • Egocentric-allocentric transformation maps self-centered video frames to environment-centered object positions and camera trajectories.
  • Different tasks weight capabilities differently; route planning relies more on perspective transformation than object size estimation.

3. VSI-Bench

VSI-Bench is a video benchmark designed to measure MLLM visual-spatial intelligence across diverse indoor environments. Its construction standardizes source datasets, generates varied tasks, and applies human review to improve question-answer quality.

  • 3.1. Overview: VSI-Bench contains over 5,000 question-answer pairs derived from 288 real videos spanning residential, professional, and industrial environments.
  • 3.1. Overview: The benchmark includes eight tasks grouped into configurational, measurement estimation, and spatiotemporal categories.
  • 3.1. Overview: Configurational tasks test spatial arrangement, measurement tasks estimate object, room, or absolute distance size, and spatiotemporal tasks test memory of video-observed spaces.
  • 3.2. Benchmark Construction: The construction pipeline standardizes diverse datasets into unified metadata and semantic spaces before generating question-answer pairs through human annotation and templates.
  • 3.2. Benchmark Construction: Human verification is applied throughout construction to filter low-quality videos, annotations, ambiguous questions, and erroneous data.

4. Evaluation on VSI-Bench

The evaluation compares 15 video-supporting MLLMs with chance baselines and human performance on VSI-Bench. Humans lead overall, while Gemini-1.5 Pro and top open-source models achieve competitive but subhuman results.

  • Evaluation Setup: 15 video-supporting MLLMs are evaluated across model families, parameter scales, and training recipes under zero-shot settings.The evaluation includes Gemini-1.5, GPT-4o, InternVL2, ViLA, LongViLA, LongVA, LLaVA-OneVision, and LLaVA-Video models.
  • Evaluation Setup: The evaluation uses random and frequency chance baselines, plus a 400-question VSI-Bench (tiny) subset for human comparison.The frequency baseline captures gains from long-tailed answers or imbalanced multiple-choice distributions.
  • Main Results: 79% average human accuracy exceeds the best model by 33%, with humans reaching 94%–100% on configuration and spatiotemporal tasks.The human-model gap is narrower on measurement tasks requiring absolute distance or size estimation.
  • Main Results: Gemini-1.5 Pro delivers competitive results, substantially exceeds chance baselines, and approaches human performance on absolute distance and room size estimation.These results remain below overall human performance despite the model’s competitive task-level results.
  • Main Results: LLaVA-Video-72B and LLaVA-OneVision-72B trail Gemini-1.5 Pro by only 4%–5%, while 7 of 12 open-source models fall below chance level.Top-tier open-source models therefore perform competitively with closed-source models, but most open-source models show substantial limitations.

5. How MLLMs Think in Space Linguistically

Self-explanations and error analysis identify spatial reasoning as the main weakness in MLLMs’ visual-spatial performance. Linguistic prompting degrades VSI-Bench results, whereas the analyses reveal partial spatial understanding and local cognitive maps.

  • Probing via Self-Explanations: Self-explanations are collected from a randomly sampled subset of 163 incorrect answers and reviewed manually to analyze model behavior.The analysis probes linguistic reasoning alongside visual cognitive maps.
  • Probing via Self-Explanations: MLLMs show accurate timestamped video descriptions and step-by-step reasoning, but incorrect cases expose failures in perspective transformation and relational reasoning.One error follows the camera’s egocentric view instead of constructing the allocentric route from door to bed.
  • Error Analysis: Around 71% of errors are attributed to spatial reasoning, making it the primary bottleneck for MLLM performance on VSI-Bench.The error taxonomy separates visual perception, linguistic intelligence, relational reasoning, and egocentric-allocentric transformation errors.
  • Limits of CoT Methods in Visuospatial Tasks: CoT, self-consistency, and Tree-of-Thoughts all degrade average VSI-Bench performance rather than improving spatial reasoning.Zero-Shot CoT and ToT reduce average performance by about 4%, while self-consistency remains 1.1% below the no-prompting baseline.
  • Limits of CoT Methods in Visuospatial Tasks: CoT improves VideoMME by 1.6% but harms VSI-Bench tasks, including 8%–21% decreases on room-size and object-size estimation.The contrast suggests that linguistic prompting can help general video understanding without addressing visual-spatial reasoning.
  • Visual Cognitive Maps: Cognitive-map visualizations are used to examine MLLMs’ implicit spatial world models, whose map-distance accuracy decreases as object distance increases.The cognitive maps provide a visual probe of spatial awareness and local model representations.

6. How MLLMs Think in Space Visually

MLLMs form local rather than unified global spatial representations from videos. Explicit cognitive-map generation improves relative-distance reasoning, but imperfect maps remain a limitation.

  • 6.1. Probing via Cognitive Maps: 64% accuracy for adjacent-object positioning indicates robust local spatial awareness in Gemini-1.5 Pro’s cognitive maps.Accuracy deteriorates as object-pair distance increases.
  • 6.1. Probing via Cognitive Maps: MLLMs form a series of local world models rather than one unified global model when remembering spaces.Forming a global representation from discrete video frames is inherently difficult for MLLMs.
  • 6.2. Cognitive Maps for Spatial Reasoning: Cognitive-map prompting tests whether local distance awareness transfers to relative-distance recall and reasoning.The experiment prompts the model to generate a map before answering the question.
  • 6.2. Cognitive Maps for Spatial Reasoning: 46.0 relative-distance accuracy without a cognitive map rises to 56.0 with a predicted map and 66.0 with a ground-truth map.The results show a 10-point gain from using the predicted cognitive map and a larger gain with an accurate map.

7. Related Works

Prior work studies MLLM visual understanding, spatial awareness, and video capabilities, often using images or language. This paper evaluates visual-spatial intelligence directly from real-world videos.

  • MLLMs with Visual-Spatial Awareness: MLLMs combine language-model reasoning with vision-encoder feature extraction and are viewed as promising foundations for world models and embodied agents.The related work emphasizes their visual understanding capabilities.
  • MLLMs with Visual-Spatial Awareness: Earlier studies primarily assess spatial information through 2D images or language, whereas this work uses real-world videos.The video setting more closely mirrors human world understanding and embodied-agent applications.
  • Benchmarking MLLMs on Video: Video benchmarks increasingly evaluate MLLMs on recognition, perception, and broader video-understanding abilities.The cited examples include Video-MME, EgoSchema, and OpenEQA.

8. Discussion and Future Work

The paper frames VSI-Bench as a basis for diagnosing MLLM spatial strengths and bottlenecks, then points toward spatially specialized training and prompting. Its supplementary materials document benchmark construction and analysis procedures.

  • Discussion: VSI-Bench analysis identifies perceptual, temporal, and linguistic strengths alongside egocentric-allocentric transformation and relational-reasoning bottlenecks.The paper reports that linguistic prompting methods fail to improve spatial reasoning, while explicit cognitive maps enhance spatial-distance reasoning.
  • Future Work: Future improvement avenues include task-specific fine-tuning, self-supervised objectives for spatial reasoning, and visuospatial-tailored prompting.These are presented as future directions rather than demonstrated results.
  • Supplementary Scope: The supplementary materials cover benchmark construction, evaluation setup, full sub-experiment results, input sequencing, repetition, and additional visualizations.They also describe analysis of self-explanations, Chain-of-Thought methods, and cognitive maps.
  • Benchmark Construction: The benchmark unifies indoor-scene data from ARKitScenes, ScanNet++, and ScanNet into standardized scene metadata and video formats.The pipeline includes 150 ARKitScenes, 50 ScanNet++, and 88 ScanNet samples, standardized at 640 × 480 resolution.
  • QA-Pair Generation: VSI-Bench QA pairs encode task type, video, question, answer options, and verbal or numerical ground truth across eight tasks.Multiple-choice tasks use accuracy, while numerical-answer tasks use mean relative accuracy.
  • Quality Review: Human review filters incomplete reconstructions, verifies scene metadata, and flags ambiguous or erroneous QA pairs for correction or removal.Review focuses especially on object-count correctness and question-answer validity.

B.3. Implementation Details of CoT Methods

The implementation details describe linguistic prompting, cognitive-map generation, distance-locality evaluation, and evaluation protocols. Additional analyses test model capacity, human comparison, frame sampling, and sampled-frame sensitivity.

  • CoT Prompting: Chain-of-Thought prompting appends “Let’s think step by step.” to each benchmark question to elicit step-by-step reasoning.The supplied passage introduces the implementation of the evaluated linguistic prompting methods.
  • Cognitive Map Generation: Cognitive maps are generated by predicting object centers on a 10x10 grid, with outputs listing coordinates for each object category.The prompt requires preserving relative spatial relationships and including all instances of requested categories.
  • Distance Locality: Inter-category distances use the shortest object-pair distance for multi-instance categories and count as correct within one grid unit of ground truth.Distances are grouped into eight bins for average-accuracy analysis.
  • Additional MLLM Results: LLaVA-Video-72B gains 8% on the cognitive-map relative-distance experiment, whereas LLaVA-Video-7B declines, likely because limited capacity impairs map prediction.The comparison is presented as validating stronger local than global cognitive-map accuracy.
  • Human Evaluation: Human evaluators receive videos and questions simultaneously, can review videos repeatedly, and answer with unlimited time in the human-level evaluation.A 560-sample subset differs from the full set by 0.5% on average and at most 2.9%.
  • Video Evaluation Setup: Open-source models and GPT-4 use evenly spaced frame sampling, while Gemini receives the entire video; frame counts vary by model.The paper treats frame sampling as a model design choice separate from benchmark design.

C.4. More Evaluation Results

Additional evaluations show that visual input generally benefits VSI-Bench performance, while several spatial tasks remain difficult and blind models can fall below chance. Results also expose task-specific biases and consistent benchmark behavior on smaller subsets.

  • Blind models perform below chance overall, while absolute distance, route plan, and relative direction remain difficult with or without vision.
  • Object-size performance exceeds chance even without vision, likely because language-model training provides relevant common-sense knowledge.
  • Larger variants often perform better in blind evaluations, but task-specific biases can inflate results, such as LongVILA-8B predicting 2 for object count and reaching 47.5% accuracy.
  • The Socratic GPT-4o variant trails standard GPT-4o by 4.7%, while VSI-Bench (tiny) preserves the rankings and average-accuracy pattern of the full benchmark.
  • Video improves performance across almost all MLLMs, especially on object count, room size, relative distance, and appearance order.

D. Input Sequencing and Repetition Analysis

Input ordering and repetition affect MLLM performance on VSI-Bench. For Gemini-1.5 Pro, video-first ordering reduces performance, whereas repeating the video improves it.

  • The experiments test whether question timing and repeated visual exposure alter MLLM performance, motivated by human attention and review behavior.
  • 2.5% lower overall performance results when Gemini switches from question-first to video-first input ordering.
  • A second presentation of the identical video produces a 2.1% performance gain for Gemini.

E. Visualization Results

The visualization materials illustrate benchmark task formats, reasoning traces, error analyses, and predicted cognitive maps. They include examples spanning object counting, measurement estimation, and other VSI-Bench evaluations.

  • VSI-Bench examples display the structure and format of its tasks, questions, and answers.
  • Error-analysis examples categorize model errors and highlight the explanation segments associated with each error type.
  • CoT visualizations show reasoning procedures for object count, object size, and room size under zero-shot, self-consistency, and tree-of-thought prompting.
  • Cognitive-map examples pair ten additional predicted maps with corresponding ground-truth layouts to inspect their alignment.
  • The displayed examples include sofa counting and fireplace-dimension estimation, while benchmark prompts distinguish numerical-answer and multiple-choice-answer questions.
Loading 2412.14171v2…