PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks

Xu Zheng, Zihao Dongfang, Lutao Jiang, Boyuan Zheng, Yulong Guo, Zhenquan Zhang, Giuliano Albanese, Runyi Yang, Mengjiao Ma, Zixin Zhang, Chenfei Liao, Dingcheng Zhen, Yuanhuiyi Lyu, Yuqian Fu, Bin Ren, Linfeng Zhang, Danda Pani Paudel, Nicu Sebe, Luc Van Gool, Xuming Hu

arXiv:2510.25760v2cs.CV

TL;DR

Systematic reviews and public benchmarks for multimodal spatial reasoning remain limited despite promising progress by large multimodal models. This survey synthesizes methods and tasks across spatial reasoning, embodied AI, and emerging modalities; its synthesis highlights the value of structural geometric bias and balanced spatial representations.

  • Problem

    Systematic reviews and evaluation of multimodal models for spatial reasoning remain limited despite progress across spatial tasks.

    Source: A. Background · p. 1

  • Method

    The survey reviews spatial reasoning methods and tasks, including test-time scaling, post-training, model design, explainability, 3D vision, embodied AI, video, and audio.

    Source: §IX · p. 24

  • Results

    The survey finds that structural geometric bias, including spatial feature fusion, relation graphs, and spatial cross-attention, helps preserve layout through reasoning.

    Source: 2) Redesigning Spatial Reasoning Modules · p. 9

  • Takeaways & Limitations

    The review points toward balancing semantic and spatial representations, with explicit 3D-aware modules and cross-view fusion, rather than relying on scale alone.

    Source: D. Explainability of Multimodal… · p. 9

    Large-scale datasets for spatially informative modalities remain scarce relative to vision-language corpora, and approximations often underperform.

    Source: 1) Spatially informative input… · p. 14

Abstract

from arXiv · show

Humans possess spatial reasoning abilities that enable them to understand spaces through multimodal observations, such as vision and sound. Large multimodal reasoning models extend these abilities by learning to perceive and reason, showing promising performance across diverse spatial tasks. However, systematic reviews and publicly available benchmarks for these models remain limited. In this survey, we provide a comprehensive review of multimodal spatial reasoning tasks with large models, categorizing recent progress in multimodal large language models (MLLMs) and introducing open benchmarks for evaluation. We begin by outlining general spatial reasoning, focusing on post-training techniques, explainability, and architecture. Beyond classical 2D tasks, we examine spatial relationship reasoning, scene and layout understanding, as well as visual question answering and grounding in 3D space. We also review advances in embodied AI, including vision-language navigation and action models. Additionally, we consider emerging modalities such as audio and egocentric video, which contribute to novel spatial understanding through new sensors. We believe this survey establishes a solid foundation and offers insights into the growing field of multimodal spatial reasoning. Updated information about this survey, codes and implementation of the open benchmarks can be found at https://github.com/zhengxuJosh/Awesome-Spatial-Reasoning.

I. INTRODUCTION

Multimodal models show promise across spatial tasks, but systematic reviews and evaluation remain limited. This survey maps the field across reasoning methods, spatial tasks, embodied AI, emerging modalities, and open benchmarks.

A. Background

  • Although multimodal models show promise across spatial tasks, systematic reviews and performance evaluation for multimodal spatial reasoning remain limited.Source: A. Background · pp. 1–2

B. Contributions

  • The survey reviews general spatial reasoning through test-time techniques, post-training, explainability, and model architecture, then covers spatial relationships, scene understanding, and 3D grounding.Source: B. Contributions · p. 2
  • It also examines embodied AI, including vision-language navigation and action models for tasks in dynamic environments.Source: B. Contributions · p. 2
  • The survey extends to audio and egocentric video and introduces open benchmarks and evaluation protocols for comparing models.Source: B. Contributions · p. 2

C. Related Works

Prior surveys cover related reasoning and multimodal topics but leave gaps in cross-modal spatial evaluation. This section defines the task broadly and organizes its types, evaluation protocols, and roadmap.

C. Related Works

  • Related surveys focus variously on unimodal reasoning, inference scaling, 3D implementation, or broad multimodal advances, without systematic spatial evaluation frameworks for dynamic real-world settings.Source: C. Related Works · p. 3

II. PROBLEM SETUP: MULTIMODAL SPATIAL REASONING

  • Multimodal spatial reasoning maps heterogeneous inputs to textual, geometric, or executable outputs under a specified reference frame.Source: §II · p. 3

A. Types of Spatial Reasoning in MLLMs

B. Evaluation Protocols for Spatial Reasoning

The survey frames spatial-reasoning evaluation across modalities, tasks, process transparency, robustness, and interactive settings. It organizes general multimodal spatial reasoning around test-time scaling, post-training, architecture, and explainability, with prompting and tool use among the inference-time strategies.

B. Evaluation Protocols for Spatial Reasoning

  • Evaluation should span modality integration, task coverage, process transparency, generalization and robustness, interactive testing, and reproducible synthetic and real-world suites.The roadmap applies these dimensions across general 2D, 3D, embodied, and emerging-modality settings.Source: B. Evaluation Protocols for Spatial… · p. 4

III. GENERAL MULTIMODAL SPATIAL REASONING

  • General multimodal spatial reasoning aligns visual perception with language to answer spatial questions, localize objects, understand perspectives and 3D scenes, and navigate.The survey groups research directions into test-time scaling, post-training, architecture, and explainability.Source: §III · p. 4

A. Test-Time Scaling Methods

  • Test-time scaling enhances spatial reasoning during inference without retraining or fine-tuning, using prompting, tool-assisted reasoning, and external modality integration.The survey groups existing work into three methodological categories.Source: A. Test-Time Scaling Methods · p. 4

1) Prompt Engineering:

  • Spatial prompts outperform reliance on long textual reasoning by using explicit geometric cues, visual traces, structured representations, and reference-based transformations.Cognitive graphs and coordinates help with spatial questions; bounding boxes or scene graphs support multi-hop reasoning, while visualization of reasoning traces boosts 2D accuracy.Source: 1) Prompt Engineering · p. 5

2) Tool Usage:

  • Inference-time tools expose spatial evidence through visual operations, structured perception outputs, and reconstructed views, sometimes orchestrated by agents.The approaches append images or traces, serialize perception as tokens or sketches, or render views from reconstructed geometry; unresolved issues include error propagation, weak persistent memory, inconsistent units and frames, and added cost and latency.Source: 2) Tool Usage · pp. 5–6

3) Others:

The survey reviews inference-time strategies alongside post-training approaches for spatial reasoning. It describes supervised fine-tuning as structured spatial supervision and reinforcement learning as reward-driven optimization, while noting trade-offs and current limits.

3) Others:

  • Self-consistency, visual-textual search, and retrieval-augmented generation expand inference beyond static prompts; retrieved spatial facts can reduce hallucinations and sharpen relations.LOGIC-RAG builds a dynamic first-order-logic knowledge base from visual input, while multi-sampling and adaptation increase compute and retrieval depends on knowledge quality.Source: 3) Others · p. 6

B. Post-Training Methods

  • Post-training adapts MLLMs after pretraining through supervised fine-tuning and reinforcement learning using spatial datasets, rewards, and curricula.Source: B. Post-Training Methods · p. 6

1) Supervised Fine-tuning (SFT):

  • SFT uses static and dynamic spatial data, explicit coordinates and temporal anchors, and curricula that progress from perception to multi-hop reasoning.Human-labeled or synthetic data constrain scale and diversity; motion-aware and temporally annotated data support reasoning about static configurations and dynamic evolution.Source: 1) Supervised Fine-tuning (SFT) · pp. 6–7

2) Reinforcement Learning (RL):

  • RL methods encode geometric and temporal correctness in rewards, use staged SFT-to-RL training and self-play, and target more stable spatial learning.Current frameworks face high computational cost, reward sparsity, and limited generalization across 2D, 3D, and temporal domains.Source: 2) Reinforcement Learning (RL) · pp. 7–8

C. MLLM Architectural Modifications

MLLMs’ standard components do not by themselves preserve the positional and geometric information spatial reasoning needs. The survey reviews input-level cues and dedicated spatial modules, then summarizes evidence that current models often lack genuine geometric grounding.

C. MLLM Architectural Modifications

1) Enhancing Input Representations:

2) Redesigning Spatial Reasoning Modules:

  • Dedicated spatial components preserve geometric structure through specialized encoders, 2D+3D fusion, depth modules, and spatially aware visual aggregation.SPATIAL-MLLM [17] reports 35–45% relative improvements over strong baselines; CAMBRIAN-1 uses its Spatial Vision Aggregator to reduce visual-token demands while retaining fine spatial detail.Source: 2) Redesigning Spatial Reasoning Modules · pp. 8–9

D. Explainability of Multimodal Spatial Reasoning

  • Rajabi et al. find that MLLMs often rely on object co-occurrence instead of geometric grounding, and propose grounded subject–object–relation triplets linked by a relational bridge.The bridge links detection and positional features.Source: D. Explainability of Multimodal… · p. 9
  • Qi et al. [20] attribute erased spatial order to dominant vision embeddings suppressing positional encodings, and propose vision-token normalization plus mid-layer geometric features.Source: D. Explainability of Multimodal… · p. 9
  • Chen et al. [21] report that 15–20% of attention weights target spatial-relation regions, while Wen et al. find reliance on bounding-box heuristics; Zhang et al. [133] report diminishing gains from scaling multimodal data.The survey’s synthesis points to representational balance, relation-aware attention, and geometric priors as complementary directions.Source: D. Explainability of Multimodal… · p. 9

IV. MULTIMODAL SPATIAL REASONING IN 3D SPACE

The survey groups 3D spatial reasoning around grounding, scene reasoning, and generation. For 3D visual grounding, it reviews direct 3D inputs, multi-view images, and hybrid 2D–3D methods, alongside challenges in generalization and spatial alignment.

IV. MULTIMODAL SPATIAL REASONING IN 3D SPACE

  • Core 3D spatial reasoning tasks include visual grounding, scene reasoning, and generation, with links to downstream navigation and vision-language-action tasks.Source: §IV · p. 9, Figure 4 · p. 10

A. 3D Visual Grounding

  • 3D grounding localizes an object from a natural-language description, while traditional supervised methods on limited datasets struggle with unseen objects and complex texts.MLLM-based approaches are grouped by direct 3D inputs, rendered multi-view images, or hybrid 2D–3D inputs.Source: A. 3D Visual Grounding · p. 10

1) 3D Input:

  • Direct-input methods embed point clouds, voxels, or volumetric features in MLLMs, using strategies such as coarse-to-fine proposals and alignment training.The survey notes that complex 3D structures can hinder interpretability and limited labeled 3D data constrains robust open-world development.Source: 1) 3D Input · pp. 10–11

2) Multi-view Input:

  • Multi-view methods use 2D MLLMs with minimal modification, but view discrepancy can misalign the model’s perspective with the grounding instruction.Approaches include learnable view prototypes, dynamically stitched image sequences, and inserted 3D coordinate axes; global-image models may also struggle with object regions and RGB lacks depth or coordinates.Source: 2) Multi-view Input · p. 11

3) Hybrid of 2D and 3D:

  • Hybrid methods combine 2D and 3D representations through depth maps, spatial descriptions, visual programs, or lifted detections to support 3D grounding.The survey presents combining multiview images and 3D structures as a way to enhance grounding performance and robustness.Source: 3) Hybrid of 2D and 3D · p. 11

B. 3D Scene Reasoning and Question Answering (QA)

The survey distinguishes training-required and training-free approaches to 3D scene reasoning and QA, then reviews 3D generation through layout and programmatic methods. It describes alignment, efficiency, representation, and inference strategies alongside their constraints.

B. 3D Scene Reasoning and Question Answering (QA)

1) Training-required:

  • Training-required methods address feature alignment, training efficiency, and alternative 3D representations such as scene graphs, 3DGS, and BEV.Examples include detector- or Q-Former-based alignment, projection-layer approaches, instruction tuning, and structured 3D representations; the survey notes computational cost, limited annotated data, and interpretability challenges.Source: 1) Training-required · pp. 11–12

2) Training-free Methods:

  • Training-free methods use MLLM priors and prompting strategies, including progressive prompting, viewpoint selection, and iterative self-correction, but can depend on base-model quality and slow inference.Source: 2) Training-free Methods · p. 12

C. 3D Generation with Spatial Reasoning

1) 3D Layout Generation:

  • Layout pipelines use direct LLM-generated spatial configurations or indirect semantic representations such as scene graphs, with optimization, iterative refinement, and interactive interfaces among the approaches reviewed.Direct mappings can produce implausible configurations such as overlapping objects; both paradigms aim for semantically coherent and physically feasible environments.Source: 1) 3D Layout Generation · p. 12

2) 3D Generation as Program:

MLLM-based 3D generation can specify geometry and layout through programs, offering a more controllable alternative to direct 3D representation generation. In embodied AI, spatial reasoning supports action and navigation, with VLA systems using different strategies to connect multimodal inputs to actions.

2) 3D Generation as Program:

  • Programmatic 3D generation represents geometry and layout as code, with current approaches targeting Blender scripts, CAD parametric programs, and mesh-generation pipelines.The survey describes this route as more controllable and better suited to real applications than directly generating 3D representations.Source: 2) 3D Generation as Program · pp. 12–13
  • 3D-GPT [32] generates Blender scripts from natural-language commands, while SceneCraft refines scenes through MLLM feedback and accumulates spatial knowledge across iterations.Other approaches include example-derived visual programs in SceneMotifCoder [177], CAD-focused methods, and the ShapeLib [181] hybrid human-AI workflow for mesh libraries.Source: 2) 3D Generation as Program · p. 13

V. MULTIMODAL SPATIAL REASONING IN EMBODIED AI

  • Embodied AI tasks include VLA, VLN, and other agent tasks, where geometric reasoning, navigation, and perspective-taking rely on spatial reasoning.VLA models generate executable actions from visual observations and language instructions, using end-to-end reactive policies, task decomposition, or intermediate affordance and goal-state predictions.Source: §V · p. 13, A. Multimodal Spatial Reasoning in VLA… · pp. 13–14

1) Spatially informative input modalities:

The surveyed VLA approaches enrich spatial inputs, train across related tasks, or add explicit reasoning steps to support action generation. Their upstream vision-language backbones also show spatial reasoning abilities on benchmarks relevant to embodied tasks.

1) Spatially informative input modalities:

  • 3D-VLA, PointVLA [192], SpatialVLA [193], and BridgeVLA [194] incorporate depth, point clouds, or 3D-aware representations to enrich spatial context for action prediction.The survey notes that these inputs compensate for geometric cues often missing from 2D visual data.Source: 1) Spatially informative input… · p. 14
  • Extra spatial modalities show promise, but scarce large-scale datasets and underperforming modality imputations remain challenges; large-scale 2D vision-language-trained models still lead overall.The survey identifies targeted pre-training and more data-efficient architectures as needed to better leverage extra modalities.Source: 1) Spatially informative input… · p. 14

2) Multi-task Pre- and Co-training:

3) Explicit Reasoning:

  • Explicit-reasoning VLAs use intermediate plans or affordances before action: ECoT, Chat-VLA2 [46], Chain-of-Affordance [198], and RT-Affordance illustrate this direction.The survey associates reasoning augmentation with improved robustness, generalization, and interpretability, while noting that selective reasoning and asynchronous pipelines can mitigate inference overhead.Source: 3) Explicit Reasoning · p. 15

4) Multimodal Spatial Reasoning in Vision Language backbone:

B. Multimodal Spatial Reasoning in VLN Models

  • VLN is a cooperative multimodal task in which an agent follows human instructions to navigate 3D environments under ambiguity.Its components include visual perception, language understanding, decision-making, and navigation execution.Source: B. Multimodal Spatial Reasoning in VLN… · pp. 15–16

1) Visual Environment Understanding and Generalization:

VLN methods preserve or structure scene information, improve grounding of human instructions, and combine spatial grounding with planning. Persistent challenges include alignment between visual features and language, particularly under unfamiliar views or domain shifts.

1) Visual Environment Understanding and Generalization:

  • VLN scene-understanding methods use multiview images, depth, scene graphs, and 3D representations to preserve spatial features, align language with 3D scenes, and support spatial reasoning and long-term perception.Examples include NaviLLM for multiview images, SpatialBot [127] for depth-based geometric querying, and ConceptGraphs for open-vocabulary 3D scene representation; Spartun3D-LLM [34] links 3D representations with text, while 3D-Mem retrieves spatial information from multiview 3D snapshots.Source: 1) Visual Environment Understanding… · p. 16, Figure 9 · p. 16
  • Visual-language alignment remains challenging for VLN agents, especially with unfamiliar views or domain shifts.The survey identifies scene graphs, BEV maps, and multiview memory as structured representations used to link perception with reasoning and planning.Source: 1) Visual Environment Understanding… · p. 16

2) Human Intention Interpretation and Instruction Comprehension:

3) Path Planning and Navigation for VLN Agents:

C. Multimodal Spatial Reasoning in Other Embodied Tasks

Other embodied tasks use spatial reasoning for question answering, grasping, and world modeling, while video work targets spatial understanding across time. The survey notes remaining challenges in spatial detail, memory, and generalization across video domains.

1) Embodied Question Answering (EQA):

  • Embodied question answering requires exploring an environment, gathering visual evidence, and grounding language in spatial perception and reasoning.Approaches include scene-graph memory for multi-turn QA, CoT reasoning in Embosr, and separating VLM perception from lightweight-model reasoning [217].Source: 1) Embodied Question Answering (EQA) · p. 17

2) Embodied Grasping:

  • Grasping methods address clutter and occlusion using obstruction-aware planning, keypoints, affordance reasoning, or segmentation and diffusion-based generation.Examples include THINKGRASP [42], FREEGRASP [43], AFFORDGRASP [219], and UNIDIFFGRASP [220].Source: 2) Embodied Grasping · p. 18

3) Embodied World Models:

  • Embodied world models need structurally consistent scenes because 2D pixel-only models can miss spatial relationships and provide weak depth or pose estimates.EVA [48], TESSERACT [47], and 3DFLOWACTION [221] model video synthesis, temporal evolution, or object-level scene flow.Source: 3) Embodied World Models · p. 18

A. Spatial Reasoning with Video

  • Video spatial reasoning must track properties and correspondences through dynamic scenes; current approaches use spatially grounded training, tracking tokens, explicit memory, or geometric-aware representations.The survey describes persistent challenges from token compression, limited spatial memory, and generalization across video domains; egocentric video also requires distinguishing agent motion from object motion.Source: A. Spatial Reasoning with Video · pp. 18–19, Table XV · p. 19

B. Spatial Reasoning with Audio

Audio spatial reasoning asks models to infer where sounds originate, while the survey reviews audio and audio-visual benchmarks and the broader evolution of multimodal spatial reasoning datasets.

B. Spatial Reasoning with Audio

  • SpatialSoundQA offers over 21,000 simulated binaural clips with questions on direction, distance, and multiple sources; SAVVY-Bench tests 3D reasoning in dynamic scenes using synchronized spatial audio.BAT combines a spatial-audio encoder with an LLM and curriculum learning; SAVVY extracts spatial tracks to build a global spatial map.Source: B. Spatial Reasoning with Audio · p. 19
  • Current audio models still struggle with open-world scenes containing multiple dynamic sound sources, and scarce high-quality annotated datasets hinder performance beyond controlled or simulated environments.The survey points to richer real-world egocentric recordings, improved simulation, and specialized architectures as promising directions.Source: B. Spatial Reasoning with Audio · p. 20

VII. BENCHMARKS

  • The benchmark landscape expanded from basic spatial tasks toward richer domains and modalities, with the timeline marking benchmarks by modality combinations.The survey describes coverage spanning panoramic imagery, video, CAD, and GIS.Source: §VII · p. 20, Figure 11 · p. 21

A. Early Multimodal Spatial Reasoning Benchmarks

  • Early benchmarks moved beyond basic image-text localization and relationship detection to include video-temporal reasoning and varied spatial relationships.Visual Genome supplies annotated images and graphs; TVQA+ [233] combines video clips with object-detection annotations, while VSR defines explicit spatial reasoning tasks.Source: A. Early Multimodal Spatial Reasoning… · pp. 20–21

B. Image-Text Spatial Reasoning Benchmarks

Image-text spatial benchmarks now span varied 2D reasoning tasks, while 3D datasets increasingly use generated annotations and real-world validation; synthetic-data generalization remains a concern.

1) 2D Spatial Reasoning Tasks:

  • 2D benchmarks assess navigation, localization, and layout generation, using settings that range from driving questions and controlled synthetic tasks to viewpoint transformations.DriveMLLM [238] uses question-answer pairs for driving scenarios; SpatialEval includes Spatial-Map and Maze-Nav, and SPACE [204] spans large- and small-scale tasks.Source: 1) 2D Spatial Reasoning Tasks · p. 21

2) Hybrid Approaches and Abstract Representations:

  • Hybrid and abstract benchmarks include positional relationships, context, navigation, dynamic reasoning, and real-world tasks such as autonomous driving and robotics.OmniSpatial [96] includes dynamic reasoning, traffic analysis, and geometric decomposition.Source: 2) Hybrid Approaches and Abstract… · p. 21

3) Insights & Discussion:

  • Synthetic data can accelerate benchmarking, but the survey notes challenges in generalization and real-world applicability and calls for dynamic real-world and hybrid datasets.Source: 3) Insights & Discussion · p. 21

4) 3D Spatial Reasoning Benchmarks:

  • 3D benchmarks have progressed toward dynamic scenes and multi-perspective reasoning, using synthesis-driven annotation and automated 2D-to-3D conversion alongside real-world validation.SpatialRGPT-Bench [18] derives 3D tasks from 2D scenes using instance segmentation and depth estimation; Spatial457 [241] covers 6D localization, orientation, and multi-object relationships.Source: 4) 3D Spatial Reasoning Benchmarks · pp. 21–22

C. Video-Text Spatial Reasoning Benchmarks

1) Fundamental Spatial Perception Tasks:

The survey tracks benchmark development from fundamental spatial perception to temporally grounded reasoning and broader multimodal tasks, while identifying data, alignment, and generalization challenges.

1) Fundamental Spatial Perception Tasks:

  • VIS-100K contains 100,000 video-question-answer pairs across six spatial tasks, while VIS-BENCH and SPACER-151K broaden evaluation to multiple tasks and spatial metadata.SPACER-151K includes 91K spatial QA pairs and 60K general video-understanding examples, with bounding boxes, temporal indices, and 10×10 grid maps; its quality control targets balanced, unambiguous data.Source: 1) Fundamental Spatial Perception Tasks · p. 22

2) Advanced Spatiotemporal Reasoning Tasks:

  • Advanced benchmarks evaluate spatial and temporal localization jointly, reverse egocentric reasoning, and linked What→When→Where reasoning chains.V-STAR [225] uses the Logarithmic Geometric Mean to jointly measure accuracy, temporal IoU, and spatial IoU.Source: 2) Advanced Spatiotemporal Reasoning… · pp. 22–23

3) Mixed-Task Benchmarking:

  • Video-R1-COT-165k provides chain-of-thought samples for supervised fine-tuning, while Video-R1-260k supports reinforcement learning; only about 8% of their samples involve explicit spatial reasoning.The survey also identifies annotation cost, inconsistent semi-automated annotation quality, and homogeneous templated data as constraints.Source: 3) Mixed-Task Benchmarking · p. 23

D. Other Modal Benchmarks

  • Other modalities and settings include urban-scene navigation, CAD model synthesis, open-vocabulary reasoning over scene graphs and point clouds, and interactive goal-driven spatial tasks.CITYINSTRUCTION and CITYEVAL accompany CITYGPT [107]; MM-ESCAPE [253] simulates an interactive escape room requiring sequential spatial reasoning and actions.Source: D. Other Modal Benchmarks · p. 23

VIII. CHALLENGES AND FUTURE DIRECTIONS

  • Future work faces limits in generalization, annotation scale, 3D representation scalability, and benchmark scope, including narrow modality coverage and restricted interaction.The survey calls for unified multimodal benchmarks with explicit coordinate frames and orientation labels, and for closer integration of perception and reasoning in dynamic environments.Source: §VIII · pp. 23–24

IX. CONCLUSION

  • The survey synthesizes work on general spatial reasoning, 3D vision, embodied AI, video, audio, and benchmarks to establish a knowledge foundation for multimodal spatial reasoning research.Source: §IX · p. 24
Loading 2510.25760v2…