PaperlayerView exact revision on arXiv

Source-linked AI summary

Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning

Jiacheng Hua, Yishu Yin, Yuhang Wu, Tai Wang, Yifei Huang, Miao Liu

arXiv:2603.23404v2cs.CVcs.CL

TL;DR

MLLMs struggle with 3D spatial reasoning because they fail to construct structured abstractions from video observations. TRACE prompts them to generate textual allocentric representations containing scene context, camera trajectories, and entities as intermediate reasoning traces, and experiments report consistent improvements across VSI-Bench and OST-Bench and diverse MLLM backbones.

  • Problem

    MLLMs struggle with 3D spatial reasoning because they rely on 2D cues and shortcut correlations instead of constructing structured 3D scene abstractions.

  • Method

    TRACE prompts MLLMs to generate a text-based allocentric representation of the 3D environment, integrating meta-context, camera trajectory, and entity-level information.

  • Results

    TRACE consistently improves performance on VSI-Bench and OST-Bench across diverse proprietary and open-source MLLM backbones.

  • Takeaways & Limitations

    Structured textual allocentric representations provide an effective intermediate reasoning interface for video-based spatial QA in MLLMs.

  • Takeaways & Limitations

    TRACE is static, which can decouple global scene consistency from dynamic egocentric updates needed for precise real-time agent tracking in multi-turn settings.

Abstract

from arXiv · show

Existing Multimodal Large Language Models (MLLMs) struggle with 3D spatial reasoning, as they fail to construct structured abstractions of the 3D environment depicted in video inputs. To bridge this gap, drawing inspiration from cognitive theories of allocentric spatial reasoning, we investigate how to enable MLLMs to model and reason over text-based spatial representations of video. Specifically, we introduce Textual Representation of Allocentric Context from Egocentric Video (TRACE), a prompting method that induces MLLMs to generate text-based representations of 3D environments as intermediate reasoning traces for more accurate spatial question answering. TRACE encodes meta-context, camera trajectories, and detailed object entities to support structured spatial reasoning over egocentric videos. Extensive experiments on VSI-Bench and OST-Bench demonstrate that TRACE yields notable and consistent improvements over prior prompting strategies across a diverse range of MLLM backbones, spanning different parameter scales and training schemas. We further present ablation studies to validate our design choices, along with detailed analyses that probe the bottlenecks of 3D spatial reasoning in MLLMs.

1 Introduction

Existing MLLMs struggle with 3D spatial QA because they rely on 2D cues instead of constructing structured scene abstractions. TRACE addresses this gap by using textual allocentric representations as intermediate reasoning traces, with gains reported on VSI-Bench and OST-Bench.

  • Human allocentric reasoning motivates representing room layouts and spatial relations textually instead of reasoning directly from individual egocentric frames.
  • MLLMs often rely on 2D visual signals and shortcut correlations rather than building hierarchical abstractions of 3D scenes.
  • TRACE uses structured textual descriptions of 3D spatial structure as intermediate reasoning traces for MLLM spatial question answering.
  • TRACE integrates room meta-context, camera trajectory, and an explicit entity registry to support structured spatial reasoning.
  • Experiments on VSI-Bench and OST-Bench report clear and consistent gains over prior prompting strategies across diverse MLLM backbones.

2 Related Work

Related work addresses spatial reasoning through supervised data, geometric modalities, intermediate representations, and general prompting strategies. TRACE instead focuses on prompting MLLMs to explicitly reason over textual representations of 3D geometry.

  • Spatial Representation: Spatial reasoning research spans vision-language models, embodied navigation, and systems augmented with explicit 3D or geometric modalities.
  • Intermediate Representations: Thinking in Space reports that externalizing a cognitive-map-like spatial representation improves reasoning, whereas standard chain-of-thought provides limited benefit.
  • Intermediate Representations: SpatialEval finds that strong MLLMs may fail to construct consistent internal 3D representations and instead rely on shortcut correlations from 2D pretraining.
  • Text-based Description of Video: Video-description research includes captioning, dense event narration, paragraph storytelling, and video-language pretraining for retrieval and QA.
  • Prompting in M/LLM: TRACE differs by designing textual representations that enable explicit reasoning over 3D geometry cues rather than optimizing long-context evidence retrieval.

3 Method

TRACE reformulates spatial reasoning around a structured textual representation generated from egocentric video and queries. Its components encode scene context, camera motion, and entities so the model can use the representation to answer spatial questions.

  • Problem Formulation: Standard chain-of-thought often fails on complex spatial tasks, motivating a reasoning trace that captures geometric structure.
  • Problem Formulation: TRACE generates an allocentric 3D-environment representation before producing the spatial answer.
  • Problem Formulation: The task conditions answer generation on an egocentric video V and query Q, with TRACE G serving as the structured intermediate representation.
  • Problem Formulation: The spatial descriptor produces TRACE from the video and query, after which the reasoning parser uses TRACE to generate the final answer.
  • TRACE Components: TRACE is defined as G = ⟨M, T, E⟩, comprising meta-context M, camera trajectory T, and entity set E.
  • Meta Context: Meta-context establishes room topology, grid alignment, and the observer’s initial heading within a room-aligned coordinate system.
  • Camera Trajectory: The trajectory records timestamps, estimated positions, and facing directions across discrete temporal steps, approximating orientation with eight directions.
  • Entity Registry: The entity registry tracks observed objects using timestamps, visual signatures, estimated metric coordinates, and spatial relations to support localization across time.

4 Experiments

Experiments evaluate TRACE on VSI-Bench and OST-Bench across multiple MLLM backbones, prompting strategies, inference settings, and text-based spatial representations. TRACE generally improves spatial reasoning, while analyses identify model, representation, efficiency, and dynamic-tracking boundaries.

  • Experimental Setup: VSI-Bench contains 5,130 question-answer pairs across 288 egocentric indoor-scene videos and eight configurational, measurement-estimation, and spatiotemporal tasks.OST-Bench contains 1,386 scenes and 10,165 question-answer pairs and evaluates online spatio-temporal understanding from an embodied-agent perspective.
  • Experimental Setup: VSI-Bench reports Accuracy for multiple-choice questions and Mean Relative Accuracy for numerical questions, with MRA averaging correctness across thresholds from 0.5 to 0.95.A numerical prediction is correct at threshold θ only when its relative error is less than 1 − θ.
  • Comparison of Different Prompting Methods: +7.54%, +3.10% and +1.63% performance improvements are achieved by TRACE for Gemini, Qwen and MiMo, respectively, on VSI-Bench.Advanced prompting helps Gemini consistently but yields marginal gains or performance compromises for Qwen; TRACE remains robust across the evaluated base models and additional state-of-the-art models.
  • Comparison of Different Prompting Methods: +1.2% absolute performance gain on Gemini and +2.4% on MiMo are achieved by TRACE on OST-Bench.Step-by-step prompts may hinder grounding and spatial-context updates across OST’s multi-turn reasoning, whereas spatially specific prompting is superior for compact MiMo.
  • Comparison of Different Prediction Setting: One-stage inference outperforms two-stage inference for both Qwen and Gemini, while text-only inference matches direct video-based performance with Gemini.The results indicate that both the generated representation and the reasoning process used to produce it contribute to prediction accuracy.
  • Comparison of Different Prediction Setting: TRACE outperforms Cognitive Map by 9.7% and Spatial Caption by 3.53% on VSI-Bench under text-only inference with Qwen2.5-VL-72B.The qualitative comparison attributes the advantage to TRACE’s ability to capture the 3D granularity needed for complex spatial reasoning.
  • Ablation Studies: Replacing the reasoning parser with Qwen2.5-VL-7B causes substantially larger degradation than replacing the spatial descriptor, indicating a stronger sensitivity to parser capacity.The comparison uses Gemini 3 Pro as a theoretical upper bound and finds especially large drops when components are replaced with Qwen2.5-VL-72B.
  • Token Efficiency: TRACE is more token-efficient than advanced baselines such as ToT and LtM while delivering better performance on compact MiMo models.The authors identify token-efficiency optimization as an orthogonal direction left for future work.

5 Conclusions

TRACE uses structured textual allocentric representations as intermediate reasoning traces for spatial understanding, consistently improving MLLM performance across benchmarks and diverse backbones.

  • TRACE enables MLLMs to use textual allocentric context from egocentric video as an intermediate reasoning trace for spatial understanding.The representation models meta-context, camera trajectory, and entity-level grounding.
  • TRACE consistently improves performance on VSI-Bench and OST-Bench across proprietary and open-source model backbones.The paper also compares TRACE with prior linguistic prompting methods and other text-based spatial representations.
  • Ablation studies, comparative evaluations, and decompositional analyses validate TRACE’s design choices and reveal common MLLM spatial-reasoning failure modes.The analyses examine how text-based representations can be leveraged and where spatial reasoning bottlenecks arise.
  • The authors present TRACE as a simple, widely applicable interface for studying structured spatial reasoning in off-the-shelf MLLMs.They describe the approach as a promising direction for advancing spatial reasoning and motivating cognitively inspired representations.

Limitations and Future Work

The current TRACE framework is static and relies on the vision-language model to generate scene representations, motivating dynamic, specialized, and more efficient future extensions.

  • Limitations: TRACE’s static allocentric representation preserves global topological consistency but is decoupled from dynamic egocentric updates needed for precise real-time agent tracking in multi-turn settings.
  • Limitations: The current implementation uses the vision-language model itself to generate scene representations, while specialized visual experts may improve their accuracy.
  • Future Work: Future work could develop dynamic streaming TRACE that incrementally updates camera trajectories and entity registries as new observations arrive.The proposed framework would maintain a persistent world model and recursively re-project the agent’s pose within the map.
  • Future Work: Future work could internalize TRACE-like reasoning through supervised fine-tuning and reinforcement learning so structured spatial representations become part of native model reasoning.
  • Future Work: The authors propose integrating specialized 3D perception modules while studying representation compression and token efficiency for low-latency embodied agents.

A Prompting Details

The appendix standardizes prompt assembly and compares linguistic reasoning protocols, while TRACE adds a structured intermediate representation before final answering.

  • A.1 Overall Structure: Most prompting methods share a base system prompt and differ mainly in the user prompt, enabling controlled comparison across strategies.The system prompt frames egocentric indoor-video spatial reasoning and requires an accurate answer without extra final-line text.
  • A.1 Overall Structure: Final prompts concatenate the user prompt with explicit question and, when applicable, options blocks, followed by a required answer format.Multiple-choice questions append an options string; open-ended and multiple-choice tasks use distinct post-prompts.
  • A.2 User Prompts for Linguistic Reasoning Methods: Chain-of-Thought prompting requests step-by-step natural-language reasoning followed by a final answer line.
  • A.2 User Prompts for Linguistic Reasoning Methods: Direct prompting suppresses explicit reasoning and instructs the model to output only the final answer.
  • A.2 User Prompts for Linguistic Reasoning Methods: Tree-of-Thoughts generates three reasoning branches, evaluates them for consistency and spatial coherence, and answers from the selected branch.
  • A.2 User Prompts for Linguistic Reasoning Methods: Least-to-Most decomposes questions from easier to harder subproblems, solves them sequentially, reuses intermediate results, and maintains layout consistency.Stages include identifying objects and regions, solving local relations, integrating a global layout, and making the final decision.
  • A.2 User Prompts for Linguistic Reasoning Methods: Cognitive Map prompting estimates object-instance centers on a 10 × 10 grid for fixed indoor categories and outputs a strict JSON dictionary.
  • A.2 User Prompts for Linguistic Reasoning Methods: TRACE first generates a YAML intermediate representation containing Meta_Context, Trajectory, and Entity_Registry before answering.It summarizes a room-aligned coordinate system, camera trajectory, and timestamped entity positions.

B.1 Details on Decomposition Analysis

The decomposition analysis breaks down spatial QA by task and descriptor–parser combinations, identifying bottlenecks across object-, relation-, and layout-level concepts.

  • Tables 5 and 6 provide per-category and per-task compositional breakdowns of spatial reasoning performance.
  • The decomposition results highlight bottlenecks in modeling object-, relation-, and layout-level spatial concepts.
  • The analysis compares combinations of visual descriptors and spatial knowledge parsers in a text-only prediction setting.Table 5 reports average results with multiple-choice and numerical-answer components; Table 6 presents a grouped breakdown.
  • Qwen-VL consistently underperforms the language-only Qwen model under both Gemini-generated and Qwen-generated textual representations.The comparison uses the same textual representations across the two model variants.

B.3 Detailed Results on Stratified Analysis

TRACE improves performance over the Direct baseline across ARKitScenes, ScanNet, and ScanNetPP for both proprietary and open-weights architectures, indicating cross-environment generalization.

  • The gains occur across both proprietary and open-weights architectures.
  • TRACE robustly improves performance over the Direct baseline across ARKitScenes, ScanNet, and ScanNetPP.The comparison covers three distinct indoor scene datasets comprising VSI-Bench.
  • Balanced improvements across environment distributions indicate that TRACE does not overfit to one dataset’s spatial characteristics.

B.4 Full Evaluation Results on VSI-Bench

The complete VSI-Bench results are reported in Table 9 using model versions selected to reduce the risk of data contamination.

  • The complete VSI-Bench results are detailed in Table 9.
  • The evaluation restricts model versions to those released no later than six months after VSI-Bench and OST-Bench were published.This restriction is intended to mitigate data contamination risk.
  • The final model selection includes Gemini 3 Pro, o3, Qwen2.5-VL, MiMo-VL-7B, and GLM-4.5V.

B.5 Token Efficiency

TRACE generally offers a favorable performance-to-cost trade-off, but its token usage varies by backbone and can exceed some baselines for larger foundation models.

  • TRACE generally maintains a favorable performance-to-cost trade-off relative to highly branching reasoning methods.
  • 737.72 tokens let TRACE outperform Tree-of-Thoughts at 1132.86 and Least-to-Most at 1097.05 on MiMo-VL-7B.
  • TRACE is more token-intensive than some baselines on Gemini 3 Pro, o3, and Qwen2.5-VL-72B.The additional consumption reflects explicitly generating an exhaustive structured allocentric representation.
  • Optimizing token efficiency during structured reasoning is left for future work.
Loading 2603.23404v2…