Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models

Runsen Xu; Weiyao Wang; Hao Tang; Xingyu Chen; Xiaodong Wang; Fu-Jen Chu; Matt Feiszli; Kevin J. Liang

Source-linked AI summary

Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models

Runsen Xu, Weiyao Wang, Hao Tang, Xingyu Chen, Xiaodong Wang, Fu-Jen Chu, Matt Feiszli, Kevin J. Liang

arXiv:2505.17015v2May 22, 2025cs.CVcs.CL

TL;DR

MLLMs have limited spatial understanding beyond single images, despite their intended use in physical-world applications. The paper introduces MultiSPA and Multi-SpatialMLLM to train and evaluate multi-frame spatial reasoning, achieving broad gains and showing scalability, generalization, multi-task benefits, and potential for robotic reward annotation.

Problem
Existing MLLMs struggle with spatial understanding, while prior spatial-training approaches focus on single-image scenarios rather than multi-frame reasoning.
Source: §1 · p. 1
Method
The paper builds a data pipeline and MultiSPA dataset integrating depth, visual correspondence, and dynamic perception from annotated 3D and 4D scenes, alongside a unified multi-frame benchmark.
Source: §1 · p. 2, §3.1 · p. 3
Results
Multi-SpatialMLLM substantially outperforms base models and proprietary systems, matches specialized 3D perception models, and generalizes and scales across multi-frame spatial tasks.
Source: §1 · p. 2
Takeaways & Limitations
The model provides a scalable multi-frame perception capability with multi-task benefits, preliminary emergent spatial behavior, and potential use as a robot-learning reward annotator.
Source: §1 · p. 2, §5 · p. 8
The current framework focuses on image pairs; extending it to more images and explaining which spatial abilities drive emergence remain open problems.
Source: M. Limitations · p. 12

Abstract

from arXiv · show

Multi-modal large language models (MLLMs) have rapidly advanced in visual tasks, yet their spatial understanding remains limited to single images, leaving them ill-suited for physical-world applications that require multi-frame reasoning. In this paper, we propose a framework to equip MLLMs with multi-frame spatial understanding by integrating fundamental spatial skills, including depth perception, visual correspondence, and dynamic perception. We design a novel data pipeline and collect the MultiSPA dataset of more than 27 million samples spanning diverse 3D and 4D scenes to enable training. Alongside MultiSPA, we introduce a comprehensive benchmark that tests a wide spectrum of spatial tasks under uniform metrics. Our resulting model, Multi-SpatialMLLM, achieves significant gains over baselines and proprietary systems, demonstrating scalable and generalizable multi-frame perception. We further observe multi-task benefits and emergent spatial capabilities in challenging scenarios, and showcase how our model can serve as a multi-frame reward annotator for robotics.

1. Introduction

The paper extends MLLM spatial understanding from single images to multi-frame reasoning by integrating depth, correspondence, and dynamic perception. It introduces MultiSPA data and benchmarking, and reports strong, scalable performance from Multi-SpatialMLLM.

Current MLLMs struggle with basic spatial understanding and remain limited by single-image training for real-world applications.Source: §1 · p. 1
The framework integrates depth perception, visual correspondence, and dynamic perception to support reasoning across multiple images.Source: §1 · pp. 1–2
The data pipeline uses annotated 3D and 4D datasets, balanced image-pair sampling, spatial and temporal alignment, and LLM-generated question–answer templates.Source: §1 · p. 2
MultiSPA contains more than 27 million samples, while its benchmark evaluates diverse multi-frame spatial tasks and output formats under a unified metric.Source: §1 · p. 2, Table 1 · p. 2
Multi-SpatialMLLM substantially outperforms base models and proprietary systems, matches specialized 3D perception models, and generalizes and scales without degrading general abilities.Source: §1 · p. 2
Multi-task training provides notable benefits, reveals preliminary emergent spatial behavior, and supports multi-frame reward annotation for robot learning.Source: §1 · p. 2

2. Related Work

Prior MLLM research addresses spatial understanding through benchmarks and spatial fine-tuning, but much of the literature remains focused on limited image settings and task formats.

MLLMs combine image encoders with language models and are trained with language-modeling objectives, but web-scale image–text data provide insufficient spatial annotations.Source: §2 · p. 2
Existing benchmarks include single-image spatial tasks, image-pair reasoning, and video-based scene-level reasoning with overlapping tasks such as camera movement estimation and keypoint matching.Source: §2 · p. 3
MultiSPA adds object movement perception and supports more diverse input and output formats than related benchmarks described here.Source: §2 · p. 3

3. MultiSPA Dataset and Benchmark

MultiSPA combines annotated 3D and 4D scenes with a data-generation pipeline for diverse multi-frame spatial QA. Its benchmark covers five spatial task families and uses held-out scene data for evaluation.

Tasks definitions

The dataset targets depth, visual correspondence, camera movement, object movement, and object size perception across multiple images.Source: §3.1 · p. 3
Referencing uses pixel coordinates or visual annotations, while outputs include qualitative answers, scalar quantities, pixel coordinates, and displacement vectors.Source: §3.1 · p. 3

Data Format

The data are formatted as image-conditioned question–answer pairs with GPT-4o-generated templates, millimeter rounding, and normalized pixel coordinates.Source: §3.2 · p. 4

Source datasets

The pipeline uses ADT and Panoptic Studio for 4D data, TAPVid3D tracking annotations for object movement, and ScanNet for other tasks.Source: §3.2 · p. 4

Image pairs sampling

Image-pair overlap is computed as the IoU of visible points and sampled between 6% and 35% with balanced overlap-ratio bins.Source: §3.2.1 · p. 4

Dynamic data generation

Camera movement labels derive from translation displacement, depth ordering, rotation, and ground-plane tilt, while object movement uses tracked point sequences and balanced motion-distance sampling.Source: §3.2.1 · pp. 4–5, §3.2.2 · p. 5

Dataset and benchmark scale

The pipeline produces over 27M QA samples from 1.1M unique images and a 7,800-sample benchmark with scene-disjoint evaluation data.Source: §3.3 · p. 5

4. Experimental Results

Multi-SpatialMLLM substantially improves multi-frame spatial understanding, generalizes beyond its training data, and preserves standard VQA ability. Experiments also show scalability, multi-task benefits, capacity-dependent learning on difficult correspondence tasks, and potential for robotic reward annotation.

Multi-Frame Spatial Understanding

36% average gain over the base model across MultiSPA tasks, with 80–90% accuracy on easier qualitative tasks.The model outperforms proprietary systems and remains stronger on challenging camera-movement-vector prediction, where baselines stay near zero.Source: §4.1 · p. 6

Generalization of Multi-SpatialMLLM

18.7% and 27.7% accuracy for the 8B and 26B models, respectively, on camera-vector prediction in unseen 3RScan scenes, versus 0% for zero-shot baselines.The evaluation uses synthetic multi-view images rendered from unseen scene meshes.Source: §4.2 · p. 7
The model retains rough parity on standard VQA benchmarks spanning perception, OCR, reasoning, and Chinese VQA.These results indicate that spatial training does not substantially reduce its original general-purpose VQA proficiency.Source: §4.2 · p. 7, Table 4 · p. 7

Multi-Task Generalization and Synergy

Additional task data raises camera-movement accuracy from 9.3% to 18.0% and object-movement accuracy from 17.5% to 22.04%.The comparisons support synergistic benefits from multi-task training rather than isolated task-specific training.Source: §4.2 · p. 7, Table 5 · p. 7

Emergence of Spatial Understanding

Only the 26B model improves over the base model after training on hard visual-correspondence samples; 8B and 13B models decline.An 8B model trained on the same number of easy samples reaches 93.33% accuracy, suggesting difficult spatial learning may require greater capacity.Source: §4.3 · p. 8, Table 6 · p. 8

Demonstrations of Multi-SpatialMLLM

In out-of-distribution robot scenes, the model identifies a static cube correctly and estimates increasing object-movement distances aligned with ground truth.The predicted distances are not exact, but the demonstration supports potential use as a multi-frame reward annotator.Source: §4.4 · p. 8, Figure 5 · p. 8

5. Conclusion

The work extends MLLM spatial understanding to multiple frames through a dedicated dataset, benchmark, and data-generation pipeline. Experiments support effectiveness, scalability, generalization, multi-task benefits, emergent spatial behaviors, and a reward-annotation application.

Multi-SpatialMLLM extends MLLM spatial understanding to multiple frames, an overlooked capability in prior research.Source: §5 · p. 8
The authors develop a data-generation pipeline, the first large-scale dataset, and a benchmark dedicated to multi-frame spatial understanding.Source: §5 · p. 8
Experiments demonstrate the model’s effectiveness, scalability, and generalization while revealing multi-task benefits and emergent behaviors on challenging spatial tasks.Source: §5 · p. 8
The model also supports new applications, including multi-frame reward annotation.Source: §5 · p. 8

A. MultiSPA Data Samples and Distributions

MultiSPA contains 26 subtasks, with examples and task-type distributions documented across the supplementary material. Its data engine is scalable and can generate additional training samples.

MultiSPA contains 26 subtasks spanning its multi-frame spatial data collection.Source: A. MultiSPA Data Samples and… · p. 9
Examples for each task are shown in Figures 7–15, while sample distributions across task types appear in Table 7.Source: A. MultiSPA Data Samples and… · p. 9
The scalable data engine can generate more training samples for these tasks.Source: A. MultiSPA Data Samples and… · p. 9
Only part of the task templates is shown because of paper length limits; the remaining templates are described as similar.Source: B. MultiSPA Data Templates · p. 9

C. Details of Source Datasets

The data pipeline draws on annotated 3D and 4D datasets and includes procedures for balanced pair sampling, camera-motion labeling, multi-image object coverage, and rigid-body motion segmentation.

ScanNet, PStudio, ADT, and TAPVid3D provide reconstructed geometry, camera poses, depth, segmentation, human poses, and temporal 3D point tracking for data generation.Source: C. Details of Source Datasets · p. 9
Image pairs are sampled by separating zero-overlap pairs and distributing quotas across bins of nonzero overlap ratios.Source: D. Image Pairs Sampling · p. 9
Camera yaw and pitch are derived from the rotated forward z-axis, determining horizontal rotation and vertical tilt directions.Source: E. Rotation Angles · p. 9
A breadth-first search identifies minimal image sets whose combined visible points satisfy object dimension coverage thresholds.Source: F. BFS-Based Minimum-Coverage-Set Search · p. 10
TAPVid3D point clouds are clustered into coherent rigid bodies using inter-point distance changes over time, with small groups filtered as noise.Source: G. Clustering-Based Rigid Body… · p. 10

H. Different Fine-Tuning Strategies

Fine-tuning results show that broader parameter updates yield larger gains from MultiSPA, with vision-encoder tuning especially helpful for fine-grained spatial prediction. The resulting models remain competitive with specialized expert systems while retaining general vision-language capabilities.

The resource-light setup fine-tunes the LLM with LoRA while freezing the vision encoder and MLP projection layer.Source: H. Different Fine-Tuning Strategies · pp. 10–11
Increasing trainable parameters yields larger gains from MultiSPA, and tuning the vision encoder particularly improves fine-grained spatial perception.Source: H. Different Fine-Tuning Strategies · p. 11
24.67% to 43.00%: vision-encoder tuning improves camera translation vector prediction accuracy.Source: H. Different Fine-Tuning Strategies · p. 11
Multi-SpatialMLLM matches VGGT and LoFTR and remains competitive with Depth-Anything while preserving broad vision-language capabilities.Source: §I · p. 11, Table 9 · p. 11

J. More on VQA Benchmarks

Additional evaluations show that Multi-SpatialMLLM generalizes to unseen spatial benchmarks while largely preserving general VLM capabilities during fine-tuning.

The model outperforms the baseline on CVBench-3D and ERQA, with particularly strong gains on ERQA’s multi-frame subset.Source: J. More on VQA Benchmarks · p. 11
The spatial benchmark evaluation is summarized in Table 10, while standard VQA evaluation is summarized in Table 11.Source: Table 10 · p. 11, Table 11 · p. 11
The evaluation extends to three additional widely used VQA benchmarks to test whether fine-tuning preserves general VLM capabilities.Source: J. More on VQA Benchmarks · p. 11
Average performance across 10 general VQA benchmarks remains stable throughout training, alleviating concerns about forgetting.Source: J. More on VQA Benchmarks · p. 11

K. Multi-Task Synergy

Multi-task training improves visual correspondence, with gains from camera movement, object perception, and depth data; depth perception contributes the largest gains in the reported analysis.

Adding camera movement, object perception, and depth perception data consistently boosts visual correspondence performance.The setup includes 1K visual-correspondence examples and 50K samples from each auxiliary task.Source: K. Multi-Task Synergy · pp. 11–12
Table 12 reports multi-task synergy for visual correspondence using camera, object, and depth training data.Source: Table 12 · p. 12
Depth perception contributes the largest gains to visual correspondence in the multi-task analysis.Source: K. Multi-Task Synergy · p. 12
Table 13 compares single-task training with configurations incorporating visual correspondence, depth, or object-perception samples.Source: Table 13 · p. 12, K. Multi-Task Synergy · p. 12

L. Generalization to More Images.

Models trained primarily on two-frame samples remain robust when intermediate frames are inserted, indicating generalization across varying frame counts in camera-direction prediction.

85% accuracy is maintained when 1–4 intermediate frames are inserted between the first and last frames for camera-direction prediction.The model predicts motion between the first and last frame while the number of intermediate frames varies.Source: L. Generalization to More Images · p. 12
Table 14 reports camera-prediction performance with different numbers of frames.Source: Table 14 · p. 12

M. Limitations

The main limitation is that the generated training data uses two-frame scenarios, although the pipeline scales to more images; extending beyond pairs and explaining emergent spatial abilities remain future work.

The generated data in the main paper uses two-frame scenarios.Source: M. Limitations · p. 12
The data-generation pipeline naturally scales to additional frames, but future work is needed to extend beyond pairs for stronger multi-view spatial reasoning.Source: M. Limitations · p. 12
Further investigation is required to clarify which spatial abilities drive the observed signs of emergent behavior.Source: M. Limitations · p. 12