Source-linked AI summary
Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models
Runsen Xu, Weiyao Wang, Hao Tang, Xingyu Chen, Xiaodong Wang, Fu-Jen Chu, Matt Feiszli, Kevin J. Liang
TL;DR
MLLMs have limited spatial understanding beyond single images, despite their intended use in physical-world applications. The paper introduces MultiSPA and Multi-SpatialMLLM to train and evaluate multi-frame spatial reasoning, achieving broad gains and showing scalability, generalization, multi-task benefits, and potential for robotic reward annotation.
Problem
Existing MLLMs struggle with spatial understanding, while prior spatial-training approaches focus on single-image scenarios rather than multi-frame reasoning.
Method
The paper builds a data pipeline and MultiSPA dataset integrating depth, visual correspondence, and dynamic perception from annotated 3D and 4D scenes, alongside a unified multi-frame benchmark.
Results
Multi-SpatialMLLM substantially outperforms base models and proprietary systems, matches specialized 3D perception models, and generalizes and scales across multi-frame spatial tasks.
Takeaways & Limitations
The model provides a scalable multi-frame perception capability with multi-task benefits, preliminary emergent spatial behavior, and potential use as a robot-learning reward annotator.
Takeaways & Limitations
The current framework focuses on image pairs; extending it to more images and explaining which spatial abilities drive emergence remain open problems.
Abstract
from arXiv · showhide
Multi-modal large language models (MLLMs) have rapidly advanced in visual tasks, yet their spatial understanding remains limited to single images, leaving them ill-suited for physical-world applications that require multi-frame reasoning. In this paper, we propose a framework to equip MLLMs with multi-frame spatial understanding by integrating fundamental spatial skills, including depth perception, visual correspondence, and dynamic perception. We design a novel data pipeline and collect the MultiSPA dataset of more than 27 million samples spanning diverse 3D and 4D scenes to enable training. Alongside MultiSPA, we introduce a comprehensive benchmark that tests a wide spectrum of spatial tasks under uniform metrics. Our resulting model, Multi-SpatialMLLM, achieves significant gains over baselines and proprietary systems, demonstrating scalable and generalizable multi-frame perception. We further observe multi-task benefits and emergent spatial capabilities in challenging scenarios, and showcase how our model can serve as a multi-frame reward annotator for robotics.
1. Introduction
The paper extends MLLM spatial understanding from single images to multi-frame reasoning by integrating depth, correspondence, and dynamic perception. It introduces MultiSPA data and benchmarking, and reports strong, scalable performance from Multi-SpatialMLLM.
- Current MLLMs struggle with basic spatial understanding and remain limited by single-image training for real-world applications.
- The framework integrates depth perception, visual correspondence, and dynamic perception to support reasoning across multiple images.
- The data pipeline uses annotated 3D and 4D datasets, balanced image-pair sampling, spatial and temporal alignment, and LLM-generated question–answer templates.
- MultiSPA contains more than 27 million samples, while its benchmark evaluates diverse multi-frame spatial tasks and output formats under a unified metric.
- Multi-SpatialMLLM substantially outperforms base models and proprietary systems, matches specialized 3D perception models, and generalizes and scales without degrading general abilities.
- Multi-task training provides notable benefits, reveals preliminary emergent spatial behavior, and supports multi-frame reward annotation for robot learning.
2. Related Work
Prior MLLM research addresses spatial understanding through benchmarks and spatial fine-tuning, but much of the literature remains focused on limited image settings and task formats.
- MLLMs combine image encoders with language models and are trained with language-modeling objectives, but web-scale image–text data provide insufficient spatial annotations.
- Existing benchmarks include single-image spatial tasks, image-pair reasoning, and video-based scene-level reasoning with overlapping tasks such as camera movement estimation and keypoint matching.
- MultiSPA adds object movement perception and supports more diverse input and output formats than related benchmarks described here.
3. MultiSPA Dataset and Benchmark
MultiSPA combines annotated 3D and 4D scenes with a data-generation pipeline for diverse multi-frame spatial QA. Its benchmark covers five spatial task families and uses held-out scene data for evaluation.
- Tasks definitions: The dataset targets depth, visual correspondence, camera movement, object movement, and object size perception across multiple images.
- Tasks definitions: Referencing uses pixel coordinates or visual annotations, while outputs include qualitative answers, scalar quantities, pixel coordinates, and displacement vectors.
- Data Format: The data are formatted as image-conditioned question–answer pairs with GPT-4o-generated templates, millimeter rounding, and normalized pixel coordinates.
- Source datasets: The pipeline uses ADT and Panoptic Studio for 4D data, TAPVid3D tracking annotations for object movement, and ScanNet for other tasks.
- Image pairs sampling: Image-pair overlap is computed as the IoU of visible points and sampled between 6% and 35% with balanced overlap-ratio bins.
- Dynamic data generation: Camera movement labels derive from translation displacement, depth ordering, rotation, and ground-plane tilt, while object movement uses tracked point sequences and balanced motion-distance sampling.
- Dataset and benchmark scale: The pipeline produces over 27M QA samples from 1.1M unique images and a 7,800-sample benchmark with scene-disjoint evaluation data.
4. Experimental Results
Multi-SpatialMLLM substantially improves multi-frame spatial understanding, generalizes beyond its training data, and preserves standard VQA ability. Experiments also show scalability, multi-task benefits, capacity-dependent learning on difficult correspondence tasks, and potential for robotic reward annotation.
- Multi-Frame Spatial Understanding: 36% average gain over the base model across MultiSPA tasks, with 80–90% accuracy on easier qualitative tasks.The model outperforms proprietary systems and remains stronger on challenging camera-movement-vector prediction, where baselines stay near zero.
- Generalization of Multi-SpatialMLLM: 18.7% and 27.7% accuracy for the 8B and 26B models, respectively, on camera-vector prediction in unseen 3RScan scenes, versus 0% for zero-shot baselines.The evaluation uses synthetic multi-view images rendered from unseen scene meshes.
- Generalization of Multi-SpatialMLLM: The model retains rough parity on standard VQA benchmarks spanning perception, OCR, reasoning, and Chinese VQA.These results indicate that spatial training does not substantially reduce its original general-purpose VQA proficiency.
- Multi-Task Generalization and Synergy: Additional task data raises camera-movement accuracy from 9.3% to 18.0% and object-movement accuracy from 17.5% to 22.04%.The comparisons support synergistic benefits from multi-task training rather than isolated task-specific training.
- Emergence of Spatial Understanding: Only the 26B model improves over the base model after training on hard visual-correspondence samples; 8B and 13B models decline.An 8B model trained on the same number of easy samples reaches 93.33% accuracy, suggesting difficult spatial learning may require greater capacity.
- Demonstrations of Multi-SpatialMLLM: In out-of-distribution robot scenes, the model identifies a static cube correctly and estimates increasing object-movement distances aligned with ground truth.The predicted distances are not exact, but the demonstration supports potential use as a multi-frame reward annotator.
5. Conclusion
The work extends MLLM spatial understanding to multiple frames through a dedicated dataset, benchmark, and data-generation pipeline. Experiments support effectiveness, scalability, generalization, multi-task benefits, emergent spatial behaviors, and a reward-annotation application.
- Multi-SpatialMLLM extends MLLM spatial understanding to multiple frames, an overlooked capability in prior research.
- The authors develop a data-generation pipeline, the first large-scale dataset, and a benchmark dedicated to multi-frame spatial understanding.
- Experiments demonstrate the model’s effectiveness, scalability, and generalization while revealing multi-task benefits and emergent behaviors on challenging spatial tasks.
- The model also supports new applications, including multi-frame reward annotation.
A. MultiSPA Data Samples and Distributions
MultiSPA contains 26 subtasks, with examples and task-type distributions documented across the supplementary material. Its data engine is scalable and can generate additional training samples.
- MultiSPA contains 26 subtasks spanning its multi-frame spatial data collection.
- Examples for each task are shown in Figures 7–15, while sample distributions across task types appear in Table 7.
- The scalable data engine can generate more training samples for these tasks.
- Only part of the task templates is shown because of paper length limits; the remaining templates are described as similar.
C. Details of Source Datasets
The data pipeline draws on annotated 3D and 4D datasets and includes procedures for balanced pair sampling, camera-motion labeling, multi-image object coverage, and rigid-body motion segmentation.
- ScanNet, PStudio, ADT, and TAPVid3D provide reconstructed geometry, camera poses, depth, segmentation, human poses, and temporal 3D point tracking for data generation.
- Image pairs are sampled by separating zero-overlap pairs and distributing quotas across bins of nonzero overlap ratios.
- Camera yaw and pitch are derived from the rotated forward z-axis, determining horizontal rotation and vertical tilt directions.
- A breadth-first search identifies minimal image sets whose combined visible points satisfy object dimension coverage thresholds.
- TAPVid3D point clouds are clustered into coherent rigid bodies using inter-point distance changes over time, with small groups filtered as noise.
H. Different Fine-Tuning Strategies
Fine-tuning results show that broader parameter updates yield larger gains from MultiSPA, with vision-encoder tuning especially helpful for fine-grained spatial prediction. The resulting models remain competitive with specialized expert systems while retaining general vision-language capabilities.
- The resource-light setup fine-tunes the LLM with LoRA while freezing the vision encoder and MLP projection layer.
- Increasing trainable parameters yields larger gains from MultiSPA, and tuning the vision encoder particularly improves fine-grained spatial perception.
- 24.67% to 43.00%: vision-encoder tuning improves camera translation vector prediction accuracy.
- Multi-SpatialMLLM matches VGGT and LoFTR and remains competitive with Depth-Anything while preserving broad vision-language capabilities.
J. More on VQA Benchmarks
Additional evaluations show that Multi-SpatialMLLM generalizes to unseen spatial benchmarks while largely preserving general VLM capabilities during fine-tuning.
- The model outperforms the baseline on CVBench-3D and ERQA, with particularly strong gains on ERQA’s multi-frame subset.
- The spatial benchmark evaluation is summarized in Table 10, while standard VQA evaluation is summarized in Table 11.
- The evaluation extends to three additional widely used VQA benchmarks to test whether fine-tuning preserves general VLM capabilities.
- Average performance across 10 general VQA benchmarks remains stable throughout training, alleviating concerns about forgetting.
K. Multi-Task Synergy
Multi-task training improves visual correspondence, with gains from camera movement, object perception, and depth data; depth perception contributes the largest gains in the reported analysis.
- Adding camera movement, object perception, and depth perception data consistently boosts visual correspondence performance.The setup includes 1K visual-correspondence examples and 50K samples from each auxiliary task.
- Table 12 reports multi-task synergy for visual correspondence using camera, object, and depth training data.
- Depth perception contributes the largest gains to visual correspondence in the multi-task analysis.
- Table 13 compares single-task training with configurations incorporating visual correspondence, depth, or object-perception samples.
L. Generalization to More Images.
Models trained primarily on two-frame samples remain robust when intermediate frames are inserted, indicating generalization across varying frame counts in camera-direction prediction.
- 85% accuracy is maintained when 1–4 intermediate frames are inserted between the first and last frames for camera-direction prediction.The model predicts motion between the first and last frame while the number of intermediate frames varies.
- Table 14 reports camera-prediction performance with different numbers of frames.
M. Limitations
The main limitation is that the generated training data uses two-frame scenarios, although the pipeline scales to more images; extending beyond pairs and explaining emergent spatial abilities remain future work.
- The generated data in the main paper uses two-frame scenarios.
- The data-generation pipeline naturally scales to additional frames, but future work is needed to extend beyond pairs for stronger multi-view spatial reasoning.
- Further investigation is required to clarify which spatial abilities drive the observed signs of emergent behavior.