PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence

Diankun Wu, Fangfu Liu, Yi-Hsin Hung, Yueqi Duan

arXiv:2505.23747v2cs.CVcs.AIcs.LG

TL;DR

Visual-based 3D spatial intelligence remains difficult when only monocular video is available. Spatial-MLLM combines semantic and structure-aware visual encoders with spatial frame sampling, achieving state-of-the-art results across multiple spatial understanding benchmarks.

  • Problem

    Visual-based 3D spatial intelligence from monocular video remains challenging because each frame provides only a partial scene observation and standard encoders lack fine-grained structural cues.

  • Method

    Spatial-MLLM combines a 2D semantic encoder, a geometry-initialized spatial encoder, connector-based fusion, space-aware frame sampling, and supervised training on spatial question-answering data.

  • Results

    Spatial-MLLM achieves state-of-the-art performance across multiple visual-based spatial understanding and reasoning benchmarks.

  • Takeaways & Limitations

    The model demonstrates spatial understanding and reasoning from purely 2D visual inputs without requiring additional 3D or 2.5D input.

  • Takeaways & Limitations

    The evaluation focuses on visual-based spatial intelligence, leaving the benefits of spatial structural information for general video understanding and reasoning unexplored.

Abstract

from arXiv · show

Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced performance on 2D visual tasks. However, improving their spatial intelligence remains a challenge. Existing 3D MLLMs always rely on additional 3D or 2.5D data to incorporate spatial awareness, restricting their utility in scenarios with only 2D inputs, such as images or videos. In this paper, we present Spatial-MLLM, a novel framework for visual-based spatial reasoning from purely 2D observations. Unlike conventional video MLLMs which rely on CLIP-based visual encoders optimized for semantic understanding, our key insight is to unleash the strong structure prior from the feed-forward visual geometry foundation model. Specifically, we propose a dual-encoder architecture: a pretrained 2D visual encoder to extract semantic features, and a 3D spatial encoder-initialized from the backbone of the visual geometry model-to extract 3D structure features. A connector then integrates both features into unified visual tokens for enhanced spatial understanding. Furthermore, we propose a space-aware frame sampling strategy at inference time, which selects the spatially informative frames of a video sequence, ensuring that even under limited token length, the model focuses on frames critical for spatial reasoning. Beyond architecture improvements, we construct a training dataset from multiple sources and train the model on it using supervised fine-tuning and GRPO. Extensive experiments on various real-world datasets demonstrate that Spatial-MLLM achieves state-of-the-art performance in a wide range of visual-based spatial understanding and reasoning tasks. Project page: https://diankun-wu.github.io/Spatial-MLLM/.

1 Introduction

Spatial-MLLM targets visual-based 3D spatial intelligence from only 2D inputs, addressing the limitations of approaches that depend on additional 3D or 2.5D data. It combines semantic and structural visual features with space-aware frame sampling and achieves state-of-the-art performance across visual-based spatial understanding and reasoning tasks.

  • Motivation and contribution: Spatial-MLLM improves visual-based 3D spatial intelligence without requiring 3D or 2.5D inputs.This targets real-world settings where only 2D visual inputs are available.
  • Architecture: A dual-encoder and connector integrate semantic features from a standard 2D visual encoder with structural features from a spatial encoder initialized from a feed-forward visual geometry foundation model.The geometry model supplies a strong structure prior that complements general-purpose video MLLM visual encoders.
  • Inference strategy: A space-aware frame-sampling strategy selects spatially informative video frames when input length is constrained, such as by VRAM limitations.The strategy exploits information from the feed-forward visual geometry model to improve performance under limited input length.
  • Results: Spatial-MLLM achieves state-of-the-art performance across ScanQA, SQA3D, and other visual-based spatial understanding and reasoning benchmarks.The model is evaluated on a wide range of visual-based spatial understanding and reasoning tasks.

2 Related Work

Related work progresses from general vision-language fusion toward 3D scene understanding, while visual-based 3D spatial reasoning from purely 2D inputs and its evaluation remain comparatively underexplored. New benchmarks increasingly assess spatial, kinematic, egocentric, and spatiotemporal reasoning in video MLLMs.

  • 2D Multimodal Large Language Models: MLLMs bridge vision and language through token-level fusion, feature-level fusion, or MLP projection into the language space.BLIP-2 and Flamingo use token- and feature-level fusion, while LLaVA, MiniGPT-4 [41], and subsequent models [42] [43] use MLPs.
  • 3D Scene Understanding: 3D MLLMs extend these models with point-cloud features, 3D detectors, object-centric representations, or other scene-level 3D inputs.The passage identifies LL3DA, Grounded 3D-LLM [45], Chat3D [46], LEO [47], Chat-Scene, 3D-LLM, and Scene-LLM [49] as representative approaches.
  • Visual-based 3D Spatial Intelligence: Visual-based 3D spatial intelligence seeks to infer 3D structures and spatial relationships from purely 2D visual inputs, but this direction remains relatively underexplored.Existing video MLLMs primarily target 2D understanding, with visual-based 3D reasoning studied through tasks such as 3D question answering [6] [53] and robotic manipulation.
  • Visual-based 3D Spatial Intelligence: New benchmarks broaden evaluation from spatial reasoning to physics, egocentric intelligence, motion dynamics, and 4D spatiotemporal interactions.Physics-aware questions include velocity estimation; Ego-ST Bench [55] evaluates egocentric spatial intelligence, and VLM4D evaluates trajectory prediction and motion dynamics.

3 Method

Spatial-MLLM combines semantic 2D features with dense 3D structural features through a dual-encoder architecture and connector for spatial understanding and reasoning. It also selects spatially informative video frames and uses a constructed spatial QA dataset with supervised fine-tuning followed by GRPO training.

  • 3.1 Architecture: Spatial-MLLM uses a dual-encoder architecture with a Qwen2.5-VL-based 2D encoder, a VGGT-initialized 3D spatial encoder, a connector, and an LLM backbone.The 2D branch extracts semantically rich features, while the spatial branch aggregates information across frames using frame-wise and global self-attention.
  • 3.1 Architecture: The connector aligns 3D features with 2D features across spatial and temporal dimensions, then fuses them into unified visual tokens using a lightweight MLP-based design.Only the dense 3D feature is used for fusion because it captures the input frames’ structural information.
  • 3.2 Space-Aware Frame Sampling: Space-aware frame sampling selects Nk spatially informative frames from long scene videos by maximizing covered voxels, addressing inputs where only 8–32 frames represent videos exceeding 2,000 frames.The method uniformly subsamples Nm frames, decodes camera and depth information with VGGT heads, and uses greedy maximum coverage; in practice, Nm = 128 and Nk = 16.
  • 3.3 Training: The training dataset contains approximately 120k visual-based 3D spatial QA pairs from ScanQA, SQA3D, and additional self-created data based on ScanNet training scenes.Each item is represented by a question, answer, video ID, and metadata.
  • 3.3 Training: Training first applies supervised fine-tuning with both visual encoders frozen while the connector and LLM backbone learn to fuse 2D and 3D features, followed by cold start and GRPO for long-CoT spatial reasoning.GRPO samples multiple outputs per question and uses formatting plus task-dependent rewards.

4 Experiments

Spatial-MLLM substantially outperforms proprietary and open-source video MLLMs on VSI-Bench despite using 4.9B parameters and only 16 frames, and surpasses video-input and task-specific models on ScanQA and SQA3D. Ablations show that space-aware sampling improves spatial coverage and performance, while the proposed architecture and dataset enhance spatial reasoning.

  • VSI-Bench Results: Despite 4.9B parameters and only 16 input frames, Spatial-MLLM significantly outperforms all proprietary and open-source MLLMs on VSI-Bench, including substantially larger models.Gemini-1.5 Pro is the strongest remaining model, while sampling videos at 1 FPS rather than using Spatial-MLLM’s 16 frames.
  • ScanQA and SQA3D Results: Spatial-MLLM significantly outperforms all video-input and task-specific models across every reported metric on both ScanQA and SQA3D.Evaluation uses the ScanQA validation set and SQA3D test set following common practice.
  • Effectiveness of Space-aware Frame Sampling: Space-aware frame sampling consistently outperforms uniform sampling at equal frame counts, while increasing sampled frames improves performance for both strategies.The comparison covers 8, 16, and 32 frames on VSI-Bench.
  • Effectiveness of Space-aware Frame Sampling: Space-aware sampling selects frames with greater spatial coverage, whereas uniform sampling can miss transient regions and produce redundant viewpoints during static-camera periods.The visualization uses point maps predicted by VGGT.

5 Conclusion

Spatial-MLLM enables spatial understanding and reasoning from purely 2D visual inputs by combining semantic and structure-aware encoders with space-aware frame sampling. The model improves over previous video MLLMs, while future work includes scaling model size and training data and extending evaluation beyond visual-based spatial intelligence.

  • Conclusion: Spatial-MLLM enables spatial understanding and reasoning from purely 2D visual inputs through a dual-encoder design combining semantic and spatial cues.The spatial encoder is initialized from a visual geometry foundation model.
  • Conclusion: Space-aware frame sampling further enhances performance under limited input constraints.
  • Limitations and Future Work: The model improves over previous video MLLMs across a wide range of visual-based spatial understanding and reasoning tasks.
  • Limitations and Future Work: Future work could scale Spatial-MLLM in model size and training data and extend its scope beyond visual-based spatial intelligence.The current model is trained and evaluated specifically on relevant visual-based datasets and benchmarks.

Technical Appendices and Supplementary Material · A Broader Impacts

Spatial-MLLM enables 3D spatial reasoning from purely 2D visual inputs, potentially broadening access to spatially aware AI without costly 3D data. Its deployment should nevertheless address data privacy and ethical considerations.

  • A Broader Impacts: Spatial-MLLM enables 3D understanding purely from 2D visual inputs.This capability advances spatial reasoning in multimodal large language models.
  • A Broader Impacts: The approach may broaden access to spatially aware AI without requiring costly 3D data.
  • A Broader Impacts: Potential application domains include robotics, autonomous systems, and visual content understanding.
  • A Broader Impacts: The capability could support spatially aware AI in robotics.
  • A Broader Impacts: The capability could support spatially aware AI in autonomous systems.
  • A Broader Impacts: The capability could support spatially aware AI for visual content understanding.
  • A Broader Impacts: Data privacy and ethical deployment remain important considerations for ensuring positive social outcomes.These considerations apply, as with other vision-language models.

B Additional Method Details · B.1 Details of Space-Aware Frame Sampling · B.2 Details of Feature Fusion

The method adds space-aware frame sampling by reconstructing scene geometry, voxelizing 3D coverage, and greedily selecting informative frames, while feature fusion aligns the 2D and 3D encoder tokens through matched spatial-temporal merging. The sampling procedure uses VGGT-derived geometry and selects up to 16 frames from 128 candidates.

  • B.1 Details of Space-Aware Frame Sampling: Space-aware frame sampling comprises scene-geometry preprocessing, voxelization and coverage calculation, and greedy maximum-coverage selection.The procedure is designed to identify spatially informative video frames through three sequential stages.
  • B.1 Details of Space-Aware Frame Sampling: The sampler uniformly subsamples each video to 128 candidate frames, then uses VGGT to compute geometry features and reconstruct 3D point maps with depth reprojection.Each reconstructed point also receives a depth-head confidence value; depth and camera reconstruction is reported as more accurate than directly decoding point maps.
  • B.1 Details of Space-Aware Frame Sampling: The sampler builds a 3D bounding box around valid scene points, discretizes it into voxels, and sets voxel size adaptively as 1/20 of the box’s minimum dimension.The scale factor is controlled by λ, which is set to 20, and each frame’s coverage is computed from its valid points.
  • B.2 Details of Feature Fusion: Both encoders use spatial patch size 14, while the 2D encoder merges adjacent spatial tokens and every two consecutive-frame tokens before fusion.This merging reduces the 2D encoder sequence to exactly one-eighth the 3D encoder’s token count, excluding register and camera tokens.
  • B.2 Details of Feature Fusion: The greedy algorithm initializes selected frames, covered voxels, and remaining candidates, then iteratively adds the candidate with maximum coverage gain and updates these sets.Selection terminates when the candidate pool is exhausted or no remaining frame adds new voxel coverage.
  • B.2 Details of Feature Fusion: After each greedy choice, the algorithm unions the selected frame’s voxels into the covered set, removes that frame from the candidates, and returns the selected indices.These updates implement the iterative coverage maximization procedure described for space-aware sampling.

B.3 Details of Dataset Construction · B.4 Details of Cold Start

The dataset is constructed from ScanNet videos through preprocessing, spatial-semantic metadata computation, and diverse spatial QA generation. Before GRPO, a 200-step cold start uses filtered multi-path reasoning generations to produce 2,459 training items.

  • B.3 Details of Dataset Construction: The dataset construction follows three stages: video preprocessing, metadata computation, and QA pair generation, following a similar approach to.
  • B.3 Details of Dataset Construction: Raw ScanNet scans are converted into 24-FPS videos with 640 × 480 resolution, while scene geometry is aligned and converted into Open3D point clouds.
  • B.3 Details of Dataset Construction: Metadata computation derives room dimensions and centers plus object-level oriented bounding boxes and semantic information from aligned scans and annotations.
  • B.3 Details of Dataset Construction: Generated QA tasks cover object counting, object and room size, absolute and relative distance, appearance order, and relative direction.
  • B.3 Details of Dataset Construction: Relative-direction questions compute horizontal angles from position, facing, and query triples, then discretize them into difficulty-dependent directional classes.
  • B.4 Details of Cold Start: Before GRPO, the model undergoes a 200-step cold start using a spatial reasoning dataset with chain-of-thought annotations.
  • B.4 Details of Cold Start: For each training item, Qwen2.5-VL-72B generates K independent reasoning-answer pairs, which are scored and adaptively filtered by question type rather than a global threshold.
  • B.4 Details of Cold Start: 2459 items remain in the cold start set from Ns = 5000 items with K = 3, preserving approximately the top 50% of generations per question type while discarding zero-reward outputs.

B.5 Details of SFT and GRPO Training

SFT and GRPO use format and task-specific rewards, supplemented by a reasoning-length reward, with stage-specific prompts based on Qwen2.5-VL. Task rewards vary by answer type, including exact match, numerical accuracy, and normalized verbal-answer similarity.

  • Reward Calculation: Task-specific rewards use exact match for multiple-choice questions, mean relative accuracy for numerical tasks, and Levenshtein-ratio similarity for verbal answers.Numerical scoring normalizes values, uses ϵ = 10−8 to avoid division by zero, and evaluates thresholds T = {0.50, 0.55, ..., 0.95}; verbal answers use normalized text and the Levenshtein library.
  • Other Details: Both stages use Qwen2.5-VL’s default system prompt, while SFT adds a question and type template and GRPO adds a question post string and type template.Experiments run on Intel(R) Xeon(R) Gold 6430 systems with 80G NVIDIA A800 GPUs.

C Additional Experiments · C.1 Additional Results on VSI-Bench

Figures 7–10 provide qualitative VSI-Bench examples showing that Spatial-MLLM reasons across task types, produces final answers, and uses self-verification and task decomposition.

  • C.1 Additional Results on VSI-Bench: Spatial-MLLM is evaluated qualitatively on the VSI-Bench dataset in Figures 7–10.
  • C.1 Additional Results on VSI-Bench: The examples cover different VSI-Bench task types.
  • C.1 Additional Results on VSI-Bench: Spatial-MLLM reasons with visual information across these tasks.
  • C.1 Additional Results on VSI-Bench: The model produces final answers from its visual reasoning.
  • C Additional Experiments: Spatial-MLLM demonstrates self-verification during its reasoning process.
  • C Additional Experiments: Spatial-MLLM demonstrates task decomposition during its reasoning process.

C.2 Additional Results on ScanQA and SQA3D

Spatial-MLLM consistently outperforms video-input models on ScanQA and SQA3D, including models trained with spatial reasoning data. Despite having 4.9 billion parameters, it also surpasses Qwen2.5-VL-72B across both benchmarks and multiple metrics.

  • C.2 Additional Results on ScanQA and SQA3D: Spatial-MLLM consistently outperforms all video-input models on ScanQA and SQA3D, including LLaVA-Video-7B and Oryx-34B [53].These comparison models incorporate spatial reasoning datasets such as ScanQA during training.
  • C.2 Additional Results on ScanQA and SQA3D: +2.3 EM-1, +17.6 BLEU-1, and +24.9 CIDEr: Spatial-MLLM surpasses Qwen2.5-VL-72B on ScanQA despite using only 4.9 billion parameters.The gains span multiple reported ScanQA metrics.
  • C.2 Additional Results on ScanQA and SQA3D: +4.2 EM-1 and +7.8 EM-R1 overall: Spatial-MLLM outperforms Qwen2.5-VL-72B across all SQA3D question types, including Is (+15.3) and Which (+13.9).The reported SQA3D improvements include average performance across all questions and question-type categories.
Loading 2505.23747v2…