Visual Spatial Tuning

Rui Yang; Ziyu Zhu; Yanwei Li; Jingjia Huang; Shen Yan; Siyuan Zhou; Zhe Liu; Xiangtai Li; Shuangye Li; Wenqian Wang; Yi Lin; Hengshuang Zhao

Source-linked AI summary

Visual Spatial Tuning

Rui Yang, Ziyu Zhu, Yanwei Li, Jingjia Huang, Shen Yan, Siyuan Zhou, Zhe Liu, Xiangtai Li, Shuangye Li, Wenqian Wang, Yi Lin, Hengshuang Zhao

arXiv:2511.05491v1Nov 7, 2025cs.CV

TL;DR

VLMs have limited spatial understanding, while extra spatial encoders add complexity and can harm general capabilities. VST addresses this gap with perception and reasoning datasets trained progressively, achieving state-of-the-art spatial performance while preserving general capabilities and improving VLA tasks.

Problem
VLMs have limited ability to capture spatial relationships from sequential visual observations, while added expert encoders can introduce complexity and negatively affect general capabilities.
Method
VST combines VST-P perception data and VST-R reasoning data with supervised fine-tuning followed by reinforcement learning.
Results
VST achieves state-of-the-art performance across multiple spatial benchmarks while preserving general multimodal capabilities.
Takeaways & Limitations
VST enhances visuospatial skills in VLA models and supports more grounded interaction with the physical world.
Takeaways & Limitations
The method uses Qwen2.5-VL as its base model because of its existing visual recognition capabilities.

Abstract

from arXiv · show

Capturing spatial relationships from visual inputs is a cornerstone of human-like general intelligence. Several previous studies have tried to enhance the spatial awareness of Vision-Language Models (VLMs) by adding extra expert encoders, which brings extra overhead and usually harms general capabilities. To enhance the spatial ability in general architectures, we introduce Visual Spatial Tuning (VST), a comprehensive framework to cultivate VLMs with human-like visuospatial abilities, from spatial perception to reasoning. We first attempt to enhance spatial perception in VLMs by constructing a large-scale dataset termed VST-P, which comprises 4.1 million samples spanning 19 skills across single views, multiple images, and videos. Then, we present VST-R, a curated dataset with 135K samples that instruct models to reason in space. In particular, we adopt a progressive training pipeline: supervised fine-tuning to build foundational spatial knowledge, followed by reinforcement learning to further improve spatial reasoning abilities. Without the side-effect to general capabilities, the proposed VST consistently achieves state-of-the-art results on several spatial benchmarks, including $34.8\%$ on MMSI-Bench and $61.2\%$ on VSIBench. It turns out that the Vision-Language-Action models can be significantly enhanced with the proposed spatial tuning paradigm, paving the way for more physically grounded AI.

1 Introduction

VST addresses VLMs’ limited spatial understanding with a comprehensive framework spanning spatial perception and reasoning. It combines diverse datasets with progressive training to improve spatial capabilities without degrading general capabilities.

Motivation: VLMs struggle to capture spatial relationships from sequential visual observations, limiting physical-world interaction and applications.Additional expert encoders can add complexity and negatively impact general capabilities.
Framework: VST targets limitations in prior work by covering diverse visual inputs and training stages within one integrated framework.The framework is designed to cultivate human-like visuospatial abilities holistically.
Framework: VST separates spatial ability into spatial perception and spatial reasoning.Perception concerns discerning object relationships, while reasoning involves building and mentally manipulating an internal environmental model.
Datasets: VST-P contains 4.1 million samples across 19 tasks, using single-image, multi-image, and video data to develop spatial perception.These inputs target relationships beyond pixel space, multiple viewpoints, and spatiotemporal structure.
Datasets: VST-R uses chain-of-thought and rule-checkable samples to strengthen spatial reasoning, especially in multi-image scenarios.The data supports reasoning processes and verifiable answers.
Training: The progressive pipeline uses supervised fine-tuning for foundational spatial knowledge and reinforcement learning for stronger spatial reasoning.This sequencing establishes perception before higher-level reasoning without requiring a specialized 3D-inductive-bias encoder.

CoT (77.8%)

The VST datasets distribute spatial and general reasoning data across multiple task categories. Reported results include strong performance on spatial benchmarks while preserving general multimodal capabilities.

Dataset distribution: VST-P and VST-R distributions distinguish spatial reasoning from general reasoning across their respective dataset components.The captions label SR as spatial reasoning and GR as general reasoning.
Results: VST reaches 87.8% on CVBench, 34.8% on MMSI-Bench, and 61.2% on VSIBench.These are reported as performance levels on multiple spatial benchmarks.
Results: VST preserves general multimodal capabilities while achieving strong spatial-benchmark performance.The reported benchmark results are accompanied by preservation of general capabilities.

2 Dataset

The VST dataset is organized into a large perception component and a smaller reasoning component. VST-P covers varied visual scenarios and 3D-oriented tasks to build spatial knowledge from common visual inputs.

VST-P: VST-P contains 4.1 million samples across 19 tasks for supervised fine-tuning.Its scenarios are single-image, multi-image, and video.
VST-P: Single-image, multi-image, and video data comprise 64.8%, 33.1%, and 2.1% of VST-P, respectively.Single-image data is the largest category.
Single-image data: Single-image tasks include relative depth estimation, 3D object detection, and distance estimation.These tasks connect 2D pixel coordinates with the 3D physical world.
Single-image data: The single-image data is collected through dedicated engines using depth maps and 3D bounding-box annotations.The 3D data engine also incorporates several open-source datasets.

RGB-D Scan

VST builds spatial understanding from single images, multiple viewpoints, and videos using dedicated data engines and diverse spatial tasks. Its perception and reasoning datasets improve performance across several spatial benchmarks.

Data processing: FoV unification projects images onto a virtual camera with a uniform field of view, reducing camera-intrinsic inconsistencies in 3D detection.Multi-turn instructions also let later boxes reference earlier ones to teach layout information.
Data processing: Scene captions describe layout and spatial arrangements alongside object content, reducing reliance on template-based numerical detection data.A large VLM generates captions from ground-truth 3D boxes and scene-graph relationships.
Multi-image: Multi-image data targets cross-view detection, correspondence, object relationships, and camera motion using RGB-D scans from ScanNet, ScanNet++, and ARKitScenes.These tasks are designed to improve spatial understanding across viewpoints.
Video: Video data captures spatiotemporal relationships through appearance-order and object-counting tasks built from the multi-image data engine.Video samples add each object’s appearance time.
Results: ∼20% improvement on CVBench-3D, ∼5% on BLINK, and ∼16% on VSIBench follow the introduction of VST-P.These gains are reported for fundamental spatial-relationship comprehension.
Reasoning data: VST-R contains 135K samples combining chain-of-thought reasoning with rule-checkable data, primarily for reconstructing scenes and inferring relations across images.The rule-checkable portion supports online reinforcement learning.
Results: 8.9% improvement on MMSI-Bench follows the introduction of VST-R, demonstrating enhanced spatial reasoning.

3 Method

VST continues a general VLM with spatial perception and reasoning through staged training, then adapts the spatial-enhanced model for action prediction in VLA tasks. The training combines supervised learning, chain-of-thought instruction, and reinforcement learning.

Model: Qwen2.5-VL is used as the base model in a ViT-MLP-LLM architecture for adding spatial capabilities.The model already identifies objects and locates them in pixel space.
Training Strategy: Stage 1 uses VST-P supervised fine-tuning plus general multimodal data to build foundational spatial perception while preserving base capabilities.The objective is standard autoregressive text prediction conditioned on visual tokens.
Training Strategy: Stage 2 uses chain-of-thought data to teach spatial reasoning patterns, producing the initial reinforcement-learning actor.General reasoning data is included to retain reasoning ability on general tasks.
Training Strategy: Stage 3 applies GRPO reinforcement learning with verification data spanning spatial understanding, 3D detection, and general multimodal understanding.The reward combines response accuracy with adherence to a specified format.
Training Strategy: For 3D detection, the accuracy reward combines 3D IoU and F1 through a linear combination, with α defaulting to 0.5.The model outperforms the cold-start model after this stage.
VLA adaptation: The spatial-enhanced VLM is adapted into a VLA model that autoregressively predicts tokenized actions from an observation image and natural-language instruction.The action space is discretized into 256 bins, each represented by a special tokenizer token.

4 Experiment

Experiments show that VST improves spatial and general benchmark performance through diverse data, progressive SFT-to-RL training, and targeted design choices, without requiring specialized 3D encoders.

Main Results: VST-7B-RL achieves the best overall average among comparable VLMs on spatial benchmarks, reaching 61.2 without a specialized 3D encoder.It approaches Gemini-2.5-Pro’s 36.9% on MMSI-Bench with 34.8%.
Main Results: VST-7B-RL reaches 44.2 AP@15 on SUN RGB-D, ranking first among general VLMs and expert methods.The model surpasses both general and specialized systems despite using no auxiliary 3D encoder.
Ablation Study: Single-image auxiliary tasks improve spatial benchmarks, but their gains remain limited on MMSI-Bench, which requires more advanced spatial reasoning.Depth data yields a 10.4% gain on CVBench-3D and a 3.2% gain on VSIBench, while single-image data fails to significantly improve MMSI-Bench.
Ablation Study: Multi-image data improves cross-view spatial understanding, including a 2.1% MMSI-Bench gain from multi-image object detection.Rule-based data further improves BLINK, MMSI-Bench, and VSI-Bench.
Ablation Study: VST-video data improves VSI-Bench by 16.6%, whereas general video data decreases the score by 0.3%.The model maintains performance on other benchmarks after introducing VST-video data.
Scaling Analysis: Scaling model size improves spatial benchmark averages, but 3D object detection gains saturate at 7B parameters.Increasing from 3B to 7B improves SUN RGB-D by 4.2 AP, while 7B-to-32B scaling shows no positive correlation.
Scaling Analysis: Tripling the dataset improves all model sizes, with spatial-benchmark gains of 1.1%, 1.5%, and 1.7% for 3B, 7B, and 32B models.The scaling pattern favors larger models on spatial benchmarks and smaller models on 3D object detection.
Reasoning Training: Reinforcement learning raises MMSI-Bench performance to 35.3% with CoT inference after cold-start training.Before RL, inference without CoT reaches 33.6%, outperforming the CoT setting by 1.9%.

5 Related Work

Related work shows strong progress in general vision-language modeling but persistent spatial challenges, motivating datasets and methods dedicated to spatial understanding and reasoning.

Large Vision-Language Models: Large vision-language models integrate text, images, and video using a common vision-encoder, projector, and language-model architecture.Prior improvements include stronger vision encoders and higher input resolution.
Spatial Understanding and Reasoning: Spatial benchmarks continue to expose challenges in spatial understanding and reasoning despite progress on visual tasks.Prior work such as SpatialVLM and SpatialRGPT addresses these challenges with specialized spatial datasets and RGB-D extensions.

6 Conclusion

VST is presented as a general, scalable framework for developing human-like spatial perception and reasoning in vision-language models. It preserves general capabilities, reaches strong spatial benchmark performance, and improves visuospatial skills in vision-language-action models.

VST combines large-scale perception data VST-P with curated reasoning instructions VST-R to cultivate spatial awareness in vision-language models.The framework is designed to support both spatial perception and reasoning.
VST acquires spatial awareness without degrading general capabilities.
VST enhances visuospatial skills in Vision-Language-Action models, supporting more grounded interaction with the physical world.

A More Implementation Details

The implementation details define coordinate conventions, prompt formatting, field-of-view unification, training-stage configurations, and the evaluation protocol for VLA tasks. The appendix also specifies the simulator and summarizes training parameters across stages.

Coordinate conventions: The camera coordinate system places the origin at the camera center, with X right, Y downward, and Z forward along the optical axis.
Coordinate conventions: A 3D bounding box is represented by center coordinates, dimensions, and orientation under a convention where X is front–back, Y vertical, and Z lateral.
FoV unification: FoV unification resizes input images so images used for 3D object detection share a predefined unified focal length.This is intended to mitigate challenges from differing camera parameters in mixed-dataset training.
Training configuration: The appendix summarizes training parameters separately for supervised fine-tuning, CoT cold start, and reinforcement learning stages.
VLA evaluation: For LIBERO evaluation, unsuccessful demonstrations and no-operation actions are filtered, inputs are resized to 256 × 256 pixels, and each task uses 50 independent rollouts.Performance is reported as the average success rate across rollouts.
Simulation: Isaac Sim-4.2 is used to generate 3D bounding boxes for scenes.

B More Results

Scaling foundational spatial training data progressively improves the baseline VLM’s spatial perception performance on 3D datasets. The experiment uses monocular 3D object detection and depth estimation as foundational tasks.

Increasing the volume of training data progressively improves Qwen2.5-VL-3B’s AP@15 on SUN RGB-D and ARKitScenes.The experiment evaluates whether spatial perception capabilities emerge as foundational-task data is scaled.

C More Data Engine Details

The data-engine details augment multi-view object information with a visualized bird’s-eye-view image to support spatial-relation reasoning. Figure 5 summarizes the scaling of spatial foundational-task data.

A visualized BEV image is added to multi-view object information as contextual input for generating spatial-reasoning data.The auxiliary view is intended to help models comprehend relationships among objects in RGB images.
Figure 5 presents the data scaling of spatial foundational tasks.

C.2 Dataset Visualization

The dataset visualization spans spatial perception, object and camera relationships, motion, video, and spatial reasoning data. Examples use structured visual representations, geometric checks, multi-turn organization, and BEV- or parallax-based reasoning prompts.

Spatial perception: Depth, distance, 3D detection, measurement, scene captioning, and correspondence data cover core spatial perception tasks.Depth uses multiple object-reference formats and sampled points; 3D detection uses 9-DoF boxes, while captions incorporate object and scene-graph information.
Spatial perception: Correspondence points are retained only after image-boundary, positive-depth, and occlusion checks.The occlusion criterion compares ground-truth depth with the depth-map value, using a 5% threshold to determine visibility.
Image relationships: Object-object and camera-camera data use shared objects across image pairs to formulate spatial relationships between objects or camera positions.Anchor objects support direction and distance questions, while camera relationships are inferred from the layouts in the paired images.
Motion and video: Camera-motion data distinguishes six rotation motions from six translation directions, and video data models object relationships, counting, and spatiotemporal information.Video object relationships are organized in multi-turn form to learn relative relationships while reducing video-loading overhead.
Spatial reasoning: Spatial reasoning samples generate reasoning processes for object-camera relationships and camera motion using BEV annotations, parallax prompts, and verified examples.General reasoning data covers math, OCR, and knowledge samples selected for partial correctness, while camera prompts separately address rotation and translation.