PaperlayerView exact revision on arXiv

Source-linked AI summary

InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models

Nianchen Deng, Lixin Gu, Shenglong Ye, Yinan He, Zhe Chen, Songze Li, Haomin Wang, Xingguang Wei, Tianshuo Yang, Min Dou, Tong He, Wenqi Shao, Kaipeng Zhang, Yi Wang, Botian Shi, Yanting Zhang, Jifeng Dai, Yu Qiao, Hongjie Zhang, Wenhai Wang

arXiv:2506.18385v1cs.CV

TL;DR

Existing open spatial-reasoning resources are limited in scale, visual diversity, and instruction formats. InternSpatial pairs a large, diverse single- and multi-view dataset with evaluation benchmarks, and training on it improves spatial-benchmark results while preserving general multimodal performance.

  • Problem

    Existing open spatial-reasoning resources are limited in scale, visual diversity, instruction formats, and multi-view supervision.

    Source: §1 · p. 1

  • Method

    The paper constructs InternSpatial from diverse visual sources with single- and multi-view QA pairs and 19 instruction formats, and introduces benchmarks for single-view and rotation-angle evaluation.

    Source: §1 · p. 2

  • Results

    12.1% improvement on InternSpatial-Bench and 10.7% on VSI-Bench are reported after training on InternSpatial, while preserving general multimodal performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Spatial reasoning is important for VLM applications in robotics, autonomous navigation, and augmented reality.

    Source: §1 · p. 1

Abstract

from arXiv · show

Recent benchmarks and datasets have been proposed to improve spatial reasoning in vision-language models (VLMs), yet existing open resources remain limited in scale, visual diversity, and instruction expressiveness. In this work, we introduce InternSpatial, the largest open-source dataset for spatial reasoning in VLMs, along with InternSpatial-Bench, a corresponding evaluation benchmark designed to assess spatial understanding under diverse instruction formats. InternSpatial comprises 12 million QA pairs spanning both single-view and multi-view settings, drawn from diverse visual environments and supporting 19 instruction formats that reflect varied query styles. For evaluation, we propose InternSpatial-Bench for single-view tasks and expand multi-view reasoning by introducing a novel rotation angle prediction task that has not been explored in prior work. Experimental results show that models trained on InternSpatial achieve 12.1% improvement on InternSpatial-Bench and 10.7% on VSI-Bench, while maintaining strong performance on general-purpose benchmarks. We hope these resources will support the development of spatially capable VLMs in practical applications such as robotics and embodied AI.

1 Introduction

Existing spatial datasets and benchmarks are limited in scene diversity, instruction formats, and multi-view supervision. InternSpatial and InternSpatial-Bench address these gaps, and training on the dataset improves spatial-benchmark performance.

1 Introduction

  • Prior resources are limited in scene diversity, instruction formats, and multi-view supervision, although spatial reasoning matters in robotics, navigation, and augmented reality.Source: §1 · pp. 1–2
  • InternSpatial contains 9.5M single-view and 2.5M multi-view QA pairs from varied visual environments, with 19 instruction formats.Source: §1 · p. 2
  • InternSpatial-Bench evaluates single-view tasks, while the multi-view VSI benchmark is extended with a novel rotation angle prediction task.Source: §1 · p. 2
  • 12.1% improvement on InternSpatial-Bench and 10.7% on VSI-Bench are reported after training on InternSpatial, while general multimodal performance is preserved.Source: §1 · p. 2

2 Related Work

Prior work explores spatial reasoning through added representations and specialized datasets, but the reviewed approaches and benchmarks retain limitations in end-to-end reasoning, data availability, and multi-view performance.

2.1 Spatial Reasoning via Vision Language Models

  • Prior VLM approaches add 3D representations, dense features, masks, or depth, but have not enabled effective end-to-end spatial reasoning.Source: §2.1 · p. 2

2.2 Spatial Reasoning Datasets

  • Spatial datasets and benchmarks span 2D, top-down, 3D, and video settings, but some rely on auxiliary inputs, one dataset is not public, and models still struggle with multi-frame reasoning.Source: §2.2 · pp. 2–3

3 Dataset

InternSpatial is built with an automated pipeline that converts heterogeneous visual sources into spatial QA pairs and varied instruction formats. Its single-view benchmark is curated and manually checked, while dataset statistics cover diverse tasks, sources, and formats.

3.1 Data Engine for InternSpatial

  • InternSpatial aggregates nearly 12 million QA pairs from in-the-wild, indoor, urban, object-centric, and embodied-navigation sources.Source: §3.1 · p. 3
  • An automated modular pipeline extracts annotations, maps them into canonical view space, and uses templates to synthesize QA pairs across sources.Source: §3.1 · p. 3
  • Image-only sources require estimated camera parameters and depth maps to lift 2D annotations into 3D; available 3D annotations are projected to images for visual references.Source: §3.1 · p. 4
  • Each QA pair can yield up to 19 training samples across textual and image formats, with unsuitable variants filtered and segmentation positions used to resolve numbered-object references.Source: §3.1 · p. 5
  • Multi-view samples from ScanNet, MultiScan, R2R, and Objaverse encode object relations and cross-view properties such as rotation.Source: §3.1 · p. 5

3.2 InternSpatial-Bench

  • InternSpatial-Bench expands prior benchmarks with diverse formats and curated QA pairs, while excluding single-view tasks that become under-constrained without depth or camera information.Source: §3.2 · p. 5
  • Benchmark construction adds format variants to existing questions and manually verifies generated entries, correcting errors and removing ambiguous or ill-formed questions.Source: §3.2 · pp. 5–6

3.3 Dataset Statistics

  • InternSpatial contains 12,035,415 QA pairs across single-view and multi-view tasks; its 6,008-pair benchmark covers five single-view tasks and varied image domains.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate InternVL-Spatial-8B on single-view and multi-view spatial benchmarks, test instruction-format expansion, and check whether spatial training affects general VQA performance.

4.1 Experiment Setup

  • The experiments fine-tune InternVL2.5-8B using downsampled general datasets together with InternSpatial, and evaluate spatial and general-purpose benchmarks.VSI-Bench testing uses 32 sampled frames per video.Source: §4.1 · p. 7

4.2 Evaluation on InternSpatial-Bench

  • 12%: InternVL-Spatial-8B outperforms InternVL2.5-8B in average accuracy on InternSpatial-Bench.It also improves Position Comparison by 25% and Size Comparison by 20.9%, and surpasses GPT-4o and Claude 3.5 Sonnet across all tasks.Source: Table 2 · p. 7, §4.2 · p. 8

4.3 Evaluation on VSI-Bench

  • InternVL-Spatial-8B ranks first on four VSI-Bench tasks and achieves the highest average score among evaluated models, including GPT-4o and Gemini-1.5 Pro.It ranks second on the remaining tasks.Source: Table 3 · p. 8, §4.3 · p. 8
  • InternVL-Spatial-8B improves over InternVL2.5-8B by more than 10% on VSI-Bench Object Counting, Object Size Estimation, and Appearance Order.The reported gains are across all benchmark tasks.Source: Table 3 · p. 8, §4.3 · p. 8

4.4 Effect of the various question formats

  • InternSpatial training with instruction-format expansion yields the best performance across image and text formats, while training without expansion also outperforms the baseline across formats.The baseline performs best on original images and natural-language instructions, but worse on formats such as <box>; the analysis excludes Rotation Estimation.Source: §4.4 · p. 8, Figure 5 · p. 9

4.5 General VQA

  • Compared with InternVL2.5-8B, InternVL-Spatial-8B changes by +1.8% on MathVista, -0.1% on OCRBench, +0.9% on TextVQA, -1.6% on ChartQA, and +0.2% on MMStar.The authors report comparable general-benchmark performance and no compromise to general reasoning capabilities.Source: Table 4 · p. 9, §4.5 · p. 9

5 Conclusions

InternSpatial provides a large, diverse spatial QA dataset, and InternSpatial-Bench evaluates spatial understanding across single-view and multi-view settings. The paper reports improved spatial benchmark performance while noting that template-based generation may underrepresent natural-language richness.

  • InternSpatial contains 12 million QA pairs across single-view and multi-view settings, with broad scene diversity and 19 instruction formats.The formats are intended to reflect varied ways users express spatial queries.Source: §5 · p. 9
  • InternSpatial-Bench provides a diagnostic single-view benchmark and extends multi-view evaluation with rotation angle prediction, a task not addressed in prior work.Source: §5 · p. 9
  • Experiments report substantial spatial-benchmark improvements while maintaining strong performance on general multimodal tasks.Source: §5 · p. 9
  • The template-based QA generation pipeline may underrepresent natural-language richness; future work will explore more expressive generation and open-ended spatial reasoning in interactive environments.Source: §5 · p. 9

A Explanation and Statistics of Tasks

InternSpatial and InternSpatial-Bench cover 10 spatial reasoning tasks, with task explanations and QA counts organized in separate tables.

B Templates for generating QAs in InternSpatial

This section presents templates for generating InternSpatial QA pairs, using placeholders and varied phrasings across question-and-answer formats. The examples cover spatial comparisons, existence estimation and object counting, as well as two-image rotation and route planning and video questions about distances, objects, room properties, and appearance order.

B Templates for generating QAs in InternSpatial

  • Template placeholders are filled with object references, values, and choices, and candidate phrasings are randomly selected to diversify generated sentences.The passage describes this as the QA-generation method and says it provides the full template list.Source: B Templates for generating QAs in… · p. 13
  • Position-comparison templates ask whether one object is above, below, left, right, nearer, or farther than another, using binary and choice-style questions.Other examples ask for the higher, lower, leftmost, rightmost, frontmost, or rearmost object, or offer paired alternatives such as above/below and near/far.Source: B Templates for generating QAs in… · pp. 14, 15, 17
  • Size-comparison templates ask binary questions or request a choice between objects based on width, height, or overall size.The examples include broader/narrower, taller/shorter, larger/smaller, and a choice of which object is wider.Source: B Templates for generating QAs in… · pp. 18–20
  • Comparison templates ask which of two objects is wider, taller, or larger, or whether one has more or less of an attribute than the other.They provide multiple phrasings for questions and corresponding answers.Source: B Templates for generating QAs in… · pp. 21–22
  • The templates frame existence questions about whether [B] is left of, above, near, or in front of [A], and comparisons asking whether [B] is wider, narrower, or larger in volume than [A].Source: B Templates for generating QAs in… · pp. 22–24
  • Object-counting templates ask how many objects are above, left of, or closer to the observer than an anchor, or taller or larger than it.Other listed examples count objects by right/left and front/behind relations, as well as relative width, height, and size.Source: B Templates for generating QAs in… · pp. 24–26
  • Two-image templates ask for an object's relative rotation, with variants for clockwise, counterclockwise, and 180-degree answers.Question and answer wording allows approximate angle estimates.Source: B Templates for generating QAs in… · p. 26
  • A multi-image route-planning template asks a robot to choose a turn while navigating toward a staircase.The example presents an action sequence with turn-back, turn-left, or turn-right choices and requests the option letter.Source: B Templates for generating QAs in… · p. 27
  • Templates ask for closest-point object distances and video-frame comparisons of which candidate object is nearest.The distance question requests an answer in meters, while the video example uses multiple-choice options.Source: B Templates for generating QAs in… · p. 27
  • Video-frame prompts ask about room object counts, room area, and an object's longest dimension.The room-area prompt specifies combined space when multiple rooms appear; answers are requested as a word or phrase.Source: B Templates for generating QAs in… · pp. 27–28
  • A video-frame template asks for the first-appearance order of specified categories using multiple-choice answers.The example requests the answer as the letter of the selected option.Source: B Templates for generating QAs in… · p. 28

C VLM-assisted Annotation for InternSpatial

InternSpatial’s annotation pipeline uses QWen2.5-VL 72B as an assistant for object detection, captioning, and grounding. Its prompts specify how each operation should be performed and formatted.

C VLM-assisted Annotation for InternSpatial

  • QWen2.5-VL 72B serves as the assistant for object detection, captioning, and grounding, with a corresponding prompt for each process.The prompts specify the requested outputs.Source: C VLM-assisted Annotation for… · p. 28
  • Object detection identifies living and non-living things, assigning each a label, detailed phrase description, and grounding box.The requested JSON contains a list of objects, each with a label, caption, and box; box coordinates run from top-left to bottom-right.Source: C VLM-assisted Annotation for… · p. 28
  • Captioning prompts use bounding boxes and optional labels to generate brief object descriptions of fewer than ten words, one per line.The supplied example pairs a bus label and box with a short description.Source: C VLM-assisted Annotation for… · pp. 28–29
  • Grounding prompts match a phrase to objects in an image and return their boxes as [x1, y1, x2, y2] coordinates.The response is specified as a JSON object containing a list of grounding boxes.Source: C VLM-assisted Annotation for… · p. 29

D More Details about InternSpatial-Bench Generation Pipeline

InternSpatial-Bench QA pairs are generated from images without relying on existing annotations, then manually verified. Its prompts cover image filtering, captioning, and top-level-object questions across several spatial tasks.

D More Details about InternSpatial-Bench Generation Pipeline

  • The benchmark pipeline filters images, captions them, designs questions, grounds objects, generates answers and instruction variants, then has experienced annotators verify every QA pair.It produces the necessary 2D annotations without relying on existing annotations.Source: D More Details about… · p. 29
  • Image filtering checks for at least five objects, cartoons, groups of images, screenshots, and realistic camera-captured images.Source: D More Details about… · p. 29
  • Image captioning prompts request a detailed image description and descriptions of all top-level objects, excluding objects that are parts of other objects.Source: D More Details about… · p. 29
  • Position questions compare randomly selected top-level object pairs using left-right, near-far, or higher-lower relationships, while excluding environment objects.The prompts also create two variants by changing the relationship while keeping the other question content unchanged.Source: D More Details about… · p. 30
  • Size questions compare top-level objects using larger-smaller, taller-shorter, or wider-narrower relationships and can be expanded by changing the relationship.Source: D More Details about… · p. 31
  • Existence and counting prompts ask two questions of each type about top-level objects, then generate four variants by changing the relationship.Relationship choices include relative position, distance, size, and dimensions; environment objects are excluded.Source: D More Details about… · pp. 31–32

E Training Details

InternVL-Spatial-8B is trained with LoRA on general and InternSpatial samples, and the paper reports the training setup and hyperparameters.

E Training Details

Loading 2506.18385v1…