PaperlayerView exact revision on arXiv

Source-linked AI summary

SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models

Arijit Ray, Jiafei Duan, Ellis Brown, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A. Plummer, Ranjay Krishna, Kuo-Hao Zeng, Kate Saenko

arXiv:2412.07755v3cs.CVcs.AIcs.GRcs.RO

TL;DR

MLMs struggle with spatial reasoning, and existing studies focus mainly on static relationships despite real-world needs to reason about motion. SAT uses simulated scenes to generate static and dynamic spatial training data and test sets. Simulation training improves performance across spatial benchmarks, but dynamic reasoning remains challenging.

  • Problem

    MLMs struggle with spatial reasoning, while much prior work addresses static relationships rather than how object or camera motion changes them.

    Source: §1 · p. 2

  • Method

    SAT procedurally generates static and dynamic spatial QA from simulated scenes, complemented by a small expert-annotated real-image dynamic test set.

    Source: §1 · pp. 2–3

  • Results

    11% for LLaVA-13B and 8% for LLaVA-Video-7B: SAT improves average performance across spatial benchmarks, including real-image dynamic tests and long-video spatial reasoning.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Simulation-generated spatial training data can improve MLM performance on real-image benchmarks, although dynamic reasoning still has considerable room for improvement.

    Source: §1 · p. 3

    The study instruction-tunes image-only LLaVA and video LLaVA, does not evaluate math or science reasoning or other MLMs, and reports camera movement as hard to improve.

    Source: §4.1 · p. 8, §5 · p. 10

Abstract

from arXiv · show

Reasoning about motion and space is a fundamental cognitive capability that is required by multiple real-world applications. While many studies highlight that large multimodal language models (MLMs) struggle to reason about space, they only focus on static spatial relationships, and not dynamic awareness of motion and space, i.e., reasoning about the effect of egocentric and object motions on spatial relationships. Manually annotating such object and camera movements is expensive. Hence, we introduce SAT, a simulated spatial aptitude training dataset utilizing 3D simulators, comprising both static and dynamic spatial reasoning across 175K question-answer (QA) pairs and 20K scenes. Complementing this, we also construct a small (150 image-QAs) yet challenging dynamic spatial test set using real-world images. Leveraging our SAT datasets and 6 existing static spatial benchmarks, we systematically investigate what improves both static and dynamic spatial awareness. Our results reveal that simulations are surprisingly effective at imparting spatial aptitude to MLMs that translate to real images. We show that perfect annotations in simulation are more effective than existing approaches of pseudo-annotating real images. For instance, SAT training improves a LLaVA-13B model by an average 11% and a LLaVA-Video-7B model by an average 8% on multiple spatial benchmarks, including our real-image dynamic test set and spatial reasoning on long videos -- even outperforming some large proprietary models. While reasoning over static relationships improves with synthetic training data, there is still considerable room for improvement for dynamic reasoning questions.

1 Introduction

The paper addresses MLMs’ limited spatial reasoning, especially about motion, and introduces simulation-generated training data for static and dynamic spatial tasks. SAT improves spatial benchmark performance, including on a real-image dynamic test set, while dynamic reasoning remains difficult.

1 Introduction

  • MLMs struggle with static spatial relations, while real-world applications also require reasoning about how camera and object motion changes spatial relationships.Pseudo-annotating dynamic motion on real images is non-trivial because it requires predicting action causality.Source: §1 · p. 2
  • SAT generates 175K spatial QA pairs from 22K simulated scenes, using 3D information to create static and motion-related questions without human supervision.The dataset is procedurally generated and can be scaled by composing assets.Source: §1 · p. 2
  • The evaluation spans static image benchmarks and video benchmarks, alongside a challenging 150-question real-image test set for dynamic spatial reasoning.The real-image set addresses the absence of image datasets for dynamic questions about action causality.Source: §1 · pp. 2–3
  • 11% for LLaVA-13B and 8% for LLaVA-Video-7B: SAT improves average performance across spatial benchmarks, including the real-image dynamic test set.The reported gains are absolute; the best model scores 62% on SAT dynamic questions against 50% random chance.Source: §1 · p. 3

2 Related work

Related work covers 3D and spatial datasets, synthetic training data, and multimodal models’ spatial limitations. This paper investigates whether synthetic images with perfect 2D/3D information can improve spatial understanding.

2 Related work

  • Many 3D datasets use real-world scans, while SAT focuses on spatial reasoning from 2D inputs and supports dynamic tasks through compatibility with physics simulators.The cited work notes that recomputing scans for dynamic scenes is expensive and many MLMs take 2D images as input.Source: §2 · p. 3
  • Prior synthetic-data studies report benefits for several visual tasks, and this work tests whether synthetic data with perfect 2D/3D information improves spatial reasoning.The cited applications include classification, semantic understanding, bias correction, and navigation.Source: §2 · pp. 3–4
  • Recent work identifies weaknesses in VLMs’ 3D pose, location, and compositional spatial understanding; this paper examines whether synthetic images with perfect 3D information can help.The related work also situates the task among image- and video-based vision-language models.Source: §2 · p. 4

3 SAT: Spatial Aptitude Training

SAT procedurally generates static and dynamic spatial QA from simulated scenes, then uses the data for instruction tuning and evaluation. Its tasks range from object relations to motion, perspective, and action consequences.

3 SAT: Spatial Aptitude Training

  • 175K questions across 22K procedurally generated ProcTHOR-10K scenes form SAT’s training dataset, which serves both instruction tuning and dynamic-spatial evaluation.The pipeline automatically curates data because varied 3D annotations involving camera and object movement are expensive to obtain on real images.Source: §3 · p. 4

3.1 Generating Static Spatial QAs.

  • 127K static QA pairs cover object left/right/above/below relations, relative closeness, camera-relative depth, and object counts.Object relations are computed by projecting poses into camera coordinates; relative depth uses simulator distances, and counts come from simulator metadata.Source: §3.1 · pp. 4–5

3.2 Generating Dynamic Spatial QA

  • 48K dynamic QA pairs across five tasks probe egocentric movement, object movement, allocentric perspective, goal aiming, and action consequences.Egocentric questions infer camera actions across frames; object-movement questions compare positions in camera coordinates, including when camera and object move together; perspective questions relocate a viewer to a marked point.Source: §3.2 · p. 5
  • Goal-aiming questions ask which rough direction to turn toward a target, while action-consequence questions ask how a movement would change spatial relations to an object.Aiming uses the target’s angle relative to the camera; the action-consequence set rephrases some questions to balance answer choices.Source: §3.2 · pp. 5–6
  • The real-image dynamic test set contains 150 expert-annotated image-QAs, while a separate synthetic test set supports analysis of individual task splits.The synthetic set covers five task types across 805 images.Source: §3.2 · p. 6

3.3 Finetuning MLMs with SAT

  • Models choose between a correct answer and a distractor, and tuning mixes in original LLaVA instruction data to prevent catastrophic forgetting.Distractors are formed by switching the correct answer to an opposite or distractor word.Source: §3.3 · p. 6

4 Experiments

The experiments evaluate LLaVA image- and video-based models on spatial benchmarks, then compare simulated, pseudo-annotated, and human-annotated training data. SAT improves spatial reasoning across real-image and video evaluations, while training-data comparisons identify further gains and persistent challenges.

4.1 Can SAT improve spatial reasoning in MLMs?

  • SAT tuning improves static spatial reasoning on real-image benchmarks, often matching or outperforming closed-source and spatially tuned models.The comparisons cover LLaVA-1.5 and LLaVA-Video models against closed-source models and RoboPoint.Source: §4.1 · p. 7, Table 3 · p. 7
  • 12% for LLaVA-1.5 and 10% for LLaVA-Video on the real-image dynamic test set; SAT also improves long-video spatial reasoning, especially route planning.SAT additionally improves outdoor-video reasoning involving humans despite training on indoor scenes.Source: §4.1 · pp. 7–8, Table 4 · p. 7, Table 5 · p. 8
  • Camera movement and abstract relations remain difficult: BLINK SpRel gains are modest, with no improvement for LLaVA-Video.The authors suggest subtle camera motion may produce minimal feature changes because of translation invariance in ViT.Source: §4.1 · p. 8
  • SAT synthetic exposes split-specific weaknesses, while SAT tuning maintains performance on tested non-spatial VQA benchmarks.LLaVA-Video's smaller GoalAim gain reflects its weaker object-relation gains; the non-spatial evaluation covers GQA, VQAv2, and OK-VQA.Source: §4.1 · p. 8, Table 6 · p. 8

4.2 What are effective types of spatial training data?

  • Simulated SAT outperforms pseudo-annotated and human-annotated alternatives overall, and adding dynamic QAs further improves static and dynamic performance.Pseudo-annotations improve some splits but can contain bounding-box errors; human-labeled relations help in-domain VSR and BLINK SpRel but are finite and less composable for varied 3D and dynamic data.Source: §4.2 · p. 9, Table 7 · p. 9
  • Static-only training tends to saturate, whereas performance scales positively when dynamic questions are mixed in; adding original pretraining data further improves results.Simulation-only training remains effective on real images, and the comparison holds the amount of new SAT data constant.Source: Figure 3 · p. 9, §4.2 · p. 10, Figure 4 · p. 10

5 Discussion

SAT improves MLM performance on numerous spatial benchmarks while maintaining pretrained commonsense. The authors limit the study to spatial performance and suggest embodied-navigation benefits warrant further evaluation.

  • The study evaluates only image-only LLaVA and video LLaVA-Video, and does not test improvements in math or science reasoning.Its stated scope is analyzing data that improves spatial performance, not large-scale training of a new MLM.Source: §5 · p. 10
  • SAT Dynamic scores 51% versus 40% for static-only training on SPOC Easy-ObjectNav action prediction, but broader embodied evaluation remains future work.The action choice is left, right, or forward for a given frame.Source: §5 · p. 10
  • SAT improves MLMs on numerous spatial benchmarks while maintaining pretrained commonsense.Source: §5 · p. 10

A.1 Discussion

The study examines spatial performance in an image-only LLaVA and a video LLaVA-Video, while leaving other capabilities and broader embodied applications for future investigation.

  • The analysis is limited to image-only LLaVA and video LLaVA-Video, and does not evaluate math or science reasoning.The authors frame the work as analysis of data for spatial performance, not large-scale training of a new MLM.Source: §A.1 · p. 18
  • SAT Dynamic reaches 51% versus 40% for static-only training on frame-level SPOC Easy-ObjectNav action prediction, while broader embodied evaluation remains future work.The choices are left, right, or forward; the authors suggest navigation might benefit but leave thorough evaluation open.Source: §A.1 · p. 18

A.2 Tuning details

The experiments tune LLaVA-1.5-13B with LoRA and fully fine-tune LLaVA-Video-7B, mixing instruction-tuning examples during synthetic-data training to reduce catastrophic forgetting.

  • LLaVA-1.5-13B uses LoRA with rank 256 and alpha 512, while the smaller LLaVA-Video-7B model is fully fine-tuned.The authors report minimal trend differences with more LoRA parameters or full fine-tuning for the image model.Source: §A.2 · p. 18
  • To prevent catastrophic forgetting, training randomly mixes LLaVA Instruct Tuning examples with synthetic QA pairs at 40% probability.Models train for around one epoch of synthetic data.Source: §A.2 · p. 18
  • Training uses batch size 8, learning rate 5e−6, cosine annealing, 1K warm-up steps, and zero weight decay.Training requires two 48GB NVIDIA GPUs; inference uses one GPU.Source: §A.2 · p. 18

A.3 Dataset Details

Experts scored 92.8% on the automatically generated SAT test questions, while the best existing zero-shot MLM scored around 53%; the authors use the synthetic test set diagnostically.

  • 92.8% of 200 randomly sampled SAT questions were answered correctly by graduate-student experts, versus around 53% for the best existing zero-shot MLM.The authors note that synthetic images can be out of domain for MLMs.Source: §A.3 · p. 18
  • The authors use SAT-synthetic as a diagnostic set, noting that gains there indicate performance gains on the related task with real images.Source: §A.3 · p. 18
  • Four graduate students cross-checked the real-image SAT dataset, and the authors state that all datasets will be publicly released.Source: §A.3 · p. 18

More details on dataset creation

The camera is placed at one of 20 sampled points with the most objects visible, then object coordinates are normalized relative to its position and orientation. The y coordinate remains unchanged because it represents height.

More details on dataset creation

  • The camera is placed at the point with the most visible objects among 20 randomly chosen positions.The camera view is normalized by subtracting its x and z position and rotating the x-z plane so it points along positive z; y remains unchanged as height.Source: More details on dataset creation · p. 19
  • After normalization, x′ indicates left/right and z′ indicates depth, providing coordinates for relative spatial relations.Positive x′ is right, negative x′ is left, and positive z′ is deeper into the scene.Source: More details on dataset creation · p. 19

SAT Static Spatial QAs

SAT generates static spatial QAs about relative object positions and distances using object coordinates in the camera frame.

SAT Static Spatial QAs

  • For relative-position questions, object poses are projected into camera coordinates; pairwise distances support closer-or-farther comparisons.The example procedure randomly selects three objects and compares their pairwise 3D distances.Source: SAT Static Spatial QAs · p. 19
  • Relative-depth questions compare objects’ distances from the camera to determine which is closer.The section also describes relative depth by comparing objects’ z′ coordinates.Source: SAT Static Spatial QAs · p. 19

SAT Dynamic Spatial QAs

SAT combines simulated dynamic spatial questions with pseudo-annotated static QAs and evaluates them across spatial benchmarks. Its ablations report benefits from dynamic data and sensitivity to answer quality, alongside remaining dynamic-reasoning errors.

SAT Dynamic Spatial QAs

  • Dynamic QAs cover camera and object movement, allocentric perspective, goal aiming, and action consequences.Camera movement and rotation are treated separately, and some examples combine camera and object motion.Source: SAT Dynamic Spatial QAs · pp. 19–20

Pseudo-annotated QAs

  • Pseudo-annotated QAs use estimated depth and object boxes, selecting three visible objects after excluding small or partially occluded ones.The selected objects support questions about depth and relative location using 2.5D information.Source: Pseudo-annotated QAs · p. 20

Evaluating on VSR and GQA-Sp

  • VSR captions are reformulated as yes/no question-answer pairs to keep evaluation formats consistent.The stated aim is to make performance differences reflect spatial-relation difficulty rather than QA format.Source: Evaluating on VSR and GQA-Sp · p. 20

Evaluating on VSI-Bench

  • VSI-Bench evaluation uses only its multiple-choice split, excluding numerical distance and dimension estimates from RGB images.The authors consider those numerical splits outside SAT’s focus on relationships in images and dynamic scenes.Source: Evaluating on VSI-Bench · p. 21

Question-answer format

  • Questions use binary choices; evaluation randomizes answer order, and SAT Real uses both orders to average performance.The authors also report that specifying answer choices in text improved performance compared with letter or number options.Source: Question-answer format · p. 21

A.4 Extra ablations and analysis.

  • At matched data size, adding dynamic data maintains static spatial reasoning while improving dynamic and video spatial reasoning.For LLaVA-Video, gains on video and SAT Real accompany maintained BLINK and CVBench performance.Source: Table 10 · p. 22, §A.4 · pp. 22–23
  • Post-tuning errors cluster around counts above two and outdoor concepts; some dynamic accuracies fall below 50%, with the explanation based on qualitative analysis.The authors identify broader simulators as future work and leave more thorough analysis of dynamic errors for future work.Source: §A.4 · p. 22
  • 10% on CVB and 18% on BLINK are the performance reductions reported when training with 20% flipped answers.The authors argue that these drops indicate performance depends on spatial answer quality, not merely question formats or object/attribute-name biases.Source: §A.4 · p. 23
Loading 2412.07755v3…