PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial-Interactor: Learning Spatial Reasoning through Interaction with the Observable Physical World

Kaixiang Yao, Xu Wang, Miao Pan, Hu Xiyue, Weishi Wang, Daniel Dahlmeier, Jintao Chen, Yongliang Shen, Xuhong Zhang, Wenqi Zhang

arXiv:2609.23038v1cs.AIcs.LG

TL;DR

VLMs struggle to track spatial-state changes across viewpoints and integrate them over long trajectories, while spatial training often centers on static QA. Spatial-Interactor trains on interaction trajectories through a three-level curriculum and two-stage SFT/OPD strategy. It reports consistent gains across VLMs and spatial benchmarks.

  • Problem

    VLMs remain limited on physical-world tasks requiring them to track spatial-state changes across viewpoints and time, while most spatial training focuses on static QA.

    Source: §1 · pp. 1–2

  • Method

    Spatial-Interactor organizes interaction-based transition learning into a three-level curriculum, using SFT for local transitions and OPD with privileged transition traces for long-horizon integration.

    Source: §1 · p. 3, §2.4 · p. 6

  • Results

    Spatial-Interactor consistently improves spatial reasoning across VLMs and benchmarks, with overall gains of 16.4 to 25.0 points over corresponding Base models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The approach improves spatial reasoning across backbones and benchmarks while increasing frame-order sensitivity.

    Source: §4 · p. 13

Abstract

from arXiv · show

Spatial reasoning is essential for vision-language models (VLMs) to understand and act in the physical world. Reasoning in dynamic environments requires VLMs to perceive local state transitions caused by object motion and viewpoint changes and integrate them over long trajectories to maintain an updated spatial state, yet existing VLMs remain limited in both capabilities. Current spatial training primarily focuses on static questions about object attributes and spatial relations, providing limited direct supervision for state transitions; in contrast, interaction trajectories naturally connect a preceding observation, an action, and a subsequent observation, offering direct supervision for local state transitions, while complete trajectories reveal dependencies among consecutive transitions. We therefore introduce Spatial-Interactor, a framework that trains VLMs to model physical-world state transitions through interaction, organizing this learning process into a three-level curriculum covering L1 passive world-state transitions, L2 active self-state transitions, and L3 long-horizon interaction trajectories. Accordingly, we construct the Learning from Spatial Interaction dataset (LSI-108K) from simulated and real interaction trajectories, with tasks aligned with the objective of each level. Our two-stage training strategy applies Supervised Fine-Tuning (SFT) to L1 and L2 for local transition modeling, and On-Policy Distillation (OPD) then uses privileged self-distillation: a teacher branch given segment-level transition descriptions supervises the student's on-policy CoT, helping the student learn to integrate consecutive transitions over L3 long trajectories. Experiments across multiple VLMs and spatial benchmarks show consistent gains in local transition modeling and long-horizon integration.

1 Introduction

Spatial-Interactor addresses VLMs’ difficulty tracking changing spatial states and integrating transitions over time. It trains models on interaction trajectories, from local state changes to long-horizon integration, and reports gains across spatial benchmarks.

1 Introduction

  • Shuffling 32 video frames lowers Qwen2.5-VL-3B and Qwen2.5-VL-7B accuracy by only 0.9 and 0.8 points, respectively, suggesting weak temporal-order sensitivity.No sub-task category changes by more than 1.4 points.Source: §1 · p. 2, Figure 2 · p. 3
  • Long-horizon camera-displacement performance drops from 54.7 to 8.6 for Qwen2.5-VL-7B and from 88.7 to 23.9 for GPT-5.5; GPT-5.5 reaches 35.5 with transition descriptions.The local-to-long-horizon comparison uses SAT-Real and VSTI-Bench; the trace raises GPT-5.5’s camera-displacement score from 23.9 to 35.5.Source: §1 · p. 2
  • Most spatial training focuses on static QA, while dynamic spatial reasoning requires tracking state changes across viewpoints and integrating consecutive changes over time.VLMs often lose track of spatial state, confuse prior observations, and miss fine-grained changes.Source: §1 · pp. 1–2
  • Spatial-Interactor uses interaction trajectories as direct supervision for local transitions, linking a preceding observation, an action, and a subsequent observation.Complete trajectories support learning to integrate consecutive transitions over long horizons.Source: §1 · p. 2
  • Its three-level curriculum covers passive world-state transitions, active self-state transitions, and long-horizon trajectories; SFT trains local transitions and OPD trains their integration.LSI-108K contains 108K verifiable QA pairs from simulated and real interaction trajectories.Source: §1 · p. 3
  • Spatial-Interactor improves Overall by 16.4–25.0 points over Base models and achieves gains of 4.3 to 10.6 points in cross-benchmark evaluation.The reported improvements span multiple VLMs and benchmarks.Source: §1 · p. 3

2 Method

The method treats interaction as supervision for local state changes and long-horizon trajectory integration, using a three-level curriculum and staged SFT/OPD training.

2.1 State-Transition Learning from Spatial Interaction

  • Each interaction record aligns consecutive observations with an action or event, and ordered records support reasoning from local transitions to full trajectories.A trajectory is represented as an ordered sequence of these records, with changes verifiable from interaction metadata.Source: §2.1 · p. 4

2.2 Progressive Spatial Interaction Curriculum

  • L1 covers external-world changes, L2 viewpoint-induced observation changes, and L3 integration of successive transitions across complete trajectories.The curriculum reuses verified simulated and real interaction records to generate QA pairs, filtering negligible, discontinuous, redundant, or visually unanswerable instances.Source: §2.2 · pp. 4–5, Figure 4 · p. 5, Figure 5 · p. 6

2.3 Local State-Transition Modeling

  • For L1 and L2, supervised fine-tuning predicts responses from observations before and after a transition, linking visual differences to physical causes or spatial consequences.Joint supervision over world-state and self-state changes provides local transition representations for long-horizon learning.Source: §2.3 · p. 5

2.4 Long-Horizon State-Transition Integration with OPD

  • For L3, OPD gives a privileged transition trace to the teacher, which evaluates student-generated reasoning prefixes while answer rewards and reasoning-token distillation optimize the policy.The trace and teacher are training-only; the process weight decays so later optimization increasingly relies on the policy’s own reasoning and verifiable outcomes.Source: §2.4 · pp. 6–7, Figure 6 · p. 7

3 Experiments

Experiments compare Spatial-Interactor with Base models and evaluate transfer, training-stage contributions, temporal evidence use, closed-loop interaction, and qualitative reasoning. Across four backbones, the method improves Overall by 16.4–25.0 points.

Contribution of Privileged Process Supervision.

  • From the same Full-SFT checkpoint, OPD reaches 60.1 Overall versus 59.2 for GRPO and exceeds it by 5.1 points on Route Planning and 3.1 on Camera Displacement.The comparison matches initialization, L3 questions, rollout groups, answer rewards, and optimization schedules, isolating privileged conditioning and same-prefix distillation.Source: Contribution of Privileged Process… · pp. 8–9

3.2 Main Results

  • Spatial-Interactor improves Overall by 16.4–25.0 points across four backbones, with gains spanning spatial relations, long-horizon reasoning, and multi-view tasks.Its variants achieve the highest VSI-Bench, MindCube, and VSTI-Bench averages, while SpatialLadder remains strongest on SPBench-MV.Source: §3.2 · p. 8, Table 1 · p. 9

3.3 Cross-Benchmark Generalization

  • On four benchmarks absent from training, Overall improves by 4.3–10.6 points; OPD adds 1.0–1.4 points over SFT for every backbone.Gains appear on both multi-view and interaction-oriented benchmarks.Source: §3.3 · p. 8, Table 2 · p. 9

Contribution of Local-Transition Supervision.

3.5 Process and Temporal Analysis

  • Across four backbones, mean on-policy task reward rises 6.8–8.9 points, while divergence between plain and privileged branches falls 27–56% during active process supervision.Rollout diversity decreases but remains nonzero.Source: §3.5 · p. 10, Figure 8 · p. 11
  • An ordered four-segment trace scores 35.4 versus 33.8 when shuffled; eight ordered segments score 35.3, while frame-order sensitivity also increases after training.For Route Planning, OPD performs best at each tested frame budget, with accuracy rising from 33.4 to 34.4 as more frames are provided.Source: §3.5 · pp. 10–11, Figure 9 · p. 11, Figure 10 · p. 12

3.6 Closed-Loop Interaction

  • On WalkerBench, success rises from 7.0 to 14.0 as mean actions fall from 17.71 to 6.52; on ESI-Bench, average success rises from 32.9 to 41.2.The mean rounds among solved ESI-Bench episodes fall from 12.34 to 10.55.Source: Table 4 · p. 12, Table 5 · p. 12, §3.6 · p. 12

3.7 Qualitative Analysis

  • In qualitative cases, Spatial-Interactor correctly identifies camera rotation and sustained lateral parallax where the Base model predicts the opposite rotation or forward movement.The examples use cross-view correspondence and stable office landmarks to connect local cues with viewpoint or camera motion.Source: Figure 11 · p. 12, Figure 12 · p. 12, §3.7 · p. 13

4 Conclusion

Spatial-Interactor learns physical-world state transitions through interaction, using LSI-108K's three-level curriculum and SFT followed by OPD. The paper reports consistent gains across backbones and benchmarks, alongside greater frame-order sensitivity.

4 Conclusion

  • LSI-108K organizes passive world-state changes, active self-state changes, and long-horizon trajectories into a three-level curriculum, with SFT for local transitions and OPD for trajectory integration.Future work will explore adaptive trajectory segmentation and more diverse real interactions.Source: §4 · p. 13

A.1 Evaluating Spatial Reasoning in VLMs

  • Spatial reasoning benchmarks reveal a persistent gap between recognizing spatial relations in isolated observations and maintaining coherent spatial states throughout an interaction trajectory.The evaluation landscape extends from object-centric properties to multi-view correspondence, video-based spatial structure, and embodied interaction.Source: §A.1 · p. 17

A.2 Methods for Improving Spatial Reasoning

The related-work discussion situates observable state transitions and on-policy distillation as complementary approaches for spatial reasoning. Spatial-Interactor uses trajectory descriptions available only during training to supervise long-horizon reasoning.

A.2 Methods for Improving Spatial Reasoning

  • Learning from observable state transitions complements data-centric, geometry-aware, and intermediate-representation approaches by connecting local physical changes with spatial consequences over extended trajectories.The related approaches use instruction corpora, explicit spatial structure, or representations of viewpoints, trajectories, and evolving scene states.Source: §A.2 · p. 17

A.3 On-Policy Distillation

  • On-policy distillation supervises responses sampled from the current student policy; Spatial-Interactor uses training-only trajectory descriptions as process supervision for long-horizon spatial reasoning.Prior work explores self-distillation, privileged-context supervision, and visually augmented teacher signals.Source: §A.3 · p. 17

B.1 Data Sources and Interaction Records

LSI-108K combines simulated and real interaction sources. Its records preserve local before-and-after observations and the full sequence needed for long-horizon tasks.

Simulated and Real Sources

  • LSI-108K combines simulated camera movements and object operations with rendered trajectories, real camera-pose or object-track observations, and robot end-effector and gripper states.Sources include AI2-THOR, ProcTHOR, HSSD, Replica, SIMS-V, ScanNet, ScanNet++, MultiScan, ARKitScenes, RoomTour3D, and BridgeData V2.Source: Simulated and Real Sources · p. 17

Interaction Records

  • Each local interaction record links observations before and after an executed or measured change, while retaining the full observation sequence for long-horizon tasks.Records also store action, state, pose, or trajectory metadata used to derive the supervision before question-and-answer generation.Source: Interaction Records · p. 18

B.2 Ground-Truth Derivation and QA Instantiation

Algorithm 1 converts simulated actions or annotated real trajectories into verified interaction QA records using task templates.

Construction Overview

  • Construction starts from a controllable simulated scene or an annotated real trajectory and a task-template bank.Simulated actions are executed or real camera and robot trajectories are normalized.Source: Construction Overview · p. 18
  • A canonical target is derived from the recorded action, state, or geometry, and failed, weak, discontinuous, ambiguous, or non-unique interactions are rejected.Accepted records are rendered into ordered media, questions, options, and answers with deterministic templates, then checked for visual answerability and organized by source, task, answer, and curriculum level.Source: Construction Overview · p. 18

Ground-Truth Derivation

The section derives spatial targets from simulated actions and annotated real trajectories, converts them into verified QA records, and organizes tasks across three curriculum levels.

Ground-Truth Derivation

  • Simulated interactions record actions, observations, camera poses, and object states; pose-annotated videos and robot footage supply motion and operation targets from their annotations.Camera trajectories also provide long-horizon displacement, path length, bearing, turns, revisits, path shape, and reverse-path targets.Source: Ground-Truth Derivation · p. 18

Canonical Records and QA Instantiation

  • Canonical records retain ordered visual evidence, action sequences, and ground-truth targets, which deterministic templates render as QA pairs without language-model generation.Exact distances, angles, paths, and state changes remain in metadata even when metric values are quantized for display.Source: Canonical Records and QA Instantiation · p. 19

B.3 Quality Filtering

  • Records are rejected for failed execution, negligible or discontinuous motion, ambiguous targets, invalid action sequences, or visually unanswerable evidence.A frozen VLM checks evidence visibility and consistency but cannot generate or revise the target.Source: §B.3 · p. 19

Three-Level Task Taxonomy

  • L1 covers external-world changes, L2 observer-induced changes, and L3 global path understanding; multi-step operations remain in L1 because they change the external world.Local tasks query parts of an interaction record, while L3 tasks integrate camera-motion transitions to recover displacement, path shape, turns, revisits, or reverse paths.Source: Three-Level Task Taxonomy · pp. 19–20

Final Composition and Numeric Targets

C Method and Training Details

The appendix details how segment-level traces, student rollouts, privileged process supervision, and verifiable rewards implement the two-stage training procedure.

Frozen Visual-Language Annotator and Prompt

  • A frozen visual-language annotator describes four consecutive eight-frame segments from 32 chronological frames, without seeing the question or reference answer.Descriptions identify visible environment and spatial anchors, visual changes, and coarse camera motion.Source: Frozen Visual-Language Annotator and… · p. 21

Trace Assembly and Validation

  • The four descriptions are concatenated chronologically into privileged trace z, and the student receives the video and question while the teacher also receives z.Automated checks reject hidden-target references and unsupported metric claims; the question is attached only after trace generation.Source: Trace Assembly and Validation · pp. 21–22

Student Rollouts

  • The student produces a reasoning paragraph followed by an answer, and each prompt generates eight on-policy responses.The response format separates reasoning tokens used for same-prefix distillation from the final answer.Source: Student Rollouts · p. 22

Privileged Teacher and Process Loss

  • The teacher and student evaluate the same student-generated prefixes, with the privileged trace as their only input difference; the teacher is stop-gradient and fixed during the student update.Process loss uses the teacher’s top 100 non-special tokens and excludes answer, padding, and special-token positions.Source: Privileged Teacher and Process Loss · p. 23

C.3 Verifiable Reward and Joint Objective

  • Rule-based task and format rewards are standardized within each rollout group, while a separate frozen SFT reference policy supplies the KL regularizer.Task rewards use exact match for single-choice questions, mean field accuracy for multiple fields, and a numeric prediction score for numeric questions.Source: §C.3 · p. 23

Process-Loss Schedule

  • The process-loss coefficient starts at 0.05, stays fixed for 100 steps, decreases linearly over the next 1,100 steps, and reaches zero at step 1,200.Privileged guidance is concentrated in the early and middle stages of on-policy optimization.Source: Process-Loss Schedule · p. 23

Optimization Hyperparameters

  • Both stages keep the visual encoder frozen; SFT trains for one epoch, and its final checkpoint initializes GRPO and OPD.GRPO and OPD also train for one epoch and share settings except privileged conditioning and the process objective.Source: Optimization Hyperparameters · p. 23

Compute Environment

  • Training uses one Linux node with eight NVIDIA B200 GPUs, with SFT implemented in LLaMA-Factory and GRPO and OPD in EasyR1/verl.Released code includes training entrypoints, rewards, the reasoning-token process mask, configurations, and environment manifest.Source: Compute Environment · p. 23

D Additional Analysis

The appendix specifies a trace-assisted diagnostic, a controlled trace perturbation protocol, and an expanded ESI-Bench category breakdown.

D.1 Trace-Assisted Discovery Diagnostic

  • The trace-assisted diagnostic provides four descriptions directly at inference alongside the original video and question, unlike OPD, which uses traces only in the teacher branch during training.The descriptions are generated before the question is attached, and evaluation uses the official numeric parser.Source: §D.1 · p. 24

Trace Perturbation Protocol

  • The trace perturbation comparison evaluates a frozen Qwen2.5-VL-7B SFT model on 839 camera-displacement questions from 310 videos across four trace conditions.The conditions vary trace order and segment count while holding video input, question, decoding, and answer parsing fixed; no retraining is performed.Source: Trace Perturbation Protocol · p. 24

D.2 Expanded ESI-Bench Breakdown

  • Table 7 extends the ESI-Bench comparison to 27 categories using matched subsets of 30 questions per category; Deformable and Liquid Volume are omitted because paired runs were unavailable.The table reports success rate percentages.Source: §D.2 · p. 24, Table 7 · p. 24

E Complete Visual Task Gallery

The gallery presents task examples spanning passive world-state changes (L1), active self-state transitions (L2), and long-horizon trajectory integration (L3) across simulated and real settings.

L1 passive world-state transitions

  • L1 examples test passive changes including object visibility or occlusion, movement toward or away from the camera, and robot end-effector movement magnitude.One robot example asks how far the end effector moved, with a ground-truth answer of about 10 cm.Source: E Complete Visual Task Gallery · pp. 25–26

L2 active self-state transitions

  • L2 camera-motion tasks ask models to identify single or composed movements, compare displacement magnitudes, and order views chronologically.Examples include a real ScanNet camera-motion estimate, a composed AI2-THOR action, and simulated or real temporal-ordering tasks.Source: E Complete Visual Task Gallery · pp. 27–28
  • Other L2 examples test viewpoint-dependent spatial judgments, including imagined object direction and whether an object remains visible after camera motion.The imagined-direction examples use a mirror-facing perspective and a specified rotation followed by forward movement.Source: E Complete Visual Task Gallery · p. 29

L3 long-horizon transition integration

  • L3 examples ask for a reverse path to the start, trajectory displacement, turn, and path length, or the video quarter containing a turn.The metric-trajectory examples include real ScanNet++ and RoomTour3D tasks.Source: E Complete Visual Task Gallery · pp. 30–31

Related papers

Loading 2609.23038v1…