PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialSpeak: QA-Native Reconstruction with Local and Global Context for Spatial Chain-of-Thought Reasoning

Yang Cao, Jiaxin Zhang, Dave Zhenyu Chen, Yingji Zhong, Ruiyuan Gao, Lanqing Hong, Dan Xu

arXiv:2609.33616v1cs.CVcs.AI

TL;DR

Answer-only training does not directly supervise the geometric estimates and derivations needed for quantitative spatial QA. SpatialSpeak pairs QA-native multi-view reconstruction with explicit spatial CoT and visual compensation; it achieves state-of-the-art results across three benchmarks, with a 62.8 ReVSI score exceeding SpatialStack-4B by 8.7 points.

  • Problem

    Answer-only supervision does not directly supervise intermediate geometric estimates or their use, although quantitative spatial answers depend on accurate estimation and appropriate reasoning.

    Source: §1 · p. 1

  • Method

    SpatialSpeak combines QA-native local-point and global-object-center reconstruction pretraining with spatial CoT that expresses question-relevant geometry, derives answers, assesses reliability, and uses visual evidence for refinement when needed.

    Source: §1 · pp. 2–3

  • Results

    SpatialSpeak achieves state-of-the-art results on ReVSI, VSI-Bench, and SPAR-Bench; its 62.8 average ReVSI score exceeds SpatialStack-4B (54.1) by 8.7 points.

    Source: §4.2 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    ReVSI ablations support complementary roles for multi-view geometry learning and explicit supervision of its use in spatial reasoning.

    Source: §5 · p. 10

Abstract

from arXiv · show

Vision-language models (VLMs) can benefit from geometric priors for multi-view spatial reasoning, yet answer-only training does not directly supervise the intermediate geometric estimates and their use in deriving quantitative spatial answers. We hypothesize that spatial chain-of-thought (CoT) supervision becomes more effective when the VLM first jointly learns complementary local geometry and global scene context through multi-view reconstruction. We introduce SpatialSpeak, a two-stage framework that connects QA-native reconstruction pretraining with spatial CoT learning. In Stage I, QA-Native Reconstruction Pretraining (QA-RP) combines marked-point 3D queries for fine-grained local geometry with object-center queries for global scene context across views. Both tasks are formulated as text-based question answering, allowing geometric estimation and subsequent reasoning to share the same autoregressive output interface. In Stage II, spatial CoT with Visual Compensation (CoT-VC) trains the model to express question-relevant geometric estimates and use them to derive answers, with reliability assessment and visual compensation supporting answer refinement when needed. On ReVSI, QA-RP increases the gain from CoT-VC from 2.6 to 6.9 points, and ablations show that both local and global reconstruction supervision are beneficial. SpatialSpeak achieves state-of-the-art results on ReVSI, VSI-Bench, and SPAR-Bench, with a ReVSI score of 62.8 that exceeds the strongest compared baseline by 8.7 points.

1 INTRODUCTION

SpatialSpeak addresses the gap between geometric learning and explicit use of question-relevant geometry by pairing QA-native reconstruction with spatial chain-of-thought supervision. It reports state-of-the-art results across three benchmarks and a larger CoT-VC gain on ReVSI when preceded by QA-RP.

  • Answer-only supervision does not directly supervise intermediate geometric estimates or their use, despite quantitative spatial answers depending on both estimation and reasoning.Source: §1 · p. 1
  • 62.8 average ReVSI score exceeds SpatialStack-4B (54.1) by 8.7 points; SpatialSpeak achieves state-of-the-art results on ReVSI, VSI-Bench, and SPAR-Bench.Source: Figure 1 · p. 2, §1 · p. 2
  • The motivation for joint reconstruction is complementary: local reconstruction grounds image points in 3D, while global reconstruction captures object-instance arrangements across views.Source: §1 · p. 2
  • QA-RP trains local marked-point 3D reconstruction and global object-center reconstruction as text-based QA in a shared coordinate system.The standard next-token objective aligns geometric prediction with the autoregressive interface later used for spatial reasoning.Source: §1 · pp. 2–3
  • On ReVSI, QA-RP increases the gain from CoT-VC from 2.6 to 6.9 points, and ablating either local or global supervision degrades performance.Source: §1 · p. 2
  • CoT-VC supervises question-relevant geometric estimates and task-specific answer derivations, with reliability assessment and visual evidence supporting refinement when needed.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work provides geometric priors through reconstruction models, feature integration, and geometry-focused training, while SpatialSpeak studies QA-based learning of geometry and its explicit use in reasoning.

3D Reconstruction

  • Pairwise reconstruction methods can restrict global context and require multiple passes and costly alignment; later methods address these constraints with long-sequence or recurrent metric-scale reconstruction.Source: §2 · p. 3

Spatial MLLMs

  • Spatial MLLM approaches inject or align geometric features, distill reconstruction knowledge, or train geometric objectives; SpatialSpeak instead studies QA-based local and global reconstruction followed by explicit spatial reasoning.Source: §2 · p. 3
  • SpatialSpeak’s overview links metric-scale reconstruction in the standard QA interface to a second stage that reasons with learned geometry and refines answers using visual evidence when needed.Source: Figure 2 · p. 4

3 METHOD

SpatialSpeak uses a two-stage reconstruction-to-reasoning workflow: QA-native multi-view reconstruction precedes spatial CoT training with visual compensation.

3.1 PRELIMINARIES

  • The VG-LLM baseline fuses visual features with frozen multi-view geometry features and generates text responses autoregressively from frames and a natural-language query.Only the LLM parameters are optimized; the vision encoder, geometry encoder, and projector remain frozen.Source: §3.1 · p. 4

3.2 STAGE I: QA-NATIVE RECONSTRUCTION PRETRAINING WITH LOCAL AND GLOBAL CONTEXT

  • QA-RP mixes local and global reconstruction examples under next-token prediction, expressing both geometric targets as text without auxiliary 3D heads or losses.Both tasks use the first-frame camera coordinate system as a shared spatial reference.Source: §3.2 · pp. 5–6
  • Local queries request 3D coordinates for a marked image point, while global queries enumerate visible objects with semantic labels and 3D centers across frames.The point target is computed from ScanNet depth maps and camera extrinsics; global outputs use a deterministic ordering.Source: §3.2 · p. 5

3.3 STAGE II: SPATIAL CHAIN-OF-THOUGHT WITH VISUAL COMPENSATION

  • CoT-VC identifies question-relevant entities, states geometric estimates, and derives answers; HIGH estimates retain the derived answer, while LOW estimates invoke visual refinement and ground-truth answer supervision.Responses include a 3D reasoning chain and reliability token.Source: §3.3 · pp. 6–7
  • Stage II uses deterministic ScanNet-based CoT generators for distance, size, count, and closest-object questions, labeling estimate reliability against QA answers.Question types outside these templates retain direct answer supervision to preserve VSI-Bench task coverage.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

The experiments evaluate SpatialSpeak on ReVSI, VSI-Bench, and SPAR-Bench, with ReVSI ablations and ScanNet reconstruction evaluation. SpatialSpeak leads the reported benchmark comparisons, while ablations examine reconstruction supervision, CoT-VC components, and reliability thresholds.

4 EXPERIMENTS

  • Spatial-reasoning evaluation uses ReVSI, VSI-Bench, and SPAR-Bench; ReVSI is used for all spatial-reasoning ablations, while reconstruction is evaluated on ScanNet.The ablations report ReVSI’s average across seven task categories; “w/o CoT-VC” retains Stage II answer-only supervision.Source: §4 · p. 7, §4.1 · p. 7

4.1 ABLATION STUDY

  • 62.8 with QA-RP and CoT-VC improves 10.4 points over the 52.4 baseline; QA-RP alone scores 55.9 and CoT-VC alone 55.0.The gain from CoT-VC rises from 2.6 points without QA-RP to 6.9 points with it.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS

  • 62.8 on ReVSI exceeds SpatialStack-4B’s 54.1 by 8.7 points, with SpatialSpeak best on five of seven task categories.The comparison uses official checkpoints and baselines under the same 32-frame setting.Source: Table 6 · p. 8, §4.2 · p. 8

4.1 ABLATION STUDY

  • 62.8 with both reconstruction tasks exceeds 59.9 without global object-center queries, 59.3 without local point queries, and 55.0 without either.The ablation supports benefits from both local and global supervision.Source: §4.1 · p. 8
  • τ = 0.3 scores 62.8, above τ = 0.1 at 59.6 and τ = 0.5 at 59.5; all three settings exceed the 55.9 variant without CoT-VC.These are the tested relative-error thresholds for constructing reliability labels.Source: §4.1 · p. 8
  • Adding spatial CoT to the 55.9 variant raises the score to 58.5, and adding visual compensation raises it further to 62.8.Visual compensation adds 4.3 points over spatial CoT without VC.Source: §4.1 · p. 8
  • Without alignment, SpatialSpeak’s Acc. and Comp. errors are 8.9 and 9.3 cm, lower than CUT3R’s 13.7 and 12.7 cm and MapAnything’s 36.3 and 28.4 cm.After GT Sim(3) alignment, SpatialSpeak records 5.0 cm on both metrics, slightly higher than CUT3R’s 4.7 and 4.6 cm.Source: §4.1 · p. 8

4.2 MAIN RESULTS

  • 63.3 is the highest average on VSI-Bench under normal training; scaled training scores 73.0 versus GeoThinker-8B’s 72.6, and SPAR-Bench scores 76.0 versus 72.0 for SpatialStack-4B.On SPAR-Bench, SpatialSpeak also exceeds GeoThinker-8B’s 68.2.Source: Table 7 · p. 9, §4.2 · p. 9, Table 8 · p. 9

5 CONCLUSION

SpatialSpeak combines QA-native multi-view reconstruction pretraining with explicit spatial reasoning supervision. It achieves state-of-the-art performance on three benchmarks, and ReVSI ablations support contributions from both local and global reconstruction supervision.

5 CONCLUSION

  • SpatialSpeak combines local-geometry and global-scene-context reconstruction pretraining with explicit spatial reasoning supervision.QA-RP uses local point and global object-center queries through the VLM’s standard QA outputs; spatial CoT finetuning teaches question-relevant estimates and their use in deriving answers.Source: §5 · p. 9
  • SpatialSpeak achieves state-of-the-art performance on ReVSI, VSI-Bench, and SPAR-Bench.ReVSI ablations show that local and global reconstruction supervision contribute to spatial reasoning and that QA-RP increases the gains from subsequent CoT-VC.Source: §5 · p. 10
  • The findings support complementary roles for learning multi-view geometry and explicitly supervising its use in spatial reasoning within this framework.Source: §5 · p. 10

A.1 EVALUATION ON SPAR-BENCH

The appendix reports a comparison with state-of-the-art models on SPAR-Bench and identifies its baseline sources.

A.1 EVALUATION ON SPAR-BENCH

  • The appendix table compares SpatialSpeak with state-of-the-art models on SPAR-Bench.Source: Table 9 · p. 14
  • The listed baselines span models including InternVL2, Qwen2.5-VL, VLM-3R, GeoThinker, and SpatialStack.The caption also lists InternVL2.5, LLaVA-OV, Qwen2-VL, LLaVA-v1.5, LLaVA-v1.6, Spatial-MLLM, UniUGG-3B, G2VLM-SR, and SenseNova-SI.Source: Table 9 · p. 14
  • Baseline results are primarily sourced from the supplementary material of G2VLM.Source: Table 9 · p. 14

A.2 MORE QUALITATIVE EXAMPLES

The additional examples illustrate spatial answers built from estimated geometry and compare Stage I reconstruction distances against ground truth and other methods.

A.2 MORE QUALITATIVE EXAMPLES

  • For the trash-bin–toilet example, SpatialSpeak estimates a closest-point distance of 3.6 m, versus 2.7 m for the variant without QA-RP or CoT-VC and 3.5 m ground truth.The estimate is derived from the objects’ estimated 3D centers and dimensions.Source: Figure 5 · p. 15, §A.2 · p. 15
  • The chair-count example enumerates four instances with distinct estimated 3D centers and predicts 4, matching ground truth; the variant without QA-RP or CoT-VC predicts 6.Source: Figure 6 · p. 15, §A.2 · p. 15
  • SpatialSpeak’s Stage I marked distances, 1.24 and 2.52 m, are closer to ground truth, 1.25 and 2.54 m, than CUT3R’s or MapAnything’s.CUT3R gives 1.38 and 2.82 m; MapAnything gives 1.63 and 2.90 m.Source: Figure 7 · p. 16

A.3 MORE IMPLEMENTATION DETAILS

This section presents training hyperparameters for both stages of SpatialSpeak, with Stage I and Stage II configurations summarized in separate tables.

A.3 MORE IMPLEMENTATION DETAILS

Loading 2609.33616v1…