PaperlayerView exact revision on arXiv

Source-linked AI summary

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

Hongxing Li, Jinyue Su, Dingming Li, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

arXiv:2610.12402v1cs.CVcs.CL

TL;DR

Existing spatial benchmarks mainly test reading relations already visible, rather than predicting how interventions change a scene. SpaceCast-Bench evaluates this capability through controlled transformations and unobserved outcomes; its evaluation finds a large human–model gap, while benchmark-aligned fine-tuning improves performance.

  • Problem

    Existing benchmarks focus on visible spatial relations, leaving predictive reasoning about post-intervention outcomes without a controlled, diagnostic benchmark.

    Source: §1 · p. 2

  • Method

    SpaceCast-Bench uses an observe–transform–infer framework to evaluate scene understanding, spatial state updating, and relations in unobserved outcomes.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    58.0% accuracy for the strongest model versus 87.2% human performance, with spatially specialized models near random chance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Predictive spatial reasoning remains a distinct and largely unsolved capability, and fine-tuning on generated data yields gains in-domain and across six out-of-domain benchmarks.

    Source: §5 · p. 10

    The benchmark uses indoor ScanNet and ScanNet++ scenes, so it does not cover outdoor settings or arrangements and scales absent from those sources.

    Source: A LIMITATIONS · p. 14

Abstract

from arXiv · show

Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input. Yet real-world spatial intelligence demands predictive spatial reasoning: constructing a scene from observations, anticipating how an intervention changes it, and reasoning about the unseen outcome. We introduce SpaceCast-Bench, the first benchmark to directly and diagnostically evaluate this capability. Built around an observe-transform-infer framework, its 3,862 questions from 182 real-world scenes span 16 task types at three levels: static perception, local prediction, and global prediction, progressively requiring scene understanding, spatial state updating, and relational inference over unobserved outcomes. Evaluating 21 models exposes a stark gap: the strongest model reaches only 58.0% against 87.2% human performance, while spatially specialized models remain near random chance. Controlled analyses further reveal that bridge views are critical for integrating distributed observations, and that explicit 3D evidence benefits models more reliably than generated outcome images or videos. Fine-tuning on our programmatically generated data lifts Qwen3-VL-4B from 34.0% to 65.7% with macro-average gains across six out-of-domain benchmarks.

1 INTRODUCTION

Existing benchmarks measure visible spatial relations rather than predicting unseen post-intervention outcomes. SpaceCast-Bench evaluates that predictive capability, and experiments reveal a substantial human–model gap and gains from benchmark-aligned training.

1 INTRODUCTION

  • Predictive spatial reasoning requires anticipating scene reconfiguration after intervention, a capability existing benchmarks do not evaluate in a controlled, diagnostically decomposable way.The paper frames this capability as a prerequisite for safe manipulation and multi-step planning.Source: §1 · p. 2
  • SpaceCast-Bench introduces 3,862 questions from 182 scenes across 16 task types to evaluate predictive spatial reasoning.Its observe–transform–infer framework progresses from scene understanding through spatial state updating to inference over unobserved outcomes.Source: §1 · p. 2
  • 58.0% accuracy for the strongest model versus 87.2% human performance, while spatially specialized models remain near random chance.The evaluation covers 21 proprietary, open-weight, and spatially specialized models.Source: §1 · p. 2
  • Staged fine-tuning on programmatically generated data raises Qwen3-VL-4B accuracy from 34.0% to 65.7% and yields consistent gains across six out-of-domain benchmarks.The paper presents this as evidence that the benchmark pipeline supports transferable model improvement.Source: §1 · p. 2

2 RELATED WORK

Earlier benchmarks chiefly measure perception of existing scenes, while newer work touches on scene-change reasoning. SpaceCast-Bench is presented as the first comprehensive benchmark for predictive spatial reasoning under controlled transformations.

2.1 BENCHMARKS FOR PERCEPTIVE SPATIAL REASONING

  • Single-image, multi-image, and video benchmarks broaden spatial perception, but evaluate relations in existing scenes rather than changes after intervention.The cited examples span basic directional questions, 3D annotations, cross-view matching, and egocentric video.Source: §2.1 · pp. 2–3

2.2 BENCHMARKS FOR PREDICTIVE SPATIAL REASONING

  • CLEVRER, SAT, MindCube, SpaMEM, and SpatialWorld address aspects of scene-change reasoning, but SpaceCast-Bench is presented as the first comprehensive benchmark under controlled scene transformations.The cited works cover synthetic collision counterfactuals, hypothetical ego actions, object motion, belief revision, and closed-loop agents.Source: §2.2 · p. 3

3 SPACECAST-BENCH

SpaceCast-Bench separates visible-scene perception from prediction about transformed, unobserved states. Its construction uses geometry-grounded observations and labels, with tasks organized by capability level and dataset statistics reported across scenes and task types.

3.1 OVERVIEW

  • The observe–transform–infer task gives pre-transformation views, a natural-language transformation, and a query, then evaluates the query in the unobserved result state.The pipeline requires integrating observations to recover relevant scene state, applying the transformation, and evaluating the query in the resulting state.Source: §3.1 · p. 3
  • Tasks progress from static perception to local object interventions and global layout rotation, testing increasingly demanding scene updating and reference-frame reasoning.L1 queries the unchanged scene; L2 translates, orbits, or removes an object and its attachment dependents; L3 applies a rigid yaw rotation to the layout.Source: §3.1 · p. 3

3.2 BENCHMARK CONSTRUCTION

  • The construction pipeline derives object geometry and attachment relations from ScanNet and ScanNet++ scenes, then selects depth-verified view chains with disjoint anchors and connecting bridge views.Candidate views are filtered for camera validity, image quality, projected area, and visibility; bridge views advance along a corridor without re-exposing both object groups.Source: §3.2 · pp. 4–5
  • A geometry-based relation engine generates labels before and after transformations, while automatic filtering and human review check validity and clarity.Ten annotators reviewed each question at least twice; pre-adjudication agreement was 98.9% for single-answer questions and 91.8% exact-set agreement for attachment-chain questions.Source: §3.2 · p. 5

3.3 DATASET STATISTICS

  • The benchmark contains 3,862 questions from 182 scenes across 16 task types, with 1,368 L1, 1,709 L2, and 785 L3 questions.Of L2 and L3 questions, 2,217 change the queried relation after transformation and 277 preserve it.Source: §3.3 · p. 5

4 EXPERIMENTS

Across 21 models, SpaceCast-Bench reveals a large human–model gap and distinct differences across capability levels, relation types, and reference frames. Further analyses examine prompting, visual and 3D evidence, and post-training, which improves both in-domain performance and out-of-domain macro averages.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers 21 proprietary, general-purpose open-weight, and spatially specialized models; overall score is the unweighted macro-average across 16 task types.Static, Local, and Global scores are macro-averages over task types in their respective transformation families.Source: §4.1 · pp. 5–6

4.2 MAIN RESULTS

  • 58.0% for Gemini 3.5 Flash trails human performance (87.2%); spatially specialized models score 25.7–29.1%, near the 25.9% random baseline.The strongest general-purpose open-weight model, Qwen3.5-27B, reaches 39.3%.Source: Table 1 · p. 6, §4.2 · pp. 6–7
  • Proprietary and general-purpose open-weight models generally drop from L1 to L2, while strong proprietary models recover on L3 and specialized models remain weaker there.The authors associate proprietary L3 recovery with global rotation tasks, where holistic spatial layout can suffice.Source: §4.2 · p. 7
  • Attachment leads selected proprietary and open-weight models (Qwen3.7-Plus: 80.4%), while direction and distance are hardest; world-centric is strongest and object-centric weakest among selected models.Spatially specialized models score especially low on attachment, including SenseNova-SI-1.2 at 13.4% and SpatialLadder at 9.3%.Source: Table 2 · p. 6, Table 3 · p. 6, §4.2 · p. 7

4.3 FURTHER ANALYSIS

  • CoT has no consistent accuracy benefit across models and levels; removing bridge views lowers accuracy across reference frames, most for world-centric direction (−5.3 pp).Camera-centric and object-centric direction decline by −1.9 pp and −1.7 pp, respectively.Source: §4.3 · pp. 7–8, Figure 5 · p. 8, Figure 6 · p. 8
  • Textual 3D metadata gains +30.7 pp for proprietary and +10.9 pp for open-weight models; among incorrect responses, 80.3% under Qwen-Image-Edit involved generation failures.Even successful generated endpoints still require alignment with the original multi-view evidence.Source: §4.3 · p. 8, Figure 7 · p. 9

4.4 TRAINING EXPERIMENTS

  • 65.7% overall for Staged SFT exceeds Mixed SFT at 64.8%; GRPO reaches 37.7%, while Mixed SFT is stronger on Global tasks (73.1% vs. 70.8%).Staged SFT raises Local performance from 51.3% to 54.0%.Source: Table 4 · p. 9, §4.4 · p. 9
  • 38.1% macro average for Staged SFT across six OOD benchmarks exceeds Mixed SFT (37.2%) and the base model (33.5%); MindCube remains slightly below the base model.Both SFT and GRPO improve the out-of-domain macro average over the base model.Source: §4.4 · p. 9, Table 4 · p. 9

5 CONCLUSION

SpaceCast-Bench targets predictive spatial reasoning, a capability the authors describe as distinct and largely unsolved. They report that generated-data fine-tuning improves performance in-domain and across six out-of-domain benchmarks.

5 CONCLUSION

  • Fine-tuning on programmatically generated data yields consistent gains in-domain and across six out-of-domain benchmarks, supporting the view that predictive spatial reasoning is distinct and largely unsolved.The benchmark progresses from understanding observed scenes to updating spatial states and inferring relations in unobserved outcomes.Source: §5 · p. 10
  • The authors report using generative AI for code implementation, figure plotting, and manuscript polishing, with all assisted content reviewed and verified.Source: AI USE STATEMENT · p. 10
  • The benchmark studies hypothetical spatial transformations and does not operate physical systems; the authors caution that model outputs do not establish readiness for safety-critical embodied deployment.Source: ETHICS STATEMENT · p. 10
  • Appendix B documents data sources, splits, task definitions, view-chain construction, transformations, question generation, and quality control; Appendix C describes evaluation, training, and prompts.Source: REPRODUCIBILITY STATEMENT · p. 10

A LIMITATIONS

The benchmark is limited to indoor scenes from ScanNet and ScanNet++, relies on reconstructed scene annotations, and applies programmatic transformations rather than physical simulation.

A LIMITATIONS

  • All 3,862 questions come from 182 indoor ScanNet and ScanNet++ scenes, so outdoor settings and object arrangements or scales absent from these datasets are not covered.Source: A LIMITATIONS · p. 14
  • Labels depend on reconstructed meshes, depth maps, camera poses, and semantic annotations, leaving residual uncertainty in occlusion and attachment labels despite filtering and independent review.The authors particularly note incomplete geometry and low reconstruction confidence as sources of uncertainty.Source: A LIMITATIONS · p. 14
  • Programmatic translations, rotations, and removals do not simulate physical dynamics, so deformation, friction, and collision-driven motion are outside the benchmark's scope.Source: A LIMITATIONS · p. 14

B.1 DATA SOURCES AND SCENE SPLITS

The evaluation set uses validation scenes from ScanNet and ScanNet++ and is separated by scene from the independently generated training pool.

B.1 DATA SOURCES AND SCENE SPLITS

  • The evaluation set contains 3,862 questions across 16 task types from 182 indoor scenes, while the 12,249-question training pool uses 54 disjoint scenes.The evaluation questions comprise 718 from 134 ScanNet scenes and 3,144 from 48 ScanNet++ scenes.Source: §B.1 · p. 14
  • Table 5 breaks down the 3,862 evaluation questions across 16 task types, with counts by ScanNet and ScanNet++ and reference frames specified for direction tasks.Source: Table 5 · p. 14

B.2 TASK DEFINITIONS AND DATASET COMPOSITION

The appendix defines task levels, constructs depth-verified multi-view chains, computes spatial relations from 3D geometry, and generates labeled questions for controlled scene transformations.

B.2 TASK DEFINITIONS AND DATASET COMPOSITION

  • L1, L2, and L3 describe the scope of spatial state change rather than one scalar difficulty scale; the evaluation includes both changed and unchanged relations after L2 and L3 interventions.The set contains 1,368 L1, 1,709 L2, and 785 L3 questions, including 2,217 changed and 277 unchanged queried relations among the prediction questions.Source: §B.2 · p. 14

B.3 ANCHOR–BRIDGE VIEW CONSTRUCTION

  • Cross-view questions use identity-disjoint anchor views connected by depth-verified corridor coverage, with retained views ordered between the anchors and no intervention outcome shown.Figure 8 summarizes retained bridge-view counts across 3,328 cross-view questions; only two questions need no intermediate bridge view.Source: §B.3 · p. 15, Figure 8 · p. 15

B.4 SPATIAL RELATIONS AND 3D TRANSFORMATIONS

  • Direction uses eight floor-plane sectors plus above or below, expressed in camera-, object-, or world-centric frames; questions near directional boundaries are excluded.Source: §B.4 · p. 15
  • Distance approximates shortest surface separation from sampled mesh points, falling back to 3D bounding boxes, then assigns one of four distance categories.Questions within 0.1 m of a category boundary are excluded.Source: §B.4 · p. 15
  • Occlusion uses mesh ray casting and camera-facing probes to distinguish not occluded, occluded, and not visible; visibility is recomputed after removal.Source: §B.4 · p. 15
  • Attachment graphs encode contact, support overlap or containment, and semantic compatibility; a moved root carries its transitive descendants, which attachment-chain questions require models to identify exactly.Source: §B.4 · p. 16
  • L2 moves or rotates an object and its attachment descendants, while L3 rotates the complete layout with the camera fixed; the relation engine labels the resulting 3D state.Candidate transformations that leave estimated room bounds or introduce a strict 3D bounding-box collision are rejected.Source: §B.4 · p. 16

B.5 QUESTION GENERATION

  • Question templates specify the transformation, object roles, reference frame, and answer choices; answer positions are balanced after semantic ground truth is computed.Attachment-chain items use exact-set matching, while direction and distance use four choices and occlusion uses three.Source: §B.5 · p. 16

B.6 QUALITY CONTROL

Quality control combines geometric filtering, independent human review, and adjudication; the evaluation uses standardized prompts and task-macro scores, with human and random baselines. Outcome-media diagnostics also examine generated endpoint images and videos.

B.6 QUALITY CONTROL

  • Reviewers reached 98.9% pairwise answer agreement on 3,801 single-answer questions; quality control corrected 157 ground truths, rewrote 143 questions or options, and removed 36 questions.Attachment-chain questions had 91.8% exact-set agreement; rewritten questions were independently reviewed by two annotators.Source: §B.6 · p. 16
  • The L1, L2, and L3 template tables cover static relations and attachment, local translation, orbit or removal, and global direction after full-layout rotation.Automatic filters remove ambiguous or weakly grounded items, while at least two annotators review each question and a third adjudicates disagreements.Source: §B.6 · p. 16, Table 6 · p. 17, Table 7 · p. 18

C.1 EVALUATION PROTOCOL AND METRICS

  • The primary Overall score equally weights accuracy across 16 task types; Static, Local, and Global scores average task types within each capability level.The primary evaluation uses deterministic decoding with a standardized chain-of-thought prompt; paired diagnostics compare the same question identifiers across conditions.Source: §C.1 · p. 19

C.2 HUMAN AND RANDOM BASELINES

  • Twelve volunteers scored 87.2% type-macro accuracy across 3,862 questions, while the random baseline samples uniformly from displayed options or valid attachment-chain answer sets.The human evaluation was independent of the ten-annotator quality-control review.Source: §C.2 · p. 19

C.3 OUTCOME MEDIA GENERATION

  • Endpoint-image generation uses an original view, intervention description, and target mask; video generation conditions on the original image, endpoint image, and intervention specification.A manual audit identified representative endpoint-image failures including object truncation, incorrect identity, layered composition, deformation, and visual artifacts.Source: §C.3 · pp. 19–20, Figure 9 · p. 20

C.4 TRAINING SETUP

The training setup uses generated questions across all task types, with task-balanced exposures and separate SFT and GRPO configurations.

C.4 TRAINING SETUP

  • Training uses 12,249 unique questions across all 16 task types from scenes disjoint from the benchmark, with task-balanced repetitions yielding 24,576 exposures.The data composition is illustrated by capability level in Figure 10.Source: §C.4 · p. 20, Figure 10 · p. 20
  • SFT trains rank-32 LoRA adapters and the multimodal aligner while freezing the vision encoder, using bfloat16, AdamW, cosine decay, and an 8,192-token maximum sequence length.Peak learning rates are 10^-4 for the language adapter and 10^-5 for the aligner.Source: §C.4 · p. 20
  • GRPO trains rank-16 language-model LoRA adapters with the vision encoder and multimodal aligner frozen, rewarding exact-answer correctness alongside a format term.It uses groups of eight completions and a 0.1-weighted format term.Source: §C.4 · p. 20

C.5 EVALUATION OF TRAINED MODELS

The appendix reports trained-model results, describes the out-of-domain question subsets, and specifies the evaluation prompts and generated world-model media inputs.

C.5 EVALUATION OF TRAINED MODELS

  • Table 9 reports per-task and capability-level accuracy for Qwen3-VL-4B after Mixed SFT, Staged SFT, and GRPO alongside base-model results.The supplied passage identifies the table's scope but does not provide its result values.Source: §C.5 · p. 20, Table 9 · p. 20
  • The six out-of-domain benchmarks use specified question-type subsets and counts; SPARTQA samples 50 questions from each of five types, and VSI samples 50 from each of three types.VSI's relative-direction subset contains 30 easy, 5 medium, and 15 hard questions; all models receive the same selected questions.Source: §C.5 · p. 20, Table 10 · p. 21

C.6 EVALUATION PROMPT

  • The main evaluation prompts request brief chain-of-thought reasoning before a final answer, while attachment-chain prompts allow multiple comma-separated choices; the BEV condition appends an initial-layout bird's-eye image.The single-choice format requests a final letter.Source: §C.6 · p. 21

C.7 WORLD-MODEL GENERATION PROMPTS

  • The endpoint-image prompt inserts a rigid moving group into a masked destination while preserving the surrounding scene; the video prompt specifies world displacement, reference frame, route views, and motion timing.For orbit questions, the prompt directs motion around a pivot while preserving the group's intrinsic world orientation.Source: §C.7 · pp. 21–22

D CASE STUDY

Representative failures expose intertwined weaknesses in spatial perception and transformation reasoning: models often fail to form a coherent 3D scene or update relations after interventions.

D CASE STUDY

  • Models often build shallow scene representations from input views, relying on salient foreground objects or image-space cues rather than consistent 3D structure.Depth ordering, ground-plane relations, and inter-object distances are frequently misestimated, and local observations are rarely integrated into a coherent whole.Source: D CASE STUDY · pp. 22–23
  • After interventions, models conflate movement direction with resulting relations, leave rigid attachments behind, or use the wrong reference frame or 2D heuristics.Removal cases also show incomplete visibility updates when models treat the removed object as the sole occluder.Source: D CASE STUDY · p. 23, Figure 12 · p. 23, Figure 13 · p. 24
  • The cases suggest current models lack a persistent, manipulable scene representation that can be queried from arbitrary viewpoints and updated under interventions.The camera-direction examples contrast superficial left–right reversal with recomputing the full layout in a fixed camera frame.Source: D CASE STUDY · p. 23, Figure 11 · p. 23, Figure 14 · p. 24
Loading 2610.12402v1…