PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialBlock: Enhancing Spatial Intelligence in LVLMs via Synthetic Block-Stacking Problem

Soohyun Ryu, Sohee Kim, Eunho Yang

arXiv:2609.07064v1cs.CVcs.AI

TL;DR

LVLMs still struggle with reconstructing 3D scene structure from 2D images, while existing real-scene supervision is costly and noisy. SpatialBlock-15k addresses this gap with synthetic block-stacking tasks and color cues, and training with either direct or reasoning-based prediction significantly outperforms baselines and generalizes to real-world scenes.

  • Problem

    LVLMs have limited ability to reconstruct and reason about 3D scene structure from 2D images, while real-scene spatial supervision requires costly, time-consuming, and noisy dense annotations.

  • Method

    SpatialBlock-15k uses synthetic block-stacking tasks covering 3D-to-2D projection, viewpoint transformation, and structural combination, with controlled color cues for anchor-based reasoning.

  • Results

    Training with SpatialBlock-15k through direct or reasoning-based prediction significantly outperforms baselines and generalizes effectively to real-world scenes despite entirely synthetic, relatively small-scale data.

  • Takeaways & Limitations

    Targeted training on fundamental spatial reasoning tasks provides a scalable alternative to annotation-heavy real-scene supervision for enhancing LVLM spatial intelligence.

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) have achieved strong performance on diverse visual tasks, yet their ability to reconstruct and reason about the 3D structure of the scene depicted in 2D images -- referred to as spatial intelligence -- remains limited. Existing approaches attempt to address this gap by using real-scene spatial question answering datasets that require dense geometric annotations. However, constructing such labels is costly, time-consuming, and often noisy due to reliance on external perception modules. In this work, we propose a novel paradigm inspired by human cognitive development: learning foundational spatial skills through structured block-manipulation tasks. We introduce SpatialBlock-15k, a synthetic dataset of 15,000 block-stacking problems covering 3D-to-2D projection, viewpoint transformation, and structural combination. The dataset further incorporates controlled color modulation as visual cues to encourage anchor-based reasoning in visually complex conditions. Experiments demonstrate that LVLMs trained on our dataset through either direct answering or reasoning-based prediction significantly outperform baselines and generalize to real-world spatial tasks, despite the dataset's synthetic and compact nature. Code and data are available at https://github.com/rsoohyun/SpatialBlock.

1 Introduction

LVLMs remain limited at reconstructing and reasoning about 3D scene structure from 2D images, motivating a synthetic block-based approach to spatial skill learning. SpatialBlock-15k covers three spatial tasks and improves LVLM performance, including generalization to real-world scenes.

  • LVLMs struggle to reconstruct 3D scene structure from 2D images despite strong performance on other visual tasks.This limitation constrains applications requiring robust spatial reasoning, including autonomous driving and robotics.
  • Existing real-scene spatial datasets require dense annotations that are costly, time-consuming, and noisy because they often rely on external perception modules.
  • The dataset covers 3D-to-2D projection, viewpoint transformation, and structural combination, with controlled color cues that support anchor-based reasoning.
  • LVLMs trained with direct or reasoning-based prediction significantly outperform baselines and generalize effectively to real-world scenes despite synthetic, relatively small-scale training data.
  • SpatialBlock-15k trains foundational spatial skills through synthetic block-stacking tasks rather than annotation-heavy real-scene supervision.The approach is inspired by structured manipulative tools used in human spatial development.
  • Block-stacking reasoning involves inferring latent 3D composition, reasoning under transformation, and integrating multiple components into a coherent structure.

2 Related Work

Prior work improves LVLM spatial intelligence through architectural modifications and specialized training strategies. Spatial datasets range from simple spatial relations to higher-level reasoning, often using external perception systems to derive geometric supervision.

  • Existing approaches modify vision architectures with spatial tokens, depth-aware modules, or spatial encoders to learn 3D features.
  • Other approaches design specialized training strategies and datasets spanning simple spatial relations to complex multi-step reasoning.Some benchmarks use external detection, segmentation, depth, or pose models to extract 3D information and generate spatial question-answer pairs.

3 SpatialBlock-15k

SpatialBlock-15k uses synthetic block-stacking tasks to target foundational spatial abilities while extending them with color cues for depth, transformation, and structural correspondence. Its three question types probe projection, viewpoint change, and structure combination.

  • SpatialBlock-15k is a scalable synthetic dataset designed to enhance foundational spatial abilities without dense real-scene 3D labels.
  • Block-stacking reasoning requires spatial composition, mental simulation, and spatial integration across visible and occluded components.
  • 3D-to-2D projection: Q1 asks models to predict a 2D projection from a 3D structure, requiring viewpoint transformation and depth-aware occlusion reasoning.
  • Viewpoint transformation: Q2 tests whether models preserve structural consistency and relative positions when a 3D structure undergoes rotation or viewpoint change.
  • Structural combination: Q3 requires simulating how two 3D components interact at contact interfaces and inferring the resulting integrated structure.
  • Visual cue extension: Controlled color cues encode depth ordering, anchor-block roles, and correspondence between overlapping blocks across separate inputs.These cues are intended to support anchor-centered scene organization and relational reasoning.

4 Method

The method trains LVLMs either to predict answers directly or to produce reasoning traces. The reasoning-based strategy combines supervised initialization, lightweight adaptation, and reinforcement learning with rewards for correctness, format, and length.

  • The framework trains direct-answer models for rapid inference and reasoning-based models to articulate intermediate logical paths before answering.
  • Direct answer prediction: Direct answer prediction optimizes ground-truth answer sequences with cross-entropy over the textual query and image inputs.
  • Reasoning-based prediction: Reasoning-based prediction first uses supervised fine-tuning, then reinforcement learning to establish task-solving ability and facilitate higher-order structural reasoning.
  • Model initialization: LoRA initializes the model instead of full-parameter fine-tuning to preserve inherent reasoning and Chain-of-Thought capabilities while aligning it with the dataset.
  • Reinforcement learning: GRPO optimizes responses using a multi-objective reward that evaluates answer correctness and reasoning-trace quality.The reward includes accuracy, required format, and length components.
  • Reinforcement learning: GRPO computes group-relative advantages from sampled candidate responses and updates the policy with clipping and a KL penalty against a reference policy.

5 Experiments

Experiments evaluate SpatialBlock-15k across in-domain, real-scene, and general visual benchmarks, including direct, reasoning-based, ablation, and Synthetic-Real comparisons. Training on the synthetic block-stacking dataset consistently improves spatial reasoning and generalizes beyond the supervised task distribution.

  • 5.2 Main Results: Training on 15k synthetic SpatialBlock samples consistently outperforms existing spatial specialists on real-scene spatial reasoning benchmarks.The authors identify block-stacking as an effective spatial-reasoning training signal despite its compact synthetic scale.
  • 5.2 Main Results: 51.3% is the best open-source performance achieved by SpatialBlock-direct Qwen3-VL-4B, improving 25.1% over its backbone on MindCube.The 3B and 7B Qwen variants also improve over their comparisons by 2.7% and 17.6%, respectively.
  • 5.2 Main Results: SpatialBlock-reason improves MMSI-Bench performance and reasoning alignment, reaching 21.1 versus 17.8 for the baseline.The 3B model exceeds SpatialLadder-3B by 3.2% using only synthetic block-stacking data.
  • 5.2 Main Results: Reasoning models improve relative spatial reasoning on SPBench despite those question types being absent from training, demonstrating generalization beyond the supervised distribution.Direct models perform best on canonical 90-degree transformations, whereas reasoning models favor diverse viewpoint changes and multi-image reasoning.
  • 5.3.2 Ablation on Visual Cues: Removing color cues reduces MindCube accuracy by 7.9% for direct models and 7.3% for reasoning models, with smaller MMSI-Bench drops of 1.9% and 4.8%.The ablation supports color modulation as a useful cue for complex-scene spatial reasoning.
  • 5.3.3 Ablation on Task Design: Under matched training conditions, SpatialLadder-26k underperforms SpatialBlock on most benchmarks except in-domain SPBench, supporting the proposed task design.Synthetic-Real preserves relative-direction and relative-distance semantics in a block-based environment, while SpatialBlock is reported to generalize better out of domain.

6 Conclusion

The paper concludes that structured synthetic block-stacking tasks can teach LVLMs foundational spatial skills without annotation-heavy real-scene supervision. SpatialBlock-15k combines three spatial task types with controlled color cues, supporting direct and reasoning-based models that generalize across real-world tasks and domains.

  • 6 Conclusion: SpatialBlock-15k trains spatial intelligence through 3D-to-2D projection, viewpoint transformation, and structural integration tasks.Controlled color variation encourages anchor-based reasoning by using specific objects as reference points.
  • Dataset construction: The dataset uses valid 3×3-grid structures with one-to-four vertically stacked blocks per position, excluding configurations whose severe occlusion makes heights ambiguous.Distractors are structurally plausible but incorrect, requiring spatial-relationship reasoning rather than simple heuristics.
  • 3D-to-2D projection: Projection tasks ask models to select a correct 2D view against distractors created by block-removal, color-consistency, or combined perturbations.Perturbations target visible columns with sufficient color diversity so alternatives remain meaningful and distinguishable.
  • Viewpoint transformation: Viewpoint-transformation tasks apply 90°, 180°, 270° rotations, or vertical flipping and require mapping a highlighted visible block to its transformed position.Choices include the correct transformation, incorrectly highlighted blocks, the original structure, and another transformation.
  • Structural combination: Structural-combination tasks require predicting the union of two structures joined at a color-marked attachment point and first-contact overlap region.Distractors replace one component while preserving the other, making the precise interaction between the input pair decisive.

A.2 Human Evaluation on SB-Bench

Human evaluation on 30 SB-Bench samples collected responses from 31 participants, while direct-model training selected an intermediate checkpoint to limit overfitting. Reinforcement-learning prompts required numbered reasoning steps and tagged final answers.

  • Human evaluation: 95% average human accuracy on 30 SB-Bench samples demonstrates that the task is simple for human participants.Responses were collected from 31 participants.
  • Training procedure: Direct-model training selected the iteration immediately before test accuracy began decreasing after exceeding 93%, reducing overfitting to block structures.The checkpoint-selection rule used the observed test-accuracy trajectory.
  • Reasoning prompt: The reinforcement-learning prompt requests numbered reasoning steps followed by a final answer enclosed in <answer> and </answer> tags.The extra <reason> tag was omitted to reduce formatting-learning load.

B.4 Reasoning quality evaluation protocol

Reasoning quality is evaluated on MMSI-Bench by comparing generated reasoning with ground-truth rationales step by step. GPT-5 assigns partial or full alignment scores that are averaged and scaled to 100.

  • B.4 Reasoning quality evaluation protocol: GPT-5 scores each generated reasoning step against the corresponding ground-truth rationale as 1, 0.5, or 0.The labels represent present, implicit, or incorrect alignment.
  • B.4 Reasoning quality evaluation protocol: The final reasoning-quality score averages the step-level alignment values and scales the result to 100.MMSI-Bench supplies the ground-truth rationales used for this evaluation.

C.1 Additional Results on Numerical Tasks

The model improves counting and size estimation on SPBench numerical tasks, while remaining less effective at absolute distance. Viewpoint generalization evaluates whether this spatial understanding transfers beyond the fixed training angle.

  • Counting and size estimation improve on SPBench, whereas absolute-distance performance remains weaker.The authors attribute this pattern to stronger structural understanding and spatial integration than precise metric regression.
  • Figure 7 compares performance across 45°, 30°, and 60° viewing angles for the same block structures.
  • The viewpoint experiment tests models on identical block configurations rendered at 30° and 60° after training exclusively on 45° views.This setup probes whether models learn underlying 3D structure rather than fixed-view appearance patterns.

C.3 Cold-start data quality

Cold-start reasoning traces are unreliable: models can answer correctly while misrepresenting the 3D structure and producing inconsistent option explanations. These defects help explain the limited benefit of cold-start fine-tuning and motivate concern about noisy teacher supervision.

  • Correct answers can coexist with incorrect 3D explanations, including block-counting errors and repeated rationales for distinct options.The resulting final choice may therefore be random despite evaluating multiple answer options.
  • Full-parameter fine-tuning with cold-start data yields only marginal improvement over LoRA-based initialization.
  • GPT-5 also frequently produces inaccurate block descriptions and reasoning, indicating that high-capability teachers may still provide unstable supervision for these tasks.

C.4 Qualitative Result

Qualitative results show that SpatialBlock training improves multi-image spatial reasoning, especially when visual cues support anchor-based integration across viewpoints. The final model identifies a shared object and reaches the correct answer where baseline variants fail.

  • SpatialBlock-3B-reason identifies the “yellow container” as a shared anchor, connects three viewpoints, and produces the correct answer.The task requires aggregating multiple images and inferring a spatial relationship from the second viewpoint.
  • The model trained without visual cues attempts multi-image reasoning but fails to identify the common object, while the baseline describes only one view.
  • Visual cues enable anchor objects to integrate information across multiple viewpoints, with additional qualitative traces reported in Figure 10.
Loading 2609.07064v1…