PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representation

Weijian Ma, Shizhao Sun, Tianyu Yu, Ruiyu Wang, Tat-Seng Chua, Jiang Bian

arXiv:2601.01984v1cs.CV

TL;DR

Spatial-reasoning VLMs need a more organized representation than local patches or isolated coordinates provide. This paper constructs object-centric JSON blueprints and trains VLMs with blueprint-based traces, rewards, and perturbations, achieving gains across in-distribution and out-of-distribution benchmarks. The approach remains limited by synthetic SAT training data and untested larger VLMs.

  • Problem

    Existing spatial-reasoning approaches use local patches or isolated coordinates, limiting explicit global organization of object relations.

  • Method

    The method builds a JSON-style blueprint of relevant objects, then applies SFT with blueprint traces, RL with blueprint-aware rewards, and anti-shortcut augmentation.

  • Results

    The method consistently outperforms existing VLMs and specialized spatial reasoning models, including a 35.9% improvement over base Qwen2.5-VL on SAT and gains on three out-of-distribution tests.

  • Takeaways & Limitations

    Organizing observations into an object-centric blueprint provides a structured basis for spatial reasoning in VLMs.

  • Takeaways & Limitations

    Training relies on synthetic SAT scenes, and larger VLMs such as 32B models were not explored because of limited computational resources.

Abstract

from arXiv · show

Spatial reasoning -- the ability to perceive and reason about relationships in space -- advances vision-language models (VLMs) from visual perception toward spatial semantic understanding. Existing approaches either revisit local image patches, improving fine-grained perception but weakening global spatial awareness, or mark isolated coordinates, which capture object locations but overlook their overall organization. In this work, we integrate the cognitive concept of an object-centric blueprint into VLMs to enhance spatial reasoning. Given an image and a question, the model first constructs a JSON-style blueprint that records the positions, sizes, and attributes of relevant objects, and then reasons over this structured representation to produce the final answer. To achieve this, we introduce three key techniques: (1) blueprint-embedded reasoning traces for supervised fine-tuning to elicit basic reasoning skills; (2) blueprint-aware rewards in reinforcement learning to encourage the blueprint to include an appropriate number of objects and to align final answers with this causal reasoning; and (3) anti-shortcut data augmentation that applies targeted perturbations to images and questions, discouraging reliance on superficial visual or linguistic cues. Experiments show that our method consistently outperforms existing VLMs and specialized spatial reasoning models.

1 Introduction

The paper frames spatial reasoning as essential for grounded VLMs and identifies weaknesses in local-patch and isolated-coordinate approaches. It proposes object-centric blueprints, combined with SFT, RL, and anti-shortcut augmentation, and reports consistent gains across benchmarks.

  • Spatial reasoning supports VLM understanding of surroundings and is important for robotic perception, manipulation, and more grounded artificial intelligence.
  • Local image-patch methods improve fine-grained perception but weaken global spatial awareness, while coordinate-based methods overlook organized relations among objects.
  • The method constructs a JSON-style blueprint of question-relevant objects’ positions, sizes, and attributes before analyzing it to answer the question.
  • Blueprint-embedded traces support SFT, while blueprint-aware rewards and anti-shortcut perturbations strengthen reasoning and discourage superficial visual or linguistic correlations.
  • 35.9% improvement over base Qwen2.5-VL on SAT was accompanied by gains of 4.3%, 3.5%, and 1.2% on BLINK, RoboSpatial, and VSR out-of-distribution tests.
  • The contributions integrate object-centric blueprints into VLM reasoning and report consistent gains over existing VLMs and specialized spatial reasoning models.

2 Related Works

Related work covers spatial-reasoning benchmarks, structural adaptations of VLMs, and reasoning-oriented methods. The approach overview positions blueprint-based reasoning alongside these existing directions.

  • Spatial-reasoning benchmarks span images, videos, and 3D scenes, with many datasets providing only test sets.
  • Structural adaptations modify attention or add depth, 3D, or spatial-temporal encoders to improve VLM spatial reasoning.
  • Reasoning-capability methods extend Visual-CoT-style approaches to wayfinding, GUI manipulation, document reasoning, and robot manipulation, including spatial methods that use images or coordinates.
  • The approach overview organizes the proposed pipeline around blueprint-embedded traces, supervised fine-tuning, reinforcement learning, and anti-shortcut augmentation.

3 Method

The method trains a VLM to construct a JSON-style object blueprint, reason over it, and produce an answer through supervised fine-tuning and reinforcement learning. Blueprint-aware rewards and targeted perturbations encourage sufficient object coverage, causal grounding, and resistance to shortcuts.

  • 3.1 Problem Formulation: The reasoning trace contains an object blueprint O, analysis z, and final answer a, with each object recording identification thought, entity name, bounding box, and attribute.
  • 3.2 Approach Overview: The two-stage training recipe combines supervised fine-tuning, reinforcement learning, blueprint-embedded traces, consistency-preserving rewards, and anti-shortcut augmentation.SFT elicits basic reasoning skills, while RL further optimizes reasoning behavior.
  • 3.2 Approach Overview: Blueprint-embedded traces are constructed by prompting a teacher VLM for atomic steps, assembling them with MCTS, and retaining correct and backtracking trajectories as SFT supervision.Search nodes add objects, analyze the blueprint, or summarize reasoning; rollouts are scored by answer correctness and linearized into traces.
  • 3.4 Blueprint-Aware Rewards: The RL reward combines answer correctness, trace format, object cardinality, and causal consistency to keep blueprints valid, appropriately sized, and connected to final answers.The method multiplies format and cardinality rewards so that cardinality remains optimized after formatting stabilizes.
  • 3.4 Blueprint-Aware Rewards: Object cardinality uses the number of objects referenced in the question or answer as K, rewards blueprints up to λK objects, and caps rewards beyond that threshold.λ is set to 2; distinct objects are counted using bounding-box positions and sizes with an IoU threshold of ≤0.3.
  • 3.5 Anti-Shortcut Data Augmentation: Anti-shortcut augmentation perturbs images by removing referenced objects or questions by reversing spatial predicates, then changes answers and filters noisy examples.This targets memorization of visual or linguistic patterns during RL.

4 Experiment

Experiments evaluate the method on iid and out-of-distribution spatial reasoning benchmarks, with qualitative analyses, attention visualizations, ablations, and a video extension. The method consistently outperforms baselines, while each proposed component contributes to performance or generalization.

  • Experiment Setup: The evaluation covers SAT validation and test sets, BLINK, Robospatial, and VSR, with training and configuration details specified for SFT and RL.The method is trained on SAT data and evaluated across benchmarks that include distribution shifts, real-world images, or unseen question types.
  • Quantitative Results: 35.9% improvement over base Qwen2.5-VL on SAT-val, with roughly 15% gain over ViGoRL and higher performance than all baselines on the out-of-distribution benchmarks.The reported out-of-distribution comparison includes SAT-test, BLINK, Robospatial, and VSR, despite no task-specific finetuning.
  • Qualitative Results: A structured blueprint outperforms methods that interleave scattered observations and reasoning, indicating that organizing perceived content matters for spatial reasoning.The qualitative comparison includes cases where the model uses a fixed wall socket to account for camera motion and infer the kettle’s actual movement.
  • Qualitative Results: Attention maps show that the method concentrates high-relevance patches around the true region of interest, whereas other reasoning methods produce scattered attention.The authors state that this tighter, more semantically aligned focus may help explain the improved performance.
  • Ablation Study: Removing anti-shortcut augmentation mainly harms out-of-distribution performance, while removing blueprint-aware rewards or RL degrades results across benchmarks.Vanilla GRPO also performs significantly worse, supporting the full blueprint-based spatial reasoning workflow.
  • Potential Extension: The image-trained method extends to video without additional retraining by selecting the four frames most similar to the question.The authors observe good performance when selected frames clearly show all mentioned objects.

5 Conclusion

The paper enhances VLM spatial reasoning by constructing object-centric blueprints and training models to reason over them with supervised and reinforcement learning techniques.

  • The method constructs a JSON-style blueprint of relevant objects before analyzing it to produce the final answer.The blueprint records object positions, sizes, and attributes.
  • Supervised fine-tuning uses blueprint-embedded reasoning traces, while reinforcement learning adds blueprint-aware rewards and anti-shortcut augmentation.These techniques are intended to improve reasoning and generalization.
  • GRPO stabilizes reinforcement learning on long-form trajectories through group-wise normalized advantages and a clipped token-level PPO-style objective.The method uses GRPO as the reinforcement-learning advantage estimator.

B Details of Data Augementation Pipeline

The augmentation pipeline perturbs either the image or the question while preserving a corresponding altered answer, then verifies the resulting triplets.

  • The pipeline either edits the image while keeping the question unchanged or perturbs the question while keeping the image unchanged.Each perturbation is designed to reverse the original answer.
  • Image augmentation identifies question-relevant objects and removes them sequentially with generated editing instructions.A VLM detects relevant objects, writes removal prompts, and an image editor applies them one by one.
  • Question augmentation uses a prompt template with reversal examples to generate a modified question.The image and original question are provided to a VLM alongside the template.
  • The resulting image–question–answer pairs are checked again to ensure that the altered answer is plausible and consistent with the triplet.The checked pairs are (I, Q′, a′) and (I′, Q, a′).

C.1 Details of data selection in the training set.

The supplementary experiments describe benchmark prompts, training-data selection, qualitative findings, and extensions of blueprint-based reasoning across models and video settings.

  • Data selection: Training samples are drawn from several SAT task categories, including static scenes, actions, object movement, goal aiming, and perspective.The reported sample counts include 24,373 static, 3,071 action, 2,313 action-sequence, 1,290 object-movement, 1,233 goal-aim, and 442 perspective examples.
  • Evaluation setup: The method is tested on BLINK using 11 of its 14 categories.The selected categories include visual correspondence, jigsaw, spatial relation, relative depth, object localization, and counting.
  • Qualitative findings: Qualitative results report stronger performance across different spatial reasoning tasks and identify inconsistencies between reasoning and final answers in GPT-4o and GPT-5-Thinking.The inconsistency finding motivates the consistency reward.
  • Video extension: A video extension samples 128 frames, reranks them with SigLIP, and selects four candidate frames for sequential model processing.The extension is described as an adaptation of the method to video spatial reasoning.
  • Video extension: The model can exclude irrelevant frames and navigate to the correct frame when the most relevant frame is not ranked first.The qualitative results attribute this behavior to the adversarial-image design.

D.3 Failure cases.

The reported failures arise from incomplete object perception, missing real-world priors, and difficulty handling 3D spatial structure and camera geometry.

  • Failures occur when the model does not perceive all objects in the input image.One example omits a distant person while recognizing a prominent foreground person.
  • The model can fail without real-world physical priors, such as understanding that opaque wagon shells obstruct the view of luggage.This failure concerns visibility reasoning rather than object localization alone.
  • Ignoring 3D space and camera geometry can cause incorrect judgments about object movement under zoom or multi-axis camera rotation.Examples involve a lamp judged to move because of zoom and a plant judged stationary during camera rotation.
  • The authors identify stronger grounding, physical priors, spatial senses, and camera-geometry modeling as future directions.They also mention distance reasoning and 4D reconstruction as ways to perceive plant movement.

E.1 Ablation on adding image diversity.

The experiments distinguish in-distribution from out-of-distribution settings and examine whether adding real-world images during reinforcement learning improves generalizability. The evaluation spans synthetic and real-image datasets, including tasks and questions not covered during training.

  • Experimental setting: The study defines iid examples as those whose image style and task both appeared during training, with all other cases classified as ood.The SAT training set uses synthetic indoor images, whereas SAT test, Blink, Robospatial, and VSR include real indoor or outdoor images.
  • Experimental setting: SAT test, Blink, Robospatial, and VSR create image-domain gaps relative to the synthetic indoor SAT training set.The passage specifically identifies the latter datasets as containing real images and therefore out-of-distribution images under the stated definition.
  • Real-image ablation: The real-image ablation adds VSR training data to sampled SAT data and augmented data during reinforcement learning.This setup tests whether familiarity with spatial reasoning on real-world images affects generalizability.

E.2 Ablation on different model architectures.

The blueprint-based reasoning strategy was also tested through in-context prompting on GPT-4o, extending the evaluation beyond the original model architecture. The reported results show performance gains from test-time prompting.

  • In-context learning: Prompting GPT-4o with blueprint-based thinking at test time yields performance gains.The strategy was inserted into prompt templates and evaluated through in-context learning.
  • In-context learning: The GPT-4o prompting experiment is presented as evidence that the blueprint-based reasoning method applies across model architectures.The passage describes this as showcasing broad applicability.
  • In-context learning: The architecture ablation uses prompt templates rather than retraining GPT-4o with the blueprint strategy.The experiment conducts in-context learning at test time.

F More performance comparison and analysis.

Additional analysis compares the method with competing systems, reports reasoning failures, and presents qualitative examples and data-augmentation materials. The evidence highlights differences in comparison settings and failure modes rather than providing a complete quantitative summary here.

  • Comparison analysis: Comparison with Robix and Mirage is limited by differences in datasets, private evaluation samples, and subtask-specific fine-tuning.Mirage uses private test samples of size 500 and fine-tunes separately on subtasks, complicating direct comparison with the proposed model.
  • Reasoning failures: 14.9% of GPT-5-thinking samples fail to produce final answers, while Vilasr has about 8.9% tool-call failures.The reported Vilasr failures mainly involve incorrect indexing on image pixels during image operations.
  • Qualitative analysis: Qualitative examples include applications of the method to video spatial reasoning.Figures 6 and 7 are labeled as qualitative examples for video spatial reasoning.
  • Data augmentation: Tables 6 and 7 document the prompt template and algorithm used throughout the data-augmentation process.The supplied passages identify these materials as a two-part table presentation.
  • Qualitative analysis: The appendix presents multiple qualitative examples alongside four failure cases for the proposed model.The failure cases concern distraction by salient objects, transparency reasoning, zoom recognition, and camera rotation.
Loading 2601.01984v1…