PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representation

Weijian Ma, Shizhao Sun, Tianyu Yu, Ruiyu Wang, Tat-Seng Chua, Jiang Bian

arXiv:2601.01984v1cs.CV

TL;DR

Current VLM approaches lack principled spatial-layout modeling for rigorous spatial reasoning. This paper has VLMs build and reason over object-centric JSON blueprints, reporting a 35.9% improvement over base Qwen2.5-VL on SAT.

  • Problem

    Current VLM approaches lack principled spatial-layout modeling for rigorous spatial reasoning, despite its importance for developing more general and grounded AI.

    Source: §1 · p. 1

  • Method

    The method has VLMs construct object-centric JSON blueprints of relevant objects’ positions, sizes, and attributes, then reason over those representations to answer questions.

    Source: §1 · p. 3, §3.1 · p. 4

  • Results

    35.9% improvement over base Qwen2.5-VL on SAT, with further gains on BLINK, RoboSpatial, and VSR and superiority to proprietary and specialized spatial reasoning models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Systematically organizing spatial observations before analysis is associated with stronger benchmark performance than interleaving observations and reasoning.

    Source: §4.2 · p. 9

    Training relies on SAT's synthetic scene images, and the authors note that a more diverse dataset could further improve performance.

    Source: §5 · p. 10

Abstract

from arXiv · show

Spatial reasoning -- the ability to perceive and reason about relationships in space -- advances vision-language models (VLMs) from visual perception toward spatial semantic understanding. Existing approaches either revisit local image patches, improving fine-grained perception but weakening global spatial awareness, or mark isolated coordinates, which capture object locations but overlook their overall organization. In this work, we integrate the cognitive concept of an object-centric blueprint into VLMs to enhance spatial reasoning. Given an image and a question, the model first constructs a JSON-style blueprint that records the positions, sizes, and attributes of relevant objects, and then reasons over this structured representation to produce the final answer. To achieve this, we introduce three key techniques: (1) blueprint-embedded reasoning traces for supervised fine-tuning to elicit basic reasoning skills; (2) blueprint-aware rewards in reinforcement learning to encourage the blueprint to include an appropriate number of objects and to align final answers with this causal reasoning; and (3) anti-shortcut data augmentation that applies targeted perturbations to images and questions, discouraging reliance on superficial visual or linguistic cues. Experiments show that our method consistently outperforms existing VLMs and specialized spatial reasoning models.

1 Introduction

The paper proposes object-centric blueprints to give VLMs a coherent spatial representation for reasoning, and reports gains over base and specialized models. It combines blueprint-based supervised fine-tuning with reinforcement learning and anti-shortcut augmentation.

Motivation and prior approaches

  • Patch-based approaches [Rose et al., 2023, Zheng et al., 2025, Fu et al., 2025] improve fine-grained perception but weaken global spatial awareness; coordinate-based methods [Sarch et al., 2025, Wu et al., 2025c, Peng et al., 2023, Chen et al., 2023, You et al., 2023, Zhang et al., 2024] leave relations unorganized.The paper frames spatial reasoning as essential for intelligent robotic perception and manipulation and for more general, grounded AI.Source: §1 · pp. 1–2

Blueprint-based reasoning

  • Given an image and question, the method records relevant objects’ rough positions, sizes, and attributes in a JSON-style blueprint, then reasons over that coherent representation.Separating observation from reflection provides global spatial context rather than relying on fragmented or local cues.Source: §1 · pp. 2–3

Training techniques

  • SFT uses blueprint-embedded reasoning traces to elicit basic reasoning skills; a teacher VLM generates atomic steps that MCTS assembles into coherent, goal-directed traces.The pipeline includes steps such as adding objects to the blueprint, analyzing it, and summarizing the answer.Source: §1 · pp. 2–3
  • RL rewards encourage blueprints to include an appropriate number of objects and align final answers with blueprint-guided reasoning; image or question perturbations discourage shortcuts.The reward penalizes incomplete blueprints, caps rewards for excessive irrelevant objects, and encourages conclusions grounded in the blueprint rather than superficial correlations.Source: §1 · pp. 2–3

Results

  • 35.9% improvement over base Qwen2.5-VL on SAT; gains of 4.3%, 3.5%, and 1.2% on BLINK, RoboSpatial, and VSR, respectively.The method also surpasses GPT-5-Thinking and specialized spatial reasoning models.Source: §1 · p. 3

2 Related Works

Related work addresses spatial reasoning through dataset development, VLM structural adaptations, and explicit reasoning cues; this paper positions object-centric blueprints as a distinct approach.

Benchmarks

Adapting VLM Structures

Enabling Reasoning Capability

3 Method

The method equips VLMs with object-centric JSON blueprints that support reasoning from structured object information. It trains models to construct these blueprints and reason from them using supervised fine-tuning, reinforcement learning, and anti-shortcut augmentation.

3.1 Problem Formulation

  • Each blueprint records question-relevant objects’ identification thoughts, names, bounding boxes, and attributes before analysis produces the final answer.The reasoning trace comprises the blueprint, analysis over it, and answer.Source: §3.1 · p. 4

3.2 Approach Overview

  • Training uses supervised fine-tuning followed by reinforcement learning, combining blueprint-embedded traces, blueprint-aware rewards, and anti-shortcut augmentation.The implementation fine-tunes Qwen2.5-VL and uses GRPO for reinforcement learning [Shao et al., 2024].Source: §3.2 · p. 5

3.3 Constructing Blueprint-Embedded Traces

  • Monte Carlo Tree Search builds supervised traces by expanding atomic object-addition, analysis, and answer steps, scoring rollouts for answer correctness, and linearizing paths.The retained traces include correct paths and paths that backtrack from failed initial rollouts [Sarch et al., 2025].Source: §3.3 · pp. 5–6

3.4 Blueprint-Aware Rewards

  • Reinforcement learning combines answer-correctness and trace-format rewards with object-cardinality and causal-consistency rewards.Cardinality encourages enough objects and caps reward beyond λK, with λ=2; causal consistency compares answer-token logits conditioned on the full reasoning context versus only the image and question. The method multiplies cardinality by format reward because format tends to saturate early; distinct-object counting uses bounding-box IoU ≤0.3.Source: §3.4 · p. 6

3.5 Anti-Shortcut Data Augmentation

  • Anti-shortcut augmentation perturbs images or spatial predicates in questions, changes the answer accordingly, and filters examples whose altered input does not support that answer.Image edits remove question-mentioned objects; altered answers are “0” for counting questions and “question and image do not match” otherwise. Question edits invert spatial predicates and flip the answer.Source: §3.5 · pp. 6–7

4 Experiment

Across five benchmarks, the blueprint-based method outperforms the reported baselines, including on out-of-distribution evaluations without task-specific fine-tuning. The experiments also examine reasoning strategy, blueprint organization, ablations, qualitative behavior, and video extension.

4.1 Experiment Setup

  • The method is trained on SAT and evaluated on five benchmarks, including out-of-distribution sets, without task-specific fine-tuning.SFT uses 65k blueprint-embedded traces from MCTS rollouts on 1.5k samples; RL uses 32k SAT question–answer pairs, each with an augmented variant. Baselines include proprietary and open VLMs and specialized spatial-reasoning methods.Source: §4.1 · pp. 7–8

4.3 Qualitative Results

  • In the kettle example, our model uses the blueprint as a well-structured representation, identifies the socket as a fixed reference, and correctly infers the kettle’s movement while other methods miss it or fail to account for camera motion.Source: Figure 3 · p. 8

4.2 Quantitative Results

  • The method improves SAT-val by 35.9% over base Qwen2.5-VL and roughly 15% over ViGoRL, and outperforms all baselines on four out-of-distribution benchmarks.The out-of-distribution comparison includes substantially larger models such as the GPT series and Robix-32B.Source: §4.2 · p. 8
  • Qwen2.5-VL’s native CoT underperforms its direct-answer version across benchmarks, while GPT-4o CoT has limited SAT-val gain and degrades Blink performance.ViGoRL and ViLASR can also perform worse than base Qwen2.5-VL; cited failure causes include ViLASR tool-call errors and GPT-5-Thinking token-limit overruns.Source: §4.2 · pp. 8–9
  • Unlike ViGoRL and ViLASR, which interleave scattered observations and reasoning, the method organizes observations in a blueprint and outperforms them across benchmarks.The authors suggest that this systematic organization of perceived content is crucial to performance.Source: §4.2 · p. 9

4.3 Qualitative Results

  • Attention maps show that the method’s high-attention patches cluster around the true region of interest, unlike the scattered patches of other reasoning-based methods.The authors propose that this tighter, semantically aligned visual focus may help explain improved performance.Source: §4.3 · p. 9, Figure 4 · p. 9

4.4 Ablation Study

  • Removing anti-shortcut augmentation modestly reduces SAT-val performance but substantially harms SAT-test and Robospatial; removing blueprint-aware rewards, RL, or the full workflow also degrades results.Vanilla GRPO, with no SFT, blueprint, or blueprint-related techniques, performs significantly worse than the full method.Source: Table 2 · p. 9, §4.4 · p. 10

4.5 Potential Extension

  • The image-trained method can extend to video without retraining by selecting the four frames most similar to the question using CLIP; it works when those frames clearly show all mentioned objects.The authors identify improving frame selection as a direction for adapting image-trained spatial reasoning models to video.Source: §4.5 · p. 10, Figure 5 · p. 10

5 Conclusion · A Details of the Reinforcement Learning Stage

The paper proposes object-centric blueprints to support VLM spatial reasoning, combining blueprint-based supervised fine-tuning with reinforcement learning and anti-shortcut augmentation. Its conclusion notes training-data and model-scale limitations, while the appendix describes GRPO’s reward normalization, clipped objective, and advantage handling.

5 Conclusion

  • The method builds JSON-style blueprints of object positions, sizes, and attributes, then uses blueprint-embedded reasoning traces, blueprint-aware RL rewards, and anti-shortcut augmentation to improve spatial reasoning and generalization.The authors propose extending the approach to video and 3D reasoning and downstream tasks such as robotic manipulation.Source: §5 · p. 10
  • Training relies on SAT’s synthetic scene images, and the authors have not explored larger VLMs such as 32B models because of limited computational resources.The paper identifies more diverse training data as a possible route to better performance.Source: §5 · p. 10

A Details of the Reinforcement Learning Stage

  • GRPO uses group-wise normalized advantages and a clipped, token-level PPO-style objective to stabilize policy training on long-form trajectories.For each trajectory, its scalar reward is centered by subtracting the group’s average reward.Source: A Details of the Reinforcement… · p. 15
  • The GRPO objective uses a clipped surrogate loss with a clipping threshold of ε = 0.2, while β controls KL regularization toward the reference policy.The appendix describes this formulation as particularly effective for stabilizing optimization in long-horizon, multimodal reasoning tasks.Source: A Details of the Reinforcement… · p. 15
  • The advantage estimator sets advantages to zero when their rewards fall within one standard deviation of the mean reward.The passage specifies this as the standard-deviation range around the mean.Source: A Details of the Reinforcement… · p. 15

B Details of Data Augementation Pipeline

The augmentation pipeline creates either image-perturbed or question-perturbed examples for SAT training, then checks that the augmented image-question-answer triplets remain plausible.

B Details of Data Augementation Pipeline

  • For SAT training, GPT-4o chooses whether to perturb the image while keeping the question unchanged or modify the question to reverse the original answer while keeping the image unchanged.The selected augmentation yields either (I′, Q, a′) or (I, Q′, a′).Source: B Details of Data Augementation Pipeline · p. 15
  • For image augmentation, the VLM identifies question-relevant objects, writes a removal prompt for each, and sends them one by one to an image editor until all are removed.The image and question are used together to identify the relevant object subset.Source: B Details of Data Augementation Pipeline · p. 15
  • For question augmentation, the VLM uses the image, original question, and a prompt template with reversal examples to produce the modified question Q′.The full image- and question-editing prompt templates are shown in Tables 6 and 7.Source: B Details of Data Augementation Pipeline · p. 15
  • The VLM rechecks each augmented image-question-answer triplet to ensure the new answer adheres to it and the augmented question or answer is plausible.Source: B Details of Data Augementation Pipeline · p. 15

C More Details about Experiment Settings … D.2 About Video Spatial Reasoning.

The appendices detail training and evaluation settings and proprietary-model prompts, then illustrate reasoning traces and video spatial reasoning, including cases of reasoning–answer inconsistency and frame selection.

C.1 Details of data selection in the training set.

  • RL training samples 24373 SAT-static, 3071 action-consequence, 2313 action-sequence, 1290 object-movement, 1233 goal-aim, and 442 perspective examples.SAT is used throughout reinforcement learning, with an even sample of the same total number of QA pairs as VigoRL for a fair comparison.Source: §C.1 · pp. 15–16

C.2 Full list of subcategories used in Blink.

  • Blink testing uses 11 of 14 categories: Visual Correspondence, Jigsaw, Spatial Relation, Semantic Correspondence, Visual Similarity, Multi-view reasoning, Functional Correspondence, Relative Depth, Object Localization, and Counting.Source: §C.2 · p. 16

C.3 Full prompts of proprietary models.

D.1 More Visualization of Reasoning Traces.

  • Qualitative examples across spatial-reasoning tasks show stronger performance from the method, which combines blueprint-based thinking patterns with corresponding training strategies.The reasoning trajectories are shown in Figures 8–35.Source: §D.1 · p. 16
  • GPT-4o and GPT-5-Thinking exhibit reasoning–answer inconsistency, supporting the use of a consistency reward beyond any single base model.The inconsistency is illustrated in Figure 24.Source: §D.1 · p. 16

D.2 About Video Spatial Reasoning.

  • The video method samples 128 frames, reranks them with SigLIP to choose four candidates, then checks frames sequentially until it finds relevant information or exhausts the candidates.If a frame is irrelevant, the model says “Questions and image do not match” and moves to the next one.Source: §D.2 · p. 16
  • Video examples show the model answering when the correct frame ranks first and navigating to it after excluding irrelevant frames when Q-Frame ranks it lower.The passage attributes this ability to adversarial-image design that helps distinguish irrelevant frames.Source: §D.2 · pp. 16–17

D.3 Failure cases. · E More ablation studies.

The appendix identifies missed objects, missing real-world priors, and weak reasoning about 3D space and camera geometry as recurring failure sources, motivating stronger physical, spatial, and camera-geometry capabilities.

D.3 Failure cases.

  • Failures arise from missed objects and absent real-world priors, including overlooking a distant man and failing to account for luggage hidden by opaque wagon shells.The paper says these cases call for stronger visual grounding and real-world priors.Source: §D.3 · p. 17
  • The model mistakes camera zoom for lamp movement, showing difficulty distinguishing changes in viewpoint from motion in 3D space.In Figure 38, it concludes the lamp moved right toward the cameras because it fails to perceive the zoom.Source: §D.3 · p. 17
  • Under complex two-axis camera rotation, the model treats a moving plant as stationary despite cues from its distance to the desk.The passage notes that 4D reconstruction methods could also reveal the plant's movement.Source: §D.3 · p. 17
  • The paper identifies stronger physical priors, real-world spatial senses, and camera-geometry understanding as future directions for spatially aware VLMs.These directions follow from the failure cases discussed in the appendix.Source: §D.3 · p. 17

E.1 Ablation on adding image diversity.

Adding real-world images during reinforcement learning improved SAT-test and VSR performance, with negligible SAT-validation loss; smaller gains on OOD Blink and Robospatial indicate that task diversity is also needed.

E.1 Ablation on adding image diversity.

  • Adding real-world images during reinforcement learning improved SAT-test and VSR performance, with only negligible loss on SAT validation.The ablation added sampled SAT training data, VSR training data, and augmented data; the results are reported in Table 4, whose values are not included in the supplied passages.Source: §E.1 · p. 18
  • The gains on OOD Blink and Robospatial were smaller than on SAT-test and VSR, indicating image diversity helps generalize across image domains but does not replace task diversity.The authors call for task diversity during training-data construction because the OOD benchmarks also test questions or capabilities unseen during training.Source: §E.1 · p. 18
  • SAT-test and VSR became iid after real-world images were added, whereas Blink and Robospatial remained OOD because their tested questions or capabilities were unseen in training.The paper defines iid as a covered combination of image style and task; other cases are OOD. SAT training uses synthetic indoor images, unlike the real indoor and outdoor images in the other benchmarks.Source: §E.1 · pp. 17–18

E.2 Ablation on different model architectures.

The authors test blueprint-based reasoning on GPT-4o through prompting and report performance gains, indicating applicability beyond the original model architecture.

E.2 Ablation on different model architectures.

  • Prompting GPT-4o at test time with blueprint-based thinking yields performance gains.The experiment places the strategy in prompt templates and uses in-context learning.Source: §E.2 · p. 18

F More performance comparison and analysis. · F.1 Additional notes to Table 1 in the main paper. · F.2 About failures in comparative models.

The appendix reports a weighted comparison with Robix while cautioning that datasets and training setups limit direct comparison, and details reasoning failures in comparative models and the authors’ model.

F.1 Additional notes to Table 1 in the main paper.

  • 85.13% weighted average on two subsets surpasses Robix 32B and is comparable with Robix 7B despite using far less training data.The authors describe this as evidence of their method’s effectiveness, while noting that the comparisons are not direct.Source: §F.1 · p. 18
  • Direct comparison is limited because Mirage uses private 500-sample test sets and per-subtask fine-tuning, while Robix covers only Blink spatial relations and depth perception.The authors’ method is pretrained on all SAT tasks and tested zero-shot on Blink without fine-tuning.Source: §F.1 · p. 18

F.2 About failures in comparative models.

  • 8.9% of Vilasr samples have tool-call failures, mainly from incorrect indexing of image pixels during image operations.The appendix reports these as detailed failure statistics for the comparative models.Source: §F.2 · p. 18
  • 14.9% of gpt-5-thinking samples fail to produce final answers, including cases where reasoning reaches correct conclusions but continues without answering.Such cases are shown in Figures 12, 15, 18, 26, and 34.Source: §F.2 · p. 18
  • The authors’ model misses a seated man in red with sunglasses when distracted by a standing man, and fails to infer luggage is hidden by a nontransparent wagon.These are two of the appendix’s examples of failures by the authors’ model.Source: Figure 36 · p. 50, Figure 37 · p. 51
  • The authors’ model also fails to recognize a zoomed-in camera view and to sense the correct camera rotation along two axes.These are additional failure examples reported in the appendix.Source: Figure 38 · p. 52, Figure 39 · p. 53

Related papers

Loading 2601.01984v1…