PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing

Junfei Wu, Jian Guan, Kaituo Feng, Qiang Liu, Shu Wu, Liang Wang, Wei Wu, Tieniu Tan

arXiv:2506.09965v2cs.CVcs.AI

TL;DR

LVLMs often rely on text-based reasoning for spatial tasks that require tracking relationships and changing positions. VILASR reasons through visual drawing operations and improves over strong baselines across spatial-reasoning benchmarks, with particular strengths in maze navigation and temporal-spatial understanding.

  • Problem

    LVLMs often use text-based reasoning for spatial tasks requiring object relationships and dynamic tracking, despite the difficulty of translating spatial details into text.

    Source: §1 · p. 2

  • Method

    VILASR uses interpretable visual drawing operations, including bounding boxes and auxiliary lines, to reason directly about spatial relationships.

    Source: §1 · pp. 2–3

  • Results

    18.4% average improvement over strong baselines across five diverse spatial-reasoning benchmarks, including maze navigation, video-based reasoning, and multi-perspective integration.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments particularly validate the approach on maze navigation and temporal-spatial understanding tasks.

    Source: §5 · p. 10

    The approach focuses on inter-object spatial relationships rather than object-specific details and therefore uses basic drawing operations instead of zooming or cropping.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

As textual reasoning with large language models (LLMs) has advanced significantly, there has been growing interest in enhancing the multimodal reasoning capabilities of large vision-language models (LVLMs). However, existing methods primarily approach multimodal reasoning in a straightforward, text-centric manner, where both reasoning and answer derivation are conducted purely through text, with the only difference being the presence of multimodal input. As a result, these methods often encounter fundamental limitations in spatial reasoning tasks that demand precise geometric understanding and continuous spatial tracking-capabilities that humans achieve through mental visualization and manipulation. To address the limitations, we propose drawing to reason in space, a novel paradigm that enables LVLMs to reason through elementary drawing operations in the visual space. By equipping models with basic drawing operations, including annotating bounding boxes and drawing auxiliary lines, we empower them to express and analyze spatial relationships through direct visual manipulation, meanwhile avoiding the performance ceiling imposed by specialized perception tools in previous tool-integrated reasoning approaches. To cultivate this capability, we develop a three-stage training framework: cold-start training with synthetic data to establish basic drawing abilities, reflective rejection sampling to enhance self-reflection behaviors, and reinforcement learning to directly optimize for target rewards. Extensive experiments demonstrate that our model, named VILASR, consistently outperforms existing methods across diverse spatial reasoning benchmarks, involving maze navigation, static spatial reasoning, video-based reasoning, and multi-view-based reasoning tasks, with an average improvement of 18.4%.

1 Introduction

The introduction argues that spatial reasoning exposes limits in text-centric LVLM reasoning and proposes visual drawing operations as a way to reason directly in visual space. It presents VILASR’s staged training and reports improved performance across spatial reasoning benchmarks.

Introduction

  • Current LVLMs often struggle with spatial reasoning because text-only reasoning loses spatial detail and makes dynamic object-position changes difficult to describe.The introduction contrasts this with human spatial reasoning through visualization and manipulation, and notes that “thinking with images” remains underexplored in open-source research.Source: §1 · p. 2
  • Visual-tool approaches [8] [50] [21] [31] can constrain LVLM reasoning through fixed black-box perception, fragmented tool calls, and simplistic prescribed reasoning patterns.These patterns can limit critical reflection on tool outputs, motivating a more flexible intrinsic approach to vision-centric reasoning.Source: §1 · p. 2
  • VILASR reasons through visual drawing operations, including bounding boxes for object localization and auxiliary lines for spatial relationship analysis.Its three-stage training uses synthetic-data cold-start training, reflective rejection sampling, and reinforcement learning with rewards balancing answer correctness and reasoning format.Source: §1 · pp. 2–3
  • VILASR improves over strong baselines by 18.4% on average across five diverse spatial reasoning benchmarks.The benchmarks include maze navigation, video-based reasoning, and tasks requiring information integration from multiple perspectives.Source: §1 · pp. 2–3
  • Reflective rejection sampling doubles reflection-behavior frequency versus training without that stage, while inference-time scaling shows progressive gains and RL consolidation of multi-attempt capability.The final RL optimization maintains strong single-attempt performance and significantly narrows the performance gap with earlier training stages.Source: §1 · p. 2

2 Related works

Prior reasoning approaches face limits from prompt sensitivity, training-data quality and scale, or constrained tool capabilities, while systematic LVLM spatial reasoning that generalizes across images and videos remains underexplored. The paper positions its staged drawing-based framework as addressing these gaps.

2.1 Reasoning in LLMs and LVLMs

  • LLM/LVLM reasoning work spans prompt engineering [65], supervised fine-tuning [74], and reinforcement learning [29] [55], but multimodal tool-augmented RL remains underexplored.Prompt-based methods depend on instruction following and can be prompt-sensitive; supervised methods are bounded by training-data quality and scale. LVLMs either reason textually [71] or depend on specialized tools [8] [50] and lack reflective reasoning [54]; this work introduces staged training with iterative drawing operations.Source: §2.1 · p. 3

2.2 Visual spatial reasoning

  • Visual spatial reasoning requires relational reasoning [38] [59] and perspective transformation [69], yet systematic LVLM methods generalizing across images and videos remain largely unexplored.Prior work includes synthetic datasets for image-based reasoning [38] [5], 3D representations for video, and object tracking across frames; this paper proposes a principled LVLM reasoning framework.Source: §2.2 · p. 4

3 Methodology

VILASR enables LVLMs to answer spatial questions through iterative visual drawing interleaved with language reasoning. Its training framework uses cold-start training, reflective rejection sampling, and reinforcement learning to cultivate this capability.

3.1 Drawing to reason in space

  • VILASR interleaves language reasoning, drawing operations, and their observed outputs [73] until a final answer, with operations able to target input images or earlier annotated outputs.The model has indexed access to original images and operation-generated images, supporting reasoning over multiple images or video frames.Source: §3.1 · pp. 4–5
  • The paradigm uses bounding boxes to anchor object locations and auxiliary lines to show spatial trajectories and relationships, focusing on inter-object relations rather than object details or zooming and cropping [8].Operations specify target-image indices, spatial coordinates, and semantic labels; execution preserves image content while overlaying annotations.Source: §3.1 · p. 4

3.2 Training framework

  • Training covers multiple-choice and numerical questions, and rewards answer correctness plus executable reasoning format, granting format rewards only above a correctness threshold.Multiple-choice correctness uses exact match; numerical correctness uses Mean Relative Accuracy (MRA) [69] across confidence thresholds.Source: §3.2 · p. 5
  • Cold-start supervised training establishes drawing abilities using synthetic reasoning paths generated by Qwen2.5-72B-VL [2] and filtered for correctness and format.The synthetic dataset is built from public image and video question-answer pairs, and training minimizes average negative log-likelihood over reasoning and operation tokens.Source: §3.2 · pp. 5–6
  • Reflective rejection sampling fine-tunes on cold-start model paths that are correct, format-valid, and reflective, with reflection defined by recurring labels across reasoning steps.The method addresses the infrequency of reflective behavior after cold-start training, encouraging self-correction before reinforcement learning.Source: §3.2 · p. 6
  • Reinforcement learning uses GRPO without a KL penalty and terminates rollouts with zero reward for missing operations, excessive images, or duplicate operations.Source: §3.2 · p. 6

4 Experiment

VILASR performs strongly across image, video, and multi-view spatial reasoning benchmarks, with reported state-of-the-art results on video and multi-view tasks. Ablations and inference-time scaling analyses show how its training stages contribute to performance and reasoning behavior.

4.1 Experimental setups

  • Evaluation covers five benchmarks across image, video, and multi-view reasoning, using accuracy for multiple-choice questions and MRA for numerical questions.The benchmarks are Maze, SpatialEval-Real, VSI-Bench, SPAR-Bench, and MMSI-Bench; the metrics follow the training objective’s formulation.Source: §4.1 · p. 7

4.2 Main results

  • VILASR achieves state-of-the-art performance on video and multi-view benchmarks and strong results on image-based spatial reasoning tasks.The authors attribute its video and multi-view results to flexible visual operations for dynamic spatial relationships.Source: Table 1 · p. 8, §4.2 · pp. 8–9
  • Proprietary models consistently outperform non-reasoning counterparts, while Qwen2.5-VL-7B shows minimal or negative gains from enabling reasoning.The authors identify a gap in open-source models’ ability to leverage visual reasoning.Source: §4.2 · p. 8
  • Iterative drawing operations help VILASR break maze navigation into interpretable visual steps and track spatial state transitions.The authors contrast this with COGCOM’s dependence on perception tools and VisCoT’s one-time grounding without revisable visual operations [50, 54].Source: §4.2 · pp. 8–9

4.3 Ablation study

  • Cold-start training improves performance across all subtasks, while the untrained drawing paradigm performs worse than the Qwen2.5-VL-7B backbone.The results indicate that spatial reasoning capabilities require dedicated training rather than prompting alone.Source: §4.3 · p. 9
  • Removing reflective rejection sampling decreases reflection-pattern exhibition by 96.5%, reasoning steps by 33.9%, and Tbox use by 85.0%, while increasing Tline use by 70.5%.Precise localization and measurement tasks decline (Absolute Distance -0.6%, Room Size -0.7%, Relative Distance -1.0%); the authors associate reflection with rechecking bounding boxes and spatial relationships before answering.Source: §4.3 · pp. 9–10
  • Without RL, drawing use rises (Tbox +159.4%, Tline +9.1%) and performance gaps are larger for numerical than multiple-choice tasks (-9.21% vs. -4.07%).The authors attribute RL’s benefit to dense rewards based on numerical proximity to ground truth, supporting precise spatial measurements.Source: §4.3 · p. 10

4.4 Inference-time scaling

  • Across training stages, pass@8 and pass@1 improve progressively, and RL narrows the gap between multi-attempt and single-attempt performance.Cold-start training and reflective rejection sampling each raise both measures relative to the preceding stage.Source: §4.4 · p. 10

5 Conclusion · A Appendix outline

VILASR uses visual drawing operations to strengthen LVLM spatial reasoning, and its three-stage training framework is validated across diverse benchmarks. The paper also identifies future research directions and outlines supplementary materials.

5 Conclusion

  • Figure 5 analyzes inference-time scaling behavior using pass@1 and pass@8 performance.Source: Figure 5 · p. 10
  • The “drawing to reason in space” paradigm enables direct visual manipulation, bridging text-centric reasoning and human-like spatial cognition.Source: §5 · p. 10
  • The three-stage training framework cultivates sophisticated reasoning, with experiments showing particular strengths in maze navigation and temporal-spatial understanding.Source: §5 · p. 10
  • Future work could extend drawing operations to more complex 3D spatial relationships and investigate more efficient training strategies.Source: §5 · p. 10

A Appendix outline

  • The supplementary materials cover dataset construction in Appendix B and experimental setup and full evaluation results in Appendix C.Source: A Appendix outline · p. 18
  • Appendix D provides visualization results, while Appendix E compares the complexity of VILASR and baselines.Source: A Appendix outline · p. 18
  • Appendices F and G discuss VILASR’s limitations and broader impact, respectively.Source: A Appendix outline · p. 18

B Dataset construction · C Experimental setup and full evaluation results

The authors construct spatial-reasoning datasets spanning maze navigation, static-image understanding, and video comprehension, with video data adding temporal and multi-view reasoning. Their pipeline creates training sets for cold-start, reflective rejection sampling, and reinforcement learning, alongside procedurally generated mazes for controlled sequential-reasoning evaluation.

B Dataset construction

  • The datasets cover maze navigation, static-image spatial relationships, and video comprehension, with video data extending static drawing-based reasoning to temporal dynamics and multi-view perspectives.Source: B Dataset construction · p. 18
  • The pipeline produced VILASR-ColdStart-33k for supervised reasoning paths, VILASR-RRS-8k for reflective rejection sampling, and VILASR-RL-40k for reinforcement learning.Source: B Dataset construction · p. 18
  • Cold-start trajectories were generated with Qwen2.5-VL-72B [2] and filtered for answer correctness, parseable answers, executable drawing operations, and at least three thinking rounds.Source: B Dataset construction · p. 18
  • VILASR-RRS-8k selects cold-started-model instances with clear self-correction and correct final answers to train reflective behavior.Source: B Dataset construction · p. 18
  • Source data draws on GQA [22], VSR [78], OpenImages, OpenSpaces [5], and SpaceLLaVA [5], with task types balanced across the collection pipeline.Source: B Dataset construction · p. 18
  • Maze examples use depth-first search to generate solvable 3×3-to-6×6 grids in roughly equal proportions, each with a marked start and four candidate destinations.Source: B Dataset construction · p. 18
  • Maze questions ask models to follow an action sequence from the start and identify the reached destination, creating a controlled test of sequential spatial reasoning.Source: B Dataset construction · p. 18

C.1 Baseline implementation · C.2 Benchmark statistics

The section describes zero-shot baseline evaluation protocols for closed- and open-source LVLMs and specialized reasoning models, and identifies Table 4 as the source of benchmark statistics.

C.1 Baseline implementation

  • The evaluation compares VILASR against various state-of-the-art models and methods.Source: §C.1 · p. 19
  • Closed-source LVLMs were queried through official APIs in zero-shot settings using input images and questions.Table 3 is cited for the evaluated models; without reasoning, models were explicitly prompted to answer directly, with format instructions for multiple-choice and numerical responses.Source: §C.1 · p. 19
  • Open-source baselines span 7B to 72B parameters and include Qwen2.5-VL [2], LLaVA-NeXT-Video [79], and LLaVA-OneVision.They were evaluated zero-shot using their standard prompting formats.Source: §C.1 · p. 19
  • Kimi-VL-A3B-Instruct has 16.0B total parameters and dynamically activates 2.8B parameters during inference.It was evaluated zero-shot using its standard prompting format.Source: §C.1 · p. 19
  • Specialized reasoning baselines use official checkpoints and prompts: COGCOM [50] uses built-in perception tools, VisCoT [54] uses bounding-box annotation, and SpaceR uses video-understanding capabilities.Source: §C.1 · p. 19

C.2 Benchmark statistics

  • Table 4 presents the benchmark statistics.The supplied passage does not specify the statistics themselves.Source: §C.2 · p. 19

C.3 Detailed results on VSI-Bench

On VSI-Bench, VILASR achieves 45.4% average accuracy, outperforming all baseline methods by 12.7%. Gains are strongest in spatial measurement, localization, and comparative reasoning, while Room Size and Route Plan remain weaker tasks.

C.3 Detailed results on VSI-Bench

  • VILASR improves Object Count (+29.0%), Absolute Distance (+15.0%), Object Size (+13.0%), Relative Distance (+16.1%), Relative Direction (+20.7%), and Appearance Order (+19.8%).The paper attributes comparative spatial reasoning gains to auxiliary lines connecting and measuring between objects, and temporal-spatial gains to systematic annotation across frames.Source: §C.3 · p. 19
  • 45.4% average accuracy on VSI-Bench lets VILASR outperform all baseline methods by +12.7%.Table 5 reports the VSI-Bench results.Source: §C.3 · p. 19, Table 5 · p. 20
  • VILASR decreases on Room Size (-9.9%) and Route Plan (-2.1%), tasks requiring holistic layout understanding or global path planning across views.The paper identifies layout reconstruction and space-completion drawing tools as possible extensions for these limitations.Source: §C.3 · p. 20

C.4 Statistical significance analysis … C.7 Query prompt template of VILASR

Paired tests found that VILASR significantly outperforms Qwen2.5-VL-7B across five spatial reasoning benchmarks, while its prompt templates support reasoning-path generation and inference through staged queries.

C.4 Statistical significance analysis

  • VILASR significantly outperforms Qwen2.5-VL-7B on MAZE, SpatialEval-Real, VSI-Bench, SPAR-Bench, and MMSI-Bench (p < 0.05) in paired t-tests.The paper describes these results as robust and consistent across spatial reasoning scenarios.Source: §C.4 · p. 20

C.5 Prompt template

  • VILASR uses system and user prompt templates both to generate cold-start reasoning paths and during inference.The templates are illustrated in Figures 6 and 7, respectively.Source: §C.5 · p. 20

C.6 System prompt of VILASR

  • The system prompt used in VILASR’s reasoning framework is presented in Figure 6.Figure 6 is captioned as the system prompt used in VILASR.Source: §C.6 · p. 20, Figure 6 · p. 21

C.7 Query prompt template of VILASR

  • The Initial Query Prompt establishes the initial visual context and query for the reasoning process.It is one of three query prompt types used by the framework.Source: §C.7 · pp. 20–21
  • The Follow-up Query Prompt guides later reasoning steps using intermediate results and accumulated context.It is one of three query prompt types used by the framework.Source: §C.7 · pp. 20–21
  • The Final Answer Prompt ends reasoning and enforces answer generation when the maximum processed images or reasoning steps is reached.Figure 7 presents the query prompt and output template used in VILASR.Source: §C.7 · p. 21, Figure 7 · p. 22

D Visualization results · E Complexity analysis

Visualization examples show VILASR using drawing operations to make spatial reasoning concrete and correct across maze, multi-view, and video tasks. The complexity section describes an evaluation of theoretical bounds and empirical resource use against Qwen2.5-VL-7B, but gives no findings here.

D Visualization results

  • The examples span MAZE, SPAR-Bench, and VSI-Bench, shown in Figures 8, 9, and 10, respectively.Path Tracer denotes drawing auxiliary lines, and Object Mapper denotes annotating objects.Source: D Visualization results · p. 22
  • VILASR uses auxiliary lines to track maze movements and reach the correct destination, while Qwen2.5-VL-7B and GPT-4o fail in the example.The Path Tracer visualizes and verifies each movement through the maze.Source: D Visualization results · p. 22, Figure 8 · p. 23
  • For multi-view reasoning, VILASR maps objects across images and verifies perspective-based position changes, while both baselines misjudge movement direction.It maps the window and air conditioner with Object Mapper, then uses Path Tracer to verify the observer’s position change.Source: D Visualization results · p. 22
  • In the video example, VILASR self-corrects to locate the telephone in image 13 and measures it using reference objects to convert pixels to centimeters.Its initial attempts in images 5 and 12 fail; Qwen2.5-VL-7B does not locate the target, while GPT-4o estimates without proper measurement.Source: D Visualization results · p. 23
  • The examples illustrate how drawing operations ground abstract spatial relationships in concrete visual representations to support more reliable reasoning.This is the stated takeaway from the maze, multi-view, and video cases.Source: D Visualization results · p. 23

E Complexity analysis

  • The complexity analysis compares VILASR with Qwen2.5-VL-7B using theoretical complexity bounds and empirical resource utilization to assess computational efficiency and scalability.The supplied passage describes the analysis scope but does not report its results.Source: E Complexity analysis · p. 23

E.1 Theoretical time complexity · E.2 Practical runtime

The analysis compares the theoretical complexity of direct answering, standard reasoning, and VILASR, then reports how VILASR’s runtime and reasoning steps are summarized across benchmarks. VILASR’s reasoning is described as having shorter per-step paths because it decomposes reasoning into interpretable visual operations.

E.1 Theoretical time complexity

  • The complexity analysis compares direct answer generation, standard reasoning followed by answer derivation, and VILASR’s iterative visual drawing and thinking.It defines M as input length, N as answer length, L as per-step reasoning-path length, and S as reasoning steps; S = 1 for the base and standard reasoning approaches.Source: §E.1 · p. 23
  • Figures 9 and 10 show spatial-reasoning examples comparing Qwen-2.5-VL-7B, GPT-4o, and VILASR on SPAR-Bench and VSI-Bench, respectively.Figure 10 highlights VILASR-generated words exhibiting reflection behavior.Source: Figure 9 · p. 24, Figure 10 · p. 25
  • Table 6 presents a comparative time-complexity analysis of the reasoning approaches, but the supplied passage does not report its cell values.The prose identifies the table as the comparative analysis of their time complexities.Source: Table 6 · p. 24, §E.1 · p. 24
  • VILASR achieves shorter per-step reasoning paths than single-step approaches by decomposing reasoning into interpretable visual operations.The passage attributes this empirical difference to VILASR’s iterative visual drawing and thinking.Source: §E.1 · pp. 23–24
  • Assuming approximately equal reasoning-path lengths and multimodal context at each step, the approximation is S^2 · L · (L + M) + S · (L + M)(L + N).Lk denotes the reasoning-path length at step k, and Mk denotes additional multimodal context incorporated at that step.Source: §E.1 · p. 26

E.2 Practical runtime

  • Table 7 summarizes VILASR’s average runtime and average reasoning steps per sample across different benchmarks; no values are provided in the supplied passages.The practical-runtime analysis evaluates these measures across benchmarks.Source: §E.2 · p. 26, Table 7 · p. 26

F Limitations · G Broader impact

The paper identifies limits in task scope, dimensionality, and training cost, while weighing potential benefits of spatial reasoning against risks from surveillance and unequal access.

F Limitations

  • Training focuses on automatically evaluated multiple-choice and numerical questions, leaving the model less equipped for open-ended spatial descriptions such as motion-trajectory analysis.The authors say this constrains performance on open-ended spatial reasoning tasks.Source: F Limitations · p. 26
  • The 2D drawing interface may inadequately capture complex 3D relationships and viewpoint changes, limiting tasks involving 3D objects or dynamic cameras.Source: F Limitations · p. 26
  • The three-stage training pipeline, especially reinforcement learning, may be inaccessible to researchers with limited computational resources.Source: F Limitations · p. 26

G Broader impact

  • Spatial reasoning improvements could benefit robotics, autonomous navigation, assistive technology, and spatial education, while drawing-based interpretability may enhance transparency and trust.The paper gives robot navigation and daily assistance for people with visual impairments as examples.Source: G Broader impact · p. 26
  • Enhanced spatial reasoning could enable privacy-infringing surveillance, and reliance on visual interfaces may exclude users unable to access or interact with them.Source: G Broader impact · p. 26
  • The authors recommend usage guidelines and access controls, alternative interaction modalities, and privacy-preserving research; release safeguards include gated access and responsible-use documentation.The API requires agreement to guidelines prohibiting surveillance or privacy-invasive uses; documentation describes capabilities, limitations, and implementation best practices.Source: G Broader impact · p. 27

Related papers

Loading 2506.09965v2…