Source-linked AI summary
Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing
Junfei Wu, Jian Guan, Kaituo Feng, Qiang Liu, Shu Wu, Liang Wang, Wei Wu, Tieniu Tan
TL;DR
Existing LVLM reasoning is largely text-centric, limiting spatial tasks that require precise geometric understanding and continuous spatial tracking. The paper introduces visual drawing operations with a three-stage training framework, and VILASR achieves an 18.4% average improvement across five diverse spatial reasoning benchmarks. The approach remains scoped to basic drawing operations and leaves more complex 3D relationships for future work.
Problem
Text-centric LVLM reasoning can lose spatial details and struggles with precise geometric understanding and continuous tracking of changing object positions.
Method
The paper trains VILASR to interweave thinking with elementary visual drawing operations through cold-start training, reflective rejection sampling, and reinforcement learning.
Results
18.4% average improvement over strong baselines was achieved across five diverse spatial reasoning benchmarks.
Takeaways & Limitations
Direct visual manipulation provides a supported approach for spatial reasoning across maze navigation, temporal-spatial understanding, and multi-view scenarios.
Takeaways & Limitations
The work uses basic drawing operations for inter-object spatial relationships and identifies more complex 3D spatial relationships as future work.
Abstract
from arXiv · showhide
As textual reasoning with large language models (LLMs) has advanced significantly, there has been growing interest in enhancing the multimodal reasoning capabilities of large vision-language models (LVLMs). However, existing methods primarily approach multimodal reasoning in a straightforward, text-centric manner, where both reasoning and answer derivation are conducted purely through text, with the only difference being the presence of multimodal input. As a result, these methods often encounter fundamental limitations in spatial reasoning tasks that demand precise geometric understanding and continuous spatial tracking-capabilities that humans achieve through mental visualization and manipulation. To address the limitations, we propose drawing to reason in space, a novel paradigm that enables LVLMs to reason through elementary drawing operations in the visual space. By equipping models with basic drawing operations, including annotating bounding boxes and drawing auxiliary lines, we empower them to express and analyze spatial relationships through direct visual manipulation, meanwhile avoiding the performance ceiling imposed by specialized perception tools in previous tool-integrated reasoning approaches. To cultivate this capability, we develop a three-stage training framework: cold-start training with synthetic data to establish basic drawing abilities, reflective rejection sampling to enhance self-reflection behaviors, and reinforcement learning to directly optimize for target rewards. Extensive experiments demonstrate that our model, named VILASR, consistently outperforms existing methods across diverse spatial reasoning benchmarks, involving maze navigation, static spatial reasoning, video-based reasoning, and multi-view-based reasoning tasks, with an average improvement of 18.4%.
1 Introduction
LVLMs often struggle with spatial reasoning because text-only reasoning can lose spatial details and inadequately track changing object positions. The paper proposes visual drawing operations and a three-stage training framework, reporting broad gains for VILASR across spatial reasoning benchmarks.
- LVLMs struggle with spatial relationships and dynamic object tracking despite strong basic visual perception, capabilities relevant to robotics and augmented reality.
- Text-only reasoning can lose spatial details when visual information is translated into textual semantic space.
- Existing visual-tool approaches can fragment reasoning through black-box perception tools and simplistic, linear tool-use patterns.
- Drawing to reason in space lets LVLMs use interpretable visual operations, including bounding boxes and auxiliary lines, for spatial problem-solving.
- VILASR uses cold-start synthetic training, reflective rejection sampling, and reinforcement learning to cultivate visual drawing and self-correction abilities.
- 18.4% average improvement over strong baselines was achieved across five spatial reasoning benchmarks involving sequential planning, temporal tracking, and multi-view integration.
2 Related works
Related work frames visual spatial reasoning as requiring relational reasoning and perspective transformation, while broader reasoning approaches include prompt engineering and tool use. These capabilities extend beyond basic visual perception.
- Existing reasoning approaches include prompt engineering to elicit latent capabilities and tool use to provide external knowledge or specialized functionality.
- Visual spatial reasoning includes relational reasoning about distances, directions, and spatial common sense between objects.
- Perspective transformation requires holding and manipulating spatial relationships across viewpoints.
3 Methodology
VILASR enables LVLMs to solve spatial reasoning problems through interleaved language reasoning, drawing operations, and observed visual results. Its three-stage training framework cultivates this capability using synthetic demonstrations, reflective sampling, and reinforcement learning.
- Drawing to reason in space: VILASR decomposes spatial reasoning into an iterative chain that interleaves natural-language reasoning, drawing operations, and their observed results.The process continues until a final answer is reached, using interaction history and indexed original or generated images.
- Drawing to reason in space: The drawing toolkit contains bounding-box annotation and auxiliary-line drawing to represent object locations, trajectories, and spatial relationships.Each operation targets an indexed image and uses spatial coordinates plus semantic labels; execution overlays the visual elements while preserving the original content.
- Drawing to reason in space: The paradigm intentionally uses basic drawing operations for inter-object spatial relationships rather than zooming or cropping.The operation set can be extended with additional visual manipulation tools in future work.
- Training framework: Training proceeds through cold-start supervised learning, reflective rejection sampling, and reinforcement learning.Cold-start data establish visual interaction capabilities; reflective sampling selects correct, format-valid paths exhibiting reflection; RL incentivizes reasoning using automated rewards.
- Training framework: The reward function combines answer correctness with reasoning-format adherence while requiring a minimum correctness threshold before granting format rewards.Multiple-choice answers use exact matching, numerical answers use Mean Relative Accuracy, and format validity requires executable operations.
4 Experiment
VILASR is evaluated across five benchmarks covering image, video, and multi-view spatial reasoning, with ablations examining its staged training framework and inference-time behavior. Results show strong performance across task types and distinct contributions from cold-start training, reflective sampling, and reinforcement learning.
- Evaluation setup: Five benchmarks cover image spatial reasoning, video temporal tracking, and multi-view spatial integration using accuracy and Mean Relative Accuracy.The evaluation includes Maze, SpatialEval-Real, VSI-Bench, and multi-view reasoning benchmarks.
- Main results: VILASR achieves strong performance on maze navigation and static spatial understanding through iterative drawing that tracks spatial state transitions.The model breaks multi-step navigation into interpretable visual steps, whereas comparison methods have tool or revision limitations.
- Main results: VILASR achieves state-of-the-art performance on all video and multi-view benchmarks, surpassing significantly larger open-source models such as LLaVA-OneVision-72B.The paper attributes this result to flexible visual operations for dynamic spatial relationships.
- Ablation study: Cold-start training is essential: the untrained drawing-enabled model performs worse than the Qwen2.5-VL-7B backbone, while Stage 1 improves all subtasks.The ablation indicates that prompting alone does not produce sophisticated spatial reasoning capabilities.
- Ablation study: 96.5% decrease in reflection pattern exhibition after removing reflective rejection sampling accompanies fewer reasoning steps and altered bbox-versus-line usage.Without Stage 2, precise localization and measurement tasks decline, while simpler directional or categorical tasks change little or improve slightly.
- Ablation study: Without reinforcement learning, drawing usage rises by 159.4% for Tbox and 9.1% for Tline, while precise numerical tasks show larger performance gaps.The paper interprets dense numerical rewards as encouraging selective drawing and more effective measurement learning.
5 Conclusion
The paper concludes that visual drawing operations provide a principled way to enhance spatial reasoning in LVLMs. Its three-stage framework is validated across diverse benchmarks, while future work targets more complex 3D relationships and efficient training.
- Conclusion: VILASR enhances LVLM spatial reasoning through direct visual manipulation using the “drawing to reason in space” paradigm.The approach is presented as bridging text-centric reasoning and human-like spatial cognition.
- Conclusion: The three-stage training framework cultivates sophisticated reasoning patterns and shows particular strengths in maze navigation and temporal-spatial understanding.The conclusion summarizes validation across diverse spatial reasoning benchmarks.
- Conclusion: Future directions include extending drawing operations to more complex 3D spatial relationships and developing more efficient training strategies.These directions are stated as remaining areas for exploration.
B Dataset construction
The dataset construction process assembles spatial reasoning data across maze, static-image, video, temporal, and multi-view settings. It produces separate datasets for cold-start training, reflective rejection sampling, and reinforcement learning.
- Dataset construction: The data span maze navigation, static image understanding, and video comprehension with temporal dynamics and multi-view spatial reasoning.Static data use bounding boxes and auxiliary lines, while video data extend these capabilities across time and viewpoints.
- Dataset construction: Three datasets support the training stages: VILASR-ColdStart-33k, VILASR-RRS-8k, and VILASR-RL-40k.They contain curated reasoning paths, reflective rejection sampling data, and reinforcement-learning data, respectively.
- Dataset construction: Table 2 reports dataset distribution across the different training stages.The passage identifies the table’s role but does not provide its individual counts.
- Dataset construction: Source data include GQA, VSR, OpenImages, OpenSpaces, and SpaceLLaVA, while mazes are procedurally generated from 3×3 to 6×6 grids.Generated mazes use depth-first search and balance grid sizes approximately equally.
C.1 Baseline implementation
The baseline evaluation compares VILASR with proprietary LVLMs, open-source models, and specialized multimodal reasoning systems. Models are evaluated using official APIs, checkpoints, prompts, or standard zero-shot formats according to their category.
- Baseline implementation: Closed-source LVLMs are queried through official APIs with zero-shot prompts that require direct answers with or without intermediate reasoning.Multiple-choice outputs use option letters, while numerical responses use single numeric values.
- Baseline implementation: Open-source LVLMs are evaluated zero-shot using their standard prompting formats across models ranging from 7B to 72B parameters.Kimi-VL-A3B-Instruct has 16.0B total parameters and activates 2.8B during inference.
- Baseline implementation: Specialized models use their officially released checkpoints and prompts, including each method’s designated perception or video-processing pipeline.COGCOM uses built-in perception tools, VisCoT uses bounding-box annotation, and SpaceR uses video understanding capabilities.
C.3 Detailed results on VSI-Bench
VILASR achieves strong VSI-Bench performance, especially on precise measurement, localization, comparative spatial reasoning, and appearance-order tasks, while remaining weaker on holistic global inference.
- Overall performance: 45.4% average accuracy, outperforming all baselines by +12.7% on VSI-Bench.The largest gains occur in Object Count (+29.0%), Absolute Distance (+15.0%), and Object Size (+13.0%).
- Comparative spatial reasoning: VILASR improves Relative Distance by +16.1% and Relative Direction by +20.7%.The results are attributed to drawing auxiliary lines that connect and measure between objects.
- Temporal-spatial reasoning: Appearance Order improves by +19.8%, highlighting VILASR’s ability to track sequential relationships across frames.Systematic object annotation across multiple frames supports temporal-spatial reasoning.
- Limitations: Room Size decreases by -9.9% and Route Plan by -2.1%, indicating limitations on holistic reasoning tasks.These tasks require integrating partial views, unseen spaces, multiple viewpoints, and global paths.
C.4 Statistical significance analysis
Paired t-tests show that VILASR significantly outperforms its base model across five spatial reasoning benchmarks, supporting consistency across evaluated scenarios.
- Statistical significance: VILASR significantly outperforms Qwen2.5-VL-7B on MAZE, SpatialEval-Real, VSI-Bench, SPAR-Bench, and MMSI-Bench.The paired t-tests report p < 0.05 for every listed benchmark.
- Evaluation setup: The prompt templates support both cold-start reasoning-path generation and VILASR inference.The framework includes system and user prompts, with distinct initial-query, follow-up, and final-answer prompts.
- Evaluation setup: After each visualization-tool use, the model must wait for and analyze the resulting visualization feedback.This prompt-level requirement directs iterative refinement during spatial reasoning.
D Visualization results
Visualization examples show VILASR decomposing maze, multi-view, and video spatial tasks into interpretable drawing-supported steps, with reflection and visual verification aiding its solutions.
- MAZE: In maze navigation, VILASR verifies each movement with auxiliary lines and reaches the correct destination.Qwen2.5-VL-7B answers incorrectly, while GPT-4o’s textual reasoning fails to track spatial transitions accurately.
- SPAR-Bench: In multi-view reasoning, VILASR maps objects across images and verifies perspective changes to determine observer movement.Both baseline models struggle with perspective transformations and misjudge the movement direction.
- VSI-Bench: In video reasoning, VILASR self-corrects after failed telephone localization and measures its size using reference objects.It identifies the telephone in image 13 and converts pixel measurements to centimeters using a monitor and headphones as scale references.
- Cross-task interpretation: Drawing operations make spatial reasoning more reliable by grounding abstract relationships in concrete visual representations.The examples cover maze navigation, multi-view reasoning, and video spatial reasoning.
E.1 Theoretical time complexity
The complexity analysis compares direct generation, standard text reasoning, and VILASR’s iterative visual drawing-and-thinking approach using reasoning-path length, multimodal context, and reasoning steps.
- Complexity formulation: The analysis compares three variants: direct-answer generation, text reasoning followed by answer derivation, and iterative visual drawing and thinking.M denotes input length, N answer length, L per-step reasoning-path length, and S reasoning steps; S = 1 for the first two variants.
- Complexity comparison: VILASR decomposes reasoning into interpretable visual operations with smaller per-step reasoning paths than single-step approaches.The comparison is presented in Table 6’s time-complexity analysis.
- Complexity formulation: The total computational complexity is expressed through a formula involving reasoning-step terms, path lengths, multimodal context, and answer length.L_k is the reasoning-path length at step k, while M_k is the additional multimodal context incorporated at that step.
- Empirical efficiency: The analysis also evaluates empirical runtime and average reasoning steps across benchmarks.Table 7 reports these measurements for VILASR.
F Limitations
The paper identifies limitations in task coverage, spatial dimensionality, and training accessibility, alongside potential societal risks and deployment concerns.
- Research scope: Training focuses on multiple-choice and numerical questions, excluding free-form scenarios such as detailed motion trajectory analysis.This constrains performance on open-ended spatial reasoning tasks.
- Representation limits: The 2D visual interface may inadequately capture complex 3D relationships and viewpoint changes, affecting tasks with dynamic camera movements.
- Computational cost: The three-stage training pipeline, especially reinforcement learning, may be inaccessible to researchers with limited computational resources.
- Societal implications: Improved spatial reasoning could benefit robotics, autonomous navigation, assistive technologies, and education, but drawing-based interaction may exclude users without suitable visual interfaces.
- Responsible deployment: Enhanced spatial reasoning could be misused for surveillance and privacy-invasive tracking, motivating usage guidelines, access controls, and privacy-preserving research.