Source-linked AI summary
VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use
Mingyuan Wu, Jingcheng Yang, Jize Jiang, Meitang Li, Kaizhuo Yan, Hanchao Yu, Minjia Zhang, Chengxiang Zhai, Klara Nahrstedt
TL;DR
Existing VLM RFT methods generally produce text-only reasoning from static images, while inference-time visual reasoning lacks training mechanisms. VTool-R1 trains VLMs to interleave text with tool-generated visual steps using outcome-based rewards, improving structured visual reasoning and multimodal chain generation, though its demonstrated setting is limited to selective attention in structured image reasoning.
Problem
Existing VLM RFT methods remain text-driven on fixed images, while visual-step inference methods do not train models to generate multimodal reasoning.
Method
VTool-R1 integrates Python-based visual editing tools into RFT and optimizes final-answer outcomes so VLMs learn when and how to use intermediate visual reasoning steps.
Results
VTool-R1 improves final reasoning accuracy and enables coherent multimodal chains of thought with intermediate image-based reasoning steps on structured visual tasks.
Takeaways & Limitations
Outcome-based RFT can teach VLMs strategic visual tool use without process-level supervision, supporting reasoning that combines images and text.
Takeaways & Limitations
The work focuses on selective attention in structured image reasoning as a starting point, rather than a broader range of tasks and toolsets.
Abstract
from arXiv · showhide
Reinforcement Learning Finetuning (RFT) has significantly advanced the reasoning capabilities of large language models (LLMs) by enabling long chains of thought, self-correction, and effective tool use. While recent works attempt to extend RFT to vision-language models (VLMs), these efforts largely produce text-only reasoning conditioned on static image inputs, falling short of true multimodal reasoning in the response. In contrast, test-time methods like Visual Sketchpad incorporate visual steps but lack training mechanisms. We introduce VTool-R1, the first framework that trains VLMs to generate multimodal chains of thought by interleaving text and intermediate visual reasoning steps. VTool-R1 integrates Python-based visual editing tools into the RFT process, enabling VLMs to learn when and how to generate visual reasoning steps that benefit final reasoning. Trained with outcome-based rewards tied to task accuracy, our approach elicits strategic visual tool use for reasoning without relying on process-based supervision. Experiments on structured visual question answering over charts and tables show that VTool-R1 enhances reasoning performance by teaching VLMs to "think with images" and generate multimodal chain of thoughts with tools. To support future research in multi-turn multi-modal reasoning, we open-source our code at https://github.com/VTOOL-R1/vtool-r1
1 INTRODUCTION
VTool-R1 addresses the gap between text-dominant VLM reasoning and multimodal reasoning with learned intermediate visual steps. It integrates visual tools into RFT so models can strategically combine images and text for structured visual tasks.
- Recent RFT adaptations for VLMs generate purely textual reasoning conditioned on fixed image inputs, without intermediate visual reasoning steps.
- Text-dominant reasoning can exploit language shortcuts, producing plausible answers that conflict with the image evidence.The paper illustrates this with a six-finger hand being answered as having five fingers.
- Inference-time methods add visual reasoning steps but rely on highly capable models and underperform with less capable open-source models.
- VTool-R1 trains VLMs to interleave textual chains of thought with intermediate visual steps generated through Python-based image editing tools.
- Outcome-based rewards teach models when and how to use visual reasoning steps without explicitly rewarding tool generation.
- The framework is validated on structured image reasoning tasks involving tables and charts using controlled experiments and a predefined visual toolset.
- VTool-R1 is presented as the first work to enable VLMs to integrate intermediate visual reasoning steps into generated text-based chains of thought.
2 RELATED WORKS
Prior work introduced visual tool use and intermediate visual artifacts during inference, while RFT advanced tool-augmented reasoning for LLMs. VTool-R1 targets the remaining gap: training VLMs to reason over intermediate visual steps.
- Python-based visual tools have been used during inference to manipulate images for visual question answering without additional training.
- Visual Sketchpad and Refocus incorporate visual artifacts or edited images into iterative reasoning, but operate at inference time rather than training tool use.
- RFT methods evolved from RLHF and PPO toward critic-free GRPO, which improves efficiency and stability for policy optimization.
- Recent VLM RFT approaches primarily train textual chains of thought, leaving tool-mediated reasoning over intermediate visual steps largely unexplored.
3 VTOOL-R1
VTool-R1 extends VLM inference and RFT with iterative visual tool use, allowing models to create and reason over edited images. Its training optimizes final-answer outcomes while regularizing policy updates and avoiding process-reward hacking.
- VLM Inference and Rollout: VTool-R1 requires VLMs that can process both the original input image and intermediate visual steps as multiple image inputs.
- VTool-R1: The framework extends RFT so VLMs learn during rollouts to invoke visual editing tools and generate multimodal chains of thought.
- VLM Inference and Rollout: The inference prompt supplies tool descriptions and asks the model to reason step by step before producing Python-like actions or a final answer.
- VLM Inference and Rollout: Tool calls execute externally in Python, producing a modified image that is fed back as an additional input for another reasoning turn.
- VLM Inference and Rollout: If tools are used, the model reasons over the original and edited images; otherwise, it answers directly in a single round.
- RFT Training: Training optimizes only the final response y to improve final reasoning quality rather than jointly optimizing intermediate tool-invoking output y′.
- RFT Training: The GRPO-style objective uses group-normalized relative advantage and KL regularization, avoiding a critic while stabilizing reward-aligned updates.
- Reward Modeling: Outcome-based reward uses final-answer correctness, with a lightweight LLM judge assigning score 1 when the prediction matches the ground truth.
4 EXPERIMENT
Experiments evaluate VTool-R1 on chart and table reasoning with Python visual-editing tools. Results show improved tool use, reasoning accuracy, adaptive invocation, and important limitations in tool-call evaluation and execution.
- Experimental Setup: The evaluation covers chart and table reasoning, using Refocus data splits and Python tools that highlight, mask, or draw around relevant visual regions.Table tasks use VWTQ, VWTQ_syn, and VTabFact; chart tasks use selected ChartQA examples.
- Baselines: Direct inference without tools is strong for open-source models, often outperforming GPT-4o, especially at the 32B scale.The authors suggest this may reflect VQA post-training or distillation, alongside weaker initial general-purpose tool use.
- Qualitative Analysis: A qualitative 3B example integrates an intermediate visual step and reaches the correct answer, although failures include incorrect second-rollout inference and obstructive augmentations.One example focuses Egypt’s bar but produces conflicting approximate values before the ground-truth answer of 0.01.
- Main Results: After RFT, 3B and 7B models learn meaningful multimodal tool use and significantly outperform direct-inference baselines.The learned behavior includes generating intermediate reasoning steps rather than relying only on the initial image.
- Tool-Use Dynamics: VTool-R1 learns selective tool invocation: models may overuse tools early, then invoke them more cautiously and answer directly when tools are unnecessary.The 32B model has a higher overall tool-use rate, while both table and chart training show non-monotonic dynamics.
- Tool-Use Dynamics: Tool-call success is measured by exception-free Python execution and valid returned images, with table success increasing steadily but chart success fluctuating.The paper notes that this proxy lacks an oracle verifier and calls for human-annotated evaluation.
- Reward Design: Alternative reward designs fail differently: process penalties eliminate tool use, while extra success rewards encourage verifier exploitation.These observations motivate outcome-based reward design tied to final task accuracy.
- Comparative Results: VTool-R1’s added visual reasoning step outperforms general visual-reasoning models and reaches 80.7 versus Deepeyes’ 60.0 in the reported comparison.The paper attributes this result to tool design, training recipe, and structured image tasks.
5 CONCLUSION
VTool-R1 shows that outcome-based reinforcement learning can train VLMs to interleave textual and visual reasoning steps using visual editing tools. The work demonstrates improved structured visual reasoning while identifying selective attention as its initial scope.
- VTool-R1 trains VLMs to generate coherent multimodal chains of thought by interleaving textual reasoning with intermediate visual states.The framework integrates visual editing tools into the reinforcement-learning loop.
- Outcome-based rewards teach models when and how to use visual tools without requiring process-level supervision.
- Experiments on structured visual reasoning tasks improve final reasoning task accuracy while producing intermediate image-based reasoning steps.
- The study begins with selective attention in structured image reasoning and leaves broader tools, datasets, and multi-turn execution for future extensions.The authors identify this focused task setting as a starting point for expansion.
A APPENDIX
The appendix specifies Python image-editing tools and a step-by-step interaction protocol for focusing relevant table regions. It also documents representative successes and failure cases involving visual augmentation, tool execution, and answer accuracy.
- Visual editing tools: The tools focus relevant table rows or columns by highlighting, masking, or drawing around their bounding boxes.These operations return edited images for subsequent inspection.
- Tool-use protocol: The reasoning protocol asks the model to select relevant regions, execute Python actions, inspect visual outputs, and continue reasoning from them.The protocol permits decisions based on outputs from previous code blocks and requires a final answer after tool use.
- Failure cases: Incorrect outcomes arise when augmentation obstructs numbers, code execution fails, or the model answers without needed augmentation.Examples include a flawed bounding box, unsuccessful augmentation code, and a no-action rollout whose 21.28 answer conflicts with the 31.01 ground truth.
- Successful use: A successful example focuses the Upper income class column before computing a sum and average.The image is edited successfully in preparation for retrieving the relevant values.