Source-linked AI summary
OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning
Zhaochen Su, Linjie Li, Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjie Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, Yu Cheng
TL;DR
Tool-augmented LVLMs lack standardized infrastructure and struggle to learn adaptive policies from static demonstrations. OpenThinkIMG unifies tool integration, trajectory generation, and training, while V-ToolRL learns tool-selection and sequencing policies through interaction feedback. On chart reasoning, V-ToolRL improves substantially over SFT initialization and supervised tool-learning baselines.
Problem
Tool-augmented LVLMs lack standardized infrastructure for integrating diverse vision tools, generating interaction data, and training robust agents, while SFT struggles with unseen tools, tasks, exploration, and dynamic adaptation.
Method
OpenThinkIMG provides unified tool interfaces, scalable trajectory generation, distributed tool deployment, and training infrastructure, while V-ToolRL learns adaptive tool-selection and sequencing policies from sampled tool outcomes and reward feedback.
Results
59.39% accuracy on CHARTGEMMA was achieved by V-ToolRL, compared with 45.67% for QWEN-SFT and 30.50% for TACO-8B.
Takeaways & Limitations
V-ToolRL outperforms SFT initialization and supervised tool-learning methods on chart reasoning while learning adaptive tool usage through direct interaction and reward feedback.
Abstract
from arXiv · showhide
While humans can flexibly leverage interactive visual cognition for complex problem-solving, enabling Large Vision-Language Models (LVLMs) to learn similarly adaptive behaviors with visual tools remains challenging. A significant hurdle is the current lack of standardized infrastructure, which hinders integrating diverse tools, generating rich interaction data, and training robust agents effectively. To address these gaps, we introduce OpenThinkIMG, the first open-source, comprehensive end-to-end framework for tool-augmented LVLMs. It features standardized vision tool interfaces, scalable trajectory generation for policy initialization, and a flexible training environment. Furthermore, considering supervised fine-tuning (SFT) on static demonstrations offers limited policy generalization for dynamic tool invocation, we propose a novel reinforcement learning (RL) framework V-ToolRL to train LVLMs to learn adaptive policies for invoking external vision tools. V-ToolRL enables LVLMs to autonomously discover optimal tool-usage strategies by directly optimizing for task success using feedback from tool interactions. We empirically validate V-ToolRL on challenging chart reasoning tasks. Our RL-trained agent, built upon a Qwen2-VL-2B, significantly outperforms its SFT-initialized counterpart (+28.83 points) and surpasses established supervised tool-learning baselines like Taco and CogCom by an average of +12.7 points. Notably, it also surpasses prominent closed-source models like GPT-4.1 by +8.68 accuracy points. We hope OpenThinkIMG can serve as a foundational framework for advancing dynamic, tool-augmented visual reasoning, helping the community develop AI agents that can genuinely "think with images".
1 Introduction
OpenThinkIMG addresses infrastructure and policy-learning gaps that limit tool-augmented LVLMs. It combines standardized tool integration, scalable trajectory construction, and reinforcement learning for adaptive visual-tool use.
- Motivation: Human visual reasoning uses sketches, highlights, and spatial cues to externalize, decompose, and manipulate complex information.These mechanisms motivate integrating visual tools into the LVLM reasoning loop.
- Challenges: SFT-centric tool-augmented approaches rely on orchestrated tool-use sequences from static datasets, limiting holistic learning across the tool-use lifecycle.The introduction identifies heterogeneous tool definitions and interfaces as a further challenge.
- Approach: OpenThinkIMG unifies standardized heterogeneous tool interfaces, scalable tool-use trajectory generation, and efficient multimodal-agent training.The framework is presented as an end-to-end foundation for tool-augmented LVLMs.
- Approach: Its open and extensible framework includes a unified registry, distributed tool inference, an integrated training pipeline, and V-ToolRL for adaptive tool use.The trajectory pipeline supports supervised fine-tuning and reinforcement learning through planning, tool-call completion, rationale parsing, filtering, validation, and human oversight.
2 OPENTHINKIMG Framework
OPENTHINKIMG provides an end-to-end infrastructure for integrating, deploying, and invoking heterogeneous vision tools with LVLMs. Its V-TOOLRL training pipeline combines supervised cold-start initialization with reinforcement learning for adaptive, iterative tool use.
- Vision tool deployment and inference: The controller feeds tool outputs back into the LVLM context, enabling iterative multi-step reasoning and final response generation.Planned actions specify tools and arguments derived from the LVLM reasoning state and the question-image input; multiple outputs can be aggregated.
- Framework overview: OPENTHINKIMG unifies vision-tool and backbone-model integration, scalable trajectory synthesis, distributed deployment, and training in one end-to-end framework.The framework includes a registry, deployment modules, and an integrated pipeline spanning supervised fine-tuning and V-TOOLRL.
- Vision tool deployment and inference: Distributed containerized services let each vision tool be independently allocated, updated, and isolated while a Tool Controller manages invocation.Each tool service listens on a dedicated local network port, and the controller handles registration, health monitoring, and inference-time orchestration.
- Cold-Start for vision tool invocation: Cold-Start trains the model with supervised fine-tuning to generate complete action-output trajectories conditioned on each question and image.Minimizing the cross-entropy objective produces a sequential vision-tool invocation policy used as the foundation for reinforcement learning.
- Reinforcement learning for adaptive tool usage: V-TOOLRL samples groups of action trajectories, executes them through vision tools, and uses final-answer quality plus intermediate outputs to learn adaptive tool selection and sequencing.Group-relative advantages are optimized with GRPO, while sampled tool outcomes enter state and reward computation.
3 Vision Trajectory Construction
OpenThinkIMG constructs high-quality vision-tool trajectories through planned action generation, batched tool execution, and multi-stage filtering. The resulting data pairs planned actions with tool outputs while removing malformed or logically inconsistent trajectories.
- Trajectory Construction: Trajectory construction uses three steps: initial symbolic action planning, batched vision-tool invocation, and trajectory-quality filtering.The process begins by generating an action plan, executes corresponding tools, and then filters the resulting trajectories.
- Action Planning: GPT-4o generates an initial action plan for each question-image example using few-shot task decomposition.The plan identifies and schedules required tool actions without executing them, guided by five demonstrations.
- Tool Invocation: Planned vision tools are batch-invoked through a tool server to obtain rollout outputs for each trajectory.Tool responses are parsed and aligned with the planned actions.
- Tool Invocation: Cached outputs are processed in parallel batches of up to B = 128, producing a complete reasoning chain that pairs each planned action with its tool result.The paired action-output chain forms the completed rationale for each example.
- Trajectory Filtering: Malformed JSON, missing outputs, failed rule-based checks, and logical inconsistencies cause trajectories to be discarded.Checks evaluate final answers and intermediate rationales, including bounding-box consistency, mask coverage, and OCR accuracy.
4 Chart Reasoning Experiments
Experiments on CHARTGEMMA evaluate V-TOOLRL against tool-augmented and multimodal baselines, training-stage ablations, learning dynamics, and qualitative cases. V-TOOLRL combines vision-integrated reinforcement learning with adaptive tool invocation, yielding stronger accuracy, efficiency, and interpretable reasoning.
- Main Results: 59.39% accuracy on CHARTGEMMA surpasses TACO-8B at 30.50%, COGCOM-13B at 15.07%, and GPT-4.1 at 50.71%.V-TOOLRL uses a 2B-parameter Qwen2-VL base, while TACO-8B and COGCOM-13B use larger models.
- Training Stages & Methods: +29.83 points over QWEN-BASE and +13.72 points over QWEN-SFT demonstrate gains from cold-start initialization and subsequent V-TOOLRL training.V-TOOLRL reaches 59.39%, compared with 29.56% for QWEN-BASE and 45.67% for QWEN-SFT.
- Training Stages & Methods: +7.76 points over TEXT-BASED RL highlights the contribution of integrating visual tool outputs directly into reinforcement learning.TEXT-BASED RL achieves 51.63% accuracy, whereas full V-TOOLRL achieves 59.39%.
- Tool Invocation Efficiency: Tool calls decrease from about 0.63 to roughly 0.10–0.12 per sample as training progresses, indicating more selective tool invocation.The decrease stabilizes around 250–300 training steps and is interpreted as reduced redundant tool use.
- Reasoning Complexity and Learning Dynamics: Average completion length rises from approximately 66 to 83–86 tokens, while V-TOOLRL maintains higher and faster-improving reward accuracy than TEXT-BASED RL.The length plateaus around 400–450 steps, and the reward-accuracy gap remains through the training plateau.
- Qualitative Case Studies: Qualitative cases show tool decomposition correcting dense-chart errors: V-TOOLRL computes a 15.0% difference and identifies a three-way tie where GPT-4.1 does not.The cases use ZOOMINSUBFIGURE, OCR, POINT, and DRAWVERTICALLINEBYX for targeted visual grounding.
5 Related Work
Related work shows that LVLMs have advanced multimodal understanding but still struggle with intricate visual reasoning and precise visual interaction. Tool-augmented approaches address these challenges, yet heterogeneous interfaces and SFT-based training limit standardization, adaptability, and generalization.
- Large Vision-Language Models (LVLMs): LVLMs have progressed from image-caption pretraining and vision-language alignment to instruction-tuned families such as Qwen-VL and LLaVA.Despite these advances, they often falter on multi-step visual reasoning and precise interaction with visual content.
- Integrating External Tools with LVLMs: External tools extend LVLMs with dedicated capabilities including OCR, calculation, grounding, and knowledge retrieval.Recent frameworks train tool interaction using synthetically generated execution traces and supervised fine-tuning.
- Integrating External Tools with LVLMs: Heterogeneous tool definitions and interfaces hinder standardization and reproducibility across tool-augmented LVLM research.The related work identifies the lack of a unified approach to tool definition and interfacing as a significant challenge.
- Integrating External Tools with LVLMs: SFT-trained tool-use policies often have limited adaptability and generalization to novel scenarios.This limitation is attributed to reliance on synthetically generated execution traces and supervised training.
6 Conclusion
The paper presents OpenThinkIMG and V-ToolRL to standardize vision-tool integration and train adaptive invocation policies beyond static supervised trajectories. Chart-reasoning experiments show improved performance over SFT initialization and supervised tool-learning methods, alongside efficient tool usage.
- 6 Conclusion: OpenThinkIMG standardizes tool integration and facilitates training for tool-augmented LVLMs.V-ToolRL is integrated as a reinforcement learning framework for learning adaptive tool invocation policies.
- 6 Conclusion: V-ToolRL optimizes tool selection and sequencing through direct interaction and reward feedback rather than mimicking static trajectories.This approach targets dynamic utilization of external vision tools.
- 6 Conclusion: Chart-reasoning experiments show that V-ToolRL significantly improves performance over SFT initialization and existing supervised tool-learning methods.The experiments also report efficient tool usage and robust interactive reasoning capabilities.
- 6 Conclusion: OpenThinkIMG and V-ToolRL are proposed as community resources for research on adaptive multimodal agents and interactive visual reasoning.The paper states that the framework is intended to accelerate further research in this area.
A Prompts for Synthetic Trajectory Generation
Figure 5 presents the prompt used to generate high-quality tool-use trajectories.
- A Prompts for Synthetic Trajectory Generation: Figure 5 shows the prompt used for generating high-quality tool-use trajectories.
[BEGIN OF GOAL]
The prompt defines a visual assistant that solves chart-based reasoning questions using its own capabilities or external tools.
- [BEGIN OF GOAL]: The prompt instructs a visual assistant to generate and solve steps for chart-based reasoning.The assistant may answer using its own capabilities or external tools.
- [BEGIN OF GOAL]: The prompt provides an available-actions interface for the chart-reasoning assistant.The supplied excerpt ends at the available-actions specification.
[BEGIN OF ACTIONS]
The framework provides vision tools for extracting text, locating points, zooming into subfigures, and segmenting regions around points.
- OCR extracts text from images, including axis labels or annotations, and returns the extracted text.The tool may return inaccurate or unordered text, and returns an empty string when no text is present.
- Point identifies a described location in an image and returns its coordinates.The input combines an image with a description such as an x-axis value.
- ZoomInSubfigure crops an image to focus on a specified subfigure.It is intended for examining smaller areas of interest.
[BEGIN OF TASK INSTRUCTIONS]
The instructions organize visual tool use as sequential, logically combined actions in strict JSON, with demonstrations showing outputs informing later reasoning and answers.
- Actions are selected from the available tools, executed one at a time, and chained so each output informs the next step.The instructions also permit an empty actions array when no action is required.
- A Point action returns coordinates rather than a new image, whereas drawing a line produces a modified image for subsequent reasoning.The example locates (0.75, -0.25), then draws a vertical line through x=12.6 and produces img2.
- The required output is strict JSON containing a thought string and an actions array with tool names and arguments.This format is explicitly delimited by format instructions.
- OCR outputs values that are compared to identify the maximum, yielding Year 3 with a value of 35.The extracted sequence is Year 1: 15, Year 2: 25, Year 3: 35, Year 4: 30, Year 5: 10.
- The trajectory-generation prompt uses explicit tool definitions and demonstration examples to guide high-quality tool-use outputs.Figure 5 describes the prompt as including definitions of alternative tools and three demonstrations.