Source-linked AI summary
Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs
Meng Lu, Ran Xu, Yi Fang, Wenxuan Zhang, Yue Yu, Gaurav Srivastava, Yuchen Zhuang, Mohamed Elhoseiny, Charles Fleming, Carl Yang, Zhengzhong Tu, Yang Xie, Guanghua Xiao, Hanrui Wang, Di Jin, Wenqi Shi, Xuan Wang
TL;DR
VLMs’ static visual representations and shallow cross-modal alignment limit fine-grained visual reasoning, while their tool selection and coordination remain inadequate. VISTA-Gym provides diverse tasks, visual tools, and an interactive training environment for VISTA-R1; across 11 benchmarks, VISTA-R1-8B outperforms similar-size open-source baselines by 9.51%–18.72%.
Problem
VLMs’ static visual embeddings and shallow cross-modal alignment limit fine-grained visual reasoning, and open-source VLMs remain inadequate at effectively selecting, invoking, and coordinating visual tools.
Source: §1 · p. 1Method
VISTA-Gym unifies diverse visual reasoning tasks, a standardized interface for visual tools, executable interaction loops, and verifiable feedback to train VISTA-R1 to interleave tool use with reasoning.
Source: §1 · pp. 1–2Results
Across five in-domain and six out-of-domain reasoning-intensive VQA benchmarks, VISTA-R1-8B outperforms similar-size open-source baselines by 9.51%–18.72%.
Source: §1 · p. 2Takeaways & Limitations
VISTA-Gym is an effective and scalable solution for developing VLM agents that interleave reasoning and tool use on complex, multi-step visual problems.
Source: §1 · p. 2
Abstract
from arXiv · showhide
While recent vision-language models (VLMs) demonstrate strong image understanding, their ability to "think with images", i.e., to reason through multi-step visual interactions, remains limited. We introduce VISTA-Gym, a scalable training environment for incentivizing tool-integrated visual reasoning capabilities in VLMs. VISTA-Gym unifies diverse real-world multimodal reasoning tasks (7 tasks from 13 datasets in total) with a standardized interface for visual tools (e.g., grounding, parsing), executable interaction loops, verifiable feedback signals, and efficient trajectory logging, enabling visual agentic reinforcement learning at scale. While recent VLMs exhibit strong text-only reasoning, both proprietary and open-source models still struggle with tool selection, invocation, and coordination. With VISTA-Gym, we train VISTA-R1 to interleave tool-use with agentic reasoning via multi-turn trajectory sampling and end-to-end reinforcement learning. Extensive experiments across 11 public reasoning-intensive VQA benchmarks show that VISTA-R1-8B outperforms state-of-the-art baselines with similar sizes by 9.51%-18.72%, demonstrating VISTA-Gym as an effective training ground to unlock the tool-integrated reasoning capabilities for VLMs.
1. Introduction
VLMs’ static visual representations and shallow cross-modal alignment limit fine-grained visual reasoning, while existing tool-integrated approaches and training environments often cover narrow tasks or domains. VISTA-Gym combines diverse visual reasoning tasks and tools in an interactive training environment; VISTA-R1-8B outperforms comparable-size open-source baselines across in- and out-of-domain benchmarks.
1. Introduction
- VLMs’ static visual embeddings and shallow cross-modal alignment limit their ability to capture fine-grained structures, spatial relations, and quantitative dependencies in real-world scenes.The paper motivates reasoning tightly coupled with visual perception as a way to support step-by-step visual reasoning.Source: §1 · p. 1
- Open-source VLMs remain inadequate at selecting, invoking, and coordinating visual tools, while existing tool-use studies often focus on narrow tasks and open-domain visual reasoning remains underexplored.Figure 1 reports that direct tool augmentation can reduce accuracy and intrinsic reasoning yields limited gains on complex VQA; supplying tool-selection priors and interleaving reasoning with tool execution improve performance.Source: §1 · p. 1, Figure 1 · p. 2
- VISTA-R1-8B surpasses comparable-size open-source baselines by 9.51%–18.72% across five in-domain and six out-of-domain reasoning-intensive VQA benchmarks.VISTA-Gym is described as a scalable solution for developing agents that interleave reasoning and tool use on complex, multi-step visual problems.Source: §1 · p. 2
- VISTA-Gym provides seven reasoning tasks across 13 public datasets and a standardized interface for 26 tools covering perception, symbolic manipulation, charts, and documents.Its infrastructure supports multithreading, parallel execution, sequential sampling, and large-scale evaluation.Source: §1 · p. 2
2. Related Works
Prior work improves VLM reasoning through distilled reasoning chains and reinforcement learning, while tool-integrated approaches add visual search, zoom, or auxiliary-model interactions. Existing systems and environments remain limited in tool or task scope, and most agent-training environments are text-only.
RL for VLM Reasoning
RL for Tool-Integrated Reasoning in VLMs
- Tool-integrated VLM approaches use image search, zoom-in operations, or auxiliary VLMs, but most rely on a single specialized tool and narrow task domains.DeepMMSearch-R1 [48] and MMSearch-R1 [74] use image search; [16] [66] [89] [90] use zoom-in; ReLook [30] uses a VLM as an auxiliary tool.Source: §2 · p. 2
RL Training Environment for Agentic Reasoning
- Most agent-training environments are text-only; the two identified VLM environments focus on compositional visual games or embodiment tasks, while VISTA-Gym targets tool-integrated RL for visual reasoning.The passage names VLM-Gym [9] and VAGEN [70] and describes their respective focuses.Source: §2 · p. 2
3. Preliminaries
The paper frames visually grounded question answering as generating a trajectory of interleaved reasoning and tool actions, with observations and feedback arriving through interaction. Exploratory experiments motivate training for tool-integrated reasoning, and the paper introduces VISTA-Gym as the environment for that work.
3. Preliminaries
- Tool-integrated reasoning is framed as a partially observable Markov decision process in which the agent produces a thought before each action and receives a new partial observation after the environment transitions.The trajectory comprises reasoning steps, tool actions, observations, and a final answer.Source: §3 · pp. 2–3
- In exploratory experiments, naively enabling tools sharply reduces base-model accuracy, and most annotated failures involve tool-call decisions, schema or argument selection, or correctness, followed by incorrect post-tool reasoning.The analysis covers 500 tool-enabled errors across GPT-5 and InternVL3-8B.Source: §3 · p. 3
- VISTA-Gym curates verifiable visual-centric tasks with tools, provides an executable interactive environment and scalable training facilities, and supports training an open-source VLM agent to interleave reasoning with tool execution.The passage points to task curation in Sec. 4.1, the environment in Sec. 4.2, scalable facilities in Sec. 4.3, and agent training in Sec. 5.Source: §3 · p. 3
A Scalable Tool-Integrated Agentic Training Environment for VLMs
VISTA-Gym combines diverse reasoning tasks with an extensible tool interface, an executable interaction loop, and infrastructure designed to support scalable agent training.
4.1. Diverse Task and Tool Collection
- VISTA-Gym curates verifiable instances from 13 benchmarks across seven task axes, spanning perception, symbolic manipulation, and language understanding, and exposes 26 tools in four families through a standardized interface.The tool families are perception, chart understanding, diagram formalization, and math solvers.Source: §4.1 · p. 3
4.2. Executable Interactive Environment
- The Gymnasium-style API initializes an episode with the question and supporting image, constrains actions to typed tool calls, and returns execution results or runtime errors as observations.The feedback can support verification of intermediate hypotheses and adjustment of subsequent reasoning.Source: §4.2 · pp. 3–4
4.3. Scalable Training Facility
- VISTA-Gym scales training through concurrent tool services and asynchronous batched requests, with Ray managing queues and load balancing and a plug-and-play interface supporting new tools.Compute-heavy VLM tools use dedicated GPUs; lightweight utilities share CPUs.Source: §4.3 · p. 4
5. VISTA-R1: Bootstrapping Tool-Integrated Reasoning via RL for VLMs
VISTA-R1 uses a two-stage training framework: behavioral cloning establishes interleaved thought–action behavior, then multi-turn online RL refines reasoning and tool orchestration. Its reward design enforces a think→tool_call→answer protocol and rewards repetition-free, well-formed, correct outputs.
5.1. Two-Stage Training Framework
- Behavioral cloning bootstraps the policy on synthesized expert trajectories that interleave thoughts and actions.Candidates are generated with GPT-5 and retained when their final answers exactly match ground truth; concise rationales are expanded using Qwen3-VL-235B-A22B-Thinking.Source: §5.1 · p. 4
- The SFT corpus uses a diverse tool mix, providing a prior for tool syntax and selection.Figure 3 shows the top tool-call distribution across tasks in the SFT data.Source: §5.1 · p. 4, Figure 3 · p. 5
- Online RL runs multi-turn rollouts in an executable environment, appending structured tool feedback to the context before continued reasoning.The policy is improved with GRPO [60] using group-normalized advantages.Source: §5.1 · p. 4
5.2. Reward Design
- The interaction protocol requires reasoning followed by a tool call on intermediate turns and reasoning followed by an answer on the final turn.The protocol specifies the think→tool_call→answer sequence.Source: §5.2 · p. 5
- A repetition detector assigns severity-dependent penalties for contiguous token, phrase, or character repeats before subsequent reward logic is applied.The detector's possible rewards are −3.0, −2.0, −1.5, or 0.Source: §5.2 · p. 5
- The rollout reward sums repetition, format, and correctness components, with correctness credited only for repetition-free, well-formed outputs.The paper describes the design as sparse and format-aware, rewarding generations that are repetition-free, structurally valid, and correct.Source: §5.2 · p. 5
6. Experiments
Across eleven reasoning-intensive VQA benchmarks, VISTA-R1-8B outperforms similarly sized baselines, while ablations show stronger performance when reasoning and tool use are interleaved. Additional studies examine training mixtures, curricula, error patterns, and trajectory quality.
6.1. Experiment Setups
- Evaluation covers five in-distribution datasets with held-out test splits and six additional out-of-distribution benchmarks; baselines generally lack tool access.The experiments use accuracy as the primary metric and evaluate VISTA-R1 with InternVL3-2B/8B/14B and Qwen2.5-VL-7B backbones.Source: §6.1 · p. 5
6.2. Main Experiment Results
- 9.51%-18.72% with tools and 2.03%-11.24% without tools: VISTA-R1-8B outperforms similarly sized baselines across eleven visual-reasoning benchmarks.VISTA-R1-8B also performs comparably to substantially larger proprietary models on out-of-distribution benchmarks.Source: Table 2 · p. 6, §6.2 · p. 6
6.3. Ablation Studies of VISTA-R1
- Full VISTA-R1 interleaves reasoning and tools, reaching 71.14%, versus 63.66% without tools and 48.40% without reasoning; SFT adds 3.46%, then RL gains 10.19% over SFT.Source: §6.3 · pp. 6–7
- Without RL, directly adding tools degrades vanilla VLM accuracy, while chain-of-thought alone yields only modest gains on complex VQA.Tool-selection priors improve performance, but gains vary by task and smaller open-source models remain particularly challenged.Source: §6.3 · p. 7, Table 3 · p. 7
6.4. Additional Studies of VISTA-Gym
- 69.54% → 71.27%: continuing RL on samples with prior pass rates in [0.125, 0.375] improves performance beyond the initial convergence point.The additional studies report that diverse task mixtures improve generalization, heterogeneous tool schemas broaden the learned action policy, and longer expert SFT trajectories contribute to higher post-RL performance.Source: §6.4 · p. 8, Figure 7 · p. 8
6.5. Error Analysis and Case Study
- Training within VISTA-Gym resolves most tool-call and reasoning failures observed in base models on the re-evaluated 500 samples.Case studies illustrate schema- and argument-correct calls, reasoning grounded in tool outputs, cross-tool generalization, and handling of imperfect outputs; human ratings cover 40 sampled trajectories per dataset on a 1–5 scale.Source: §6.5 · p. 8, Table 4 · p. 9
7. Conclusion
The conclusion presents VISTA-Gym as a scalable training ground for tool-integrated, visual-centric agentic RL and VISTA-R1 as an agent that interleaves multi-turn reasoning with structured tool use. It offers training recommendations and notes that richer stepwise semantics and broader tool ecosystems may benefit long-horizon tool-integrated reasoning.
7. Conclusion
- VISTA-Gym provides a scalable training environment for visual-centric agentic RL, and VISTA-R1 interleaves multi-turn reasoning with structured tool use.The environment combines unified interfaces, executable feedback, and efficient logging.Source: §7 · p. 9
- Practitioners may train policies to interleave reasoning and tools, warm up with SFT before online RL, diversify tasks and tools, and allocate more RL to hard-but-learnable slices via tail-patching.The recommendations target tool-call syntax, in-depth tool-integrated reasoning, action-space coverage, and difficult training slices.Source: §7 · p. 9
- The verifiers emphasize terminal correctness and structural validity; richer stepwise semantics and broader tool ecosystems may further benefit long-horizon tool-integrated reasoning.Source: §7 · p. 9
Supplementary Material
The supplementary material introduces dataset statistics for in-distribution and out-of-distribution data and describes a diverse in-distribution suite spanning multiple reasoning domains.
A. Task and Data Information
- The supplementary material presents statistics for the in-distribution and out-of-distribution datasets in Table 5.Source: A. Task and Data Information · p. 14
A.1. Information for In-Distribution Datasets
- The in-distribution suite covers fifteen datasets across eight reasoning domains, including chart, geometric, mathematical, geospatial, scientific, document, spatial, and general visual reasoning.Source: §A.1 · p. 14
- Tasks include quantitative chart extraction, diagram-grounded geometry, layout-aware document extraction, compositional 3D-scene reasoning, and open-ended questions requiring external knowledge.Source: §A.1 · p. 14
A.2. Information for Out-of-Distribution Datasets
The supplementary material describes six out-of-distribution benchmarks, VISTA-Gym's 26-tool collection, baseline systems, and a prompt template.
A.2. Information for Out-of-Distribution Datasets
- Six additional benchmarks assess distinct reasoning, including table-based math, diagram science QA, plot reading, 3D-scene arithmetic, abstract visual relations, and holistic mathematical reasoning.Source: §A.2 · p. 14
B. Toolset Information
- VISTA-Gym provides 26 tools across four categories, including perception, chart understanding, diagram formalization, and math solvers.The tools support operations such as detection, segmentation, OCR, structured chart extraction, symbolic diagram parsing, and multimodal mathematical reasoning.Source: B. Toolset Information · pp. 15–16
C. Baseline Details
- The listed baselines include systems using tool invocation with RL, active-perception policies, intermediate-step rewards, symbolic representations, and unified visual-task training.Source: C. Baseline Details · p. 16
D. Prompt Templates
- The prompt template is provided in Figure 9.Source: D. Prompt Templates · p. 16
E. Additional Method Details of VISTA-Gym
VISTA-Gym uses a Gymnasium-style reset-and-step interface for multi-turn tool interaction. Its system prompt specifies how agents reason, call tools, process outputs, and format final answers.
E. Additional Method Details of VISTA-Gym
- On reset, the environment samples a question-and-image task and returns an initial observation; each step supplies an observation and receives a thought–action pair.Source: E. Additional Method Details of… · p. 16
- The environment executes each action, transitions under the POMDP formalization, and returns the next observation to support multi-turn interaction.Source: E. Additional Method Details of… · p. 16
- The system prompt directs agents to reason before tool calls, analyze tool results afterward, and place final answers in answer tags.It also specifies a standard JSON tool-call format and instructs agents not to answer without tool results.Source: E. Additional Method Details of… · p. 16
- Figure 9 contains the prompt template.Source: Figure 9 · p. 16
F. Additional Method Details of VISTA-R1
VISTA-R1’s InternVL3 adaptation preserves the model’s native visual processing while accommodating distributed multimodal training. The error analysis distinguishes six failure types spanning tool-call construction, execution, and reasoning over results.
F. Additional Method Details of VISTA-R1
- The InternVL3 adaptation preserves raw image placeholders for its native processor and shards only language-decoder layers to support stable, efficient multimodal GRPO training.The processor expands placeholders into 256 visual-context tokens; attention masks and position IDs are adjusted for the extended sequence.Source: F. Additional Method Details of VISTA-R1 · p. 17
G.1. Detailed Error Types for Error Analysis
- The error taxonomy separates six failures, from malformed tool calls and arguments to incorrect interpretation of valid tool output.It distinguishes schema, argument-name, argument-value, tool-output-format, and post-execution reasoning errors.Source: §G.1 · p. 17
G.2. Additional Training Setup Details
Training uses bounded multi-turn interactions, asynchronous rollouts, and distributed GPU optimization across several model backbones.
G.2. Additional Training Setup Details
- Interactions allow at most three turns, retain conversation and tool logs, and process up to 24 trajectories concurrently.The setup balances exploration depth and computational cost while maximizing throughput.Source: §G.2 · p. 17
- Training runs on eight NVIDIA H200 GPUs with FSDP2, using asynchronous rollouts with batch size 128.Agent-side code execution uses Python 3.10, and a toolrouter service provides real-time execution and feedback.Source: §G.2 · p. 17
- The main experiments use InternVL3-2B, InternVL3-8B, InternVL3-14B, and Qwen2.5-VL-7B backbones.Ablations and additional analyses use InternVL3-8B.Source: §G.2 · p. 17