Source-linked AI summary
VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use
Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, Tianyu Pang, Wenhu Chen
TL;DR
RLVR-based LLM reasoning remains largely single-turn and disconnected from external tools, while existing ARLT systems are fragmented and difficult to extend across domains. VerlTool introduces a unified, modular framework with standardized multimodal tool APIs, upstream VeRL alignment, and asynchronous rollouts, achieving competitive performance across six ARLT tasks while providing unified infrastructure.
Problem
RLVR is limited to single-turn interactions, while existing ARLT codebases suffer from fragmented tool integration, synchronous rollout bottlenecks, and limited extensibility.
Method
VerlTool extends VeRL for multi-turn ARLT using standardized APIs for diverse tools, multimodal observations, and trajectory-level asynchronous rollout execution.
Results
VerlTool-trained models achieve competitive performance with specialized systems across six ARLT tasks, including 62.2% average mathematical-benchmark performance and 45.9% knowledge-QA accuracy.
Takeaways & Limitations
VerlTool provides unified infrastructure for studying and deploying tool-using agents across diverse tasks, modalities, and interaction settings.
Abstract
from arXiv · showhide
Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated success in enhancing LLM reasoning capabilities, but remains limited to single-turn interactions without tool integration. While recent Agentic Reinforcement Learning with Tool use (ARLT) approaches have emerged to address multi-turn tool interactions, existing works develop task-specific codebases that suffer from fragmentation, synchronous execution bottlenecks, and limited extensibility across domains. These inefficiencies hinder broader community adoption and algorithmic innovation. We introduce VerlTool, a unified and modular framework that addresses these limitations through systematic design principles. VerlTool provides four key contributions: (1) upstream alignment with VeRL ensuring compatibility and simplified maintenance, (2) unified tool management via standardized APIs supporting diverse modalities including code execution, search, SQL databases, and vision processing, (3) asynchronous rollout execution achieving near 2$\times$ speedup by eliminating synchronization bottlenecks, and (4) comprehensive evaluation demonstrating competitive performance across 6 ARLT domains. Our framework formalizes ARLT as multi-turn trajectories with multi-modal observation tokens (text/image/video), extending beyond single-turn RLVR paradigms. We train and evaluate models on mathematical reasoning, knowledge QA, SQL generation, visual reasoning, web search, and software engineering tasks, achieving results comparable to specialized systems while providing unified training infrastructure. The modular plugin architecture enables rapid tool integration requiring only lightweight Python definitions, significantly reducing development overhead and providing a scalable foundation for tool-augmented RL research. Our code is open-sourced at https://github.com/TIGER-AI-Lab/verl-tool.
1 INTRODUCTION
VERLTOOL addresses the limitations of isolated, single-turn LLM reasoning by providing a unified framework for multi-turn, tool-using, multimodal reinforcement learning. Its upstream alignment, standardized tool APIs, asynchronous rollouts, and six-task evaluation target reproducibility, extensibility, and efficient agentic training.
- RLVR strengthens long-context reasoning and supports behaviors such as reflection, backtracking, and multi-step reasoning.
- Single-turn LLM reasoning lacks external-world interaction and environmental feedback, contributing to brittle overthinking and hallucination.
- ARLT extends LLM training with multi-turn interactions using tools such as code execution, search, image manipulation, and domain-specific APIs.
- Existing ARLT systems face asynchronous rollout, fragmented tool management, and underdeveloped multimodal-support challenges.
- VERLTOOL builds on VERL to provide open-source, modular, multimodal ARLT training with upstream compatibility and standardized tool interaction.
- VERLTOOL supports code, search, SQL, and vision tools through lightweight Python definitions, while trajectory-level asynchronous execution yields over 2× rollout speedup.
- Across six ARLT tasks, VERLTOOL provides unified training infrastructure and competitive performance relative to specialized systems.
- The framework combines upstream-aligned RL infrastructure, unified tool integration, asynchronous execution, and diverse tasks to support scalable ARLT research and deployment.
2 RELATED WORK
Related work develops reinforcement learning for long-horizon tool use, but existing training frameworks vary in tool coverage and extensibility. Domain-specific systems and synchronous or limited integrations leave a need for comprehensive, deployable support across diverse tools and modalities.
- ARLT optimizes tool-calling policies using execution outcomes and environmental feedback in long-horizon, multi-turn interactions.
- ARLT requires credit assignment across sequential tool calls, explicit tool-response observation tokens, and failure-aware execution environments.
- Tool-using reinforcement learning has been applied to mathematical reasoning, information retrieval, NL2SQL generation, and visual reasoning.
- These applications require iterative environmental probing, adaptation to tool feedback, and strategy refinement beyond purely supervised approaches.
- Existing frameworks provide uneven tool coverage: traditional RL systems offer basic search and code support, while others emphasize core training or search with limited executors.
- SKYRL broadens tool support with bash and SQL executors but requires complex containerized environments that add deployment overhead.
- Task-specific systems use hard-coded integrations that limit extensibility and adaptability, motivating comprehensive frameworks with easier deployment and development.
3 VERLTOOL FRAMEWORK
VERLTOOL extends RLVR into multi-turn tool interaction and provides a modular framework for training agents with asynchronous execution, standardized tools, and multimodal observations.
- Agentic Reinforcement Learning with Tool Use: ARLT represents trajectories as alternating LLM actions and tool-returned observation tokens across multiple interaction steps.Unlike single-turn RLVR, tool observations enter the reasoning process and are masked during policy optimization because they are off-policy.
- Framework Design: VERLTOOL addresses fragmented ARLT infrastructure through a unified API, modular plugins, and separation of reinforcement learning from tool execution.Its architecture connects the VeRL Workflow and Tool Server, allowing tools to be registered through a common interface and managed independently.
- Framework Design: Asynchronous rollouts let each trajectory submit tool calls immediately after action generation instead of waiting for the whole batch.This design reduces idle GPU and CPU periods caused by synchronous batch-based execution.
- Framework Design: Separate tokenization of actions and observations preserves consistent rollout token prefixes across multi-turn interactions.Joint tokenization can merge boundary symbols differently, such as producing token IDs 29,198 versus 397 for the illustrated example.
- Framework Design: The framework evaluates tool-augmented training across six tasks, with training dynamics reported for AIME24, NQ, Spider-Test, VStar, GAIA, and SWE-Verified.Figure 5 notes that final evaluation performance may exceed training curves because train-evaluation settings differ, while action counts are averaged over sampled responses.
- Framework Design: VERLTOOL supports parallel tool execution through Python multithreading for lightweight calls and Ray-based asynchronous execution for resource-intensive tools.The Ray backend supports distributed and fault-tolerant execution across machines or GPU nodes.
4 EXPERIMENTS
VERLTOOL is evaluated across six ARLT tasks and diverse tool modalities, achieving competitive task-specific performance while revealing task-dependent tool-use dynamics and agentic behaviors.
- Evaluation scope: VERLTOOL evaluates six ARLT tasks spanning mathematical reasoning, knowledge QA, SQL, visual reasoning, agentic search, and software engineering.The framework supports diverse tools and modalities, including text, image, and video processing.
- Overall performance: VERLTOOL-trained models consistently match or exceed existing baselines across the six evaluated tasks.The reported results compare unified-framework training with specialized systems and divergent codebases.
- Benchmark results: 45.9% accuracy on VT-Search surpasses Search-R1 by 10.9%, while VT-VisualReasoner reaches 82.7% on V* Bench and VT-DeepSearch reaches 34.0% on GAIA.VT-SQL matches specialized systems such as SkyRL-SQL; mathematical results surpass expert models on AIME24, AMC23, and Olympiad Bench.
- Tool-use dynamics: Tool-use frequency varies substantially across tasks, from typically 1 ∼4 mathematical interactions to over 100 software-engineering interactions.Effective tool use depends on appropriate reward design and initialization strategies.
- Tool-use dynamics: VT-SQL tool usage decreases after several dozen RL steps as the model learns SQL executors are non-essential for many straightforward queries.The model increasingly memorizes expected execution results and uses fewer verification calls.
- Tool-use dynamics: VT-DeepSearch tool usage increases dramatically because information-rich search results cannot be easily memorized and remain necessary for task completion.The contrast with SQL demonstrates that tool utility shapes learning dynamics across computational domains.
- Emergent agent behavior: VERLTOOL-trained agents exhibit self-correction, iterative refinement, and strategic tool selection beyond simple function-calling.Examples include mathematical verification and backtracking, query refinement, and debugging through incremental fixes.
5 CONCLUSION
The conclusion presents VERLTOOL as unified, modular infrastructure for efficient multimodal ARLT training, validated across six domains with competitive performance.
- 5 CONCLUSION: VERLTOOL addresses ARLT training limitations through standardized multimodal tool management, asynchronous rollouts, and upstream alignment with VeRL.The framework extends single-turn RLVR to multi-turn ARLT and supports diverse tool integrations.
- 5 CONCLUSION: Across six domains, agents trained with VERLTOOL demonstrate competitive performance compared with specialized systems under unified training infrastructure.The paper positions VERLTOOL as scalable foundational infrastructure for ARLT research.
A DETAILED EXPERIMENT SETUP
The experiments evaluate VERLTOOL across six diverse domains, each requiring distinct tools and integration strategies to test adaptability and performance.
- A DETAILED EXPERIMENT SETUP: VERLTOOL is evaluated across six diverse domains to assess its effectiveness in tool-augmented reasoning.The domains present different challenges and require different tool integration strategies.
A.1 MATHEMATICAL REASONING WITH PYTHON EXECUTOR (VT-MATH)
The detailed setup describes tool-based mathematical reasoning and knowledge question answering, using verifiable rewards and benchmark evaluations tailored to each task.
- Mathematical reasoning: VT-Math integrates a Python interpreter so agents can execute calculations and verify intermediate mathematical results.The mathematical-coding agent sends Python snippets to a sandboxed interpreter and processes execution traces.
- Mathematical reasoning: VT-Math trains on DeepMath with a reward combining answer accuracy and tool-use incentives.The reward encourages Python-executor exploration while preserving an accuracy focus.
- Mathematical evaluation: VT-Math is evaluated on MATH-500, OLYMPIAD, MINERVA, GSM8K, AMC, AIME24, and AIME25 using MATH-EVALUATION-HARNESS1.The evaluation covers multiple mathematical benchmarks with standardized scoring.
- Knowledge question answering: Knowledge-QA uses a FAISS-based retriever to query a local knowledge base for relevant information during complex question answering.The agent alternates search operations with reasoning steps to construct answers.
- Knowledge-QA evaluation: The knowledge-QA evaluation uses Exact Match on general and multi-hop benchmarks, including NQ, TriviaQA, PopQA, HotpotQA, 2Wiki, MuSiQue, and Bamboogle.The benchmark suite tests both general factual answering and multi-hop reasoning.
A.3 MULTI-TURN SQL QUERY GENERATION (VT-SQL)
VT-SQL evaluates an agent that converts natural-language questions into executable SQL using schema hints, tool-calling instructions, and execution feedback. Performance is assessed by execution accuracy across multiple Spider benchmarks.
- Motivation: Tool integration supports iterative query refinement through execution feedback and error correction.
- Task and setup: The agent translates natural-language questions into executable SQL using schema hints and tool-calling instructions.The setup uses Qwen2.5-7B-Instruct and the SkyRL-SQL training set.
- Reward: Execution accuracy is the sole focus of the reward function.
- Evaluation: Evaluation uses execution accuracy on SPIDER-1.0 Dev and Test, SPIDER-DK, and SPIDER-SYN.
A.4 VISUAL REASONING WITH IMAGE OPERATIONS (VT-VISUALREASONER)
VT-VISUALREASONER equips agents with image-operation tools for reasoning over dense visual information and evaluates reward designs that combine accuracy with tool-use complexity. The setup includes visual search evaluation on V-Star and a separate web-search configuration using retrieval rewards.
- Visual tools: Image-operation tools let agents zoom into regions, select key frames, and manipulate images during visual reasoning.
- Reward design: The reward combines answer accuracy with curiosity and action-count penalties, using H = 0.3, N = 1, α = 0.5, and β = 0.05.
- Reward design: The study compares GRPO-acc, using accuracy-only rewards, with GRPO-complex, using the original complexity-driven reward.
- Evaluation: The visual-reasoning evaluation primarily uses V-Star to assess multimodal visual-search capabilities.
- Web search: The web-search setup combines accuracy and tool-usage rewards, trains on 1K mixed examples, and evaluates on GAIA and HLE.Snippet-Only uses retrieved snippets, whereas QwQ-32B summarizes raw content from retrieved URLs.
A.6 SOFTWARE ENGINEERING BENCHMARK (VT-SWE)
VT-SWE integrates terminal and code-execution tools for iterative software development tasks, rewarding only successful completion with passing tests. Training runs in sandboxed environments with explicit timeout and resource controls.
- Task and tools: Bash-terminal and code-execution tools support code understanding, localization, debugging, modification, and iterative testing.
- Reward: The reward is 1 only when execution terminates successfully and all verification tests pass; otherwise it is 0.
- Evaluation: Performance is evaluated on SWE-Verified by measuring task resolution and verification-test success.
- Infrastructure: Sandbox services run on eight servers and scale each task from 1 CPU and 2 GB memory up to 2 CPUs and 4 GB memory.
- Infrastructure: Interactions time out after 90 seconds, reward computation after 300 seconds, and trajectories after 20 minutes; failed trajectories receive zero reward and masked gradients.
- Supported tools: Table 8 lists the tools currently supported by the VERLTOOL framework.
A.8 TRAINING AND EVALUATION CONFIGURATIONS
The paper presents task-specific training and evaluation configurations alongside a progression from prompt-based and supervised tool use to multi-turn reinforcement learning with tool feedback. It defines agentic LLMs and ARLT as stateful, interactive systems whose actions respond to environmental observations.
- Training and evaluation configurations: Configuration differences across tasks can create gaps between validation curves and final downstream evaluation performance.
- Training and evaluation configurations: Math-TIR and Pixel-Reasoner use smaller batches, Search-R1 uses larger batches, and SWE tasks may require up to 100 interaction turns.
- Training and evaluation configurations: Evaluation temperatures range from 0.0 for deterministic SQL generation to 0.6 for exploratory tasks, with turn limits matched to task complexity.
- From tool integration to agentic LLMs: Tool integration progressed from prompt-based orchestration and supervised traces toward reinforcement learning for adaptive, multi-turn tool use.
- Agentic LLMs: Agentic LLMs reason, act through tools, maintain state, and revise plans according to environmental observations.
- Reinforcement learning foundations: GRPO supports long reasoning chains and integration of multi-turn tool responses with verifiable rewards.
- Agentic reinforcement learning with tool use: ARLT trains on dynamic multi-turn trajectories where tool responses are environmental observations conditioning future actions.
- Agentic reinforcement learning with tool use: ARLT explicitly handles observation tokens, assigns credit across tool calls, and uses asynchronous, failure-aware executors.
C CASE STUDY
The case study demonstrates multi-turn tool calling through representative agents for mathematical reasoning, knowledge QA, SQL generation, and web search. It also explains the response tags used to distinguish reasoning, tool calls, observations, and final outputs.
- Response formatting distinguishes internal reasoning, tool calls, tool responses, and final answers through separate tags and colors.
- Four demonstrations cover mathematical reasoning with Python, knowledge Q&A with search, multi-turn SQL generation, and agentic web search.
- Tool responses for web search, SQL generation, and knowledge Q&A are truncated or simplified in the demonstrations.
- Example-answer tables provide demonstrations for VT-MATH, VT-SEARCH, SQL reasoning, and VT-DEEPSEARCH.
- Multi-turn SQL Generation: The SQL example uses a NOT IN clause to exclude student IDs associated with cats from the full student list.