PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning

Joykirat Singh, Raghav Magazine, Yash Pandya, Akshay Nambi

arXiv:2505.01441v1cs.AI

TL;DR

LLMs remain limited by static knowledge and text-only reasoning when tasks require dynamic reasoning, adaptive decisions, or external tools. ARTIST unifies agentic reasoning, reinforcement learning, and tool integration, enabling autonomous tool decisions without step-level supervision. Across mathematical reasoning and multi-turn function calling, it consistently outperforms baselines, with up to 22% absolute improvement over base models and emergent adaptive behaviors.

  • Problem

    Static knowledge and text-only reasoning limit LLMs on time-sensitive, knowledge-intensive, complex, and open-ended tasks requiring external tools.

  • Method

    ARTIST interleaves reasoning, tool queries, and tool outputs, training these trajectories with outcome-based reinforcement learning without intermediate-step or tool-call supervision.

  • Results

    ARTIST consistently outperforms baselines across mathematical reasoning and multi-turn function calling, achieving up to 22% absolute improvement over base models.

  • Takeaways & Limitations

    Agentic RL with tool integration yields adaptive tool selection, iterative self-correction, and deeper multi-step reasoning in the evaluated tasks.

  • Takeaways & Limitations

    Future work must address scaling to more diverse domains, richer feedback, and safety and reliability in open-ended environments.

Abstract

from arXiv · show

Large language models (LLMs) have achieved remarkable progress in complex reasoning tasks, yet they remain fundamentally limited by their reliance on static internal knowledge and text-only reasoning. Real-world problem solving often demands dynamic, multi-step reasoning, adaptive decision making, and the ability to interact with external tools and environments. In this work, we introduce ARTIST (Agentic Reasoning and Tool Integration in Self-improving Transformers), a unified framework that tightly couples agentic reasoning, reinforcement learning, and tool integration for LLMs. ARTIST enables models to autonomously decide when, how, and which tools to invoke within multi-turn reasoning chains, leveraging outcome-based RL to learn robust strategies for tool use and environment interaction without requiring step-level supervision. Extensive experiments on mathematical reasoning and multi-turn function calling benchmarks show that ARTIST consistently outperforms state-of-the-art baselines, with up to 22% absolute improvement over base models and strong gains on the most challenging tasks. Detailed studies and metric analyses reveal that agentic RL training leads to deeper reasoning, more effective tool use, and higher-quality solutions. Our results establish agentic RL with tool integration as a powerful new frontier for robust, interpretable, and generalizable problem-solving in LLMs.

1 Introduction

ARTIST addresses the limits of static, text-only reasoning by integrating agentic reasoning, dynamic tool use, and reinforcement learning. It evaluates this framework across mathematical reasoning and multi-turn function calling, reporting broad gains and emergent adaptive behaviors.

  • Motivation: Static internal knowledge and text-only reasoning limit performance on time-sensitive, knowledge-intensive, domain-specific, and tool-dependent tasks.These limitations can produce inaccuracies, hallucinations, brittle tool use, and poor adaptation to unseen scenarios.
  • Framework: ARTIST interleaves internal thinking, tool queries, and tool outputs so models can decide which tools to use, when to invoke them, and how to incorporate results.The framework treats tool usage and environment interaction as first-class operations within multi-turn reasoning chains.
  • Training: ARTIST uses outcome-based reinforcement learning without intermediate-step or tool-call supervision to learn adaptive tool-use strategies.GRPO guides learning from final outcomes rather than requiring step-level labels.
  • Evaluation: ARTIST is evaluated on mathematical reasoning and multi-turn function calling across challenging benchmarks, model scales, and baseline categories.The evaluation includes MATH-500, AIME, AMC, Olympiad Bench, τ-bench, and BFCL v3 subcategories.
  • Results: 22% absolute improvement over base models is achieved on mathematical reasoning, while ARTIST also surpasses GPT-4o and leading models on AMC, AIME, and Olympiad.For multi-turn function calling, ARTIST more than doubles base and prompt-based accuracy on τ-bench and gains on difficult BFCL v3 subsets.
  • Results: ARTIST exhibits adaptive tool selection, iterative self-correction, and context-aware multi-step reasoning across both evaluated domains.These behaviors are reported as emergent from the unified agentic RL framework.

2 ARTIST Overview

ARTIST implements agentic reasoning as structured rollouts that alternate between model reasoning and external interactions. Its GRPO-based training uses outcome rewards for answers, formatting, and successful tool execution while masking deterministic tool outputs.

  • Framework: ARTIST integrates code interpreters, web search engines, and domain-specific tools into reinforcement-learning-based reasoning across external environments.The framework is presented as general and extensible beyond isolated tool use or narrow domains.
  • GRPO Training: GRPO samples groups of responses, computes relative outcome-based advantages, and optimizes the policy without a critic or value-function approximation.The algorithm uses groupwise rewards and advantages for sampled responses from the old policy.
  • GRPO Training: Tool-output tokens are masked during loss computation so optimization focuses on the agent’s reasoning and decisions rather than deterministic responses.This loss-masking strategy is part of ARTIST’s GRPO adaptation for tool-integrated rollouts.
  • Rollout Process: Each rollout alternates internal reasoning, tool or environment queries, returned outputs, and a final answer, with the policy choosing whether and how to interact at each step.Supported interactions include code execution, API calls, web search, file operations, and browser or operating-system actions.
  • Reward Design: The composite reward includes answer correctness, prescribed output format, and successful tool execution.Format rewards enforce the ordering of reasoning, tool calls, and outputs, while answer rewards target correctness.
  • Reward Design: Tool Execution Reward is defined as the fraction of successful tool calls, encouraging syntactically correct and executable queries.Tools_success and Tools_total denote successful and total tool calls, respectively.

3 Case Study

ARTIST applies agentic reasoning and tool integration to complex mathematical reasoning and multi-turn function calling. Across both case studies, it combines structured planning, adaptive tool use, and iterative correction to manage computations and interactive workflows.

  • 3.1 Complex Mathematical Reasoning with Agentic Tool Use: ARTIST augments mathematical reasoning with a Python interpreter, allowing complex computations and intermediate-result verification outside text-only inference.The rollout uses structured reasoning, Python code, interpreter outputs, and a final answer.
  • 3.1 Complex Mathematical Reasoning with Agentic Tool Use: The mathematical reward design combines exact-answer, structured-format, and successful-tool-execution signals.Answer reward is 2 for an exact match; format rewards enforce the required tag sequence and tool execution rewards successful Python runs.
  • 3.1 Complex Mathematical Reasoning with Agentic Tool Use: ARTIST decomposes complex problems, alternates language reasoning with external computation, and refines its approach using intermediate results.The examples associate this process with self-refinement, self-correction, and self-reflection that emerge without explicit step-level supervision.
  • 3.2 Multi-Turn Function Calling with Agentic Reasoning and Tool Use: Function-calling prompts require explicit reasoning before tool calls, while rewards target state maintenance and correct function-call sequences.The state reward tracks relevant variables across interactions; the function reward matches expected function names and arguments.
  • 3.2 Multi-Turn Function Calling with Agentic Reasoning and Tool Use: In multi-turn function calling, ARTIST sequences and adapts multiple calls while managing dependencies, state, ambiguity, and tool errors.The case studies cover vehicle control, travel booking, item exchange, and customer support scenarios.
  • 3.2 Multi-Turn Function Calling with Agentic Reasoning and Tool Use: The function-calling examples show self-refinement, self-correction, and self-reflection during evolving user requirements and environment feedback.The model can reorder actions, retry after errors, confirm details, and validate outcomes before proceeding.

4 Experimental Setup

ARTIST is evaluated across mathematical reasoning and multi-turn function calling using established benchmarks, outcome-based metrics, and comparisons with diverse baseline categories. The setup combines curated training data with held-out validation and broad evaluation coverage.

  • 4 Experimental Setup: The evaluation spans complex mathematical problem solving and multi-turn function calling, with datasets, metrics, and baselines specified for each setting.The study includes both reasoning and tool-use domains.
  • 4 Experimental Setup: The math training set contains 20,000 NuminaMath problems paired with ground-truth final answers for outcome-based reinforcement learning.The problems range from elementary arithmetic and algebra to advanced competition-level tasks.
  • 4 Experimental Setup: Math evaluation uses MATH-500, AIME, AMC, and Olympiad Bench, covering competition-style, standardized, and olympiad-level problems.Olympiad Bench specifically targets multi-step reasoning.
  • 4 Experimental Setup: Pass@1 accuracy measures the percentage of math problems whose final answer exactly matches the ground truth in one attempt.The same metric is used for function calling with correctness additionally requiring the ground-truth environment state.
  • 4 Experimental Setup: Function-calling training uses 100 BFCL v3 base-category tasks, with another 100 annotated tasks reserved for validation.Tasks require multiple coordinated calls, state tracking, and error recovery.
  • 4 Experimental Setup: Function-calling evaluation covers BFCL v3 domains and subcategories plus τ-bench airline and retail dialogues requiring domain-specific APIs and policy compliance.BFCL v3 includes vehicle control, trading, travel, file systems, cross-functional APIs, missing parameters, missing functions, and long context.
  • 4 Experimental Setup: Training compares 7B and 14B Qwen2.5-Instruct models for math and uses Qwen2.5-7B-Instruct for function calling, alongside frontier and tool-augmented baselines.Math rollouts sample six trajectories with an 8,000-token budget; function-calling training samples eight rollouts with a 16,384-token context window and 2,048-token responses.

5 Results

ARTIST improves mathematical reasoning and multi-turn function calling, with the largest gains on complex tasks where adaptive tool use and multi-step reasoning matter most.

  • Mathematical Reasoning: Up to 22% absolute improvement over base models appears on challenging mathematical benchmarks, including AMC, AIME, and Olympiad.Qwen2.5-14B-ARTIST reaches 0.55 on AMC, improving 22.0% over the base model.
  • Mathematical Reasoning: ARTIST’s advantage increases with task complexity, while gains on the less challenging MATH-500 are more modest.The paper attributes the pattern to greater value from dynamic tool integration and multi-step reasoning on complex problems.
  • Mathematical Reasoning: Prompting base models to use tools yields limited improvements, whereas explicit agentic RL produces consistently higher accuracy across mathematical tasks.The reported AMC advantage over the prompt-based baseline reaches 12.1%.
  • Mathematical Reasoning: ARTIST outperforms open-source tool-augmented models at the same scale, with average gains up to 37.7% across mathematical benchmarks.For Qwen2.5-7B, the reported average improvements are 26.7% over ToRA-7B, 16.5% over NuminaMath-TIR, 24% over ToRA-Code, and 37.7% over PAL.
  • Mathematical Reasoning: ARTIST surpasses GPT-4o across all reported mathematical benchmarks, including an 8.9% gain on Olympiad at the 7B scale.At 14B, the reported gaps widen further across Olympiad, AMC, MATH-500, and AIME.
  • Metrics Analysis: ARTIST improves solution quality, tool utilization, and reasoning depth, with reward scores nearly tripling on AMC and quadrupling on AIME.It averages over 3 tool calls per AIME query versus 0.3 for the baseline, and response length can reach twice the baseline on harder datasets.
  • Multi-Turn Function Calling: More than doubling base-model accuracy on τ-bench, ARTIST also improves the hardest BFCL v3 subsets, especially Long Context.It reaches 0.260 on Airline and 0.240 on Retail, versus base-model scores of 0.120 and 0.180; Long Context improves by 9.0%.
  • Multi-Turn Function Calling: ARTIST outperforms prompt-based reasoning most clearly on difficult multi-turn tasks, including a 7.5% gain on BFCL v3 Long Context.On τ-bench, the reported improvements are 14.0% on Airline and 4.0% on Retail.

6 Related Work

Prior work advances LLM reasoning through chain-of-thought, tool integration, and reinforcement learning, but their combination for adaptive agentic tool use remains underexplored. ARTIST addresses this gap by treating tool interaction as a first-class operation learned through outcome-based RL.

  • Chain-of-Thought Reasoning and Prompting Approaches: Chain-of-thought prompting improves reasoning by encouraging explicit intermediate steps and decomposition into smaller subproblems.
  • Tool-Based Reasoning: Tool-based reasoning gives LLMs access to external resources such as web search, code interpreters, and APIs.
  • RL-Based Reasoning: RL-based reasoning enhances LLM capabilities through reward-driven policy optimization, including methods such as PPO and GRPO.
  • Reasoning and Agentic Systems: Agentic systems extend LLMs beyond static, single-turn inference through autonomous planning, adaptation, and interaction with external environments.
  • Reasoning and Agentic Systems: Existing tool-augmented reasoning methods often use search or code tools, but may rely on retrieval tasks or supervised fine-tuning with annotated data.
  • Our Contribution: ARTIST unifies agentic reasoning, tool integration, and outcome-based RL so models learn adaptive, context-aware tool invocation and environment interaction.

7 Conclusion

ARTIST unifies agentic reasoning, reinforcement learning, and dynamic tool integration to train models that autonomously plan, adapt, and solve multi-step tasks. Across mathematical and function-calling settings, it produces richer reasoning and purposeful tool use without step-level supervision, while future work targets broader domains, feedback, safety, and reliability.

  • 7 Conclusion: ARTIST unifies agentic reasoning, reinforcement learning, and dynamic tool integration within a framework for autonomous multi-step problem solving.
  • 7 Conclusion: The framework supports adaptive tool selection, iterative self-correction, and deeper reasoning without step-level supervision.
  • 7 Conclusion: Future work should scale ARTIST to more diverse domains, integrate richer feedback including human preferences, and address safety and reliability in open-ended environments.
  • 7 Conclusion: ARTIST coordinates internal reasoning with dynamic tool and environment interaction to support complex problem solving across diverse domains.

C Implementation Details

ARTIST uses Qwen2.5 models trained with GRPO, sampling multiple exploratory rollouts and alternating text generation with tool invocation. The implementation records successful, empty-output, and failed tool executions, and reports the training hardware and reproducibility plan.

  • Model and Training: ARTIST trains Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct models with GRPO using multiple sampled rollouts per question.Training uses a learning rate of 10^-6, six rollouts per question, and an 8,000-token maximum response length.
  • Rollout and Tool Execution: Each rollout alternates between text generation and tool invocation, with Python code executed through the built-in exec() function.
  • Rollout and Tool Execution: Tool feedback distinguishes successful execution with output, successful execution without output, and failed execution.Failures return compilation errors, while successful executions return a compilation-success message.
  • Hardware: The experiments use 4×A100 80 GB GPUs, with reported training times of 20 hours for one setup and 34 hours for another.
  • Reproducibility: The codebase is built on Verl and verifier libraries, with code, hyperparameters, and configuration files planned for release upon publication.

D.1 Example-1

In the apple-probability example, ARTIST decomposes the problem into conditional probabilities, computes candidate values with Python, and iteratively adjusts n based on tool feedback. This process yields n = 6 and illustrates self-refinement, self-correction, and self-reflection emerging without explicit supervision.

  • Example-1: The task asks for the smallest n making the probability of drawing red apples n times consecutively from 11 apples less than 0.5.
  • Example-1: ARTIST expresses the probability as a product of conditional fractions and translates that reasoning into Python computation.
  • Example-1: The model evaluates each computed result, increases n when the threshold is unmet, and repeats until the probability falls below 0.5.
  • Example-1: The example identifies self-refinement, self-correction, and self-reflection as emergent capabilities of the agentic process and reward design.
  • Example-1: At n = 5, the probability is 0.5454545454545453, while at n = 6 it is 0.4545454545454544, satisfying the condition.

D.2 Example-2

ARTIST solves the class-enrollment problem by combining inclusion-exclusion with tool-assisted computation. After recovering from a variable-scope error, it recomputes and verifies the correct total of 50 students.

  • The example illustrates self-refinement, self-correction, and self-reflection through rerunning, diagnosing, and validating the calculation.ARTIST improves its approach after the error and repeats the calculation before answering.
  • ARTIST applies inclusion-exclusion to compute 42 students enrolled in at least one course, then adds 8 students enrolled in neither course.The calculation uses 18 chorus students, 26 band students, and 2 students in both courses.
  • A stateless tool execution causes a compilation error because students_at_least_one_course is undefined in the second code block.The model identifies the variable-scope problem and reruns the computation with all variables defined together.
  • 50 students is the corrected total after recomputing the enrollment union and adding the 8 students in neither course.The result is produced and then confirmed again in the final answer.

E.1 Example-1

In multi-step vehicle and travel tasks, ARTIST sequences tool calls, responds to execution feedback, and recovers from errors or missing prerequisites. These interactions demonstrate adaptive planning for dynamic environments.

  • Vehicle-control example: ARTIST locks all car doors before attempting to start the engine, then adapts when the environment requires the brake pedal to be pressed.It presses the brake pedal and successfully retries engine startup in START mode.
  • Emergent behavior: Across the examples, ARTIST plans, adapts to tool errors, recovers from failed actions, and communicates intermediate and final task status.The passages characterize these capabilities as practical for automation and workflow orchestration.
  • Emergent behavior: The examples present RL-driven tool integration as supporting multi-function task handling in dynamic environments without explicit supervision.The stated value is robust agentic reasoning for real-world automation.
  • Vehicle-control example: The engine starts successfully after ARTIST presses the brake pedal, completing the vehicle-control task.The tool reports a running engine with fuel level 15.5 and battery voltage 12.8.
  • Travel example: ARTIST handles a failed flight-cost query by booking the flight with corrected parameters and then proceeding to cancellation.The initial get_flight_cost call fails because access_token is an unexpected keyword argument.

Emergent Agentic Capabilities

The shopping workflow shows ARTIST tracking user and order information, correcting mistaken assumptions, and filtering product choices against nuanced preferences. It repeatedly validates details before proceeding.

  • Emergent capabilities: The example characterizes ARTIST’s behavior as incremental refinement, error diagnosis, and preference-sensitive option filtering.These capabilities are presented as emergent agentic behaviors in multi-turn function calling.
  • Self-reflection: The agent confirms details with the user and summarizes its understanding before advancing the exchange workflow.This includes confirming the target order, item, and exchange requirements.
  • Preference handling: ARTIST filters the desk-lamp search according to the requested lower brightness and preferred battery power source.It uses product-type and product-detail tools to inspect available options.
  • State tracking: The agent maintains user context by retrieving a user ID, recent orders, order details, and the desk lamp’s item ID.The workflow begins with the user’s name and zip code and uses tool results to narrow the target item.
  • Self-correction: ARTIST identifies a mismatch when the initially selected order lacks the desk lamp and redirects the search to another order.It queries order details again and finds the desk lamp in order #W6390527.

Emergent Agentic Capabilities

In the insurance-support example, ARTIST verifies reservation details, handles persistent disagreement, and works around an unavailable update operation. It ultimately adds insurance through a rebooking workflow.

  • Tool limitation recovery: When direct insurance updating is unavailable, ARTIST diagnoses the unsupported action and pivots to rebooking the reservation with insurance.The update_reservation_insurance call returns an unknown-action error.
  • Emergent capabilities: The example presents ARTIST as handling persistent requests, constrained tools, failed actions, and user-centered communication in customer support workflows.The stated application scope is real-world customer support and workflow automation.
  • Reservation verification: ARTIST discovers that the reservation lacks insurance, despite the user’s belief that it was selected during booking.The reservation details show no insurance, and the agent explains the available paid option.
  • User communication: The agent repeatedly reviews confirmation details and clarifies the user’s options before processing the requested insurance purchase.It maintains the reservation context while responding to the user’s insistence.
  • Task completion: ARTIST successfully adds travel insurance for $30 per passenger and reports the updated reservation to the user.The final booking call uses insurance: yes and processes $30 on the specified credit card.
Loading 2505.01441v1…