PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Context Reinforcement Learning for Tool Use in Large Language Models

Yaoqi Ye, Yiran Zhao, Keyu Duan, Zeyu Zheng, Kenji Kawaguchi, Cihang Xie, Michael Qizhe Shieh

arXiv:2603.08068v1cs.AI

TL;DR

Training LLMs to use tools effectively either struggles with exploration from scratch or requires costly labeled data for supervised initialization. ICRL uses few-shot demonstrations during reinforcement-learning rollouts, gradually removes them, and achieves state-of-the-art QA performance, outperforming the strongest Qwen2.5-3B baseline by +8.94 average exact match.

  • Problem

    Direct RL often produces poor tool-use exploration, while SFT-based initialization requires large amounts of expensive labeled data.

  • Method

    ICRL teaches tool use through few-shot demonstrations in RL rollout prompts and gradually reduces them until the model operates without demonstrations.

  • Results

    ICRL achieves state-of-the-art QA performance across models and datasets, exceeding the strongest Qwen2.5-3B baseline by +8.94 average exact match.

  • Takeaways & Limitations

    ICRL offers a scalable, data-efficient alternative to SFT-based pipelines for learning effective tool-use behavior without labeled tool traces.

Abstract

from arXiv · show

While large language models (LLMs) exhibit strong reasoning abilities, their performance on complex tasks is often constrained by the limitations of their internal knowledge. A compelling approach to overcome this challenge is to augment these models with external tools -- such as Python interpreters for mathematical computations or search engines for retrieving factual information. However, enabling models to use these tools effectively remains a significant challenge. Existing methods typically rely on cold-start pipelines that begin with supervised fine-tuning (SFT), followed by reinforcement learning (RL). These approaches often require substantial amounts of labeled data for SFT, which is expensive to annotate or synthesize. In this work, we propose In-Context Reinforcement Learning (ICRL), an RL-only framework that eliminates the need for SFT by leveraging few-shot prompting during the rollout stage of RL. Specifically, ICRL introduces in-context examples within the rollout prompts to teach the model how to invoke external tools. Furthermore, as training progresses, the number of in-context examples is gradually reduced, eventually reaching a zero-shot setting where the model learns to call tools independently. We conduct extensive experiments across a range of reasoning and tool-use benchmarks. Results show that ICRL achieves state-of-the-art performance, demonstrating its effectiveness as a scalable, data-efficient alternative to traditional SFT-based pipelines.

1. Introduction

ICRL is an RL-only framework that teaches LLMs tool-augmented reasoning through few-shot demonstrations during rollouts, then gradually removes those demonstrations to enable autonomous tool use. Across QA and reasoning benchmarks, it achieves state-of-the-art performance without SFT or ground-truth tool traces.

  • Motivation: Existing RL-from-scratch approaches often perform poorly because models lack initial tool-use abilities and struggle with ineffective exploration, while SFT-then-RL pipelines require supervised data.These are the dominant training paradigms described in the introduction.
  • Method: ICRL teaches tool use directly through RL rollouts augmented with few-shot examples showing step-by-step reasoning and tool invocation.The rollout prompt is constructed by prepending a small number of in-context demonstrations.
  • Method: A multi-stage curriculum gradually reduces rollout demonstrations from few-shot to zero-shot, encouraging the model to internalize tool-use strategies and produce structured outputs autonomously.The training reward balances task accuracy and format correctness.
  • Results: Up to 8.9 on Qwen2.5-3B and 7.3 on Qwen2.5-7B in average exact match accuracy, ICRL outperforms strong search baselines on challenging QA datasets.The cited baselines are ZeroSearch, Search-R1, and ParallelSearch.
  • Results: ICRL generalizes across diverse tool-augmented reasoning domains and offers a unified, scalable framework without costly supervision.The experiments cover a range of QA and reasoning benchmarks.

2. In-Context Reinforcement Learning (ICRL)

ICRL formulates tool-augmented reasoning as sequential model–tool interaction and trains it with reinforcement learning using masked optimization and GRPO. Its rollout process begins with few-shot tool-use demonstrations, progressively removes them, and uses accuracy plus format correctness as the reward signal.

  • Tool-use formulation: Tool-augmented generation conditions each response token on the query, prior tokens, and the history of model–tool interactions.The history contains previous actions and tool observations up to each step.
  • Tool-use formulation: At each step, the model can reason internally, query an external tool, or return a final answer, with actions represented in structured text.The format can use XML tags such as <think>, <search>, <information>, and <answer>.
  • RL objective and optimization: Loss masking excludes retrieved tokens from optimization so that only language-model-generated tokens contribute to the reinforcement-learning objective.Retrieved content is not produced by the model and therefore does not represent its reasoning or decisions.
  • RL objective and optimization: ICRL applies GRPO by sampling N responses with the previous policy and optimizing their group-relative rewards.The method trains the policy πθ on the RL dataset using πθold to generate response groups.
  • Training process: ICRL gradually reduces rollout demonstrations from an initial few-shot prompt until no examples remain, combining in-context guidance with reinforcement-learning exploration.The prompt changes from PN to PN−1 after sufficient learning progress, and this reduction repeats iteratively.
  • Reward design: The composite reward combines exact-match answer accuracy with structured-format correctness based on XML-tag adherence and penalties for violations.The accuracy reward is 1 for an exact match and 0 otherwise; format violations receive predefined costs.

3. Experiment

The experiments evaluate ICRL across Qwen instruction-tuned backbones, diverse QA benchmarks, and strong tool-use baselines. ICRL achieves state-of-the-art accuracy without SFT or labeled tool traces, while learning structured multi-turn search behavior.

  • Experimental Setup: ICRL is evaluated primarily with Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct, plus Qwen2.5-14B-Instruct and Qwen3-8B.The models are instruction-tuned backbones used for question answering and reasoning tasks.
  • Experimental Setup: Evaluation covers TriviaQA, HotpotQA, 2Wiki, Musique, and Bamboogle, excluding training-set NQ to avoid data leakage.The benchmarks span diverse domains and reasoning types.
  • Qualitative Analysis: An ICRL-Qwen2.5-7B example answers a compositional Bamboogle query through sequential searches, identifying George Washington and retrieving his inauguration date.The example demonstrates structured multi-turn tool-use behavior learned through in-context reinforcement learning.
  • Main Results: 40.16 average EM on Qwen2.5-3B surpasses Search-R1’s 31.10 by +8.94 across five QA datasets.The largest improvements occur on 2Wiki (+7.3), Musique (+9.7), and Bamboogle (+7.2).
  • Main Results: 49.12 average EM on Qwen2.5-7B exceeds ParallelSearch’s 41.78 by +7.34, with the best results on four of five datasets.ICRL scores 75.4 on TriviaQA, 53.6 on 2Wiki, 26.0 on Musique, and 48.0 on Bamboogle.
  • Main Results: 40.16 average EM without SFT or labeled tool traces exceeds O2-Searcher’s 37.26, with gains on four of five datasets.The reported gains include TriviaQA (+12.9) and Musique (+4.0).

4. Further Analysis

Further analysis shows that the 3~2~0 rollout curriculum preserves answer quality better than 3~2~1~0, while ICRL scales to Qwen2.5-14B and learns tool use under sparse rewards. The method also extends beyond web search to code-writing and Python-based math solving with greater data efficiency than cold-start SFT approaches.

  • Curriculum design: The three-stage 3~2~0 curriculum achieves substantially higher EM accuracy than 3~2~1~0 across all five QA datasets.On TriviaQA and 2Wiki, 3~2~0 reaches 75.4 and 53.6, versus 20.8 and 26.8 for 3~2~1~0.
  • Curriculum design: The four-stage curriculum produces faster decisions but weakens answer quality by encouraging premature stopping and shorter reasoning paths.Over 80% of queries finish within two search turns, whereas 3~2~0 allows longer reasoning paths.
  • Model scaling: On Qwen2.5-14B-Instruct, ICRL achieves 51.84 average EM, surpassing CoT by +20.7 and direct prompting by +27.0.It reaches 75.0 EM on TriviaQA and 61.8 on 2Wiki across the five QA datasets.
  • Training dynamics: During the 0-shot stage, response length initially drops after demonstrations are removed and then gradually increases as training progresses.Across the 3-shot, 2-shot, and 0-shot stages, early responses remain relatively stable and well structured.
  • Training dynamics: Despite sparse rewards based only on format validity and final-answer accuracy, ICRL increases valid tool calls during 0-shot training.This indicates that the model internalizes tool-use behavior without dense or step-level supervision.
  • Beyond web search: ICRL also trains code-writing and Python tool-calling for complex math problems and is more data-efficient than methods requiring thousands of annotated cold-start SFT examples.The comparison is made against ReTool, an SFT-RL framework for code-writing and tool-calling.

5. Conclusion

ICRL trains LLMs to use external tools through reinforcement learning with few-shot rollout demonstrations, eliminating the need for SFT and labeled tool traces. By phasing out demonstrations, it supports autonomous tool use and performs strongly across QA, reasoning, web-search, and code-execution tasks.

  • 5. Conclusion: ICRL trains LLMs for tool use without requiring supervised fine-tuning or labeled tool traces.It incorporates few-shot demonstrations directly into reinforcement-learning rollout prompts.
  • 5. Conclusion: Gradually phasing out few-shot demonstrations enables models to transition from imitation to autonomous tool use through reward-driven learning.The rollout process eventually removes the demonstrations as training progresses.
  • 5. Conclusion: ICRL achieves strong performance across QA and reasoning benchmarks while outperforming approaches relying on supervised data or frozen tool-use policies.The supplied conclusion characterizes these results as strong performance across a range of benchmarks.
  • 5. Conclusion: ICRL generalizes across web search and code execution, demonstrating flexibility and effectiveness as a scalable, data-efficient alternative to traditional SFT+RL pipelines.The conclusion identifies these domains as evidence of the framework’s broader applicability.
Loading 2603.08068v1…