PaperlayerView exact revision on arXiv

Source-linked AI summary

Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Data

Emre Can Acikgoz, Cheng Qian, Jonas Hübotter, Heng Ji, Dilek Hakkani-Tür, Gokhan Tur

arXiv:2602.21320v1cs.LG

TL;DR

Tool-calling agents typically require curated task-solution pairs and human supervision, limiting zero-data self-evolution. Tool-R0 uses self-play reinforcement learning to co-evolve task generation and tool-use solving, achieving broad benchmark gains and surpassing supervised baselines under the same setting.

  • Problem

    Existing self-play methods have not established whether tool-calling agents can learn across domains from scratch under a zero-data assumption.

  • Method

    Tool-R0 co-evolves a Generator that creates verifiable frontier tasks and a Solver that learns to complete them through dual reinforcement learning from the same base model.

  • Results

    92.52% relative gain over the base model spans diverse tool-use benchmarks, while Tool-R0 achieves 47.84% average accuracy versus 46.06% for the strongest supervised baseline.

  • Takeaways & Limitations

    Self-play can induce broad tool-calling skills from weak base models and match or surpass supervised agents trained on static human data.

  • Takeaways & Limitations

    The approach relies on latent knowledge from a shared base model, so external signals may be needed to break its knowledge boundary.

Abstract

from arXiv · show

Large language models (LLMs) are becoming the foundation for autonomous agents that can use tools to solve complex tasks. Reinforcement learning (RL) has emerged as a common approach for injecting such agentic capabilities, but typically under tightly controlled training setups. It often depends on carefully constructed task-solution pairs and substantial human supervision, which creates a fundamental obstacle to open-ended self-evolution toward superintelligent systems. In this paper, we propose Tool-R0 framework for training general purpose tool-calling agents from scratch with self-play RL, under a zero-data assumption. Initialized from the same base LLM, Tool-R0 co-evolves a Generator and a Solver with complementary rewards: one proposes targeted challenging tasks at the other's competence frontier and the other learns to solve them with real-world tool calls. This creates a self-evolving cycle that requires no pre-existing tasks or datasets. Evaluation on different tool-use benchmarks show that Tool-R0 yields 92.5 relative improvement over the base model and surpasses fully supervised tool-calling baselines under the same setting. Our work further provides empirical insights into self-play LLM agents by analyzing co-evolution, curriculum dynamics, and scaling behavior.

1 Introduction

Tool-R0 proposes zero-data self-play reinforcement learning in which independently trained Generator and Solver roles co-evolve to teach general-purpose tool calling from scratch. It reports consistent gains across architectures, scales, and benchmarks, surpassing fully supervised tool agents while identifying mechanisms governing this self-evolution.

  • Framework: Tool-R0 initializes one base model as independently trained Generator and Solver roles that co-evolve through complementary rewards without human data.The Generator creates challenging tasks near the Solver’s evolving competence, while the Solver learns from outcome-based rewards.
  • Findings: Self-play alone can teach complex tool-calling skills from weak priors, producing consistent gains across model scales, architectures, and benchmarks.The introduction presents this as evidence that self-play reinforcement learning is sufficient for complex tool-use learning from zero data.
  • Findings: Adaptive self-generated curricula generalize better than fixed human-distribution datasets, while separate Generator and Solver parameters prevent role conflicts in high-entropy action spaces.Effective co-evolution also requires smooth difficulty-aware rewards; static generation or miscalibrated difficulty does not sustain tool-use improvement.
  • Performance: Tool-R0 consistently improves base models across architectures, scales, and benchmarks, surpassing fully supervised tool agents in a data-free setting.These results support self-evolution into general tool-calling agents without human data.
  • Framework: The framework uses generated tool-calling tasks, difficulty-aware rewards, filtering, and easy-to-hard curricula to drive the Solver’s learning cycle.The Generator targets the Solver’s competence frontier, and the Solver trains on the resulting curated task pool.

2 Related Work

The related work frames tool-learning as structured interaction with external tools and situates self-play within both historical game-playing and recent open-domain LLM reinforcement learning.

  • Tool-Learning with LLMs: Tool-learning extends LLM capabilities beyond parametric knowledge by requiring tool selection, context-grounded arguments, and multi-step function-call composition.The passage identifies tool-learning as a paradigm involving structured reasoning over available tool schemas.
  • Self-Evolution through Self-Play: Self-play has produced superhuman performance in closed-loop games and has recently been applied to open-domain reasoning with LLMs trained via RLVR.The cited recent work extends self-play beyond games into open-domain reasoning settings.

3 Method

Tool-R0 is a zero-data self-play framework that co-evolves a Generator and Solver from the same base LLM through dual reinforcement learning. The Generator creates verifiable tasks at the Solver’s evolving competence frontier, while the Solver learns from a curated curriculum and feeds difficulty signals back into generation.

  • Overview: Tool-R0 co-evolves a Generator and Solver through K self-play iterations, with the Generator synthesizing tasks and the Solver learning to solve them from the same base LLM.Each iteration trains the Generator, freezes it to build a deduplicated and cross-verified dataset, then trains the Solver before carrying it forward.
  • Task Generation: Lightweight task specifications ground generation to prevent mode collapse while preserving domain-agnostic adaptability.The framework addresses the tendency of generic diversity prompts to concentrate generations on a small set of high-likelihood patterns.
  • Task Generation: The Generator uses GRPO to produce JSON-verifiable user requests, tool menus, and gold tool-calls from scratch without external data.The structured format enables execution-based feedback and verification rather than relying on non-verifiable natural-language supervision.
  • Generator Rewards: Generator rewards enforce parseable structure, consistency between tools, calls, and request values, and curriculum quality based on Solver-calibrated difficulty and semantic alignment.The curriculum reward suppresses unsolvable tasks and favors tasks near the Solver’s competence frontier while requiring semantic faithfulness.
  • Dataset Construction and Solver Training: After Generator training, Tool-R0 removes near-duplicate tasks, cross-verifies pseudo-labels with multiple Solver predictions, and trains the Solver on the resulting curriculum.Solver statistics directly influence Generator difficulty rewards, creating a feedback loop that produces progressively harder yet solvable tasks without external supervision.

4 Experiments

Tool-R0 enables base LLMs to self-evolve into general-purpose tool-calling agents from scratch without human data, producing gains across benchmarks, model scales, and architectures. It also outperforms supervised baselines through adaptive self-generated curricula and depends on active, difficulty-aware co-evolution.

  • Main results: +22.99 points (92.52% relative gain) is the primary Qwen2.5-1.5B-Instruct model’s average improvement from self-play across evaluated tool-calling benchmarks.The results show that self-play alone induces substantial gains across diverse task types.
  • Scaling: 30.57 average accuracy lets the trained 0.5B model surpass the 1.5B base model, while 47.84 lets the trained 1.5B model exceed the 3B base model.Tool-R0 improves models across scales, with larger absolute gains for smaller models and evidence of latent tool-use capabilities at 0.5B.
  • Model families: +4.53; ↑10.30% for Qwen2.5-3B-Instruct and +4.35; ↑12.04% for Llama-3.2-3B-Instruct show consistent benefits across model families.Both families reach comparable post-training levels despite different starting baselines, supporting model-agnostic behavior.
  • Comparison with supervised models: 47.84% average accuracy with zero curated data surpasses supervised methods trained on 4k-210k human-annotated examples.Tool-R0’s curriculum targets evolving weaknesses, while self-generated tasks provide broader and more balanced benchmark coverage than static curated datasets.
  • Co-evolution ablations: −17.42 pp from shared parameters and −6.19 points from freezing the Generator show that separate, actively learning Generator–Solver roles are essential for stable co-evolution.The need for separation is attributed to open-ended high-entropy tool actions and fundamentally different reward objectives.
  • Curriculum dynamics: 4.30 pp (↓8.99% relative) lower accuracy after removing difficulty reward shows that calibrated, smooth curriculum signals are necessary for sustained improvement.The Generator learns progressively harder but semantically coherent tasks, while performance typically peaks around the third iteration before saturating or slightly degrading.

5 Discussion · Appendix · A Related Work Extended

Tool-R0 enables base LLMs to self-evolve into general-purpose tool-calling agents through zero-data self-play, matching or surpassing supervised baselines across benchmarks and model scales. The discussion identifies reward robustness and richer curriculum feedback as key limitations and future directions, while positioning the framework against curated-supervision tool learning and prior self-play methods.

  • 5.1 Conclusion: Tool-R0 converts base LLMs into general-purpose tool-calling agents through zero-human-data self-play and can match or surpass supervised baselines across benchmarks and model scales.The authors describe these results as preliminary but suggest Tool-R0 supports self-improvement across domains without external data.
  • 5.2 Challenges: Tool-R0 remains an early step because smaller models can exhibit imperfect instruction or policy following that produces rare reward-hacking supervision failures.Semantic coherence checks and validity rewards mitigate these cases, which become less frequent with stronger base models but may require greater care at larger scale.
  • 5.3 What is Next: The proposed next step is richer difficulty feedback because Solver stochastic consistency can be noisy and sparse for guiding Generator learning.Suggested alternatives include loss- or gradient-based signals, semantic error attribution, and smoother plateau-shaped or calibrated-margin objectives.
  • A Related Work Extended: Tool-integrated reasoning grounds LLM knowledge through APIs, databases, and software functions, while early research emphasized curated benchmarks for tool selection, arguments, and execution correctness.The related literature cites Qu et al. (2025) for tool-integrated reasoning and Li et al. (2023), Tang et al. (2023), Srinivasan et al. (2023), Wu et al. (2024), and Patil et al. (2024) for benchmark-focused work.
  • A Related Work Extended: Existing tool-learning approaches rely on demonstrations, synthetic instructions, or static task distributions, limiting scalability where realistic requests, tool traces, and verifiable outcomes are unavailable.Tool-R0 instead studies a strict zero-data setting without demonstrations, prompts, or external task corpora.
  • A Related Work Extended: Prior LLM self-play progressed from alignment methods such as SPIN (Chen et al., 2024b) toward capability enhancement in verifiable domains such as code generation.The passage situates this progression alongside earlier AI self-play systems including TD-Gammon, AlphaGo (Silver et al., 2016; 2017), and CICERO.
  • A Related Work Extended: Tool-R0 fills this gap with co-evolving Generator and Solver agents whose complementary rewards create adaptive self-generated curricula for zero-data tool-integrated reasoning across diverse domains.This differs from prior self-play by targeting tool use rather than relying on static task generation.

B Grounded Task Specification … B.3 Sampling Strategy and Hyperparameters

The grounded task specification controls task semantics, interaction structure, domain sampling, and gold tool-call composition. It combines explicit domain configurations with user-defined sampling weights and fixed context and call-count probabilities.

  • B.1 Formal Definitions: The Generator is conditioned on explicit domain configurations that constrain generated tasks’ semantic scope and structural properties.The setup uses standardized domain-configuration templates for controllable, domain-agnostic generation.
  • B.1 Formal Definitions: The specification represents each task using its domain, interaction-context type, available-tool count, and number of gold tool calls.
  • B.2 Domain Configurations: Figure 9 specifies uniform sampling coefficients across functional and agentic domain categories rather than normalized probabilities.The configuration includes categories such as finance, healthcare, productivity, scheduling, databases, and cloud infrastructure.
  • B.2 Domain Configurations: The task prompt forbids meta-instructions, placeholders, and abstract descriptions while requiring gold-call argument values to appear verbatim in the user question.
  • B.3 Sampling Strategy and Hyperparameters: Domains are sampled from a fixed set of more than 30 functional and agentic categories using a user-defined weighted distribution.Practitioners specify both the domain set and relative weights to align the curriculum with task importance, deployment priorities, or safety constraints.
  • B.3 Sampling Strategy and Hyperparameters: Interaction contexts are sampled as 90% single-turn and 10% multi-turn, with multi-turn prompts embedding short histories and allowing one gold tool call.
  • B.3 Sampling Strategy and Hyperparameters: For single-turn tasks, the gold-call count is n = 1 with probability 0.8 and n = 2 with probability 0.2, while multi-turn tasks fix n = 1.

C Generator Implementation Details · C.1 Training Setup · C.2 Reward Hyperparameters

The Generator uses GRPO-based mixed-precision distributed training and emits a rigid four-block interface for deterministic execution. Its reward combines format, validity, and curriculum signals, with explicit checks for tool-call correctness, difficulty, and semantic alignment.

  • C.1 Training Setup: The Generator is trained with GRPO using TRL, HuggingFace Accelerate, and DeepSpeed ZeRO-3 in bfloat16 mixed precision on three GPUs.Training uses per-device batch size 2 with gradient accumulation over 4 steps, yielding a global batch size of 24 sequences per training configuration.
  • C.2 Reward Hyperparameters: Generator completions must contain exactly four tagged blocks—<think>, <question>, <available tools>, and <tool call answer>—with JSON-parsable tool specifications and calls.Tool calls are restricted to flat primitive arguments, enabling deterministic parsing, execution, and reward computation.
  • C.2 Reward Hyperparameters: The Generator reward combines format, validity, and curriculum components: rfmt, rvalid, and rcurr.These three components define the reward structure used during Generator training.
  • C.2 Reward Hyperparameters: The format reward sums binary indicators for tag completeness, tool-menu JSON validity, and gold-call JSON validity.These checks operationalize whether the structured completion conforms to the required output interface.
  • C.2 Reward Hyperparameters: Validity weighting is (λmenu, λgold, λvalue) = (0.4, 0.4, 0.2), checking tool existence, required parameters, and question-value matches.The validity reward acts as a compiler-like gate against nonexistent tools and malformed or unsupported calls.
  • C.2 Reward Hyperparameters: The curriculum reward is the unweighted sum rcurr = rdiff + rsem of difficulty and semantic-alignment components.This combines solver-calibrated challenge with correspondence between the generated question and gold tool call.
  • C.2 Reward Hyperparameters: Difficulty estimation queries the current Solver K = 8 times per task at temperature 0.7 with a maximum generation length of 2048 tokens.The temperature 0.7 setting is exclusive to difficulty estimation and differs from the Solver’s training rollout temperature of 1.0.
  • C.2 Reward Hyperparameters: Semantic alignment uses Solver judgments from 1 to 5, normalized as rsem = (s −1)/4 to penalize vague or templated questions.Gold tool calls are canonicalized before verification, while schema validation checks required parameters without strict type checking.

D Details of Solver Dataset Construction … D.3 Difficulty Probing and Curriculum Selection

The Solver dataset is built through generation and deduplication, Solver-based cross-verification, and difficulty probing with curriculum selection. This pipeline produces a diverse, semantically valid, reliable, and appropriately challenging training set grounded in Solver behavior.

  • D Details of Solver Dataset Construction: The frozen Generator synthesizes Solver data from control specifications through generation and deduplication, cross-verification, and difficulty probing and selection.This three-stage pipeline follows Generator training and uses the Generator purely as a task synthesizer.
  • D.1 Specification-Grounded Generation and Deduplication: The pipeline samples 10,000 structured task triples and removes near-duplicates using canonicalized question–tool–call signatures.Each triple contains a user request, tool menu, and gold tool calls, producing a large but non-redundant candidate pool.
  • D.2 Solver-Based Cross-Verification: Repeated Solver sampling retains candidates with consistent predicted and generated tool calls while discarding low-agreement instances as unreliable supervision.The procedure follows the principle that reproducible answers are more likely to provide correct supervision signals (Huang et al., 2023).
  • D.3 Difficulty Probing and Curriculum Selection: Difficulty probing estimates Solver pass@K success rates, assigns easy, medium, and hard buckets, and selects domain-diverse samples with balanced difficulty.This prevents the dataset from becoming biased toward trivially solvable tasks.
  • D.3 Difficulty Probing and Curriculum Selection: The initial 10,000 candidates are filtered to 2,000 samples forming the final Solver training set for each self-play iteration.Selection preserves domain diversity while maintaining a balanced mix of difficulty levels.
  • D.3 Difficulty Probing and Curriculum Selection: The selected data follow a batch-level curriculum that introduces mostly easy samples first and progressively adds harder tasks, including compositional multi-tool scenarios.This progression builds foundational tool-calling skills before exposing the Solver to harder problems that require them.
  • D.3 Difficulty Probing and Curriculum Selection: Overall, the procedure yields a diverse, semantically valid, appropriately challenging, and self-consistent distribution while avoiding noisy or degenerate pseudo-labels that could destabilize Solver training.Unlike static synthetic pipelines, generation quality remains continuously grounded in Solver behavior for stable agentic learning.

E Solver Implementation Details

Solver training uses GRPO with verifiable format and accuracy rewards, while the Solver follows a Tool-Integrated Reasoning interface that separates reasoning from predicted tool calls.

  • Solver Implementation Details: Solver training uses GRPO with a graded format reward rfmt and a dense accuracy reward racc.These are the two verifiable rewards used to instantiate Solver training.
  • Solver Implementation Details: The Solver prompt follows a Tool-Integrated Reasoning interface, emitting reasoning in <think> and a predicted tool-call list in <tool call answer>.This interface differs from the Generator’s prompt format.

E.1 Training Setup · E.2 Reward Hyperparameters

Solver training uses a larger rollout batch than the Generator and combines structured-format, dense accuracy, robust value-matching, and extra-call penalties. The reward design provides early parseability feedback while favoring correct, non-spurious tool calls.

  • E.1 Training Setup: The Solver uses the same Accelerate, DeepSpeed ZeRO-3, bfloat16, and three-GPU infrastructure as the Generator, with per-device batch size 2 and five-step gradient accumulation.This yields a global batch size of 32 sequences per update, larger than the Generator’s 24, to address higher variance in open-ended tool-call prediction.
  • E.2 Reward Hyperparameters: A completion is structurally valid only when it contains a non-empty <tool call answer> block, parsed using a relaxed loader that accepts JSON, Python-literal structures, and fenced JSON.Ellipsis-like placeholders are rejected and receive zero reward; parsed calls are normalized into canonical tool calls.
  • E.2 Reward Hyperparameters: The format reward combines tag presence, parseability, and successful normalization as rfmt = 0.3 · Itag + 0.3 · Iparse + 0.4 · Inorm.This graded reward supplies non-zero feedback before full functional correctness becomes common.
  • E.2 Reward Hyperparameters: Table 6 summarizes the fixed reward-related hyperparameters used for Solver training.The table covers the reward settings described for format and accuracy evaluation.
  • E.2 Reward Hyperparameters: Function-call accuracy uses greedy matching between predicted and gold calls, scoring tool-name exactness, argument-key F1 overlap, and matching argument values.The component weights are (λname, λkey, λval) = (0.2, 0.3, 0.5), and matched scores are averaged over gold calls.
  • E.2 Reward Hyperparameters: Value matching treats exact matches and normalized numeric or whitespace-equivalent values as equal, while long numeric strings are compared as identifiers to avoid floating-point artifacts.When those normalizations do not apply, comparison falls back to canonical JSON.
  • E.2 Reward Hyperparameters: The accuracy reward penalizes extra predicted calls multiplicatively using α = 0.25, leaving correct-length predictions unchanged while downweighting spurious calls.The penalty is applied through r_acc = s̄ / (1 + α · max(0, |Ĉ| − |C⋆|)).

F Further Details on Evaluation

The evaluation spans diverse agentic tool-calling tasks to assess multiple aspects of function invocation, including generalization, scalability, and robustness under distribution shifts.

  • Evaluation Scope: The evaluation comprehensively covers different agentic tool-calling tasks to assess diverse aspects of function invocation.The supplied passage frames the evaluation as collective and broad rather than focused on a single tool-use setting.
  • Evaluation Benchmarks: Tool-Alpaca (Tang et al., 2023) tests generalization across heterogeneous tool categories under synthetic distribution shifts.Its emphasis is robustness to shifted synthetic distributions.
  • Evaluation Benchmarks: Seal-Tools (Wu et al., 2024) evaluates large-scale APIs across diverse domains while reducing potential data contamination and stressing scalability.The passage positions Seal-Tools as an expanded setting beyond heterogeneous tool categories.

G Training Details of Supervised Baseline Tool-Calling Agents

The study re-trains supervised and reinforcement-learning tool-calling baselines on the same Qwen-2.5-1.5B-Instruct model using author-provided datasets and closely matched procedures, enabling controlled comparison with Tool-R0.

  • Baseline training details: xLAM uses supervised fine-tuning on approximately 60,000 diverse agent trajectories aggregated from multiple tool-use and agent benchmarks.Its sources include ToolBench, Webshop, ToolAlpaca, HotpotQA, AlfWorld, APIBank, Mind2Web, AgentBoard, AgentBench, API-GEN, and SpecTools.
  • Baseline training details: Hammer applies function masking and supervised fine-tuning to an augmented dataset of around 210,000 samples, including 7,500 irrelevance-detection samples.It trains Qwen 2.0 series models from 0.5B to 7B parameters and emphasizes training methods over data refinement.
  • Baseline training details: ToolACE generates over 500,000 multi-turn tool-learning dialogs from 26,507 APIs through self-evolution synthesis and multi-agent interactions.Its supervised fine-tuning pipeline uses Llama-3.1-8B-Instruct and dual-layer rule-based and model-based verification.
  • Baseline training details: ToolRL uses GRPO on a mixed 4,000-sample dataset, emphasizing reward design for multi-step interactions, irrelevant-tool detection, and generalization to unseen scenarios.The mixture contains 2,000 ToolACE samples, 1,000 masked Hammer samples, and 1,000 xLAM samples.
  • Controlled baseline evaluation: The baselines are re-trained on Qwen-2.5-1.5B-Instruct to isolate training-data and algorithmic effects from base-model differences.The authors use officially released datasets and aim for a controlled, fair evaluation of Tool-R0 against data-intensive approaches.
  • Implementation procedure: The re-implementations preserve each method’s training paradigm: SFT for xLAM, Hammer, and ToolACE, and GRPO with the authors’ default hyperparameters for ToolRL.SFT baselines are trained with LLaMA-Factory while ToolRL uses the authors’ official repository and codebase.

H Full Algorithm of Tool-R0 … Late Training (Iter 3): Complex User Request with Multi-step Tool-Calling

Tool-R0 alternates Generator training, curriculum construction, and Solver training across K co-evolutionary iterations, using rewards and tool-call correctness to drive zero-data self-play. The Generator’s outputs evolve from simple single-call tasks to complex multi-step requests with multiple constraints.

  • H Full Algorithm of Tool-R0: Algorithm 1 alternates Generator training, dataset construction, and Solver training for K co-evolutionary iterations, returning a trained Solver policy from a base LLM.The Generator is trained against a frozen Solver, generated tasks are curated into curricula, and the Solver learns from those curricula.
  • H Full Algorithm of Tool-R0: Generator rewards combine format and validity checks with curriculum rewards based on Solver difficulty and semantic alignment.The algorithm probes the frozen Solver M times, estimates success probability, computes a band-pass difficulty reward, and adds semantic alignment scoring.
  • H Full Algorithm of Tool-R0: Generated tasks are deduplicated, difficulty-estimated with pass@M, bucketed as easy, medium, or hard, and ordered into a domain-balanced easy-to-hard curriculum.The curriculum is constructed from the frozen Generator’s outputs before Solver training.
  • H Full Algorithm of Tool-R0: Solver rewards combine output-format compliance with answer-acceptance scoring over tool-call names, keys, values, and extra-call penalties.Solver training samples minibatches from the curated easy-to-hard curriculum.
  • Solver Prompt: The Generator prompt constrains domains, context, tool-menu cardinality, primitive-only arguments, and a strict four-block output schema.The required blocks are <think>, <question>, <available tools>, and <tool call answer>.
  • Early Training (Iter 1): Surface-level User Request: In Iteration 1, the Generator produces surface-level tasks with simple underspecified requests, one available function, and a single straightforward tool call.The example uses canonical arguments.
  • Late Training (Iter 3): Complex User Request with Multi-step Tool-Calling: By Iteration 3, the Generator synthesizes multi-constraint requests requiring ordered tool calls, such as booking a round-trip flight and centrally located hotel.The example incorporates temporal constraints, passenger count, and cabin-class preferences, with a solution that decomposes the request into multiple tool calls.
Loading 2602.21320v1…