PaperlayerView exact revision on arXiv

Source-linked AI summary

From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents

Jiaxuan Gao, Jiaao Chen, Chuyi He, Shusheng Xu, Di Jin, Yi Wu

arXiv:2601.22607v3cs.AIcs.CL

TL;DR

Interactive tool-using agents require scalable multi-turn data and stable reinforcement learning despite uncertain user simulation. The paper combines self-evolving synthetic-data generation with verifier-based RL, achieving strong τ^2-bench results for open-weight Qwen3 models. The framework offers a scalable route to training complex tool-use behavior, while evaluation remains constrained to benchmark environments with explicit schemas and verifiers.

  • Problem

    Scalable high-quality multi-turn tool-use data is difficult to obtain, while user-simulator instability introduces noisy RL signals during interactive rollouts.

  • Method

    The framework combines AReaL-SEA self-evolving synthesis of tool-grounded dialogues and executable checkers with user-model SFT and verifier-based GRPO training.

  • Results

    73.0 pass^1 on Airline and 98.3 pass^1 on Telecom are achieved by Qwen3-235B-A22B-2507 with RL, with competitive performance relative to reported proprietary baselines.

  • Takeaways & Limitations

    Self-evolving synthetic data and stabilized verifier-based RL can reliably improve multi-turn tool-use capabilities in open-weight models.

  • Takeaways & Limitations

    The approach is trained and evaluated within constrained benchmark environments with explicit tool schemas and verifiers, while broader deployment may introduce misuse risks.

Abstract

from arXiv · show

Interactive tool-using agents must solve real-world tasks via multi-turn interaction with both humans and external environments, requiring dialogue state tracking, multi-step tool execution, while following complex instructions. Post-training such agents is challenging because synthesis for high-quality multi-turn tool-use data is difficult to scale, and reinforcement learning (RL) could face noisy signals caused by user simulation, leading to degraded training efficiency. We propose a unified framework that combines a self-evolving data agent with verifier-based RL. Our system, EigenData, is a hierarchical multi-agent engine that synthesizes tool-grounded dialogues together with executable per-instance checkers, and improves generation reliability via closed-loop self-evolving process that updates prompts and workflow. Building on the synthetic data, we develop an RL recipe that first fine-tunes the user model and then applies GRPO-style training with trajectory-level group-relative advantages and dynamic filtering, yielding consistent improvements beyond SFT. Evaluated on tau^2-bench, our best model reaches 73.0% pass^1 on Airline and 98.3% pass^1 on Telecom, matching or exceeding frontier models. Overall, our results suggest a scalable pathway for bootstrapping complex tool-using behaviors without expensive human annotation.

1. Introduction

Interactive tool-use agents must gather information through multi-turn dialogue while executing complex tool actions, making scalable data synthesis and stable RL difficult. The framework combines self-evolving synthetic data with verifier-based RL and reports strong τ^2-bench performance using open-weight models.

  • Challenges: Interactive agents must elicit preferences and private details over multiple turns while handling uncertain user behavior.Users may provide information incrementally, change their minds, or respond unexpectedly.
  • Challenges: High-quality multi-turn tool-use data is difficult to obtain at scale because complex domain rules and coherent simulated-user information must be satisfied simultaneously.The paper identifies scalable data acquisition as a primary post-training bottleneck.
  • Challenges: User-simulator instability introduces noisy RL signals and undermines rollout success, especially when users also invoke tools.The framework addresses this by first fine-tuning the user model, then using stabilized GRPO-style training with large batches, dynamic sampling, and verifier-based rewards.
  • Results: 73.0 pass^1 on Airline and 98.3 pass^1 on Telecom are achieved by Qwen3-235B-A22B-2507 with RL, matching or exceeding frontier models.The framework reports improvements across Airline, Retail, and Telecom on τ^2-bench.
  • Framework: AReaL-SEA generates verifiable, complex, high-quality multi-turn tool-use instances through a self-evolving data synthesis system.Its execution layer produces trajectories and executable per-instance verification functions for RL rewards.
  • Contributions: The contributions include an RL recipe with user-model fine-tuning, large-batch training, dynamic sampling, and verifier-based outcome rewards.The paper also reports extensive ablations and state-of-the-art τ^2-bench results using fully open-weight models.

2. Related Work

Related work establishes tool-using agents, grounded synthetic trajectory generation, and verifier-based reinforcement learning as the main foundations for this framework.

  • Tool-using Language Agents: Tool-using agents extend language models with APIs, browsers, and search engines, with ReAct interleaving reasoning traces and actions.Toolformer further demonstrated self-supervised learning of when and how to invoke tools.
  • Synthetic Data Generation: Synthetic-data research increasingly grounds trajectory generation with execution-based validation and extends function-calling data toward multi-turn tool use.APIGen and APIGen-MT exemplify this progression.
  • Reinforcement Learning for LLM: RLHF uses learned reward models for human preferences, while RLVR uses programmatic verifiers to evaluate model outputs on complex reasoning tasks.The paper positions agentic RL as an extension of this verifier-based direction.

3. Preliminary

The paper models interactive tool use as a two-player Dec-POMDP in which agent and user policies act through partially observed, state-changing interactions. Training optimizes terminal task correctness.

  • Problem Formulation: Interactive tool use is formulated as a Dec-POMDP with environment states, actions, transitions, rewards, observations, and agent and user policies.The agent policy is πθ and the user policy is πuser.
  • State and Observation: The state captures all environment information, while transitions model updates caused by actions such as tool calls.Both players observe local observations derived from the current state.
  • State and Observation: At each turn, agent and user observations include local context such as task instructions, tool specifications, and interaction history.Either participant may act on a turn, while the non-acting party contributes an empty action.
  • Reward: The outcome-reward formulation gives feedback only at the terminal turn, where a scalar evaluates the correctness of the final state.Intermediate turns receive zero reward.
  • Policies: The agent policy generates text from its observation, and a heuristic parser extracts actions such as tool calls from that output.The user policy similarly generates actions conditioned on its own observation.
  • Objective: The objective is to maximize expected terminal reward over trajectories generated jointly by the agent and user policies.The trajectory includes states and joint actions through the terminal turn.

4. AReaL-SEA: Self-Evolving Data Synthesis Framework

AReaL-SEA generates diverse, tool-grounded training data through independently evolving synthesis streams. Each stream synthesizes and verifies tasks and trajectories, routes failures to reflection, and contributes verified trajectories to the final dataset.

  • Framework overview: AReaL-SEA combines meta-planning, sequential synthesis and verification, trajectory collection, and reflection into a self-evolving data pipeline.The pipeline generates diverse plan pairs, processes each through specialized agents, and iteratively improves generation using failure feedback.
  • Diversified plan generation: The meta-planning module creates distinct synthesis–evaluation plan pairs covering domains, complexity levels, tool-use patterns, and interaction styles.Synthesis plans specify task distributions and diversity constraints, while evaluation plans define quality criteria and failure taxonomies.
  • Diversified plan generation: Each plan pair becomes an independent stream, decoupling diversity from one stochastic generation process and enabling plan-specific reflection.The final dataset is the union of verified trajectories produced across streams.
  • Agent pipeline: The agent pipeline synthesizes structured tasks, verifies them, rolls out assistant–user interactions, and verifies complete trajectories before dataset inclusion.Failed tasks and trajectories are routed to reflection with rationales; trajectory failures also receive TASK or TRAJECTORY attribution.
  • Reflection and self-evolution: Reflection aggregates verification failures and revises synthesis and evaluation plans, allowing successive iterations to reduce failures and improve trajectory quality.Independent streams maintain separate reflection loops so domain-specific failures drive targeted refinements without cross-contamination.

5. Reinforcement Learning for Interactive Tool-using Agent

The framework adapts GRPO to interactive agents using trajectory-level group-relative advantages, token-level objective normalization, dynamic filtering, and state-based binary rewards. It also fine-tunes the user simulator because unreliable simulated users can turn correct agent behavior into zero-reward failures.

  • Group-relative optimization: The method adapts Group Relative Policy Optimization with group-relative advantages for interactive-agent training.For each task, independent trajectories are sampled and rewards are normalized relative to their group.
  • RL objective: The RL objective uses token-level clipping and normalization and sums surrogate loss across tokens from all trajectories.Trajectory token counts determine the normalization scope in the complete objective.
  • State-based reward: State-based verification compares trajectory entities and actions with the ground-truth final state, assigning a binary reward only to full matches.Partial matches are not evaluated as successful under this verification process.
  • Dynamic filtering: Dynamic filtering excludes tasks whose sampled trajectories all receive identical rewards, retaining groups with meaningful outcome variation.Identical all-success or all-failure groups yield zero group-relative advantage for every trajectory.
  • User model fine-tuning: Fine-tuning the user simulator is necessary because unreliable tool-using user behavior can cause task failures and noisy RL rewards.The supplied figure contrasts a base user model that exhausts irrelevant tools with an SFT-trained model that follows instructions and enables success.

6. Experiment

Experiments on τ 2-bench show consistent gains from SFT and RL across domains, while ablations identify data quality, user-model quality, batch size, and dynamic filtering as important factors.

  • Main Results: SFT and RL consistently improve performance across τ 2-bench domains and model sizes.
  • Mix Training: 81.3% average pass^1 and 68.5% average pass^4 in mix training surpass Qwen3-Max-Thinking and GPT-5.
  • Data Ablation: 56.0% pass^1 from the full synthetic pipeline approaches 52.0% from human expert data, while removing validation, evolution, or diversity reduces performance.
  • User Model Ablation: 95.6% pass^1 with a fine-tuned user model contrasts with 75.6% using a base user model, a 20% performance gap.
  • Algorithm Ablation: Increasing total batch size from 256 to 512 improves pass^1 to 70.5%, while disabling dynamic filtering lowers pass^1 from 70.5% to 65.0%.

7. Conclusion

The paper presents a scalable framework combining self-evolving synthetic data with verifier-based reinforcement learning for long-horizon tool-using agents. Across τ 2-bench, it improves open-weight Qwen3 models, with ablations supporting the necessity of both data synthesis and RL stabilization components.

  • AReaL-SEA combines self-evolving multi-agent data synthesis with reinforcement learning under verifiable rewards.
  • The framework synthesizes multi-turn tool-use dialogues and executable per-instance checkers for SFT and simulator-in-the-loop GRPO training.
  • Across Airline, Retail, and Telecom, the approach substantially improves open-weight Qwen3 models and competes with reported proprietary baselines.
  • Ablations indicate that self-evolving data synthesis and RL stabilization components are necessary for reliable gains.

Impact Statement

The framework offers a scalable post-training approach for long-horizon tool-using agents while reducing reliance on human annotation. It also highlights misuse risks and recommends safeguards for deployment.

  • The framework combines self-evolving synthetic data generation with reinforcement learning under verifiable rewards.It uses constrained benchmark environments with explicit tool schemas and verifiers during training and evaluation.
  • The approach can lower reliance on expensive human annotation and support reproducible, execution-grounded training signals.
  • More capable tool-using systems may increase misuse risks, including harmful workflows or unauthorized actions.The authors recommend strict permissioning, auditing, policy enforcement, monitoring, and responsible release practices.

1. Experiments Details

The experiments evaluate the approach on τ 2-bench across Airline, Retail, and Telecom using two Qwen3 MoE model sizes and standardized pass^k evaluation. Training uses an asynchronous infrastructure and reported reproducibility settings.

  • Experiment Setup: τ 2-bench tests multi-turn dialogue management and multi-step tool execution across Airline, Retail, and Telecom.The benchmark covers flight services, e-commerce operations, and mobile-plan management.
  • Evaluation: Evaluation uses GPT-4.1 as the user simulator and pass^k, requiring all k independent attempts to succeed.
  • Models: Experiments compare Qwen3-30B-A3B and Qwen3-235B-A22B against proprietary frontier models.
  • Training Infrastructure: The training framework asynchronously overlaps rollout generation with policy training to improve GPU utilization.The 30B and 235B models use 64 and additional multi-node H200 resources, respectively.
  • Training Details: SFT and RL use long-context training configurations with maximum context length up to 32,768 tokens.The reported settings include batch sizes, learning rates, trajectory counts, and an 8,192-token generation cap per turn.

2. More Experimental Results

Additional experiments examine RL training curves and the effect of mixing domains. Larger models retain nearly identical SFT performance under mixed training, whereas smaller models show performance decay attributed to domain interference.

  • Training Curves: The training curves report p1 and p4 against training steps for separate and mixed training.Separate-training step counts differ because domains contain varying amounts of training data.
  • Separate vs. Mix Training: 74.7% versus 74.5% average pass^1 shows nearly identical mix- and separate-training performance for Qwen3-235B-A22B.Individual-domain differences are marginal for the larger model.
  • Separate vs. Mix Training: 71.5% to 63.7% average pass^1 marks the performance drop for Qwen3-30B-A3B under mix training.The results suggest smaller models have limited capacity for diverse multi-domain data, causing interference between domains.
  • Separate vs. Mix Training: Table 1 compares SFT performance between domain-specific and combined-domain training.

3. More Details on AReaL-SEA

AReaL-SEA uses structured prompts and staged workflows to synthesize and evaluate tool-grounded tasks. Its instructions constrain task categories, tool-call formats, evaluation rules, and response sequencing.

  • Synthetic Task Generation: Synthetic-task generation uses task descriptions, examples, proposed plans, and available file or environment tools.
  • Synthesis Planning: The synthesis workflow generates a plan focused on exactly one task category, followed by information gathering, optional instance creation, and task synthesis.
  • Evaluation Rules: Evaluation rules require concise criteria and an [ACCEPT] or [REJECT] outcome without being overly restrictive.
  • Tool-Use Constraints: Tool-use prompts require valid JSON calls, complete parameters, matching types, descriptive values, and parameter validation.
  • Response Protocol: Generation instructions separate function calls from final task generation and permit either one or more calls or a generated task.
  • Quality Evaluation: Quality evaluation assesses the synthetic task against rules, produces an evaluation report, and identifies up to five rule relaxations when appropriate.
  • Quality Evaluation: Evaluation prompts also require explicit reasoning before calls or reports and constrain task-specific tool use.
  • Rule Improvement: The final improvement process may declare existing rules sufficient or recommend changes to avoid excessive restrictiveness.
Loading 2601.22607v3…