PaperlayerView exact revision on arXiv

Source-linked AI summary

RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Ruishan Fang, Siyuan Lu, Chenyi Zhuang, Tao Lin

arXiv:2606.19047v1cs.AI

TL;DR

Static datasets lose informative samples as multi-turn tool-use agents improve, creating a data-scarcity challenge for RL. RODS dynamically detects capability-boundary tasks, synthesizes structurally matched variants, and achieves comparable performance to a 17K-sample offline pipeline with roughly 20× fewer trajectories.

  • Problem

    Static multi-turn tool-use RL datasets become less informative as agent capability boundaries shift, while high-quality data remains scarce.

  • Method

    RODS uses rollout reward variance to detect capability-boundary samples, then synthesizes structurally matched multi-turn variants and manages them in a co-evolving replay buffer.

  • Results

    RODS achieves comparable performance to large-scale offline synthesis with roughly 20× fewer trajectories and improves over fixed-data RL and environment augmentation in controlled runs.

  • Takeaways & Limitations

    The controlled results support boundary-targeted synthesis as a dynamic curriculum that sustains informative policy gradients under data scarcity.

  • Takeaways & Limitations

    RODS currently relies on deterministic simulation environments, limiting direct adaptation to opaque environments or remote MCP servers.

Abstract

from arXiv · show

Multi-turn tool-use RL is bottlenecked by the rapid depletion of informative samples in static datasets. We observe that the gradient signal in GRPO concentrates on tasks with the highest rollout reward variance, a consequence of the Popoviciu upper bound. Consequently, samples near the agent's capability boundary -- where successes and failures are roughly balanced -- contribute disproportionately large policy gradients. As training progresses, this boundary continuously shifts, which gradually depletes the pool of informative samples in a static dataset. We propose RODS (Reward-driven Online Data Synthesis) to resolve this depletion. RODS closes the loop between RL training and data generation by repurposing the progress reward variance as a practical, zero-cost boundary detector that requires no extra inference beyond the rollouts already computed for training. It continuously identifies such boundary samples, synthesizes new multi-turn variants matching their structural complexity (e.g., API topology and dependency depth) via a skill-aligned resampling pipeline, and manages a dynamic replay buffer that co-evolves with the policy. Starting from 400 human seeds and maintaining an active training pool of ~800 samples, RODS achieves comparable performance to a 17K-sample offline pipeline while requiring roughly 20x fewer trajectories, and improves over fixed-data RL and environment augmentation in our controlled setting.

1 Introduction

Multi-turn tool-use RL is limited by scarce high-quality data, shifting capability boundaries, and incoherent online synthesis. RODS addresses this gap by coupling data generation to RL training and using rollout reward variance as a capability-boundary detector.

  • Challenges: High-quality multi-turn datasets are scarce, static data loses learning signal as agent capabilities evolve, and online synthesis can lack semantic coherence.These challenges correspond to data scarcity (C1), shifting boundaries and signal depletion (C2), and semantic disjointedness (C3).
  • Prior approaches: Offline synthesis remains decoupled from training, while fixed-seed RL approaches are constrained by signal depletion as the model improves.Online self-play and self-evolution aim to close the loop but unconstrained zero-data generation can fail to preserve coherence in complex multi-turn interactions.
  • RODS: RODS tightly couples data generation with the RL training loop to dynamically synthesize data that tracks the evolving capability boundary while maintaining multi-turn semantic coherence.The framework is proposed specifically for training under extreme data scarcity.
  • RODS: Progress Reward in GRPO serves as a boundary detector because existing rollout statistics are already computed for advantage estimation and rollout variance peaks near the capability boundary.The passage identifies the boundary region as µ ≈0.5 and connects the variance behavior to the Popoviciu upper bound.

2 Related Work

Prior work expands tool-use data through large-scale offline synthesis, directed failure-mode targeting, environment simulation, curriculum learning, and self-play. These approaches leave a gap for methods that adapt data generation to a policy’s shifting capability boundary while preserving structural anchors from scarce human data.

  • Data synthesis for tool use: Human-annotated multi-turn tool-use benchmarks typically contain only hundreds of instances, limiting their usefulness for training reliable agentic policies.This scarcity has motivated a shift toward large-scale offline synthesis.
  • Data synthesis for tool use: Directed synthesis targets specific weaknesses: FunReason-MT addresses hard query generation using environment-API graphs, while LoopTool uses feedback to correct algorithmic errors.Both methods remain offline snapshots and cannot adapt to a shifting capability boundary during active RL training.
  • Online RL and curriculum learning: Simulation-based RL methods construct high-fidelity interactive loops to extract more signal from existing tasks, while EnvTuning improves efficiency through a four-stage curriculum.These approaches address agentic-RL data efficiency through environment simulation or corrective feedback.
  • Online RL and curriculum learning: Self-play generates tasks from scratch with zero data, but unconstrained generation often struggles with the long-horizon logical chains and interdependent API calls required for multi-turn tool use.RODS instead uses a critically small set of human data as structural anchors.

3 RODS

RODS addresses multi-turn data scarcity by coupling reward-based boundary detection, structurally aligned task synthesis, and dynamic replay management in a closed training-data loop. It targets tasks near the shifting capability boundary to sustain informative gradient signals.

  • RODS framework: RODS maintains a saturated learning signal through reward-based seed detection, skill-aligned synthesis, and dynamic replay buffer management.These three modules co-evolve to track and replenish informative training data.
  • Boundary detection: Average Progress Reward partitions tasks into mastered, boundary, and hard regions using thresholds α− = 0.20 and α+ = 0.85.Tasks with α− ≤ r̄_i ≤ α+ are treated as boundary seeds near the model’s immediate capability boundary.
  • Boundary detection: Boundary seeds are ranked by ϕ(r̄_i) = 4r̄_i(1 − r̄_i), prioritizing tasks nearest the capability midpoint r̄_i = 0.5.Per-type quotas promote diverse skill coverage, while a temporal exclusion window limits redundant sampling.
  • Skill-aligned synthesis: The synthesis pipeline generates novel variants that preserve each seed’s API dependency depth and parameter-flow complexity rather than merely paraphrasing it.It extracts a complexity profile Φ(x_seed) and samples x′ conditioned on an approximately matching profile.
  • Replay management: RODS uses staged injection and multi-layer retirement to keep the replay pool aligned with the shifting capability boundary while limiting distributional instability.Retirement includes burn-in filtering, boundary-drift eviction, variance-prioritized pruning, and removal of stale unsampled variants.

4 Experiments

RODS outperforms fixed-data RL and environment augmentation in controlled BFCL V3 experiments by continuously expanding training near the capability boundary. It achieves comparable performance to large-scale offline synthesis with a much smaller active pool, while ablations show that boundary targeting, continuous rewards, coherence rewriting, and pool refresh are important.

  • Controlled comparisons: All controlled RL methods use the same 400 training samples, GRPO configuration, and progress reward; only the Stage 3 strategy for gradient-signal depletion differs.The comparison isolates the effect of fixed data, environment enrichment, and dynamic boundary expansion.
  • Controlled comparisons: +33.87%: RODS raises Qwen3-4B-Instruct overall multi-turn performance to 56.00%, versus 50.00% for Static dataset and 50.50% for EnvTuning.RODS attains the highest overall scores across the three model families, with gains consistent across all four sub-splits.
  • Dynamic expansion: Over 800 unique tasks: RODS breaks the 400-task static limit by injecting variants and retiring mastered tasks while keeping the active pool bounded by Pmax.The sliding-window pool concentrates training data near the capability boundary, alleviating gradient starvation.
  • Dynamic expansion: Pmax =50 already improves meaningfully over the static baseline, while performance shows diminishing returns beyond Pmax =200 and supports Pmax =400 as a practical operating point.The boundary region of the 400 original samples becomes fully covered at larger pool sizes, indicating higher per-sample value for targeted boundary data.
  • Data efficiency: 56.00%: RODS matches FunReason-MT-4B on Long-Context and exceeds it on Missing Functions and Missing Parameters using an active pool of ∼800 samples versus 17K offline trajectories.The active pool comprises 400 original seeds plus up to Pmax =400 generated variants.
  • Ablations: −5.13%: removing coherence rewrite causes the largest overall drop, while random seed selection, binary accuracy, and retirement reduce performance by −4.75%, −3.25%, and −3.38%, respectively.These ablations support the roles of usable synthesis, boundary targeting, continuous credit, and continuous pool refresh.

5 Conclusion … B.1 Hyperparameter Specifications

RODS frames multi-turn tool-use RL as a dynamic curriculum problem, using progress-reward variance to detect shifting capability boundaries and sustain informative gradients. The paper formalizes sequential tool-use episodes and specifies implementation safeguards, while noting current limitations in opaque and remote environments.

  • 5 Conclusion: RODS treats static-data depletion as a dynamic curriculum problem and uses progress-reward variance near the shifting capability boundary for boundary detection.This signal is reused to sustain informative policy gradients as the agent improves.
  • 5 Conclusion: RODS currently relies on deterministic simulation environments implemented with executable Python objects to verify execution correctness and provide feedback.Extending the framework to opaque environments and remote MCP servers remains future work.
  • A Formulation of the Sequential Multi-turn Decision Process: A multi-turn tool-use episode is modeled as a POMDP representing one user task composed of sequential instructions called turns.The episode continues across turns until all instructions are completed.
  • A Formulation of the Sequential Multi-turn Decision Process: At each timestep, the policy samples an action from a predefined action space containing tool invocations and task-resolution responses.Tool invocations obtain API context, while task resolution completes the current sub-task and advances the interaction.
  • A Formulation of the Sequential Multi-turn Decision Process: Only the terminal timestep receives a sparse binary reward RT ∈{0, 1} reflecting overall task success or failure.This delayed feedback makes credit assignment and exploration difficult in long-horizon scenarios.
  • B System Implementation and Hyperparameters: The system implementation details the hyperparameters, engineering optimizations, and fault-tolerance mechanisms used in the experiments.These choices are presented as part of the system implementation and hyperparameter specification.
  • B.1 Hyperparameter Specifications: α−= 0.20 and α+ = 0.85 define the Boundary Zone, α+ retire = 0.95 defines the Mastered Zone, and α− retire = 0.20 defines the Too Hard Zone.The Boundary Zone is intended to capture partial but incomplete mastery and maximize gradient variance.
  • B.1 Hyperparameter Specifications: 20% of the active pool size is the per-epoch injection cap, while Pmax = 400 generated items bounds the active dataset and triggers priority-based eviction when exceeded.The original seed dataset is preserved and exempt from retirement.

B.2 Data Injection Latency Analysis … H Reward Details

The paper supplements boundary-targeted synthesis with asynchronous data handling, fault-tolerant execution, feedback-driven repair, validation, and staged reward definitions. Its analysis motivates targeting tasks near µ ≈0.5, where reward variance and potential GRPO gradient signal are highest.

  • B.2 Data Injection Latency Analysis: Variants enter training after an average 1-step synthesis delay and 13-step staging delay, while pass@16 changes by less than 0.10 on average.The delay is roughly 1–2 full epochs, yet variants remain in their originally classified zone and relevant to the evolving model.
  • B.3 System Engineering and Concurrency: The system uses a faithful simulation replica, isolated asynchronous generation processes, filesystem locking, structured <reason> prompts, and restartable append-only state.These mechanisms provide deterministic execution guarantees, prevent shared-memory races, stabilize structured outputs, and support independent daemon scaling.
  • B.4 Crash Recovery and Fault Tolerance: Zero data loss across restart boundaries is achieved by serializing tracker state, replaying immutable variant logs, and deterministically recomputing retirement logic.The curriculum resumes from the interrupted training state after daemon or trainer restarts.
  • B.5 Motivating Analysis: Gradient Variance at the Capability Boundary: Targeting tasks with average progress reward near µ ≈0.5 is motivated by maximal reward-variance potential and GRPO signal density at the capability boundary.For binary rewards, signal density is proportional to p(1 −p), vanishes as p →0 or p →1, and is maximized at p = 0.5; the continuous-reward analogue is a practical guideline.
  • C Agent Prompt Design: The synthesis pipeline generates structurally matched function plans, applies configuration patches to execution failures, rewrites coherent user queries, and classifies or repairs quality defects.The agents preserve structural complexity, diagnose environment-level failures, enforce natural multi-turn semantics, and distinguish query-fixable from ground-truth-unfixable samples.
  • D Deterministic Execution Pipeline Internals: The deterministic execution pipeline samples and decomposes functions, generates state-conditioned parameters, executes turns sequentially, and routes structured failures into repair feedback.Failures record their type, function, turn, and diagnostic detail for Config Patch Agent and Planner re-invocation.
  • E Error Taxonomy for Feedback-Driven Re-synthesis: The error taxonomy separates parameter, decomposition, function-sampling, and VM-execution failures from query-level errors, enabling targeted recovery strategies.Only the first four error types invoke configuration patching; query-level errors trigger Planner re-invocation with action-space constraints.

I Combination Experiment: RODS + EnvTuning

This experiment examines whether RODS and EnvTuning provide complementary gradient signals. RODS sustains climbing progress rewards, while combining both methods is presented as improving on either augmentation alone.

  • Combination Experiment: RODS + EnvTuning: RODS and EnvTuning are evaluated for complementary gradient signals through training progress rewards.The comparison is reported in Table 1.
  • Combination Experiment: RODS + EnvTuning: The static baseline without augmentation suffers from rapid reward saturation.
  • Combination Experiment: RODS + EnvTuning: RODS maintains a continuously climbing progress reward by introducing boundary tasks, whereas replacing static tasks alone lacks granular execution hints.

J Training Details · K Theoretical Extension to PPO

The paper provides system hyperparameters for RL training, data synthesis, and the dynamic replay buffer, then extends RODS’s capability probe from GRPO to PPO. The PPO extension uses advantage or reward variance to identify uncertain boundary tasks without architectural changes.

  • J Training Details: System hyperparameters cover RL training, data synthesis, and the dynamic replay buffer.The section presents the full RL-pipeline training hyperparameters and a configuration table for these components.
  • K Theoretical Extension to PPO: RODS’s capability probe naturally extends to PPO, although experiments focus on GRPO for memory efficiency.The empirical focus on GRPO is attributed to its memory efficiency, while PPO is treated as a compatible extension.
  • K Theoretical Extension to PPO: PPO estimates the advantage function using Generalized Advantage Estimation driven by TD-error δt = rt + γV(st+1) −V(st).The TD-error uses the reward, discount factor, and successive value estimates.
  • K Theoretical Extension to PPO: For binary or bounded task-level rewards, TD-error variance remains tightly coupled to final-reward variance across trajectories from the same prompt.The Progress Reward RP is given as an example of a bounded task-level reward.
  • K Theoretical Extension to PPO: At mastery or consistent failure, accurate value predictions produce near-zero TD-errors and vanishing gradients; uncertainty near p ≈0.5 marks the capability boundary.The passage contrasts RP →1 or RP →0 with highly uncertain outcomes around p ≈0.5.
  • K Theoretical Extension to PPO: Tracking Var(Â) or raw reward variance in the PPO rollout buffer serves the same function as the GRPO variance probe.Both signals can be used to detect the high-uncertainty region relevant to RODS.
  • K Theoretical Extension to PPO: RODS can therefore dynamically identify and synthesize boundary tasks in PPO without architectural changes.The extension preserves the capability-probe mechanism while operating on PPO rollout statistics.

L Out-of-Distribution Generalization Results

The section evaluates out-of-distribution generalization across BFCL V4, τ2-bench, and ACEBench Agent, finding that boundary-expanded training supports generalizable reasoning rather than only in-distribution pattern matching.

  • Evaluation scope: OOD performance is evaluated on BFCL V4, τ2-bench, and ACEBench Agent benchmarks.The section reports detailed performance metrics across all three benchmarks.
  • Generalization finding: Boundary-expanded training yields generalizable reasoning capabilities rather than mere in-distribution pattern matching.This is the section’s central conclusion about OOD generalization.
  • Comparison and validity: Table 4 compares results against Llama-3.1-8B-Instruct and marks RODS-trained models as improved on OOD tasks.xLAM’s Retail and Airline scores are invalid for OOD evaluation because it was trained on the original τ-bench.

M Synthesis LLM Robustness · N Synthesis Computational Cost

RODS remains robust when its synthesis backbone changes, indicating that performance comes from the framework rather than a particular LLM. Its asynchronous synthesis doubles GPU-hours relative to training but achieves substantial data efficiency with no training-loop slowdown.

  • M Synthesis LLM Robustness: Replacing Qwen3-32B with GLM-4.5-Air while holding RODS components fixed preserves nearly identical performance, supporting model-agnostic framework gains.The generated pool cap remains Pmax =400, with boundary detection, structural isomorphism, replay buffering, and training unchanged.
  • M Synthesis LLM Robustness: 0.75% overall performance difference (56.00% →55.25%) shows that the stronger GLM-4.5-Air backbone adds no further gains.Schema-guided planning, deterministic VM validation, and multi-tier quality filtering may normalize outputs across synthesis backbones under this setup.
  • M Synthesis LLM Robustness: GLM-4.5-Air leads on Missing Parameter (+2.00%) and Long Context (+2.00%), whereas Qwen3-32B leads on Base (+3.00%) and Missing Functions (+4.00%).These complementary sub-split biases suggest that multi-backbone synthesis ensembles could produce structurally diverse variants.
  • N Synthesis Computational Cost: Synthesis overhead equals 1× the training cost in GPU-hours, but its single vLLM instance performs no gradient computation and consumes fewer FLOPs.The synthesis cluster operates in parallel, so it does not block or slow the training loop.
  • N Synthesis Computational Cost: Training reaches 56.00% at approximately 56 hours wall-clock while training and synthesis clusters operate concurrently.The systems communicate asynchronously through filesystem-based queues, with seed emission by the trainer and variant ingestion at epoch boundaries.
  • N Synthesis Computational Cost: 896 GPU-hours total comprise 448 GPU-hours for training and 448 GPU-hours for synthesis on 16×A100 GPUs for approximately 56 hours.Training uses 8×A100 with Qwen3-4B-Instruct and synthesis uses 8×A100 hosting Qwen3-32B via vLLM with 64 workers.
  • N Synthesis Computational Cost: A successful base variant requires approximately 9–15 LLM calls, with miss_func and miss_param variants adding 2–5 adversarial-transform and verification calls.A 3-turn base variant includes planning, six query-generation and verification calls, coherence rewriting, and quality judging; 64 workers process calls concurrently.
  • N Synthesis Computational Cost: 20× data efficiency matches 17K-sample offline pipelines using an active training pool of approximately 800 samples despite doubled GPU footprint.RODS therefore trades additional synthesis compute for substantially fewer training data requirements.

O Benchmark and Evaluation Details

The evaluation covers BFCL V3 in-distribution learning and OOD generalization across unseen APIs, modalities, domains, and complex interaction patterns. RODS is assessed using established data splits and official benchmark evaluation procedures.

  • In-Distribution Evaluation (BFCL V3): BFCL V3 provides an 800-sample multi-turn in-distribution testbed divided equally among Base, Missing Function, Missing Parameter, and Long-Context categories.The categories test standard dependencies, missing capabilities, missing arguments, and extended conversational context, respectively.
  • In-Distribution Evaluation (BFCL V3): The in-distribution evaluation uses the exact 400/400 train/test split established by Lu et al. (2025) and the official BFCL AST matching evaluator.This setup is intended to ensure a fair comparison.
  • Out-of-Distribution Evaluation: OOD evaluation tests unseen APIs and interaction modalities using BFCL V4 Web Search and Memory, τ2-bench Retail, Airline, and Telecom, and ACEBench Multi-turn and Multi-step Agent splits.These benchmarks cover dynamic information retrieval, long-term state tracking, constrained business logic, and complex API topologies.
  • Out-of-Distribution Evaluation: All OOD evaluations strictly follow their respective official evaluation protocols and scoring scripts.This applies across BFCL V4, τ2-bench, and ACEBench.

P Synthesized Data Examples … Q Full Benchmark Results

RODS synthesizes novel multi-turn tool-use variants by preserving each seed’s structural skill pattern while changing content, parameters, and failure conditions. The examples span vehicle control, missing-function recovery, and missing-parameter clarification, followed by a full benchmark-results section whose supplied passage identifies controlled RL comparisons but reports no values.

  • P Synthesized Data Examples: RODS preserves the seed’s structural complexity while generating novel content in seed-to-variant pairs.The examples are presented side by side to show how structural complexity is retained during synthesis.
  • P.1 Base Type: VehicleControlAPI (Seed →Variant): The VehicleControlAPI seed combines fuel conversion, engine startup safety checks, and distance estimation across three turns.The seed anticipates converting 166 liters to gallons, starting the engine with safety checks, and estimating travel distance.
  • P.1 Base Type: VehicleControlAPI (Seed →Variant): The VehicleControlAPI variant retains the sequence of unit conversion, vehicle operation, and multi-step distance planning while changing functions and parameters.Both examples use 1, 2, and 4 calls respectively, with a dependency chain in distance planning.
  • P.2 Missing Function Type: GorillaFileSystem (Seed →Variant): The GorillaFileSystem seed requires refusing a removed rm function and then recovering to inspect the current directory, including hidden files.The task begins by locating and attempting to remove a file and directory before the unavailable function forces refusal.
  • P.2 Missing Function Type: GorillaFileSystem (Seed →Variant): The GorillaFileSystem variant changes file operations by removing mkdir instead of rm, requiring refusal and recovery in Turn 4.The agent refuses the unavailable folder-creation request, then executes mkdir after the function is restored.
  • P.3 Missing Parameter Type: TradingBot + MathAPI (Seed →Variant): The TradingBot + MathAPI seed requires clarification when a vague stock reference leaves the requested average under-specified.The user initially requests an average of price, trading volume, MA5, and MA20, then identifies AAPL as the intended stock.
  • P.3 Missing Parameter Type: TradingBot + MathAPI (Seed →Variant): The TradingBot + MathAPI variant preserves clarification of vague references while changing the stock, computation, and missing numerical values.The agent asks for clarification instead of executing the under-specified average request, then computes the mean after receiving −3.4 and −1.0.
  • Q Full Benchmark Results: Table 6 defines full in-distribution BFCL V3 multi-turn results for Tier 1 controlled RL comparisons using 400 shared training samples and GRPO.The supplied table passage states that only the data/environment strategy differs and that red text marks improvement over the base model, but provides no numerical results.

R Ablation Configuration Details

The ablations isolate RODS components across boundary detection, synthesis quality, and lifecycle management. Conditions replace boundary-focused sampling or progress rewards, remove synthesis mechanisms, or disable pool pruning and refresh.

  • Boundary detection ablations: Boundary detection ablations sample seeds uniformly from the training pool or replace continuous progress reward RP with binary task accuracy.These conditions test boundary-targeted capability tracking and whether progress reward’s partial credit is necessary.
  • Synthesis pipeline ablations: Synthesis pipeline ablations independently generate per-turn queries without coherence rewrite or produce function sequences without narrative planning.These variants remove holistic semantic grounding or cross-turn thematic coherence anchored to a unified goal.
  • Synthesis pipeline ablations: The blind-retry ablation removes the Error Critic and Config Patch Agent, retrying failed executions with a fresh random plan instead of corrective signals.This tests the contribution of feedback accumulated after execution failures.
  • Lifecycle management ablations: The no-retirement ablation disables all three retirement layers (L1–L3), allowing the pool to grow without pruning.This isolates the effect of lifecycle-based removal.
  • Lifecycle management ablations: The static-pool ablation generates variants once at the beginning of Stage 3 and freezes the pool, preventing refresh during training.No new variants are synthesized as training progresses.
Loading 2606.19047v1…