PaperlayerView exact revision on arXiv

Source-linked AI summary

Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits

Chen Xi, Ding Ruixue, Wang Zhigaoyuan, Zhang Fanrui, Zhang Qiang, Chen Boli, Wang Shihang, Wang Tianfu, Ren Ailin, Wang Pengyong, Xie Pengjun, Xiong Hui

arXiv:2610.13438v1cs.AI

TL;DR

On-policy agent RL can produce sparse successful trajectories beyond a policy’s capability boundary, while interventions introduce off-policy shifts. RTI uses reward-aware corrections, filters excessive shifts, and reweights consequential tokens; across benchmarks, it improves performance, with especially large gains on difficult tasks.

  • Problem

    When tasks exceed a model’s capability boundary, on-policy successful trajectories become extremely sparse, and teacher-guided methods remain bounded by teacher capability.

    Source: §1 · p. 1

  • Method

    RTI corrects failed rollouts using reward-side evidence, filters interventions with excessive distribution shifts, and reweights tokens that carry reasoning changes.

    Source: §1 · p. 2, §4.2 · p. 5, §4.3 · p. 6

  • Results

    Across five benchmarks and two backbones, RTI substantially improves task success, with particularly large gains on Navigation, τ2-Telecom, and Claw-Eval-Hard20.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Training analyses show that intervention supplies successful trajectories early, while the policy increasingly succeeds without intervention as learning proceeds.

    Source: §6 · p. 10

    Intervention trajectories are off-policy and can leak privileged information or cause distribution shifts that suppress useful learning signals and destabilize optimization.

    Source: §1 · p. 2

Abstract

from arXiv · show

Reinforcement learning (RL) has emerged as a key paradigm for improving large language model (LLM) agents, enabling stronger reasoning, planning, and tool-use in interactive environments. However, existing agent RL methods remain fundamentally constrained by an on-policy reasoning bottleneck: when tasks exceed the model's current capability boundary, effective trajectories become extremely sparse and training often collapses. We introduce Rollout-Time Intervention (RTI), a technique designed to elicit successful trajectories beyond the policy's intrinsic exploration capacity. RTI operates by leveraging reward-side evidence (i.e., evaluation rubrics) during rollout to convert erroneous trajectories into successful ones, providing effective learning signals for tasks that the policy rarely solves on its own. Although intervention trajectories are inherently off-policy, we find that they can effectively guide policy optimization when trajectories with excessive distributional shifts are filtered out. Extensive experiments across five benchmarks demonstrate that RTI consistently outperforms RL baselines, enabling the post-trained policy to approach the performance of substantially stronger contemporary LLMs and, more importantly, transcend its original capability boundary. The code is available at https://github.com/Alibaba-NLP/qqr.

1 Introduction

On-policy RL produces sparse successes beyond a model’s capability boundary, while teacher-guided gains remain bounded by teacher capability. RTI corrects failed rollouts and filters and reweights interventions to support learning beyond unaided exploration.

1 Introduction

  • When tasks exceed the model’s capability boundary, on-policy successes become extremely sparse and training can become unstable or collapse; teacher-guided gains remain bounded by teacher capability.The paper links many failures to early task-critical reasoning or action errors despite the agent possessing the needed knowledge and tools.Source: §1 · p. 1
  • The motivating observation is that correcting an early task-critical error can let the remaining trajectory proceed normally, raising the possibility of learning beyond the policy’s original reasoning limits.Figure 1 is described as showing RTI enabling successful trajectories where on-policy and teacher-guided RL fail.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Intervention creates off-policy trajectories, risking privileged-information leakage, implausible shortcuts, excessive distribution shifts, and unstable PPO-style updates.Large shifts can push importance ratios beyond the clipping range, while low-probability actions may produce high-variance gradients.Source: §1 · p. 2
  • RTI combines reward-aware multi-agent intervention, perplexity-based distribution gating, and reasoning-shift-aware token reweighting to learn from successful trajectories beyond unaided exploration.The intervention transforms failed rollouts while preventing privileged-information leakage; gating filters excessive shifts, and reweighting focuses updates on critical tokens.Source: §1 · p. 2

2 Related Work

Prior agent-training work includes data-synthesis-driven supervised fine-tuning, pure RL with verifiable rewards, and teacher-guided training. These approaches have different dependencies, including trajectory synthesis and stronger teacher capability.

Agentic Reasoning for LLMs

  • Agentic reasoning combines planning, acting, and learning through interaction, with long-horizon planning, partial observability, and feedback as central demands.The section situates the paradigm in tool-use, search-based, and workflow-oriented systems.Source: §2 · p. 3
  • Data-synthesis-driven SFT improves tool use with generated interaction trajectories, while pure RL with verifiable rewards has shown promise when outcomes are easy to verify.The cited SFT approaches depend on large-scale trajectory synthesis; pure-RL examples include code execution, search, simulated retrieval, and fine-grained tool-use rewards.Source: §2 · p. 3

Teacher-Guided Training for LLMs

  • Teacher-guided methods provide token-level credit assignment or off-policy successful trajectories, but their gains fundamentally depend on stronger teachers and remain bounded by teacher capability.LUFFY combines offline expert data with online rollouts and uses importance sampling for distribution shift.Source: §2 · p. 3

3 Preliminaries

The paper formulates agentic tasks as rewarded trajectory generation and describes GRPO as a clipped-surrogate policy optimization method with KL regularization.

Problem Setting

  • An agentic task generates a multi-step trajectory interleaving reasoning, tool invocations, environmental feedback, and a final answer, evaluated by a binary verifiable reward.The policy generates trajectories for queries sampled from a task distribution.Source: §3 · p. 3

Group Relative Policy Optimization

  • GRPO samples a group of trajectories for each query and updates the policy by maximizing a clipped surrogate objective.The objective uses per-token probability ratios against the behavior policy and a KL-regularization strength parameter.Source: §3 · p. 3

4 Rollout-Time Intervention

RTI turns failed rollouts into policy-aligned successful trajectories and uses distribution gating and token reweighting to make their learning signals more targeted. The intervention pipeline checks for task-critical errors, while gating and reweighting address off-policy shifts.

4 Rollout-Time Intervention

  • RTI’s three components are reward-aware multi-agent intervention, perplexity-based distribution gating, and reasoning-shift-aware token reweighting.They correct task-critical errors, filter excessive deviation from the rollout policy, and emphasize consequential intervention-induced tokens.Source: §4 · p. 4

4.1 Reward-Aware MAS-Based Intervention

  • Intervention is invoked only when every trajectory in a query’s group fails; the validator flags harmful deviations likely to derail success, rather than suboptimal but recoverable choices.It uses reward-side evidence and observable history to judge whether the current step remains compatible with task completion.Source: §4.1 · p. 4
  • The corrector minimally rewrites a detected critical error while preserving the policy’s style and local reasoning; the verifier checks gold-path consistency, information leakage, and behavioral plausibility.Corrections should remain derivable from visible history and stay close to the original policy behavior.Source: §4.1 · p. 5

4.2 Perplexity-Based Distribution Gating

  • Perplexity-based gating rescales an intervention under the rollout policy using only original observable history, then retains responses below a threshold because higher perplexity signals greater deviation.The perplexity is computed over tokens in the intervened response.Source: §4.2 · p. 5

4.3 Reasoning-Shift-Aware Token Reweighting

  • Token reweighting focuses updates on corrected tokens that carry key reasoning changes, using their likelihood ratio under correction and rollout prompts, with extreme ratios clipped around 1.A larger ratio indicates a token is more characteristic of the corrected reasoning path than the original rollout path.Source: §4.3 · p. 6
  • Reweighting applies only to intervened trajectories; when a group already contains a successful rollout, RTI sets the coefficient to 1 and reduces exactly to GRPO.The reweighting is applied to the token-level GRPO clipped surrogate.Source: §4.3 · p. 6

5 Experiments

Across five benchmarks, RTI improves agent performance, with its largest gains on difficult tasks. The experiments assess the method across backbones, component ablations, threshold settings, training dynamics, and reasoning-pattern changes.

5.1 Experimental Setup

  • Evaluation spans Navigation, three τ2-Bench domains, and Claw-Eval-Hard20, using Pass@k and Passk success metrics for k = 1, . . . , 4.Navigation uses real APIs, and a successful trajectory must invoke the direction tool with correct parameters; Claw-Eval-Hard20 comprises 20 challenging tasks on which the base policy rarely succeeds.Source: §5.1 · p. 6

5.2 Main Results

  • RTI consistently improves performance, with gains of 24.56 and 22.00 Pass@1 points over GRPO on τ2-Telecom and Navigation, respectively.On Claw-Eval-Hard20 with Qwen3-8B, RTI reaches 50.00% Pass@1 versus 26.25% for OPD and 30.00% for LUFFY; with Qwen3.5-9B, it reaches 85.00%, 23.75 points above LUFFY. Tables 1 and 2 report main results for the Qwen3-8B and Qwen3.5-9B backbones.Source: §5.2 · p. 7, Table 1 · p. 7, Table 2 · p. 8

5.3 Ablation Studies

  • Removing Post-Check nearly eliminates RTI’s τ2-Telecom gains and reduces Navigation Pass@1 below GRPO, to 26.00 versus 35.33.Without integrity verification, admitted interventions are frequently leak-contaminated or otherwise unlearnable, and can harm the policy.Source: §5.3 · pp. 7–8, Table 3 · p. 8
  • Removing PPL gating lowers τ2-Telecom Pass@1 from 61.40 to 54.39, while removing reweighting causes systematic but milder degradation across sixteen metrics.The gating ablation also reduces Pass2 from 46.49 to 27.19 and Pass4 from 31.58 to 18.42; reweighting’s largest gaps occur on all-of-k metrics.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 Hyperparameter Study

  • On Navigation, performance improves as δ rises from 3 to 5 and degrades at 20; δ = 5 gives the highest final success rate.δ = 0 rejects all interventions, while δ = +∞ applies no filtering; δ = 20 has the highest volatility, and the experiments adopt δ = 5.Source: §5.4 · pp. 8–9, Figure 3 · p. 9

5.5 Further Analysis

  • On Navigation, RTI’s preintervention rollout success rises above GRPO’s, alongside improving evaluation Pass@4.The early gap between RTI’s pre- and post-intervention rollout Pass@8 shows that intervention converts many failed rollout groups into successful trajectories.Source: Figure 4 · p. 9, §5.5 · p. 9
  • During training, PPL on successful intervened steps falls, while RTI maintains a broader interaction-turn distribution than GRPO, including trajectories exceeding three turns.The authors interpret the PPL trend as evidence that intervened reasoning is absorbed into native behavior; the turn distribution retains non-trivial mass above three turns at later stages.Source: Figure 5 · p. 9, §5.5 · pp. 9–10
  • On Claw-Eval-Hard20, both self-intervention RTI and stronger-model intervention sustain gains while GRPO’s scores collapse, but standard RTI performs strongest.The authors attribute the stronger model’s weaker learnability to corrections that deviate further from the policy distribution.Source: Figure 6 · p. 10, §5.5 · p. 10

6 Conclusion

RTI combines reward-aware self-intervention, perplexity-based filtering, and token reweighting to help policies learn beyond unaided exploration. Across five benchmarks and two backbones, it improves task success, with training analyses showing increasing success without intervention.

6 Conclusion

  • Across five benchmarks and two backbones, RTI substantially improves task success, with particularly large gains on Navigation, τ2-Telecom, and Claw-Eval-Hard20.The conclusion also reports that a comparison using a much larger intervening model confirms RTI does not require a stronger teacher to achieve these gains.Source: §6 · p. 10
  • RTI combines reward-aware self-intervention, perplexity-based filtering, and reasoning-shift-aware token reweighting to focus updates on consequential decisions.The intervention corrects failed rollouts, the gate filters excessively shifted trajectories, and token reweighting focuses policy updates.Source: §6 · p. 10
  • Training analyses show that intervention supplies successful trajectories early, while the policy increasingly succeeds without intervention as learning proceeds.The conclusion reports that RTI enables agent policies to learn beyond their unaided exploration capacity.Source: §6 · p. 10

AI use statement

The appendices document the ethics and reproducibility statements, RTI’s multi-agent prompts, and the full training algorithm.

AI use statement

  • The authors used generative AI for proofreading, grammar checking, and language refinement, while stating that the research ideas, experiments, claims, and final content were authored, verified, and approved by them.Source: AI use statement · p. 11

Ethics statement

  • The ethics statement says the authors intend to follow applicable rules and conduct the work with integrity; no personally identifiable information was collected.The data are intended for research purposes.Source: Ethics statement · p. 11

Reproducibility statement

  • The reproducibility statement says the appendix contains the RTI algorithm, implementation details, hyperparameters, benchmark and baseline descriptions, proofs, prompts, and case studies.Source: Reproducibility statement · p. 11

A.1 RTI MAS Prompt Templates

  • The RTI prompt templates specify three multi-agent roles: a validator, a corrector, and a verifier.The validator checks whether a step remains on track; the corrector repairs errors while concealing privileged information; and the verifier checks for leakage or hallucination.Source: §A.1 · p. 14, §A.1.1 · pp. 14–15

A.1.1 τ2-Bench Prompts (Telecom / Retail / Airline)

  • The τ2-Bench prompts make the validator reject hard-rule violations, including user-side device actions called by the agent and missing confirmations before database updates.The corrector minimally repairs clear errors, while the verifier rejects privileged-information leakage or fabricated content.Source: §A.1.1 · pp. 14–15

A.1.2 Navigation Domain Prompts

  • The Navigation prompts use reward-side evidence as a silent reference, with validation focused on hard rules and correction designed to preserve style while avoiding leakage.The verifier rejects leakage and content that cannot be derived from the visible question and history.Source: §A.1.2 · p. 16

A.2 The Full RTI Algorithm

  • RTI first runs an on-policy GRPO group; only if every rollout fails does it try an intervention, and only a successful, accepted correction replaces one failed member.Accepted corrections must pass the perplexity gate, and the update uses cached prompt log probabilities for the intervened trajectory.Source: §A.2 · p. 17

B.1 Convergence Analysis of RTI

The convergence analysis considers RTI’s mixture of on-policy and intervened updates, which introduces distribution shift. Under stated conditions, it establishes an O(1/K) stationarity guarantee.

B.1 Convergence Analysis of RTI

  • RTI’s off-policy trajectories and token reweighting raise a distribution-shift concern, motivating boundedness assumptions for the intervened updates.The theorem also assumes the expected update direction is aligned with the true gradient, with a lower bound governed by the on-policy clip margin.Source: §B.1 · p. 17
  • Under the stated alignment and boundedness conditions, RTI has an O(1/K) stationarity guarantee.Source: §B.1 · p. 17
  • The analysis models each update as a mixture of off-policy intervened and on-policy GRPO updates, with λ representing the expected training mix.Intervention occurs only for fully failed groups, so λ is data-dependent in practice but treated as constant in the analysis.Source: §B.1 · p. 17

B.2 Proof of Theorem 1

The proof bounds RTI’s convergence under assumptions on the on-policy and intervened update sources. It obtains the same asymptotic stationarity order as clipped policy-gradient methods, with an additive intervention-dependent term.

B.2 Proof of Theorem 1

  • The objective combines off-policy intervened and on-policy rollout data using a proportion λ, and applies bounded token-level reweighting to intervened rollouts.The intervened-rollout weight is bounded within [1 − ϵcal, 1 + ϵcal] and is detached from the computation graph.Source: §B.2 · p. 19
  • The proof assumes cumulative second-moment boundedness for off-policy updates and step-wise second-moment control for on-policy updates.The stated assumptions abstract, rather than rigorously derive from, the reweighting and clipping mechanisms.Source: §B.2 · pp. 19–20
  • Under the stated assumptions, RTI retains the O(1/K) stationarity rate, with an additive bound term that grows with intervention mixing and clip width and vanishes when λ = 0.At λ = 0, the bound reduces to the GRPO-style bound.Source: §B.2 · pp. 19, 21

C.1 Implementation Details.

This section details the training and evaluation setup, including the construction and assessment of Navigation, τ2-Bench, and Claw-Eval-Hard20 under strict success criteria. It also presents a τ2-Telecom MAS walkthrough of intervention repairing a rollout error while retaining the policy’s reasoning direction, alongside broader experimental-detail analyses.

C.1 Implementation Details.

  • RTI uses Qwen3-8B and Qwen3.5-9B backbones, with the policy model filling all three intervention roles; RL uses binary rewards and masks environment feedback from the loss.Training uses Adam at 1 × 10−6, weight decay 0.1, rollout temperature 0.8, group size 8, and clipping thresholds 0.2; training runs on eight NVIDIA H20 GPUs. Teacher-based baselines use larger same-generation models, while τ2-Bench uses Qwen3.5-397B-A17B as user simulator.Source: §C.1 · p. 21

C.2.3 Claw-Eval-Hard20

  • Claw-Eval-Hard20 uses an official LLM judge and counts a task as passed only when its safety-gated score exceeds 0.75.The score is task_score = safety × (0.8 × completion + 0.2 × robustness), so unsafe behavior overrides partial completion.Source: §C.1 · p. 21, §C.2.3 · p. 24

C.2.1 Navigation: Real-world Map Routing

  • Navigation data are built by sampling geographically coherent waypoint sets, generating natural-language route requests, and filtering out ambiguous or implausible query–route pairs.Waypoint sets contain 3–9 POIs sampled around a shared map anchor; the held-out test and training sets contain 150 and 600 instances, respectively. Evaluation exact-matches coordinates and applies order-aware waypoint matching, with up to six turns per episode.Source: §C.2.1 · pp. 22–23
  • Navigation requires multi-turn route planning over 3–9 waypoints, where ambiguous place names and exact coordinate-level matching make a single incorrect resolution yield zero reward.Agents must resolve all locations before calling direction; ambiguous POI results may require region-refined searches, and the evaluator gives no partial credit for incorrect coordinates.Source: §C.2.1 · p. 23

C.2.2 τ2-Bench (Telecom / Retail / Airline)

  • τ2-Bench evaluates multi-turn customer-service agents across telecom, retail, and airline domains, counting success when the final state satisfies the user’s request.Episodes allow up to 30 interaction turns, and Pass@k and Passk are reported separately for each domain.Source: §C.2.2 · p. 24

C.2.3 Claw-Eval-Hard20

  • Claw-Eval-Hard20 contains 20 human-verified, real-world tool-use tasks selected to be difficult, solvable, and reliably evaluable, directly testing the exploration bottleneck motivating RTI.Tasks span tool orchestration, failure recovery, cross-source synthesis, entity disambiguation, safety, document understanding, and numerical reasoning. The selection process excludes inaccessible evidence, unstable tasks, and unreliable graders; 17 tasks have zero successful base-policy rollouts across three attempts.Source: §C.2.3 · pp. 24–25

C.3 More details for Baselines

  • Baselines include untrained and larger reference models, on-policy RL methods, and teacher-guided methods including OPD and LUFFY.OPD weights updates using teacher scores on policy-generated trajectories; LUFFY substitutes successful teacher rollouts when an entire policy rollout group fails.Source: §C.3 · pp. 26–27

D More Details for Experimental Results

  • The appendix outlines analyses of transfer, a complete τ2-Telecom intervention, pipeline ablations, training-time statistics, GRPO-versus-RTI transcripts, and reasoning-shift reweighting.Section D.6 addresses transfer beyond training-data synthesis task families; Sections D.1–D.5 cover the intervention walkthrough, ablations, statistics, transcript comparison, and reweighting visualization.Source: D More Details for Experimental Results · p. 27

D.1 A Complete MAS Intervention on τ2-Telecom

  • MAS intervention converts a failed rollout into a valid, learnable trajectory by correcting a decisive local error while preserving the policy’s reasoning direction.The validator rejects a user-side device action; the corrector preserves the policy’s intent by turning it into a user-facing instruction, and the verifier checks for leakage or fabricated tool results.Source: §D.1 · pp. 27–28

D.2 Ablating Leakage Suppression

The leakage-suppression ablation contrasts a ground-truth-aware correction that is rejected for exposing hidden information with a clean redirection that repairs the decisive mistake using visible evidence. The comparison shows that task success alone does not make an intervened trajectory suitable for policy training.

D.2 Ablating Leakage Suppression

  • In the example, the policy makes an illegal user-side tool call, while the leakage-suppressed Corrector redirects the user to perform the action.The original response attempts toggle_data directly; the corrected reply asks the user to run it and then check network status.Source: §D.2 · pp. 28–29
  • A ground-truth-aware correction can satisfy the task outcome yet expose backend-only information, so the Verifier rejects it as a training signal.The leaked reply reveals the customer name, internal line identifier, and data-usage value, none of which was supported by the visible conversation.Source: §D.2 · p. 28
  • The clean redirection uses visible evidence, stays close to the policy’s intent, and turns the failed action into a valid continuation suitable for optimization.The Verifier rationale confirms that the user-side action is correctly assigned to the user and that no privileged information or fabricated tool result is included.Source: §D.2 · pp. 28–29

D.3 Training-Time Statistics of MAS Intervention

The section reports intermediate statistics for the MAS intervention pipeline on τ2-Telecom with Qwen3-8B. These statistics cover intervention and verification activity as well as resulting trajectory characteristics, with a high acceptance rate for intervened trajectories.

D.3 Training-Time Statistics of MAS Intervention

  • The reported MAS pipeline statistics show a high acceptance rate for intervened trajectories, indicating that corrections are often accepted while the Verifier screens for leakage and learnability.The table summarizes intermediate pipeline statistics on τ2-Telecom using Qwen3-8B.Source: §D.3 · p. 29, Table 6 · p. 30
  • The statistics track how often intervention is triggered and how many rollout steps the Validator flags as invalid.The passage describes these as intermediate statistics of the intervention pipeline.Source: §D.3 · p. 29, Table 6 · p. 30
  • The reported statistics also count corrections accepted or rejected by the Verifier and describe resulting trajectory statistics.No individual cell values are supplied in the passages.Source: §D.3 · p. 29, Table 6 · p. 30

D.4 Reasoning Paradigm Difference: RTI vs. GRPO on Navigation

RTI’s Navigation trajectory repairs an incorrect POI retrieval before route planning, unlike GRPO’s failed route; the section also visualizes decisive token edits and reports generalization to Claw-Eval Rest-OOD tasks.

D.4 Reasoning Paradigm Difference: RTI vs. GRPO on Navigation

  • On Navigation, GRPO proceeds with a wrong destination despite noticing the mismatch, while RTI refines the search before making the correct direction call.GRPO receives reward 0, whereas RTI's corrected direction call matches ground truth and receives reward 1.Source: §D.4 · p. 30

D.5 Visualization Evidence for Reasoning-Shift-Aware Token Reweighting

  • Reasoning-shift reweighting emphasizes local edits that change the action plan, correct role boundaries, or redirect the action target.The visualization highlights these tokens as consequential edits in the τ2-Telecom domain.Source: §D.5 · p. 31

D.6 Out-of-Distribution Generalization on Claw-Eval

  • RTI raises Hard20-ID Pass@1 from 26.25 (base) to 85.00, above GRPO's 52.50, and Rest-OOD Pass@1 from 54.34 to 62.92.RTI also raises Rest-OOD avg Passk from 43.62 to 54.48 and full-suite Pass@1 from 51.51 to 65.20. Rest-OOD is unscreened, and environment-limited failures cap attainable scores.Source: §D.6 · p. 31, Table 7 · p. 31
Loading 2610.13438v1…