PaperlayerView exact revision on arXiv

Source-linked AI summary

RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training

Yugu Li, Jimmy Cao, Jianglin Qiao, Siyi Hu

arXiv:2608.18682v1cs.AI

TL;DR

Multi-turn agentic RL training is unstable because flattened trajectory optimization creates context mismatch, weak turn-level credit assignment, and asynchronous policy drift. RTPO addresses these issues with reverse-order turn-level updates over sparse reverse trees and improves accuracy over GRPO by 21.50% and SeeUPO by 10.76% across eight benchmarks.

  • Problem

    Multi-turn agentic RL training remains unstable because flattened optimization creates context mismatch, weak turn-level credit assignment, and asynchronous policy drift.

  • Method

    RTPO organizes interactions as sparse reverse trees, optimizes turn-level policies in reverse temporal order, and regenerates downstream sibling continuations on-policy.

  • Results

    RTPO achieves the best performance across all eight benchmarks, outperforming GRPO by 21.50% and SeeUPO by 10.76% in overall accuracy.

  • Takeaways & Limitations

    RTPO provides a theoretically supported framework for improving multi-turn optimization stability in long-horizon tool-using agents.

  • Takeaways & Limitations

    Turn-level advantage estimates can be contaminated by downstream errors before downstream policies are optimized, while on-policy continuation requires sibling rollouts at each turn.

Abstract

from arXiv · show

Training multi-turn agentic workflows with reinforcement learning (RL) enables large language models to perform complex reasoning, use external tools, and conduct iterative search beyond single-turn settings. Yet multi-turn RL training remains highly unstable, often causing severe performance degradation as the number of turns increases. Through theoretical analysis, we identify three tightly coupled sources of instability: rollout-training context mismatch, weak turn-level credit assignment under sparse terminal rewards, and asynchronous policy drift when short and long trajectories are optimized under different policy versions. We show that these issues share a common structural origin in flattened trajectory optimization and address them through a unified reverse-turn formulation. We propose Reverse-Turn Policy Optimization (RTPO), which organizes multi-turn rollouts as sparse reverse trees and performs turn-level policy updates in temporal reverse order, aligning each decision with its downstream continuation. RTPO enables causally consistent turn-level credit assignment and on-policy continuation to control asynchronous drift. We provide theoretical guarantees showing that RTPO eliminates context mismatch and asynchronous drift under the proposed turn-level formulation, reduces credit bias, and converges to recursive optimality. Experiments on multi-turn agentic RL benchmarks show that RTPO improves upon trajectory- and turn-level baselines by 21.50% and 10.76%, respectively, highlighting its potential to support more stable training for tool-using agents.

1 Introduction

Multi-turn agentic RL becomes increasingly unstable as workflows lengthen because temporal dependencies amplify context mismatch, weak turn-level credit assignment, and asynchronous policy drift. RTPO addresses these coupled problems by optimizing sparse reverse-tree rollouts in temporal reverse order with turn-level on-policy updates and theoretical guarantees.

  • Instability: Multi-turn RL can improve short workflows yet degrade as turns increase, because each decision reshapes context, environment state, and future decision distributions.The instability is amplified by cross-turn temporal dependencies rather than sequence length alone.
  • Instability: The paper identifies three coupled instability sources: rollout–training context mismatch, weak turn-level credit assignment, and asynchronous policy drift across policy versions.Sparse terminal rewards obscure individual decision contributions, while short and long trajectories may be optimized under different evolving policies.
  • Core Challenge: These problems share a common structural origin in flattened trajectory optimization, motivating a unified turn-wise stabilization approach.The stated challenge is to restore rollout–training consistency, improve turn-level credit assignment, and control asynchronous policy drift.
  • RTPO: RTPO represents sampled interactions as sparse reverse trees and optimizes turn-level policies in temporal reverse order.Continuation values are propagated from later turns to earlier ones through a reverse optimality guarantee, with sibling continuations constructed for each turn.
  • RTPO: RTPO combines sparse reverse trees, turn-level on-policy updates, and theoretical guarantees on recursive optimality, context consistency, and controlled policy drift.The framework is presented as a policy optimization method for stabilizing multi-turn agentic RL training.

2 Theoretical Analysis: Training Instability

The analysis attributes multi-turn RL instability to three coupled failures of flattened trajectory optimization: rollout–training context mismatch, weak turn-level credit assignment, and long-horizon policy drift. These failures bias policy updates, obscure causal turn contributions, and can destabilize or collapse training as horizons lengthen.

  • Sources of instability: Flattened multi-turn RL suffers from context mismatch, trajectory-only credit assignment, and long-horizon policy drift.Figure 1 identifies these three sources of instability.
  • Rollout–training context mismatch: Truncated or summarized rollout contexts differ from full-history training contexts, biasing importance ratios and worsening mismatch in later turns.Non-injective truncation can also alias distinct full-history states and restrict optimization to a potentially inferior observation-induced policy class.
  • Turn-level credit assignment: A single trajectory-level advantage can disagree with the true turn-level advantage when upstream state effects or downstream stochasticity dominate, causing incorrect or reversed updates.Group-relative baselines may be determined by upstream trajectories without causal connection to the current action.
  • Long-horizon policy drift: Asynchronous optimization makes longer trajectories off-policy, while token-level clipping cannot faithfully correct drift and exact full-trajectory correction has rapidly growing variance.The resulting instability can cause standard PPO- or GRPO-style training to become unstable or collapse in long-horizon agentic RL.

3 Method: Reverse-Turn Policy Optimization (RTPO)

RTPO decomposes multi-turn interaction into turn-level sub-policies, preserves rollout-training contexts, and trains turns in reverse order with sparse-tree sibling rollouts. This design provides recursive-optimality guarantees, causally consistent credit assignment, and on-policy continuation that removes policy drift and reduces estimation error.

  • Turn-boundary MDP and context matching: RTPO models episodes as turn-boundary MDPs with independent sub-policies, using the identical recorded context c_k during rollout and training.The loss is computed only over turn-k output tokens, so importance-sampling denominators use the same conditioning context as rollout.
  • Reverse-order training: Reverse-order optimization freezes completed turn policies while downstream policies are already optimized, and uses sparse-tree sibling rollouts to estimate turn-level advantages.For turn k, RTPO generates G−1 sibling rollouts from the trunk boundary state and excludes the trunk trajectory from gradient updates.
  • Theoretical guarantees: Under stated assumptions, reverse-order turn optimization converges per turn to recursive optimality, which equals global trajectory optimality when macro-action policy classes are complete.On-policy continuation also dynamically reduces Monte Carlo value-estimation variance as success probabilities move away from one-half.
  • Causally consistent advantage estimation: RTPO’s turn-level advantages are locally unbiased up to O(1/G), remove upstream state contamination, and outperform trajectory advantages when cross-state value variance is non-zero.Shared-boundary-state siblings make return differences attributable to turn-k macro-actions, assigning advantages only to turn-k output tokens.
  • On-policy continuation: Sibling continuations are regenerated on-policy under the current downstream policy, making trajectory-level importance-sampling weights identically one and eliminating correction-induced variance explosion.The same current policy generates and evaluates each continuation, so terminal returns provide unbiased Monte Carlo estimates of the turn-level Q-value.

4 Experimental Results and Analysis

The experiments evaluate RTPO against trajectory-, turn-, and tree-level baselines across mathematical and knowledge-intensive agentic RL benchmarks. Analyses separately examine rollout–training consistency, turn-level credit assignment, and on-policy continuation.

  • Experimental Setting: Experiments use Qwen3-8B with thinking mode and compare RTPO against GRPO, ARPO, TreeGRPO, SeeUPO, and an untuned vanilla reference across mathematical and knowledge tasks.The benchmark suite spans easy, medium, and hard difficulty tiers and includes 12 experiments.
  • Rollout–Training Consistency: RTPO’s rollout–training consistency is assessed with geometric-mean token-logprob ratios and KL divergence, where ratios near 1 and smaller KL indicate closer distributional alignment.The comparison covers mathematical and knowledge reasoning tasks.
  • Rollout–Training Consistency: Baseline rollout–training ratios drift slowly toward 1, but this empirical alignment remains incomplete, task-dependent, noisy, and optimization-budget intensive compared with RTPO’s structural consistency.The passage relates this behavior to bootstrapped alignment mechanisms in DAgger and SCoRe.
  • Turn-Level Credit Assignment: Under full-history rollout inputs that control context mismatch, RTPO-CA achieves the highest overall accuracy, isolating the contribution of turn-level credit assignment to advantage estimation.Controlled-setting accuracies are substantially higher than those in Table 1 because full interaction histories are used.
  • On-Policy Continuation: On-policy continuation is most beneficial when downstream policies change substantially, particularly on longer-horizon GAIA, WebWalkerQA, and XBench tasks where stale-rollout importance-sampling correction can introduce clamp-truncation bias.HLE’s shorter, more closed-ended search horizons produce smaller gains.

5 Concluding Remarks · A Related Work · B Full Theoretical Analysis: Training Instability

The paper identifies rollout–training context mismatch, weak turn-level credit assignment, and asynchronous policy drift as coupled sources of instability in multi-turn agentic RL. RTPO addresses them through a unified reverse-turn framework, while the theoretical analysis formalizes how flattened trajectory optimization creates biased training signals.

  • 5 Concluding Remarks: RTPO integrates rollout–training consistency, turn-level credit assignment, and on-policy continuation within a unified training pipeline for unstable multi-turn agentic RL.The framework targets tool-augmented mathematical reasoning and deep-search tasks.
  • A Related Work: Existing agentic RL methods such as GRPO and GSPO optimize increasingly refined token- or sequence-level objectives for reinforcement learning with verifiable rewards.GRPO uses token-level importance ratios and group-relative advantage estimation, while GSPO provides sequence-level refinement.
  • A Related Work: SeeUPO models multi-turn interaction as sequential multi-agent bandits and updates turn-level virtual agents in reverse execution order, motivating RTPO’s reverse-turn design.SeeUPO is described as inheriting monotonic improvement and proving convergence to the globally optimal policy.
  • A Related Work: Tree-based methods improve credit granularity through shared-prefix rollouts, but reconstructed trees can still propagate trajectory-level credit mismatch to individual turns.ARPO branches adaptively at high-entropy nodes after tool calls and estimates branch advantages independently.
  • A Related Work: Shared-prefix gradients remain biased because descendant outcomes perturb common tokens with trajectory-level signals, while branch averaging can dilute advantages and learned critics introduce bias.These limitations are reported for Tree-GRPO, SEEA-R1, TreeAdv, ARPO, and ATPO under rollout–training contextual mismatch.
  • B Full Theoretical Analysis: Training Instability: In the theoretical analysis, standard flattened optimization re-evaluates tokens under concatenated full-history contexts although rollouts use truncated or summarized contexts, producing biased importance-sampling ratios.This rollout–training context mismatch undermines stable policy optimization.
  • B Full Theoretical Analysis: Training Instability: The analysis formulates multi-turn tool interaction as a hierarchical Markov decision process separating turn-level planning from token-level execution, unlike methods that flatten all interaction into one token sequence.Turn states include interaction history, while token states combine turn history with the generated token prefix.

B.1 Single flattened-trajectory policy optimization: mismatch from rollout to training … C.1 Notation, Formal Problem Setup, and Technical Assumptions

The paper identifies three instability mechanisms in flattened multi-turn RL—rollout-training mismatch, unreliable turn-level credit, and asynchronous policy drift—and formalizes RTPO’s turn-boundary setting for matched-context optimization and reverse-order convergence analysis.

  • B.1 Single flattened-trajectory policy optimization: mismatch from rollout to training: Long multi-turn histories force rollouts to truncate or summarize context, while flat training can reevaluate tokens under the full history, creating a rollout-training mismatch.The mismatch produces a biased importance-sampling ratio and increasingly distorts later-turn updates as omitted history grows.
  • B.1 Single flattened-trajectory policy optimization: mismatch from rollout to training: Truncated observations can alias distinct full-history states, restricting policies to an observation-induced class whose projected optimum may be strictly below the true optimum.This loss arises because the truncated process may not preserve the original Markov property.
  • B.2 Trajectory-only credit assignment: mismatch across low- and high-quality turns: Trajectory-level advantages assign a shared return signal across turns, so downstream stochasticity and upstream state variation can reverse the sign of the true turn-level advantage.The problem is especially severe when trajectories reach semantically different states before the evaluated turn.
  • B.2 Trajectory-only credit assignment: mismatch across low- and high-quality turns: Cross-state group baselines mix returns from different turn-level states, introducing upstream bias unrelated to the action at the current turn.Same-state grouping instead anchors the baseline to the local context and differs from the true turn advantage only by (G −1)/G.
  • B.3 Long-horizon policy drift: PPO clipping mismatch asynchronous turns: Asynchronous completion lets short trajectories update the policy while longer trajectories continue rolling out under older policy versions, producing turn-wise policy drift that token clipping does not explicitly resolve.Per-token clipped ratios cannot reproduce the full-trajectory correction: their product is constrained to [(1−ϵ)^T, (1+ϵ)^T], while exact trajectory weighting has variance that grows exponentially with T.
  • C.1 Notation, Formal Problem Setup, and Technical Assumptions: RTPO’s theoretical framework models episodes as K-turn boundary MDPs with complete responses as high-level actions and tool feedback defining transitions.The framework separates turn-level decisions from token generation and represents each augmented state as the interaction state together with its turn index.
  • C.1 Notation, Formal Problem Setup, and Technical Assumptions: Using the same compressed or truncated conditioning context during rollout and training restores matched token-level importance sampling, while turn values decompose into local actions plus downstream continuation value.In sparse-reward settings, downstream continuation value often carries most of a nonterminal turn’s quality signal.
  • C Method: RTPO Theoretical Proofs: The convergence analysis establishes recursive optimality in a tabular finite-horizon setting under standard stochastic-approximation conditions and reverse-order updates with downstream policies frozen.These assumptions are intentionally stronger than those used by the practical shared-neural-network implementation.

C.2 Proof of Theorem 1: Convergence to Recursive Optimality

Under assumptions including finite spaces, Robbins–Monro step sizes, sufficient exploration, frozen downstream policies, and bounded rewards, reverse-order turn-level optimization converges per turn and recursively from the final turn backward. With context-sufficient macro-action completeness, recursive optimality is equivalent to global trajectory optimality, and the guarantee extends asymptotically to neural function approximation under a realizability assumption.

  • Theorem 1: Under Assumptions (A1)–(A10), reverse-order backward induction provides the conditions for per-turn convergence and recursive optimality.The assumptions include finite state and macro-action spaces, Robbins–Monro step sizes, sufficient exploration, exact freezing of optimized downstream policies, and bounded rewards.
  • Per-turn convergence: With optimized downstream policies fixed, each turn’s target is stationary, so tabular stochastic approximation converges to the corresponding augmented turn-level action value.The proof uses stationary conditional targets, bounded second moments, infinite visitation of relevant state–macro-action pairs, and Robbins–Monro updates.
  • Recursive optimality: Reverse-order training applies per-turn convergence from turn K−1 backward, yielding every turn-level policy’s optimal selector given optimized and frozen downstream policies.The induction begins at the final turn and proceeds backward, freezing converged downstream policies before optimizing each preceding turn.
  • Global optimality: Under context-sufficient macro-action completeness, recursive optimality is equivalent to global optimality over the full trajectory.Every globally feasible trajectory-level policy must be representable by turn-level policies acting on contexts c_k = ψ(S_k).
  • Neural function approximation: Under neural function approximation and the replacement realizability assumption (A4’), reverse-order RTPO asymptotically realizes the recursively optimal policy on every reachable state.The argument reduces updates to projected Q-learning, shows vanishing projection error on the visitation support, and preserves the reverse induction structure.

C.3 Proof of Theorem 2: Causally Consistent Turn-Level Advantage Estimation

Theorem 2 shows that RTPO’s state-matched sibling estimator provides locally unbiased turn-level advantages up to O(1/G) finite-group bias, avoids upstream state contamination, and achieves lower value-estimation error than flat-trajectory estimation. Because siblings share a boundary state, their credit signals reflect turn-k actions and exclude prefix-token gradients.

  • Local unbiasedness up to finite-group bias: RTPO’s sibling estimator has only O(1/G) finite-group bias and no upstream state-contamination term, unlike the trajectory-level estimator.Its conditional bias is −1/(G−1) times the true turn-level advantage.
  • Reduced value estimation error: When σ2_V > 0, state-matched sibling baselines have strictly lower MSE because they remove the cross-state value component present in trajectory-level baselines.The trajectory estimator incurs an additional MSE component proportional to σ2_V, whereas the sibling estimator removes it under matched downstream-noise conditions.
  • State-matched causal actions: Shared boundary states make return differences attributable to sampled turn-k macro-actions and downstream noise rather than upstream trajectory differences.Conditioned on S_k, the expected return difference equals the difference between the corresponding action values.
  • State-matched causal actions: RTPO assigns the advantage only to sampled turn-k output tokens, while prefix tokens in the shared context receive zero gradient.The gradient support includes only turn-k action tokens and excludes the shared prefix.
  • Interpretation: The proof attributes RTPO’s advantage to comparing alternative turn-k actions from the same boundary state, rather than centering returns around a global mean from different states.Flat-trajectory baselines introduce the upstream term Vπ(S_i,k) − µ_R.

C.4 Proof of Theorem 3: On-Policy Continuation under Asynchronous Turns

Theorem 3 shows that RTPO prevents asynchronous off-policy drift by regenerating sibling continuations under synchronized current downstream policies. This makes turn-level value estimates unbiased, removes trajectory-level importance-sampling correction, and improves advantage-estimation reliability as downstream policies become more confident.

  • Drift-free on-policy continuation: RTPO synchronizes θ>k, forks sibling rollouts from Sk, and continues them to termination under the same policy πθ>k.This reverse-order procedure establishes the theorem’s on-policy continuation condition.
  • Drift-free on-policy continuation: Each sibling terminal return is an unbiased Monte Carlo estimate of the current-policy turn-level Q-value.The downstream continuation is sampled from the same current policy after the turn-k macro-action.
  • Drift-free on-policy continuation: The trajectory-level importance-sampling weight equals 1, so RTPO requires no trajectory-level IS correction.Sampling and evaluation use the identical policy throughout each sibling continuation.
  • Dynamic error reduction in advantage estimation: As downstream success probabilities move away from 1/2, Q-value variance decreases and the centered turn-level advantage estimator’s signal-to-noise ratio improves.For normalized rewards, the analogous result is a reduction in the worst-case bounded-variance control.
  • Comparison with stale trajectory-level IS correction: Stale trajectory-level IS correction can compound small per-token policy mismatches into high-variance products over long continuations.Per-token PPO clipping does not eliminate this trajectory-level mismatch because clipping and multiplication do not commute.

D Experimental Setup … E.2 Source Code

The experiments evaluate RTPO on multi-turn mathematical and web-search reasoning with Qwen3-based implementations, standardized infrastructure, and repeated stability analyses. The implementation uses reverse-order, on-policy turn updates and provides pseudocode and source-code support for these components.

  • D Experimental Setup: Qwen3 models provide the backbone for RTPO and all baselines, with up to 32,768-token contexts and thinking mode enabled for multi-turn experiments.This setup accommodates long interaction histories and complete tool-integrated reasoning trajectories.
  • D Experimental Setup: RTPO is compared against trajectory-only methods GRPO and turn-level or tree-based methods ARPO, Tree-GRPO, and SeeUPO.The baselines differ in token-level, tree-level, or sequential per-turn advantage estimation and updates.
  • D Experimental Setup: Evaluation covers mathematical reasoning with MATH and web-search knowledge reasoning using 1,000 hard-search samples from SimpleDeepSearcher and WebSailor.Both tasks require external-tool interaction, multi-turn reasoning, and adaptation to intermediate feedback; the search task additionally stresses retrieval and evidence integration.
  • E Implementation Details: Experiments use VeRL, vLLM, and FSDP on 8× NVIDIA A100 GPUs, with batch size 64, 4096-token single-turn responses, and horizons of K = 3 or K = 6 turns.The horizons correspond to mathematical reasoning and web search, respectively.
  • D Experimental Setup: Pass@1 measures final policy quality, tool calls measure behavior under fixed compute, and log-probability ratio plus KL divergence assess rollout–training consistency.Controlled experiments also examine turn-level credit assignment and on-policy continuation; each experiment is repeated three times and averaged.
  • E.1 Pseudocode: RTPO first generates a complete trunk trajectory, records boundary states and environment snapshots, then trains turns in reverse order while synchronizing the current policy before on-policy sibling generation.The trunk supplies anchors without receiving gradients, while downstream optimized parameters are inherited by earlier-turn updates.
  • E.1 Pseudocode: Each reverse-turn step computes sibling-based advantages and applies a clipped PPO update only to the current turn’s tokens, with the completed turn frozen before proceeding upstream.Shared states make the turn-level advantage free of upstream state contamination, and each turn uses importance ratios against the synchronized old policy.
  • E.2 Source Code: The supplementary repository provides installation instructions, versioned dependencies, and VERL-based additions for round summaries, on-policy tree rollouts, per-turn advantages, and reverse updates.These additions let later rounds roll out without access to the complete interaction history.

E.3 Training Details · E.4 Qwen3 Chat Template · F Additional Results and Insights

The paper specifies shared rollout budgets, optimizer settings, and hardware conditions for its experiments, while Qwen3’s chat template filters internal reasoning traces before constructing the next-turn input. These details define the computational setup and interaction-context processing used across the reported experiments.

  • E.3 Training Details: 80k rollouts for mathematical reasoning and 16k for knowledge reasoning define the global sampling budgets used for computational fairness.The mathematical budget is 5k prompts × 16 rollouts per prompt, while the knowledge budget is 1k prompts × 16 rollouts per prompt.
  • E.3 Training Details: Qwen3 experiments use sampling temperature 0.9, AdamW, and learning rate 1 × 10−6.
  • E.3 Training Details: For GRPO and SeeUPO, training samples 16 chains per prompt, with SeeUPO updating sequentially by turn in reverse execution order.
  • E.3 Training Details: RTPO enforces the same global sampling budgets as baselines, limiting aggregate rollout compute to 80k for mathematical reasoning and 16k for knowledge reasoning.Per-prompt rollout counts can vary with early termination, but the global constraint keeps total generated rollouts within the baseline budget.
  • E.3 Training Details: Experiments run on a single node with 8× NVIDIA A100 80GB GPUs, 256 CPU cores, and 256GB of system memory.Training requires approximately 15 hours for the math task and approximately 5 hours for the knowledge task.
  • E.4 Qwen3 Chat Template: At each turn, Qwen3 generates an internal reasoning trace, textual commentary, and a tool-call instruction whose execution returns an environmental observation.The three components are enclosed using <think>...</think> and <tool_call>...</tool_call> markers as applicable.
  • E.4 Qwen3 Chat Template: Before the next-turn input is constructed, the chat-template filter removes the <think>...</think> reasoning segment while retaining textual commentary.

F.1 Discussion of Main Results

RTPO improves multi-turn tool-use performance by learning task-dependent calling strategies: fewer calls with stronger internal reasoning for mathematics and denser calls for knowledge reasoning. Its gains are especially clear on hard, long-horizon tasks, where it reduces context-mismatch bias and supports more stable optimization.

  • Mathematical reasoning: 33.33% Pass@1 on AIME24 lets RTPO surpass SeeUPO (20.00%) and GRPO (10.00%) while using fewer tool calls.RTPO solves ten AIME24 problems with four Python calls, whereas Vanilla makes 17 calls and solves one.
  • Failure modes: RTPO can still fail when rollouts terminate before final answer emission, even after obtaining a correct intermediate result.Examples include computing the correct mathematical average 64.0 but producing an empty extracted answer, and stopping after identifying a film director without completing the birthplace lookup.
  • Knowledge reasoning: 64.89 F1 on HotpotQA lets RTPO exceed SeeUPO (54.27) and GRPO (53.39), with a clear lead also on 2Wiki.Knowledge-intensive tasks require longer interaction horizons and more frequent reasoning revision, amplifying rollout–training context mismatch as turns accumulate.
  • Task-dependent tool use: RTPO’s tool-use behavior differs by task category rather than uniformly encouraging or suppressing calls.It uses fewer calls with more reliable internal reasoning on mathematics, but denser calls and more thorough external information integration for knowledge reasoning.

F.2 Insights from Rollout–Training Consistency Analysis · F.3 Additional Findings for Hard-Search Scenarios

Baselines can gradually approach rollout–training consistency, but alignment remains incomplete, oscillatory, and costly on long-horizon tasks, whereas RTPO removes this gap structurally. Additional experiments isolate on-policy continuation by comparing standard RTPO with an otherwise matched off-policy variant across hard-search benchmarks.

  • F.2 Insights from Rollout–Training Consistency Analysis: Baselines’ rollout–training ratios slowly drift toward 1 despite differing rollout and training contexts, consistent with training-data distribution adaptation and learnable summary states.Prior work suggests decision-preserving summaries can support convergence toward equivalent optimal policies and compact decision states.
  • F.2 Insights from Rollout–Training Consistency Analysis: RTPO’s structural consistency differs fundamentally from baselines’ empirical alignment.The distinction is developed through incompleteness, oscillation, and optimization-budget costs.
  • F.2 Insights from Rollout–Training Consistency Analysis: 0.97 is GRPO’s recovery on math after 30 steps, versus 0.83 on search after 14 steps, with alignment becoming harder as horizons and accumulated contexts grow.The longer-horizon difficulty is also reflected in the knowledge-reasoning results in Table 1.
  • F.2 Insights from Rollout–Training Consistency Analysis: 1.02 is SeeUPO’s math ratio spike at step 9, followed by sustained fluctuations from advantage-estimate variance, while RTPO is affected only by training–inference engine differences.RTPO’s ratio is structurally guaranteed rather than bootstrapped through an advantage-driven loop.
  • F.2 Insights from Rollout–Training Consistency Analysis: Baselines spend part of their optimization budget compensating for summary-versus-full-history gaps, whereas RTPO directs that budget toward the task objective.This hidden cost is consistent with baselines making fewer search calls on long-horizon knowledge reasoning.
  • F.2 Insights from Rollout–Training Consistency Analysis: Baselines do not necessarily fail, but their compensatory training dynamics become substantially less effective as task horizons lengthen, unlike RTPO’s construction-based consistency.The ratio therefore exposes a gap that baselines must learn to compensate for and RTPO does not have by construction.
  • F.3 Additional Findings for Hard-Search Scenarios: The on-policy ablation uses Qwen3-8B and preserves reverse-order turn-level training and state-matched sibling structure, changing only how downstream sibling continuations are obtained.Standard RTPO synchronizes latest parameters θ>k to the inference engine at turn k and regenerates continuations under πθ>k.
  • F.3 Additional Findings for Hard-Search Scenarios: The off-policy variant reuses continuations generated under πθ0 during initial rollout and corrects staleness with a clamped trajectory-level importance weight.The comparison evaluates output-hit accuracy across GAIA, WebWalkerQA, XBench, and HLE subject subsets.

F.4 Limitations and Future Work · F.5 Broader Impacts · G Supplementary Theoretical Clarifications, Implementation Details, and Extended Experiments

RTPO’s design has limitations involving trunk quality, computational overhead, and training-token utilization. Its potential broader impacts include more stable, analyzable, and reliable agentic RL, alongside increased autonomous capabilities.

  • F.4 Limitations and Future Work: RTPO depends on a single trunk trajectory, so poor early decisions can place later boundary states in low-value regions and weaken turn-level learning signals.Near-zero turn-level advantages can make corresponding gradient updates ineffective.
  • F.4 Limitations and Future Work: Multi-trunk sampling or retroactive trunk selection could improve trunk diversity and enable earlier removal of low-quality paths.The paper contrasts RTPO’s single trunk anchor with beam search and best-of-N methods that maintain multiple candidate prefixes.
  • F.4 Limitations and Future Work: RTPO incurs additional computational cost because each of its K reverse phases requires sibling generation, environment restoration, on-policy continuation, and a PPO-style update.The overhead scales with the number of turns K and sibling rollouts.
  • F.4 Limitations and Future Work: RTPO under-utilizes training tokens by assigning advantages only to sibling turn-k output tokens while excluding prefixes, downstream continuations, and trunk tokens from gradient updates.Sibling downstream tokens are used for terminal-return estimation but discarded during policy optimization.
  • F.5 Broader Impacts: More stable agentic RL could reduce failed tool-use trajectories, improve sample efficiency, and support more reliable LLM-agent deployment in research, education, software engineering, and decision support.These are identified as potential positive impacts of RTPO.
  • F.5 Broader Impacts: Separating turn-level decisions from full-trajectory outcomes may make multi-turn RL easier to analyze and improve monitoring of planning, search, and tool use.Turn-level monitoring can deepen understanding of how agentic workflows learn across multiple turns.
  • F.5 Broader Impacts: Stronger turn-refined agentic workflows may increase LLM agents’ capability to act autonomously across broader settings.The passage presents this as a potential accompanying risk of RTPO’s broader impacts.

G.1 Stateful Tool-Agent Evaluation on τ 3-Airline · G.2 Sensitivity to the Sibling Group Size · G.3 Trunk Quality and Failure Dynamics

Across stateful τ 3-Airline evaluation, RTPO improves held-out success, tool-execution quality, and response efficiency, while sibling-group size trades training cost against signal density and inference behavior. Training analyses show that failures increasingly become shorter incorrect terminations rather than prolonged stalls, without heuristic trunk filtering.

  • G.1 Stateful Tool-Agent Evaluation on τ 3-Airline: RTPO evaluates state-changing airline-database interactions across 30 training tasks and 20 held-out test tasks.Earlier tool actions alter the state observed in later turns.
  • G.1 Stateful Tool-Agent Evaluation on τ 3-Airline: 56.25% is RTPO’s peak online success rate at Step 40, while success falls to 40.63% at Step 50.The late decline suggests degradation from continued optimization on a small training set with sparse rewards.
  • G.1 Stateful Tool-Agent Evaluation on τ 3-Airline: RTPO achieves the highest held-out Pass@1, Pass@4, and normal environment-termination rate.Against Tree-GRPO, generation truncation falls from 47.50% to 42.50% and average response length from 2,538.2 to 1,688.8 tokens; against REFUEL, Pass@1 rises from 11.25% to 17.50% and Pass@4 from 20.00% to 30.00%.
  • G.1 Stateful Tool-Agent Evaluation on τ 3-Airline: RTPO reduces tool-error calls from 50 to 9 and raises execution success rate from 57.63% to 89.16%.It also uses approximately 9,650.5 response tokens per successful trajectory, 38.2% below Tree-GRPO and 56.0% below GRPO.
  • G.2 Sensitivity to the Sibling Group Size: The sibling-group estimator has finite-group bias O(1/G), while larger G reduces Monte Carlo baseline noise without upstream-state contamination.All sibling continuations begin from the same turn-boundary state.
  • G.2 Sensitivity to the Sibling Group Size: Increasing G from 3 to 4 lowers the zero-advantage rate from 89.61% to 82.66%, raises Pass@1 from 17.50% to 20.00%, and cuts inference tokens from 9,650.5 to 7,477.9.The larger group supplies denser relative signals but requires more offline exploration.
  • G.2 Sensitivity to the Sibling Group Size: Moving from G = 3 to G = 4 increases GPU-hours by approximately 30.6% and training-generation tokens by approximately 152.8%, while improving Pass@1 by 2.5 percentage points and leaving Pass@4 unchanged.G = 3 is the stronger default cost–performance trade-off, whereas G = 4 favors Pass@1 and concise inference.
  • G.3 Trunk Quality and Failure Dynamics: Only 6.88% of trunks fail within the first turn, while trajectories without terminal signals fall from 37.50% at Step 10 to 0% at Step 50 and failed-trajectory length from 13.30 to 2.84 turns.Later failures increasingly appear as fast but incorrect termination; RTPO samples all trunks on-policy without heuristic filtering and uses same-state sibling returns for relative signals.

G.4 Selective-Gradient Optimization and Training Efficiency … G.9 Additional Limitations

The supplementary sections explain RTPO’s selective-gradient and efficiency trade-offs, tool-use behavior, synchronization effects, and training-budget dynamics, then distinguish it from related methods and identify theoretical, computational, reward-sparsity, optimization, and evaluation limitations.

  • G.4 Selective-Gradient Optimization and Training Efficiency: RTPO applies policy loss only to current-turn sibling output tokens, excluding prefixes and downstream continuations to prevent trajectory-level credit contamination.The trunk and prefix define the reached state, while downstream continuation supplies the return for evaluating the current action.
  • G.4 Selective-Gradient Optimization and Training Efficiency: 41.3% higher wall-clock time and GPU-hours and 40.0% more training-generation tokens reflect sibling-continuation sampling, reverse updates, and policy synchronization.Despite higher training cost, RTPO reduces successful-trajectory response tokens by 38.2% versus Tree-GRPO and approximately 56.0% versus GRPO.
  • G.4 Selective-Gradient Optimization and Training Efficiency: With approximately 2% more GPU-hours, RTPO matches GRPO on Pass@1, raises Pass@4 from 25.00% to 35.00%, and increases environment completion from 20.00% to 30.00%.The compute-matched comparison also reduces truncation, total tool calls, and average response length while improving tool-execution success.
  • G.6 On-Policy Continuation and Synchronization Frequency: Per-stage on-policy continuation consistently improves long-horizon deep-search performance, with the largest gain on GAIA Level 3, while remaining broadly comparable to off-policy RTPO on HLE.The results attribute stale-continuation costs mainly to long-horizon settings; fixed-interval and policy-KL synchronization lie between the two endpoints.
  • G.7 Budget Consumption Across Training: From Step 10 to Step 50, tool calls fall from 7.34 to 0.94 per trunk, generated tokens from 8,013.5 to 1,669.3, and tool-execution success rises from 74.04% to 93.33%.Every stage follows the predefined sampling and update schedule, while per-trunk cost decreases as the policy becomes more efficient.
  • G.8 Further Distinctions from Related Methods: RTPO differs from SeeUPO, ArCHer, REFUEL, R3, and SRL in its theoretical target and mechanism: outcome-supervised, on-policy, critic-free turn-level advantages from sibling continuations at shared boundary states.The cited comparisons distinguish RTPO’s flattened-pipeline inconsistencies from SeeUPO’s multi-agent bandits, ArCHer’s actor–critic hierarchy, REFUEL’s covariate shift, and expert-trajectory methods.
  • G.9 Additional Limitations: RTPO’s limitations include no global neural convergence guarantee, 41.3% higher training cost, over 80% zero-advantage rates, a fall from 56.25% to 40.63% online success, and limited experimental breadth.The experiments average three runs and use Qwen3-4B/Qwen3-8B with K = 3 or K = 6; G = 4 raises training-generation tokens by approximately 152.8%.
Loading 2608.18682v1…