RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training

• Multi-turn RL can degrade as the number of workflow turns increases because temporal dependencies reshape contexts, environment states, and future decision distributions.Existing trajectory-, turn-, and tree-based methods only partially address this instability and do not jointly resolve its shared training-pipeline origin.Source: §1 · p. 2
• The analysis identifies three coupled instability sources: rollout–training context mismatch, weak turn-level credit assignment under sparse rewards, and asynchronous policy drift across trajectory lengths.These issues share a structural origin in flattened trajectory optimization.Source: §1 · p. 2
• RTPO organizes interactions as sparse reverse trees and optimizes turn-level policies in temporal reverse order, propagating continuation values from later to earlier decisions.Sibling continuations provide matched downstream conditions, while on-policy continuation controls asynchronous drift and improves causal turn-level credit assignment.Source: §1 · p. 2
• RTPO provides theoretical guarantees for recursive optimality, context consistency, and reduced credit bias, and outperforms strong baselines including GRPO, TreeGRPO, ARPO, and SeeUPO on multi-turn agentic RL benchmarks.The framework is designed to stabilize the training pipeline while improving multi-turn performance.Source: §1 · p. 2

2 Theoretical Analysis: Training Instability

• Multi-turn RL instability arises from context mismatch, weak turn-level credit assignment, and asynchronous policy drift in flattened trajectory optimization.Rollouts may use truncated or summarized contexts while training recomputes likelihoods on full histories; terminal rewards are assigned at the trajectory level; and long trajectories can be optimized after policy updates from shorter ones.Source: §2 · p. 2, Figure 1 · p. 3

2.1 Rollout–Training Mismatch

• Later-turn rollout–training mismatch biases policy gradients and can restrict optimization to an observation-induced policy class with a lower optimum.Training evaluates probabilities under full histories although rollouts use truncated or summarized contexts; omitted history grows in later turns, and non-injective truncation causes state aliasing.Source: §2.1 · p. 3

2.2 Trajectory-Only Credit Assignment

• Trajectory-level advantages can reverse the correct turn-level update, while cross-state grouping adds bias from upstream states unrelated to the current action.The trajectory signal contains the true turn-level advantage plus upstream state effects and downstream stochasticity; group-relative baselines are valid only when trajectories share the same turn-level state.Source: §2.2 · pp. 3–4

2.3 Long-Horizon Policy Drift

• Token-level clipping cannot faithfully correct long-horizon policy drift, whose discrepancy compounds with trajectory length and can make PPO- or GRPO-style training unstable or collapse.Asynchronous updates make longer trajectories off-policy, while exact full-trajectory importance ratios are impractical because their variance grows rapidly with trajectory length.Source: §2.3 · p. 4

3 Method: Reverse-Turn Policy Optimization (RTPO)

• RTPO combines turn-boundary policy decomposition, reverse-order optimization, sparse-tree credit assignment, and on-policy continuation to stabilize multi-turn agentic RL training.The method targets rollout-training mismatch, weak turn-level credit assignment, and asynchronous policy drift through three coordinated components.Source: §3 · p. 4, Figure 2 · p. 5

3.1 Turn-Level Policy Optimization

• Reverse-order optimization keeps rollout and training contexts identical, freezes optimized downstream policies, and guarantees convergence from per-turn policies to recursive, and conditionally global, optimality.The global-optimality conclusion requires complete turn-level macro-action spaces and a policy class capable of representing every feasible trajectory-level policy.Source: §3.1 · pp. 4–6

3.2 Turn-Level Credit Assignment

• State-matched sparse-tree siblings provide causally consistent turn-level advantages, with finite-group bias O(1/G) and lower error than trajectory advantages when cross-state value variance is nonzero.Advantages are assigned only to the sampled turn’s output tokens, while shared prefixes are excluded from gradient updates.Source: §3.2 · pp. 6–7

3.3 On-Policy Continuation

• On-policy continuation regenerates siblings with optimized downstream policies, making terminal returns unbiased Q-value estimates with unit importance weights and no trajectory-level correction or multiplicative variance explosion.As downstream policies improve, continuation estimates align with the current policy and can reduce Monte Carlo variance when success probabilities move away from one-half.Source: §3.3 · pp. 7–8

4 Experimental Results and Analysis

• Experiments use Qwen3-8B with thinking mode across 12 mathematical and knowledge reasoning settings, comparing RTPO against trajectory- and turn/tree-level baselines.Rollouts use external calculation and web-search tools, with VeRL, vLLM, and FSDP supporting implementation and distributed training.Source: §4 · p. 8

4.1 Main Results

• 66.78% higher overall accuracy than vanilla, RTPO outperforms GRPO by 21.50% and SeeUPO by 10.76% across all eight benchmarks.Table 1 compares mathematical and knowledge-intensive question answering across easy, medium, and hard difficulty tiers.Source: §4.1 · p. 8, Table 1 · p. 8

4.2 Rollout–Training Consistency Analysis

• RTPO maintains a 1.0 log-probability ratio and the lowest average KL divergence, indicating stronger rollout–training consistency than GRPO and SeeUPO.On mathematical tasks, GRPO recovers from 0.89 to 0.97, while on knowledge tasks baselines recover only from approximately 0.80 to 0.89 and 0.83.Source: §4.2 · p. 9, Figure 3 · p. 9

4.3 Effect of Turn-Level Credit Assignment

• Under matched full-history rollouts, RTPO-CA achieves the highest overall average across four mathematical benchmarks and leads on AMC23, AIME25, and MATH500.It matches the best Pass@4 on AIME24; sibling rollouts from the same boundary state provide a local baseline for current-turn advantages.Source: §4.3 · p. 9, Table 2 · p. 9

4.4 Policy Drift Correction

• +7.00% on XBench, +5.83% on GAIA, and +3.50% on WebWalkerQA, default RTPO outperforms off-policy RTPO, while HLE is nearly tied at −0.33%.The gains are larger on longer retrieval and interaction horizons, where stale-rollout importance-sampling correction can suffer clamp-truncation bias.Source: §4.4 · pp. 9–10, Table 3 · p. 10

5 Concluding Remarks

• RTPO integrates rollout–training consistency, turn-level credit assignment, and on-policy continuation to improve optimization stability for long-horizon tool-using agents.The paper presents theoretical guarantees and empirical results supporting RTPO as a framework for multi-turn agentic RL.Source: §5 · p. 10

A Related Work

• Existing multi-turn adaptations of GRPO and GSPO flatten interactions, uniformly assign trajectory rewards, and recompute ratios under full histories despite truncated or summarized rollout contexts.This conditioning discrepancy weakens PPO-style stability assumptions and can lead to divergence or policy collapse in long-horizon tasks.Source: A Related Work · p. 16
• SeeUPO [Hu et al., 2026] updates turn-level agents in reverse order and proves global optimality, but retains full-history training contexts and multiplicative importance-sampling variance with clipping bias.These limitations motivate RTPO’s reverse-order updates under the turn-level conditioning context observed during rollout.Source: A Related Work · p. 16
• Tree-based methods such as ARPO [Dong et al., 2026], TreeGRPO [Ji et al., 2026], SEEA-R1 [Tian et al., 2026], TreeRPO [Yang et al., 2025b], TreeAdv [Cao et al., 2026a], and ATPO [Cao et al., 2026b] refine credit granularity but retain residual trajectory-level bias.Shared-prefix signals can be perturbed, diluted, attenuated near roots, or affected by critic bias under rollout–training contextual mismatch.Source: A Related Work · pp. 16–17
• SimpleTIR [Xue et al., 2026] filters void-turn trajectories and RAGEN [Wang et al., 2025b] applies numerical mismatch correction, whereas RTPO restructures training around turn-boundary MDPs without dropping trajectories or post-hoc correction.SimpleTIR is tied to code-execution reasoning and discards training data, while both approaches overlook contextual mismatch between rollout conditioning and training recomputation.Source: A Related Work · p. 17

B Full Theoretical Analysis: Training Instability

• The theoretical analysis identifies three instability sources: biased importance ratios from flattened context mismatch, inadequate action comparison from trajectory-only advantages, and asynchronous off-policy drift in long-horizon tasks.Later-turn rollout states depend on earlier actions and tool feedback, so policy drift can make them inconsistent with states induced by the current policy.Source: B Full Theoretical Analysis: Training… · p. 17
• The H-MDP formulation separates turn-level response planning from token-level execution, while standard PPO-family methods flatten both into one token sequence with a trajectory-level advantage shared across unmasked tokens.Each turn response is the high-level action, token generation is low-level execution with zero intermediate reward, and the flattened surrogate uses context-dependent importance ratios.Source: B Full Theoretical Analysis: Training… · pp. 17–18

B.1 Single flattened-trajectory policy optimization: mismatch from rollout to training

• Flattened training re-evaluates rollout tokens under concatenated histories rather than the truncated or summarized contexts that generated them, biasing importance ratios and gradients.The discrepancy worsens in later turns as omitted rollout history grows, potentially causing large token-probability errors and unstable updates.Source: §B.1 · p. 18

B.2 Trajectory-only credit assignment: mismatch across low- and high-quality turns

• Trajectory-level advantages can reverse turn-level credit because upstream state variation and downstream stochasticity may dominate the current turn’s true advantage.A cross-trajectory baseline is locally valid only when compared rollouts share the same turn-level state; otherwise, state-value variation introduces causal bias unrelated to the current action.Source: §B.2 · pp. 19–20

B.3 Long-horizon policy drift: PPO clipping mismatch asynchronous turns

• Per-token PPO clipping cannot reproduce the full-trajectory importance correction under asynchronous long-horizon updates, and exact trajectory weighting has variance that grows exponentially with length.Short trajectories may update the policy while longer trajectories are still generated under an older version, creating turn-wise drift that accumulates over time.Source: §B.3 · p. 20

C.1 Notation, Formal Problem Setup, and Technical Assumptions

• The formal model represents each episode as K turn-boundary macro-actions, with complete responses as actions, tool feedback as transitions, and sparse rewards typically arriving only at the terminal turn.Variable response lengths make the process equivalently a finite-horizon semi-MDP, while optimization and credit assignment remain defined at turn boundaries.Source: §C.1 · p. 21
• Rollout–training consistency requires reusing the exact conditioning context c_k=ψ(S_k) recorded during rollout, whether it is full, truncated, or summarized.Using the same context makes turn-level token importance ratios conditionally matched rather than recomputed under a different history representation.Source: §C.1 · p. 21
• Recursive optimality is defined by optimizing the final-turn sub-policy first, then optimizing earlier turns while downstream policies remain fixed.The convergence claim is established only in a tabular finite-horizon setting under stated stochastic-approximation conditions.Source: §C.1 · p. 22
• The theoretical assumptions include finite turn-boundary spaces and horizons, sufficient exploration and state coverage, valid step sizes, bounded rewards and iterates, and decision-sufficient conditioning contexts.These assumptions isolate reverse-order tabular dynamics and are stronger than those guaranteed by the practical shared neural implementation.Source: §C.1 · p. 22
• The tabular convergence analysis is an idealized limiting case rather than a global-convergence claim for neural function approximation, where freezing is implemented by masking completed-turn tokens from later losses.The conditioning map is decision-sufficient when states indistinguishable under ψ have identical optimal action values; full-history conditioning satisfies this automatically.Source: §C.1 · pp. 22–23
• With γ_H = 1, the formulation reduces to the undiscounted finite-horizon setting used in many sparse-reward agentic RL tasks.The exponent accounts for token-generation steps elapsed before a future reward.Source: §C.1 · p. 23
• The augmented action-value function evaluates a complete turn-k response together with downstream policy execution from S_{k+1} onward.This captures the continuation-dependent quality of a turn-level action.Source: §C.1 · p. 23
• In sparse-reward settings, nonterminal immediate rewards are often zero, so turn-level action quality primarily comes from downstream continuation value.Source: §C.1 · p. 23
• The optimal value is characterized by a backward Bellman optimality recursion over turn boundaries.Source: §C.1 · p. 23

Theorem 1(a): Per-turn convergence

• Each turn’s tabular update converges under fixed downstream policies because its continuation target is stationary and satisfies Robbins–Monro conditions.Finite state and macro-action spaces, sufficient exploration, bounded targets, and appropriate step sizes support convergence to the augmented turn-level action value.Source: §C.2 · pp. 24–25

Theorem 1(c): Global optimality

• Context-sufficient macro-action completeness makes recursive optimality equivalent to global optimality over the full trajectory.Every globally feasible trajectory-level policy must be representable through turn-level policies acting on contexts ck = ψ(Sk).Source: §C.2 · p. 24

Theorem 1(b): Recursive optimality

• Reverse-order training converges to recursively optimal turn policies by optimizing each turn with downstream policies optimized and frozen.Backward induction proceeds from turn K−1 to turn 0, with each per-turn stochastic approximation converging under the theorem’s assumptions.Source: §C.2 · pp. 23, 24, 26

Interpretation

• Without reverse-order freezing, changing downstream continuation values would make the turn-level target nonstationary and invalidate the single-turn stochastic-approximation argument.Freezing optimized downstream policies is therefore the condition that enables the convergence proof.Source: §C.2 · p. 26

Neural-function-approximation extension

• Under a realizable neural function class, reverse-order RTPO asymptotically realizes the recursively optimal policy on every reachable state in the visitation support.The extension replaces tabular updates with projected Q-learning and assumes the optimal sub-policy is representable asymptotically.Source: §C.2 · p. 27

• Flat-trajectory baselines center returns around the global mean μR, whereas RTPO estimates the local value Vπ(Sk), removing upstream state contamination.The contaminating term Vπ(Si,k) − μR can dominate the true turn-level advantage despite being unrelated to the current turn action.Source: §C.3 · pp. 27, 28, 31
• RTPO’s sibling estimator has only O(1/G) finite-group bias and no upstream state-contamination term, unlike the trajectory-level estimator.The sibling estimator is locally unbiased up to the finite-group factor (G −2)/(G −1), whereas flat trajectories introduce a cross-state term unrelated to the current action.Source: §C.3 · pp. 28–30
• When σ2_V > 0, state-matched sibling baselines have strictly lower MSE because they remove cross-state value error and variance.Under matched downstream-noise conditions, trajectory-level baselines retain a σ2_V contribution, while sibling baselines do not.Source: §C.3 · pp. 28, 30
• Shared boundary states make return differences attributable to turn-k macro-actions and downstream noise rather than upstream trajectory differences.Conditioned on the common state S_k, sibling expected-return differences equal differences in their turn-level action values.Source: §C.3 · pp. 28, 31
• RTPO assigns each turn-k advantage only to that turn’s output tokens, excluding shared-prefix tokens from the gradient.The resulting gradient support contains only sibling rollout tokens sampled at turn k; prefix tokens receive zero gradient.Source: §C.3 · pp. 28, 31

On-Policy Continuation without Drift

• RTPO synchronizes current downstream parameters before generating sibling rollouts, so each continuation is sampled and evaluated under the same policy.Sibling rollouts fork from the shared boundary state and continue to termination under πθ>k.Source: §C.4 · p. 32
• Each sibling return is an unbiased current-policy turn-level Q-value estimate, and the trajectory-level importance-sampling weight is exactly 1.Consequently, RTPO requires no trajectory-level importance-sampling correction and avoids policy-drift contamination.Source: §C.4 · pp. 32–34

Dynamic Error Reduction in Advantage Estimation

• Reverse-order training reduces Q-value estimation variance when downstream success probabilities move away from the high-uncertainty region around 1/2.For normalized rewards, the analogous result is a reduced worst-case bounded-variance estimate.Source: §C.4 · pp. 32–33
• Lower Monte Carlo noise in sibling Q-values improves the signal-to-noise ratio of the centered turn-level advantage estimator.By contrast, stale trajectory-level importance-sampling weights can compound multiplicatively over long continuations, causing high variance even under small per-token policy mismatch.Source: §C.4 · pp. 32–34

D Experimental Setup

• Experiments use Qwen3-8B as the main backbone and Qwen3-4B for mathematical credit-assignment studies, with thinking mode enabled across agentic RL experiments.Both models support contexts up to 32,768 tokens; the smaller model isolates turn-level credit assignment from external knowledge retrieval.Source: D Experimental Setup · p. 34
• RTPO is compared with trajectory-only GRPO and turn-level or tree-based ARPO, Tree-GRPO, and SeeUPO baselines.The baselines respectively use group-relative advantages, adaptive branching, shared-prefix tree advantages, or sequential per-turn updates.Source: D Experimental Setup · pp. 34–35
• The evaluation covers tool-using mathematical reasoning and web-search knowledge reasoning, both requiring multi-turn interaction, external tools, and adaptation to intermediate feedback.Mathematical evaluation includes standard and competition-level benchmarks, while web-search evaluation includes multi-hop QA and held-out deep-search datasets.Source: D Experimental Setup · p. 35
• Training uses VeRL, vLLM, and FSDP on 8× NVIDIA A100 GPUs with batch size 64, maximum response length 4096 tokens, and horizons of K = 3 or K = 6 turns.The learning rate is 1 × 10−6 with AdamW and weight decay 0.01; branching baselines use their reported hyperparameters.Source: D Experimental Setup · p. 35
• Evaluation reports Pass@1, tool-call counts, rollout–training consistency, KL divergence, and controlled analyses of turn-level credit assignment and on-policy continuation.Each experiment is repeated three times, with average performance reported.Source: D Experimental Setup · pp. 35–36

E.1 Pseudocode

• RTPO first generates a complete trunk trajectory, records boundary states and environment snapshots, then trains turns in reverse order without updating the trunk.At each turn, the method restores snapshots, generates on-policy siblings, continues them under the current policy to terminal reward, computes sibling-based advantages, and applies PPO only to that turn’s tokens.Source: §E.1 · p. 36
• Shared sibling states make the turn-level advantage free of upstream state contamination, while each completed turn freezes its policy before the next reverse update inherits it.The pseudocode synchronizes the inference engine and uses the updated policy as the importance-sampling denominator for each turn.Source: §E.1 · p. 36

E.2 Source Code

• The supplementary repository provides installation instructions, versioned dependencies, reproducible training and evaluation scripts, and VERL-based additions for summarized contexts, on-policy tree rollouts, per-turn advantages, and reverse updates.These materials support reproducing the mathematical and knowledge-reasoning experiments.Source: §E.2 · p. 36

E.3 Training Details

• All methods receive 80k mathematical-reasoning rollouts or 16k knowledge-reasoning rollouts, with 16 rollouts per prompt and no greater aggregate compute for RTPO.The mathematical and knowledge tasks use 5k and 1k prompts, respectively; γ = 1 for all methods.Source: §E.3 · p. 37
• The experiments use maximum interaction horizons of K = 3 turns for mathematics and K = 6 turns for web search, with 8192-token prompts and 4096-token single-turn responses.Training and rollout batch sizes are both 64, and the Qwen3 learning rate is 1 × 10−6 with initial KL coefficient 0.02.Source: §E.3 · p. 37
• RTPO uses 4 trunks for mathematics and 2 for web search, then forks 2 sibling rollouts at each realized turn boundary until termination.The differing trunk counts account for early termination before the maximum horizon.Source: §E.3 · p. 37
• Baselines use 16-chain or structured-tree sampling, while RTPO’s reverse-tree construction is configured to reach the same per-prompt rollout budget.GRPO and SeeUPO sample 16 chains; ARPO uses N = 8 and M = 16; TreeGRPO uses M = 4, N = 3, and L = 1.Source: §E.3 · p. 37
• The experiments run on 8× NVIDIA A100 80GB GPUs, requiring approximately 15 hours for mathematics and 5 hours for knowledge reasoning.The system uses 256 CPU cores, 256GB memory, and Ubuntu 22.04 LTS.Source: §E.3 · p. 37

E.4 Qwen3 Chat Template

• Qwen3 removes each turn’s <think>...</think> trace before the next input while retaining commentary, the tool call, and the tool result.The next-turn context therefore contains the original request plus the prior action history and environment feedback, preserving decision-relevant information while limiting context growth.Source: §E.4 · p. 37

Mathematical reasoning tasks

• RTPO reduces redundant verification on mathematical tasks, solving ten AIME24 problems with four Python calls versus Vanilla’s 17 calls and one solved problem.The discussion links this pattern to stronger control over reasoning, with tools used mainly for verification and numerical computation.Source: §F.1 · p. 38
• 33.33% Pass@1 on AIME24 lets RTPO surpass SeeUPO (20.00%) and GRPO (10.00%) by 13.3 and 23.3 absolute points.RTPO also maintains a clear lead on AIME25 and OE-Math while using the fewest tool calls on hard mathematical problems.Source: §F.1 · p. 39
• Short mathematical horizons limit baseline disadvantages because context discrepancy remains manageable, making standard training dynamics sufficient for near-optimal performance.Accordingly, overall method gaps on mathematical tasks remain relatively small despite RTPO’s clearer advantage on the hardest benchmarks.Source: §F.1 · p. 39

Knowledge reasoning tasks

• 64.89 F1 on HotpotQA lets RTPO exceed SeeUPO (54.27) and GRPO (53.39), while maintaining a clear lead on 2Wiki.RTPO issues 613 and 867 search calls on HotpotQA and 2Wiki, respectively, versus roughly 200 for other methods.Source: §F.1 · p. 39
• Long-horizon knowledge tasks amplify rollout–training context mismatch, whereas RTPO structurally eliminates this bias and directs optimization toward the task objective.The passage attributes baseline underperformance to insufficient sustained follow-up retrieval and multi-turn evidence integration.Source: §F.1 · pp. 39–40
• A multi-hop 2Wiki example shows a hop-completion error: the rollout identifies director Roy Mack but skips retrieving his birthplace and returns the intermediate entity.The error occurs after the first search succeeds, demonstrating that early termination can produce an incorrect answer despite partial retrieval success.Source: §F.1 · p. 40

Main results

• Figure 6 illustrates differing policy performance on a knowledge-reasoning task using web-search tools.The supplied caption identifies the task and tool setting but does not provide quantitative cell values.Source: Figure 6 · p. 40
• RTPO uses fewer calls with more reliable internal reasoning for mathematical tasks, but denser calls with more thorough external information integration for knowledge tasks.This bidirectional adaptation spans both task categories.Source: §F.1 · p. 40
• RTPO’s advantage comes from its turn-level optimization objective rather than simply encouraging or suppressing tool calls.The objective learns tool-use strategies matched to the underlying nature of each task.Source: §F.1 · p. 40

F.2 Insights from Rollout–Training Consistency Analysis

• Baselines’ ratios can drift toward 1 because trajectories generated from summary rollouts are reinforced under full-history training, indirectly aligning the two conditionings.This mechanism resembles DAgger [Ross et al., 2011] and the bootstrapped alignment observed by SCoRe [Kumar et al., 2025].Source: §F.2 · p. 40
• Summary contexts may become learnable sufficient decision states when they preserve decision-relevant information, enabling information-theoretic policy convergence [Allen et al., 2021].Related multi-turn RL work reports that end-to-end optimization can turn summaries into compact decision states [Lu et al., 2025, Yu et al., 2026a].Source: §F.2 · p. 41
• GRPO recovers only to 0.97 on math after 30 steps and 0.83 on search after 14 steps, showing that alignment is incomplete and task-dependent.Longer horizons and faster context accumulation make empirical alignment harder, as also reflected in knowledge reasoning results in Table 1.Source: §F.2 · p. 41
• SeeUPO reaches a 1.02 math-ratio spike at step 9 and sustains fluctuations, whereas RTPO’s ratio is structurally guaranteed apart from training–inference engine differences.Advantage-estimate variance propagates through the bootstrapped loop and causes step-to-step jitter.Source: §F.2 · p. 41
• RTPO avoids the compensatory rollout–training mismatch that baselines must increasingly repair on long-horizon tasks.The ratio analysis indicates that baselines do not necessarily fail, but their compensation becomes less effective as task horizons grow; RTPO removes this gap by construction.Source: §F.2 · p. 41

F.3 Additional Findings for Hard-Search Scenarios

• Standard RTPO synchronizes latest parameters θ>k with the inference engine at turn k and regenerates sibling continuations on-policy, making trajectory-level IS weight identically one.The comparison holds reverse-order turn-level training and state-matched sibling structure fixed while isolating on-policy continuation.Source: §F.3 · p. 41
• The off-policy RTPO variant reuses initial-rollout continuations generated under πθ0 and corrects their staleness with a clamped trajectory-level importance weight.Thus, the variant differs from standard RTPO only in how sibling downstream continuations are obtained.Source: §F.3 · pp. 41–42
• The on-policy and off-policy variants are compared using output-hit accuracy across GAIA, WebWalkerQA, XBench, and HLE deep-search benchmarks.GAIA includes three difficulty levels, while HLE contains eight subject subsets; output-hit counts a sample correct when its full output contains the gold answer or an alias.Source: Table 6 · p. 41, §F.3 · p. 42

F.4 Limitations and Future Work

• RTPO’s algorithmic design has limitations despite stronger theoretical guarantees and empirical performance than existing flat-trajectory methods.Source: §F.4 · p. 42
• Poor early trunk decisions can place later boundary states in low-value regions, causing near-zero turn-level advantages and ineffective gradient updates.RTPO currently uses a single trunk anchor; multi-trunk sampling or retroactive trunk selection could improve coverage but increase rollout cost.Source: §F.4 · p. 42
• Reverse multi-turn training adds cost that scales with turns and sibling rollouts, while policy synchronization introduces inference-training latency.Batched parallel sibling generation and asynchronous scheduling partially mitigate, but do not eliminate, this overhead.Source: §F.4 · p. 42
• RTPO under-utilizes training tokens by updating only turn-k sibling tokens, discarding prefixes, downstream continuations, and trunk tokens from gradient updates.This preserves causal credit assignment; auxiliary language-modeling losses or self-play rewards could improve token efficiency without changing that principle.Source: §F.4 · p. 42

F.5 Broader Impacts

• More stable agentic RL could reduce failed tool-use trajectories, improve sample efficiency, and support reliable deployment in research, education, software engineering, and decision support.Turn-level monitoring may also clarify how agents learn planning, search, and tool use across multiple turns.Source: §F.5 · pp. 42–43
• More autonomous long-horizon agents could produce confident errors, misuse tools, or amplify harmful automation without appropriate safeguards.Deployment should include safety constraints, tool-use monitoring, privacy-preserving data handling, and human oversight, especially in high-stakes domains.Source: §F.5 · p. 43

G.1 Stateful Tool-Agent Evaluation on τ 3-Airline

• 56.25% peak online success at Step 40 makes RTPO best from Step 20 onward, but success falls to 40.63% at Step 50 under sparse rewards and a small training set.The late decline indicates that continued optimization can degrade performance in this setting.Source: Table 7 · p. 44, §G.1 · p. 44
• RTPO achieves the highest Pass@1, Pass@4, and normal termination rate, while reducing truncation, response length, and REFUEL-comparison metrics on held-out τ 3-Airline tasks.Relative to Tree-GRPO, truncation falls from 47.50% to 42.50% and response length from 2,538.2 to 1,688.8 tokens; relative to REFUEL, Pass@1 rises from 11.25% to 17.50% and Pass@4 from 20.00% to 30.00%.Source: Table 8 · p. 44, §G.1 · p. 44
• RTPO reduces tool-error calls from 50 to 9, raises execution success from 57.63% to 89.16%, and uses 9,650.5 response tokens per successful trajectory.This token cost is 38.2% lower than Tree-GRPO and 56.0% lower than GRPO, while RTPO attempts fewer calls than GRPO.Source: Table 9 · p. 44, §G.1 · p. 44

G.2 Sensitivity to the Sibling Group Size

• Increasing sibling group size G from 3 to 4 lowers zero-advantage cases from 89.61% to 82.66%, raises Pass@1 from 17.50% to 20.00%, and cuts inference tokens from 9,650.5 to 7,477.9.The larger group provides denser relative signals and more Monte Carlo averaging, with finite-group bias of order O(1/G), but requires more offline exploration.Source: Table 10 · p. 45, §G.2 · p. 45
• Moving from G = 3 to G = 4 increases GPU-hours 30.6% and training-generation tokens 152.8%, while improving Pass@1 by 2.5 points and leaving Pass@4 unchanged.G = 3 is the stronger default cost–performance trade-off; G = 4 suits prioritized Pass@1 and concise inference, while G = 3 is the minimum informative configuration because G = 2 degenerates.Source: Table 11 · p. 45, §G.2 · p. 45

G.3 Trunk Quality and Failure Dynamics

• Only 6.88% of trunks fail within the first turn, so most errors occur later; finite-sample early failures do not prove strict dead states.An observed early failure cannot establish that every possible future policy is unable to recover.Source: Table 12 · p. 45, §G.3 · p. 45
• Trajectories without terminal signals fall from 37.50% at Step 10 to 0% at Step 50, while failed-trajectory length drops from 13.30 to 2.84 turns.Later failures increasingly become fast but incorrect terminations rather than persistent generation until truncation; the post-Step-40 decline may reflect sparse-reward over-optimization.Source: §G.3 · pp. 45–46, Table 13 · p. 46
• RTPO samples all trunks on-policy without heuristic filtering, using same-state sibling returns to identify informative relative signals while preserving the visited state distribution.Filtering low-quality trunks could reduce sibling-sampling cost but would concentrate optimization on manually selected states.Source: §G.3 · p. 46

G.4 Selective-Gradient Optimization and Training Efficiency

• RTPO applies loss only to current-turn sibling outputs, preserving reached contexts and downstream returns for causal credit while optimizing every turn across the reverse sweep.The method trades approximately 41.3% higher wall-clock time and GPU-hours, plus approximately 40.0% more training-generation tokens, for state-matched credit and current-policy continuations.Source: §G.4 · pp. 46–47
• 8.3% more training tokens than Tree-GRPO and 40.0% more than GRPO accompany the highest Pass@1, while successful-trajectory response tokens fall 38.2% and 56.0%, respectively.These results indicate that additional offline continuation sampling increases training computation without producing more verbose deployment-time inference.Source: §G.4 · p. 46, Table 14 · p. 46

G.5 Tool-Use Behavior on GSM8K

• RTPO improves accuracy while reducing total tool calls across GSM8K, AIME24, and AIME25 relative to the vanilla model.In an audit of 100 GSM8K problems, all 36 calls executed successfully, and 34 of 36 tool-using trajectories had already derived the answer before one Python verification call.Source: §G.5 · pp. 47–48, Table 18 · p. 47, Table 19 · p. 48

G.6 On-Policy Continuation and Synchronization Frequency

• Per-stage on-policy continuation improves long-horizon deep-search performance most on GAIA Level 3, while remaining broadly comparable to off-policy RTPO on shorter, more static HLE retrieval.Fixed-interval and policy-KL synchronization reduce cost but no longer guarantee ω_j,k ≡ 1.Source: §G.6 · p. 48

G.7 Budget Consumption Across Training

• From Step 10 to Step 50, tool calls decrease from 7.34 to 0.94 per trunk, generated tokens from 8,013.5 to 1,669.3, and tool-execution success rises from 74.04% to 93.33%.Early stages remain more expensive because the initial policy generates longer trajectories and more tool calls, but actual per-trunk cost declines as training improves efficiency.Source: §G.7 · pp. 48–49, Table 21 · p. 49

G.8 Further Distinctions from Related Methods

• RTPO differs from SeeUPO, ArCHer, REFUEL, R3, and SRL by unifying state-matched sibling comparison, outcome-supervised turn advantages, reverse updates, and on-policy continuation to address three coupled inconsistencies.Unlike these methods, RTPO targets rollout–training context mismatch, trajectory-only credit contamination, and asynchronous downstream-continuation drift without an explicit critic or expert trajectories.Source: §G.8 · p. 49

G.9 Additional Limitations

• RTPO’s strict convergence guarantee applies only to the fixed-downstream-policy tabular formulation, while shared neural policies lack a global guarantee for non-convex optimization.Training also costs approximately 41.3% more wall-clock time and GPU-hours than GRPO; increasing G = 4 raises training-generation tokens by approximately 152.8%.Source: §G.9 · pp. 49–50
• Above 80% of Airline samples still have zero advantage at G = 4, online success falls from 56.25% at Step 40 to 40.63% at Step 50, and experiments average only three runs.The paper therefore motivates denser rewards, adaptive sibling sampling, broader model and turn-length coverage, and intermediate synchronization studies.Source: §G.9 · p. 50