PaperlayerView exact revision on arXiv

Source-linked AI summary

Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction

Zhong Guan, Yongjian Guo, Haoran Sun, Wen Huang, Shuai Di, Likang Wu, Xiong Jun Wu, Hongke Zhao

arXiv:2605.12070v2cs.LGcs.AI

TL;DR

Asynchronous Agentic RL can lose historical training-side logits, undermining the semantic separation of discrepancy repair and policy-staleness correction. This paper analyzes exact recovery and low-cost approximation strategies, finding that revised PPO-EWMA consistently improves over proxy baselines while often approaching exact recovery.

  • Problem

    Asynchronous Agentic RL may discard historical training-side logits, leaving decoupled off-policy correction without its intended semantic separation.

  • Method

    The paper decomposes correction into discrepancy and staleness terms, studies three exact old-logit recovery routes, and develops revised PPO-EWMA as a low-cost approximation.

  • Results

    PPO-EWMA consistently improves over Decoupled PPO and Linear_prox and often approaches Snapshot results on held-out Agentic benchmarks.

  • Takeaways & Limitations

    Exact recovery improves correction fidelity, while PPO-EWMA offers a practical alternative when exact old-logit recovery is too expensive.

  • Takeaways & Limitations

    Synchronization via partial rollout interruption introduces synchronization stalls, resource reconfiguration overhead, and disruption to rollout parallelism.

Abstract

from arXiv · show

Asynchronous reinforcement learning improves rollout throughput for large language model agents by decoupling sample generation from policy optimization, but it also introduces a critical failure mode for PPO-style off-policy correction. In heterogeneous training systems, the total importance ratio should ideally be decomposed into two semantically distinct factors: a \emph{training--inference discrepancy term} that aligns inference-side and training-side distributions at the same behavior-policy version, and a \emph{policy-staleness term} that constrains the update from the historical policy to the current policy. We show that practical asynchronous pipelines with delayed updates and partial rollouts often lose the required historical training-side logits, or old logits. This missing-old-logit problem entangles discrepancy repair with staleness correction, breaks the intended semantics of decoupled correction, and makes clipping and masking thresholds interact undesirably. To address this issue, we study both exact and approximate correction routes. We propose three exact old-logit acquisition strategies: snapshot-based version tracking, a dedicated old-logit model, and synchronization via partial rollout interruption, and compare their system trade-offs. From the perspective of approximate correction, we focus on preserving the benefits of decoupled correction through a more appropriate approximate policy when exact old logits cannot be recovered at low cost, without incurring extra system overhead. Following this analysis, we adopt a revised PPO-EWMA method, which achieves significant gains in both training speed and optimization performance.

1 Introduction

Asynchronous Agentic RL can lose the historical training-side logits needed to separate training–inference discrepancy repair from policy-staleness correction. This paper analyzes the resulting semantic failure and compares exact recovery routes with a low-cost PPO-EWMA approximation.

  • Introduction: Asynchronous rollouts, delayed updates, and partial trajectories can make the training-side old policy unavailable, causing an old-logit mismatch.The mismatch arises when trajectories span multiple parameter versions and the actor advances beyond the version that generated earlier tokens.
  • Introduction: The intended correction decomposes the total ratio into discrepancy repair at a shared old version and policy-staleness correction from that version to the current policy.The two ratios have distinct meanings and should therefore be controlled differently.
  • Introduction: Missing old logits invalidate this decomposition, entangling discrepancy repair with staleness correction and making shared ratios or thresholds insufficient.Discrepancy repair filters numerically inconsistent tokens, whereas staleness correction uses sign-dependent PPO clipping.
  • Introduction: Interpolation-based proxy policies mainly re-parameterize effective clipping boundaries rather than recovering the missing reference policy.The paper evaluates this limitation within a unified dual-constraint analysis of PPO-style objectives.
  • Introduction: The paper studies three exact old-logit acquisition routes and a revised PPO-EWMA reference as a low-cost approximation without extra system overhead.Experiments compare exact recovery, proxy references, and PPO-EWMA across optimization behavior and system overhead on dense and MoE LLMs.

2 Related Work

Prior work stabilizes LLM reinforcement learning through clipped objectives, importance-weight modifications, training–inference mismatch mitigation, and reference policies for stale updates. This work distinguishes itself by examining whether asynchronous systems retain a semantically correct reference policy when old training-side logits are missing.

  • On- and off-policy reinforcement learning: PPO and GRPO stabilize LLM reinforcement learning with clipped objectives, but on-policy training can be inefficient for long-horizon Agentic tasks.Expensive rollout generation and synchronization-limited GPU utilization motivate off-policy and asynchronous pipelines.
  • Off-policy correction: Recent methods improve off-policy robustness by modifying importance-sampling weights through clipping, gradient-constraint separation, second-moment control, or effective sample size.The cited methods include CISPO, GPPO, M2PO, VESPO, VCPO, MiniRL, and TOPR.
  • Positioning of this work: This work asks whether asynchronous reference policies are semantically correct: proxy references can help when old training-side logits are absent but do not exactly recover Eq. (1).This distinction motivates the paper’s decoupled view of correction and proximal stabilization.
  • Training–inference mismatch: Training–inference mismatch methods mitigate numerical divergence using masking, clipping, or routing replay, with mismatch especially visible in MoE models.Masked Importance Sampling (MIS) masks tokens with severe training–inference divergence.
  • Reference policies for stale updates: Reference-policy methods stabilize stale updates by separating importance correction from proximal constraints, approximating proximal policies, or maintaining smoothed policy anchors.Examples include Decoupled PPO, A-3PO, and PPO-EWMA-style references.

3 Preliminaries

This section defines token-level PPO importance ratios, clipped objectives, and advantage-dependent masking for LLM RL fine-tuning. It then distinguishes inference- and training-side policies in asynchronous systems and identifies missing historical policies as a source of semantically entangled correction.

  • PPO formulation: PPO uses the token-level ratio r_t(θ)=π_θ(y_t|x,y_<t)/π_old(y_t|x,y_<t) in its clipped surrogate objective.The response is sampled from π_old, and an advantage estimate A_t is computed for each token or sequence.
  • PPO formulation: PPO clipping clips ratios above 1+ϵ when A_t>0 and below 1−ϵ when A_t<0.The resulting gradient can be written in masked importance-sampling form using the PPO-side active mask.
  • PPO formulation: The active mask combines advantage-sign conditions with ratio bounds to constrain policy updates and prevent excessively large changes.It is defined as I{A_t≥0}I{r_t(θ)≤1+ϵ}+I{A_t<0}I{r_t(θ)≥1−ϵ}.
  • Asynchronous distributions: Asynchronous LLM RL systems can produce distinct inference-side μ_v and training-side π_v distributions at the same parameter version.Inference runs on engines such as vLLM or SGLang, while training runs on systems such as Megatron or FSDP.
  • Old-logit mismatch: When π_old is discarded after delayed or partial asynchronous rollouts, replacing it with an interpolated surrogate entangles discrepancy repair with staleness correction.The loss remains algebraically correct, but the two correction factors no longer retain their separate meanings.

4 A Unified Analysis of Decoupled Correction

The section explains why decoupled discrepancy repair and staleness correction cannot be collapsed into standard PPO clipping, then unifies existing corrections under the MIS formulation. It shows that asynchronous version mismatch causes missing-old-logit failures and motivates exact acquisition or delay-aware approximate references.

  • Decoupled correction semantics: Discrepancy repair requires a symmetric constraint around 1, whereas PPO uses an advantage-sign-dependent asymmetric filter; combining them forces an incompatible shared threshold.The resulting interaction compromises optimization and makes discrepancy masking and PPO-CLIP act through the same final active-token set.
  • Unified MIS analysis: Existing LLM off-policy methods generally decompose correction into discrepancy and staleness ratios, but asynchronous version latency creates a mismatch between current, asynchronous, and old policy versions.In synchronous training, an accessible πold supports an exact semantic decomposition; asynchronous latency breaks that reference alignment.
  • Approximate reference limits: Arithmetic or token-wise log-linear interpolation does not restore an exact reference: it merely re-parameterizes the effective clipping and masking boundaries of the total ratio.This result is formalized in Proposition 1 for r(θ) = πθ/µold.
  • Repair directions: The section therefore considers infrastructure-supported acquisition of ground-truth old logits alongside a lower-overhead approximate reference based on a revised exponential moving average that accounts for asynchronous delays.The exact acquisition options include snapshot-based version tracking, a dedicated old-logit model, and partial rollout interruption synchronization.

5 Recovering and Approximating Old Logits

This section presents three exact strategies for recovering old logits, each trading off semantic fidelity against system cost, and then adopts PPO-EWMA as a lower-cost approximation when exact acquisition is too expensive. PPO-EWMA replaces unavailable old logits with a smoother reference policy and uses staleness-window tuning plus automatic resetting to limit accumulated staleness.

  • Exact acquisition: Exact old-logit acquisition is framed through three strategies: snapshot-based version tracking, a dedicated old-logit model, and partial rollout interruption.All three target πold, the training-side token probability under the rollout version.
  • Exact acquisition: Snapshot tracking restores the semantic decomposition most cleanly but requires additional memory and potentially frequent actor-side version switching.Partial rollouts can span multiple policy versions, complicating exact recovery.
  • Exact acquisition: A dedicated old-logit model overlaps old-logit computation with gradient updates and reduces contention on the actor path, but requires resource partitioning.Its separation from update training can reduce end-to-end actor-stage time.
  • Exact acquisition: Partial rollout interruption computes exact logits with the still-resident historical version and avoids storing old weights, but introduces synchronization stalls and resource reconfiguration overhead.The system temporarily switches rollout resources to actor-side old-logit computation before resuming generation.
  • Approximate correction: When exact acquisition is too expensive, PPO-EWMA uses an exponentially averaged reference policy as a low-cost approximation instead of claiming exact recovery.The proxy replaces unavailable πold for both staleness correction and discrepancy repair.
  • Approximate correction: PPO-EWMA sets βprox ≈Wstale/(Wstale + 2) to center the reference in the asynchronous version window and resets it when stale history causes masking to reject many tokens.The reset re-centers the proxy around the current actor.

6 Experiments

Experiments compare exact old-logit recovery, proxy references, and PPO-EWMA across dense and MoE Agentic RL models. PPO-EWMA improves practical-method performance while offering a more favorable performance–cost trade-off and stabilizing training through threshold-aware masking and automatic reset.

  • Experimental setup: Experiments evaluate dense Qwen3-4B and MoE Qwen3-30B-A3B models on retail, airline, telecom, in-store, and delivery Agentic RL benchmarks.The asynchronous setup caps the rollout–actor version gap at three while minimizing additional staleness from minibatch reuse and repeated PPO epochs.
  • Main benchmark results: PPO-EWMA consistently improves over Decoupled PPO and Linear_prox and often approaches the idealized Snapshot results.Snapshot assumes exact old logits are available, whereas PPO-EWMA uses a maintained EWMA reference.
  • Main benchmark results: PPO-EWMA achieves the best pass@4 on 4B retail, best pass@2 on 4B VitaBench in-store, and strongest 30B performance on the airline split, while tying several best scores.It ties the best telecom scores on 4B and the best retail pass@4 and telecom avg@2 on 30B, indicating substantial recovery of exact-correction benefits.
  • System overhead: Snapshot exact recovery is accurate but costly because it requires version switching, snapshot storage, and extra recovery time, especially for the 30B MoE model.A dedicated old-logit model can overlap computation, but its benefit depends on resource partitioning and requires additional model-side infrastructure.
  • Threshold analysis: Looser discrepancy thresholds improve early learning but can destabilize later trajectories, whereas stricter thresholds slow the start and reduce late collapse.Discrepancy masking and PPO clipping are coupled: looser stale-policy thresholds allow more problematic tokens early, while stronger clipping later helps mask recovery.
  • EWMA reset: With β = 0.75, automatic EWMA reset recovers a high Train-Infer Mask after reference drift while preserving most of the early Task success gain.Only a few re-centering events are needed, supporting reset as a low-cost response to an excessively stale proxy reference.

7 Conclusion

The paper identifies the missing-old-logit problem as a threat to decoupled off-policy correction in asynchronous Agentic RL and addresses it through exact recovery and low-cost approximation. Exact recovery improves correction fidelity, while revised PPO-EWMA offers a practical alternative when recovery is too expensive.

  • The missing-old-logit problem undermines decoupled off-policy correction in asynchronous Agentic RL for LLMs.
  • The paper addresses the problem through infrastructure-level exact old-logit recovery and low-cost approximation with a revised PPO-EWMA reference.
  • Exact old-logit recovery improves correction fidelity, whereas PPO-EWMA provides a practical alternative when exact recovery is too expensive.

A Detailed Derivations for Interpolation-Based Proxies · B PPO-EWMA Update Details · C Full PPO Variant Comparison

The appendices show that interpolation proxies reparameterize effective PPO constraints without reconstructing missing old logits, while PPO-EWMA provides a normalized historical-policy reference and the variant comparison distinguishes staleness from discrepancy correction.

  • A Detailed Derivations for Interpolation-Based Proxies: Log-linear interpolation changes effective thresholds to ϵ1,eff ≈ ϵ1/(1−α) and ϵ2,eff ≈ ϵ2/α, reparameterizing the active region of rθ.The derivation treats the proxy as changing constraints on the original total ratio rather than introducing a new correction factor.
  • A Detailed Derivations for Interpolation-Based Proxies: Arithmetic interpolation likewise produces scaled constraints whose active region is determined by intersections of mask and PPO clipping bounds.The resulting constraints remain functions of the single total ratio rθ.
  • A Detailed Derivations for Interpolation-Based Proxies: For version gaps n ∈ {1, 2, 3}, the numerical mappings use α = 1/(n + 1) and permit asymmetric lower and upper clipping thresholds.The settings include experimental thresholds and nearby alternatives, with symmetric PPO clipping as a special case.
  • A Detailed Derivations for Interpolation-Based Proxies: Removing interpolation and applying reparameterized constraints directly yields a training curve almost identical to log-linear interpolation, supporting the reparameterization interpretation.The appendix therefore treats interpolation as a stabilizer, not exact training–inference discrepancy repair, because it does not reconstruct missing old logits.
  • B PPO-EWMA Update Details: PPO-EWMA maintains a normalized exponentially weighted average of historical actor states, with larger βprox retaining longer policy memory and smaller values emphasizing the current actor.The appendix records the standard parameterization and motivates its reference-policy behavior.
  • C Full PPO Variant Comparison: The full comparison distinguishes r1 as the policy-update or staleness ratio and r2 as the training–inference discrepancy or proxy-reference ratio.For positive advantages PPO clipping keeps samples before the upper boundary, while for negative advantages it keeps samples after the lower boundary.
  • C Full PPO Variant Comparison: Standard PPO-clip uses one mixed ratio, whereas decoupled PPO separates staleness and discrepancy when πold is available; asynchronous training may instead require πasync or πprox.The comparison frames missing training-side old logits as the obstacle to restoring the intended two-factor structure.

D Additional Threshold Examples

Additional threshold comparisons show a recurring trade-off between faster early learning from looser stale-policy filtering and smoother late training from stricter filtering. The examples also show that discrepancy repair and PPO-CLIP interact through the active-token set.

  • Threshold trade-offs: Across threshold pairs, looser stale-policy filtering improves early speed, whereas stricter filtering supports smoother late stability.Run names encode discrepancy and stale-policy thresholds; success figures show task curves and their differences.
  • Figure 6: With discrepancy threshold 1.005, the looser stale-policy threshold retains more update signal and improves early progress, but later becomes more oscillatory.The stricter setting catches up more smoothly later.
  • Figure 7: With discrepancy threshold 1.003, stricter discrepancy filtering narrows the difference, while the looser stale-policy setting still improves early speed and the stricter setting controls the trajectory better.Both settings discard more biased tokens than the 1.005 pair.
  • Figure 8: At discrepancy threshold 1.002, aggressive filtering reduces biased tokens and stabilizes training but limits useful off-policy signal, producing slower optimization and a lower early success curve.This example illustrates the cost of stronger filtering for policy improvement.
  • Figure 9: With shared discrepancy threshold 1.004, changing the stale-policy threshold alters both PPO clip fraction and the training–inference mask trajectory.The mechanisms interact through the active-token set, affecting which questionable tokens remain for discrepancy repair.

E Additional PPO-EWMA Ablations

The ablations show that PPO-EWMA must balance memory, threshold strictness, and reset behavior. Longer memory stabilizes but accumulates stale-policy bias, while shorter memory and looser thresholds improve early progress but can weaken correction or cause instability.

  • Decay-factor behavior: β = 0.9 accumulates stale history, misaligns πprox with recent rollout policies, and makes the Train-Infer Mask progressively more aggressive.Tightening either the discrepancy threshold or PPO-CLIP ratio threshold does not remove this failure mode.
  • Decay-factor behavior: β = 0.5 keeps more tokens active but has a less efficient early Task success curve, illustrating the trade-off between adaptive memory and proximal anchoring.Longer memory provides a stronger stabilizing anchor but can accumulate stale-policy bias, whereas shorter memory weakens the proximal reference.
  • Threshold interaction: For β = 0.4 and β = 0.5, looser Train-Infer Mask or PPO-CLIP ratio thresholds retain more early update signal but can cause a mid-training success drop.Looser settings admit more mismatched tokens into optimization, while coupled mask and clipping dynamics can later recover part of the trajectory.
  • Automatic reset behavior: Automatic reset resolves β = 0.75’s late collapse while preserving most early efficiency, triggering only three times at τ = 0.9 and two times at τ = 0.8.The few resets clear stale history in θprox and restore a high Train-Infer Mask value; later steps mostly have no reset trigger.

F Experimental Details · G Additional System Measurements

The experiments use ROLL as the training framework and define τ 2-Bench evaluation splits by domain.

  • F Experimental Details: τ 2-Bench uses all available base samples for the retail and airline domains.
  • F Experimental Details: τ 2-Bench uses the test split for the telecom domain.
  • F Experimental Details: ROLL serves as the base training framework for the experiments.
  • F Experimental Details: ROLL provides a well-structured agent abstraction.

G.1 Snapshot Old-Logit Overhead

Snapshot-based old-logit recovery materially increases actor-side compute and memory costs, primarily through historical-version restoration and extra forward passes. In the observed runs, both configurations eventually terminated after Raylet exits, although the logs do not establish snapshot recovery as the sole cause.

  • Compute overhead: Snapshot-old computation adds roughly 95 seconds per measured step in the 4B run and roughly 178 seconds per measured step in the 30B-A3B run.The measurements cover 77 steps for 4B and 11 steps for 30B-A3B, so they reflect observed run windows rather than final training results.
  • Compute overhead: The dominant overhead comes from loading or restoring historical versions and running extra forward passes to reconstruct old logits, rather than from snapshot saving itself.These operations materially increase actor-side compute load.
  • Memory overhead: The 30B-A3B run retains up to 76.43GB of resident snapshot state when five snapshots are kept, creating substantial memory pressure.Snapshot retention adds especially high resident storage demand in the 30B-A3B setting.
  • Observed limitation: Both runs terminate with Ray actor unavailability after the Raylet exits unexpectedly, but the logs do not prove snapshot recovery is the sole cause.The measured resident snapshot storage and additional forward time nevertheless demonstrate material system pressure.

H Limitations · NeurIPS Paper Checklist

The paper identifies limits in scale validation, systems analysis, and throughput evaluation, while the checklist notes incomplete reporting of hyperparameters, compute resources, broader impacts, and asset licensing. It also records several non-applicable concerns, including human-subject research and high-risk asset safeguards.

  • H Limitations: The correction methods have not been validated on models at the several-hundred-billion-parameter scale, where system behavior may differ qualitatively.Potential differences include memory pressure, communication overhead, expert routing, and rollout–training scheduling.
  • H Limitations: The infrastructure analysis does not provide a fine-grained end-to-end study of scheduler, networking, placement, swapping, worker-idleness, or failure effects.The analysis instead focuses on overheads from snapshot retention, version switching, historical forward computation, and partial rollout interruption.
  • H Limitations: Throughput measurements are indicative rather than comprehensive because they lack broad sweeps across cluster sizes, model scales, sequence lengths, rollout lengths, and staleness windows.A larger systems evaluation is needed to characterize the throughput frontier and identify engineering trade-offs.
  • NeurIPS Paper Checklist: The manuscript specifies evaluated backbones, domains, metrics, and maximum staleness, but does not yet include the full set of training and evaluation hyperparameters.Appendix F records implementation framework information.
  • NeurIPS Paper Checklist: The paper reports repeated runs and averaged aggregate task metrics, but omits error bars, confidence intervals, and statistical significance tests because the experiments are computationally costly.The checklist marks this criterion [Yes] while explaining the omitted uncertainty reporting.
  • NeurIPS Paper Checklist: Compute-resource accounting is incomplete for every training and evaluation run, despite system measurements for exact old-logit acquisition in Section 6.3 and Appendix G.The checklist therefore identifies missing reproducibility information about the resources required by individual runs and in total.
  • NeurIPS Paper Checklist: The manuscript does not currently include a dedicated broader-impact discussion, although potential impacts are described as indirect through more efficient and stable training of LLM agents.The checklist notes that the work is primarily methodological and systems-focused.
  • NeurIPS Paper Checklist: The current manuscript does not explicitly enumerate licenses and terms of use for all existing assets, while it introduces no new dataset, model, or software asset.The paper also states that it involves no human-subject experiments and releases no high-risk asset requiring safeguards.
Loading 2605.12070v2…