Source-linked AI summary
Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
Haizhong Zheng, Jiawei Zhao, Beidi Chen
TL;DR
The paper asks whether RL for LLMs can reuse highly stale rollouts without sacrificing stability or performance. It introduces M2PO, which constrains the second moment of importance weights to suppress extreme outliers while preserving informative updates, and finds stable training matching on-policy performance under staleness of at least 256 updates.
Problem
On-policy RL requires fresh rollouts, while highly stale off-policy training forces a trade-off between performance degradation and training collapse.
Method
M2PO constrains the second moment of importance weights, masking extreme outliers while preserving informative high-entropy-token updates.
Results
M2PO matches on-policy performance with data stale by at least 256 model updates and reduces average clipping on Qwen-2.5-32B from 1.22% to 0.06%.
Takeaways & Limitations
Stale trajectories can support stable, scalable off-policy RL when the trust region suppresses only extreme variance rather than broadly clipping updates.
Takeaways & Limitations
The batch-level KL baseline can suffer cancellation effects and fail to constrain large importance ratios, motivating the second-moment measure.
Abstract
from arXiv · showhide
Reinforcement learning has been central to recent advances in large language model reasoning, but most algorithms rely on on-policy training that demands fresh rollouts at every update, limiting efficiency and scalability. Asynchronous RL systems alleviate this by decoupling rollout generation from training, yet their effectiveness hinges on tolerating large staleness in rollout data, a setting where existing methods either degrade in performance or collapse. We revisit this challenge and uncover a prosperity-before-collapse phenomenon: stale data can be as informative as on-policy data if exploited properly. Building on this insight, we introduce M2PO (Second-Moment Trust Policy Optimization), which constrains the second moment of importance weights to suppress only extreme outliers while preserving informative updates. Notably, M2PO sharply reduces the fraction of clipped tokens under high staleness (from 1.22% to 0.06% over training), precisely masking high-variance tokens while maintaining stable optimization. Extensive evaluation across six models (from 1.7B to 32B) and eight benchmarks shows that M2PO delivers stable off-policy training even with data stale by at least 256 model updates and matches on-policy performance.
1 Introduction
On-policy RL offers stable performance but requires fresh rollouts, limiting scalability, while stale-data methods trade performance against stability. The paper identifies how informative stale data can be preserved and introduces M2PO for stable off-policy training.
- Fresh-rollout requirements make on-policy RL inefficient and limit scalability, motivating asynchronous systems that decouple rollout generation from training.
- Large staleness exposes a trade-off: stability-oriented methods degrade, whereas performance-oriented methods often collapse.
- M2PO constrains the second moment of importance weights to mask extreme outliers while preserving informative high-entropy-token updates.
- Under staleness of at least 256 model updates, M2PO matches on-policy accuracy while reducing clipped tokens from 1.22% to 0.06%.The evaluation covers six model scales from 1.7B to 32B and eight math reasoning benchmarks.
2 Related Work
Related work covers RL methods for LLM reasoning and trust-region strategies for stabilizing policy updates. These approaches motivate the need for better control of stale-data training.
- RLVR uses verifiable rewards and policy-optimization methods such as PPO and GRPO to improve LLM reasoning.
- Recent RLVR methods also improve reasoning by optimizing value functions, including VinePPO, VC-PPO, and VAPO.
- Trust-region research explores ratio, approximate, sequence-level, asymmetric, and gradient-preserving clipping strategies for RLVR.
3 Background
GRPO adapts PPO for LLM fine-tuning by normalizing rewards within response groups and clipping policy ratios. The paper studies how this baseline behaves as training data become increasingly stale.
- 3.1 Group Relative Policy Optimization (GRPO): GRPO replaces value-function advantages with normalized reward scores computed within groups of responses to the same prompt.
- 3.1 Group Relative Policy Optimization (GRPO): GRPO clips the policy ratio to [1−ϵ,1+ϵ] to limit excessive probability increases or overpenalization during updates.
- 3.2 Performance Degradation from Training with Stale Data: Stale-k RL trains each iteration on data generated k model updates earlier, with four model updates per training step in the stated setup.
- 3.2 Performance Degradation from Training with Stale Data: As staleness increases, GRPO training converges more slowly and reaches lower test accuracy on Qwen2.5-Math-7B.
4 Prosperity before Collapse: Stale Data Contain Enough Training Information in RL on LLMs
Stale trajectories can initially outperform clipped training, revealing useful information that conventional trust regions discard, but unconstrained training eventually collapses. The analysis links this trade-off to clipping of informative high-entropy tokens.
- Prosperity before collapse: training without a trust region: Under staleness s=256, removing the trust region initially outperforms clipped training and can match on-policy baselines before eventual collapse.
- Pivotal token masking by ϵ-clipping when training with stale data: Large staleness sharply increases clipping, while the importance-ratio distance from 1 rises with average token entropy.
- Pivotal token masking by ϵ-clipping when training with stale data: Because high-entropy tokens are typically most informative, ϵ-clipping disproportionately prunes useful training signals and degrades stale-data performance.
- The resulting challenge is to preserve stale-data benefits with an adaptive trust region while maintaining stable training.
5 Second-Moment Trust Policy Optimization
M2PO measures policy mismatch with the second moment of log importance ratios, targeting instability from distributional differences without overconstraining token-level variation. It masks tokens until the remaining batch satisfies a fixed second-moment threshold and applies the constraint only where PPO would actually clip.
- 5.1 Measuring Distribution Gap with the Second Moment: Batch-level KL can hide large policy deviations through cancellation and inadequately constrain tokens with ratios above one.Single-sample per-token KL estimates may be positive or negative, allowing opposing deviations to offset each other.
- 5.1 Measuring Distribution Gap with the Second Moment: M2PO uses the second moment of log importance ratios to constrain behavior–current policy mismatch while retaining sensitivity to variance and extreme-ratio outliers.Unlike batch KL, the per-token squared log-ratio is non-negative and reflects variance in importance weights.
- 5.1 Measuring Distribution Gap with the Second Moment: M2 also upper-bounds the Pearson chi-square divergence between the new and behavior policies despite not directly constraining r−1 like ϵ-clipping.The theorem establishes this relationship for the policy distributions used in the method.
- 5.2 Second-Moment Trust Policy Optimization: M2PO masks tokens selectively until the remaining batch’s second moment falls below threshold τM2, using τM2 = 0.04 consistently across experiments.The authors report that this threshold was effective across all training scenarios and was not sensitive in their experiments.
- 5.2 Second-Moment Trust Policy Optimization: The second-moment constraint is applied only to tokens satisfying PPO’s effective clipping conditions: A>0 with r>1 or A<0 with r<1.The final policy update uses the resulting mask and grouped advantages.
6 Experiments
Across six models and eight benchmarks, M2PO remains stable and competitive under extreme rollout staleness, while reducing unnecessary clipping and remaining robust to its threshold choice.
- 6.2 Performance Comparison on Training with Staleness: 36.6% accuracy with M2PO at s=256 exceeds GRPO at s=0, which reaches 33.0% on Qwen3-Base-1.7B.The authors attribute this comparison partly to M2PO’s lower clipping ratio under stale training.
- 6.2 Performance Comparison on Training with Staleness: M2PO with s=256 quickly catches up to the on-policy trajectory and converges faster than GRPO with the same staleness on Qwen-2.5-32B.Its reward trajectory similarly catches up and aligns closely with the s=0 trajectory, whereas GRPO with s=256 underperforms throughout.
- 6.2 Performance Comparison on Training with Staleness: Existing methods generally struggle at s=256: GSPO preserves stability but loses performance, while most other baselines break down early.These comparisons motivate a more robust solution for larger staleness.
- 6.3 Analysis and Ablation Study: 1.22% versus 0.06% average clipping on Qwen-2.5-32B shows that M2PO reduces clipping by over an order of magnitude under s=256.On Qwen3-Base-1.7B, the corresponding ratios are 0.66% for GRPO at s=256 and 0.02% for M2PO at s=256.
- 6.3 Analysis and Ablation Study: Accuracy remains stable across a broad range of τM2 values, declining only when the threshold is extremely small or large.The paper uses τM2 = 0.04 across its training settings.
7 Conclusion
The paper identifies prosperity before collapse as evidence that stale data can initially be highly informative but eventually destabilizes training, then proposes M2PO to retain that signal safely. M2PO constrains importance-weight second moments, matches on-policy performance under extreme staleness, and reduces clipping while remaining threshold-insensitive.
- Stale data can be as informative as on-policy trajectories, but training without a trust region eventually collapses because of instability.
- M2PO constrains the second moment of importance weights to suppress extreme outliers while preserving informative high-entropy tokens.This provides a variance-sensitive trust region for stable off-policy training.
- M2PO matches on-policy performance under extreme staleness, significantly reduces clipping, and is highly insensitive to its threshold.
Ethics Statement
The study develops reinforcement learning algorithms for LLMs using publicly available datasets without human subjects, personally identifiable information, or sensitive data. It limits the work to controlled academic settings while acknowledging potential misuse of more capable LLMs.
- The study uses no human subjects, personally identifiable information, or sensitive data, and relies on publicly available datasets.
- The research is confined to controlled academic settings to mitigate potential misuse of more capable LLMs.
The Use of Large Language Models (LLMs)
The authors used LLMs only for manuscript polishing and routine coding assistance, not for scientific ideation, algorithm design, analysis, or conclusions.
- LLMs assisted with grammar, readability, presentation, debugging syntax errors, and refactoring simple functions.
- LLMs were not used for research ideation, algorithm design, experimental analysis, or drawing conclusions.
- The appendix provides experimental details, theoretical proofs, and additional results, but does not identify LLM assistance beyond the stated uses.
B Detailed Experimental Setting
The experiments evaluate M2PO across multiple model and benchmark settings, with formal analysis bounding the divergence between new and behavior policies.
- Models & Datasets: M2PO is evaluated across six models, using DeepScaleR math data and eight mathematical reasoning benchmarks.The models range from 1.7B to 32B, with context lengths of 4k or 16k depending on the model series.
- Training: Training uses verl with vLLM rollouts, 256-sample batches, eight responses per prompt, and 1,000 training steps.The actor is optimized with FSDP, and the M2PO threshold is fixed at 0.04.
- Theoretical Proof: The theoretical analysis bounds chi-square divergence by the second moment of log importance ratios under a bounded-ratio assumption.The proof introduces z = log r and uses the bound |z| <= log R.
D Additional Experiments
Additional experiments examine which tokens standard clipping suppresses and test M2PO across staleness levels, finding stable performance even at extreme staleness.
- Commonly clipped tokens in GRPO: Standard epsilon-clipping frequently targets semantically and structurally critical high-entropy tokens in reasoning traces.Examples include tokens that initiate, connect, or conclude reasoning steps.
- Additional figures: The additional visualizations include a word cloud of frequently clipped tokens and a staleness-performance figure for Qwen2.5-Math-7B.The supplied descriptions identify the figures' topics but do not provide further numerical comparisons.
- M2PO under different staleness: At staleness s=256, M2PO remains on par with on-policy performance without collapse or degradation.Across s = 0, 32, 64, 128, and 256, larger staleness mainly slows the earliest accuracy gains.