RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning

• Rollout occupies 70-80% of on-policy RL time, and long-tailed response lengths leave GPUs idle behind stragglers.Source: §1 · p. 1
• Partial rollout reuses freed compute by pausing unfinished trajectories, improving throughput but risking stale-policy samples that may degrade accuracy.Source: §1 · p. 2
• Loss-level reweighting adjusts gradients after generation, leaving highly off-policy trajectories in the training data.Source: §1 · p. 2
• RollVerify uses OPS-guided sequence- and token-level verification to truncate and regenerate highly off-policy parts before training.Source: §1 · p. 2
• On DAPO-Math [33] with Qwen3-8B, RollVerify reduces training cost from 98.1 to 57.5 GPU days while preserving accuracy.Source: §1 · p. 2

• PPO [24] [21] became an early LLM post-training foundation, while GRPO [25] and its variants [33] [5] [35] avoid explicit value modeling and show strong empirical performance.Source: §2 · p. 2
• Because these methods rely on large rollout sets, rollout quality, diversity, and throughput are critical to scalable LLM reinforcement learning.Source: §2 · pp. 2–3
• Long-tail acceleration work either preserves on-policy training through pipeline or scheduling optimizations [30] [31] [38] [6] or uses asynchronous designs [4], which may degrade accuracy.Source: §2 · p. 3
• GSPO [35], SAPO [5], and VESPO [26] reweight off-policy samples during training, whereas RollVerify repairs trajectories before they enter training.Source: §2 · p. 3

3.1 Preliminary

• Partial rollout assigns freed engines to new tasks and pauses unfinished trajectories once a batch is ready; GRPO uses a clipped surrogate objective with an importance ratio.The ratio corrects mismatch between current and behavior policies, and clipping restricts updates when it deviates too far from 1.Source: §3.1 · p. 3

3.2 Measuring Off-Policy Deviation

• Off-Policy Shift measures trajectory deviation from the current policy using token-level importance-ratio offsets and their sequence-level average.A zero sequence-level shift indicates an on-policy trajectory; larger shifts indicate greater off-policy deviation.Source: §3.2 · pp. 3–4

3.3 Sample Filtering Analysis

• The filtering analysis discards samples that fail the chosen criterion, comparing staleness thresholds with sequence-level OPS thresholds.The comparison is reported for DAPO-Math with Qwen3-4B-Base and batch size 128.Source: §3.3 · p. 4, Table 1 · p. 4
• Staleness filtering still produces entropy explosion and accuracy degradation, while OPS remains high during training.Staleness is a coarse proxy for off-policy deviation; the reported comparison uses Table 1 and Figure 2.Source: §3.3 · p. 4, Figure 2 · p. 4
• OPS-based filtering substantially improves accuracy and stability, with stricter thresholds generally producing larger gains but reducing effective training throughput.The analysis compares filtering strategies under partial rollout on DAPO-Math with Qwen3-4B-Base and batch size 128.Source: §3.3 · p. 4, Table 1 · p. 4

4 RollVerify

• RollVerify retains the verified prefix and removes only the suffix that violates OPS, avoiding waste of partially valid rollouts while protecting accuracy.The approach is motivated by the trade-off between controlling off-policy degree and preserving training throughput.Source: §4 · pp. 4–5

4.1 Overall Procedure

• Each cycle trains on completed samples, then verifies cached unfinished trajectories, truncates off-policy suffixes, and resumes retained samples under the updated policy.Repeated verification keeps resumed samples nearly on-policy with respect to the latest policy.Source: §4.1 · p. 5

4.2 Two-Stage Verification

• Two-stage verification truncates at the first segment whose average OPS reaches ds, then scans retained tokens and stops at the first token whose OPS reaches dt.The procedure returns the verified prefix before the first token-level threshold violation.Source: §4.2 · pp. 5–6

4.3 Conditional Switching

• Acceptance rate is the fraction of partially generated tokens retained; when its recent average falls below a threshold such as 0.5, partial rollout switches to on-policy training.The implementation estimates the average with a sliding window of size 8.Source: §4.3 · pp. 6–7

5.2 Main Results

• 57.2 average score matches on-policy GRPO's 57.3 while reducing training cost from 98.1 to 57.5 GPU days, a 1.7× speedup; on the MoE model, RollVerify also matches on-policy performance with a 1.7× speedup.Naive partial rollout reaches 45.9 average accuracy on Qwen3-8B-Base and substantially degrades accuracy on the MoE model.Source: §5.2 · p. 8
• RollVerify keeps entropy, reward, and OPS stable for dense and MoE models, whereas naive partial rollout shows rising entropy and OPS alongside reward and response-length drops.The partial-rollout instability worsens over training and entropy eventually explodes.Source: §5.2 · p. 8

5.3 Ablation Study

• Sequence-level verification raises average score from 45.9 to 53.9, and adding token-level verification raises it to 57.4 while reducing training cost from 68.1 to 61.5 GPU days.Sequence-level checks can leave outlier tokens; token-level verification protects against highly off-policy tokens, lowering OPS and raising acceptance rate.Source: §5.3 · pp. 8–9, Table 3 · p. 8
• When acceptance falls to about 0.5, partial-rollout acceleration vanishes; conditional switching back to on-policy training reduces GPU days from 61.5 to 57.5.The acceptance-rate decline occurs in later training as more samples become highly off-policy.Source: Table 3 · p. 8, §5.3 · p. 9
• With sequence-level verification, smaller ds improves performance but can sharply lower acceptance and slow training; ds ≈0.01 provides a practical trade-off.Token-level verification improves accuracy and efficiency, results remain stable across a reasonably wide dt range, and dt = 2 gives the reported best overall trade-off.Source: §5.3 · p. 9, Table 4 · p. 9, Table 5 · p. 9

5.4 Analysis Experiments

• Against GSPO, SAPO, and VESPO, RollVerify outperforms naive partial rollout, matches GRPO with fewer GPU days, and has the lowest OPS among partial-rollout methods.The comparison distinguishes methods that change the training loss from methods that operate on rollout samples.Source: §5.4 · pp. 9–10, Table 6 · p. 9
• At 64K context, RollVerify reduces training cost from 261 to 137 GPU days while maintaining accuracy; the reported saving is smaller at 8K, from 16.6 to 15.1 GPU days.Longer contexts intensify the long-tail effect, while RollVerify's speedups grow with context length.Source: §5.4 · p. 10, Table 7 · p. 10
• Verification adds modest overhead, with two-stage checks requiring O(N) scans; across DeepScaleR, ReTool, and code generation, RollVerify beats partial rollout and has accuracy comparable to on-policy training.The code-generation results are preliminary evidence beyond mathematics.Source: §5.4 · p. 10, Table 8 · p. 10

6 Conclusion and Limitations

• RollVerify improves partial-rollout sample quality through sequence- and token-level verification while preserving on-policy performance and improving training efficiency.The paper reports effectiveness and robustness across different models, tasks, and context lengths.Source: §6 · p. 10
• Broader evaluation on non-math and agentic tasks, and validation in fully asynchronous RL training, remain future work.Source: §6 · p. 10

A Effect of Conditional Switching

• At step 700, RollVerify without switching records reward 0.659 and entropy 0.122 versus on-policy 0.665 and 0.113, with AIME24/25 scores of 38.7/32.5 versus 38.7/32.8, and reduces cost from 81.3 to 64.3 GPU days.With conditional switching, cost falls further to 60.2 GPU days and AIME24/25 scores are 38.8/32.8; switching is triggered at step 598.Source: A Effect of Conditional Switching · p. 17

B Generalization Experiments

• Across DeepScaleR mathematical reasoning, ReTool tool-assisted reasoning, and code generation, RollVerify outperforms partial rollout and achieves accuracy comparable to on-policy training while reducing cost relative to on-policy training.The code-generation results are preliminary evidence beyond mathematics.Source: B Generalization Experiments · p. 17, Table 10 · p. 17, Table 11 · p. 17

NeurIPS Paper Checklist

• The work is primarily empirical and uses heuristic approaches without fully stated theoretical assumptions or formal mathematical proofs.The checklist notes that a brief limitations discussion appears in the Conclusion.Source: NeurIPS Paper Checklist · pp. 18–19
• The code and data are not yet publicly available, and the supplemental material lacks sufficient instructions for faithful reproduction.The checklist says the authors plan to release the code later.Source: NeurIPS Paper Checklist · p. 20
• The checklist states that full experimental details and information about computational resources are provided in the main text and supplemental material.Source: NeurIPS Paper Checklist · pp. 20–21
• The paper reports benchmark accuracies and training costs but provides no error bars, confidence intervals, or statistical significance tests.Source: NeurIPS Paper Checklist · p. 20