A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization

1 Introduction

• Removing more divergent tokens does not reliably improve stability and can cause collapse, motivating analysis of how updates alter the current policy’s local distribution rather than modeling old-policy drift.This offers an alternative lens for understanding instability when off-policy updates are unavoidable.Source: §1 · p. 1
• The proposed taxonomy predicts that exploration-like updates increase entropy and lead to collapsed generations, while prior-aligning tokens in negative rollouts can also cause collapse.It distinguishes prior-aligned tokens, which reduce entropy, from prior-opposed tokens, which increase it; selecting useful negative-advantage tokens under coarse rewards is outside its scope.Source: §1 · pp. 1–2
• WAPO is an online clipped policy-gradient modification to GRPO that masks non-winning completions, uses group-normalized advantages, and gives no policy-gradient contribution to prompts without winners.Prompts with at least one positive-advantage completion are normalized over the group; unlike rejection fine-tuning or simple filtering, WAPO retains policy ratios.Source: §1 · p. 2
• In an idealized binary-reward setting, WAPO’s adaptive factor is 1 − q_x, reducing update strength as prompt success probability saturates while preserving reinforcement from sampled successes.The analyzed normalization choices over positive rollouts ascend the same success probability q_x, with adaptive factors depending on q_x.Source: §1 · p. 2
• Across mathematical reasoning and multi-hop QA settings, WAPO matches or improves on GRPO (Shao et al., 2024), DAPO (Yu et al., 2025), and GSPO (Zheng et al., 2025b) in final accuracy and pass@k.Gains are clearest when baselines collapse; when they remain stable, WAPO performs on par. The paper also evaluates OOD generalization on 2-wiki and AIME’25.Source: §1 · p. 2

Background

• GRPO-style RLVR uses group-relative sequence advantages in clipped updates; DAPO (Yu et al., 2025) and Dr. GRPO (Liu et al., 2025) refine it, yet collapse persists under coarse rewards.The framework can optionally include KL regularization to πθold (Shao et al., 2024).Source: §2 · p. 2
• Prior work attributes instability to policy mismatch: GSPO (Zheng et al., 2025b) and Qi et al. (2026) refine trust regions, while engineering fixes address train-inference mismatch.The cited fixes include FP16 precision alignment (Qi et al., 2025), adaptive learning-rate scheduling (Zhang et al., 2026), and router synchronization (Ma et al., 2025; Ye et al., 2026).Source: §2 · p. 2
• These works motivate a token-gradient taxonomy unifying positive- and negative-advantage updates and WAPO’s positive-only update.The analysis focuses on how advantage sign interacts with the current token distribution during GRPO-style training.Source: §2 · p. 2

3 A Gradient-Motivated Token Taxonomy

• Lowering DAPO’s negative clipping threshold removes divergent updates but training still collapses, and ratio-based clipping adversely affects low-probability tokens without improving stability or reducing entropy.The importance-ratio spread is higher at lower sampled probabilities.Source: §3 · pp. 2–3, Figure 2 · p. 3
• A non-sampled token’s probability rises when its probability sum with the sampled token is below C(p) for positive advantage, with the inequality reversed for negative advantage.Thus, reinforcing a low-probability successful token can raise other low-probability alternatives and broaden the local distribution.Source: §3 · p. 3
• The taxonomy classifies sampled tokens as peak or valley relative to adaptive C(p), then combines that class with advantage sign into Pos-peak, Pos-valley, Neg-peak, and Neg-valley regimes.Unlike entropy, peak/valley status depends on the sampled token; C(p) adapts to distribution sharpness rather than imposing a fixed probability cutoff.Source: §3 · p. 3
• Maximum-probability tokens are peaks, while minimum-probability tokens are valleys except when the distribution is uniform on its support; advantage sign determines whether the update reinforces or suppresses that region.Nearly every minimum-probability token is a valley.Source: §3 · p. 3
• Pos-valley and Neg-peak updates tend to increase entropy, whereas Neg-valley and usually Pos-peak updates decrease it; across three models, valleys are sampled less often but have higher entropy.The three models are SmolLM3-3B, Qwen3-4B, and Gemma3-4B.Source: §3 · p. 4, Figure 3 · p. 4

Isolating the four token regimes

• Negative updates on low-probability tokens can suppress spurious continuations, but in high-entropy contexts they may concentrate probability prematurely, producing confident, repetitive, or generic generations.This is a low-entropy collapse mode, distinct from the random drift caused by entropy-increasing regimes.Source: §4 · p. 4

Taxonomy-based masking

• Pos, Neg, Peak, and Valley masking each combine an entropy-increasing and entropy-decreasing regime, retaining exploration and exploitation pressure; their group dynamics depend on the model’s prior distribution for the dataset.Pos-peak alone is stable but does not reach optimal rewards, motivating use of exploratory quadrants.Source: §4 · p. 4
• Peak training collapses rapidly despite including Pos-peak; Neg training learns quickly then collapses, while Valley is more stable but underperforms Pos and lenient-clipping baselines.Neg-peak appears in both Peak and Neg masks; Valley tokens are sampled less often and are aggressively removed by ratio-based clipping.Source: §4 · pp. 4–5

Isolating the four token regimes

• Pos-peak is stable but conservative, Neg-peak and Pos-valley trigger random collapse, and Neg-valley can improve reward early before overconfident collapse.The quadrant experiments match the taxonomy’s entropy predictions: Pos-peak and Neg-valley decrease entropy, while Neg-peak and Pos-valley increase it.Source: §4 · p. 4, Figure 4 · p. 5

Taxonomy-based masking

• Pos training performs on par with DAPO, suggesting rewarded rollouts already provide sufficient learning signal; restricting advantage signs adds no regularizer or trust-region threshold.The intervention simply limits which advantage signs can update the model.Source: §4 · p. 5

• WAPO removes non-positive advantage terms because positive-advantage updates reinforce successful high-probability tokens and explore successful low-probability tokens.Source: §5 · p. 5
• WAPO remains online GRPO-style: it uses sampled rollouts, group-normalized advantages, token-level importance ratios, and clipping, while retaining only positive-advantage completions.For continuous rewards, this selects completions scoring above the group baseline.Source: §5 · pp. 5–6
• For binary rewards, the positive-only group update follows the policy-gradient direction scaled by 1-qx; WAPO defaults to this bounded factor rather than the more aggressive (1-qx)/qx weighting.Both factors upweight harder prompts, but the alternative can overemphasize rare or noisy successes when qx is small.Source: §5 · p. 6
• Both WAPO normalizations outperform PSR and RAFT++ on NuminaMath-LEAN with SmolLM3-3B using exact-match binary reward.Figure 5 describes RAFT++ as exhibiting short-answer bias and PSR as plateauing without importance ratios.Source: §5 · p. 6, Figure 5 · p. 6
• Unlike PSR and RAFT++, WAPO preserves clipped GRPO-style objectives and mean-centered group-relative advantages, avoiding per-sequence normalization that can bias training toward terse templates.PSR lacks a clipped group-relative policy-gradient objective, while RAFT++ uses per-sequence normalization without mean-centered group-relative advantages.Source: §5 · p. 6

• The evaluation covers Math-500 and NuminaMath-LEAN for mathematical reasoning, Hotpot-QA and OTT-QA for multi-hop QA, and Qwen3-4B, SmolLM3-3B, and Gemma3-4B against RLVR baselines.It measures exact-match accuracy, stability, pass@k, and out-of-domain generalization; QA models interact with a search environment before answering.Source: §6 · pp. 6–7
• WAPO remains stable and continues improving across evaluated settings, while DAPO collapses in several cases and GRPO collapses on Gemma3-4B for NuminaMath-LEAN.The results suggest that removing non-positive-advantage updates reduces a major source of instability without hurting learning capability.Source: §6 · pp. 7–8, Figure 6 · p. 7
• WAPO beats the next-best stable baseline by 9.9% (Qwen3-4B) and 3.2% (Gemma3-4B) on OTT-QA, and by 4.5% and 10.6%, respectively, on Hotpot-QA.On Math-500 and NuminaMath-LEAN, WAPO nearly matches other baselines.Source: §6 · p. 8
• On NuminaMath-LEAN, SmolLM3-3B and Gemma3-4B initially trail baselines but close the gap later; WAPO may lag early because its adaptive factor emphasizes lower-success prompts and yields more conservative updates.WAPO appears most beneficial on harder datasets, performing best on OTT-QA and remaining largely comparable to strong baselines on Math-500.Source: §6 · p. 8

WAPO preserves exploration

• WAPO achieves the best pass@1 and strongest pass@k across all three model families on Hotpot-QA and OTT-QA, while retaining competitive higher-pass@k performance on both math datasets.On NuminaMath-LEAN with SmolLM3-3B, WAPO reaches 81.02% pass@16, matching GSPO’s 81.0%.Source: Figure 7 · p. 7, §6 · p. 8

WAPO shows robust out-of-domain transfer

• On Hotpot-QA→2-wiki, WAPO performs best across all three model families; on NuminaMath-LEAN→AIME’25, it remains competitive and increasingly so at larger k, especially for Qwen3-4B and SmolLM3-3B.At low k on NuminaMath-LEAN→AIME’25, WAPO is not always strongest.Source: §6 · p. 8, Table 1 · p. 8, Table 2 · p. 8

7 Conclusion

• WAPO improves training stability, preserves pass@k, and outperforms strong RLVR baselines where they collapse or saturate early, while remaining competitive when they are stable.The reported scope is mathematical reasoning and multi-hop QA benchmarks.Source: §7 · p. 8
• The paper introduces a peak–valley taxonomy for understanding how advantage-weighted updates shape RLVR stability and exploration.Its analysis is at the token-gradient level.Source: §7 · p. 8
• WAPO removes non-positive-advantage updates while retaining online rollouts, group-normalized advantages, importance ratios, and clipping.The paper describes it as a minimal GRPO-style objective.Source: §7 · p. 8
• Future work includes additional verifiable domains such as text-to-SQL (Pourreza and Rafiei, 2024; Gorti et al., 2025) and code generation (Le et al., 2022; Chen et al., 2022), larger models, and MoE architectures.Source: §7 · p. 8

A Additional Collapse Examples

• In Qwen3-4B with GRPO and Gemma3-4B with DAPO, more aggressive clipping suppresses more divergent tokens but does not improve stability, so instability is not explained solely by the trust region.The passage compares Qwen3-4B trained with GRPO (Yang et al., 2025; Deng et al., 2025) and Gemma3-4B trained with DAPO (Kamath et al., 2025; Yu et al., 2025).Source: A Additional Collapse Examples · p. 12

B.1 First-Order Gradient Prediction

• A small gradient-descent step on the advantage-weighted negative log-likelihood changes the probability of a non-sampled token, as derived in the appendix.The derivation uses a first-order Taylor expansion of the softmax probability and its Jacobian.Source: §B.1 · p. 12
• The appendix derives the non-sampled-token probability change by substituting the logit update into the Taylor expansion and simplifying sums that include C(p), the sum of squared probabilities.The supplied passages describe the derivation steps but do not show the final equation.Source: §B.1 · pp. 12–13

B.2 Extremal Tokens and the Reference Level

• For a valid probability distribution, pmax ≥ C(p) and pmin ≤ C(p), with equality in the minimum bound only for a uniform distribution.Here C(p) is the sum of squared probabilities.Source: §B.2 · p. 13
• Any token attaining the maximum probability is a peak because its probability is at least C(p).Source: §B.2 · p. 13
• A minimum-probability token is a valley unless the distribution is perfectly uniform.Source: §B.2 · p. 13

B.3 First-Order Entropy Direction

• The derivation linearizes next-token entropy, uses probability-mass conservation, and substitutes the general token update to obtain a bracketed expression for the change.The sign analysis then examines that bracket as a function of the sampled token probability.Source: §B.3 · p. 13
• For valley tokens, B(ps) ≤ 0; positive-advantage valleys increase entropy, whereas negative-advantage valleys decrease it.The proof bounds B(x) above by zero across x ∈ [0, C(p)].Source: §B.3 · p. 14, Figure 9 · p. 15
• For maximum-probability tokens, B(ps) ≥ 0; positive-advantage peaks decrease entropy, whereas negative-advantage peaks increase it.The proof uses non-negative weights summing to ps and the fact that −log pk ≥ −log ps for every token k.Source: §B.3 · p. 14, Figure 9 · p. 15
• A peak token that is not the global maximum can have B(ps) < 0, so peak status alone does not guarantee the maximum-token sign result; such cases are described as rare.The experiments are reported to follow the expected trend for positive- and negative-advantage peak tokens.Source: §B.3 · pp. 14–15

C Additional Token Dynamics Plots

• For NuminaMath-LEAN runs on Qwen3-4B, Figure 9 shows pos-valley and neg-peak updates increase entropy, while pos-peak and neg-valley updates decrease it.The passage describes these as confirming Section 3's entropy direction predictions.Source: C Additional Token Dynamics Plots · p. 15

D WAPO Gradient Analysis

• With binary rewards, the success probability equals expected reward, and its policy gradient is expressed as a sum of token-level log-probability gradients estimated from sampled rollouts.The derivation uses the log-derivative trick and Monte Carlo sampling over G rollouts.Source: D WAPO Gradient Analysis · pp. 15–16
• For binary rewards, positive-only GRPO-style updates mask advantages to successful trajectories and approximate the success-probability gradient when the group mean reward approximates q.The analysis evaluates the resulting gradient for normalizers D = G and D = G · r̄.Source: D WAPO Gradient Analysis · p. 16

E Baseline Formulations

• GRPO uses token-level clipped ratios, GSPO uses a geometric-mean sequence ratio, and DAPO uses mean-centered, unnormalized advantages with asymmetric clipping.GRPO and GSPO use group-normalized advantages.Source: E Baseline Formulations · p. 17

F Experiment Details

• Experiments use two NVIDIA A6000 GPUs, assigning one to vLLM rollout generation and one to policy updates.Source: F Experiment Details · p. 17

F.1 Math Environment

• Math responses receive reward 1 only when they follow the required format and the extracted final answer exactly matches the ground truth; otherwise, reward is 0.The parser extracts the final answer after “Answer:”.Source: §F.1 · p. 17

F.1.1 Math-500 dataset.

• Math-500 experiments use the augmented PRM800K split with 12,000 training problems and 500 held-out test problems, using only problem statements and final answers.The split adds 4,500 original test problems to the original 7,500-problem training set to reduce overfitting.Source: §F.1.1 · p. 18

F.1.2 NuminaMath-LEAN dataset.

• 21,251 NuminaMath-LEAN examples remain after filtering; the fixed-seed split uses 1,000 for evaluation and 20,251 for training.The filtering keeps math-wordproblem and multiple-choice examples with nonempty problem and answer fields and simple numeric answers; the retained set contains 18,273 math-wordproblem and 2,978 multiple-choice examples.Source: §F.1.2 · p. 18

F.2 Multi-hop QA Environment

• Multi-hop QA uses multi-turn retrieval: models reason in <think>, issue searches in <search>, and provide final responses in <answer>, with max_turns set to 10.The final reward combines answer F1 and a format reward that checks structured search or answer output.Source: §F.2 · p. 18

F.2.1 OTT-QA

• They generated SFT trajectories from OTT-QA training data, then built RL data from a disjoint 10,000-sample subset, retaining questions with six-rollout Qwen-SFT accuracy between 0% and 50%, exclusive.Checkpoints were evaluated on 1,000 held-out samples. The retriever encoded unchunked tables and passages with BAAI/bgem3 (Chen et al., 2024).Source: §F.2.1 · p. 19

F.2.2 Hotpot-QA & 2wiki

• 10,000 HotpotQ samples form the SFT coldstart set, with the remaining data used for RL; checkpoints are evaluated on 1,000 Hotpot-QA test samples and 1,000 2wiki OOD samples.Training and evaluation use the wiki-18 corpus with an E5 embedding model (Wang et al., 2022).Source: §F.2.2 · p. 19

F.3 Hyperparameters

• The experiments use a 1 × 10−5 learning rate across checkpoints within each task family, justified as 10× the full-fine-tuning rate for LoRA RL.Table 3 summarizes training hyperparameters for math and multi-hop QA.Source: §F.3 · p. 19, Table 3 · p. 19

F.4 Supplementary Experiments

• WAPO’s MATH-500 and NuminaMath-LEAN pass@k curves show competitive performance in most settings.Figure 10 omits the base model because its substantially lower performance would compress the y-axis and obscure differences among trained baselines.Source: §F.4 · p. 19, Figure 10 · p. 20
• WAPO achieves the strongest multi-hop QA performance across model families; DAPO collapses in 4 of 6 runs, while WAPO is competitive but not uniformly best on math.DAPO collapses within 100 steps on OTT-QA with Qwen3-4B, so its score is not reported; overall, WAPO offers the strongest stability-performance tradeoff across tasks and model families.Source: §F.4 · p. 19, Table 4 · p. 20