PaperlayerView exact revision on arXiv

Source-linked AI summary

A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization

Prasanth YSS, Zhichen Ren, Rasa Hosseinzadeh, Ilan Gofman, Yuqi Chen, Zhaoyan Liu, Guangwei Yu, Jesse C. Cresswell, Satya Krishna Gorti

arXiv:2606.16154v1cs.LG

TL;DR

GRPO-style RLVR optimization can collapse, motivating a clearer account of how token-level updates affect stability. The paper develops a gradient-based taxonomy and WAPO, which updates only positive-advantage completions; across math and multi-hop QA, WAPO improves stability and remains competitive or outperforms baselines.

  • Problem

    Off-policy GRPO-style RLVR optimization remains fragile and susceptible to training collapse.

  • Method

    The paper derives a peak–valley taxonomy of token-level gradient effects and proposes WAPO, which removes non-positive advantage updates.

  • Results

    Across mathematical reasoning and multi-hop QA benchmarks, WAPO improves stability, preserves pass@k, and matches or outperforms RLVR baselines across model families.

  • Takeaways & Limitations

    WAPO remains stable and competitive when baselines collapse or saturate early, while remaining competitive where they are stable.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but GRPO-style optimization remains prone to collapse. We analyse this instability through token-level gradient dynamics, deriving a taxonomy that predicts how updates affect next-token probabilities and entropy. The taxonomy shows that stability depends jointly on the advantage sign and token distribution under the current policy. Motivated by this finding, we propose Winner Advantage Policy Optimization (WAPO), a simple online clipped policy-gradient objective that updates only on positive-advantage completions. Across mathematical reasoning and multi-hop QA benchmarks, WAPO improves training stability and matches or outperforms baselines across multiple model families. Full code can be found at https://github.com/layer6ai-labs/wapo.

1 Introduction

The paper reframes RLVR instability through local gradient effects on token probabilities and entropy, rather than policy drift alone. This analysis motivates WAPO, an online clipped policy-gradient method that updates only positive-advantage completions and matches or improves established baselines across reasoning and multi-hop QA settings.

  • Motivation: GRPO-style RLVR reuses rollouts from an old policy for multiple gradient updates, creating an off-policy setting associated with training instability.Prior work commonly attributes collapse to old–new policy drift and training–inference engine mismatch.
  • Motivation: Removing more divergent tokens does not consistently improve stability and can itself cause collapse.This motivates analyzing how individual gradient updates alter the current policy’s local distribution instead of assuming divergence is uniformly harmful.
  • Gradient taxonomy: The proposed taxonomy predicts how policy updates affect probability and entropy, distinguishing prior-aligned tokens that reduce entropy from prior-opposed tokens that increase it.Exploration-like updates and prior-aligning tokens in negative rollouts can produce collapsed generations, while selecting useful negative-advantage tokens is difficult under coarse rewards.
  • WAPO: WAPO is a minimal GRPO modification that masks policy-gradient terms from non-winning completions while retaining policy ratios and group-normalized advantages.Prompts without a positive-advantage completion contribute no policy-gradient; in the binary-reward analysis, its adaptive factor is 1 − qx, weakening updates as success probability saturates.
  • Empirical evaluation: Across NuminaMath-LEAN, Math-500, Hotpot-QA, and OTT-QA, WAPO matches or improves over GRPO, DAPO, and GSPO across Qwen3-4B, SmolLM3-3B, and Gemma3-4B.The evaluation covers mathematical reasoning and multi-hop question answering.

2 Background

GRPO-style RLVR uses grouped rollouts, group-relative advantages, and clipped importance-ratio updates, but off-policy optimization with coarse rewards remains vulnerable to collapse. Prior work links instability to policy mismatch, harmful low-probability-token gradients, and distributional squeezing, motivating a token-gradient taxonomy based on advantage sign and current token distributions.

  • GRPO-style RLVR: GRPO-style objectives sample grouped completions, convert rewards into group-relative advantages, and apply sequence-level advantages through clipped importance-ratio updates, optionally with KL regularization.DAPO modifies length normalization and clipping, while Dr. GRPO analyzes reward and length-normalization biases.
  • GRPO-style RLVR: Off-policy optimization under coarse rewards remains prone to collapse despite refinements such as DAPO and Dr. GRPO.These variants refine the GRPO framework through clipping, length normalization, and bias analysis.
  • Policy mismatch and trust-regions: Instability is commonly attributed to mismatch between the rollout policy πθold and current policy πθ, motivating trust-region and train-inference-mismatch remedies.GSPO addresses token-level importance sampling with sequence-level clipping; other work reports engineering fixes including FP16 precision alignment.
  • Gradient analysis of training dynamics: Failed low-likelihood trajectories can induce likelihood displacement and collapse, while DPO-style negative updates can reduce every response likelihood and create a degenerate peaked distribution.The latter phenomenon is described as a “squeezing effect,” extending beyond penalized responses.
  • Motivation: These findings motivate analyzing how advantage sign interacts with current token distributions and using a unified taxonomy to motivate WAPO’s positive-only update.The taxonomy is introduced in Section 3 under a token-gradient view of positive- and negative-advantage updates.

3 A Gradient-Motivated Token Taxonomy

This section shows that clipping divergence alone does not ensure stability and develops a token-level taxonomy based on advantage sign and sampled-token probability. The resulting four regimes explain how updates alter token probabilities and entropy.

  • Gradient-motivated motivation: Lowering the negative clipping threshold removes divergent updates but causes training collapse, while ratio clipping adversely affects lower-probability tokens.The spread of importance ratios is larger at lower sampled probabilities, so clipping can harm those tokens.
  • Gradient-motivated token taxonomy: For positive-advantage samples, non-sampled-token probability increases when ps+pi < C(p) and decreases when ps+pi > C(p), with reversed inequalities for negative advantages.The adaptive reference C(p) determines whether a local update redistributes probability toward or away from each non-sampled token.
  • Gradient-motivated token taxonomy: Combining advantage sign with peak-versus-valley status yields four regimes: Pos-peak, Pos-valley, Neg-peak, and Neg-valley.Peak/valley status is conditioned on the sampled token and adapts to the current next-token distribution rather than using a fixed probability cutoff.
  • Entropy implications: Pos-valley updates increase entropy, Neg-valley updates decrease it, Pos-peak updates usually decrease it, and Neg-peak updates tend to increase it.These first-order entropy predictions follow from the sign of the entropy-change factors for valley and peak tokens.

4 Diagnosing Token-Level Instability in RLVR

Token-level instability depends jointly on advantage sign and token probability: Pos-peak is stable but conservative, while entropy-increasing regimes and Neg-valley can produce collapse. Advantage-sign filtering emerges as the most stable coarse intervention, with Pos training matching DAPO without an added regularizer or trust-region threshold.

  • Token-level regimes: Pos-peak is stable but quickly plateaus, reinforcing likely continuations, reducing entropy, and providing limited pressure to explore alternative solution paths.The quadrant experiments characterize Pos-peak as stable but conservative.
  • Token-level regimes: Neg-valley often improves reward early but eventually induces low-entropy collapse, yielding confident, repetitive, or generic generations.Negative updates on low-probability tokens can suppress spurious continuations, but in high-entropy contexts they may prematurely concentrate probability mass.
  • Token-level regimes: Neg-peak and Pos-valley increase entropy and are prone to random collapse by shifting probability mass toward non-sampled alternatives.Figure 4 validates that both regimes increase entropy, while entropy-increasing groups collapse rapidly across hyperparameters.
  • Masking ablations: Peak training collapses rapidly because Neg-peak suppresses high-probability sampled tokens and sharply redistributes mass to alternatives.The observation suggests that Neg-peak dominates despite the stable Pos-peak component.
  • Masking ablations: Valley training is more stable and resembles high-entropy filtering, but it underperforms Pos and lenient-clipping baselines because valley tokens are sampled less often.Valley tokens are also sensitive to ratio-based clipping.
  • Advantage-sign filtering: Pos training performs on par with DAPO, and restricting updates to positive advantages adds neither a regularizer nor a trust-region threshold.The passage identifies advantage sign as the most stable coarse filter and says rewarded rollouts already contain sufficient learning signal.

5 Winner Advantage Policy Optimization

WAPO removes non-positive advantage terms from an online GRPO-style policy-gradient update, retaining sampled rollouts, group-normalized advantages, importance ratios, and clipping. In binary-reward settings, its positive-only update follows the binary policy-gradient direction while adaptively emphasizing harder prompts.

  • Motivation: WAPO removes non-positive advantage terms, retaining positive-advantage updates that reinforce successful high- and low-probability tokens.The method is motivated by exploitation from Pos-peak updates and rewarded exploration from Pos-valley updates.
  • Method: WAPO remains an online GRPO-style policy-gradient method using sampled rollouts, group-normalized advantages, token-level importance ratios, and clipping.
  • Binary-reward intuition: The pos-only group-normalized update ascends the same direction as the binary policy gradient, with an adaptive factor 1 − q_x.
  • Binary-reward intuition: Adaptive factors weight prompts by current-policy success, emphasizing harder prompts while attenuating easier ones.The bounded 1 − q_x factor is WAPO’s default because an alternative factor can overemphasize rare or noisy successes when q_x is small.
  • WAPO objective: For continuous rewards, WAPO retains only completions with positive group-normalized advantage and applies uniform maximum-length normalization.Because only positive advantages remain, the lower clipping bound is inactive for retained terms.

6 Results

Across mathematical reasoning and multi-hop QA benchmarks, WAPO is stable across model families, delivers especially strong multi-hop QA gains, and preserves exploratory behavior and out-of-domain transfer. It remains competitive on math tasks while avoiding the collapses observed with DAPO and some GRPO settings.

  • Evaluation setup: The evaluation covers Math-500 and NuminaMath-LEAN for mathematical reasoning, plus Hotpot-QA and OTT-QA for multi-hop QA, across Qwen3-4B, SmolLM3-3B, and Gemma3-4B.Math tasks use direct final-answer generation, whereas multi-hop QA involves interaction with a search environment.
  • Training stability: WAPO remains stable across all evaluated datasets and model families, while DAPO collapses in several settings and GRPO collapses on NuminaMath-LEAN with Gemma3-4B.GRPO and GSPO are generally more stable than DAPO but often saturate early, especially on multi-hop QA.
  • Training stability: WAPO’s removal of non-positive-advantage updates reduces a major source of RLVR instability without hurting learning capability.This design is associated with WAPO’s stable training behavior across tasks and model families.
  • In-domain performance: 9.9% and 3.2% are WAPO’s OTT-QA gains over the next best stable baseline for Qwen3-4B and Gemma3-4B, respectively.On Hotpot-QA, the corresponding margins are 4.5% and 10.6%; WAPO nearly matches other baselines on Math-500 and NuminaMath-LEAN.
  • Exploration: WAPO achieves the best pass@1 and strongest pass@k performance across all three model families on Hotpot-QA and OTT-QA.It also maintains competitive higher-pass@k performance on Math-500 and NuminaMath-LEAN.
  • Out-of-domain transfer: WAPO achieves the best Hotpot-QA→2-wiki performance across all three model families and remains competitive on NuminaMath-LEAN→AIME’25.The out-of-domain pass@k results show a similar trend to the in-domain setting.

7 Conclusion

The paper analyzes RLVR instability through token-level gradients and introduces a peak–valley taxonomy linking advantage-weighted updates to stability and exploration. It then proposes WAPO, a minimal GRPO-style objective that removes non-positive advantage updates and improves stability across reasoning benchmarks.

  • The work presents a token-level gradient analysis of RLVR instability.
  • The peak–valley taxonomy explains how advantage-weighted updates shape stability and exploration.
  • WAPO is a minimal GRPO-style objective that removes non-positive advantage updates while retaining online rollouts, group-normalized advantages, importance ratios, and clipping.
  • Across mathematical reasoning and multi-hop QA benchmarks, WAPO improves training stability, preserves pass@k, and outperforms strong RLVR baselines.

A Additional Collapse Examples … B.2 Extremal Tokens and the Reference Level

Additional collapse experiments show that aggressive clipping does not ensure RLVR stability, while the proofs establish first-order probability changes and characterize extremal tokens relative to C(p).

  • A Additional Collapse Examples: More aggressive clipping suppresses a larger fraction of divergent tokens but does not improve stability in Qwen3-4B with GRPO or Gemma3-4B with DAPO.This indicates RLVR instability is not explained solely by the trust region.
  • A Additional Collapse Examples: The additional collapse examples use Qwen3-4B trained with GRPO and Gemma3-4B trained with DAPO.
  • B.1 First-Order Gradient Prediction: The gradient derivation starts from logits z, sampled-token probability ps, non-sampled probability pi, and advantage-weighted negative log-likelihood ℓs(z) = −A log ps.
  • B Proofs for the Gradient-Motivated Taxonomy: A small gradient-descent step with step size η > 0 changes the probability of a non-sampled token through the softmax gradient and a first-order Taylor expansion.
  • B.2 Extremal Tokens and the Reference Level: The reference level C(p) bounds the token distribution’s extrema: pmax ≥ C(p) and pmin ≤ C(p), with equality for pmin only under a uniform distribution.
  • B.2 Extremal Tokens and the Reference Level: Any token attaining maximum probability is a peak, whereas a minimum-probability token is always a valley unless the distribution is perfectly uniform.

B.3 First-Order Entropy Direction

The section derives a first-order entropy-change expression from token-probability updates and establishes how its bracketed term behaves for valley and maximum-probability peak tokens. It also notes rare non-global-maximum exceptions while confirming the expected entropy trends empirically.

  • Derivation: The first-order Taylor expansion substitutes the general token update into entropy change, using probability-mass conservation and −∑_k p_k log p_k = H(p).The update is Δp_k = ηA p_k(C(p) − p_s − p_k + δ_sk).
  • Valley tokens: For valley tokens with p_s < C(p), strict convexity and non-positive endpoint values imply B(p_s) ≤ 0.The proof establishes B(x) ≤ 0 throughout x ∈ [0, C(p)].
  • Peak tokens: For maximum-probability peak tokens, non-negative weights and −log p_k ≥ −log p_s imply B(p_s) ≥ 0.This conclusion applies when p_s = max_k p_k and C(p) ≤ p_s.
  • Exceptions and empirical trend: Rare non-global-maximum peaks can violate the expected sign: with p = (0.14, 0.46, 0.40), p_s = 0.40, C(p) ≈ 0.391, and B(p_s) ≈ −0.0083.Despite this corner case, experiments follow the expected trend that Pos-peak tokens decrease entropy while Neg-peak tokens increase entropy.

C Additional Token Dynamics Plots

Additional entropy and reward plots from NuminaMath-LEAN runs on Qwen3-4B confirm the entropy-direction predictions from Section 3. Pos-valley and neg-peak updates increase entropy, whereas pos-peak and neg-valley updates decrease it.

  • Additional entropy and reward plots are provided for NuminaMath-LEAN runs on Qwen3-4B.
  • The plots confirm Section 3’s predictions about update-induced entropy directions.
  • Pos-valley and neg-peak updates increase entropy, while pos-peak and neg-valley updates decrease entropy.

D WAPO Gradient Analysis … F.1.1 Math-500 dataset.

The paper derives WAPO’s gradient from binary-reward policy gradients, contrasts its baselines, and specifies mathematical-reasoning experiments using strict format-and-answer rewards and an augmented Math-500 split.

  • D WAPO Gradient Analysis: With binary rewards, the probability of a correct answer equals the expected reward over generated sequences.This identity underpins the policy-gradient derivation for WAPO.
  • D WAPO Gradient Analysis: The exact gradient expands sequence log-probabilities into sums of token-level conditional log-probability gradients.GRPO approximates this expectation using G independent rollouts and their empirical mean.
  • D WAPO Gradient Analysis: Positive-only updates mask advantages by reward, and under the approximation r̄ ≈ q, the resulting objective depends on the normalizer D.The derivation evaluates the cases D = G and D = G · r̄, confirming the corresponding table rows.
  • E Baseline Formulations: The baselines define token-level importance ratios as ρit(θ) = πθ(yi,t | x, yi,<t) / πθold(yi,t | x, yi,<t).GRPO and GSPO use group-normalized advantages, while GSPO uses a sequence-level geometric-mean ratio.
  • E Baseline Formulations: DAPO uses mean-centered, non-variance-normalized advantages with token-level ratios and asymmetric clipping.This distinguishes DAPO’s formulation from the other reviewed baselines.
  • F Experiment Details: Experiments run on two NVIDIA A6000 GPUs, assigning one GPU to vllm rollout generation and the other to policy updates.This passage specifies the experimental hardware allocation.
  • F.1 Math Environment: Mathematical responses must place reasoning inside <think>...</think> tags and provide a final answer in the form Answer: <your answer>.The parser extracts the text following Answer:, and the binary reward is 1 only when formatting and exact answer correctness both hold.
  • F.1.1 Math-500 dataset.: Math-500 uses the PRM800K-augmented split, with 12,000 training problems and 500 held-out test problems.The augmentation adds 4,500 problems from the original Math-500 test split to reduce overfitting to the original 7,500-problem training set.

F.1.2 NuminaMath-LEAN dataset. … F.4 Supplementary Experiments

The supplementary sections define filtered math and multi-hop QA datasets, retrieval-based environments, shared training settings, and comparative results. WAPO is strongest on multi-hop QA, avoids DAPO’s frequent collapse, and remains competitive on math benchmarks.

  • F.1.2 NuminaMath-LEAN dataset.: 21,251 examples remain after filtering NuminaMath-LEAN, with 18,273 math-word-problem and 2,978 multiple-choice-question examples.The first 1,000 shuffled examples are evaluation data, and 20,251 are training data.
  • F.2 Multi-hop QA Environment: Multi-hop QA uses Search-R1-style interaction, requiring reasoning in <think>, searches in <search>, and final answers in <answer>, with max_turns set to 10.The environment includes retrieval evidence and a format reward alongside normalized-answer F1.
  • F.2.1 OTT-QA: OTT-QA uses 20,000 samples for SFT trajectories and a disjoint 10,000-sample subset for RL, retaining questions with exploration-supporting rollouts.Six rollouts were generated per question on a Qwen-SFT model before selection.
  • F.2.1 OTT-QA: The OTT-QA retriever encodes unchunked Wikipedia passages and serialized tables with the BAAI/bgem3 embedding model.Wikipedia passages use normalized titles plus text, while tables include title, section title, introduction, and markdown content.
  • F.2.2 Hotpot-QA & 2wiki: Hotpot-QA uses 10,000 samples for SFT cold-start data and evaluates checkpoints on a held-out 1000-sample test set.Out-of-distribution evaluation uses a 1000-sample subset of the 2wiki test dataset.
  • F.2.2 Hotpot-QA & 2wiki: HotpotQA and 2wiki use a wiki-18 retriever with an E5 embedding model following Search-R1 instructions.The same retrieval setup supports their training and evaluation.
  • F.3 Hyperparameters: The experiments use shared settings within each task family and a larger learning rate of 1 × 10−5 for LoRA RL training.The learning-rate choice follows the claim that LoRA RL training needs 10× the full-fine-tuning rate.
  • F.4 Supplementary Experiments: WAPO achieves the strongest multi-hop QA performance across model families, whereas DAPO collapses in 4 out of 6 runs and within 100 steps on OTT-QA with Qwen3-4B.On math benchmarks, WAPO is competitive, sometimes trailing GRPO or GSPO, but consistently outperforms DAPO.
Loading 2606.16154v1…