PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning

Junkang Wu, Kexin Huang, Jiancan Wu, An Zhang, Xiang Wang, Xiangnan He

arXiv:2509.22611v2cs.LGcs.AI

TL;DR

RLVR training faces both entropy collapse and entropy explosion, while prior work has largely focused on collapse. QAE replaces the mean baseline with a group-wise quantile baseline and a response-level gate; the paper reports stabilized entropy, sparse credit assignment, and sustained pass@1 gains across reasoning benchmarks.

  • Problem

    Prior approaches largely target entropy collapse, leaving entropy explosion comparatively underexplored even though effective RLVR requires keeping entropy within a productive range.

    Source: §3.1 · p. 3

  • Method

    QAE replaces the mean with a group-wise K-quantile baseline that routes updates to rare successes on hard queries and remaining failures on easy ones.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    QAE stabilizes entropy and improves pass@1 across reasoning benchmarks while updating only a small subset of samples (∼20%).

    Source: §6 · p. 9, §8 · p. 10

  • Takeaways & Limitations

    Baseline design provides an entropy-control lever that composes with existing sequence- and token-level controls.

    Source: §6 · p. 10, §8 · p. 10

    The approach uses a fixed K; dynamic or automatic K selection and PPO integration remain future work.

    Source: §8 · p. 10

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM reasoning, but training often oscillates between {entropy collapse} and {entropy explosion}. We trace both hazards to the mean baseline used in value-free RL (e.g., GRPO and DAPO), which improperly penalizes negative-advantage samples under reward outliers. We propose {Quantile Advantage Estimation} (QAE), replacing the mean with a group-wise K-quantile baseline. QAE induces a response-level, two-regime gate: on hard queries (p <= 1 - K) it reinforces rare successes, while on easy queries (p > 1 - K) it targets remaining failures. Under first-order softmax updates, we prove {two-sided entropy safety}, giving lower and upper bounds on one-step entropy change that curb explosion and prevent collapse. Empirically, this minimal modification stabilizes entropy, sparsifies credit assignment (with tuned K, roughly 80% of responses receive zero advantage), and yields sustained pass@1 gains on Qwen3-8B/14B-Base across AIME 2024/2025 and AMC 2023. These results identify {baseline design} -- rather than token-level heuristics -- as the primary mechanism for scaling RLVR.

1 INTRODUCTION

RLVR must balance policy diversity with reward optimization: methods aimed at preventing entropy collapse can overlook harmful entropy explosion. The introduction proposes QAE’s quantile baseline as a way to regulate both hazards and sustain reasoning gains.

1 INTRODUCTION

  • Prior approaches largely target entropy collapse, while uncontrolled entropy growth can also make exploration inefficient and stall progress.The introduction describes collapse as suppressing exploration and limiting performance.Source: §1 · p. 1
  • On Qwen3-8B-Base, DAPO with Clip-Higher shows an early entropy spike followed by a performance plateau, whereas QAE stabilizes entropy and sustains pass@1 gains.The spike occurs at steps 10–80.Source: §1 · p. 1, Figure 1 · p. 2
  • QAE replaces the mean reward baseline with a group-wise K-quantile, regulating entropy and sparsifying credit assignment while sustaining performance gains.With tuned K, roughly 80% of responses receive zero advantage.Source: §1 · p. 2
  • QAE is traced to the mean baseline in value-free RL and routes updates to rare successes on hard queries and residual failures on easy ones.The paper reports pass@1 gains and solid pass@16 performance across AIME’24, AIME’25, and AMC’23.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries introduce PPO and its value-free variants, GRPO and DAPO, as the policy-optimization foundations for the paper. They describe PPO’s trust-region constraint and the group-based reward estimation used by GRPO and DAPO.

2 PRELIMINARIES

  • PPO constrains policy updates to a trust region around the previous policy and typically estimates advantages with a value network.The passage gives a clipping hyperparameter such as 0.2.Source: §2 · pp. 2–3
  • GRPO removes the value-network requirement by estimating each response’s advantage relative to reward statistics from a group of sampled responses.Responses receive binary correctness rewards.Source: §2 · p. 3
  • DAPO builds on GRPO by removing the KL penalty and using asymmetric clipping with token-level objective normalization.Its clipping range permits larger updates for advantageous actions.Source: §2 · p. 3
  • The dynamic sampling constraint requires each training batch to contain both positive and negative examples for a meaningful advantage signal and stable gradients.Source: §2 · p. 3

3 THE ENTROPY DILEMMA IN RL SCALING: FROM COLLAPSE TO EXPLOSION

The section frames entropy collapse and explosion as opposing hazards for RLVR, then examines how token-level controls can avert collapse yet still leave entropy growth and performance stalls. Its analysis attributes the imbalance to negative-advantage samples and motivates baseline design as the remedy.

3 THE ENTROPY DILEMMA IN RL SCALING: FROM COLLAPSE TO EXPLOSION

  • Entropy collapse narrows exploration, while entropy explosion swamps gradients with noise and destabilizes learning; prior work has focused comparatively less on explosion.The section argues that RLVR needs entropy within a productive, stable range because mitigating collapse can inadvertently induce explosion.Source: §3.1 · p. 3, §3 · p. 3

3.2 AN ANALYSIS OF ENTROPY EXPLOSION IN RLVR

  • Clip-Higher coincides with an early rise in anthropomorphic tokens and pass@1, but token frequency later returns toward baseline as performance plateaus.The figure compares DAPO without and with Clip-Higher; the plateau persists with Clip-Higher.Source: §3.2 · p. 4, Figure 2 · p. 4
  • High-entropy token usage shifts from diverse markers such as “wait” and “perhaps” to formulaic “so” and “let,” reflecting reduced exploratory diversity.This convergence aligns with the observed plateau.Source: §3.2 · p. 4, Figure 3 · p. 4
  • Negative-advantage samples dominate early entropy growth in DAPO, while positive-advantage samples remain comparatively stable.The analysis links this imbalance to early over-exploration and insufficient exploitation later.Source: §3.2 · p. 5, Figure 4 · p. 5
  • Changing DAPO’s high clipping threshold from 0.20 to 0.28 has only marginal effects, with performance peaking near 0.26 while the late-stage plateau persists.The analysis concludes that token-level clipping adjustments do not resolve the exploration–exploitation tension.Source: §3.2 · p. 5

TAKEAWAY

  • The section’s takeaway is that token-level controls can avert collapse yet induce performance-limiting explosion, which it attributes to the baseline’s handling of negative-advantage samples under reward outliers.It therefore frames the issue as baseline design rather than token-level hyperparameter tuning.Source: TAKEAWAY · p. 5

4 METHOD: QUANTILE-BASED ADVANTAGE ESTIMATION FOR ENTROPY REGULATION

QAE replaces the mean reward baseline with a group-wise K-quantile to make advantage assignment controllable. Its query-difficulty gate selects which responses to update, and under first-order softmax updates it provides two-sided entropy safety.

4 METHOD: QUANTILE-BASED ADVANTAGE ESTIMATION FOR ENTROPY REGULATION

  • The mean baseline can let reward outliers turn competent responses into negative-advantage examples, while QAE replaces it with a robust, controllable quantile baseline.K shifts the update focus between exploration and exploitation.Source: §4 · p. 5

4.1 FORMULATION AND INTUITION

  • For binary rewards, the K-quantile baseline switches at success rate p(q) = 1−K: hard queries update rare successes, while easy queries target remaining failures.On hard queries, incorrect responses receive zero advantage and rare correct responses positive advantage; on easy queries, correct responses receive zero and remaining failures negative advantage.Source: §4.1 · p. 6

4.2 GRADIENT ANALYSIS

  • QAE's GRPO-style objective selectively nullifies one discriminative term according to query difficulty, shifting updates toward signals from rare successes or residual failures.The objective assumes binary rewards, group size G ≥ 2, and a right-continuous empirical quantile.Source: §4.2 · pp. 6–7

4.3 THEORETICAL ANALYSIS: TWO-REGIME ENTROPY SAFETY

  • Under first-order softmax updates, entropy change increases with the baseline, and the K-quantile baseline selects the low- or high-success regime to provide two-sided entropy safety.The response-level baseline is zero when p(q) ≤ 1−K and one when p(q) > 1−K; token-level controls leave that baseline unchanged and cannot prevent explosion driven by negative-advantage samples.Source: §4.3 · p. 7

TAKEAWAY

  • The quantile gate's one-step entropy shift is minimal on low-success queries and maximal on high-success queries, preventing explosion and collapse, respectively.The guarantee is stated for first-order softmax updates.Source: TAKEAWAY · p. 7

5 EXPERIMENTS

Experiments report consistent pass@1 gains with comparable pass@16, alongside analyses of training dynamics, response sparsity, and clipping-dependent ablations.

5 EXPERIMENTS

  • Evaluations use zero-shot AIME’24, AIME’25, and AMC’23, sampling 32 completions per query at temperature 0.7.Source: §5 · p. 7

5.1 OVERALL PERFORMANCE ACROSS MODELS & RECIPES

5.2 TRAINING DYNAMICS & ENTROPY SAFETY

  • On AIME’24 Qwen3–8B, QAE sustains pass@1 improvement after DAPO stalls, keeps pass@16 similar, and suppresses negative-advantage entropy growth.Source: Figure 5 · p. 8, §5.2 · p. 8
  • Approximately 80% of sampled responses have zero advantage, concentrating updates on the informative minority.Source: Figure 5 · p. 8, §5.2 · p. 9

5.1 OVERALL PERFORMANCE ACROSS MODELS & RECIPES

  • QAE consistently adds gains when combined with token-level controls or sequence-level optimization, without changing their hyperparameters.Source: §5.1 · p. 8

5.3 ABLATIONS & COMPOSITION

  • POS-MASK and NEG-MASK isolate positive- and negative-update masking; NEG-MASK tracks QAE at ϵhigh=0.28, while POS-MASK dominates at ϵhigh=0.20.Source: Figure 6 · p. 9, §5.3 · p. 9

6 DISCUSSION

The discussion frames K as an entropy-guided exploration–exploitation control and positions group-wise quantile baseline design as an additional entropy lever.

6 DISCUSSION

  • K controls how many responses receive nonzero advantage; the suggested choice is 0.6 at low entropy and 0.4 at high entropy.Source: §6 · p. 9
  • QAE updates roughly 20% of samples and routes credit to successes on difficult queries or failures on easy ones.Source: §6 · p. 9
  • The group-level, data-adaptive quantile baseline adds an entropy-control lever that composes with existing techniques while preserving standard policy-gradient updates.Source: §6 · p. 10

7 RELATED WORK

The section situates DAPO and GRPO among foundational value-free RLVR methods for eliciting LLM reasoning. It identifies baseline-level, data-adaptive entropy control that preserves standard policy-gradient updates as a gap addressed by QAE.

7 RELATED WORK

  • GRPO and DAPO are foundational online, value-free baselines for RLVR reasoning research.The paper uses DAPO to investigate instability from dysregulated policy entropy.Source: §7 · p. 10
  • Prior work spans mechanistic analyses, objective-level regulation, and recipe or system-level exploration heuristics.Examples include analyses of high-entropy tokens, modified optimization targets, advantage shaping, Pass@k training, and agentic scaffolds.Source: §7 · p. 10
  • A data-adaptive, baseline-level entropy control that preserves standard policy-gradient updates remains missing.QAE addresses this gap with a quantile baseline and binary masking as a complementary drop-in lever.Source: §7 · p. 10

8 CONCLUSION

QAE replaces the mean baseline with a group-wise K-quantile to gate updates across two query regimes. The conclusion reports bounded one-step entropy change, stabilized entropy, sparser credit assignment, and improved pass@1 across reasoning benchmarks; future work includes adaptive K and PPO integration.

8 CONCLUSION

  • QAE stabilizes entropy, sparsifies credit assignment, and improves pass@1 across reasoning benchmarks while composing with sequence- and token-level controls.The conclusion describes its two-regime gate as amplifying rare successes and suppressing residual failures.Source: §8 · p. 10
  • Under first-order policy updates, QAE bounds one-step entropy change to curb both collapse and explosion.It replaces the mean baseline with a group-wise K-quantile.Source: §8 · p. 10
  • Future work includes dynamic or state-adaptive K and integrating quantile baselines into PPO.Suggested K signals include success rate, entropy, and gradient variance.Source: §8 · p. 10

A.1 PROOF OF PROPOSITION 4.1

The proof derives a quantile-regulated objective from standardized advantages under binary rewards and a right-continuous empirical quantile. Its query-difficulty gate masks one outcome type per regime and applies asymmetric weighting.

A.1 PROOF OF PROPOSITION 4.1

  • For binary rewards and group size G ≥ 2, the right-continuous empirical quantile yields the quantile-regulated objective up to an ε-dependent constant.The proof substitutes the standardized advantage into the GRPO-style objective and decomposes it by reward outcome.Source: §A.1 · p. 15
  • Only one outcome type has non-zero standardized advantage in each success-rate regime.Responses are masked asymmetrically according to whether p is above or below 1 − K.Source: §A.1 · p. 15
  • The quantile-regulated objective masks one discriminative term according to query difficulty and replaces the symmetric weight with asymmetric factors.For p ≤ 1 − K the scalar is positive; for p > 1 − K it is negative, with reweighting by (1 − p)/p.Source: §A.1 · pp. 15–16

A.2 PROOF OF PROPOSITION 4.2

The appendix proves that the K-quantile baseline selects entropy-extremizing choices in low- and high-success regimes. Experiments and parameter analysis report controlled token exploration and show that K tunes entropy, accuracy, and response length.

A.2 PROOF OF PROPOSITION 4.2

  • In the low-success regime, the K-quantile choice minimizes entropy change; in the high-success regime, it maximizes it.The proof identifies these extrema as explosion-proof and collapse-proof, respectively, for a non-uniform softmax policy under first-order updates.Source: §A.2 · p. 16

B.2 MORE EXPERIMENTS

  • Under QAE, high-entropy token counts stabilize while pass@1 continues improving in later training.The token snapshots at steps 20, 80, and 200 describe exploration activation followed by targeted refinement.Source: Figure 7 · p. 17, §B.2 · p. 17, Figure 8 · p. 18

B.1 IMPLEMENTATION DETAILS

  • The reported setup uses DAPO components and recommended clip-higher settings, with no KL divergence or entropy losses.Training uses a global batch size of 512 and a fixed learning rate of 10−6; additional experiments include Qwen3-32B and Qwen3-8B scaling and a GSPO cold start.Source: §B.1 · p. 17

B.3 QUANTILE PARAMETER ANALYSIS

  • Large K drives higher entropy, longer responses, and volatile training, while small K yields low-entropy, exploration-poor training with limited accuracy gains.The reported K = 0.4 configuration avoids the high-entropy instability of K = 0.8 while maintaining stochasticity and sustained accuracy improvements.Source: §B.3 · p. 18, Figure 9 · p. 18

B.4 ANALYSIS OF TRAINING DYNAMICS ON 8B AND 14B MODELS

Across Qwen3-8B-Base and Qwen3-14B-Base, QAE stabilizes entropy, bounds response length, and sustains smoother accuracy gains than DAPO. The training dynamics show QAE moderating the mean baseline’s entropy volatility at both scales.

B.4 ANALYSIS OF TRAINING DYNAMICS ON 8B AND 14B MODELS

  • Across Qwen3-8B-Base and Qwen3-14B-Base, QAE stabilizes entropy, bounds response length, and sustains smoother accuracy gains than DAPO.The curves track entropy, AIME24@32 accuracy, and response length for each model size.Source: §B.4 · pp. 18–19, Figure 10 · p. 19, Figure 11 · p. 19
  • On Qwen3-8B, DAPO’s entropy surge around step 100 accompanies response-length divergence and an accuracy plateau, while QAE maintains productive entropy and later accuracy gains.Source: §B.4 · p. 18, Figure 10 · p. 19
  • On Qwen3-14B, DAPO entropy remains higher and more volatile than with QAE, whose training moderates entropy and response length while producing smoother, more monotonic accuracy.Source: §B.4 · p. 19, Figure 11 · p. 19

Related papers

Loading 2509.22611v2…