PaperlayerView exact revision on arXiv

Source-linked AI summary

GRPODropout: Less is More for Online Reinforcement Learning Rollouts

Hexuan Deng, Zihao Yan, Xuebo Liu, Shuo Nie, Yue Wang, Chen Wang, Zhaohua Zhang, Tianwen Jiang, Qiuyong Xiao, Jihong Zhang, Min Zhang

arXiv:2610.11854v1cs.LGcs.AIcs.CL

TL;DR

Online RL can suffer entropy collapse that reduces sampling diversity and weakens exploration, while guidance for selecting complete rollouts remains limited. GRPODropout selectively deletes rollouts and recenters retained advantages before the standard update. Across three models and ten benchmarks, it improves average score over original GRPO while using fewer retained rollouts in policy updates.

  • Problem

    Online RL can suffer entropy collapse that reduces sampling diversity and weakens exploration, and existing work offers limited guidance for selecting complete rollouts.

    Source: §1 · p. 1

  • Method

    GRPODropout uses rollout-level analysis to selectively remove high-probability positive-advantage rollouts, recenter retained advantages, and update with the original RL loss.

    Source: §1 · p. 1, §3.2 · p. 4

  • Results

    Across three models and ten benchmarks, average score improves over original GRPO by 1.55 percentage points while updates use 12.1% fewer retained rollouts on average.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support the insight that selectively removing some rollouts can improve performance over using all sampled rollouts in policy updates.

    Source: §5 · p. 9

    GRPODropout does not outperform the strongest baseline at every model scale.

    Source: §5 · p. 10

Abstract

from arXiv · show

Reinforcement learning (RL) methods such as GRPO substantially improve large language model reasoning but often suffer from policy entropy collapse: the loss of sampling diversity weakens exploration and limits further improvement. Existing methods address this issue either through algorithm-level interventions, such as reward modification and entropy/KL regularization, or through token-level reweighting. We investigate a complementary perspective: entropy collapse can also be mitigated by changing which generated rollouts contribute to policy updates. Under the same sampling budget, not all rollouts contribute positively to an update, and selectively excluding some can improve learning. To address this, we propose GRPODropout: before the standard update, we use a simple strategy that selectively removes a small number of high-probability positive-advantage rollouts and recenters the retained advantages. To motivate this design, we develop a rollout-level theoretical analysis that guides method design and threshold selection. The method changes only rollout usage, and adds negligible computational overhead. Experiments show higher accuracy than original GRPO and higher actor entropy while using fewer rollout samples for updates, illustrating "less is more." This work provides insight into RL rollout usage: removing some rollouts can improve performance. Code is available at https://github.com/hexuandeng/GRPODropout/.

1 INTRODUCTION

The paper frames entropy collapse as a challenge to exploration in online RL and argues that rollout selection offers a complementary way to address it. GRPODropout removes selected rollouts and recenters retained advantages; experiments report improved average scores and higher actor entropy with fewer update rollouts.

1 INTRODUCTION

  • Online RL can suffer policy entropy collapse, reducing sampling diversity and weakening exploration of alternative reasoning paths.The reduced diversity can limit opportunities to discover new solutions and improve further.Source: §1 · p. 1
  • Existing approaches modify algorithm signals or rules, or reweight, branch, or calibrate token-level data; theoretical and methodological guidance for selecting complete rollouts remains limited.The paper proposes changing rollout usage alone as another way to mitigate entropy collapse.Source: §1 · p. 1
  • Across three models and ten benchmarks, average score improves over original GRPO by 1.55 percentage points while policy updates use 12.1% fewer retained rollouts on average.At training’s end, training and held-out actor entropy are 446.1% and 256.0% higher than original GRPO, respectively.Source: §1 · p. 2
  • The paper contributes rollout-level analysis, a filtering-and-recentering algorithm, and evaluations of accuracy and entropy across three model sizes.The analysis characterizes which rollouts can reduce policy entropy and informs the data-selection method.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses entropy collapse through token-level interventions and changes to training signals or optimization. This paper positions rollout selection and advantage recentering as a data-level complement that preserves the RL objective form.

2 RELATED WORK

  • Token-level methods mask or reweight selected tokens, branch at critical positions, or construct token-level advantages, while leaving open how to select complete rollouts.The paper identifies rollout-level entropy analysis and selection as its response to that gap.Source: §2 · p. 2
  • Entropy-aware algorithms alter rewards, advantages, clipping, temperature, or regularization to mitigate entropy collapse.The cited approaches include reweighting updates and modifying optimization rules directly.Source: §2 · p. 2
  • The paper proposes data-level rollout selection and advantage recentering as a complementary intervention that preserves the RL objective form.This contrasts with prior approaches that mitigate entropy collapse through algorithmic changes.Source: §2 · p. 3

3 METHOD

The method analyzes entropy effects at the rollout level, then filters selected positive-advantage rollouts and recenters retained advantages before applying the original RL loss. Its design aims to reduce entropy-decreasing updates while limiting changes to the existing training process.

3 METHOD

  • GRPODropout selects rollouts and recenters their advantages, while experiments evaluate the complete method; Section 3 develops theory for each design choice.The section proceeds from preliminaries through entropy analysis, positive-only deletion, and recentering.Source: §3 · p. 3

3.1 PRELIMINARIES

  • A rollout is a complete output sequence sampled for a prompt under the old policy, and the method seeks task return while mitigating entropy collapse and preserving exploration.The policy objective uses rollout rewards and advantages, with reference-policy regularization in the sampled surrogate.Source: §3.1 · p. 3

3.2 ROLLOUT-LEVEL ENTROPY EFFECT AND CANDIDATE DELETION

  • Under the local update analysis, high-probability positive-advantage and low-probability negative-advantage rollouts can decrease entropy; positive association between advantages and log-probabilities also predicts first-order entropy decrease.The analysis characterizes each rollout’s first-order contribution and the group’s entropy change.Source: §3.2 · pp. 3–4
  • Filtering greedily removes rollouts with the most negative estimated entropy contributions until the retained set meets the nondecreasing-entropy criterion.The method estimates entropy using the sampled group’s mean surprisal and length-normalizes log-probabilities to avoid direct length bias.Source: §3.2 · p. 4

3.3 DELETING ONLY POSITIVE-ADVANTAGE ROLLOUTS

  • In a shared-learning-rate independent-logit comparison, a high-probability positive rollout contributes more to first-order entropy reduction than a low-probability negative rollout when their proximal-update contributions are equal.The method therefore restores excluded nonpositive-advantage rollouts and preferentially filters positive-advantage candidates.Source: §3.3 · p. 5

3.4 RECENTERING AFTER DELETION AND THE FULL PROCEDURE

  • Deletion can shift all zero-advantage paths in the same direction, regardless of their quality, with the shift determined by the retained advantages’ old-policy-weighted mean.The analysis links this normalization offset to potentially unwarranted probability changes and training instability.Source: §3.4 · pp. 5–6
  • Probability-weighted recentering is designed to cancel the first-order normalization shift, and the full procedure applies the original RL loss to retained rollouts.The method reuses rewards and log-probabilities, requires no value network, and adds negligible computational overhead.Source: §3.4 · p. 6

4 EXPERIMENTS

Across three model sizes and ten benchmarks, GRPODropout improves average accuracy over GRPO while using fewer retained rollouts for updates. Ablations favor deleting positive-advantage rollouts with recentering, and the method ranks first among six systems on the reported accuracy-and-diversity evaluation.

4 EXPERIMENTS

  • Experiments train three model sizes on matched mathematics prompts and evaluate ten benchmarks covering mathematical reasoning and general QA.AMC23, AIME24, AIME25, and HMMT25 use Mean@32; other benchmarks use Mean@1.Source: §4 · p. 6

4.1 MAIN RESULTS

  • GRPODropout improves average accuracy over GRPO by 1.55 percentage points and exceeds KL-Cov by 0.11 points on average across three model sizes.Table 1 reports the main results across three backbones, averaging ten benchmarks.Source: §4.1 · pp. 6–7, Table 1 · p. 7
  • At the final Qwen3-4B checkpoint, GRPODropout has 446.1% higher entropy on DAPO-17K and 256.0% higher entropy on MMLU-Pro, retaining 7.03 update rollouts per question versus GRPO’s eight.Across ten recorded checkpoints, 7.03 rollouts is 12.1% fewer than the eight-rollout baseline; Figure 2 tracks entropy and rollout usage.Source: §4.1 · p. 7, Figure 2 · p. 7

4.2 ABLATION STUDIES

  • Deleting only positive-advantage rollouts achieves higher entropy and accuracy than deleting only negative candidates; deleting both raises entropy most but slightly lowers accuracy.Table 2 ablates deletion patterns, recentering, and probability weighting on Qwen3-1.7B and Qwen3-4B.Source: Table 2 · p. 8, §4.2 · p. 8
  • Without recentering, positive-rollout deletion causes severe performance degradation and training collapse; probability weighting adds 0.03 points on Qwen3-4B and 2.82 points on Qwen3-1.7B.The latter values compare probability-weighted with equal-weight recentering.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 MORE PRECISE EXPLORATION AND MORE DIVERSE REASONING PATHS

  • GRPODropout ranks first among six systems (overall mean rank 1.63) and is best or second-best on all eight metrics, jointly preserving accuracy, entropy, and reasoning-path diversity.Table 3 reports Qwen3-4B solution coverage, wrong-answer dispersion, and reasoning diversity.Source: §4.3 · p. 9, Table 3 · p. 9

5 CONCLUSION

GRPODropout filters high-probability positive-advantage rollouts and recenters retained advantages, leaving the original RL loss unchanged. Experiments show it outperforms original GRPO with fewer retained rollouts, though it does not beat the strongest baseline at every scale.

5 CONCLUSION

  • Across three model scales and ten benchmarks, GRPODropout outperforms original GRPO while using fewer retained rollouts in policy updates.It does not outperform the strongest baseline at every model scale.Source: §5 · pp. 9–10
  • GRPODropout selectively removes high-probability positive-advantage rollouts and recenters retained advantages to stabilize training while leaving the original RL loss unchanged.The authors report that removing some such rollouts mitigates entropy collapse.Source: §5 · p. 9
  • The rollout-level theory motivates the deletion strategy but cannot fully explain the method; the authors validate it through comparisons with original GRPO and ablations.The theory is described as an initial starting point for method design.Source: §5 · pp. 9–10
  • The method relies on within-prompt differences across multiple rollouts, so it does not directly apply to single-rollout training, and its effectiveness with very small groups remains unevaluated.Source: §5 · p. 10

A.1 PROOFS OF PROPOSITION 1 AND PROPOSITION 2

The appendix derives rollout-level entropy contributions and a group-level entropy derivative for a fixed sampled group under a local KL-proximal update. It then gives a Monte Carlo filtering criterion and clarifies how implementation proxies and policy-gradient correspondences are scoped.

A.1 PROOFS OF PROPOSITION 1 AND PROPOSITION 2

  • The local model updates sequence distributions with a KL penalty relative to the old policy, holding a finite sampled group and its finite advantages fixed around η = 0.The analysis assumes finite old-policy entropy and uses rollout advantages without length normalization; its regularity argument is conditional on the fixed group.Source: §A.1 · p. 15
  • The sequence-space sampled score assigns direct update signal only to sampled sequences, with repeated samples contributing additively; its inverse-probability factor is needed for mathematical correspondence but is not computed by the algorithm.Source: §A.1 · p. 15
  • A high-probability positive-advantage rollout or low-probability negative-advantage rollout decreases entropy in the local model, while the reverse combinations increase it.The contribution includes normalization-induced changes to other sequences and is zero when the advantage is zero or surprisal equals entropy.Source: §A.1 · p. 16
  • A negative empirical covariance between advantages and surprisal means the fixed-group KL-proximal update decreases entropy at the starting point; zero covariance leaves finite-step change undetermined.This derivative is not an entropy guarantee for a neural-network update.Source: §A.1 · p. 17
  • The Monte Carlo filtering constraint uses the original group’s mean surprisal; removing the most negative contributions until the retained sum is nonnegative yields the fewest removals and retains the most rollouts.Recomputing the mean on the retained subset generally changes the criterion.Source: §A.1 · p. 17
  • Implementation uses length-normalized surprisal for filtering and recentering, with the original-group mean as the filtering baseline.The recentering weights are computed by a softmax over retained rollouts.Source: §A.1 · p. 17
  • These normalized quantities are implementation proxies, not complete-sequence entropy estimates; the theoretical results retain full sequence probabilities.Responses with equal mean token log-probability receive equal recentering weights regardless of length.Source: §A.1 · p. 17
  • At the starting point, outcome-supervised GRPO and GSPO correspond to the rollout-level reward gradient with length-normalized advantages under the stated conditions, but their clipped objectives and later gradients need not match.The correspondence ignores reference KL and assumes clipping is inactive.Source: §A.1 · pp. 17–18

A.2 PROOF OF PROPOSITION 3

Proposition 3 compares rollout-level entropy contributions under a local independent-logit model. Its conclusions motivate the positive-advantage filter but are limited to stated assumptions and do not establish behavior for shared-parameter neural-network updates.

A.2 PROOF OF PROPOSITION 3

  • Under the stated equal-magnitude condition, both a high-probability positive rollout and a low-probability negative rollout decrease entropy, with the comparison conditional on that pair.The assumptions include A+ > 0 > A− and s+ < H(p) < s−.Source: §A.2 · p. 19
  • The independent-logit update scales a global step size by η/pi, whereas the shared-gradient update uses α and retains the probability factor in its entropy effect.The inverse-probability factor cancels the pi in the entropy gradient; this distinguishes update rules, not whether they have a step size.Source: §A.2 · p. 19
  • The comparison is first-order and assumes independent logits, zero-sum whole-group advantages, and the stated equal-magnitude condition; it does not guarantee entropy preservation under shared-parameter AdamW updates.It also does not establish positive-sample dominance for arbitrary finite steps or actual neural-network updates.Source: §A.2 · p. 19

A.3 PROOF OF PROPOSITION 4

This proof analyzes deletion-induced changes to retained advantages and zero-advantage sequences. Probability-weighted recentering imposes a weighted zero-mean condition, while the resulting claims remain local to the specified update model.

A.3 PROOF OF PROPOSITION 4

  • Deleting positive-advantage rollouts from a zero-sum group leaves the retained advantages with a negative sum, and retention depends on rollout content rather than preserving an i.i.d. sample.This motivates reconsidering the advantage baseline after selection, but does not determine the probability weights.Source: §A.3 · p. 19
  • For zero-advantage sequences, the retained-score update applies a shared normalization factor, preserving their probability ratios while shifting their log probabilities equally.These sequences can include sampled paths with zero or canceling retained signals, not only unsampled paths.Source: §A.3 · p. 20
  • The zero-mean result does not imply finite-step invariance, and the entropy direction cannot be inferred from the retained-score mean alone.The global entropy expression remains a covariance under the stated analysis.Source: §A.3 · p. 20
  • Equation (9) imposes a probability-weighted zero mean on retained advantages, eliminating first-order normalization drift for zero-advantage sequences in the stated KL-proximal model.The weights come from the old-policy expectation and calibrate the retained score rather than reconstructing the original sampling distribution.Source: §A.3 · pp. 20–21

B.1 EXPERIMENTAL SETUP

Experiments train three Qwen3 model sizes on DAPO-17K and compare GRPODropout with original GRPO and representative entropy-collapse interventions. Evaluation spans ten mathematics and general-question-answering benchmarks.

B.1 EXPERIMENTAL SETUP

  • Training uses Qwen3-1.7B, Qwen3-4B, and Qwen3-8B on DAPO-17K for one epoch, with eight rollouts per prompt and selected rollouts removed before GRPODropout updates.Original GRPO uses all eight rollouts; the baseline adds neither an entropy bonus nor reference-policy KL regularization.Source: §B.1 · p. 21
  • Comparisons include original GRPO, token-level Beyond8020 and Clip-Cov, and objective or update-rule methods KL-Cov and DAPO.GRPODropout retains the original RL objective form while applying it to selected rollouts with recentered advantages.Source: §B.1 · p. 21
  • Evaluation covers ten benchmarks across mathematical reasoning and general question answering, reporting Mean@1 or Mean@32 and an unweighted mean accuracy across benchmarks.Mean@32 is used for AMC23, AIME24, AIME25, and HMMT25; the other six benchmarks report Mean@1.Source: §B.1 · p. 21

B.2 DIVERSITY-ANALYSIS PROTOCOL

The diversity analysis measures path diversity and actor entropy across four small mathematics benchmarks. It also summarizes sampled reasoning paths, distinguishing the derivation producing each answer from abandoned attempts.

B.2 DIVERSITY-ANALYSIS PROTOCOL

  • The protocol covers 130 unique questions across AMC23 (40), AIME24 (30), AIME25 (30), and HMMT25 (30), with 32 sampled responses per question; Pass@32 and final-checkpoint token-mean actor entropy are averaged equally across benchmarks.Source: §B.2 · p. 21

GLM-5.2 SUMMARIZER PROMPT

  • For each rollout, GLM-5.2 is prompted to summarize the actual answer-producing reasoning path separately from abandoned ideas; summary distance and Vendi use all 32 summaries.The summaries are embedded with bge-m3; summary distance is mean pairwise cosine distance, while Vendi is computed from the normalized Gram-matrix eigenvalues.Source: §B.2 · p. 22, GLM-5.2 SUMMARIZER PROMPT · p. 22
Loading 2610.11854v1…